27350 Commits

Author SHA1 Message Date
sundb
362f2a84bd
Improve overflow check of expire time (#8519)
When milliseconds == LLONG_MAX / 1000, *1000 will not overflow.
2021-02-22 08:45:26 +02:00
Yossi Gottlieb
f8fed5ffd1 Fix allowed length for REPLCONF ip-address. (#8517)
Originally this was limited to IPv6 address length, but effectively it
has been used for host names and now that Sentinel accepts that as well
we need to be able to store full hostnames.

Fixes #8507
2021-02-21 11:22:36 +02:00
Yossi Gottlieb
d828f90c26
Fix allowed length for REPLCONF ip-address. (#8517)
Originally this was limited to IPv6 address length, but effectively it
has been used for host names and now that Sentinel accepts that as well
we need to be able to store full hostnames.

Fixes #8507
2021-02-21 11:22:36 +02:00
Huang Zw
4f4cd78fed Client tracking tracking-redir-broken push len is 2 not 3 (#8456)
When redis responds with tracking-redir-broken push message (RESP3),
it was responding with a broken protocol: an array of 3 elements, but only
pushes 2 elements.

Some bugs in the test make this pass. Read the push reply
will consume an extra reply, because the reply length is 3, but there
are only two elements, so the next reply will be treated as third
element. So the test is corrected too.

Other changes:
* checkPrefixCollisionsOrReply success should return 1 instead of -1,
  this bug didn't have any implications.
* improve client tracking tests to validate more of the response it reads.
2021-02-21 09:34:46 +02:00
Huang Zw
f687ac0c32
Client tracking tracking-redir-broken push len is 2 not 3 (#8456)
When redis responds with tracking-redir-broken push message (RESP3),
it was responding with a broken protocol: an array of 3 elements, but only
pushes 2 elements.

Some bugs in the test make this pass. Read the push reply
will consume an extra reply, because the reply length is 3, but there
are only two elements, so the next reply will be treated as third
element. So the test is corrected too.

Other changes:
* checkPrefixCollisionsOrReply success should return 1 instead of -1,
  this bug didn't have any implications.
* improve client tracking tests to validate more of the response it reads.
2021-02-21 09:34:46 +02:00
Gnanesh
1621966712 EXPIRE, EXPIREAT, SETEX, GETEX: Return error when expire time overflows (#8287)
Respond with error if expire time overflows from positive to negative of vice versa.

* `SETEX`, `SET EX`, `GETEX` etc would have already error on negative value,
but now they would also error on overflows (i.e. when the input was positive but
after the manipulation it becomes negative, which would have passed before)
* `EXPIRE` and `EXPIREAT` was ok taking negative values (would implicitly delete
the key), we keep that, but we do error if the user provided a value that changes
sign when manipulated (except the case of changing sign when `basetime` is added)

Signed-off-by: Gnanesh <gnaneshkunal@outlook.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-02-21 09:09:54 +02:00
Gnanesh
0772098b1b
EXPIRE, EXPIREAT, SETEX, GETEX: Return error when expire time overflows (#8287)
Respond with error if expire time overflows from positive to negative of vice versa.

* `SETEX`, `SET EX`, `GETEX` etc would have already error on negative value,
but now they would also error on overflows (i.e. when the input was positive but
after the manipulation it becomes negative, which would have passed before)
* `EXPIRE` and `EXPIREAT` was ok taking negative values (would implicitly delete
the key), we keep that, but we do error if the user provided a value that changes
sign when manipulated (except the case of changing sign when `basetime` is added)

Signed-off-by: Gnanesh <gnaneshkunal@outlook.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-02-21 09:09:54 +02:00
Jim Brunner
d8b623ddde dict: pause rehash, minor readability refactor (#8515)
The `dict` field `iterators` is misleading and incorrect. 
This variable is used for 1 purpose - to pause rehashing.

The current `iterators` field doesn't actually count "iterators".
It counts "safe iterators".  But - it doesn't actually count safe iterators
either.  For one, it's only incremented once the safe iterator begins to
iterate, not when it's created.  It's also incremented in `dictScan` to
prevent rehashing (and commented to make it clear why `iterators` is
being incremented during a scan).

This update renames the field as `pauserehash` and creates 2 helper
macros `dictPauseRehashing(d)` and `dictResumeRehashing(d)`
2021-02-20 12:56:30 +02:00
Jim Brunner
06966d2a0e
dict: pause rehash, minor readability refactor (#8515)
The `dict` field `iterators` is misleading and incorrect. 
This variable is used for 1 purpose - to pause rehashing.

The current `iterators` field doesn't actually count "iterators".
It counts "safe iterators".  But - it doesn't actually count safe iterators
either.  For one, it's only incremented once the safe iterator begins to
iterate, not when it's created.  It's also incremented in `dictScan` to
prevent rehashing (and commented to make it clear why `iterators` is
being incremented during a scan).

This update renames the field as `pauserehash` and creates 2 helper
macros `dictPauseRehashing(d)` and `dictResumeRehashing(d)`
2021-02-20 12:56:30 +02:00
sundb
ad0e2455f7 Fix timing error oom-score-adj test (#8513)
fixes timing issue, fork didn't always get to set the oom score before the test verified it.
2021-02-19 13:01:25 +02:00
sundb
46346e9e3a
Fix timing error oom-score-adj test (#8513)
fixes timing issue, fork didn't always get to set the oom score before the test verified it.
2021-02-19 13:01:25 +02:00
John Sully
87fc018415 Implement maxstorage config
Former-commit-id: 79e07859dfec14edf2a39646013e1a82db033d4f
2021-02-19 05:17:41 +00:00
John Sully
7fb41ba1bd Implement maxstorage config
Former-commit-id: 79e07859dfec14edf2a39646013e1a82db033d4f
2021-02-19 05:17:41 +00:00
Harkrishn Patro
b7ee65f3c4 Remove redundant pubsub list to store the patterns. (#8472)
Remove redundant pubsub list to store the patterns.
2021-02-17 14:13:50 -08:00
Harkrishn Patro
303465af35
Remove redundant pubsub list to store the patterns. (#8472)
Remove redundant pubsub list to store the patterns.
2021-02-17 14:13:50 -08:00
Oran Agra
f6d1af1c07 solve valgrind warning in child_info (#8505)
Valgrind warns about `write` accessing uninitialized memory, which was the struct padding.
2021-02-17 12:30:29 +02:00
Oran Agra
71ab81ec69
solve valgrind warning in child_info (#8505)
Valgrind warns about `write` accessing uninitialized memory, which was the struct padding.
2021-02-17 12:30:29 +02:00
John Sully
3ac42ec80c Disable async rehash with single threads, it causes slowdowns as the rehash never completes
Former-commit-id: 5d08dbdf76c0fd1e0cfcf86b97ef3e656f0e4f5d
2021-02-16 21:07:43 +00:00
John Sully
dea064712b Disable async rehash with single threads, it causes slowdowns as the rehash never completes
Former-commit-id: 5d08dbdf76c0fd1e0cfcf86b97ef3e656f0e4f5d
2021-02-16 21:07:43 +00:00
yihuang
ae4d2fe1b3 Optimize listpack for stream usage to avoid repeated reallocs (#6281)
Avoid repeated reallocs growing the listpack while entries are being added.
This is done by pre-allocating the listpack to near maximum size, and using
malloc_size to check if it needs realloc or not.
When the listpack reaches the maximum number of entries, we shrink it to fit it's used size.

Co-authored-by: Viktor Söderqvist <viktor@zuiderkwast.se>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-02-16 16:17:38 +02:00
yihuang
aab479f8cf
Optimize listpack for stream usage to avoid repeated reallocs (#6281)
Avoid repeated reallocs growing the listpack while entries are being added.
This is done by pre-allocating the listpack to near maximum size, and using
malloc_size to check if it needs realloc or not.
When the listpack reaches the maximum number of entries, we shrink it to fit it's used size.

Co-authored-by: Viktor Söderqvist <viktor@zuiderkwast.se>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-02-16 16:17:38 +02:00
uriyage
a8d6c6493a Adds INFO fields to track fork child progress (#8414)
* Adding current_save_keys_total and current_save_keys_processed info fields.
  Present in replication, BGSAVE and AOFRW.
* Changing RM_SendChildCOWInfo() to RM_SendChildHeartbeat(double progress)
* Adding new info field current_fork_perc. Present in Replication, BGSAVE, AOFRW,
  and module forks.
2021-02-16 16:06:51 +02:00
uriyage
fd052d2a86
Adds INFO fields to track fork child progress (#8414)
* Adding current_save_keys_total and current_save_keys_processed info fields.
  Present in replication, BGSAVE and AOFRW.
* Changing RM_SendChildCOWInfo() to RM_SendChildHeartbeat(double progress)
* Adding new info field current_fork_perc. Present in Replication, BGSAVE, AOFRW,
  and module forks.
2021-02-16 16:06:51 +02:00
Viktor Söderqvist
685a3e1fa3 Add ziplistReplace, in-place optimized for elements of same size
Avoids memmove and reallocs when replacing a ziplist element of the
same encoded size as the new value.

Affects HSET, HINRBY, HINCRBYFLOAT (via hashTypeSet) and LSET (via
quicklistReplaceAtIndex).
2021-02-16 13:01:14 +02:00
Viktor Söderqvist
acb32d472d Add ziplistReplace, in-place optimized for elements of same size
Avoids memmove and reallocs when replacing a ziplist element of the
same encoded size as the new value.

Affects HSET, HINRBY, HINCRBYFLOAT (via hashTypeSet) and LSET (via
quicklistReplaceAtIndex).
2021-02-16 13:01:14 +02:00
Viktor Söderqvist
ce1d737772 Use stack for decoding integer-encoded values in list push
Less heap allocations when commands like LMOVE push integer values.
2021-02-16 13:01:14 +02:00
Viktor Söderqvist
683e530cf3 Use stack for decoding integer-encoded values in list push
Less heap allocations when commands like LMOVE push integer values.
2021-02-16 13:01:14 +02:00
Viktor Söderqvist
ea97f6aba7 Avoid useless copying in LINDEX command for reply 2021-02-16 13:01:14 +02:00
Viktor Söderqvist
f521498b43 Avoid useless copying in LINDEX command for reply 2021-02-16 13:01:14 +02:00
Oran Agra
91bc7597c7 minor test suite cleanup, revive old test (#8497)
There are two tests in other.tcl that were dependant of the sha1 package
import which meant that they didn't usually run.
The reason it was like that was that prior to the creation of DEBUG
DIGEST, the test suite used to have an equivalent function, but that's
no longer the case and this dependency isn't needed.

The other change is to revert config changes done by the test before the
test suite continues. can be useful if using `--host` to run multiple
units against the same server
2021-02-15 17:20:03 +02:00
Oran Agra
fb3457d157
minor test suite cleanup, revive old test (#8497)
There are two tests in other.tcl that were dependant of the sha1 package
import which meant that they didn't usually run.
The reason it was like that was that prior to the creation of DEBUG
DIGEST, the test suite used to have an equivalent function, but that's
no longer the case and this dependency isn't needed.

The other change is to revert config changes done by the test before the
test suite continues. can be useful if using `--host` to run multiple
units against the same server
2021-02-15 17:20:03 +02:00
Yossi Gottlieb
11dce0a0cb Fix incorrect shared ping length. (#8498) 2021-02-15 17:09:33 +02:00
Yossi Gottlieb
b1929bb2b6
Fix incorrect shared ping length. (#8498) 2021-02-15 17:09:33 +02:00
Yossi Gottlieb
064d807db3 Escape unsafe field name characters in INFO. (#8492)
Fixes #8489
2021-02-15 17:08:53 +02:00
Yossi Gottlieb
141ac8df59
Escape unsafe field name characters in INFO. (#8492)
Fixes #8489
2021-02-15 17:08:53 +02:00
Oran Agra
dffb64f5cc solve race in replication-2 test - again (#8491)
this should make it timing independent and also faster in most cases
2021-02-15 12:50:23 +02:00
Oran Agra
30775bc3e3
solve race in replication-2 test - again (#8491)
this should make it timing independent and also faster in most cases
2021-02-15 12:50:23 +02:00
Viktor Söderqvist
8a002f470a Modules: In RM_HashSet, add COUNT_ALL flag and set errno (#8446)
The added flag affects the return value of RM_HashSet() to include
the number of inserted fields, in addition to updated and deleted
fields.

errno is set on errors, tests are added and documentation updated.
2021-02-15 11:40:05 +02:00
Viktor Söderqvist
0bc8c9c8f9
Modules: In RM_HashSet, add COUNT_ALL flag and set errno (#8446)
The added flag affects the return value of RM_HashSet() to include
the number of inserted fields, in addition to updated and deleted
fields.

errno is set on errors, tests are added and documentation updated.
2021-02-15 11:40:05 +02:00
filipe oliveira
05d2b269a9 redis-benchmark: Fix broken protocol when used with -a or --dbnum (#8486)
Fix the pointers to the slot hash tags in the case of prefixed commands usage
i.e. AUTH / SELECT

It adjusts the pointers to the slot hash tags in the case of prefixed commands
usage as soon as we get the 1st reply (same like we already did for the random
strings within the command )
2021-02-14 14:42:41 +02:00
filipe oliveira
efccd6353b
redis-benchmark: Fix broken protocol when used with -a or --dbnum (#8486)
Fix the pointers to the slot hash tags in the case of prefixed commands usage
i.e. AUTH / SELECT

It adjusts the pointers to the slot hash tags in the case of prefixed commands
usage as soon as we get the 1st reply (same like we already did for the random
strings within the command )
2021-02-14 14:42:41 +02:00
John Sully
c43bb8e8b5 Fix deadlock in storage prefetch
Former-commit-id: 9e6162de08248f7726b97b73aee2f23808ff4e7b
2021-02-12 19:38:57 +00:00
John Sully
f3853b6485 Fix deadlock in storage prefetch
Former-commit-id: 9e6162de08248f7726b97b73aee2f23808ff4e7b
2021-02-12 19:38:57 +00:00
Yossi Gottlieb
f74fd77280 Fix errors with sentinel leaked fds test. (#8482)
* Don't run test script on non-Linux.
* Verify that reported fds do indeed exist also in parent, to avoid
  false negatives on some systems (namely CentOS).

Co-authored-by: Andy Pan <panjf2000@gmail.com>
2021-02-11 15:25:01 +02:00
Yossi Gottlieb
8c42d1257f
Fix errors with sentinel leaked fds test. (#8482)
* Don't run test script on non-Linux.
* Verify that reported fds do indeed exist also in parent, to avoid
  false negatives on some systems (namely CentOS).

Co-authored-by: Andy Pan <panjf2000@gmail.com>
2021-02-11 15:25:01 +02:00
Yossi Gottlieb
91ab2fbda6 Fix duplicate replicas issue. (#8481)
We need to store replicas referenced by their announced address (IP or
address). Before that, if hostnames were used and the IP address
changed, duplicate entries would have been created.
2021-02-11 11:50:47 +02:00
Yossi Gottlieb
94bc26e652
Fix duplicate replicas issue. (#8481)
We need to store replicas referenced by their announced address (IP or
address). Before that, if hostnames were used and the IP address
changed, duplicate entries would have been created.
2021-02-11 11:50:47 +02:00
John Sully
ef471e6e53 Enable prefetch with storage providers
Former-commit-id: 21f9c52c446ad1b460a340c3d873839b2355728f
2021-02-11 00:50:21 +00:00
John Sully
bb767ad61d Enable prefetch with storage providers
Former-commit-id: 21f9c52c446ad1b460a340c3d873839b2355728f
2021-02-11 00:50:21 +00:00
Yossi Gottlieb
0e603e228c Fix Sentinel configuration rewrite. (#8480)
The `resolve-hostnames` and `announce-hostnames` parameters were not
specified correctly according to the new convention introduced by
ec447de18.
2021-02-10 18:38:10 +02:00