21038 Commits

Author SHA1 Message Date
zhaozhao.zz
f0f4ef4423 config: proto-max-bulk-len must be 1mb or greater
(cherry picked from commit 46323cec56793c66273005b72f1f29146c108840)
2020-09-01 09:27:58 +03:00
zhaozhao.zz
d3831559cf config: proto-max-bulk-len must be 1mb or greater
(cherry picked from commit bd4b33d7a2b00fddcaaccee9705d21b92d97311e)
2020-09-01 09:27:58 +03:00
zhaozhao.zz
0d78615532 using proto-max-bulk-len in checkStringLength for SETRANGE and APPEND
(cherry picked from commit 912b48e4b9d9a0738ab3010c4f30e47a3beec3fc)
2020-09-01 09:27:58 +03:00
zhaozhao.zz
44ef19743e using proto-max-bulk-len in checkStringLength for SETRANGE and APPEND
(cherry picked from commit 2e69bfe44d154d6739da48ff87f1c66045f68e93)
2020-09-01 09:27:58 +03:00
WuYunlong
5240ad73ef see #7250, fix signature of RedisModule_DeauthenticateAndCloseClient (#7645)
In redismodule.h, RedisModule_DeauthenticateAndCloseClient returns void
`void REDISMODULE_API_FUNC(RedisModule_DeauthenticateAndCloseClient)(RedisModuleCtx *ctx, uint64_t client_id);`
But in module.c, RM_DeauthenticateAndCloseClient returns int
`int RM_DeauthenticateAndCloseClient(RedisModuleCtx *ctx, uint64_t client_id)`

It it safe to change return value from `void` to `int` from the user's perspective.

(cherry picked from commit dd08aec539f8a8b35ccb8b8c953a76c07c739d62)
2020-09-01 09:27:58 +03:00
WuYunlong
086707315f see #7250, fix signature of RedisModule_DeauthenticateAndCloseClient (#7645)
In redismodule.h, RedisModule_DeauthenticateAndCloseClient returns void
`void REDISMODULE_API_FUNC(RedisModule_DeauthenticateAndCloseClient)(RedisModuleCtx *ctx, uint64_t client_id);`
But in module.c, RM_DeauthenticateAndCloseClient returns int
`int RM_DeauthenticateAndCloseClient(RedisModuleCtx *ctx, uint64_t client_id)`

It it safe to change return value from `void` to `int` from the user's perspective.

(cherry picked from commit d6220f12a9fafeb07ad86fa91e88383ca2a524c8)
2020-09-01 09:27:58 +03:00
Wang Yuan
753ff31fa3 Print error info if failed opening config file (#6943)
(cherry picked from commit 514a1e223df5b62cd1fc6aff38126d4a74ebef7e)
2020-09-01 09:27:58 +03:00
Wang Yuan
9fd0c027e7 Print error info if failed opening config file (#6943)
(cherry picked from commit ea7eeb2fd2e7a73d951f394d5d14a5ba5a75d4b2)
2020-09-01 09:27:58 +03:00
Wen Hui
9b7da52c3b fix memory leak in ACLLoadFromFile error handling (#7623)
(cherry picked from commit 30ead1edaee1cea93ee5cdeb975880df12b41f5c)
2020-09-01 09:27:58 +03:00
Wen Hui
0e581e0044 fix memory leak in ACLLoadFromFile error handling (#7623)
(cherry picked from commit 3f67b0337833cbd2a2746dad7cd8411a31b75f70)
2020-09-01 09:27:58 +03:00
Oran Agra
6f6658ab60 redis-cli --cluster-yes - negate force flag for clarity
this internal flag is there so that some commands do not comply to `--cluster-yes`

(cherry picked from commit f519dcb21626e5fd214960c7d83ee4fab7a3929d)
2020-09-01 09:27:58 +03:00
Oran Agra
e8801f7eaa redis-cli --cluster-yes - negate force flag for clarity
this internal flag is there so that some commands do not comply to `--cluster-yes`

(cherry picked from commit 1aa31e4da9a2daf9058dd923738e331bc5c91e31)
2020-09-01 09:27:58 +03:00
Frank Meier
d5f7ec90bc reintroduce REDISCLI_CLUSTER_YES env variable in redis-cli
the variable was introduced only in the 5.0 branch in #5879 bc6c1c40db

(cherry picked from commit c6ac2588db27de9a71effa6dc21417f7c787ac69)
2020-09-01 09:27:58 +03:00
Frank Meier
3a677b45ac reintroduce REDISCLI_CLUSTER_YES env variable in redis-cli
the variable was introduced only in the 5.0 branch in #5879 bc6c1c40db

(cherry picked from commit 51077c821236eba5223b1e624f2462214a0e5062)
2020-09-01 09:27:58 +03:00
Oran Agra
3a4ee4b6d6 module hook for master link up missing on successful psync (#7584)
besides, hooks test was time sensitive. when the replica managed to
reconnect quickly after the client kill, the test would fail

(cherry picked from commit c5d85c69c75438f98f84e549877c2999a2e450a8)
2020-09-01 09:27:58 +03:00
Oran Agra
a5294c4e52 module hook for master link up missing on successful psync (#7584)
besides, hooks test was time sensitive. when the replica managed to
reconnect quickly after the client kill, the test would fail

(cherry picked from commit f7e77759902aa19cfa537ed454e6bc987498e8c5)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb
2428b64d5a Clarify RM_BlockClient() error condition. (#6093)
(cherry picked from commit 342d9a642f2d21fe906bf1be24d841fa6f314601)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb
e8efcdee0a Clarify RM_BlockClient() error condition. (#6093)
(cherry picked from commit 7af05f07ff72a4246ddaf986a1e815eadd0cbfef)
2020-09-01 09:27:58 +03:00
namtsui
3d98418f4a Avoid an out-of-bounds read in the redis-sentinel (#7443)
The Redis sentinel would crash with a segfault after a few minutes
because it tried to read from a page without read permissions. Check up
front whether the sds is long enough to contain redis:slave or
redis:master before memcmp() as is done everywhere else in
sentinelRefreshInstanceInfo().

Bug report and commit message from Theo Buehler. Fix from Nam Nguyen.

Co-authored-by: Nam Nguyen <namn@berkeley.edu>
(cherry picked from commit 8c03eb90da3951ed92d8d2729fc6d2ce4feb9e45)
2020-09-01 09:27:58 +03:00
namtsui
2bcc056a41 Avoid an out-of-bounds read in the redis-sentinel (#7443)
The Redis sentinel would crash with a segfault after a few minutes
because it tried to read from a page without read permissions. Check up
front whether the sds is long enough to contain redis:slave or
redis:master before memcmp() as is done everywhere else in
sentinelRefreshInstanceInfo().

Bug report and commit message from Theo Buehler. Fix from Nam Nguyen.

Co-authored-by: Nam Nguyen <namn@berkeley.edu>
(cherry picked from commit 63dae5232415d216dfc1acce8b5335e20aa3b178)
2020-09-01 09:27:58 +03:00
Wen Hui
46686af7f5 Add SignalModifiedKey hook in XGROUP CREATE with MKSTREAM option (#7562)
(cherry picked from commit 0a2b019b79831e6c593c2de943af61ceef4327e1)
2020-09-01 09:27:58 +03:00
Wen Hui
b2bd3c0653 Add SignalModifiedKey hook in XGROUP CREATE with MKSTREAM option (#7562)
(cherry picked from commit f33acb3f029d7f5aa68e22e612036e04417e3e91)
2020-09-01 09:27:58 +03:00
Wen Hui
9f873c6c2a fix leak in error handling of debug populate command (#7062)
valsize was not modified during the for loop below instead of getting from c->argv[4], therefore there is no need to put inside the for loop.. Moreover, putting the check outside loop will also avoid memory leaking, decrRefCount(key) should be called in the original code if we put the check in for loop

(cherry picked from commit 2afa308306fc641204f10a2bbe2fe35e28b6d259)
2020-09-01 09:27:58 +03:00
Wen Hui
4238df5436 fix leak in error handling of debug populate command (#7062)
valsize was not modified during the for loop below instead of getting from c->argv[4], therefore there is no need to put inside the for loop.. Moreover, putting the check outside loop will also avoid memory leaking, decrRefCount(key) should be called in the original code if we put the check in for loop

(cherry picked from commit c69a9b2f61e0747fabb8120f03c9e2a29b43b472)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb
ebdfa71584 TLS: Propagate and handle SSL_new() failures. (#7576)
The connection API may create an accepted connection object in an error
state, and callers are expected to check it before attempting to use it.

Co-authored-by: mrpre <mrpre@163.com>
(cherry picked from commit bc450c5f63d39d0f0b8c97fa91d15bb8d688b86d)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb
1c05b87e30 TLS: Propagate and handle SSL_new() failures. (#7576)
The connection API may create an accepted connection object in an error
state, and callers are expected to check it before attempting to use it.

Co-authored-by: mrpre <mrpre@163.com>
(cherry picked from commit 784ceeb90d84bbc49fc2f2e2e6c7b9fae2524bd5)
2020-09-01 09:27:58 +03:00
Jiayuan Chen
7b2af98316 Add optional tls verification (#7502)
Adds an `optional` value to the previously boolean `tls-auth-clients` configuration keyword.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
(cherry picked from commit 198770751fdc4c46eb4971ead9b5787fd6ce39fd)
2020-09-01 09:27:58 +03:00
Jiayuan Chen
096285ab64 Add optional tls verification (#7502)
Adds an `optional` value to the previously boolean `tls-auth-clients` configuration keyword.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
(cherry picked from commit f31260b0445f5649449da41555e1272a40ae4af7)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb
ba559dc903 TLS: support cluster/replication without tls-port.
Initialize and configure OpenSSL even when tls-port is not used, because
we may still have tls-cluster or tls-replication.

Also, make sure to reconfigure OpenSSL when these parameters are changed
as TLS could have been enabled for the first time.

(cherry picked from commit b76a93c362091daafd8a8d15a45d527b7437d013)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb
8a4240b1fa TLS: support cluster/replication without tls-port.
Initialize and configure OpenSSL even when tls-port is not used, because
we may still have tls-cluster or tls-replication.

Also, make sure to reconfigure OpenSSL when these parameters are changed
as TLS could have been enabled for the first time.

(cherry picked from commit c75512d89d1697cf782cdc826acffab5b6adc1c7)
2020-09-01 09:27:58 +03:00
grishaf
53f36dc5b6 Fix prepareForShutdown function declaration (#7566)
(cherry picked from commit f8751d03ba9635064c89844d5915c2ec2dcdc827)
2020-09-01 09:27:58 +03:00
grishaf
395d339234 Fix prepareForShutdown function declaration (#7566)
(cherry picked from commit 4126ca466fb9fbf0d84a66e31c9f59f28e2fbff6)
2020-09-01 09:27:58 +03:00
Madelyn Olson
32d225c644 Properly reset errno for rdbLoad (#7542)
(cherry picked from commit 9615c7480de53c920baadf6279b527b60de8f0d4)
2020-09-01 09:27:58 +03:00
Madelyn Olson
e00b6248c9 Properly reset errno for rdbLoad (#7542)
(cherry picked from commit 818dc3a0898125d621288df671994eed34f849f2)
2020-09-01 09:27:58 +03:00
Wen Hui
fb9b6e0919 Add missing calls to raxStop (#7532)
Since the dynamic allocations in raxIterator are only used for deep walks, memory
leak due to missing call to raxStop can only happen for rax with key names longer
than 32 bytes.

Out of all the missing calls, the only ones that may lead to a leak are the rax
for consumer groups and consumers, and these were only in AOFRW and rdbSave, which
normally only happen in fork or at shutdown.

(cherry picked from commit 0b8d47a9857142203c03846fc4284746695d3dc3)
2020-09-01 09:27:58 +03:00
Wen Hui
d2f09a054c Add missing calls to raxStop (#7532)
Since the dynamic allocations in raxIterator are only used for deep walks, memory
leak due to missing call to raxStop can only happen for rax with key names longer
than 32 bytes.

Out of all the missing calls, the only ones that may lead to a leak are the rax
for consumer groups and consumers, and these were only in AOFRW and rdbSave, which
normally only happen in fork or at shutdown.

(cherry picked from commit 4e8f2d6881a38397bfbf0d7d161959163a5f6e88)
2020-09-01 09:27:58 +03:00
Wen Hui
25ab97cbe3 add missing caching command in client help (#7399)
(cherry picked from commit e035e5218f4245f5fa5eb0cafc56c572d9f4fa15)
2020-09-01 09:27:58 +03:00
Wen Hui
cef46edebc add missing caching command in client help (#7399)
(cherry picked from commit 2fbd0271f6fc0a93b9323b6478ec1a7a4ca77614)
2020-09-01 09:27:58 +03:00
zhaozhao.zz
8739408147 replication: need handle -NOPERM error after send ping (#7538)
(cherry picked from commit 57fbe4cbafb07aa9f036d7e2f5cf88830c27a921)
2020-09-01 09:27:58 +03:00
zhaozhao.zz
9e997cd0d1 replication: need handle -NOPERM error after send ping (#7538)
(cherry picked from commit 13e50935a84c319763a0d3b2be6ce64962092541)
2020-09-01 09:27:58 +03:00
maohuazhu
3155c35180 Optimize __ziplistCascadeUpdate algorithm (#6886)
The previous algorithm is of O(n^2) time complexity.
It would have run through the ziplist entries one by one, each time doing a `realloc` and a
`memmove` (moving the entire tail of the ziplist).

The new algorithm is O(n), it runs over all the records once, computing the size of the `realloc`
needed, then does one `realloc`, and run thought the records again doing many smaller `memmove`s,
each time moving just one record.

So this change reduces many reallocs, and moves each record just once.

Co-authored-by: zhumaohua <zhumaohua@megvii.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-08-28 17:22:35 +03:00
maohuazhu
ee4a15aae0
Optimize __ziplistCascadeUpdate algorithm (#6886)
The previous algorithm is of O(n^2) time complexity.
It would have run through the ziplist entries one by one, each time doing a `realloc` and a
`memmove` (moving the entire tail of the ziplist).

The new algorithm is O(n), it runs over all the records once, computing the size of the `realloc`
needed, then does one `realloc`, and run thought the records again doing many smaller `memmove`s,
each time moving just one record.

So this change reduces many reallocs, and moves each record just once.

Co-authored-by: zhumaohua <zhumaohua@megvii.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-08-28 17:22:35 +03:00
Jim Brunner
2ce59d3a1b Use H/W Monotonic clock and updates to AE (#7644)
Update adds a general source for retrieving a monotonic time.
In addition, AE has been updated to utilize the new monotonic
clock for timer processing.

This performance improvement is **not** enabled in a default build due to various H/W compatibility
concerns, see README.md for details. It does however change the default use of gettimeofday with
clock_gettime and somewhat improves performance.

This update provides the following
1. An interface for retrieving a monotonic clock. getMonotonicUs returns a uint64_t (aka monotime)
   with the number of micro-seconds from an arbitrary point. No more messing with tv_sec/tv_usec.
   Simple routines are provided for measuring elapsed milli-seconds or elapsed micro-seconds (the
   most common use case for a monotonic timer). No worries about time moving backwards.
2. High-speed assembler implementation for x86 and ARM. The standard method for retrieving the
   monotonic clock is POSIX.1b (1993): clock_gettime(CLOCK_MONOTONIC, timespec*). However, most
   modern processors provide a constant speed instruction clock which can be retrieved in a fraction
   of the time that it takes to call clock_gettime. For x86, this is provided by the RDTSC
   instruction. For ARM, this is provided by the CNTVCT_EL0 instruction. As a compile-time option,
   these high-speed timers can be chosen. (Default is POSIX clock_gettime.)
3. Refactor of event loop timers. The timer processing in ae.c has been refactored to use the new
   monotonic clock interface. This results in simpler/cleaner logic and improved performance.
2020-08-28 11:54:10 +03:00
Jim Brunner
c01e94a431
Use H/W Monotonic clock and updates to AE (#7644)
Update adds a general source for retrieving a monotonic time.
In addition, AE has been updated to utilize the new monotonic
clock for timer processing.

This performance improvement is **not** enabled in a default build due to various H/W compatibility
concerns, see README.md for details. It does however change the default use of gettimeofday with
clock_gettime and somewhat improves performance.

This update provides the following
1. An interface for retrieving a monotonic clock. getMonotonicUs returns a uint64_t (aka monotime)
   with the number of micro-seconds from an arbitrary point. No more messing with tv_sec/tv_usec.
   Simple routines are provided for measuring elapsed milli-seconds or elapsed micro-seconds (the
   most common use case for a monotonic timer). No worries about time moving backwards.
2. High-speed assembler implementation for x86 and ARM. The standard method for retrieving the
   monotonic clock is POSIX.1b (1993): clock_gettime(CLOCK_MONOTONIC, timespec*). However, most
   modern processors provide a constant speed instruction clock which can be retrieved in a fraction
   of the time that it takes to call clock_gettime. For x86, this is provided by the RDTSC
   instruction. For ARM, this is provided by the CNTVCT_EL0 instruction. As a compile-time option,
   these high-speed timers can be chosen. (Default is POSIX clock_gettime.)
3. Refactor of event loop timers. The timer processing in ae.c has been refactored to use the new
   monotonic clock interface. This results in simpler/cleaner logic and improved performance.
2020-08-28 11:54:10 +03:00
Oran Agra
2640897e3a Fix rejectCommand trims newline in shared error objects, hung clients (#7714)
fe8d6fe74 (released in 6.0.6) has a side effect, when processCommand
rejects a command with pre-made shared object error string, it trims the
newlines from the end of the string. if that string is later used with
addReply, the newline will be missing, breaking the protocol, and
leaving the client hung.

It seems that the only scenario which this happens is when replying with
-LOADING to some command, and later using that reply from the CONFIG
SET command (still during loading). this will result in hung client.

Refactoring the code in order to avoid trimming these newlines from
shared string objects, and do the newline trimming only in other cases
where it's needed.

Co-authored-by: Guy Benoish <guy.benoish@redislabs.com>
2020-08-27 12:54:01 +03:00
Oran Agra
9fcd9e191e
Fix rejectCommand trims newline in shared error objects, hung clients (#7714)
65a3307bc (released in 6.0.6) has a side effect, when processCommand
rejects a command with pre-made shared object error string, it trims the
newlines from the end of the string. if that string is later used with
addReply, the newline will be missing, breaking the protocol, and
leaving the client hung.

It seems that the only scenario which this happens is when replying with
-LOADING to some command, and later using that reply from the CONFIG
SET command (still during loading). this will result in hung client.

Refactoring the code in order to avoid trimming these newlines from
shared string objects, and do the newline trimming only in other cases
where it's needed.

Co-authored-by: Guy Benoish <guy.benoish@redislabs.com>
2020-08-27 12:54:01 +03:00
Oran Agra
8a4d10a3a5 Update memory metrics for INFO during loading (#7690)
During a long AOF or RDB loading, the memory stats were not updated, and
INFO would return stale data, specifically about fragmentation and RSS.
In the past some of these were sampled directly inside the INFO command,
but were moved to cron as an optimization.

This commit introduces a concept of loadingCron which should take
some of the responsibilities of serverCron.
It attempts to limit it's rate to approximately the server Hz, but may
not be very accurate.

In order to avoid too many system call, we use the cached ustime, and
also make sure to update it in both AOF loading and RDB loading inside
processEventsWhileBlocked (it seems AOF loading was missing it).
2020-08-27 11:09:32 +03:00
Oran Agra
8bdcbbb085
Update memory metrics for INFO during loading (#7690)
During a long AOF or RDB loading, the memory stats were not updated, and
INFO would return stale data, specifically about fragmentation and RSS.
In the past some of these were sampled directly inside the INFO command,
but were moved to cron as an optimization.

This commit introduces a concept of loadingCron which should take
some of the responsibilities of serverCron.
It attempts to limit it's rate to approximately the server Hz, but may
not be very accurate.

In order to avoid too many system call, we use the cached ustime, and
also make sure to update it in both AOF loading and RDB loading inside
processEventsWhileBlocked (it seems AOF loading was missing it).
2020-08-27 11:09:32 +03:00
valentinogeron
0292720ccb EXEC with only read commands should not be rejected when OOM (#7696)
If the server gets MULTI command followed by only read
commands, and right before it gets the EXEC it reaches OOM,
the client will get OOM response.

So, from now on, it will get OOM response only if there was
at least one command that was tagged with `use-memory` flag
2020-08-27 09:19:24 +03:00
valentinogeron
b7289e912c
EXEC with only read commands should not be rejected when OOM (#7696)
If the server gets MULTI command followed by only read
commands, and right before it gets the EXEC it reaches OOM,
the client will get OOM response.

So, from now on, it will get OOM response only if there was
at least one command that was tagged with `use-memory` flag
2020-08-27 09:19:24 +03:00