6954 Commits

Author SHA1 Message Date
Oran Agra
f519dcb216 redis-cli --cluster-yes - negate force flag for clarity
this internal flag is there so that some commands do not comply to `--cluster-yes`
2020-08-05 18:30:43 +03:00
Frank Meier
c6ac2588db reintroduce REDISCLI_CLUSTER_YES env variable in redis-cli
the variable was introduced only in the 5.0 branch in #5879 bc6c1c40db
2020-08-05 18:30:43 +03:00
Tyson Andre
486e39e86e Add a ZMSCORE command returning an array of scores. (#7593)
Syntax: `ZMSCORE KEY MEMBER [MEMBER ...]`

This is an extension of #2359
amended by Tyson Andre to work with the changed unstable API,
add more tests, and consistently return an array.

- It seemed as if it would be more likely to get reviewed
  after updating the implementation.

Currently, multi commands or lua scripting to call zscore multiple times
would almost definitely be less efficient than a native ZMSCORE
for the following reasons:

- Need to fetch the set from the string every time instead of reusing the C
  pointer.
- Using pipelining or multi-commands would result in more bytes sent by
  the client for the repeated `ZMSCORE KEY` sections.
- Need to specially encode the data and decode it from the client
  for lua-based solutions.
- The fastest solution I've seen for large sets(thousands or millions)
  involves lua and a variadic ZADD, then a ZINTERSECT, then a ZRANGE 0 -1,
  then UNLINK of a temporary set (or lua). This is still inefficient.

Co-authored-by: Tyson Andre <tysonandre775@hotmail.com>
2020-08-04 17:49:33 +03:00
hujie
46bff240bf remove duplicate semicolon (#7438) 2020-08-02 13:59:51 +03:00
Oran Agra
c5d85c69c7 module hook for master link up missing on successful psync (#7584)
besides, hooks test was time sensitive. when the replica managed to
reconnect quickly after the client kill, the test would fail
2020-07-31 13:14:29 +03:00
Oran Agra
27a8ded3db Remove dead code from update_zmalloc_stat_alloc (#7589)
this seems like leftover from before 9d8ceca
2020-07-31 13:01:39 +03:00
fayadexinqing
323d92b689 broadcast a PONG message when slot's migration is over, which may reduce the moved request of clients (#7571) 2020-07-29 18:05:27 -07:00
Kevin McGehee
b6acefddb7 Call propagate instead of writing directly to AOF/replicas (#6658)
Use higher-level API to funnel all generic propagation through
single function call.
2020-07-29 17:54:37 -07:00
Yossi Gottlieb
342d9a642f Clarify RM_BlockClient() error condition. (#6093) 2020-07-29 17:03:38 +03:00
Arun Ranganathan
444b53e640 Show threading configuration in INFO output (#7446)
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-07-29 08:46:44 +03:00
namtsui
8c03eb90da Avoid an out-of-bounds read in the redis-sentinel (#7443)
The Redis sentinel would crash with a segfault after a few minutes
because it tried to read from a page without read permissions. Check up
front whether the sds is long enough to contain redis:slave or
redis:master before memcmp() as is done everywhere else in
sentinelRefreshInstanceInfo().

Bug report and commit message from Theo Buehler. Fix from Nam Nguyen.

Co-authored-by: Nam Nguyen <namn@berkeley.edu>
2020-07-29 08:25:56 +03:00
Wen Hui
0a2b019b79 Add SignalModifiedKey hook in XGROUP CREATE with MKSTREAM option (#7562) 2020-07-29 08:22:54 +03:00
Wen Hui
2afa308306 fix leak in error handling of debug populate command (#7062)
valsize was not modified during the for loop below instead of getting from c->argv[4], therefore there is no need to put inside the for loop.. Moreover, putting the check outside loop will also avoid memory leaking, decrRefCount(key) should be called in the original code if we put the check in for loop
2020-07-28 22:05:48 +03:00
Yossi Gottlieb
bc450c5f63 TLS: Propagate and handle SSL_new() failures. (#7576)
The connection API may create an accepted connection object in an error
state, and callers are expected to check it before attempting to use it.

Co-authored-by: mrpre <mrpre@163.com>
2020-07-28 11:32:47 +03:00
Jiayuan Chen
198770751f Add optional tls verification (#7502)
Adds an `optional` value to the previously boolean `tls-auth-clients` configuration keyword.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2020-07-28 10:45:21 +03:00
Yossi Gottlieb
b76a93c362 TLS: support cluster/replication without tls-port.
Initialize and configure OpenSSL even when tls-port is not used, because
we may still have tls-cluster or tls-replication.

Also, make sure to reconfigure OpenSSL when these parameters are changed
as TLS could have been enabled for the first time.
2020-07-27 13:26:02 +03:00
grishaf
f8751d03ba Fix prepareForShutdown function declaration (#7566) 2020-07-26 08:27:30 +03:00
zhaozhao.zz
f4e2525c86 more strict check in rioConnRead (#7564) 2020-07-24 14:40:19 +08:00
Meir Shpilraien (Spielrein)
73198c5019 This PR introduces a new loaded keyspace event (#7536)
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
2020-07-23 12:38:51 +03:00
Oran Agra
4a118ab30b Fix harmless bug in rioConnRead (#7557)
this code is in use only if the master is disk-based, and the replica is
diskless. In this case we use a buffered reader, but we must avoid reading
past the rdb file, into the command stream. which Luckly rdb.c doesn't
really attempt to do (it knows how much it should read).

When rioConnRead detects that the extra buffering attempt reaches beyond
the read limit it should read less, but if the caller actually requested
more, then it should return with an error rather than a short read. the
bug would have resulted in short read.

in order to fix it, the code must consider the real requested size, and
not the extra buffering size.
2020-07-23 12:37:43 +03:00
Madelyn Olson
9615c7480d Properly reset errno for rdbLoad (#7542) 2020-07-21 17:00:13 -07:00
WuYunlong
2054cf46c4 Clarification on the bug that was fixed in PR #7539. (#7541)
Before that PR, processCommand() did not notice that cmd could be a module
command in which case getkeys_proc member has a different meaning.

The outcome was that a module command which doesn't take any key names in its
arguments (similar to SLOWLOG) would be handled as if it might have key name arguments
(similar to MEMORY), would consider cluster redirect but will end up with 0 keys
after an excessive call to getKeysFromCommand, and eventually do the right thing.
2020-07-21 09:41:44 +03:00
Wen Hui
0b8d47a985 Add missing calls to raxStop (#7532)
Since the dynamic allocations in raxIterator are only used for deep walks, memory
leak due to missing call to raxStop can only happen for rax with key names longer
than 32 bytes.

Out of all the missing calls, the only ones that may lead to a leak are the rax
for consumer groups and consumers, and these were only in AOFRW and rdbSave, which
normally only happen in fork or at shutdown.
2020-07-21 08:13:05 +03:00
Wen Hui
e035e5218f add missing caching command in client help (#7399) 2020-07-20 18:53:03 -07:00
zhaozhao.zz
57fbe4cbaf replication: need handle -NOPERM error after send ping (#7538) 2020-07-20 22:21:55 +08:00
WuYunlong
a06d7eda9a Fix cluster redirect for module command with no firstkey. (#7539)
Before this commit, processCommand() did not notice that cmd could be a module command
which declared `getkeys-api` and handled it for the purpose of cluster redirect it
as if it doesn't use any keys.

This commit fixed it by reusing the codes in addReplyCommand().
2020-07-20 15:33:06 +03:00
WuYunlong
6efda6a476 Refactor streamAppendItem() by deleting redundancy condition. (#7487)
It will never happen that "lp != NULL && lp_bytes >= server.stream_node_max_bytes".
Assume that "lp != NULL && lp_bytes >= server.stream_node_max_bytes",
we got the following conditions:
a. lp != NULL
b. lp_bytes >= server.stream_node_max_bytes

If server.stream_node_max_bytes is 0, given condition a, condition b is always satisfied
If server.stream_node_max_bytes is not 0, given condition a and condition b, the codes just a
	few lines above set lp to NULL, a controdiction with condition a

So that condition b is recundant. We could delete it safely.
2020-07-20 13:14:27 +03:00
yoav-steinberg
8a2b0472a7 Support passing stack allocated module strings to moduleCreateArgvFromUserFormat (#7528)
Specifically, the key passed to the module aof_rewrite callback is a stack allocated robj. When passing it to RedisModule_EmitAOF (with appropriate "s" fmt string) redis used to panic when trying to inc the ref count of the stack allocated robj. Now support such robjs by coying them to a new heap robj. This doesn't affect performance because using the alternative "c" or "b" format strings also copies the input to a new heap robj.
2020-07-16 20:59:38 +03:00
杨博东
2028ad5a12 Stream avoid duplicate parse id (#7450) 2020-07-16 08:57:27 +03:00
Luke Palmer
df4c74ef07 Send null for invalidate on flush (#7469) 2020-07-15 10:53:41 -07:00
dmurnane
c292d43fec Notify systemd on sentinel startup (#7168)
Co-authored-by: Daniel Murnane <dmurnane@eitccorp.com>
2020-07-15 13:29:26 +03:00
Developer-Ecosystem-Engineering
004479c184 Add registers dump support for Apple silicon (#7453)
Export following environment variables before building on macOS on Apple silicon

export ARCH_FLAGS="-arch arm64"
export SDK_NAME=macosx
export SDK_PATH=$(xcrun --show-sdk-path --sdk $SDK_NAME)
export CFLAGS="$ARCH_FLAGS -isysroot $SDK_PATH -I$SDK_PATH/usr/include"
export CXXFLAGS=$CFLAGS
export LDFLAGS="$ARCH_FLAGS"
export CC="$(xcrun -sdk $SDK_PATH --find clang) $CFLAGS"
export CXX="$(xcrun -sdk $SDK_PATH --find clang++) $CXXFLAGS"
export LD="$(xcrun -sdk $SDK_PATH --find ld) $LDFLAGS"

make
make test
..
All tests passed without errors!

Backtrack logging assumes x86 and required updating
2020-07-15 12:44:03 +03:00
Wen Hui
6c5f98b24b correct error msg for num connections reaching maxclients in cluster mode (#7444) 2020-07-15 12:38:47 +03:00
WuYunlong
e5166eccee Fix command help for unexpected options (#7476) 2020-07-15 12:38:22 +03:00
WuYunlong
7da8c062d5 Refactor RM_KeyType() by using macro. (#7486) 2020-07-15 12:37:44 +03:00
Oran Agra
a3df709234 diskless master disconnect replicas when rdb child failed (#7518)
in case the rdb child failed, crashed or terminated unexpectedly redis
would have marked the replica clients with repl_put_online_on_ack and
then kill them only after a minute when no ack was received.

it would not stream anything to these connections, so the only effect of
this bug is a delay of 1 minute in the replicas attempt to re-connect.
2020-07-14 20:21:59 +03:00
Qu Chen
a517043c7a Replica always reports master's config epoch in CLUSTER NODES output. (#7235) 2020-07-13 07:16:06 -07:00
Oran Agra
c10eabeeca RESTORE ABSTTL skip expired keys - leak (#7511) 2020-07-13 16:40:19 +03:00
jimgreen2013
c05d8a0a31 fix description about ziplist, the code is ok (#6318)
* fix description about ZIP_BIG_PREVLEN(the code is ok), it's similar to
antirez#4705

* fix description about ziplist entry encoding field (the code is ok),
the max length should be 2^32 - 1 when encoding is 5 bytes
2020-07-11 14:51:44 -05:00
杨博东
a4b4df5fe6 STORE variants: SINTER,SUNION,SDIFF,ZUNION use setKey instead of dbDelete+dbAdd (#7489)
one of the differences (other than consistent code with SORT, GEORADIUS), is that the LFU of the old key is retained.
2020-07-11 15:52:41 +03:00
马永泽
40e930dab5 fix benchmark in cluster mode fails to authenticate (#7488)
Co-authored-by: Oran Agra <oran@redislabs.com> (styling)
2020-07-10 16:37:11 +03:00
Yossi Gottlieb
c611a836f6 TLS: Session caching configuration support. (#7420)
* TLS: Session caching configuration support.
* TLS: Remove redundant config initialization.
2020-07-10 11:33:47 +03:00
Yossi Gottlieb
6dfbccc212 TLS: Ignore client cert when tls-auth-clients off. (#7457) 2020-07-10 10:32:21 +03:00
James Hilliard
5f16b65a5f Use pkg-config to properly detect libssl and libcrypto libraries (#7452) 2020-07-10 10:30:09 +03:00
Yossi Gottlieb
99b920534f TLS: Add missing redis-cli options. (#7456)
* Tests: fix and reintroduce redis-cli tests.

These tests have been broken and disabled for 10 years now!

* TLS: add remaining redis-cli support.

This adds support for the redis-cli --pipe, --rdb and --replica options
previously unsupported in --tls mode.

* Fix writeConn().
2020-07-10 10:25:55 +03:00
Oran Agra
a11940b112 redis-cli --hotkeys fixed to handle non-printable key names 2020-07-10 10:07:47 +03:00
Oran Agra
3d760eaea2 redis-cli --bigkeys fixed to handle non-printable key names 2020-07-10 10:07:47 +03:00
Oran Agra
1c35f8741b RESTORE ABSTTL won't store expired keys into the db (#7472)
Similarly to EXPIREAT with TTL in the past, which implicitly deletes the
key and return success, RESTORE should not store key that are already
expired into the db.
When used together with REPLACE it should emit a DEL to keyspace
notification and replication stream.
2020-07-10 10:02:37 +03:00
huangzhw
84bef6691b defrag.c activeDefragSdsListAndDict when defrag sdsele, We can't use (#7492)
it to calculate hash, we should use newsds.
2020-07-10 08:29:44 +03:00
Oran Agra
8d4f055e43 tests/valgrind: don't use debug restart (#7404)
* tests/valgrind: don't use debug restart

DEBUG REATART causes two issues:
1. it uses execve which replaces the original process and valgrind doesn't
   have a chance to check for errors, so leaks go unreported.
2. valgrind report invalid calls to close() which we're unable to resolve.

So now the tests use restart_server mechanism in the tests, that terminates
the old server and starts a new one, new PID, but same stdout, stderr.

since the stderr can contain two or more valgrind report, it is not enough
to just check for the absence of leaks, we also need to check for some known
errors, we do both, and fail if we either find an error, or can't find a
report saying there are no leaks.

other changes:
- when killing a server that was already terminated we check for leaks too.
- adding DEBUG LEAK which was used to test it.
- adding --trace-children to valgrind, although no longer needed.
- since the stdout contains two or more runs, we need slightly different way
  of checking if the new process is up (explicitly looking for the new PID)
- move the code that handles --wait-server to happen earlier (before
  watching the startup message in the log), and serve the restarted server too.

* squashme - CR fixes
2020-07-10 08:26:52 +03:00