9954 Commits

Author SHA1 Message Date
Kevin McGehee
b6acefddb7 Call propagate instead of writing directly to AOF/replicas (#6658)
Use higher-level API to funnel all generic propagation through
single function call.
2020-07-29 17:54:37 -07:00
Yossi Gottlieb
342d9a642f Clarify RM_BlockClient() error condition. (#6093) 2020-07-29 17:03:38 +03:00
Arun Ranganathan
444b53e640 Show threading configuration in INFO output (#7446)
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-07-29 08:46:44 +03:00
namtsui
8c03eb90da Avoid an out-of-bounds read in the redis-sentinel (#7443)
The Redis sentinel would crash with a segfault after a few minutes
because it tried to read from a page without read permissions. Check up
front whether the sds is long enough to contain redis:slave or
redis:master before memcmp() as is done everywhere else in
sentinelRefreshInstanceInfo().

Bug report and commit message from Theo Buehler. Fix from Nam Nguyen.

Co-authored-by: Nam Nguyen <namn@berkeley.edu>
2020-07-29 08:25:56 +03:00
Wen Hui
0a2b019b79 Add SignalModifiedKey hook in XGROUP CREATE with MKSTREAM option (#7562) 2020-07-29 08:22:54 +03:00
Wen Hui
2afa308306 fix leak in error handling of debug populate command (#7062)
valsize was not modified during the for loop below instead of getting from c->argv[4], therefore there is no need to put inside the for loop.. Moreover, putting the check outside loop will also avoid memory leaking, decrRefCount(key) should be called in the original code if we put the check in for loop
2020-07-28 22:05:48 +03:00
Yossi Gottlieb
675b00c7e0 Fix TLS cluster tests. (#7578)
Fix consistency test added in 0c9916d00 without considering TLS
redis-cli configuration.
2020-07-28 14:04:06 +03:00
Yossi Gottlieb
bc450c5f63 TLS: Propagate and handle SSL_new() failures. (#7576)
The connection API may create an accepted connection object in an error
state, and callers are expected to check it before attempting to use it.

Co-authored-by: mrpre <mrpre@163.com>
2020-07-28 11:32:47 +03:00
Oran Agra
06aaeabaea Fix failing tests due to issues with wait_for_log_message (#7572)
- the test now waits for specific set of log messages rather than wait for
  timeout looking for just one message.
- we don't wanna sample the current length of the log after an action, due
  to a race, we need to start the search from the line number of the last
  message we where waiting for.
- when attempting to trigger a full sync, use multi-exec to avoid a race
  where the replica manages to re-connect before we completed the set of
  actions that should force a full sync.
- fix verify_log_message which was broken and unused
2020-07-28 11:15:29 +03:00
Jiayuan Chen
198770751f Add optional tls verification (#7502)
Adds an `optional` value to the previously boolean `tls-auth-clients` configuration keyword.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2020-07-28 10:45:21 +03:00
Yossi Gottlieb
b03473f0bc TLS: support cluster/replication without tls-port. (#7573)
Initialize and configure OpenSSL even when tls-port is not used, because
we may still have tls-cluster or tls-replication.

Also, make sure to reconfigure OpenSSL when these parameters are changed
as TLS could have been enabled for the first time.
2020-07-27 15:31:53 +03:00
Oran Agra
62e84b42d2 Daily github action: run cluster and sentinel tests with tls (#7575) 2020-07-27 15:30:36 +03:00
Yossi Gottlieb
b76a93c362 TLS: support cluster/replication without tls-port.
Initialize and configure OpenSSL even when tls-port is not used, because
we may still have tls-cluster or tls-replication.

Also, make sure to reconfigure OpenSSL when these parameters are changed
as TLS could have been enabled for the first time.
2020-07-27 13:26:02 +03:00
grishaf
f8751d03ba Fix prepareForShutdown function declaration (#7566) 2020-07-26 08:27:30 +03:00
zhaozhao.zz
f4e2525c86 more strict check in rioConnRead (#7564) 2020-07-24 14:40:19 +08:00
Oran Agra
49d4aebce0 Stabilize bgsave test that sometimes fails with valgrind (#7559)
on ci.redis.io the test fails a lot, reporting that bgsave didn't end.
increaseing the timeout we wait for that bgsave to get aborted.
in addition to that, i also verify that it indeed got aborted by
checking that the save counter wasn't reset.

add another test to verify that a successful bgsave indeed resets the
change counter.
2020-07-23 13:06:24 +03:00
Meir Shpilraien (Spielrein)
73198c5019 This PR introduces a new loaded keyspace event (#7536)
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
2020-07-23 12:38:51 +03:00
Oran Agra
4a118ab30b Fix harmless bug in rioConnRead (#7557)
this code is in use only if the master is disk-based, and the replica is
diskless. In this case we use a buffered reader, but we must avoid reading
past the rdb file, into the command stream. which Luckly rdb.c doesn't
really attempt to do (it knows how much it should read).

When rioConnRead detects that the extra buffering attempt reaches beyond
the read limit it should read less, but if the caller actually requested
more, then it should return with an error rather than a short read. the
bug would have resulted in short read.

in order to fix it, the code must consider the real requested size, and
not the extra buffering size.
2020-07-23 12:37:43 +03:00
Brian P O'Rourke
7b4a39264e Add contribution guidelines for vulnerability reports 2020-07-22 16:24:39 +03:00
Brian P O'Rourke
a6de21b1a1 Outdent github issues guidelines 2020-07-22 16:24:39 +03:00
Sungho Hwang
63c5bfb6ef Fix typo in deps/README.md (#7553) 2020-07-22 11:33:57 +03:00
Madelyn Olson
9615c7480d Properly reset errno for rdbLoad (#7542) 2020-07-21 17:00:13 -07:00
Oran Agra
bb170fa06e testsuite may leave servers alive on error (#7549)
in cases where you have
test name {
  start_server {
    start_server {
      assert
    }
  }
}

the exception will be thrown to the test proc, and the servers are
supposed to be killed on the way out. but it seems there was always a
bug of not cleaning the server stack, and recently (#7404) we started
relying on that stack in order to kill them, so with that bug sometimes
we would have tried to kill the same server twice, and leave one alive.

luckly, in most cases the pattern is:
start_server {
  test name {
  }
}
2020-07-21 16:56:19 +03:00
Yossi Gottlieb
dbc0a64843 Tests: drop TCL 8.6 dependency. (#7548)
This re-implements the redis-cli --pipe test so it no longer depends on a close feature available only in TCL 8.6.

Basically what this test does is run redis-cli --pipe, generates a bunch of commands and pipes them through redis-cli, and inspects the result in both Redis and the redis-cli output.

To do that, we need to close stdin for redis-cli to indicate we're done so it can flush its buffers and exit. TCL has bi-directional channels can only offers a way to "one-way close" a channel with TCL 8.6. To work around that, we now generate the commands into a file and feed that file to redis-cli directly.

As we're writing to an actual file, the number of commands is now reduced.
2020-07-21 14:17:14 +03:00
Oran Agra
a472f35efd Fixes to release scripts (#7547) 2020-07-21 14:07:06 +03:00
WuYunlong
2054cf46c4 Clarification on the bug that was fixed in PR #7539. (#7541)
Before that PR, processCommand() did not notice that cmd could be a module
command in which case getkeys_proc member has a different meaning.

The outcome was that a module command which doesn't take any key names in its
arguments (similar to SLOWLOG) would be handled as if it might have key name arguments
(similar to MEMORY), would consider cluster redirect but will end up with 0 keys
after an excessive call to getKeysFromCommand, and eventually do the right thing.
2020-07-21 09:41:44 +03:00
Remi Collet
7853d8410b Fix deprecated tail syntax in tests (#7543) 2020-07-21 09:07:54 +03:00
Wen Hui
0b8d47a985 Add missing calls to raxStop (#7532)
Since the dynamic allocations in raxIterator are only used for deep walks, memory
leak due to missing call to raxStop can only happen for rax with key names longer
than 32 bytes.

Out of all the missing calls, the only ones that may lead to a leak are the rax
for consumer groups and consumers, and these were only in AOFRW and rdbSave, which
normally only happen in fork or at shutdown.
2020-07-21 08:13:05 +03:00
Wen Hui
e035e5218f add missing caching command in client help (#7399) 2020-07-20 18:53:03 -07:00
zhaozhao.zz
57fbe4cbaf replication: need handle -NOPERM error after send ping (#7538) 2020-07-20 22:21:55 +08:00
WuYunlong
a06d7eda9a Fix cluster redirect for module command with no firstkey. (#7539)
Before this commit, processCommand() did not notice that cmd could be a module command
which declared `getkeys-api` and handled it for the purpose of cluster redirect it
as if it doesn't use any keys.

This commit fixed it by reusing the codes in addReplyCommand().
2020-07-20 15:33:06 +03:00
Oran Agra
363347830b Run daily CI on PRs to release a branch (#7535) 2020-07-20 13:57:54 +03:00
WuYunlong
6efda6a476 Refactor streamAppendItem() by deleting redundancy condition. (#7487)
It will never happen that "lp != NULL && lp_bytes >= server.stream_node_max_bytes".
Assume that "lp != NULL && lp_bytes >= server.stream_node_max_bytes",
we got the following conditions:
a. lp != NULL
b. lp_bytes >= server.stream_node_max_bytes

If server.stream_node_max_bytes is 0, given condition a, condition b is always satisfied
If server.stream_node_max_bytes is not 0, given condition a and condition b, the codes just a
	few lines above set lp to NULL, a controdiction with condition a

So that condition b is recundant. We could delete it safely.
2020-07-20 13:14:27 +03:00
Scott Brenner
c7644eda71 GitHub Actions workflows - use latest version of actions/checkout (#7534) 2020-07-20 09:22:24 +03:00
Itamar Haber
f827ed388d Merge pull request #7529 from itamarhaber/stable-sha256sum
Adds SHA256SUM to redis-stable tarball upload
2020-07-17 15:20:54 +03:00
Itamar Haber
efb04ab4a2 Adds SHA256SUM to redis-stable tarball upload 2020-07-16 21:31:36 +03:00
Itamar Haber
9cbf4056a3 Adds GitHub issue templates (#7468)
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: yoav-steinberg <yoav@monfort.co.il>
2020-07-16 21:01:27 +03:00
yoav-steinberg
8a2b0472a7 Support passing stack allocated module strings to moduleCreateArgvFromUserFormat (#7528)
Specifically, the key passed to the module aof_rewrite callback is a stack allocated robj. When passing it to RedisModule_EmitAOF (with appropriate "s" fmt string) redis used to panic when trying to inc the ref count of the stack allocated robj. Now support such robjs by coying them to a new heap robj. This doesn't affect performance because using the alternative "c" or "b" format strings also copies the input to a new heap robj.
2020-07-16 20:59:38 +03:00
杨博东
2028ad5a12 Stream avoid duplicate parse id (#7450) 2020-07-16 08:57:27 +03:00
Luke Palmer
df4c74ef07 Send null for invalidate on flush (#7469) 2020-07-15 10:53:41 -07:00
dmurnane
c292d43fec Notify systemd on sentinel startup (#7168)
Co-authored-by: Daniel Murnane <dmurnane@eitccorp.com>
2020-07-15 13:29:26 +03:00
Developer-Ecosystem-Engineering
004479c184 Add registers dump support for Apple silicon (#7453)
Export following environment variables before building on macOS on Apple silicon

export ARCH_FLAGS="-arch arm64"
export SDK_NAME=macosx
export SDK_PATH=$(xcrun --show-sdk-path --sdk $SDK_NAME)
export CFLAGS="$ARCH_FLAGS -isysroot $SDK_PATH -I$SDK_PATH/usr/include"
export CXXFLAGS=$CFLAGS
export LDFLAGS="$ARCH_FLAGS"
export CC="$(xcrun -sdk $SDK_PATH --find clang) $CFLAGS"
export CXX="$(xcrun -sdk $SDK_PATH --find clang++) $CXXFLAGS"
export LD="$(xcrun -sdk $SDK_PATH --find ld) $LDFLAGS"

make
make test
..
All tests passed without errors!

Backtrack logging assumes x86 and required updating
2020-07-15 12:44:03 +03:00
Wen Hui
6c5f98b24b correct error msg for num connections reaching maxclients in cluster mode (#7444) 2020-07-15 12:38:47 +03:00
WuYunlong
e5166eccee Fix command help for unexpected options (#7476) 2020-07-15 12:38:22 +03:00
WuYunlong
7da8c062d5 Refactor RM_KeyType() by using macro. (#7486) 2020-07-15 12:37:44 +03:00
Oran Agra
a3df709234 diskless master disconnect replicas when rdb child failed (#7518)
in case the rdb child failed, crashed or terminated unexpectedly redis
would have marked the replica clients with repl_put_online_on_ack and
then kill them only after a minute when no ack was received.

it would not stream anything to these connections, so the only effect of
this bug is a delay of 1 minute in the replicas attempt to re-connect.
2020-07-14 20:21:59 +03:00
Oran Agra
8a14ce8634 redis-cli tests, fix valgrind timing issue (#7519)
this test when run with valgrind on github actions takes 160 seconds
2020-07-14 18:04:08 +03:00
WuYunlong
24b6f62741 Fix out of update help info in tcl tests. (#7516)
Before this commit, the output of "./runtest-cluster --help" is incorrect.
After this commit, the format of the following 3 output is consistent:
./runtest --help
./runtest-cluster --help
./runtest-sentinel --help
2020-07-14 11:35:04 +03:00
Qu Chen
a517043c7a Replica always reports master's config epoch in CLUSTER NODES output. (#7235) 2020-07-13 07:16:06 -07:00
Oran Agra
c10eabeeca RESTORE ABSTTL skip expired keys - leak (#7511) 2020-07-13 16:40:19 +03:00