10273 Commits

Author SHA1 Message Date
Leibale Eidelman
adc4748248 fix ZRANGESTORE - should return 0 when src points to an empty key (#9089)
mistakenly it used to return an empty array rather than 0.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 95274f1f8a3ef4cb4033beecfaa99ea1439ed170)
2021-07-21 21:06:49 +03:00
guybe7
3b32512dc9 Include sizeof(struct stream) in objectComputeSize (#9164)
Affects MEMORY USAGE

(cherry picked from commit 4434cebbb38ae2ef39b2d74ed3e6cd8d1a061d0c)
2021-07-21 21:06:49 +03:00
Binbin
0be66e4ebe ZRANDMEMBER WITHSCORES with negative COUNT may return bad score (#9162)
Return a bad score when used with negative count (or count of 1), and non-ziplist encoded zset.
Also add test to validate the return value and cover the issue.

(cherry picked from commit 4bc5a8324d3cb23ed2cc8a9cd19444a893a6d52c)
2021-07-21 21:06:49 +03:00
Evan
21a768990c modules: Add newlen == 0 handling to RM_StringTruncate (#3717) (#3718)
Previously, passing 0 for newlen would not truncate the string at all.
This adds handling of this case, freeing the old string and creating a new empty string.

Other changes:
- Move `src/modules/testmodule.c` to `tests/modules/basics.c`
- Introduce that basic test into the test suite
- Add tests to cover StringTruncate
- Add `test-modules` build target for the main makefile
- Extend `distclean` build target to clean modules too

(cherry picked from commit 1ccf2ca2f475a161b8ca0c574d4c0e6ef9ecf754)
2021-07-21 21:06:49 +03:00
yvette903
fde58863d5 Fix coredump after Client Unpause command when threaded I/O is enabled (#9041)
Fix crash when using io-threads-do-reads and issuing CLIENT PAUSE and
CLIENT UNPAUSE.
This issue was introduced in redis 6.2 together with the FAILOVER command.

(cherry picked from commit 096c5fd5d22caa2c21a4863cbaaf64fd8e7107d2)
2021-07-21 21:06:49 +03:00
guybe7
288ff19198 Module API for current command name (#8792)
sometimes you can be very deep in the call stack, without access to argv.
once you're there you may want your reply/log to contain the command name.

(cherry picked from commit e16d3eb998f7017c95ef17179de77aa5b6f2e272)
2021-07-21 21:06:49 +03:00
Huang Zhw
5a963f360f Fix XTRIM or XADD with LIMIT may delete more entries than Count. (#9048)
The decision to stop trimming due to LIMIT in XADD and XTRIM was after the limit was reached.
i.e. the code was deleting **at least** that count of records (from the LIMIT argument's perspective, not the MAXLEN),
instead of **up to** that count of records.
see #9046

(cherry picked from commit eaa7a7bb93c1ac6dbf347e1c29ac719a12a75158)
2021-07-21 21:06:49 +03:00
Huang Zhw
300b06f1f0 XTRIM call streamParseAddOrTrimArgsOrReply use wrong arg xadd. (#9047)
xtrimCommand call streamParseAddOrTrimArgsOrReply should use xadd==0.

When the syntax is valid, it does not cause any bugs because the params of XADD is superset of XTRIM.
Just XTRIM will not respond with error on invalid syntax. The syntax of XADD will also be accpeted by XTRIM.

(cherry picked from commit 91f3689bf5dc4ce3cf00d9d957b9677b362a205e)
2021-07-21 21:06:49 +03:00
YaacovHazan
4af7a28728 stabilize tests that involved with load handlers (#8967)
When test stop 'load handler' by killing the process that generating the load,
some commands that already in the input buffer, still might be processed by the server.
This may cause some instability in tests, that count on that no more commands
processed after we stop the `load handler'

In this commit, new proc 'wait_load_handlers_disconnected' added, to verify that no more
cammands from any 'load handler' prossesed, by checking that the clients who
genreate the load is disconnceted.

Also, replacing check of dbsize with wait_for_ofs_sync before comparing debug digest, as
it would fail in case the last key the workload wrote was an overridden key (not a new one).

Affected tests
Race fix:
- failover command to specific replica works
- Connect multiple replicas at the same time (issue #141), master diskless=$mdl, replica diskless=$sdl
- AOF rewrite during write load: RDB preamble=$rdbpre

Cleanup and speedup:
- Test replication with blocking lists and sorted sets operations
- Test replication with parallel clients writing in different DBs
- Test replication partial resync: $descr (diskless: $mdl, $sdl, reconnect: $reconnect

(cherry picked from commit 32a2584e079a1b3c2d1e6649e38239381a73a459)
2021-07-21 21:06:49 +03:00
perryitay
6ae3673b5e Fail EXEC command in case a watched key is expired (#9194)
There are two issues fixed in this commit: 
1. we want to fail the EXEC command in case there is a watched key that's logically
   expired but not yet deleted by active expire or lazy expire.
2. we saw that currently cache time is update in every `call()` (including nested calls),
   this time is being also being use for the isKeyExpired comparison, we want to update
   the cache time only in the first call (execCommand)

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit ac8b1df8850cc80fbf9ce8c2fbde0c1d3a1b4e91)
2021-07-21 21:06:49 +03:00
Huang Zhw
970302c00c Do not install a file event to send data to rewrite child when parent stop sending diff to child in aof rewrite. (#8767)
In aof rewrite, when parent stop sending data to child, if there is
new rewrite data, aofChildWriteDiffData write event will be installed.
Then this event is issued and deletes the file event without do anyting.
This will happen over and over again until aof rewrite finish.

This bug used to waste a few system calls per excessive wake-up
(epoll_ctl and epoll_wait) per cycle, each cycle triggered by receiving
a write command from a client.

(cherry picked from commit cb961d8c8e10ff3b619f1579a03336a15e9e6f45)
2021-07-21 21:06:49 +03:00
Omer Shadmi
3687f210a1 Avoid exiting to allow diskless loading to recover from RDB short read on module AUX data (#9199)
Currently a replica is able to recover from a short read (when diskless loading
is enabled) and avoid crashing/exiting, replying to the master and then the rdb
could be sent again by the master for another load attempt by the replica.
There were a few scenarios that were not behaving similarly, such as when
there is no end-of-file marker, or when module aux data failed to load, which
should be allowed to occur due to a short read.

(cherry picked from commit f06d782f5abcb30efb0117841232828ed3e129bf)
2021-07-21 21:06:49 +03:00
Oran Agra
f1ff72c7a3 Fix race in client side tracking (#9116)
The `Tracking gets notification of expired keys` test in tracking.tcl
used to hung in valgrind CI quite a lot.

It turns out the reason is that with valgrind and a busy machine, the
server cron active expire cycle could easily run in the same event loop
as the command that created `mykey`, so that when they key got expired,
there were two change events to broadcast, one that set the key and one
that expired it, but since we used raxTryInsert, the client that was
associated with the "last" change was the one that created the key, so
the NOLOOP filtered that event.

This commit adds a test that reproduces the problem by using lazy expire
in a multi-exec which makes sure the key expires in the same event loop
as the one that added it.

(cherry picked from commit 9b564b525d8ce88295ec14ffdc3bede7e5f5c33e)
2021-07-21 21:06:49 +03:00
Maxim Galushka
9562821f12 redis-cli: support for REDIS_REPLY_SET in CSV and RAW output. (#7338)
Fixes #6792. Added support of REDIS_REPLY_SET in raw and csv output of `./redis-cli`

Test:

run commands to test:
  ./redis-cli -3 --csv COMMAND
  ./redis-cli -3 --raw COMMAND

Now they are returning resuts, were failing with: "Unknown reply type: 10" before the change.

(cherry picked from commit 96bb078577ce2b0d093c873faae5d3ecca26a1de)
2021-07-21 21:06:49 +03:00
M Sazzadul Hoque
78d6597012 Fix 6.2.4 release month to June (#9027) 2021-06-01 19:03:06 +03:00
Oran Agra
8728f68f66 Redis 6.2.4 2021-06-01 17:03:36 +03:00
Oran Agra
e55cc0e9f4 Fix integer overflow in STRALGO LCS (CVE-2021-32625) (#9011)
An integer overflow bug in Redis version 6.0 or newer can be exploited using the
STRALGO LCS command to corrupt the heap and potentially result with remote code
execution. This is a result of an incomplete fix by CVE-2021-29477.

(cherry picked from commit 1ddecf1958924b178b76a31d989ef1e05af81964)
2021-06-01 17:03:36 +03:00
YaacovHazan
bd2785d18b unregister AE_READABLE from the read pipe in backgroundSaveDoneHandlerSocket (#8991)
In diskless replication, we create a read pipe for the RDB, between the child and the parent.
When we close this pipe (fd), the read handler also needs to be removed from the event loop (if it still registered).
Otherwise, next time we will use the same fd, the registration will be fail (panic), because
we will use EPOLL_CTL_MOD (the fd still register in the event loop), on fd that already removed from epoll_ctl

(cherry picked from commit 501d7755831527b4237f9ed6050ec84203934e4d)
2021-06-01 17:03:36 +03:00
Wen Hui
eb6d4da231 fix sentinel test failure (#8983)
fix for recent breakage from #8958, did a mistake when updating the pr.

(cherry picked from commit be6ce8a92a9acbecfaaa6c57a45037fc1018fefe)
2021-06-01 17:03:36 +03:00
Wen Hui
efb7a5c630 [SENTINEL] reset sentinel-user/pass to NULL when user config with empty string (#8958)
(cherry picked from commit ae6f58690b91010d003cdf5552d74b8e5b428d53)
2021-06-01 17:03:36 +03:00
Madelyn Olson
f8938d868b Hide migrate command from slowlog if they include auth (#8859)
Redact commands that include sensitive data from slowlog and monitor

(cherry picked from commit a59e75a475782d86d7ce2b5b9c6f5bb4a5ef0bd6)
2021-06-01 17:03:36 +03:00
patpatbear
2adf90348f sinterstore: add missing keyspace del event when any source set not exists. (#8949)
this patch fixes sinterstore by add missing keyspace del event when any source set not exists.

Co-authored-by: srzhao <srzhao@sysnew.com>
(cherry picked from commit 46d9f31e94355ec15b95418377677bcf75839bc9)
2021-06-01 17:03:36 +03:00
Oran Agra
eca5e75b2a Fix crash unlinking a stream with groups rax and no groups (#8932)
When estimating the effort for unlink, we try to compute the effort of
the first group and extrapolate.
If there's a groups rax that's empty, there'a an assertion.

reproduce:
xadd s * a b
xgroup create s bla $
xgroup destroy s bla
unlink s

(cherry picked from commit 97108845e2ae7661e5091c817cb03459ec81ea8c)
2021-06-01 17:03:36 +03:00
Oran Agra
f8e272f765 fix redis-benchmark to ignore unsupported configs (#8916)
Redis Enterprise supports the CONFIG GET command, but it replies with am
empty array since the save and appendonly configs are not supported.
before this fix redis-benchmark would segfault for trying to access the
error string on an array type reply.
see #8869

(cherry picked from commit 4d1094e8be3150b92b3e96d3a743c66b1a95988a)
2021-06-01 17:03:36 +03:00
Wang Yuan
435511f7c3 Fix wrong COW memory in log (#8917)
Always 0 MB of memory used by copy-on-write, introduced in #8645.

(cherry picked from commit 81e2d7272b784273099fecd85b15473277296771)
2021-06-01 17:03:36 +03:00
yoav-steinberg
7059cfb432 Enforce client output buffer soft limit when no traffic. (#8833)
When client breached the output buffer soft limit but then went idle,
we didn't disconnect on soft limit timeout, now we do.
Note this also resolves some sporadic test failures in due to Linux
buffering data which caused tests to fail if during the test we went
back under the soft COB limit.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: sundb <sundbcn@gmail.com>
(cherry picked from commit 152fce5e2cbf947a389da414a431f7331981a374)
2021-06-01 17:03:36 +03:00
Oran Agra
439c356fe6 Redis 6.2.3 2021-05-03 22:57:00 +03:00
Oran Agra
f72cad07e1 Resolve nonsense static analysis warnings
(cherry picked from commit fd7d51c353607f350c865155444bce9236f3d682)
2021-05-03 22:57:00 +03:00
Oran Agra
fe0345931c Fix integer overflow in STRALGO LCS (CVE-2021-29477)
An integer overflow bug in Redis version 6.0 or newer could be exploited using
the STRALGO LCS command to corrupt the heap and potentially result with remote
code execution.

(cherry picked from commit f0c5f920d0f88bd8aa376a2c05af4902789d1ef9)
2021-05-03 22:57:00 +03:00
Oran Agra
832b768248 Fix integer overflow in intset (CVE-2021-29478)
An integer overflow bug in Redis 6.2 could be exploited to corrupt the heap and
potentially result with remote code execution.

The vulnerability involves changing the default set-max-intset-entries
configuration value, creating a large set key that consists of integer values
and using the COPY command to duplicate it.

The integer overflow bug exists in all versions of Redis starting with 2.6,
where it could result with a corrupted RDB or DUMP payload, but not exploited
through COPY (which did not exist before 6.2).

(cherry picked from commit 29900d4e6bccdf3691bedf0ea9a5d84863fa3592)
2021-05-03 22:57:00 +03:00
sundb
23e126a993 Fix memory leak in moduleDefragGlobals (#8853)
(cherry picked from commit 5100ef9f8246dec6590f35f6b9f0b88c2dea0cfb)
2021-05-03 22:57:00 +03:00
Huang Zhw
02bd008bd1 Improve redis-cli help. When help command, we only match command (#8879)
prefix args not all args. So when we help commands with subcommands,
all subcommands will be output.

(cherry picked from commit 0b1b9edb2843730b03f78b6073cdd30873dbba95)
2021-05-03 22:57:00 +03:00
Binbin
0d10460235 redis-benchmark: Add zfree(data) and fix lrange size / text mismatch (#8872)
missing zfree(data) in redis-benchmark.

And also correct the wrong size in lrange.
the text mentioned 500, but size was 450, changed to 500

(cherry picked from commit 1eff8564c78011f7257e485796990a0d4d607a5b)
2021-05-03 22:57:00 +03:00
Binbin
e919deac12 redis-cli: Do not use hostsocket when we got redirected in cluster mode (#8870)
When redis-cli was used with both -c (cluster) and -s (unix socket),
it would have kept trying to use that unix socket, even if it got
redirected by the cluster (resulting in an infinite loop).

(cherry picked from commit 416f2773395ffcd72d8d8408e1558f49d59a0077)
2021-05-03 22:57:00 +03:00
filipe oliveira
ee542fbbd9 redis-benchmark: Error/Warning handling updates. (#8869)
- Immediately exit on errors that are not related to topology updates.
- Deprecates the `-e` option ( retro compatible ) and warns that we now
  exit immediately on errors that are not related to topology updates.
- Fixed wrongfully failing on config fetch error (warning only). This only affects RE.

Bottom line:
- MOVED and ASK errors will not show any warning (unlike the throttled error with `-e` before).
- CLUSTERDOWN still prints an error unconditionally and sleeps for 1 second.
- other errors are fatal.

(cherry picked from commit ef6f902372d4646b1894ec5dbd5f857dea5688d6)
2021-05-03 22:57:00 +03:00
Huang Zhw
9868ad801d Fix potential CONFIG SET bind test failure. (#8875)
Use an invalid IP address to trigger CONFIG SET bind failure, instead of DNS which is not guaranteed to always fail.

(cherry picked from commit 2b22fffc787e91df789dabf23ddcf19ecf34cf6f)
2021-05-03 22:57:00 +03:00
yoav-steinberg
2149c5a030 Bump freebsd-vm version to fix CI failures (#8876)
Specifically we had issues with NTP sync failure which was resolved here: 457af73456

(cherry picked from commit 2e88b0639689a3019e27f55dfa40578847443eeb)
2021-05-03 22:57:00 +03:00
Oran Agra
91c450e80a Prevent replicas from sending commands that interact with keyspace (#8868)
This solves an issue reported in #8712 in which a replica would bypass
the client write pause check and cause an assertion due to executing a
write command during failover.

The fact is that we don't expect replicas to execute any command other
than maybe REPLCONF and PING, etc. but matching against the ADMIN
command flag is insufficient, so instead i just block keyspace access
for now.

(cherry picked from commit 46f4ebbe842620f0976a36741a72482620aa4b48)
2021-05-03 22:57:00 +03:00
Yossi Gottlieb
9d71ac0a6e Remove redundant -latomic on arm64. (#8867)
(cherry picked from commit ebfbb091096b6f36cf82e9f6e6583b10fd5b5acb)
2021-05-03 22:57:00 +03:00
Yang Bodong
ce288cb559 When the password is wrong, redis-benchmark should exit (#8855)
(cherry picked from commit 8423b77f14c0d3a58e580c65a70b4f980f5cdcf6)
2021-05-03 22:57:00 +03:00
Istemi Ekin Akkus
9158a510e1 Modules: Fix RM_GetClusterNodeInfo() to correctly populate the master_id (#8846)
(cherry picked from commit af035c1e1d3bcaf662051cff4dc49f6051321c9c)
2021-05-03 22:57:00 +03:00
zyxwvu Shi
7ac26c3497 Use monotonic clock to check for Lua script timeout. (#8812)
This prevents a case where NTP moves the system clock
forward resulting in a false detection of a busy script.

Signed-off-by: zyxwvu Shi <i@shiyc.cn>
(cherry picked from commit f61c37cec900ba391541f20f7655aad44a26bafc)
2021-05-03 22:57:00 +03:00
Wang Yuan
3b9ceeb2a4 Expire key firstly and then notify keyspace event (#8830)
Modules event subscribers may get wrong things in notifyKeyspaceEvent callback,
such as wrong number of keys, or be able to lookup this key.
This commit changes the order to be like the one in evict.c.

Cleanup:
Since we know the key exists (it expires now), db*Delete is sure to return 1,
so there's no need to check it's output (misleading).

(cherry picked from commit 63acfe4b00b9d3e34a53559f965c0bc44c03db61)
2021-05-03 22:57:00 +03:00
bugwz
d7221e0135 Print the number of abnormal line in AOF (#8823)
When redis-check-aof finds an error, it prints the line number for faster troubleshooting. 

(cherry picked from commit 761d7d27711edfbf737def41ff28f5b325fb16c8)
2021-05-03 22:57:00 +03:00
Huang Zhw
5ce259333e Fix migrateCommand may migrate wrong value. (#8815)
This scene is hard to happen. When first attempt some keys expired,
only kv position is updated not ov. Then socket err happens, second
attempt is taken. This time kv items may be mismatching with ov items.

(cherry picked from commit 080d4579db40d965f8392af5b1da7a99d1a817d5)
2021-05-03 22:57:00 +03:00
Madelyn Olson
3e97074649 Fix memory leak when doing lazyfreeing client tracking table (#8822)
Interior rax pointers were not being freed

(cherry picked from commit c73b4ddfd96d00ed0d0fde17953ce63d78bc3777)
2021-05-03 22:57:00 +03:00
Oran Agra
87c5cf52df Merge 6.2.2 release
Release 6.2.2
2021-04-20 08:03:58 +03:00
Oran Agra
e0049d881d Redis 6.2.2 2021-04-19 21:39:40 +03:00
Oran Agra
9b13be9b59 Merge unstable into 6.2 2021-04-19 21:36:00 +03:00
Viktor Söderqvist
9e70361abe Fix space included in links in modules API doc (#8818) 2021-04-19 21:35:14 +03:00