286 Commits

Author SHA1 Message Date
Malavan Sotheeswaran
dc419a4e6e
keys after brpop causes crash due to timeout not being reset (#662) 2023-05-30 01:01:51 -04:00
chendianqiang
4c200e0985 fix hincrbyfloat not to create a key if the new value is invalid (#11149)
Check the validity of the value before performing the create operation,
prevents new data from being generated even if the request fails to execute.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: chendianqiang <chendianqiang@meituan.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2023-04-18 17:08:27 -07:00
Oran Agra
d4c0a970e7 Integer Overflow in RAND commands can lead to assertion (CVE-2023-25155) (#11857)
Issue happens when passing a negative long value that greater than
the max positive value that the long can store.
2023-04-18 17:08:27 -07:00
Malavan Sotheeswaran
ad0be5666c
Security fixes from redis: Avoid integer overflows in SETRANGE and SORT, Fix range issues in ZRANDMEMBER and HRANDFIELD (#547)
* Avoid integer overflows in SETRANGE and SORT (CVE-2022-35977) (#11720)

Authenticated users issuing specially crafted SETRANGE and SORT(_RO)
commands can trigger an integer overflow, resulting with Redis attempting
to allocate impossible amounts of memory and abort with an OOM panic.

* Fix range issues in ZRANDMEMBER and HRANDFIELD (CVE-2023-22458) (#11674)

missing range check in ZRANDMEMBER and HRANDIFLD leading to panic due
to protocol limitations

* use std::min/max

* add assert_not_equal

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-01-17 17:10:39 -05:00
Malavan Sotheeswaran
f5f1bd7605
Merge main with oss release sep29 2022 (#521)
* need to include stdint for uintptr_t

* need to include stdint for uintptr_t

* use atomic_load for g_pserver->mstime

* use atomic_load for g_pserver->mstime

* Integrate readwritelock with Pro Code

* Integrate readwritelock with Pro Code

* Defensive asserts for RWLock

* Defensive asserts for RWLock

* Save and restore master info in rdb to allow active replica partial sync (#371)

* save replid for all masters in rdb

* expanded rdbSaveInfo to hold multiple master structs

* parse repl-masters from rdb

* recover replid info from rdb in active replica mode, attempt partial sync

* save offset from rdb into correct variable

* don't change replid based on master in active rep

* save and load psync info from correct fields

* Save and restore master info in rdb to allow active replica partial sync (#371)

* save replid for all masters in rdb

* expanded rdbSaveInfo to hold multiple master structs

* parse repl-masters from rdb

* recover replid info from rdb in active replica mode, attempt partial sync

* save offset from rdb into correct variable

* don't change replid based on master in active rep

* save and load psync info from correct fields

* placement new instead of memcpy

* placement new instead of memcpy

* Remove asserts, RW lock can go below zero in cases of aeAcquireLock

* Remove asserts, RW lock can go below zero in cases of aeAcquireLock

* Inclusive language

* Inclusive language

* update packaging for OS merge

* update packaging for OS merge

* modify dockerfile to build within image

* modify dockerfile to build within image

* Make active client balancing a configurable option

* Make active client balancing a configurable option

* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior

* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior

* Only run the tls-name-validation test if --tls is passed into runtest

* Only run the tls-name-validation test if --tls is passed into runtest

* Fix KeyDB not building with TLS < 1.1.1

* Fix KeyDB not building with TLS < 1.1.1

* update changelog to use replica as terminology

* update changelog to use replica as terminology

* update copyright

* update copyright

* update deb copyright

* update deb copyright

* call aeThreadOnline() earlier

* call aeThreadOnline() earlier

* Removed mergeReplicationId

* Removed mergeReplicationId

* acceptTLS is threadsafe like the non TLS version

* acceptTLS is threadsafe like the non TLS version

* setup Machamp ci

* setup Machamp ci

* make build_test.sh executable

* make build_test.sh executable

* PSYNC production fixes

* PSYNC production fixes

* fix the Machamp build

* fix the Machamp build

* break into tests into steps

* break into tests into steps

* Added multimaster test

* Added multimaster test

* Update ci.yml

Change min tested version to 18.04

* Update ci.yml

Change min tested version to 18.04

* fork lock for all threads, use fastlock for readwritelock

* fork lock for all threads, use fastlock for readwritelock

* hide forklock object in ae

* hide forklock object in ae

* only need to include readwritelock in ae

* only need to include readwritelock in ae

* time thread lock uses fastlock instead of std::mutex

* time thread lock uses fastlock instead of std::mutex

* set thread as offline when waiting for time thread lock

* set thread as offline when waiting for time thread lock

* update README resource links

* update README resource links

* Fix MALLOC=memkind build issues

* Fix MALLOC=memkind build issues

* Fix module test break

* Fix module test break

* Eliminate firewall dialogs on mac for regular and cluster tests.  There are still issues with the sentinel tests but attempting to bind only to localhost causes failures

* Eliminate firewall dialogs on mac for regular and cluster tests.  There are still issues with the sentinel tests but attempting to bind only to localhost causes failures

* remove unused var in networking.cpp

* remove unused var in networking.cpp

* check ziplist len to avoid crash on empty ziplist convert

* check ziplist len to avoid crash on empty ziplist convert

* remove nullptr subtraction

* remove nullptr subtraction

* cannot mod a pointer

* cannot mod a pointer

* need to include stdint for uintptr_t

* need to include stdint for uintptr_t

* use atomic_load for g_pserver->mstime

* use atomic_load for g_pserver->mstime

* Integrate readwritelock with Pro Code

* Integrate readwritelock with Pro Code

* Defensive asserts for RWLock

* Defensive asserts for RWLock

* Save and restore master info in rdb to allow active replica partial sync (#371)

* save replid for all masters in rdb

* expanded rdbSaveInfo to hold multiple master structs

* parse repl-masters from rdb

* recover replid info from rdb in active replica mode, attempt partial sync

* save offset from rdb into correct variable

* don't change replid based on master in active rep

* save and load psync info from correct fields

* Save and restore master info in rdb to allow active replica partial sync (#371)

* save replid for all masters in rdb

* expanded rdbSaveInfo to hold multiple master structs

* parse repl-masters from rdb

* recover replid info from rdb in active replica mode, attempt partial sync

* save offset from rdb into correct variable

* don't change replid based on master in active rep

* save and load psync info from correct fields

* placement new instead of memcpy

* placement new instead of memcpy

* Remove asserts, RW lock can go below zero in cases of aeAcquireLock

* Remove asserts, RW lock can go below zero in cases of aeAcquireLock

* Inclusive language

* Inclusive language

* call aeThreadOnline() earlier

* call aeThreadOnline() earlier

* Removed mergeReplicationId

* Removed mergeReplicationId

* Make active client balancing a configurable option

* Make active client balancing a configurable option

* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior

* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior

* acceptTLS is threadsafe like the non TLS version

* acceptTLS is threadsafe like the non TLS version

* PSYNC production fixes

* PSYNC production fixes

* Ensure we are responsive during storagecache clears

* Ensure we are responsive during storagecache clears

* Ensure recreated tables use the same settings as ones made at boot

* Ensure recreated tables use the same settings as ones made at boot

* Converted some existing PSYNC tests for multimaster

* Converted some existing PSYNC tests for multimaster

* Inclusive language fix

* Inclusive language fix

* Cleanup test suite

* Cleanup test suite

* Updated test replica configs so tests make sense

* Updated test replica configs so tests make sense

* active-rep test reliability

* active-rep test reliability

* Quick fix to make psync tests work

* Quick fix to make psync tests work

* Fix PSYNC test crashes

* Fix PSYNC test crashes

* Ensure we force moves not copies when ingesting bulk insert files

* Ensure we force moves not copies when ingesting bulk insert files

* Disable async for hget commands as it is not ready

* Disable FLASH

* Fix crash in save of masterinfo

* Fix musl/Alpine build failures

* Remove unnecessary libs

* update readme

* update readme

* remove Enterprise references

* Limit max overage to 20% during RDB save

* Delete COPYING to replace with BSD license

* update deb master changelog

* Update license

* Fix Readme typo from github org transition

Replace mention of scratch-file-path with db-s3-object

* Fix reference counting failure in the dict.  This is caused by std::swap also swapping refcounts

* Fix assertion in async rehash

* Prevent crash on shutdown by avoiding dtors (they are unnecessary anyways)

* Initialize noshrink, it was dangling

* Prevent us from starting a rehash when one wasn't already in progress.  This can cause severe issues for snapshots

* Avoid unnecessary rehashing when a rehash is abandoned

* Dictionary use correct acquire/release semantics

* Add fence barriers for the repl backlog (important for AARCH64 and other weak memory models)

* Silence TSAN errors on ustime and mstime.  Every CPU we support is atomic on aligned ints, but correctness matters

* Disable async commands by default

* Fix TSAN warnings on the repl backlog

* Merge OSS back into pro

* Fix unmerged files

* Fix O(n^2) algorithm in the GC cleanup logic

* Fix crash in expire when a snapshot is in flight.  Caused by a perf optimization getting the expire map out of sync with the val

* On Alpine we must have a reasonable stack size

* Revert ci.yml to unstable branch version

* Implements the soft shutdown feature to allow clients to cooperatively disconnect preventing disruption during shutdown

* Ensure clean shutdown with multiple threads

* update dockerfiles

* update deb pkg references and changelog

* update gem reference

* lpGetInteger returns int64_t, avoid overflow (#10068)

Fix #9410

Crucial for the ms and sequence deltas, but I changed all
calls, just in case (e.g. "flags")

Before this commit:
`ms_delta` and `seq_delta` could have overflown, causing `currid` to be wrong,
which in turn would cause `streamTrim` to trim the entire rax node (see new test)

* Fix issue #454 (BSD build break)

* Do not allow commands to run in background when in eval, Issue #452

* Fix certificate leak during connection when tls-allowlists are used

* Fix issue #480

* Fix crash running INFO command while a disk based backlog is set

* check tracking per db

* fix warnings

* Fix a race when undoConnectWithMaster changes mi->repl_transfer_s but the connection is not yet closed and the event handler runs

* Fix a race in processChanges/trackChanges with rdbLoadRio by acquiring the lock when trackChanges is set

* Fix ASAN use after free

* Additional fixes

* Fix integer overflow of the track changes counter

* Fix P99 latency issue for TLS where we leave work for the next event loop

tlsProcessPendingData() needs to be called before we execute queued commands because it may enqueue more commands

* Fix race removing key cache

* Prevent crash on load in long running KeyDB instances

* Fixes a crash where the server assertion failed when the key exists in DB during RDB load

* Remove old assertion which is commented out.

* avoid from instatiating EpochHolder multiple times to improve performance and cpu utilization

* avoid from instatiating EpochHolder multiple times to improve performance and cpu utilization

* src\redis-cli.c: fix potential null pointer dereference found by cppcheck

src\redis-cli.c:5488:35: warning: Either the condition
'!table' is redundant or there is possible null pointer dereference:
table. [nullPointerRedundantCheck]

* Fix Issue #486

* Workaround bug in snapshot sync - abort don't crash

* Improve reliability of async parts of the soft shutdown tests

* Improve reliability of fragmentation tests

* Verify that partial syncs do indeed occur

* Fix O(n) algorithm in INFO command

* Remove incorrect assert that fires when the repl backlog is used fully

* Make building flash optional

* Remove unneeded gitlab CI file

* [BUG] Moves key to another DB, the source key was removed if the move failed due to the key exists in the destination db #497 (#498)

Co-authored-by: Paul Chen <mingchen@Mings-MacBook-Pro.local>

* trigger repl_curr_off!= master_repl_offset assert failure when having pending write case

* use debug for logging the message instead

* rocksdb log using up the diskspace on flash (#519)

* Fix OpenSSL 3.0.x related issues. (#10291)

* Drop obsolete initialization calls.
* Use decoder API for DH parameters.
* Enable auto DH parameters if not explicitly used, which should be the
  preferred configuration going forward.

* remove unnecessary forward declaration

* remove internal ci stuff

* remove more internal ci/publishing

* submodule update step

* use with syntax instead

* bump ci ubuntu old ver as latest is now 22.04

* include submodules on all ci jobs

* install all deps for all ci jobs

Co-authored-by: Vivek Saini <vsaini@snapchat.com>
Co-authored-by: Christian Legge <christian@eqalpha.com>
Co-authored-by: benschermel <bschermel@snapchat.com>
Co-authored-by: John Sully <john@csquare.ca>
Co-authored-by: zliang <zliang@snapchat.com>
Co-authored-by: malavan <malavan@eqalpha.com>
Co-authored-by: John Sully <jsully@snapchat.com>
Co-authored-by: jfinity <38383673+jfinity@users.noreply.github.com>
Co-authored-by: benschermel <43507366+benschermel@users.noreply.github.com>
Co-authored-by: guybe7 <guy.benoish@redislabs.com>
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
Co-authored-by: root <paul.chen1@huawei.com>
Co-authored-by: Ilya Shipitsin <chipitsine@gmail.com>
Co-authored-by: Paul Chen <32553156+paulmchen@users.noreply.github.com>
Co-authored-by: Paul Chen <mingchen@Mings-MacBook-Pro.local>
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2022-12-14 12:17:36 -05:00
VivekSainiEQ
1d882b5ddd Merge tag '6.2.6' into Redis_626_Merge
Former-commit-id: e6d7e01be6965110d487e12f40511fe0b3497695
2021-10-21 22:33:55 +00:00
Binbin
b622537199 SMOVE only notify dstset when the addition is successful. (#9244)
in case dest key already contains the member, the dest key isn't modified, so the command shouldn't invalidate watch.

(cherry picked from commit 11dc4e59b365d6cd8699604d7d1c1025b6bb6259)
2021-07-21 21:06:49 +03:00
Binbin
88655019cc Fix accidental deletion of sinterstore command when we meet wrong type error. (#9032)
SINTERSTORE would have deleted the dest key right away,
even when later on it is bound to fail on an (WRONGTYPE) error.

With this change it first picks up all the input keys, and only later
delete the dest key if one is empty.

Also add more tests for some commands.
Mainly focus on
- `wrong type error`:
	expand test case (base on sinter bug) in non-store variant
	add tests for store variant (although it exists in non-store variant, i think it would be better to have same tests)
- the dstkey result when we meet `non-exist key (empty set)` in *store

sdiff:
- improve test case about wrong type error (the one we found in sinter, although it is safe in sdiff)
- add test about using non-exist key (treat it like an empty set)
sdiffstore:
- according to sdiff test case, also add some tests about `wrong type error` and `non-exist key`
- the different is that in sdiffstore, we will consider the `dstkey` result

sunion/sunionstore add more tests (same as above)

sinter/sinterstore also same as above ...

(cherry picked from commit b8a5da80c49501773f8778aaf5cbf595cef615e4)
2021-07-21 21:06:49 +03:00
Jason Elbaum
fad44611dc Change return value type for ZPOPMAX/MIN in RESP3 (#8981)
When using RESP3, ZPOPMAX/ZPOPMIN should return nested arrays for consistency
with other commands (e.g. ZRANGE).

We do that only when COUNT argument is present (similarly to how LPOP behaves).
for reasoning see https://github.com/redis/redis/issues/8824#issuecomment-855427955

This is a breaking change only when RESP3 is used, and COUNT argument is present!

(cherry picked from commit 7f342020dcbdf9abe754d6b666efdeded7063870)
2021-07-21 21:06:49 +03:00
Binbin
c6b3966d02 hrandfield and zrandmember with count should return emptyarray when key does not exist. (#9178)
due to a copy-paste bug, it used to reply with null response rather than empty array.
this commit includes new tests that are looking at the RESP response directly in
order to be able to tell the difference between them.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit a418a2d3fc0250c094802d7e8ea64d96eedfda07)
2021-07-21 21:06:49 +03:00
Leibale Eidelman
095a6e5937 fix ZRANGESTORE - should return 0 when src points to an empty key (#9089)
mistakenly it used to return an empty array rather than 0.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 95274f1f8a3ef4cb4033beecfaa99ea1439ed170)
2021-07-21 21:06:49 +03:00
Binbin
11235c0e40 ZRANDMEMBER WITHSCORES with negative COUNT may return bad score (#9162)
Return a bad score when used with negative count (or count of 1), and non-ziplist encoded zset.
Also add test to validate the return value and cover the issue.

(cherry picked from commit 4bc5a8324d3cb23ed2cc8a9cd19444a893a6d52c)
2021-07-21 21:06:49 +03:00
Huang Zhw
e6df2f6210 Fix XTRIM or XADD with LIMIT may delete more entries than Count. (#9048)
The decision to stop trimming due to LIMIT in XADD and XTRIM was after the limit was reached.
i.e. the code was deleting **at least** that count of records (from the LIMIT argument's perspective, not the MAXLEN),
instead of **up to** that count of records.
see #9046

(cherry picked from commit eaa7a7bb93c1ac6dbf347e1c29ac719a12a75158)
2021-07-21 21:06:49 +03:00
John Sully
5267928381 Merge tag '6.2.2' into unstable
Former-commit-id: 93ebb31b17adec5d406d2e30a5b9ea71c07fce5c
2021-05-21 05:54:39 +00:00
John Sully
fe8efa916b Merge tag '6.2.1' into unstable
Former-commit-id: bfed57e3e0edaa724b9d060a6bb8edc5a6de65fa
2021-05-19 02:59:48 +00:00
Yang Bodong
4c14e8668c
Fix out of range confusing error messages (XAUTOCLAIM, RPOP count) (#8746)
Fix out of range error messages to be clearer (avoid mentioning 9223372036854775807)
* Fix XAUTOCLAIM COUNT option confusing error msg
* Fix other RPOP and alike error message to mention positive
2021-04-07 10:01:28 +03:00
Valentino Geron
44d8b039e8
Fix XAUTOCLAIM response to return the next available id as the cursor (#8725)
This command used to return the last scanned entry id as the cursor,
instead of the next one to be scanned.
so in the next call, the user could / should have sent `(cursor` and not
just `cursor` if he wanted to avoid scanning the same record twice.

Scanning the record twice would look odd if someone is checking what
exactly was scanned, but it also has a side effect of incrementing the
delivery count twice.
2021-04-01 12:13:55 +03:00
guybe7
843f769b96
zsetAdd: Fix wrong reply in case of INCR and GT/LT (#8717)
If GT/LT fails the operation we need to reply with
nill (like failure due to NX).

Other changes:
Add the missing $encoding suffix to many zset tests

Note: there's a behavior change just in case of INCR + GT/LT that fails.
The old code was replying with the wrong (rejected) score, and now it'll reply with nil.

Note that that's anyway a corner case so this "behavior change" shouldn't have too much affect.
Using GT/LT with INCR has a predictable result even before we run the command
(INCR GT will only only / always fail if the increment is negative).
2021-04-01 09:33:53 +03:00
sundb
569a3f4548
Use chi-square for random distributivity verification in test (#8709)
Problem:
Currently, when performing random distribution verification, we determine
the probability of each element occurring in the sum, but the probability is
only an estimate, these tests had rare sporadic failures, and we cannot verify
what the probability of failure will be.

Solution:
Using the chi-square distribution instead of the original random distribution
validation makes the test more reasonable and easier to find problems.
2021-04-01 08:20:15 +03:00
sundb
18ac41973b
RAND* commands: fix risk of OOM panic in hash and zset, use fair random in hash, and add tests for even distribution to all (#8429)
Changes to HRANDFIELD and ZRANDMEMBER:
* Fix risk of OOM panic when client query a very big negative count (avoid allocating huge temporary buffer).
* Fix uneven random distribution in HRANDFIELD with negative count (wasn't using dictGetFairRandomKey).
* Add tests to check an even random distribution (HRANDFIELD, SRANDMEMBER, ZRANDMEMBER).

Co-authored-by: Oran Agra <oran@redislabs.com>
2021-02-05 15:56:20 +02:00
Yang Bodong
b9a0500f16
Add HRANDFIELD and ZRANDMEMBER. improvements to SRANDMEMBER (#8297)
New commands:
`HRANDFIELD [<count> [WITHVALUES]]`
`ZRANDMEMBER [<count> [WITHSCORES]]`
Algorithms are similar to the one in SRANDMEMBER.

Both return a simple bulk response when no arguments are given, and an array otherwise.
In case values/scores are requested, RESP2 returns a long array, and RESP3 a nested array.
note: in all 3 commands, the only option that also provides random order is the one with negative count.

Changes to SRANDMEMBER
* Optimization when count is 1, we can use the more efficient algorithm of non-unique random
* optimization: work with sds strings rather than robj

Other changes:
* zzlGetScore: when zset needs to convert string to double, we use safer memcpy (in
  case the buffer is too small)
* Solve a "bug" in SRANDMEMBER test: it intended to test a positive count (case 3 or
  case 4) and by accident used a negative count

Co-authored-by: xinluton <xinluton@qq.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-01-29 10:47:28 +02:00
Raghav Muddur
0367a80819
GETEX, GETDEL and SET PXAT/EXAT (#8327)
This commit introduces two new command and two options for an existing command

GETEX <key> [PERSIST][EX seconds][PX milliseconds] [EXAT seconds-timestamp]
[PXAT milliseconds-timestamp]

The getexCommand() function implements extended options and variants of the GET
command. Unlike GET command this command is not read-only. Only one of the options
can be used at a given time.

1. PERSIST removes any TTL associated with the key.
2. EX Set expiry TTL in seconds.
3. PX Set expiry TTL in milliseconds.
4. EXAT Same like EX instead of specifying the number of seconds representing the
    TTL (time to live), it takes an absolute Unix timestamp
5. PXAT Same like PX instead of specifying the number of milliseconds representing the
    TTL (time to live), it takes an absolute Unix timestamp

Command would return either the bulk string, error or nil.

GETDEL <key>
Would delete the key after getting.

SET key value [NX] [XX] [KEEPTTL] [GET] [EX <seconds>] [PX <milliseconds>]
[EXAT <seconds-timestamp>][PXAT <milliseconds-timestamp>]

Two new options added here are EXAT and PXAT

Key implementation notes
- `SET` with `PX/EX/EXAT/PXAT` is always translated to `PXAT` in `AOF`. When relative time is
  specified (`PX/EX`), replication will always use `PX`.
- `setexCommand` and `psetexCommand` would no longer need translation in `feedAppendOnlyFile`
  as they are modified to invoke `setGenericCommand ` with appropriate flags which will take care of
  correct AOF translation.
- `GETEX` without any optional argument behaves like `GET`.
- `GETEX` command is never propagated, It is either propagated as `PEXPIRE[AT], or PERSIST`.
- `GETDEL` command is propagated as `DEL`
- Combined the validation for `SET` and `GETEX` arguments. 
- Test cases to validate AOF/Replication propagation
2021-01-27 19:47:26 +02:00
Oran Agra
9e56d3969a
Add tests for RESP3 responce of ZINTER and ZRANGE (#8391)
It was confusing as to why these don't return a map type.
the reason is that order matters, so we need to make sure the client
library knows to respect it.
Added comments in the implementation and tests to cover it.
2021-01-26 17:55:32 +02:00
guybe7
814aad65f1
XADD and XTRIM, Trim by MINID, and new LIMIT argument (#8169)
This PR adds another trimming strategy to XADD and XTRIM named MINID
(complements the existing MAXLEN).
It also adds a new LIMIT argument that allows incremental trimming by repeated
calls (rather than all at once).

This provides the ability to trim all records older than a certain ID (which makes it
possible for the user to trim by age too).
Example:
XTRIM mystream MINID ~ 1608540753 will trim entries with id < 1608540753,
but might not trim all (because of the ~ modifier)

The purpose is to ease the use of streams. many users use streams as logs and
the common case is wanting a log
of the last X seconds rather than a log that contains maximum X entries (new
MINID vs existing MAXLEN)

The new LIMIT modifier is only supported when the trim strategy uses ~.
i.e. when the user asked for exact trimming, it all happens in one go (no
possibility for incremental trimming).
However, when ~ is provided, we trim full rax nodes, up to the limit number
of records.
The default limit is 100*stream_node_max_entries (used when LIMIT is not
provided).
I.e. this is a behavior change (even if the existing MAXLEN strategy is used).
An explicit limit of 0 means unlimited (but note that it's not the default).

Other changes:

Refactor arg parsing code for XADD and XTRIM to use common code.
2021-01-08 18:13:25 +02:00
Jonah H. Harris
b5029dfdad
Add ZRANGESTORE command, and improve ZSTORE command (#7844)
Add ZRANGESTORE command, and improve ZSTORE command to deprecated Z[REV]RANGE[BYSCORE|BYLEX].

Syntax for the new ZRANGESTORE command:
ZRANGESTORE [BYSCORE | BYLEX] [REV] [LIMIT offset count]

New syntax for ZRANGE:
ZRANGE [BYSCORE | BYLEX] [REV] [WITHSCORES] [LIMIT offset count]

Old syntax for ZRANGE:
ZRANGE [WITHSCORES]

Other ZRANGE commands remain unchanged.

The implementation uses common code for all of these, by utilizing a consumer interface that in one
command response to the client, and in the other command stores a zset key.

Co-authored-by: Oran Agra <oran@redislabs.com>
2021-01-07 10:58:53 +02:00
guybe7
714e103ac3
Add XAUTOCLAIM (#7973)
New command: XAUTOCLAIM <key> <group> <consumer> <min-idle-time> <start> [COUNT <count>] [JUSTID]

The purpose is to claim entries from a stale consumer without the usual
XPENDING+XCLAIM combo which takes two round trips.

The syntax for XAUTOCLAIM is similar to scan: A cursor is returned (streamID)
by each call and should be used as start for the next call. 0-0 means the scan is complete.

This PR extends the deferred reply mechanism for any bulk string (not just counts)

This PR carries some unrelated test code changes:
- Renames the term "client" into "consumer" in the stream-cgroups test
- And also changes DEBUG SLEEP into "after"

Co-authored-by: Oran Agra <oran@redislabs.com>
2021-01-06 10:34:27 +02:00
Itamar Haber
f44186e575
Adds count to L/RPOP (#8179)
Adds: `L/RPOP <key> [count]`

Implements no. 2 of the following strategies:

1. Loop on listTypePop - this would result in multiple calls for memory freeing and allocating (see 769167a079)
2. Iterate the range to build the reply, then call quickListDelRange - this requires two iterations and **is the current choice**
3. Refactor quicklist to have a pop variant of quickListDelRange - probably optimal but more complex

Also:
* There's a historical check for NULL after calling listTypePop that was converted to an assert.
* This refactors common logic shared between LRANGE and the new form of LPOP/RPOP into addListRangeReply (adds test for b/w compat)
* Consequently, it may have made sense to have `LRANGE l -1 -2` and `LRANGE l 9 0` be legit and return a reverse reply. Due to historical reasons that would be, however, a breaking change.
* Added minimal comments to existing commands to adhere to the style, make core dev life easier and get commit karma, naturally.
2020-12-25 21:49:24 +02:00
John Sully
367067c98a Prevent test crash due to unimplemented save for nested hash
Former-commit-id: 7e0077f293c5f91f240ab401e5f167e487afbfe0
2020-12-11 00:23:44 +00:00
Wang Yuan
1acc315cea
Minor improvements for list-2 test (#8156)
had some unused variables.
now some are used to assert that they match, others were useless.
2020-12-08 16:26:38 +02:00
Itamar Haber
37f45d9e56
Adds exclusive range query intervals to XPENDING (#8130) 2020-12-08 11:43:00 +02:00
guybe7
6bb5503524
More efficient self-XCLAIM (#8098)
when the same consumer re-claim an entry that it already has, there's
no need to remove-and-insert if it's the same rax.
we do need to update the idle time though.
this commit only improves efficiency (doesn't change behavior).
2020-12-07 21:31:35 +02:00
Oran Agra
01c13bddea Sanitize dump payload: improve tests of ziplist and stream encodings
- improve stream rdb encoding test to include more types of stream metadata
- add test to cover various ziplist encoding entries (although it does
  look like the stress test above it is able to find some too
- add another test for ziplist encoding for hash with full sanitization
- add similar ziplist encoding tests for list
2020-12-06 14:54:34 +02:00
Itamar Haber
441c490024
Adds exclusive ranges to X[REV]RANGE (#8072)
Adds the ability to use exclusive (open) start and end query intervals in XRANGE and XREVRANGE queries.

Fixes #6562
2020-12-03 14:36:48 +02:00
Madelyn Olson
69b7113bb5
Getset fix (#8118)
* Fixed SET GET executing on wrong type

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2020-12-01 11:46:45 -08:00
guybe7
ada2ac9ae2
XPENDING with IDLE (#7972)
Used to filter stream pending entries by their idle-time,
useful for XCLAIMing entries that have not been processed
for some time
2020-11-29 12:08:47 +02:00
Itamar Haber
0963edbc65
Adds tests for XADD/XTRIM's MAXLEN arguments (#8083) 2020-11-23 14:37:58 +02:00
thomaston
39f716a121
ZREVRANGEBYSCORE Optimization for out of range offset (#5773)
ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
When the offset is too large, the query is very slow. Especially when the offset is greater than the length of zset it is easy to determine whether the offset is greater than the length of zset at first, and If it exceed the length of zset, then return directly.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-17 18:35:56 +02:00
Felipe Machado
d8fd48c436
Add new commands ZDIFF and ZDIFFSTORE (#7961)
- Add ZDIFF and ZDIFFSTORE which work similarly to SDIFF and SDIFFSTORE
- Make sure the new WITHSCORES argument that was added for ZUNION isn't considered valid for ZUNIONSTORE

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-15 14:14:25 +02:00
John Sully
5de36840ad Add missing file
Former-commit-id: 97fd73692363e05b80e0c0e84b7b2b8fe7f760b8
2020-11-11 03:29:38 +00:00
Wen Hui
f328194d12
support NOMKSTREAM option in xadd command (#7910)
introduces a NOMKSTREAM option for xadd command, this would be useful for some
use cases when we do not want to create new stream by default:

XADD key [MAXLEN [~|=] <count>] [NOMKSTREAM] <ID or *> [field value] [field value]
2020-10-18 10:15:43 +03:00
Felipe Machado
c3f9e01794
Adds new pop-push commands (LMOVE, BLMOVE) (#6929)
Adding [B]LMOVE <src> <dst> RIGHT|LEFT RIGHT|LEFT. deprecating [B]RPOPLPUSH.

Note that when receiving a BRPOPLPUSH we'll still propagate an RPOPLPUSH,
but on BLMOVE RIGHT LEFT we'll propagate an LMOVE

improvement to existing tests
- Replace "after 1000" with "wait_for_condition" when wait for
  clients to block/unblock.
- Add a pre-existing element to target list on basic tests so
  that we can check if the new element was added to the correct
  side of the list.
- check command stats on the replica to make sure the right
  command was replicated

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-10-08 08:33:17 +03:00
Nykolas Laurentino de Lima
66ee45b65c
Add GET parameter to SET command (#7852)
Add optional GET parameter to SET command in order to set a new value to
a key and retrieve the old key value. With this change we can deprecate
`GETSET` command and use only the SET command with the GET parameter.
2020-10-02 15:07:19 +03:00
John Sully
4f18a247e3 Merge tag '6.0.8' into unstable
Former-commit-id: 4c7e4b91a6bb2034636856b608b8c386d07f5541
2020-09-30 19:47:55 +00:00
valentinogeron
795c454db1
Stream: Inconsistency between master and replica some XREADGROUP case (#7526)
XREADGROUP auto-creates the consumer inside the consumer group the
first time it saw it.
When XREADGROUP is being used with NOACK option, the message will not
be added into the client's PEL and XGROUP SETID would be propagated.
When the replica gets the XGROUP SETID it will only update the last delivered
id of the group, but will not create the consumer.

So, in this commit XGROUP CREATECONSUMER is being added.
Command pattern: XGROUP CREATECONSUMER <key> <group> <consumer>.

When NOACK option is being used, createconsumer command would be
propagated as well.

In case of AOFREWRITE, consumer with an empty PEL would be saved with
XGROUP CREATECONSUMER whereas consumer with pending entries would be
saved with XCLAIM
2020-09-24 12:02:40 +03:00
bodong.ybd
e08bf16637 Add ZINTER/ZUNION command
Syntax: ZINTER/ZUNION numkeys key [key ...] [WEIGHTS weight [weight ...]]
[AGGREGATE SUM|MIN|MAX] [WITHSCORES]

see #7624
2020-09-24 08:59:14 +03:00
alexronke-channeladvisor
66a13ccbdf
Add GT and LT options to ZADD for conditional score updates (#7818)
Co-authored-by: Alex Ronke <w.alex.ronke@gmail.com>
2020-09-23 21:56:16 +03:00
Valentino Geron
19ef1f371d Fix LPOS command when RANK is greater than matches
When calling to LPOS command when RANK is higher than matches,
the return value is non valid response. For example:
```
LPUSH l a
:1
LPOS l b RANK 5 COUNT 10
*-4
```
It may break client-side parser.

Now, we count how many replies were replied in the array.
```
LPUSH l a
:1
LPOS l b RANK 5 COUNT 10
*0
```

(cherry picked from commit 9204a9b2c2f6eb59767ab0bddcde62c75e8c20b0)
2020-09-01 09:27:58 +03:00
Oran Agra
daef1f00c2
Add test coverage for CLIENT UNBLOCK (#7712)
plus minor other fixes to list.tcl
2020-08-27 08:09:39 +03:00
Valentino Geron
9204a9b2c2 Fix LPOS command when RANK is greater than matches
When calling to LPOS command when RANK is higher than matches,
the return value is non valid response. For example:
```
LPUSH l a
:1
LPOS l b RANK 5 COUNT 10
*-4
```
It may break client-side parser.

Now, we count how many replies were replied in the array.
```
LPUSH l a
:1
LPOS l b RANK 5 COUNT 10
*0
```
2020-08-23 16:03:30 +03:00
Mota
ddcbb628a1
Test:Fix invalid cases in hash.tcl and dump.tcl (#4611) 2020-08-12 10:25:24 +08:00