14165 Commits

Author SHA1 Message Date
malavan
eabf119fde don't use flags that aren't supported by certain versions of OPENSSL
Former-commit-id: fae6bfb80bc66351e278f8c5c4ce91bdcef64afb
2021-08-16 22:55:46 +00:00
christianEQ
fe50d680e1 install 32bit libraries in github action
Former-commit-id: b08aa26a765ddca4a398458ba6a3b91e0358aaf7
2021-08-13 11:05:12 -04:00
christianEQ
0b1212aaa1 github runner for 32bit build
Former-commit-id: 8dacbed556e858e1ccaa567373bb1466ddabe91e
2021-08-13 11:05:12 -04:00
christianEQ
314cc7d6de changed keys counter variable to unsigned to fix signedness comparison warning
Former-commit-id: b44df43fcbc6392796a9a2ec92b3aab1f43f5503
2021-08-13 11:05:12 -04:00
christianEQ
e0ff0ee45c %zu for printing type size_t
Former-commit-id: 84af829053eaf70439aaf0b7fdebfe2d9d2f3b6f
2021-08-13 11:05:12 -04:00
christianEQ
c48ae9ace7 fixed 32bit compatibility in Makefile
Former-commit-id: f5ee64f9aaeb3f3773de2435191bcff0855154e4
2021-08-13 11:05:12 -04:00
christianEQ
8bccf9bf12 fixed overly strict assert for 32bit
Former-commit-id: ce4891b33d65038bb0543eb2d3526c5310fee59b
2021-08-13 11:05:12 -04:00
MalavanEQAlpha
0750f0f2e6 Merge pull request #318 from VivekSainiEQ/redisearch_fixes
Initialized serverTL in more places in module.cpp

Former-commit-id: ffa46b8f0bd84c3f854a76e375be2108705fa8cc
2021-08-11 15:47:30 -04:00
Vivek Saini
c5b7ff7492 Merge pull request #296 from VivekSainiEQ/systemd_fix
Fixed bug where make USE_SYSTEMD=yes would not pass in correct flags

Former-commit-id: bf05c261d2689683ce89da2d062e1819ed6d772c
2021-08-11 15:39:50 -04:00
VivekSainiEQ
23c0c8171e Merge remote-tracking branch 'upstream/unstable' into redisearch_fixes
Former-commit-id: 952ae9fbe38b99593ec175da6718a5d7fae49837
2021-08-11 19:28:04 +00:00
VivekSainiEQ
2598493a62 Prevent invalid mvcc timestamps from causing critical errors
Former-commit-id: 6f2dbb00119b1d0a1f5a2543d2c6af05f83ef5de
2021-08-11 15:06:14 -04:00
Paweł Sacawa
eb5a1a5e19 Add: zsh completions with descriptions
This commit add zsh completions for the keydb `client`. They have contextual host completion and full argument descriptions.
Vendor-distributed completions for zsh should end up in `/usr/share/zsh/vendor-completions`, but unfortunatly I'm not familiar with the packaging method for *.deb archives, so these completions will need to be moved to the appropriate directory.


Former-commit-id: f77980fce87f22b59677e374e0d5c113775cc08a
2021-08-11 15:05:11 -04:00
MalavanEQAlpha
442f1075d5 Merge pull request #340 from EQ-Alpha/update_keydb.conf
update keydb.conf with new configs from redis unstable

Former-commit-id: d90c1e6eae9b0bb07fb67512e87339d948db5f72
2021-08-11 15:04:22 -04:00
MalavanEQAlpha
dda92b67c8 Merge pull request #341 from EQ-Alpha/pkg_config
update packaging config files

Former-commit-id: d987f463ebeb19ed19869ebbed4e8681bcb6dbcb
2021-08-11 15:01:37 -04:00
benschermel
3745daf938 add script to compare all active config parameters of any 2 config files
Former-commit-id: 89beaf2460e969c3ee18e03fe539de5833dba50a
2021-08-10 20:10:28 -04:00
benschermel
a7b3ab45f2 update packaging config files
Former-commit-id: 0c5d12b91455db4acad57665db8aac79180b42ab
2021-08-10 20:08:39 -04:00
malavan
6879564b48 update keydb.conf with new configs from redis unstable
Former-commit-id: 32ca2e8a7335ee7bc4889a9369c97177f34739ee
2021-08-10 21:34:52 +00:00
Christian Legge
9e562f1fe5 Add REPLPING command for use during replication (#329)
* added replping command for initiating replication

* backwards compatibility for replping (retry if not recognized)

* don't allow ping during loading (load balancer fix)

* changed replping warning to notice

Former-commit-id: d7f6bc16145206e96ffeb9941398d564c3dba6a9
2021-07-29 15:50:30 -04:00
MalavanEQAlpha
31e2a22b6f Merge pull request #328 from EQ-Alpha/backtrace_for_alpine
Backtrace for alpine using libunwind

Former-commit-id: 83d878f11183d8cd54fb34bceb7f839f73599ab1
2021-07-28 17:17:06 -04:00
malavan
c3a6be36e3 add some comments
Former-commit-id: 0fc6441c9608f0650003a7ab29dd704166345272
2021-07-28 20:32:18 +00:00
Oran Agra
db09f6eb2e Redis 6.2.5 2021-07-21 21:06:49 +03:00
Huang Zhw
835d15b536 On 32 bit platform, the bit position of GETBIT/SETBIT/BITFIELD/BITCOUNT,BITPOS may overflow (see CVE-2021-32761) (#9191)
GETBIT, SETBIT may access wrong address because of wrap.
BITCOUNT and BITPOS may return wrapped results.
BITFIELD may access the wrong address but also allocate insufficient memory and segfault (see CVE-2021-32761).

This commit uses `uint64_t` or `long long` instead of `size_t`.
related https://github.com/redis/redis/pull/8096

At 32bit platform:
> setbit bit 4294967295 1
(integer) 0
> config set proto-max-bulk-len 536870913
OK
> append bit "\xFF"
(integer) 536870913
> getbit bit 4294967296
(integer) 0

When the bit index is larger than 4294967295, size_t can't hold bit index. In the past,  `proto-max-bulk-len` is limit to 536870912, so there is no problem.

After this commit, bit position is stored in `uint64_t` or `long long`. So when `proto-max-bulk-len > 536870912`, 32bit platforms can still be correct.

For 64bit platform, this problem still exists. The major reason is bit pos 8 times of byte pos. When proto-max-bulk-len is very larger, bit pos may overflow.
But at 64bit platform, we don't have so long string. So this bug may never happen.

Additionally this commit add a test cost `512MB` memory which is tag as `large-memory`. Make freebsd ci and valgrind ci ignore this test.

(cherry picked from commit 71d452876ebf8456afaadd6b3c27988abadd1148)
2021-07-21 21:06:49 +03:00
Oran Agra
bae0512c8a longer timeout in replication test (#8963)
the test normally passes. but we saw one failure in a valgrind run in github actions

(cherry picked from commit 8458baf6a96fa6c6050bac24160f82d32a0b9ed4)
2021-07-21 21:06:49 +03:00
Huang Zhw
8d6134952a Remove testmodule in src/modules/Makefile. (#9250)
src/modules make failed. As in #3718 testmodule.c was removed. But the makefile was not updated

(cherry picked from commit d54c9086c267d20bb6981f5a60f589e93b662d62)
2021-07-21 21:06:49 +03:00
Oran Agra
1d7c0e5949 Fix failing basics moduleapi test on 32bit CI (#9140)
(cherry picked from commit 5ffdbae1f64bb66b6e2470779540fb1051dcbff1)
2021-07-21 21:06:49 +03:00
Oran Agra
37b0f3617d Adjustments to recent RM_StringTruncate fix (#3718) (#9125)
- Introduce a new sdssubstr api as a building block for sdsrange.
  The API of sdsrange is many times hard to work with and also has
  corner case that cause bugs. sdsrange is easy to work with and also
  simplifies the implementation of sdsrange.
- Revert the fix to RM_StringTruncate and just use sdssubstr instead of
  sdsrange.
- Solve valgrind warnings from the new tests introduced by the previous
  PR.

(cherry picked from commit ae418eca24ba53a7dca07b0e7065f856e625469b)
2021-07-21 21:06:49 +03:00
Huang Zhw
6866117194 Fix missing separator in module info line (usedby and using lists) (#9241)
Fix module info genModulesInfoStringRenderModulesList lack separator when there's more than one module in the list.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 1895e134a77efd789b1a6daee76a6ba5ec90e516)
2021-07-21 21:06:49 +03:00
Binbin
b622537199 SMOVE only notify dstset when the addition is successful. (#9244)
in case dest key already contains the member, the dest key isn't modified, so the command shouldn't invalidate watch.

(cherry picked from commit 11dc4e59b365d6cd8699604d7d1c1025b6bb6259)
2021-07-21 21:06:49 +03:00
qetu3790
355b1b6a57 Set TCP keepalive on inbound clusterbus connections (#9230)
Set TCP keepalive on inbound clusterbus connections to prevent memory leak

(cherry picked from commit f03af47a34ec672a7d9b18150a5be3a83681c19b)
2021-07-21 21:06:49 +03:00
Yossi Gottlieb
d6f4273241 Fix compatibility with OpenSSL 1.1.0. (#9233)
(cherry picked from commit 277e4dc2032356c7712b539e89f7e9154e0a1a86)
2021-07-21 21:06:49 +03:00
Oran Agra
de1b19ea88 fix valgrind issues with recently added test in modules/blockonbackground (#9192)
fixes test issue introduced in #9167

1. invalid reads due to accessing non-retained string (passed as unblock context).
2. leaking module blocked client context, see #6922 for info.

(cherry picked from commit a8518cce951629eaccde40fd0e51b36a5dc6321c)
2021-07-21 21:06:49 +03:00
Yossi Gottlieb
79fa5618f1 Fix CLIENT UNBLOCK crashing modules. (#9167)
Modules that use background threads with thread safe contexts are likely
to use RM_BlockClient() without a timeout function, because they do not
set up a timeout.

Before this commit, `CLIENT UNBLOCK` would result with a crash as the
`NULL` timeout callback is called. Beyond just crashing, this is also
logically wrong as it may throw the module into an unexpected client
state.

This commits makes `CLIENT UNBLOCK` on such clients behave the same as
any other client that is not in a blocked state and therefore cannot be
unblocked.

(cherry picked from commit aa139e2f02292d668370afde8c91575363c2d611)
2021-07-21 21:06:49 +03:00
Huang Zhw
91bf2ab86d redis-cli cluster import command may issue wrong MIGRATE command. (#8945)
In clusterManagerCommandImport strcat was used to concat COPY and
REPLACE, the space maybe not enough.
If we use --cluster-replace but not --cluster-copy, the MIGRATE
command contained COPY instead of REPLACE.

(cherry picked from commit a049f6295a28a20b11eff89083e91dab0738413b)
2021-07-21 21:06:49 +03:00
Binbin
88655019cc Fix accidental deletion of sinterstore command when we meet wrong type error. (#9032)
SINTERSTORE would have deleted the dest key right away,
even when later on it is bound to fail on an (WRONGTYPE) error.

With this change it first picks up all the input keys, and only later
delete the dest key if one is empty.

Also add more tests for some commands.
Mainly focus on
- `wrong type error`:
	expand test case (base on sinter bug) in non-store variant
	add tests for store variant (although it exists in non-store variant, i think it would be better to have same tests)
- the dstkey result when we meet `non-exist key (empty set)` in *store

sdiff:
- improve test case about wrong type error (the one we found in sinter, although it is safe in sdiff)
- add test about using non-exist key (treat it like an empty set)
sdiffstore:
- according to sdiff test case, also add some tests about `wrong type error` and `non-exist key`
- the different is that in sdiffstore, we will consider the `dstkey` result

sunion/sunionstore add more tests (same as above)

sinter/sinterstore also same as above ...

(cherry picked from commit b8a5da80c49501773f8778aaf5cbf595cef615e4)
2021-07-21 21:06:49 +03:00
Jason Elbaum
fad44611dc Change return value type for ZPOPMAX/MIN in RESP3 (#8981)
When using RESP3, ZPOPMAX/ZPOPMIN should return nested arrays for consistency
with other commands (e.g. ZRANGE).

We do that only when COUNT argument is present (similarly to how LPOP behaves).
for reasoning see https://github.com/redis/redis/issues/8824#issuecomment-855427955

This is a breaking change only when RESP3 is used, and COUNT argument is present!

(cherry picked from commit 7f342020dcbdf9abe754d6b666efdeded7063870)
2021-07-21 21:06:49 +03:00
Mikhail Fesenko
8884971223 Direct redis-cli repl prints to stderr, because --rdb can print to stdout. fflush stdout after responses (#9136)
1. redis-cli can output --rdb data to stdout
   but redis-cli also write some messages to stdout which will mess up the rdb.

2. Make redis-cli flush stdout when printing a reply
  This was needed in order to fix a hung in redis-cli test that uses
  --replica.
   Note that printf does flush when there's a newline, but fwrite does not.

3. fix the redis-cli --replica test which used to pass previously
   because it didn't really care what it read, and because redis-cli
   used printf to print these other things to stdout.

4. improve redis-cli --replica test to run with both diskless and disk-based.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Viktor Söderqvist <viktor@zuiderkwast.se>
(cherry picked from commit 1eb4baa5b8e76adc337ae9fab49acc2585a0cdd0)
2021-07-21 21:06:49 +03:00
Rob Snyder
07e1248686 Fix ziplist length updates on bigendian platforms (#2080)
Adds call to intrev16ifbe to ensure ZIPLIST_LENGTH is compared correctly

(cherry picked from commit eaa52719a355c4467d0383c1c9f5184c9c14fe5a)
2021-07-21 21:06:49 +03:00
Oran Agra
6cd84b64f0 Test infra, handle RESP3 attributes and big-numbers and bools (#9235)
- promote the code in DEBUG PROTOCOL to addReplyBigNum
- DEBUG PROTOCOL ATTRIB skips the attribute when client is RESP2
- networking.c addReply for push and attributes generate assertion when
  called on a RESP2 client, anything else would produce a broken
  protocol that clients can't handle.

(cherry picked from commit 6a5bac309e868deef749c36949723b415de2496f)
2021-07-21 21:06:49 +03:00
Binbin
c6b3966d02 hrandfield and zrandmember with count should return emptyarray when key does not exist. (#9178)
due to a copy-paste bug, it used to reply with null response rather than empty array.
this commit includes new tests that are looking at the RESP response directly in
order to be able to tell the difference between them.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit a418a2d3fc0250c094802d7e8ea64d96eedfda07)
2021-07-21 21:06:49 +03:00
Oran Agra
432e056659 Tests: add a way to read raw RESP protocol reponses (#9193)
This makes it possible to distinguish between null response and an empty
array (currently the tests infra translates both to an empty string/list)

(cherry picked from commit 7103367ad44b4241e59a709771cb464aa2a86b20)
2021-07-21 21:06:49 +03:00
Mikhail Fesenko
bbb6c2fb53 redis-cli --rdb: fix broken fsync/ftruncate for stdout (#9135)
A change in redis 6.2 caused redis-cli --rdb that's directed to stdout to fail because fsync fails.
This commit avoids doing ftruncate (fails with a warning) and fsync (fails with an error) when the
output file is `-`, and adds the missing documentation that `-` means stdout.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Wang Yuan <wangyuancode@163.com>
(cherry picked from commit 74fe15b3602ed7c003b5c53e45e31f7aa6d4a86f)
2021-07-21 21:06:49 +03:00
Leibale Eidelman
095a6e5937 fix ZRANGESTORE - should return 0 when src points to an empty key (#9089)
mistakenly it used to return an empty array rather than 0.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 95274f1f8a3ef4cb4033beecfaa99ea1439ed170)
2021-07-21 21:06:49 +03:00
guybe7
56020fd821 Include sizeof(struct stream) in objectComputeSize (#9164)
Affects MEMORY USAGE

(cherry picked from commit 4434cebbb38ae2ef39b2d74ed3e6cd8d1a061d0c)
2021-07-21 21:06:49 +03:00
Binbin
11235c0e40 ZRANDMEMBER WITHSCORES with negative COUNT may return bad score (#9162)
Return a bad score when used with negative count (or count of 1), and non-ziplist encoded zset.
Also add test to validate the return value and cover the issue.

(cherry picked from commit 4bc5a8324d3cb23ed2cc8a9cd19444a893a6d52c)
2021-07-21 21:06:49 +03:00
Evan
406caa5f56 modules: Add newlen == 0 handling to RM_StringTruncate (#3717) (#3718)
Previously, passing 0 for newlen would not truncate the string at all.
This adds handling of this case, freeing the old string and creating a new empty string.

Other changes:
- Move `src/modules/testmodule.c` to `tests/modules/basics.c`
- Introduce that basic test into the test suite
- Add tests to cover StringTruncate
- Add `test-modules` build target for the main makefile
- Extend `distclean` build target to clean modules too

(cherry picked from commit 1ccf2ca2f475a161b8ca0c574d4c0e6ef9ecf754)
2021-07-21 21:06:49 +03:00
yvette903
8bd5e39102 Fix coredump after Client Unpause command when threaded I/O is enabled (#9041)
Fix crash when using io-threads-do-reads and issuing CLIENT PAUSE and
CLIENT UNPAUSE.
This issue was introduced in redis 6.2 together with the FAILOVER command.

(cherry picked from commit 096c5fd5d22caa2c21a4863cbaaf64fd8e7107d2)
2021-07-21 21:06:49 +03:00
guybe7
c547dff650 Module API for current command name (#8792)
sometimes you can be very deep in the call stack, without access to argv.
once you're there you may want your reply/log to contain the command name.

(cherry picked from commit e16d3eb998f7017c95ef17179de77aa5b6f2e272)
2021-07-21 21:06:49 +03:00
Huang Zhw
e6df2f6210 Fix XTRIM or XADD with LIMIT may delete more entries than Count. (#9048)
The decision to stop trimming due to LIMIT in XADD and XTRIM was after the limit was reached.
i.e. the code was deleting **at least** that count of records (from the LIMIT argument's perspective, not the MAXLEN),
instead of **up to** that count of records.
see #9046

(cherry picked from commit eaa7a7bb93c1ac6dbf347e1c29ac719a12a75158)
2021-07-21 21:06:49 +03:00
Huang Zhw
312ccae003 XTRIM call streamParseAddOrTrimArgsOrReply use wrong arg xadd. (#9047)
xtrimCommand call streamParseAddOrTrimArgsOrReply should use xadd==0.

When the syntax is valid, it does not cause any bugs because the params of XADD is superset of XTRIM.
Just XTRIM will not respond with error on invalid syntax. The syntax of XADD will also be accpeted by XTRIM.

(cherry picked from commit 91f3689bf5dc4ce3cf00d9d957b9677b362a205e)
2021-07-21 21:06:49 +03:00
YaacovHazan
ff27217639 stabilize tests that involved with load handlers (#8967)
When test stop 'load handler' by killing the process that generating the load,
some commands that already in the input buffer, still might be processed by the server.
This may cause some instability in tests, that count on that no more commands
processed after we stop the `load handler'

In this commit, new proc 'wait_load_handlers_disconnected' added, to verify that no more
cammands from any 'load handler' prossesed, by checking that the clients who
genreate the load is disconnceted.

Also, replacing check of dbsize with wait_for_ofs_sync before comparing debug digest, as
it would fail in case the last key the workload wrote was an overridden key (not a new one).

Affected tests
Race fix:
- failover command to specific replica works
- Connect multiple replicas at the same time (issue #141), master diskless=$mdl, replica diskless=$sdl
- AOF rewrite during write load: RDB preamble=$rdbpre

Cleanup and speedup:
- Test replication with blocking lists and sorted sets operations
- Test replication with parallel clients writing in different DBs
- Test replication partial resync: $descr (diskless: $mdl, $sdl, reconnect: $reconnect

(cherry picked from commit 32a2584e079a1b3c2d1e6649e38239381a73a459)
2021-07-21 21:06:49 +03:00