gvsafronov/futriix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix bad merge in CI.yml

Browse Source 
Former-commit-id: 6311d709c39b3bacaeab77b18033010f1b548f81
This commit is contained in:
John Sully 2020-05-21 22:09:06 -04:00
commit 193d7c76cb
57 changed files with 2642 additions and 869 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.github/workflows/ci.yml vendored
View File

@ -38,14 +38,9 @@ jobs:
run: |
sudo apt-get -y install tcl8.5
./runtest --clients 2 --verbose
<<<<<<< HEAD
- name: module tests
run: |
./runtest-moduleapi
=======
- name: module api test
run: ./runtest-moduleapi --clients 2 --verbose
>>>>>>> 024c380b9da02bc4112822c0f5f9ac1388b4205b
build-ubuntu-old:
runs-on: ubuntu-16.04

532
00-RELEASENOTES
View File

@ -11,6 +11,538 @@ CRITICAL: There is a critical bug affecting MOST USERS. Upgrade ASAP.
SECURITY: There are security fixes in the release.
--------------------------------------------------------------------------------
================================================================================
Redis 6.0.0 GA Released Thu Apr 30 14:55:02 CEST 2020
================================================================================
Upgrade urgency CRITICAL: many bugs fixed compared to the last release
candidate. Better to upgrade if you see things
affecting your environment in the changelog.
Hi all, finally we have Redis 6.0.0 GA! Enjoy this new Redis release.
Most of the documentation was updated today so that you can likely
find what you are looking for about the new features at redis.io.
This is the list of what changed compared to the previoius release candidate:
* XCLAIM AOF/replicas propagation fixed.
* Client side caching: new NOLOOP option to avoid getting notified about
changes performed by ourselves.
* ACL GENPASS now uses HMAC-SHA256 and have an optional "bits" argument.
It means you can use it as a general purpose "secure random strings"
primitive!
* Cluster "SLOTS" subcommand memory optimization.
* The LCS command is now a subcommand of STRALGO.
* Meaningful offset for replicas as well. More successful partial
resynchronizations.
* Optimize memory usage of deferred replies.
* Faster CRC64 algorithm for faster RDB loading.
* XINFO STREAM FULL, a new subcommand to get the whole stream state.
* CLIENT KILL USER <username>.
* MIGRATE AUTH2 option, for ACL style authentication support.
* Other random bugfixes.
Enjoy Redis 6! :-)
Goodbye antirez
List of commits in this release:
antirez in commit 1f9b82bd5:
Update help.h again before Redis 6 GA.
1 file changed, 17 insertions(+), 12 deletions(-)
antirez in commit 3fcffe7d0:
redis-cli: fix hints with subcommands.
1 file changed, 2 insertions(+), 1 deletion(-)
antirez in commit 455d8a05c:
redis-cli command help updated.
1 file changed, 165 insertions(+), 25 deletions(-)
zhaozhao.zz in commit 70287bbc9:
lazyfree & eviction: record latency generated by lazyfree eviction
1 file changed, 18 insertions(+), 13 deletions(-)
antirez in commit 7be21139a:
MIGRATE AUTH2 for ACL support.
1 file changed, 19 insertions(+), 5 deletions(-)
antirez in commit e1ee1a49d:
CLIENT KILL USER <username>.
1 file changed, 11 insertions(+)
antirez in commit d56f058c0:
Fix tracking table max keys option in redis.conf.
1 file changed, 12 insertions(+), 9 deletions(-)
antirez in commit 96dd5fc93:
redis-cli: safer cluster fix with unreachalbe masters.
1 file changed, 26 insertions(+), 1 deletion(-)
antirez in commit 5b59d9c5d:
redis-cli: simplify cluster nodes coverage display.
1 file changed, 10 insertions(+), 17 deletions(-)
antirez in commit c163d4add:
redis-cli: try to make clusterManagerFixOpenSlot() more readable.
1 file changed, 25 insertions(+), 6 deletions(-)
Guy Benoish in commit aab74b715:
XINFO STREAM FULL should have a default COUNT of 10
1 file changed, 8 insertions(+), 4 deletions(-)
antirez in commit 606134f9d:
Comment clearly why we moved some code in #6623.
1 file changed, 4 insertions(+), 1 deletion(-)
srzhao in commit ee627bb66:
fix pipelined WAIT performance issue.
1 file changed, 13 insertions(+), 13 deletions(-)
antirez in commit 47b8a7f9b:
Fix create-cluster BIN_PATH.
1 file changed, 1 insertion(+), 1 deletion(-)
Guy Benoish in commit 6c0bc608a:
Extend XINFO STREAM output
2 files changed, 226 insertions(+), 34 deletions(-)
hwware in commit 5bfc18950:
Fix not used marco in cluster.c
1 file changed, 1 insertion(+), 1 deletion(-)
Itamar Haber in commit 56d628f85:
Update create-cluster
1 file changed, 1 insertion(+), 1 deletion(-)
Itamar Haber in commit cac9d7cf7:
Adds `BIN_PATH` to create-cluster
1 file changed, 8 insertions(+), 6 deletions(-)
Oran Agra in commit b712fba17:
hickup, re-fix dictEncObjKeyCompare
1 file changed, 4 insertions(+), 4 deletions(-)
Oran Agra in commit ea63aea72:
fix loading race in psync2 tests
3 files changed, 15 insertions(+), 1 deletion(-)
antirez in commit 64e588bfa:
Rework comment in dictEncObjKeyCompare().
1 file changed, 8 insertions(+), 9 deletions(-)
Oran Agra in commit 0d1e8c93b:
allow dictFind using static robj
1 file changed, 9 insertions(+), 4 deletions(-)
Madelyn Olson in commit a1bed447b:
Added crcspeed library
2 files changed, 341 insertions(+)
Madelyn Olson in commit a75fa3aad:
Made crc64 test consistent
1 file changed, 3 insertions(+), 2 deletions(-)
Madelyn Olson in commit 52c75e9db:
Implemented CRC64 based on slice by 4
5 files changed, 124 insertions(+), 157 deletions(-)
Oran Agra in commit 8110ba888:
optimize memory usage of deferred replies
1 file changed, 31 insertions(+)
Oran Agra in commit e4d2bb62b:
Keep track of meaningful replication offset in replicas too
5 files changed, 212 insertions(+), 92 deletions(-)
antirez in commit fea9788cc:
Fix STRALGO command flags.
1 file changed, 1 insertion(+), 1 deletion(-)
Dave-in-lafayette in commit 2144047e1:
fix for unintended crash during panic response
1 file changed, 1 insertion(+), 1 deletion(-)
Guy Benoish in commit 43329c9b6:
Add the stream tag to XSETID tests
1 file changed, 1 insertion(+), 1 deletion(-)
Dave-in-lafayette in commit 1e17d3de7:
fix for crash during panic before all threads are up
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit 3722f89f4:
LCS -> STRALGO LCS.
4 files changed, 28 insertions(+), 15 deletions(-)
antirez in commit 373ae6061:
Also use propagate() in streamPropagateGroupID().
1 file changed, 11 insertions(+), 1 deletion(-)
yanhui13 in commit f03f1fad6:
add tcl test for cluster slots
1 file changed, 44 insertions(+)
yanhui13 in commit 374ffdf1c:
optimize the output of cluster slots
1 file changed, 7 insertions(+), 4 deletions(-)
antirez in commit 4db38d2ef:
Minor aesthetic changes to #7135.
1 file changed, 5 insertions(+), 7 deletions(-)
Valentino Geron in commit f0a261448:
XREADGROUP with NOACK should propagate only one XGROUP SETID command
1 file changed, 13 insertions(+), 7 deletions(-)
antirez in commit fbdef6a9b:
ACL: re-enable command execution of disabled users.
1 file changed, 4 deletions(-)
antirez in commit 05a41da75:
getRandomBytes(): use HMAC-SHA256.
1 file changed, 30 insertions(+), 10 deletions(-)
antirez in commit 345c3768d:
ACL GENPASS: take number of bits as argument.
1 file changed, 21 insertions(+), 6 deletions(-)
antirez in commit 639c8a1d9:
ACL GENPASS: emit 256 bits instead of 128.
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit 321acea03:
ACL: deny commands execution of disabled users.
1 file changed, 4 insertions(+)
Theo Buehler in commit b0920e6e8:
TLS: Fix build with SSL_OP_NO_CLIENT_RENEGOTIATION
1 file changed, 1 insertion(+), 1 deletion(-)
Yossi Gottlieb in commit 149b658b5:
TLS: Fix build on older verisons of OpenSSL.
1 file changed, 2 insertions(+)
antirez in commit 06917e581:
Tracking: test expired keys notifications.
1 file changed, 13 insertions(+)
antirez in commit e434b2ce4:
Tracking: NOLOOP tests.
1 file changed, 32 insertions(+)
antirez in commit f3a172887:
Tracking: signal key as modified when evicting.
1 file changed, 1 insertion(+)
antirez in commit e63bb7ec8:
Tracking: NOLOOP further implementation and fixes.
2 files changed, 21 insertions(+), 6 deletions(-)
antirez in commit 6791ff052:
Tracking: NOLOOP internals implementation.
17 files changed, 174 insertions(+), 112 deletions(-)
antirez in commit 725b8cc68:
Implement redis_set_thread_title for MacOS.
1 file changed, 6 insertions(+)
zhenwei pi in commit 3575b8706:
Threaded IO: set thread name for redis-server
3 files changed, 28 insertions(+)
antirez in commit a76c67578:
Sentinel: small refactoring of sentinelCollectTerminatedScripts().
1 file changed, 1 insertion(+), 2 deletions(-)
omg-by in commit 3a27064c4:
fix(sentinel): sentinel.running_scripts will always increase more times and not reset
1 file changed, 1 insertion(+)
antirez in commit 5c4c73e2c:
A few comments and name changes for #7103.
1 file changed, 13 insertions(+), 4 deletions(-)
Oran Agra in commit 6148f9493:
testsuite run the defrag latency test solo
3 files changed, 42 insertions(+), 2 deletions(-)
Jamie Scott in commit 51d3012d4:
Adding acllog-max-len to Redis.conf
1 file changed, 9 insertions(+)
antirez in commit c39f16c42:
Fix XCLAIM propagation in AOF/replicas for blocking XREADGROUP.
2 files changed, 8 insertions(+), 3 deletions(-)
================================================================================
Redis 6.0-rc4 Released Thu Apr 16 16:10:35 CEST 2020
================================================================================
Upgrade urgency LOW: If you are using RC3 without issues, don't rush.
Hi all, this the latest release candidate of Redis 6. This is likely to
be very similar to what you'll see in Redis 6 GA. Please test it and
report any issue :-)
Main changes in this release:
* Big INFO speedup when using a lot of of clients.
* Big speedup on all the blocking commands: now blocking
on the same key is O(1) instead of being O(N).
* Stale replicas now allow MULTI/EXEC.
* New command: LCS (Longest Common Subsequence).
* Add a new configuration to make DEL like UNLINK.
* RDB loading speedup.
* Many bugs fixed (see the commit messages at the end of this node)
See you in 14 days for Redis 6 GA.
List of commits:
antirez in commit 9f594e243:
Update SDS to latest version.
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit 48781dd95:
RESP3: fix HELLO map len in Sentinel mode.
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit 371ab0cff:
Don't allow empty spaces in ACL usernames.
1 file changed, 36 insertions(+), 8 deletions(-)
antirez in commit b86140ac5:
Don't allow empty spaces in ACL key patterns.
1 file changed, 12 insertions(+), 1 deletion(-)
liumiuyong in commit a7ee3c3e7:
FIX: truncate max/min longitude,latitude related geo_point (ex: {180, 85.05112878} )
1 file changed, 4 insertions(+)
Guy Benoish in commit e5b9eb817:
Typo in getTimeoutFromObjectOrReply's error reply
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit 0f31bb5c1:
Fix HELLO reply in Sentinel mode, see #6160.
1 file changed, 1 insertion(+), 1 deletion(-)
hwware in commit b92d9a895:
fix spelling in acl.c
1 file changed, 2 insertions(+), 2 deletions(-)
antirez in commit 8f896e57a:
Fix zsetAdd() top comment spelling.
1 file changed, 3 insertions(+), 3 deletions(-)
hayleeliu in commit 8f5157058:
fix spelling mistake in bitops.c
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit ddeda9ceb:
Fix function names in zslDeleteNode() top comment.
1 file changed, 2 insertions(+), 1 deletion(-)
antirez in commit bde1f0a8e:
RESP3: change streams items from maps to arrays.
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit bec68bff2:
Use the special static refcount for stack objects.
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit 0f239e51b:
RDB: refactor some RDB loading code into dbAddRDBLoad().
3 files changed, 22 insertions(+), 4 deletions(-)
antirez in commit f855db61b:
incrRefCount(): abort on statically allocated object.
2 files changed, 12 insertions(+), 2 deletions(-)
antirez in commit 23094ba01:
More powerful DEBUG RELOAD.
3 files changed, 55 insertions(+), 16 deletions(-)
antirez in commit 8161a7a3e:
RDB: clarify a condition in rdbLoadRio().
2 files changed, 9 insertions(+), 2 deletions(-)
antirez in commit 61b153073:
RDB: load files faster avoiding useless free+realloc.
7 files changed, 40 insertions(+), 28 deletions(-)
antirez in commit 414debfd0:
Speedup: unblock clients on keys in O(1).
4 files changed, 50 insertions(+), 23 deletions(-)
antirez in commit cbcd07777:
Fix ACL HELP table missing comma.
1 file changed, 12 insertions(+), 12 deletions(-)
mymilkbottles in commit 2437455f2:
Judge the log level in advance
1 file changed, 1 insertion(+)
antirez in commit 35c64b898:
Speedup INFO by counting client memory incrementally.
4 files changed, 52 insertions(+), 26 deletions(-)
qetu3790 in commit c3ac71748:
fix comments about RESIZE DB opcode in rdb.c
1 file changed, 1 insertion(+), 4 deletions(-)
antirez in commit c8dbcff9d:
Clarify redis.conf comment about lazyfree-lazy-user-del.
1 file changed, 9 insertions(+), 5 deletions(-)
zhaozhao.zz in commit abd5156f2:
lazyfree: add a new configuration lazyfree-lazy-user-del
4 files changed, 7 insertions(+), 2 deletions(-)
antirez in commit 5719b3054:
LCS: more tests.
1 file changed, 8 insertions(+)
antirez in commit c89e1f293:
LCS: allow KEYS / STRINGS to be anywhere.
1 file changed, 6 deletions(-)
antirez in commit 0b16f8d44:
LCS tests.
1 file changed, 22 insertions(+)
antirez in commit 9254a805d:
LCS: get rid of STOREIDX option. Fix get keys helper.
2 files changed, 20 insertions(+), 21 deletions(-)
antirez in commit a4c490703:
LCS: fix stale comment.
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit cb92c23de:
LCS: output LCS len as well in IDX mode.
1 file changed, 6 insertions(+), 1 deletion(-)
antirez in commit 56a52e804:
LCS: MINMATCHLEN and WITHMATCHLEN options.
1 file changed, 24 insertions(+), 11 deletions(-)
antirez in commit ebb09a5c3:
LCS: 7x speedup by accessing the array with better locality.
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit a9f8a8cba:
LCS: implement KEYS option.
1 file changed, 18 insertions(+), 2 deletions(-)
antirez in commit 4aa24e62a:
LCS: other fixes to range emission.
1 file changed, 20 insertions(+), 16 deletions(-)
antirez in commit 2b67b6b87:
LCS: fix emission of last range starting at index 0.
1 file changed, 1 insertion(+), 1 deletion(-)
antirez in commit 420aac727:
LCS: implement range indexes option.
1 file changed, 59 insertions(+), 9 deletions(-)
antirez in commit a518a9a76:
LCS: initial functionality implemented.
4 files changed, 156 insertions(+), 1 deletion(-)
srzhao in commit 026cc11b0:
Check OOM at script start to get stable lua OOM state.
3 files changed, 11 insertions(+), 4 deletions(-)
Oran Agra in commit 02b594f6a:
diffrent fix for runtest --host --port
2 files changed, 13 insertions(+), 13 deletions(-)
Guy Benoish in commit f695d1830:
Try to fix time-sensitive tests in blockonkey.tcl
1 file changed, 54 insertions(+), 1 deletion(-)
Guy Benoish in commit 0e42cfc36:
Use __attribute__ only if __GNUC__ is defined
1 file changed, 12 insertions(+), 3 deletions(-)
Guy Benoish in commit 91ed9b3c4:
Modules: Perform printf-like format checks in variadic API
1 file changed, 3 insertions(+), 3 deletions(-)
Valentino Geron in commit 3e0d20962:
XREAD and XREADGROUP should not be allowed from scripts when BLOCK option is being used
3 files changed, 18 insertions(+), 2 deletions(-)
Guy Benoish in commit 240094c9b:
Stale replica should allow MULTI/EXEC
1 file changed, 3 insertions(+), 3 deletions(-)
Xudong Zhang in commit 209f3a1eb:
fix integer overflow
1 file changed, 2 insertions(+), 2 deletions(-)
Guy Benoish in commit 024c380b9:
Fix no-negative-zero test
1 file changed, 1 insertion(+)
Oran Agra in commit a38ff404b:
modules don't signalModifiedKey in setKey() since that's done (optionally) in RM_CloseKey
4 files changed, 8 insertions(+), 8 deletions(-)
Oran Agra in commit 814874d68:
change CI to build and run the module api tests
1 file changed, 2 insertions(+)
Oran Agra in commit 061616c1b:
fix possible warning on incomplete struct init
1 file changed, 1 insertion(+), 1 deletion(-)
Guy Benoish in commit 7764996be:
Make sure Redis does not reply with negative zero
2 files changed, 10 insertions(+)
Guy Benoish in commit eba28e2ce:
DEBUG OBJECT should pass keyname to module when loading
3 files changed, 4 insertions(+), 4 deletions(-)
David Carlier in commit 15c9e79a7:
debug, dump registers on arm too.
1 file changed, 55 insertions(+), 27 deletions(-)
hwware in commit cd2b5df97:
fix spelling in cluster.c
1 file changed, 1 insertion(+), 1 deletion(-)
Valentino Geron in commit 8cdc153f5:
XACK should be executed in a "all or nothing" fashion.
2 files changed, 23 insertions(+), 1 deletion(-)
hwware in commit b35407fa7:
add check for not switching between optin optout mode directly
1 file changed, 12 insertions(+), 1 deletion(-)
hwware in commit 4395889c9:
add check for not providing both optin optout flag
1 file changed, 8 insertions(+)
Guy Benoish in commit 1907e0f18:
PERSIST should notify a keyspace event
1 file changed, 1 insertion(+)
Guy Benoish in commit c35a53169:
streamReplyWithRange: Redundant XSETIDs to replica
1 file changed, 2 insertions(+), 1 deletion(-)
antirez in commit 6fe66e096:
Simplify comment in moduleTryServeClientBlockedOnKey().
1 file changed, 3 insertions(+), 12 deletions(-)
Guy Benoish in commit 193fc241c:
Fix memory corruption in moduleHandleBlockedClients
3 files changed, 149 insertions(+), 46 deletions(-)
================================================================================
Redis 6.0-rc3 Released Tue Mar 31 17:42:39 CEST 2020
================================================================================

30
keydb.conf
View File

@ -626,20 +626,23 @@ replica-priority 100
# to track the keys fetched by many clients.
#
# For this reason it is possible to configure a maximum fill value for the
# invalidation table. By default it is set to 10%, and once this limit is
# reached, Redis will start to evict caching slots in the invalidation table
# even if keys are not modified, just to reclaim memory: this will in turn
# invalidation table. By default it is set to 1M of keys, and once this limit
# is reached, Redis will start to evict keys in the invalidation table
# even if they were not modified, just to reclaim memory: this will in turn
# force the clients to invalidate the cached values. Basically the table
# maximum fill rate is a trade off between the memory you want to spend server
# maximum size is a trade off between the memory you want to spend server
# side to track information about who cached what, and the ability of clients
# to retain cached objects in memory.
#
# If you set the value to 0, it means there are no limits, and all the 16
# millions of caching slots can be used at the same time. In the "stats"
# INFO section, you can find information about the amount of caching slots
# used at every given moment.
# If you set the value to 0, it means there are no limits, and Redis will
# retain as many keys as needed in the invalidation table.
# In the "stats" INFO section, you can find information about the number of
# keys in the invalidation table at every given moment.
#
# tracking-table-max-fill 10
# Note: when key tracking is used in broadcasting mode, no memory is used
# in the server side so this setting is useless.
#
# tracking-table-max-keys 1000000
################################## SECURITY ###################################
@ -737,6 +740,15 @@ replica-priority 100
# For more information about ACL configuration please refer to
# the Redis web site at https://redis.io/topics/acl
# ACL LOG
#
# The ACL Log tracks failed commands and authentication events associated
# with ACLs. The ACL Log is useful to troubleshoot failed commands blocked
# by ACLs. The ACL Log is stored in and consumes memory. There is no limit
# to its length.You can reclaim memory with ACL LOG RESET or set a maximum
# length below.
acllog-max-len 128
# Using an external ACL file
#
# Instead of configuring users here in this file, it is possible to use

4
src/Makefile
View File

@ -252,9 +252,9 @@ endif
REDIS_SERVER_NAME=keydb-server
REDIS_SENTINEL_NAME=keydb-sentinel
REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o acl.o storage.o rdb-s3.o fastlock.o new.o tracking.o cron.o connection.o tls.o sha256.o motd.o timeout.o $(ASM_OBJ)
REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crcspeed.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o acl.o storage.o rdb-s3.o fastlock.o new.o tracking.o cron.o connection.o tls.o sha256.o motd.o timeout.o $(ASM_OBJ)
REDIS_CLI_NAME=keydb-cli
REDIS_CLI_OBJ=anet.o adlist.o dict.o redis-cli.o redis-cli-cpphelper.o zmalloc.o release.o anet.o ae.o crc64.o siphash.o crc16.o storage-lite.o fastlock.o new.o motd.o $(ASM_OBJ)
REDIS_CLI_OBJ=anet.o adlist.o dict.o redis-cli.o redis-cli-cpphelper.o zmalloc.o release.o anet.o ae.o crcspeed.o crc64.o siphash.o crc16.o storage-lite.o fastlock.o new.o motd.o $(ASM_OBJ)
REDIS_BENCHMARK_NAME=keydb-benchmark
REDIS_BENCHMARK_OBJ=ae.o anet.o redis-benchmark.o adlist.o dict.o zmalloc.o siphash.o redis-benchmark.o storage-lite.o fastlock.o new.o $(ASM_OBJ)
REDIS_CHECK_RDB_NAME=keydb-check-rdb

72
src/acl.cpp
View File

@ -32,6 +32,7 @@ extern "C" {
#include "sha256.h"
}
#include <fcntl.h>
#include <ctype.h>
/* =============================================================================
* Global state for ACLs
@ -171,6 +172,18 @@ sds ACLHashPassword(unsigned char *cleartext, size_t len) {
* Low level ACL API
* ==========================================================================*/
/* Return 1 if the specified string contains spaces or null characters.
* We do this for usernames and key patterns for simpler rewriting of
* ACL rules, presentation on ACL list, and to avoid subtle security bugs
* that may arise from parsing the rules in presence of escapes.
* The function returns 0 if the string has no spaces. */
int ACLStringHasSpaces(const char *s, size_t len) {
for (size_t i = 0; i < len; i++) {
if (isspace(s[i]) || s[i] == 0) return 1;
}
return 0;
}
/* Given the category name the command returns the corresponding flag, or
* zero if there is no match. */
uint64_t ACLGetCommandCategoryFlagByName(const char *name) {
@ -692,7 +705,8 @@ void ACLAddAllowedSubcommand(user *u, unsigned long id, const char *sub) {
*
* When an error is returned, errno is set to the following values:
*
* EINVAL: The specified opcode is not understood.
* EINVAL: The specified opcode is not understood or the key pattern is
* invalid (contains non allowed characters).
* ENOENT: The command name or command category provided with + or - is not
* known.
* EBUSY: The subcommand you want to add is about a command that is currently
@ -791,6 +805,10 @@ int ACLSetUser(user *u, const char *op, ssize_t oplen) {
errno = EEXIST;
return C_ERR;
}
if (ACLStringHasSpaces(op+1,oplen-1)) {
errno = EINVAL;
return C_ERR;
}
sds newpat = sdsnewlen(op+1,oplen-1);
listNode *ln = listSearchKey(u->patterns,newpat);
/* Avoid re-adding the same pattern multiple times. */
@ -1166,6 +1184,12 @@ int ACLLoadConfiguredUsers(void) {
while ((ln = listNext(&li)) != NULL) {
sds *aclrules = (sds*)listNodeValue(ln);
sds username = aclrules[0];
if (ACLStringHasSpaces(aclrules[0],sdslen(aclrules[0]))) {
serverLog(LL_WARNING,"Spaces not allowed in ACL usernames");
return C_ERR;
}
user *u = ACLCreateUser(username,sdslen(username));
if (!u) {
u = ACLGetUserByName(username,sdslen(username));
@ -1291,6 +1315,14 @@ sds ACLLoadFromFile(const char *filename) {
continue;
}
/* Spaces are not allowed in usernames. */
if (ACLStringHasSpaces(argv[1],sdslen(argv[1]))) {
errors = sdscatprintf(errors,
"'%s:%d: username '%s' contains invalid characters. ",
g_pserver->acl_filename, linenum, argv[1]);
continue;
}
/* Try to process the line using the fake user to validate iif
* the rules are able to apply cleanly. */
ACLSetUser(fakeuser,"reset",-1);
@ -1592,7 +1624,7 @@ void addACLLogEntry(client *c, int reason, int keypos, sds username) {
* ACL SETUSER <username> ... acl rules ...
* ACL DELUSER <username> [...]
* ACL GETUSER <username>
* ACL GENPASS
* ACL GENPASS [<bits>]
* ACL WHOAMI
* ACL LOG [<count> | RESET]
*/
@ -1600,6 +1632,13 @@ void aclCommand(client *c) {
char *sub = szFromObj(c->argv[1]);
if (!strcasecmp(sub,"setuser") && c->argc >= 3) {
sds username = szFromObj(c->argv[2]);
/* Check username validity. */
if (ACLStringHasSpaces(username,sdslen(username))) {
addReplyErrorFormat(c,
"Usernames can't contain spaces or null characters");
return;
}
/* Create a temporary user to validate and stage all changes against
* before applying to an existing user or creating a new user. If all
* arguments are valid the user parameters will all be applied together.
@ -1777,16 +1816,31 @@ void aclCommand(client *c) {
}
dictReleaseIterator(di);
setDeferredArrayLen(c,dl,arraylen);
} else if (!strcasecmp(sub,"genpass") && c->argc == 2) {
char pass[32]; /* 128 bits of actual pseudo random data. */
getRandomHexChars(pass,sizeof(pass));
addReplyBulkCBuffer(c,pass,sizeof(pass));
} else if (!strcasecmp(sub,"genpass") && (c->argc == 2 || c->argc == 3)) {
#define GENPASS_MAX_BITS 4096
char pass[GENPASS_MAX_BITS/8*2]; /* Hex representation. */
long bits = 256; /* By default generate 256 bits passwords. */
if (c->argc == 3 && getLongFromObjectOrReply(c,c->argv[2],&bits,NULL)
!= C_OK) return;
if (bits <= 0 || bits > GENPASS_MAX_BITS) {
addReplyErrorFormat(c,
"ACL GENPASS argument must be the number of "
"bits for the output password, a positive number "
"up to %d",GENPASS_MAX_BITS);
return;
}
long chars = (bits+3)/4; /* Round to number of characters to emit. */
getRandomHexChars(pass,chars);
addReplyBulkCBuffer(c,pass,chars);
} else if (!strcasecmp(sub,"log") && (c->argc == 2 || c->argc ==3)) {
long count = 10; /* Number of entries to emit by default. */
/* Parse the only argument that LOG may have: it could be either
* the number of entires the user wants to display, or alternatively
* the "RESET" command in order to flush the old entires. */
* the number of entries the user wants to display, or alternatively
* the "RESET" command in order to flush the old entries. */
if (c->argc == 3) {
if (!strcasecmp(szFromObj(c->argv[2]),"reset")) {
listSetFreeMethod(ACLLog,(void(*)(const void*))ACLFreeLogEntry);
@ -1858,7 +1912,7 @@ void aclCommand(client *c) {
"DELUSER <username> [...] -- Delete a list of users.",
"CAT -- List available categories.",
"CAT <category> -- List commands inside category.",
"GENPASS -- Generate a secure user password.",
"GENPASS [<bits>] -- Generate a secure user password.",
"WHOAMI -- Return the current connection username.",
"LOG [<count> | RESET] -- Show the ACL log entries.",
NULL

14
src/bio.cpp
View File

@ -154,6 +154,18 @@ void *bioProcessBackgroundJobs(void *arg) {
return NULL;
}
switch (type) {
case BIO_CLOSE_FILE:
redis_set_thread_title("bio_close_file");
break;
case BIO_AOF_FSYNC:
redis_set_thread_title("bio_aof_fsync");
break;
case BIO_LAZY_FREE:
redis_set_thread_title("bio_lazy_free");
break;
}
/* Make the thread killable at any time, so that bioKillThreads()
* can work reliably. */
pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
@ -254,7 +266,7 @@ void bioKillThreads(void) {
int err, j;
for (j = 0; j < BIO_NUM_OPS; j++) {
if (pthread_cancel(bio_threads[j]) == 0) {
if (bio_threads[j] && pthread_cancel(bio_threads[j]) == 0) {
if ((err = pthread_join(bio_threads[j],NULL)) != 0) {
serverLog(LL_WARNING,
"Bio thread for job type #%d can be joined: %s",

10
src/bitops.cpp
View File

@ -269,7 +269,7 @@ int64_t getSignedBitfield(unsigned char *p, uint64_t offset, uint64_t bits) {
* then zero is returned, otherwise in case of overflow, 1 is returned,
* otherwise in case of underflow, -1 is returned.
*
* When non-zero is returned (oferflow or underflow), if not NULL, *limit is
* When non-zero is returned (overflow or underflow), if not NULL, *limit is
* set to the value the operation should result when an overflow happens,
* depending on the specified overflow semantics:
*
@ -556,7 +556,7 @@ void setbitCommand(client *c) {
byteval &= ~(1 << bit);
byteval |= ((on & 0x1) << bit);
((uint8_t*)ptrFromObj(o))[byte] = byteval;
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STRING,"setbit",c->argv[1],c->db->id);
g_pserver->dirty++;
addReply(c, bitval ? shared.cone : shared.czero);
@ -828,11 +828,11 @@ void bitopCommand(client *c) {
/* Store the computed value into the target key */
if (maxlen) {
robj *o = createObject(OBJ_STRING,res);
setKey(c->db,targetkey,o);
setKey(c,c->db,targetkey,o);
notifyKeyspaceEvent(NOTIFY_STRING,"set",targetkey,c->db->id);
decrRefCount(o);
} else if (dbDelete(c->db,targetkey)) {
signalModifiedKey(c->db,targetkey);
signalModifiedKey(c,c->db,targetkey);
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",targetkey,c->db->id);
}
g_pserver->dirty++;
@ -1209,7 +1209,7 @@ void bitfieldGeneric(client *c, int flags) {
}
if (changes) {
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STRING,"setbit",c->argv[1],c->db->id);
g_pserver->dirty += changes;
}

2
src/blocked.cpp
View File

@ -119,7 +119,7 @@ void processUnblockedClients(int iel) {
* the code is conceptually more correct this way. */
if (!(c->flags & CLIENT_BLOCKED)) {
if (c->querybuf && sdslen(c->querybuf) > 0) {
processInputBufferAndReplicate(c);
processInputBuffer(c, CMD_CALL_FULL);
}
}
fastlock_unlock(&c->lock);

124
src/cluster.cpp
View File

@ -2148,7 +2148,7 @@ int clusterProcessPacket(clusterLink *link) {
resetManualFailover();
g_pserver->cluster->mf_end = mstime() + CLUSTER_MF_TIMEOUT;
g_pserver->cluster->mf_slave = sender;
pauseClients(mstime()+(CLUSTER_MF_TIMEOUT*2));
pauseClients(mstime()+(CLUSTER_MF_TIMEOUT*CLUSTER_MF_PAUSE_MULT));
serverLog(LL_WARNING,"Manual failover requested by replica %.40s.",
sender->name);
} else if (type == CLUSTERMSG_TYPE_UPDATE) {
@ -4238,76 +4238,60 @@ void clusterReplyMultiBulkSlots(client *c) {
dictIterator *di = dictGetSafeIterator(g_pserver->cluster->nodes);
while((de = dictNext(di)) != NULL) {
clusterNode *node = (clusterNode*)dictGetVal(de);
int start = -1;
int j = 0, start = -1;
int i, nested_elements = 0;
/* Skip slaves (that are iterated when producing the output of their
* master) and masters not serving any slot. */
if (!nodeIsMaster(node) || node->numslots == 0) continue;
static_assert((CLUSTER_SLOTS % (sizeof(uint32_t)*8)) == 0, "code below assumes the bitfield is a multiple of sizeof(unsinged)");
for (int iw = 0; iw < (CLUSTER_SLOTS/(int)sizeof(uint32_t)/8); ++iw)
{
uint32_t wordCur = reinterpret_cast<uint32_t*>(node->slots)[iw];
if (iw != ((CLUSTER_SLOTS/sizeof(uint32_t)/8)-1))
{
if (start == -1 && wordCur == 0)
continue;
if (start != -1 && (wordCur+1)==0)
continue;
for(i = 0; i < node->numslaves; i++) {
if (nodeFailed(node->slaves[i])) continue;
nested_elements++;
}
for (j = 0; j < CLUSTER_SLOTS; j++) {
int bit, i;
if ((bit = clusterNodeGetSlotBit(node,j)) != 0) {
if (start == -1) start = j;
}
if (start != -1 && (!bit || j == CLUSTER_SLOTS-1)) {
addReplyArrayLen(c, nested_elements + 3); /* slots (2) + master addr (1). */
unsigned ibitStartLoop = iw*sizeof(uint32_t)*8;
for (int j = ibitStartLoop; j < (iw+1)*(int)sizeof(uint32_t)*8; j++) {
int i;
int bit = (int)(wordCur & 1);
wordCur >>= 1;
if (bit != 0) {
if (start == -1) start = j;
if (bit && j == CLUSTER_SLOTS-1) j++;
/* If slot exists in output map, add to it's list.
* else, create a new output map for this slot */
if (start == j-1) {
addReplyLongLong(c, start); /* only one slot; low==high */
addReplyLongLong(c, start);
} else {
addReplyLongLong(c, start); /* low */
addReplyLongLong(c, j-1); /* high */
}
if (start != -1 && (!bit || j == CLUSTER_SLOTS-1)) {
int nested_elements = 3; /* slots (2) + master addr (1). */
void *nested_replylen = addReplyDeferredLen(c);
start = -1;
if (bit && j == CLUSTER_SLOTS-1) j++;
/* First node reply position is always the master */
addReplyArrayLen(c, 3);
addReplyBulkCString(c, node->ip);
addReplyLongLong(c, node->port);
addReplyBulkCBuffer(c, node->name, CLUSTER_NAMELEN);
/* If slot exists in output map, add to it's list.
* else, create a new output map for this slot */
if (start == j-1) {
addReplyLongLong(c, start); /* only one slot; low==high */
addReplyLongLong(c, start);
} else {
addReplyLongLong(c, start); /* low */
addReplyLongLong(c, j-1); /* high */
}
start = -1;
/* First node reply position is always the master */
/* Remaining nodes in reply are replicas for slot range */
for (i = 0; i < node->numslaves; i++) {
/* This loop is copy/pasted from clusterGenNodeDescription()
* with modifications for per-slot node aggregation */
if (nodeFailed(node->slaves[i])) continue;
addReplyArrayLen(c, 3);
addReplyBulkCString(c, node->ip);
addReplyLongLong(c, node->port);
addReplyBulkCBuffer(c, node->name, CLUSTER_NAMELEN);
/* Remaining nodes in reply are replicas for slot range */
for (i = 0; i < node->numslaves; i++) {
/* This loop is copy/pasted from clusterGenNodeDescription()
* with modifications for per-slot node aggregation */
if (nodeFailed(node->slaves[i])) continue;
addReplyArrayLen(c, 3);
addReplyBulkCString(c, node->slaves[i]->ip);
addReplyLongLong(c, node->slaves[i]->port);
addReplyBulkCBuffer(c, node->slaves[i]->name, CLUSTER_NAMELEN);
nested_elements++;
}
setDeferredArrayLen(c, nested_replylen, nested_elements);
num_masters++;
addReplyBulkCString(c, node->slaves[i]->ip);
addReplyLongLong(c, node->slaves[i]->port);
addReplyBulkCBuffer(c, node->slaves[i]->name, CLUSTER_NAMELEN);
}
num_masters++;
}
}
serverAssert(start == -1);
}
dictReleaseIterator(di);
setDeferredArrayLen(c, slot_replylen, num_masters);
}
@ -5049,7 +5033,7 @@ void restoreCommand(client *c) {
setExpire(c,c->db,c->argv[1],nullptr,ttl);
}
objectSetLRUOrLFU(obj,lfu_freq,lru_idle,lru_clock,1000);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_GENERIC,"restore",c->argv[1],c->db->id);
addReply(c,shared.ok);
g_pserver->dirty++;
@ -5164,15 +5148,17 @@ void migrateCloseTimedoutSockets(void) {
dictReleaseIterator(di);
}
/* MIGRATE host port key dbid timeout [COPY | REPLACE | AUTH password]
/* MIGRATE host port key dbid timeout [COPY | REPLACE | AUTH password |
* AUTH2 username password]
*
* On in the multiple keys form:
*
* MIGRATE host port "" dbid timeout [COPY | REPLACE | AUTH password] KEYS key1
* key2 ... keyN */
* MIGRATE host port "" dbid timeout [COPY | REPLACE | AUTH password |
* AUTH2 username password] KEYS key1 key2 ... keyN */
void migrateCommand(client *c) {
migrateCachedSocket *cs;
int copy = 0, replace = 0, j;
char *username = NULL;
char *password = NULL;
long timeout;
long dbid;
@ -5190,7 +5176,7 @@ void migrateCommand(client *c) {
/* Parse additional options */
for (j = 6; j < c->argc; j++) {
int moreargs = j < c->argc-1;
int moreargs = (c->argc-1) - j;
if (!strcasecmp(szFromObj(c->argv[j]),"copy")) {
copy = 1;
} else if (!strcasecmp(szFromObj(c->argv[j]),"replace")) {
@ -5202,6 +5188,13 @@ void migrateCommand(client *c) {
}
j++;
password = szFromObj(c->argv[j]);
} else if (!strcasecmp(szFromObj(c->argv[j]),"auth2")) {
if (moreargs < 2) {
addReply(c,shared.syntaxerr);
return;
}
username = szFromObj(c->argv[++j]);
password = szFromObj(c->argv[++j]);
} else if (!strcasecmp(szFromObj(c->argv[j]),"keys")) {
if (sdslen(szFromObj(c->argv[3])) != 0) {
addReplyError(c,
@ -5262,8 +5255,13 @@ try_again:
/* Authentication */
if (password) {
serverAssertWithInfo(c,NULL,rioWriteBulkCount(&cmd,'*',2));
int arity = username ? 3 : 2;
serverAssertWithInfo(c,NULL,rioWriteBulkCount(&cmd,'*',arity));
serverAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,"AUTH",4));
if (username) {
serverAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,username,
sdslen(username)));
}
serverAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,password,
sdslen(password)));
}
@ -5398,7 +5396,7 @@ try_again:
if (!copy) {
/* No COPY option: remove the local key, signal the change. */
dbDelete(c->db,kv[j]);
signalModifiedKey(c->db,kv[j]);
signalModifiedKey(c,c->db,kv[j]);
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",kv[j],c->db->id);
g_pserver->dirty++;

18
src/config.h
View File

@ -232,4 +232,22 @@ void setproctitle(const char *fmt, ...);
#define USE_ALIGNED_ACCESS
#endif
/* Define for redis_set_thread_title */
#ifdef __linux__
#define redis_set_thread_title(name) pthread_setname_np(pthread_self(), name)
#else
#if (defined __NetBSD__ || defined __FreeBSD__ || defined __OpenBSD__)
#include <pthread_np.h>
#define redis_set_thread_title(name) pthread_set_name_np(pthread_self(), name)
#else
#if (defined __APPLE__ && defined(MAC_OS_X_VERSION_10_7))
int pthread_setname_np(const char *name);
#include <pthread.h>
#define redis_set_thread_title(name) pthread_setname_np(name)
#else
#define redis_set_thread_title(name)
#endif
#endif
#endif
#endif

269
src/crc64.c
View File

@ -1,16 +1,5 @@
/* Redis uses the CRC64 variant with "Jones" coefficients and init value of 0.
*
* Specification of this CRC64 variant follows:
* Name: crc-64-jones
* Width: 64 bites
* Poly: 0xad93d23594c935a9
* Reflected In: True
* Xor_In: 0xffffffffffffffff
* Reflected_Out: True
* Xor_Out: 0x0
* Check("123456789"): 0xe9c6d914c4b8d9ca
*
* Copyright (c) 2012, Salvatore Sanfilippo <antirez at gmail dot com>
/* Copyright (c) 2014, Matt Stancliff <matt@genges.com>
* Copyright (c) 2020, Amazon Web Services
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@ -37,147 +26,100 @@
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE. */
#include <stdint.h>
#include "crc64.h"
#include "crcspeed.h"
static uint64_t crc64_table[8][256] = {{0}};
static const uint64_t crc64_tab[256] = {
UINT64_C(0x0000000000000000), UINT64_C(0x7ad870c830358979),
UINT64_C(0xf5b0e190606b12f2), UINT64_C(0x8f689158505e9b8b),
UINT64_C(0xc038e5739841b68f), UINT64_C(0xbae095bba8743ff6),
UINT64_C(0x358804e3f82aa47d), UINT64_C(0x4f50742bc81f2d04),
UINT64_C(0xab28ecb46814fe75), UINT64_C(0xd1f09c7c5821770c),
UINT64_C(0x5e980d24087fec87), UINT64_C(0x24407dec384a65fe),
UINT64_C(0x6b1009c7f05548fa), UINT64_C(0x11c8790fc060c183),
UINT64_C(0x9ea0e857903e5a08), UINT64_C(0xe478989fa00bd371),
UINT64_C(0x7d08ff3b88be6f81), UINT64_C(0x07d08ff3b88be6f8),
UINT64_C(0x88b81eabe8d57d73), UINT64_C(0xf2606e63d8e0f40a),
UINT64_C(0xbd301a4810ffd90e), UINT64_C(0xc7e86a8020ca5077),
UINT64_C(0x4880fbd87094cbfc), UINT64_C(0x32588b1040a14285),
UINT64_C(0xd620138fe0aa91f4), UINT64_C(0xacf86347d09f188d),
UINT64_C(0x2390f21f80c18306), UINT64_C(0x594882d7b0f40a7f),
UINT64_C(0x1618f6fc78eb277b), UINT64_C(0x6cc0863448deae02),
UINT64_C(0xe3a8176c18803589), UINT64_C(0x997067a428b5bcf0),
UINT64_C(0xfa11fe77117cdf02), UINT64_C(0x80c98ebf2149567b),
UINT64_C(0x0fa11fe77117cdf0), UINT64_C(0x75796f2f41224489),
UINT64_C(0x3a291b04893d698d), UINT64_C(0x40f16bccb908e0f4),
UINT64_C(0xcf99fa94e9567b7f), UINT64_C(0xb5418a5cd963f206),
UINT64_C(0x513912c379682177), UINT64_C(0x2be1620b495da80e),
UINT64_C(0xa489f35319033385), UINT64_C(0xde51839b2936bafc),
UINT64_C(0x9101f7b0e12997f8), UINT64_C(0xebd98778d11c1e81),
UINT64_C(0x64b116208142850a), UINT64_C(0x1e6966e8b1770c73),
UINT64_C(0x8719014c99c2b083), UINT64_C(0xfdc17184a9f739fa),
UINT64_C(0x72a9e0dcf9a9a271), UINT64_C(0x08719014c99c2b08),
UINT64_C(0x4721e43f0183060c), UINT64_C(0x3df994f731b68f75),
UINT64_C(0xb29105af61e814fe), UINT64_C(0xc849756751dd9d87),
UINT64_C(0x2c31edf8f1d64ef6), UINT64_C(0x56e99d30c1e3c78f),
UINT64_C(0xd9810c6891bd5c04), UINT64_C(0xa3597ca0a188d57d),
UINT64_C(0xec09088b6997f879), UINT64_C(0x96d1784359a27100),
UINT64_C(0x19b9e91b09fcea8b), UINT64_C(0x636199d339c963f2),
UINT64_C(0xdf7adabd7a6e2d6f), UINT64_C(0xa5a2aa754a5ba416),
UINT64_C(0x2aca3b2d1a053f9d), UINT64_C(0x50124be52a30b6e4),
UINT64_C(0x1f423fcee22f9be0), UINT64_C(0x659a4f06d21a1299),
UINT64_C(0xeaf2de5e82448912), UINT64_C(0x902aae96b271006b),
UINT64_C(0x74523609127ad31a), UINT64_C(0x0e8a46c1224f5a63),
UINT64_C(0x81e2d7997211c1e8), UINT64_C(0xfb3aa75142244891),
UINT64_C(0xb46ad37a8a3b6595), UINT64_C(0xceb2a3b2ba0eecec),
UINT64_C(0x41da32eaea507767), UINT64_C(0x3b024222da65fe1e),
UINT64_C(0xa2722586f2d042ee), UINT64_C(0xd8aa554ec2e5cb97),
UINT64_C(0x57c2c41692bb501c), UINT64_C(0x2d1ab4dea28ed965),
UINT64_C(0x624ac0f56a91f461), UINT64_C(0x1892b03d5aa47d18),
UINT64_C(0x97fa21650afae693), UINT64_C(0xed2251ad3acf6fea),
UINT64_C(0x095ac9329ac4bc9b), UINT64_C(0x7382b9faaaf135e2),
UINT64_C(0xfcea28a2faafae69), UINT64_C(0x8632586aca9a2710),
UINT64_C(0xc9622c4102850a14), UINT64_C(0xb3ba5c8932b0836d),
UINT64_C(0x3cd2cdd162ee18e6), UINT64_C(0x460abd1952db919f),
UINT64_C(0x256b24ca6b12f26d), UINT64_C(0x5fb354025b277b14),
UINT64_C(0xd0dbc55a0b79e09f), UINT64_C(0xaa03b5923b4c69e6),
UINT64_C(0xe553c1b9f35344e2), UINT64_C(0x9f8bb171c366cd9b),
UINT64_C(0x10e3202993385610), UINT64_C(0x6a3b50e1a30ddf69),
UINT64_C(0x8e43c87e03060c18), UINT64_C(0xf49bb8b633338561),
UINT64_C(0x7bf329ee636d1eea), UINT64_C(0x012b592653589793),
UINT64_C(0x4e7b2d0d9b47ba97), UINT64_C(0x34a35dc5ab7233ee),
UINT64_C(0xbbcbcc9dfb2ca865), UINT64_C(0xc113bc55cb19211c),
UINT64_C(0x5863dbf1e3ac9dec), UINT64_C(0x22bbab39d3991495),
UINT64_C(0xadd33a6183c78f1e), UINT64_C(0xd70b4aa9b3f20667),
UINT64_C(0x985b3e827bed2b63), UINT64_C(0xe2834e4a4bd8a21a),
UINT64_C(0x6debdf121b863991), UINT64_C(0x1733afda2bb3b0e8),
UINT64_C(0xf34b37458bb86399), UINT64_C(0x8993478dbb8deae0),
UINT64_C(0x06fbd6d5ebd3716b), UINT64_C(0x7c23a61ddbe6f812),
UINT64_C(0x3373d23613f9d516), UINT64_C(0x49aba2fe23cc5c6f),
UINT64_C(0xc6c333a67392c7e4), UINT64_C(0xbc1b436e43a74e9d),
UINT64_C(0x95ac9329ac4bc9b5), UINT64_C(0xef74e3e19c7e40cc),
UINT64_C(0x601c72b9cc20db47), UINT64_C(0x1ac40271fc15523e),
UINT64_C(0x5594765a340a7f3a), UINT64_C(0x2f4c0692043ff643),
UINT64_C(0xa02497ca54616dc8), UINT64_C(0xdafce7026454e4b1),
UINT64_C(0x3e847f9dc45f37c0), UINT64_C(0x445c0f55f46abeb9),
UINT64_C(0xcb349e0da4342532), UINT64_C(0xb1eceec59401ac4b),
UINT64_C(0xfebc9aee5c1e814f), UINT64_C(0x8464ea266c2b0836),
UINT64_C(0x0b0c7b7e3c7593bd), UINT64_C(0x71d40bb60c401ac4),
UINT64_C(0xe8a46c1224f5a634), UINT64_C(0x927c1cda14c02f4d),
UINT64_C(0x1d148d82449eb4c6), UINT64_C(0x67ccfd4a74ab3dbf),
UINT64_C(0x289c8961bcb410bb), UINT64_C(0x5244f9a98c8199c2),
UINT64_C(0xdd2c68f1dcdf0249), UINT64_C(0xa7f41839ecea8b30),
UINT64_C(0x438c80a64ce15841), UINT64_C(0x3954f06e7cd4d138),
UINT64_C(0xb63c61362c8a4ab3), UINT64_C(0xcce411fe1cbfc3ca),
UINT64_C(0x83b465d5d4a0eece), UINT64_C(0xf96c151de49567b7),
UINT64_C(0x76048445b4cbfc3c), UINT64_C(0x0cdcf48d84fe7545),
UINT64_C(0x6fbd6d5ebd3716b7), UINT64_C(0x15651d968d029fce),
UINT64_C(0x9a0d8ccedd5c0445), UINT64_C(0xe0d5fc06ed698d3c),
UINT64_C(0xaf85882d2576a038), UINT64_C(0xd55df8e515432941),
UINT64_C(0x5a3569bd451db2ca), UINT64_C(0x20ed197575283bb3),
UINT64_C(0xc49581ead523e8c2), UINT64_C(0xbe4df122e51661bb),
UINT64_C(0x3125607ab548fa30), UINT64_C(0x4bfd10b2857d7349),
UINT64_C(0x04ad64994d625e4d), UINT64_C(0x7e7514517d57d734),
UINT64_C(0xf11d85092d094cbf), UINT64_C(0x8bc5f5c11d3cc5c6),
UINT64_C(0x12b5926535897936), UINT64_C(0x686de2ad05bcf04f),
UINT64_C(0xe70573f555e26bc4), UINT64_C(0x9ddd033d65d7e2bd),
UINT64_C(0xd28d7716adc8cfb9), UINT64_C(0xa85507de9dfd46c0),
UINT64_C(0x273d9686cda3dd4b), UINT64_C(0x5de5e64efd965432),
UINT64_C(0xb99d7ed15d9d8743), UINT64_C(0xc3450e196da80e3a),
UINT64_C(0x4c2d9f413df695b1), UINT64_C(0x36f5ef890dc31cc8),
UINT64_C(0x79a59ba2c5dc31cc), UINT64_C(0x037deb6af5e9b8b5),
UINT64_C(0x8c157a32a5b7233e), UINT64_C(0xf6cd0afa9582aa47),
UINT64_C(0x4ad64994d625e4da), UINT64_C(0x300e395ce6106da3),
UINT64_C(0xbf66a804b64ef628), UINT64_C(0xc5bed8cc867b7f51),
UINT64_C(0x8aeeace74e645255), UINT64_C(0xf036dc2f7e51db2c),
UINT64_C(0x7f5e4d772e0f40a7), UINT64_C(0x05863dbf1e3ac9de),
UINT64_C(0xe1fea520be311aaf), UINT64_C(0x9b26d5e88e0493d6),
UINT64_C(0x144e44b0de5a085d), UINT64_C(0x6e963478ee6f8124),
UINT64_C(0x21c640532670ac20), UINT64_C(0x5b1e309b16452559),
UINT64_C(0xd476a1c3461bbed2), UINT64_C(0xaeaed10b762e37ab),
UINT64_C(0x37deb6af5e9b8b5b), UINT64_C(0x4d06c6676eae0222),
UINT64_C(0xc26e573f3ef099a9), UINT64_C(0xb8b627f70ec510d0),
UINT64_C(0xf7e653dcc6da3dd4), UINT64_C(0x8d3e2314f6efb4ad),
UINT64_C(0x0256b24ca6b12f26), UINT64_C(0x788ec2849684a65f),
UINT64_C(0x9cf65a1b368f752e), UINT64_C(0xe62e2ad306bafc57),
UINT64_C(0x6946bb8b56e467dc), UINT64_C(0x139ecb4366d1eea5),
UINT64_C(0x5ccebf68aecec3a1), UINT64_C(0x2616cfa09efb4ad8),
UINT64_C(0xa97e5ef8cea5d153), UINT64_C(0xd3a62e30fe90582a),
UINT64_C(0xb0c7b7e3c7593bd8), UINT64_C(0xca1fc72bf76cb2a1),
UINT64_C(0x45775673a732292a), UINT64_C(0x3faf26bb9707a053),
UINT64_C(0x70ff52905f188d57), UINT64_C(0x0a2722586f2d042e),
UINT64_C(0x854fb3003f739fa5), UINT64_C(0xff97c3c80f4616dc),
UINT64_C(0x1bef5b57af4dc5ad), UINT64_C(0x61372b9f9f784cd4),
UINT64_C(0xee5fbac7cf26d75f), UINT64_C(0x9487ca0fff135e26),
UINT64_C(0xdbd7be24370c7322), UINT64_C(0xa10fceec0739fa5b),
UINT64_C(0x2e675fb4576761d0), UINT64_C(0x54bf2f7c6752e8a9),
UINT64_C(0xcdcf48d84fe75459), UINT64_C(0xb71738107fd2dd20),
UINT64_C(0x387fa9482f8c46ab), UINT64_C(0x42a7d9801fb9cfd2),
UINT64_C(0x0df7adabd7a6e2d6), UINT64_C(0x772fdd63e7936baf),
UINT64_C(0xf8474c3bb7cdf024), UINT64_C(0x829f3cf387f8795d),
UINT64_C(0x66e7a46c27f3aa2c), UINT64_C(0x1c3fd4a417c62355),
UINT64_C(0x935745fc4798b8de), UINT64_C(0xe98f353477ad31a7),
UINT64_C(0xa6df411fbfb21ca3), UINT64_C(0xdc0731d78f8795da),
UINT64_C(0x536fa08fdfd90e51), UINT64_C(0x29b7d047efec8728),
};
#define POLY UINT64_C(0xad93d23594c935a9)
/******************** BEGIN GENERATED PYCRC FUNCTIONS ********************/
/**
* Generated on Sun Dec 21 14:14:07 2014,
* by pycrc v0.8.2, https://www.tty1.net/pycrc/
*
* LICENSE ON GENERATED CODE:
* ==========================
* As of version 0.6, pycrc is released under the terms of the MIT licence.
* The code generated by pycrc is not considered a substantial portion of the
* software, therefore the author of pycrc will not claim any copyright on
* the generated code.
* ==========================
*
* CRC configuration:
* Width = 64
* Poly = 0xad93d23594c935a9
* XorIn = 0xffffffffffffffff
* ReflectIn = True
* XorOut = 0x0000000000000000
* ReflectOut = True
* Algorithm = bit-by-bit-fast
*
* Modifications after generation (by matt):
* - included finalize step in-line with update for single-call generation
* - re-worked some inner variable architectures
* - adjusted function parameters to match expected prototypes.
*****************************************************************************/
uint64_t crc64(uint64_t crc, const unsigned char *s, uint64_t l) {
uint64_t j;
/**
* Reflect all bits of a \a data word of \a data_len bytes.
*
* \param data The data word to be reflected.
* \param data_len The width of \a data expressed in number of bits.
* \return The reflected data.
*****************************************************************************/
static inline uint_fast64_t crc_reflect(uint_fast64_t data, size_t data_len) {
uint_fast64_t ret = data & 0x01;
for (j = 0; j < l; j++) {
uint8_t byte = s[j];
crc = crc64_tab[(uint8_t)crc ^ byte] ^ (crc >> 8);
for (size_t i = 1; i < data_len; i++) {
data >>= 1;
ret = (ret << 1) | (data & 0x01);
}
return crc;
return ret;
}
/**
* Update the crc value with new data.
*
* \param crc The current crc value.
* \param data Pointer to a buffer of \a data_len bytes.
* \param data_len Number of bytes in the \a data buffer.
* \return The updated crc value.
******************************************************************************/
uint64_t _crc64(uint_fast64_t crc, const void *in_data, const uint64_t len) {
const uint8_t *data = in_data;
unsigned long long bit;
for (uint64_t offset = 0; offset < len; offset++) {
uint8_t c = data[offset];
for (uint_fast8_t i = 0x01; i & 0xff; i <<= 1) {
bit = crc & 0x8000000000000000;
if (c & i) {
bit = !bit;
}
crc <<= 1;
if (bit) {
crc ^= POLY;
}
}
crc &= 0xffffffffffffffff;
}
crc = crc & 0xffffffffffffffff;
return crc_reflect(crc, 64) ^ 0x0000000000000000;
}
/******************** END GENERATED PYCRC FUNCTIONS ********************/
/* Initializes the 16KB lookup tables. */
void crc64_init(void) {
crcspeed64native_init(_crc64, crc64_table);
}
/* Compute crc64 */
uint64_t crc64(uint64_t crc, const unsigned char *s, uint64_t l) {
return crcspeed64native(crc64_table, crc, (void *) s, l);
}
/* Test main */
@ -188,8 +130,31 @@ uint64_t crc64(uint64_t crc, const unsigned char *s, uint64_t l) {
int crc64Test(int argc, char *argv[]) {
UNUSED(argc);
UNUSED(argv);
printf("e9c6d914c4b8d9ca == %016llx\n",
(unsigned long long) crc64(0,(unsigned char*)"123456789",9));
crc64_init();
printf("[calcula]: e9c6d914c4b8d9ca == %016" PRIx64 "\n",
(uint64_t)_crc64(0, "123456789", 9));
printf("[64speed]: e9c6d914c4b8d9ca == %016" PRIx64 "\n",
(uint64_t)crc64(0, "123456789", 9));
char li[] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed "
"do eiusmod tempor incididunt ut labore et dolore magna "
"aliqua. Ut enim ad minim veniam, quis nostrud exercitation "
"ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis "
"aute irure dolor in reprehenderit in voluptate velit esse "
"cillum dolore eu fugiat nulla pariatur. Excepteur sint "
"occaecat cupidatat non proident, sunt in culpa qui officia "
"deserunt mollit anim id est laborum.";
printf("[calcula]: c7794709e69683b3 == %016" PRIx64 "\n",
(uint64_t)_crc64(0, li, sizeof(li)));
printf("[64speed]: c7794709e69683b3 == %016" PRIx64 "\n",
(uint64_t)crc64(0, li, sizeof(li)));
return 0;
}
#endif
#ifdef REDIS_TEST_MAIN
int main(int argc, char *argv[]) {
return crc64Test(argc, argv);
}
#endif

1
src/crc64.h
View File

@ -7,6 +7,7 @@
extern "C" {
#endif
void crc64_init(void);
uint64_t crc64(uint64_t crc, const unsigned char *s, uint64_t l);
#ifdef REDIS_TEST

281
src/crcspeed.c Normal file
View File

@ -0,0 +1,281 @@
/*
* Copyright (C) 2013 Mark Adler
* Originally by: crc64.c Version 1.4 16 Dec 2013 Mark Adler
* Modifications by Matt Stancliff <matt@genges.com>:
* - removed CRC64-specific behavior
* - added generation of lookup tables by parameters
* - removed inversion of CRC input/result
* - removed automatic initialization in favor of explicit initialization
This software is provided 'as-is', without any express or implied
warranty. In no event will the author be held liable for any damages
arising from the use of this software.
Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:
1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.
Mark Adler
madler@alumni.caltech.edu
*/
#include "crcspeed.h"
/* Fill in a CRC constants table. */
void crcspeed64little_init(crcfn64 crcfn, uint64_t table[8][256]) {
uint64_t crc;
/* generate CRCs for all single byte sequences */
for (int n = 0; n < 256; n++) {
table[0][n] = crcfn(0, &n, 1);
}
/* generate nested CRC table for future slice-by-8 lookup */
for (int n = 0; n < 256; n++) {
crc = table[0][n];
for (int k = 1; k < 8; k++) {
crc = table[0][crc & 0xff] ^ (crc >> 8);
table[k][n] = crc;
}
}
}
void crcspeed16little_init(crcfn16 crcfn, uint16_t table[8][256]) {
uint16_t crc;
/* generate CRCs for all single byte sequences */
for (int n = 0; n < 256; n++) {
table[0][n] = crcfn(0, &n, 1);
}
/* generate nested CRC table for future slice-by-8 lookup */
for (int n = 0; n < 256; n++) {
crc = table[0][n];
for (int k = 1; k < 8; k++) {
crc = table[0][(crc >> 8) & 0xff] ^ (crc << 8);
table[k][n] = crc;
}
}
}
/* Reverse the bytes in a 64-bit word. */
static inline uint64_t rev8(uint64_t a) {
#if defined(__GNUC__) || defined(__clang__)
return __builtin_bswap64(a);
#else
uint64_t m;
m = UINT64_C(0xff00ff00ff00ff);
a = ((a >> 8) & m) | (a & m) << 8;
m = UINT64_C(0xffff0000ffff);
a = ((a >> 16) & m) | (a & m) << 16;
return a >> 32 | a << 32;
#endif
}
/* This function is called once to initialize the CRC table for use on a
big-endian architecture. */
void crcspeed64big_init(crcfn64 fn, uint64_t big_table[8][256]) {
/* Create the little endian table then reverse all the entires. */
crcspeed64little_init(fn, big_table);
for (int k = 0; k < 8; k++) {
for (int n = 0; n < 256; n++) {
big_table[k][n] = rev8(big_table[k][n]);
}
}
}
void crcspeed16big_init(crcfn16 fn, uint16_t big_table[8][256]) {
/* Create the little endian table then reverse all the entires. */
crcspeed16little_init(fn, big_table);
for (int k = 0; k < 8; k++) {
for (int n = 0; n < 256; n++) {
big_table[k][n] = rev8(big_table[k][n]);
}
}
}
/* Calculate a non-inverted CRC multiple bytes at a time on a little-endian
* architecture. If you need inverted CRC, invert *before* calling and invert
* *after* calling.
* 64 bit crc = process 8 bytes at once;
*/
uint64_t crcspeed64little(uint64_t little_table[8][256], uint64_t crc,
void *buf, size_t len) {
unsigned char *next = buf;
/* process individual bytes until we reach an 8-byte aligned pointer */
while (len && ((uintptr_t)next & 7) != 0) {
crc = little_table[0][(crc ^ *next++) & 0xff] ^ (crc >> 8);
len--;
}
/* fast middle processing, 8 bytes (aligned!) per loop */
while (len >= 8) {
crc ^= *(uint64_t *)next;
crc = little_table[7][crc & 0xff] ^
little_table[6][(crc >> 8) & 0xff] ^
little_table[5][(crc >> 16) & 0xff] ^
little_table[4][(crc >> 24) & 0xff] ^
little_table[3][(crc >> 32) & 0xff] ^
little_table[2][(crc >> 40) & 0xff] ^
little_table[1][(crc >> 48) & 0xff] ^
little_table[0][crc >> 56];
next += 8;
len -= 8;
}
/* process remaining bytes (can't be larger than 8) */
while (len) {
crc = little_table[0][(crc ^ *next++) & 0xff] ^ (crc >> 8);
len--;
}
return crc;
}
uint16_t crcspeed16little(uint16_t little_table[8][256], uint16_t crc,
void *buf, size_t len) {
unsigned char *next = buf;
/* process individual bytes until we reach an 8-byte aligned pointer */
while (len && ((uintptr_t)next & 7) != 0) {
crc = little_table[0][((crc >> 8) ^ *next++) & 0xff] ^ (crc << 8);
len--;
}
/* fast middle processing, 8 bytes (aligned!) per loop */
while (len >= 8) {
uint64_t n = *(uint64_t *)next;
crc = little_table[7][(n & 0xff) ^ ((crc >> 8) & 0xff)] ^
little_table[6][((n >> 8) & 0xff) ^ (crc & 0xff)] ^
little_table[5][(n >> 16) & 0xff] ^
little_table[4][(n >> 24) & 0xff] ^
little_table[3][(n >> 32) & 0xff] ^
little_table[2][(n >> 40) & 0xff] ^
little_table[1][(n >> 48) & 0xff] ^
little_table[0][n >> 56];
next += 8;
len -= 8;
}
/* process remaining bytes (can't be larger than 8) */
while (len) {
crc = little_table[0][((crc >> 8) ^ *next++) & 0xff] ^ (crc << 8);
len--;
}
return crc;
}
/* Calculate a non-inverted CRC eight bytes at a time on a big-endian
* architecture.
*/
uint64_t crcspeed64big(uint64_t big_table[8][256], uint64_t crc, void *buf,
size_t len) {
unsigned char *next = buf;
crc = rev8(crc);
while (len && ((uintptr_t)next & 7) != 0) {
crc = big_table[0][(crc >> 56) ^ *next++] ^ (crc << 8);
len--;
}
while (len >= 8) {
crc ^= *(uint64_t *)next;
crc = big_table[0][crc & 0xff] ^
big_table[1][(crc >> 8) & 0xff] ^
big_table[2][(crc >> 16) & 0xff] ^
big_table[3][(crc >> 24) & 0xff] ^
big_table[4][(crc >> 32) & 0xff] ^
big_table[5][(crc >> 40) & 0xff] ^
big_table[6][(crc >> 48) & 0xff] ^
big_table[7][crc >> 56];
next += 8;
len -= 8;
}
while (len) {
crc = big_table[0][(crc >> 56) ^ *next++] ^ (crc << 8);
len--;
}
return rev8(crc);
}
/* WARNING: Completely untested on big endian architecture. Possibly broken. */
uint16_t crcspeed16big(uint16_t big_table[8][256], uint16_t crc_in, void *buf,
size_t len) {
unsigned char *next = buf;
uint64_t crc = crc_in;
crc = rev8(crc);
while (len && ((uintptr_t)next & 7) != 0) {
crc = big_table[0][((crc >> (56 - 8)) ^ *next++) & 0xff] ^ (crc >> 8);
len--;
}
while (len >= 8) {
uint64_t n = *(uint64_t *)next;
crc = big_table[0][(n & 0xff) ^ ((crc >> (56 - 8)) & 0xff)] ^
big_table[1][((n >> 8) & 0xff) ^ (crc & 0xff)] ^
big_table[2][(n >> 16) & 0xff] ^
big_table[3][(n >> 24) & 0xff] ^
big_table[4][(n >> 32) & 0xff] ^
big_table[5][(n >> 40) & 0xff] ^
big_table[6][(n >> 48) & 0xff] ^
big_table[7][n >> 56];
next += 8;
len -= 8;
}
while (len) {
crc = big_table[0][((crc >> (56 - 8)) ^ *next++) & 0xff] ^ (crc >> 8);
len--;
}
return rev8(crc);
}
/* Return the CRC of buf[0..len-1] with initial crc, processing eight bytes
at a time using passed-in lookup table.
This selects one of two routines depending on the endianess of
the architecture. */
uint64_t crcspeed64native(uint64_t table[8][256], uint64_t crc, void *buf,
size_t len) {
uint64_t n = 1;
return *(char *)&n ? crcspeed64little(table, crc, buf, len)
: crcspeed64big(table, crc, buf, len);
}
uint16_t crcspeed16native(uint16_t table[8][256], uint16_t crc, void *buf,
size_t len) {
uint64_t n = 1;
return *(char *)&n ? crcspeed16little(table, crc, buf, len)
: crcspeed16big(table, crc, buf, len);
}
/* Initialize CRC lookup table in architecture-dependent manner. */
void crcspeed64native_init(crcfn64 fn, uint64_t table[8][256]) {
uint64_t n = 1;
*(char *)&n ? crcspeed64little_init(fn, table)
: crcspeed64big_init(fn, table);
}
void crcspeed16native_init(crcfn16 fn, uint16_t table[8][256]) {
uint64_t n = 1;
*(char *)&n ? crcspeed16little_init(fn, table)
: crcspeed16big_init(fn, table);
}

60
src/crcspeed.h Normal file
View File

@ -0,0 +1,60 @@
/* Copyright (c) 2014, Matt Stancliff <matt@genges.com>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE. */
#ifndef CRCSPEED_H
#define CRCSPEED_H
#include <inttypes.h>
#include <stdio.h>
typedef uint64_t (*crcfn64)(uint64_t, const void *, const uint64_t);
typedef uint16_t (*crcfn16)(uint16_t, const void *, const uint64_t);
/* CRC-64 */
void crcspeed64little_init(crcfn64 fn, uint64_t table[8][256]);
void crcspeed64big_init(crcfn64 fn, uint64_t table[8][256]);
void crcspeed64native_init(crcfn64 fn, uint64_t table[8][256]);
uint64_t crcspeed64little(uint64_t table[8][256], uint64_t crc, void *buf,
size_t len);
uint64_t crcspeed64big(uint64_t table[8][256], uint64_t crc, void *buf,
size_t len);
uint64_t crcspeed64native(uint64_t table[8][256], uint64_t crc, void *buf,
size_t len);
/* CRC-16 */
void crcspeed16little_init(crcfn16 fn, uint16_t table[8][256]);
void crcspeed16big_init(crcfn16 fn, uint16_t table[8][256]);
void crcspeed16native_init(crcfn16 fn, uint16_t table[8][256]);
uint16_t crcspeed16little(uint16_t table[8][256], uint16_t crc, void *buf,
size_t len);
uint16_t crcspeed16big(uint16_t table[8][256], uint16_t crc, void *buf,
size_t len);
uint16_t crcspeed16native(uint16_t table[8][256], uint16_t crc, void *buf,
size_t len);
#endif

2
src/cron.cpp
View File

@ -66,7 +66,7 @@ void cronCommand(client *c)
spjob->vecargs.emplace_back(sdsdup(szFromObj(c->argv[i])));
robj *o = createObject(OBJ_CRON, spjob.release());
setKey(c->db, c->argv[ARG_NAME], o);
setKey(c, c->db, c->argv[ARG_NAME], o);
decrRefCount(o);
// use an expire to trigger execution. Note: We use a subkey expire here so legacy clients don't delete it.
setExpire(c, c->db, c->argv[ARG_NAME], c->argv[ARG_NAME], base + interval);

30
src/db.cpp
View File

@ -313,8 +313,10 @@ int dbMerge(redisDb *db, robj *key, robj *val, int fReplace)
* 3) The expire time of the key is reset (the key is made persistent),
* unless 'keepttl' is true.
*
* All the new keys in the database should be created via this interface. */
void genericSetKey(redisDb *db, robj *key, robj *val, int keepttl, int signal) {
* All the new keys in the database should be created via this interface.
* The client 'c' argument may be set to NULL if the operation is performed
* in a context where there is no clear client performing the operation. */
void genericSetKey(client *c, redisDb *db, robj *key, robj *val, int keepttl, int signal) {
dictEntry *de = dictFind(db->pdict, ptrFromObj(key));
if (de == NULL) {
dbAdd(db,key,val);
@ -323,12 +325,12 @@ void genericSetKey(redisDb *db, robj *key, robj *val, int keepttl, int signal) {
dbOverwriteCore(db,de,key,val,!!g_pserver->fActiveReplica,!keepttl);
}
incrRefCount(val);
if (signal) signalModifiedKey(db,key);
if (signal) signalModifiedKey(c,db,key);
}
/* Common case for genericSetKey() where the TTL is not retained. */
void setKey(redisDb *db, robj *key, robj *val) {
genericSetKey(db,key,val,0,1);
void setKey(client *c, redisDb *db, robj *key, robj *val) {
genericSetKey(c,db,key,val,0,1);
}
/* Return true if the specified key exists in the specified database.
@ -553,9 +555,11 @@ long long dbTotalServerKeyCount() {
* Every time a DB is flushed the function signalFlushDb() is called.
*----------------------------------------------------------------------------*/
void signalModifiedKey(redisDb *db, robj *key) {
/* Note that the 'c' argument may be NULL if the key was modified out of
* a context of a client. */
void signalModifiedKey(client *c, redisDb *db, robj *key) {
touchWatchedKey(db,key);
trackingInvalidateKey(key);
trackingInvalidateKey(c,key);
}
void signalFlushedDb(int dbid) {
@ -649,7 +653,7 @@ void delGenericCommand(client *c, int lazy) {
int deleted = lazy ? dbAsyncDelete(c->db,c->argv[j]) :
dbSyncDelete(c->db,c->argv[j]);
if (deleted) {
signalModifiedKey(c->db,c->argv[j]);
signalModifiedKey(c,c->db,c->argv[j]);
notifyKeyspaceEvent(NOTIFY_GENERIC,
"del",c->argv[j],c->db->id);
g_pserver->dirty++;
@ -1096,8 +1100,8 @@ void renameGenericCommand(client *c, int nx) {
dbAdd(c->db,c->argv[2],o);
if (spexpire != nullptr)
setExpire(c,c->db,c->argv[2],std::move(*spexpire));
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c->db,c->argv[2]);
signalModifiedKey(c,c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[2]);
notifyKeyspaceEvent(NOTIFY_GENERIC,"rename_from",
c->argv[1],c->db->id);
notifyKeyspaceEvent(NOTIFY_GENERIC,"rename_to",
@ -1174,8 +1178,8 @@ void moveCommand(client *c) {
dbAdd(dst,c->argv[1],o);
if (spexpire != nullptr) setExpire(c,dst,c->argv[1],std::move(*spexpire));
signalModifiedKey(src,c->argv[1]);
signalModifiedKey(dst,c->argv[1]);
signalModifiedKey(c,src,c->argv[1]);
signalModifiedKey(c,dst,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_GENERIC,
"move_from",c->argv[1],src->id);
notifyKeyspaceEvent(NOTIFY_GENERIC,
@ -1604,7 +1608,7 @@ int expireIfNeeded(redisDb *db, robj *key) {
"expired",key,db->id);
int retval = g_pserver->lazyfree_lazy_expire ? dbAsyncDelete(db,key) :
dbSyncDelete(db,key);
if (retval) signalModifiedKey(db,key);
if (retval) signalModifiedKey(NULL,db,key);
return retval;
}

2
src/debug.cpp
View File

@ -602,7 +602,7 @@ NULL
memcpy(ptrFromObj(val), buf, valsize<=buflen? valsize: buflen);
}
dbAdd(c->db,key,val);
signalModifiedKey(c->db,key);
signalModifiedKey(c,c->db,key);
decrRefCount(key);
}
addReply(c,shared.ok);

32
src/evict.cpp
View File

@ -466,9 +466,10 @@ int freeMemoryIfNeeded(void) {
if (listLength(g_pserver->masters) && g_pserver->repl_slave_ignore_maxmemory && !g_pserver->fActiveReplica) return C_OK;
size_t mem_reported, mem_tofree, mem_freed;
mstime_t latency, eviction_latency;
mstime_t latency, eviction_latency, lazyfree_latency;
long long delta;
int slaves = listLength(g_pserver->slaves);
int result = C_ERR;
/* When clients are paused the dataset should be static not just from the
* POV of clients not being able to write, but also from the POV of
@ -479,10 +480,10 @@ int freeMemoryIfNeeded(void) {
mem_freed = 0;
latencyStartMonitor(latency);
if (g_pserver->maxmemory_policy == MAXMEMORY_NO_EVICTION)
goto cant_free; /* We need to free memory, but policy forbids. */
latencyStartMonitor(latency);
while (mem_freed < mem_tofree) {
int j, k, i;
static unsigned int next_db = 0;
@ -598,9 +599,9 @@ int freeMemoryIfNeeded(void) {
dbAsyncDelete(db,keyobj);
else
dbSyncDelete(db,keyobj);
signalModifiedKey(NULL,db,keyobj);
latencyEndMonitor(eviction_latency);
latencyAddSampleIfNeeded("eviction-del",eviction_latency);
latencyRemoveNestedEvent(latency,eviction_latency);
delta -= (long long) zmalloc_used_memory();
mem_freed += delta;
g_pserver->stat_evictedkeys++;
@ -629,25 +630,30 @@ int freeMemoryIfNeeded(void) {
}
}
} else {
latencyEndMonitor(latency);
latencyAddSampleIfNeeded("eviction-cycle",latency);
goto cant_free; /* nothing to free... */
}
}
latencyEndMonitor(latency);
latencyAddSampleIfNeeded("eviction-cycle",latency);
return C_OK;
result = C_OK;
cant_free:
/* We are here if we are not able to reclaim memory. There is only one
* last thing we can try: check if the lazyfree thread has jobs in queue
* and wait... */
while(bioPendingJobsOfType(BIO_LAZY_FREE)) {
if (((mem_reported - zmalloc_used_memory()) + mem_freed) >= mem_tofree)
break;
usleep(1000);
if (result != C_OK) {
latencyStartMonitor(lazyfree_latency);
while(bioPendingJobsOfType(BIO_LAZY_FREE)) {
if (getMaxmemoryState(NULL,NULL,NULL,NULL) == C_OK) {
result = C_OK;
break;
}
usleep(1000);
}
latencyEndMonitor(lazyfree_latency);
latencyAddSampleIfNeeded("eviction-lazyfree",lazyfree_latency);
}
return C_ERR;
latencyEndMonitor(latency);
latencyAddSampleIfNeeded("eviction-cycle",latency);
return result;
}
/* This is a wrapper for freeMemoryIfNeeded() that only really calls the

33
src/expire.cpp
View File

@ -33,6 +33,17 @@
#include "server.h"
#include "cron.h"
/* Helper function for the activeExpireCycle() function.
* This function will try to expire the key that is stored in the hash table
* entry 'de' of the 'expires' hash table of a Redis database.
*
* If the key is found to be expired, it is removed from the database and
* 1 is returned. Otherwise no operation is performed and 0 is returned.
*
* When a key is expired, g_pserver->stat_expiredkeys is incremented.
*
* The parameter 'now' is the current time in milliseconds as is passed
* to the function to avoid too many gettimeofday() syscalls. */
void activeExpireCycleExpireFullKey(redisDb *db, const char *key) {
robj *keyobj = createStringObject(key,sdslen(key));
@ -43,7 +54,7 @@ void activeExpireCycleExpireFullKey(redisDb *db, const char *key) {
dbSyncDelete(db,keyobj);
notifyKeyspaceEvent(NOTIFY_EXPIRED,
"expired",keyobj,db->id);
if (g_pserver->tracking_clients) trackingInvalidateKey(keyobj);
trackingInvalidateKey(NULL, keyobj);
decrRefCount(keyobj);
g_pserver->stat_expiredkeys++;
}
@ -56,17 +67,7 @@ void activeExpireCycleExpireFullKey(redisDb *db, const char *key) {
* if no access is performed on them.
*----------------------------------------------------------------------------*/
/* Helper function for the activeExpireCycle() function.
* This function will try to expire the key that is stored in the hash table
* entry 'de' of the 'expires' hash table of a Redis database.
*
* If the key is found to be expired, it is removed from the database and
* 1 is returned. Otherwise no operation is performed and 0 is returned.
*
* When a key is expired, g_pserver->stat_expiredkeys is incremented.
*
* The parameter 'now' is the current time in milliseconds as is passed
* to the function to avoid too many gettimeofday() syscalls. */
void activeExpireCycleExpire(redisDb *db, expireEntry &e, long long now) {
if (!e.FFat())
{
@ -156,7 +157,7 @@ void activeExpireCycleExpire(redisDb *db, expireEntry &e, long long now) {
switch (val->type)
{
case OBJ_SET:
signalModifiedKey(db,&objKey);
signalModifiedKey(nullptr, db,&objKey);
notifyKeyspaceEvent(NOTIFY_SET,"srem",&objKey,db->id);
break;
}
@ -229,7 +230,7 @@ void expireMemberCore(client *c, robj *key, robj *subkey, long long basetime, lo
}
setExpire(c, c->db, key, subkey, when);
signalModifiedKey(c->db, key);
signalModifiedKey(c, c->db, key);
g_pserver->dirty++;
addReply(c, shared.cone);
}
@ -608,14 +609,14 @@ void expireGenericCommand(client *c, long long basetime, int unit) {
/* Replicate/AOF this as an explicit DEL or UNLINK. */
aux = g_pserver->lazyfree_lazy_expire ? shared.unlink : shared.del;
rewriteClientCommandVector(c,2,aux,key);
signalModifiedKey(c->db,key);
signalModifiedKey(c,c->db,key);
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",key,c->db->id);
addReply(c, shared.cone);
return;
} else {
setExpire(c,c->db,key,nullptr,when);
addReply(c,shared.cone);
signalModifiedKey(c->db,key);
signalModifiedKey(c,c->db,key);
notifyKeyspaceEvent(NOTIFY_GENERIC,"expire",key,c->db->id);
g_pserver->dirty++;
return;

4
src/geo.cpp
View File

@ -657,13 +657,13 @@ void georadiusGeneric(client *c, int flags) {
if (returned_items) {
zsetConvertToZiplistIfNeeded(zobj,maxelelen);
setKey(c->db,storekey,zobj);
setKey(c,c->db,storekey,zobj);
decrRefCount(zobj);
notifyKeyspaceEvent(NOTIFY_ZSET,"georadiusstore",storekey,
c->db->id);
g_pserver->dirty += returned_items;
} else if (dbDelete(c->db,storekey)) {
signalModifiedKey(c->db,storekey);
signalModifiedKey(c,c->db,storekey);
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",storekey,c->db->id);
g_pserver->dirty++;
}

4
src/geohash.c
View File

@ -206,7 +206,11 @@ int geohashDecodeWGS84(const GeoHashBits hash, GeoHashArea *area) {
int geohashDecodeAreaToLongLat(const GeoHashArea *area, double *xy) {
if (!xy) return 0;
xy[0] = (area->longitude.min + area->longitude.max) / 2;
if (xy[0] > GEO_LONG_MAX) xy[0] = GEO_LONG_MAX;
if (xy[0] < GEO_LONG_MIN) xy[0] = GEO_LONG_MIN;
xy[1] = (area->latitude.min + area->latitude.max) / 2;
if (xy[1] > GEO_LAT_MAX) xy[1] = GEO_LAT_MAX;
if (xy[1] < GEO_LAT_MIN) xy[1] = GEO_LAT_MIN;
return 1;
}

213
src/help.h
View File

@ -28,6 +28,56 @@ struct commandHelp {
int group;
char *since;
} commandHelp[] = {
{ "ACL CAT",
"[categoryname]",
"List the ACL categories or the commands inside a category",
9,
"6.0.0" },
{ "ACL DELUSER",
"username [username ...]",
"Remove the specified ACL users and the associated rules",
9,
"6.0.0" },
{ "ACL GENPASS",
"[bits]",
"Generate a pseudorandom secure password to use for ACL users",
9,
"6.0.0" },
{ "ACL LIST",
"-",
"List the current ACL rules in ACL config file format",
9,
"6.0.0" },
{ "ACL LOAD",
"-",
"Reload the ACLs from the configured ACL file",
9,
"6.0.0" },
{ "ACL LOG",
"[count or RESET]",
"List latest events denied because of ACLs in place",
9,
"6.0.0" },
{ "ACL SAVE",
"-",
"Save the current ACL rules in the configured ACL file",
9,
"6.0.0" },
{ "ACL SETUSER",
"rule [rule ...]",
"Modify or create the rules for a specific ACL user",
9,
"6.0.0" },
{ "ACL USERS",
"-",
"List the username of all the configured ACL rules",
9,
"6.0.0" },
{ "ACL WHOAMI",
"-",
"Return the name of the user associated to the current connection",
9,
"6.0.0" },
{ "APPEND",
"key value",
"Append a value to a key",
@ -44,7 +94,7 @@ struct commandHelp {
9,
"1.0.0" },
{ "BGSAVE",
"-",
"[SCHEDULE]",
"Asynchronously save the dataset to disk",
9,
"1.0.0" },
@ -80,7 +130,7 @@ struct commandHelp {
"2.0.0" },
{ "BRPOPLPUSH",
"source destination timeout",
"Pop a value from a list, push it to another list and return it; or block until one is available",
"Pop an element from a list, push it to another list and return it; or block until one is available",
2,
"2.2.0" },
{ "BZPOPMAX",
@ -93,51 +143,71 @@ struct commandHelp {
"Remove and return the member with the lowest score from one or more sorted sets, or block until one is available",
4,
"5.0.0" },
{ "CLIENT CACHING",
"YES|NO",
"Instruct the server about tracking or not keys in the next request",
8,
"6.0.0" },
{ "CLIENT GETNAME",
"-",
"Get the current connection name",
9,
8,
"2.6.9" },
{ "CLIENT GETREDIR",
"-",
"Get tracking notifications redirection client ID if any",
8,
"6.0.0" },
{ "CLIENT ID",
"-",
"Returns the client ID for the current connection",
9,
8,
"5.0.0" },
{ "CLIENT KILL",
"[ip:port] [ID client-id] [TYPE normal|master|slave|pubsub] [ADDR ip:port] [SKIPME yes/no]",
"Kill the connection of a client",
9,
8,
"2.4.0" },
{ "CLIENT LIST",
"-",
"[TYPE normal|master|replica|pubsub]",
"Get the list of client connections",
9,
8,
"2.4.0" },
{ "CLIENT PAUSE",
"timeout",
"Stop processing commands from clients for some time",
9,
8,
"2.9.50" },
{ "CLIENT REPLY",
"ON|OFF|SKIP",
"Instruct the server whether to reply to commands",
9,
8,
"3.2" },
{ "CLIENT SETNAME",
"connection-name",
"Set the current connection name",
9,
8,
"2.6.9" },
{ "CLIENT TRACKING",
"ON|OFF [REDIRECT client-id] [PREFIX prefix] [BCAST] [OPTIN] [OPTOUT] [NOLOOP]",
"Enable or disable server assisted client side caching support",
8,
"6.0.0" },
{ "CLIENT UNBLOCK",
"client-id [TIMEOUT|ERROR]",
"Unblock a client blocked in a blocking command from a different connection",
9,
8,
"5.0.0" },
{ "CLUSTER ADDSLOTS",
"slot [slot ...]",
"Assign new hash slots to receiving node",
12,
"3.0.0" },
{ "CLUSTER BUMPEPOCH",
"-",
"Advance the cluster config epoch",
12,
"3.0.0" },
{ "CLUSTER COUNT-FAILURE-REPORTS",
"node-id",
"Return the number of failure reports active for a given node",
@ -158,6 +228,11 @@ struct commandHelp {
"Forces a replica to perform a manual failover of its master.",
12,
"3.0.0" },
{ "CLUSTER FLUSHSLOTS",
"-",
"Delete a node's own slots information",
12,
"3.0.0" },
{ "CLUSTER FORGET",
"node-id",
"Remove a node from the nodes table",
@ -183,6 +258,11 @@ struct commandHelp {
"Force a node cluster to handshake with another node",
12,
"3.0.0" },
{ "CLUSTER MYID",
"-",
"Return the node id",
12,
"3.0.0" },
{ "CLUSTER NODES",
"-",
"Get Cluster config for the node",
@ -365,7 +445,7 @@ struct commandHelp {
13,
"3.2.0" },
{ "GEODIST",
"key member1 member2 [unit]",
"key member1 member2 [m|km|ft|mi]",
"Returns the distance between two members of a geospatial index",
13,
"3.2.0" },
@ -414,6 +494,11 @@ struct commandHelp {
"Delete one or more hash fields",
5,
"2.0.0" },
{ "HELLO",
"protover [AUTH username password] [SETNAME clientname]",
"switch Redis protocol",
8,
"6.0.0" },
{ "HEXISTS",
"key field",
"Determine if a hash field exists",
@ -465,7 +550,7 @@ struct commandHelp {
5,
"2.8.0" },
{ "HSET",
"key field value",
"key field value [field value ...]",
"Set the string value of a hash field",
5,
"2.0.0" },
@ -514,13 +599,43 @@ struct commandHelp {
"Get the UNIX time stamp of the last successful save to disk",
9,
"1.0.0" },
{ "LATENCY DOCTOR",
"-",
"Return a human readable latency analysis report.",
9,
"2.8.13" },
{ "LATENCY GRAPH",
"event",
"Return a latency graph for the event.",
9,
"2.8.13" },
{ "LATENCY HELP",
"-",
"Show helpful text about the different subcommands.",
9,
"2.8.13" },
{ "LATENCY HISTORY",
"event",
"Return timestamp-latency samples for the event.",
9,
"2.8.13" },
{ "LATENCY LATEST",
"-",
"Return the latest latency samples for all events.",
9,
"2.8.13" },
{ "LATENCY RESET",
"[event]",
"Reset latency data for one or more events.",
9,
"2.8.13" },
{ "LINDEX",
"key index",
"Get an element from a list by its index",
2,
"1.0.0" },
{ "LINSERT",
"key BEFORE|AFTER pivot value",
"key BEFORE|AFTER pivot element",
"Insert an element before or after another element in a list",
2,
"2.2.0" },
@ -529,19 +644,24 @@ struct commandHelp {
"Get the length of a list",
2,
"1.0.0" },
{ "LOLWUT",
"[VERSION version]",
"Display some computer art and the Redis version",
9,
"5.0.0" },
{ "LPOP",
"key",
"Remove and get the first element in a list",
2,
"1.0.0" },
{ "LPUSH",
"key value [value ...]",
"Prepend one or multiple values to a list",
"key element [element ...]",
"Prepend one or multiple elements to a list",
2,
"1.0.0" },
{ "LPUSHX",
"key value",
"Prepend a value to a list, only if the list exists",
"key element [element ...]",
"Prepend an element to a list, only if the list exists",
2,
"2.2.0" },
{ "LRANGE",
@ -550,12 +670,12 @@ struct commandHelp {
2,
"1.0.0" },
{ "LREM",
"key count value",
"key count element",
"Remove elements from a list",
2,
"1.0.0" },
{ "LSET",
"key index value",
"key index element",
"Set the value of an element in a list by its index",
2,
"1.0.0" },
@ -600,10 +720,25 @@ struct commandHelp {
1,
"1.0.0" },
{ "MIGRATE",
"host port key|"" destination-db timeout [COPY] [REPLACE] [KEYS key]",
"host port key|"" destination-db timeout [COPY] [REPLACE] [AUTH password] [KEYS key]",
"Atomically transfer a key from a Redis instance to another one.",
0,
"2.6.0" },
{ "MODULE LIST",
"-",
"List all modules loaded by the server",
9,
"4.0.0" },
{ "MODULE LOAD",
"path [arg]",
"Load a module",
9,
"4.0.0" },
{ "MODULE UNLOAD",
"name",
"Unload a module",
9,
"4.0.0" },
{ "MONITOR",
"-",
"Listen for all requests received by the server in real time",
@ -679,6 +814,11 @@ struct commandHelp {
"Listen for messages published to channels matching the given patterns",
6,
"2.0.0" },
{ "PSYNC",
"replicationid offset",
"Internal command used for replication",
9,
"2.8.0" },
{ "PTTL",
"key [subkey]",
"Get the time to live for a key or subkey in milliseconds",
@ -735,7 +875,7 @@ struct commandHelp {
9,
"5.0.0" },
{ "RESTORE",
"key ttl serialized-value [REPLACE]",
"key ttl serialized-value [REPLACE] [ABSTTL] [IDLETIME seconds] [FREQ frequency]",
"Create a key using the provided serialized value, previously obtained using DUMP.",
0,
"2.6.0" },
@ -755,13 +895,13 @@ struct commandHelp {
2,
"1.2.0" },
{ "RPUSH",
"key value [value ...]",
"Append one or multiple values to a list",
"key element [element ...]",
"Append one or multiple elements to a list",
2,
"1.0.0" },
{ "RPUSHX",
"key value",
"Append a value to a list, only if the list exists",
"key element [element ...]",
"Append an element to a list, only if the list exists",
2,
"2.2.0" },
{ "SADD",
@ -775,7 +915,7 @@ struct commandHelp {
9,
"1.0.0" },
{ "SCAN",
"cursor [MATCH pattern] [COUNT count]",
"cursor [MATCH pattern] [COUNT count] [TYPE type]",
"Incrementally iterate the keys space",
0,
"2.8.0" },
@ -825,7 +965,7 @@ struct commandHelp {
8,
"1.0.0" },
{ "SET",
"key value [expiration EX seconds|PX milliseconds] [NX|XX]",
"key value [EX seconds|PX milliseconds] [NX|XX] [KEEPTTL]",
"Set the string value of a key",
1,
"1.0.0" },
@ -914,6 +1054,11 @@ struct commandHelp {
"Incrementally iterate Set elements",
3,
"2.8.0" },
{ "STRALGO",
"LCS algo-specific-argument [algo-specific-argument ...]",
"Run algorithms (currently LCS) against strings",
1,
"6.0.0" },
{ "STRLEN",
"key",
"Get the length of the value stored in a key",
@ -935,9 +1080,9 @@ struct commandHelp {
3,
"1.0.0" },
{ "SWAPDB",
"index index",
"index1 index2",
"Swaps two Redis databases",
8,
9,
"4.0.0" },
{ "SYNC",
"-",
@ -995,7 +1140,7 @@ struct commandHelp {
14,
"5.0.0" },
{ "XADD",
"key ID field string [field string ...]",
"key ID field value [field value ...]",
"Appends a new entry to a stream",
14,
"5.0.0" },
@ -1010,7 +1155,7 @@ struct commandHelp {
14,
"5.0.0" },
{ "XGROUP",
"[CREATE key groupname id-or-$] [SETID key id-or-$] [DESTROY key groupname] [DELCONSUMER key groupname consumername]",
"[CREATE key groupname id-or-$] [SETID key groupname id-or-$] [DESTROY key groupname] [DELCONSUMER key groupname consumername]",
"Create, destroy, and manage consumer groups.",
14,
"5.0.0" },
@ -1035,12 +1180,12 @@ struct commandHelp {
14,
"5.0.0" },
{ "XREAD",
"[COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...]",
"[COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]",
"Return never seen elements in multiple streams, with IDs greater than the ones reported by the caller for each stream. Can block.",
14,
"5.0.0" },
{ "XREADGROUP",
"GROUP group consumer [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...]",
"GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] ID [ID ...]",
"Return new entries from a stream using a consumer group, or access the history of the pending entries for a given consumer. Can block.",
14,
"5.0.0" },

6
src/hyperloglog.cpp
View File

@ -1219,7 +1219,7 @@ void pfaddCommand(client *c) {
}
hdr = (hllhdr*)ptrFromObj(o);
if (updated) {
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STRING,"pfadd",c->argv[1],c->db->id);
g_pserver->dirty++;
HLL_INVALIDATE_CACHE(hdr);
@ -1310,7 +1310,7 @@ void pfcountCommand(client *c) {
* data structure is not modified, since the cached value
* may be modified and given that the HLL is a Redis string
* we need to propagate the change. */
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
g_pserver->dirty++;
}
addReplyLongLong(c,card);
@ -1383,7 +1383,7 @@ void pfmergeCommand(client *c) {
last hllSparseSet() call. */
HLL_INVALIDATE_CACHE(hdr);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
/* We generate a PFADD event for PFMERGE for semantical simplicity
* since in theory this is a mass-add of elements. */
notifyKeyspaceEvent(NOTIFY_STRING,"pfadd",c->argv[1],c->db->id);

10
src/module.cpp
View File

@ -910,7 +910,7 @@ void RM_SetModuleOptions(RedisModuleCtx *ctx, int options) {
/* Signals that the key is modified from user's perspective (i.e. invalidate WATCH
* and client side caching). */
int RM_SignalModifiedKey(RedisModuleCtx *ctx, RedisModuleString *keyname) {
signalModifiedKey(ctx->client->db,keyname);
signalModifiedKey(ctx->client,ctx->client->db,keyname);
return REDISMODULE_OK;
}
@ -2086,7 +2086,7 @@ void *RM_OpenKey(RedisModuleCtx *ctx, robj *keyname, int mode) {
static void moduleCloseKey(RedisModuleKey *key) {
int signal = SHOULD_SIGNAL_MODIFIED_KEYS(key->ctx);
if ((key->mode & REDISMODULE_WRITE) && signal)
signalModifiedKey(key->db,key->key);
signalModifiedKey(key->ctx->client,key->db,key->key);
/* TODO: if (key->iter) RM_KeyIteratorStop(kp); */
RM_ZsetRangeStop(key);
decrRefCount(key->key);
@ -2231,7 +2231,7 @@ RedisModuleString *RM_RandomKey(RedisModuleCtx *ctx) {
int RM_StringSet(RedisModuleKey *key, RedisModuleString *str) {
if (!(key->mode & REDISMODULE_WRITE) || key->iter) return REDISMODULE_ERR;
RM_DeleteKey(key);
genericSetKey(key->db,key->key,str,0,0);
genericSetKey(key->ctx->client,key->db,key->key,str,0,0);
key->value = str;
return REDISMODULE_OK;
}
@ -2311,7 +2311,7 @@ int RM_StringTruncate(RedisModuleKey *key, size_t newlen) {
if (key->value == NULL) {
/* Empty key: create it with the new size. */
robj *o = createObject(OBJ_STRING,sdsnewlen(NULL, newlen));
genericSetKey(key->db,key->key,o,0,0);
genericSetKey(key->ctx->client,key->db,key->key,o,0,0);
key->value = o;
decrRefCount(o);
} else {
@ -3701,7 +3701,7 @@ int RM_ModuleTypeSetValue(RedisModuleKey *key, moduleType *mt, void *value) {
if (!(key->mode & REDISMODULE_WRITE) || key->iter) return REDISMODULE_ERR;
RM_DeleteKey(key);
robj *o = createModuleObject(mt,value);
genericSetKey(key->db,key->key,o,0,0);
genericSetKey(key->ctx->client,key->db,key->key,o,0,0);
decrRefCount(o);
key->value = o;
return REDISMODULE_OK;

165
src/networking.cpp
View File

@ -566,6 +566,36 @@ void addReplyStatusFormat(client *c, const char *fmt, ...) {
sdsfree(s);
}
/* Sometimes we are forced to create a new reply node, and we can't append to
* the previous one, when that happens, we wanna try to trim the unused space
* at the end of the last reply node which we won't use anymore. */
void trimReplyUnusedTailSpace(client *c) {
listNode *ln = listLast(c->reply);
clientReplyBlock *tail = ln? (clientReplyBlock*)listNodeValue(ln): nullptr;
/* Note that 'tail' may be NULL even if we have a tail node, becuase when
* addDeferredMultiBulkLength() is used */
if (!tail) return;
/* We only try to trim the space is relatively high (more than a 1/4 of the
* allocation), otherwise there's a high chance realloc will NOP.
* Also, to avoid large memmove which happens as part of realloc, we only do
* that if the used part is small. */
if (tail->size - tail->used > tail->size / 4 &&
tail->used < PROTO_REPLY_CHUNK_BYTES)
{
size_t old_size = tail->size;
tail = (clientReplyBlock*)zrealloc(tail, tail->used + sizeof(clientReplyBlock));
/* If realloc was a NOP, we got the same value which has internal frag */
if (tail == listNodeValue(ln)) return;
/* take over the allocation's internal fragmentation (at least for
* memory usage tracking) */
tail->size = zmalloc_usable(tail) - sizeof(clientReplyBlock);
c->reply_bytes += tail->size - old_size;
listNodeValue(ln) = tail;
}
}
/* Adds an empty object to the reply list that will contain the multi bulk
* length, which is not known when this function is called. */
void *addReplyDeferredLen(client *c) {
@ -573,6 +603,7 @@ void *addReplyDeferredLen(client *c) {
* ready to be sent, since we are sure that before returning to the
* event loop setDeferredAggregateLen() will be called. */
if (prepareClientToWrite(c, false) != C_OK) return NULL;
trimReplyUnusedTailSpace(c);
listAddNodeTail(c->reply,NULL); /* NULL is our placeholder. */
return listLast(c->reply);
}
@ -2231,12 +2262,60 @@ int processMultibulkBuffer(client *c) {
return C_ERR;
}
/* Perform necessary tasks after a command was executed:
*
* 1. The client is reset unless there are reasons to avoid doing it.
* 2. In the case of master clients, the replication offset is updated.
* 3. Propagate commands we got from our master to replicas down the line. */
void commandProcessed(client *c) {
int cmd_is_ping = c->cmd && c->cmd->proc == pingCommand;
long long prev_offset = c->reploff;
if (c->flags & CLIENT_MASTER && !(c->flags & CLIENT_MULTI)) {
/* Update the applied replication offset of our master. */
c->reploff = c->read_reploff - sdslen(c->querybuf) + c->qb_pos;
}
/* Don't reset the client structure for clients blocked in a
* module blocking command, so that the reply callback will
* still be able to access the client argv and argc field.
* The client will be reset in unblockClientFromModule(). */
if (!(c->flags & CLIENT_BLOCKED) ||
c->btype != BLOCKED_MODULE)
{
resetClient(c);
}
/* If the client is a master we need to compute the difference
* between the applied offset before and after processing the buffer,
* to understand how much of the replication stream was actually
* applied to the master state: this quantity, and its corresponding
* part of the replication stream, will be propagated to the
* sub-replicas and to the replication backlog. */
if (c->flags & CLIENT_MASTER) {
long long applied = c->reploff - prev_offset;
long long prev_master_repl_meaningful_offset = g_pserver->master_repl_meaningful_offset;
if (applied) {
if (!g_pserver->fActiveReplica)
{
AeLocker ae;
ae.arm(c);
replicationFeedSlavesFromMasterStream(g_pserver->slaves,
c->pending_querybuf, applied);
}
sdsrange(c->pending_querybuf,applied,-1);
}
/* The g_pserver->master_repl_meaningful_offset variable represents
* the offset of the replication stream without the pending PINGs. */
if (cmd_is_ping)
g_pserver->master_repl_meaningful_offset = prev_master_repl_meaningful_offset;
}
}
/* This function calls processCommand(), but also performs a few sub tasks
* that are useful in that context:
* for the client that are useful in that context:
*
* 1. It sets the current client to the client 'c'.
* 2. In the case of master clients, the replication offset is updated.
* 3. The client is reset unless there are reasons to avoid doing it.
* 2. calls commandProcessed() if the command was handled.
*
* The function returns C_ERR in case the client was freed as a side effect
* of processing the command, otherwise C_OK is returned. */
@ -2244,20 +2323,7 @@ int processCommandAndResetClient(client *c, int flags) {
int deadclient = 0;
serverTL->current_client = c;
if (processCommand(c, flags) == C_OK) {
if (c->flags & CLIENT_MASTER && !(c->flags & CLIENT_MULTI)) {
/* Update the applied replication offset of our master. */
c->reploff = c->read_reploff - sdslen(c->querybuf) + c->qb_pos;
}
/* Don't reset the client structure for clients blocked in a
* module blocking command, so that the reply callback will
* still be able to access the client argv and argc field.
* The client will be reset in unblockClientFromModule(). */
if (!(c->flags & CLIENT_BLOCKED) ||
c->btype != BLOCKED_MODULE)
{
resetClient(c);
}
commandProcessed(c);
}
if (serverTL->current_client == NULL) deadclient = 1;
serverTL->current_client = NULL;
@ -2333,36 +2399,6 @@ void processInputBuffer(client *c, int callFlags) {
}
}
/* This is a wrapper for processInputBuffer that also cares about handling
* the replication forwarding to the sub-replicas, in case the client 'c'
* is flagged as master. Usually you want to call this instead of the
* raw processInputBuffer(). */
void processInputBufferAndReplicate(client *c) {
if (!(c->flags & CLIENT_MASTER)) {
processInputBuffer(c, CMD_CALL_FULL);
} else {
/* If the client is a master we need to compute the difference
* between the applied offset before and after processing the buffer,
* to understand how much of the replication stream was actually
* applied to the master state: this quantity, and its corresponding
* part of the replication stream, will be propagated to the
* sub-replicas and to the replication backlog. */
size_t prev_offset = c->reploff;
processInputBuffer(c, CMD_CALL_FULL);
size_t applied = c->reploff - prev_offset;
if (applied) {
if (!g_pserver->fActiveReplica)
{
AeLocker ae;
ae.arm(c);
replicationFeedSlavesFromMasterStream(g_pserver->slaves,
c->pending_querybuf, applied);
}
sdsrange(c->pending_querybuf,applied,-1);
}
}
}
void readQueryFromClient(connection *conn) {
client *c = (client*)connGetPrivateData(conn);
serverAssert(conn == c->conn);
@ -2437,13 +2473,9 @@ void readQueryFromClient(connection *conn) {
return;
}
/* Time to process the buffer. If the client is a master we need to
* compute the difference between the applied offset before and after
* processing the buffer, to understand how much of the replication stream
* was actually applied to the master state: this quantity, and its
* corresponding part of the replication stream, will be propagated to
* the sub-slaves and to the replication backlog. */
processInputBufferAndReplicate(c);
/* There is more data in the client input buffer, continue parsing it
* in case to check if there is a full command to execute. */
processInputBuffer(c, CMD_CALL_FULL);
if (listLength(serverTL->clients_pending_asyncwrite))
{
aelock.arm(c);
@ -2628,6 +2660,7 @@ void clientCommand(client *c) {
"KILL <option> <value> [option value ...] -- Kill connections. Options are:",
" ADDR <ip:port> -- Kill connection made from <ip:port>",
" TYPE (normal|master|replica|pubsub) -- Kill connections by type.",
" USER <username> -- Kill connections authenticated with such user.",
" SKIPME (yes|no) -- Skip killing current connection (default: yes).",
"LIST [options ...] -- Return information about client connections. Options:",
" TYPE (normal|master|replica|pubsub) -- Return clients of specified type.",
@ -2678,6 +2711,7 @@ NULL
/* CLIENT KILL <ip:port>
* CLIENT KILL <option> [value] ... <option> [value] */
char *addr = NULL;
user *user = NULL;
int type = -1;
uint64_t id = 0;
int skipme = 1;
@ -2707,10 +2741,18 @@ NULL
(char*) ptrFromObj(c->argv[i+1]));
return;
}
} else if (!strcasecmp((const char*)ptrFromObj(c->argv[i]),"addr") && moreargs) {
addr = (char*)ptrFromObj(c->argv[i+1]);
} else if (!strcasecmp((const char*)ptrFromObj(c->argv[i]),"skipme") && moreargs) {
if (!strcasecmp((const char*)ptrFromObj(c->argv[i+1]),"yes")) {
} else if (!strcasecmp(szFromObj(c->argv[i]),"addr") && moreargs) {
addr = szFromObj(c->argv[i+1]);
} else if (!strcasecmp(szFromObj(c->argv[i]),"user") && moreargs) {
user = ACLGetUserByName(szFromObj(c->argv[i+1]),
sdslen(szFromObj(c->argv[i+1])));
if (user == NULL) {
addReplyErrorFormat(c,"No such user '%s'",
szFromObj(c->argv[i+1]));
return;
}
} else if (!strcasecmp(szFromObj(c->argv[i]),"skipme") && moreargs) {
if (!strcasecmp(szFromObj(c->argv[i+1]),"yes")) {
skipme = 1;
} else if (!strcasecmp((const char*)ptrFromObj(c->argv[i+1]),"no")) {
skipme = 0;
@ -2736,6 +2778,7 @@ NULL
if (addr && strcmp(getClientPeerId(client),addr) != 0) continue;
if (type != -1 && getClientType(client) != type) continue;
if (id != 0 && client->id != id) continue;
if (user && client->puser != user) continue;
if (c == client && skipme) continue;
/* Kill it. */
@ -2864,6 +2907,8 @@ NULL
options |= CLIENT_TRACKING_OPTIN;
} else if (!strcasecmp(szFromObj(c->argv[j]),"optout")) {
options |= CLIENT_TRACKING_OPTOUT;
} else if (!strcasecmp(szFromObj(c->argv[j]),"noloop")) {
options |= CLIENT_TRACKING_NOLOOP;
} else if (!strcasecmp(szFromObj(c->argv[j]),"prefix") && moreargs) {
j++;
prefix = (robj**)zrealloc(prefix,sizeof(robj*)*(numprefix+1), MALLOC_LOCAL);
@ -3019,7 +3064,7 @@ void helloCommand(client *c) {
/* Let's switch to the specified RESP mode. */
c->resp = ver;
addReplyMapLen(c,7);
addReplyMapLen(c,6 + !g_pserver->sentinel_mode);
addReplyBulkCString(c,"server");
addReplyBulkCString(c,"redis");
@ -3035,7 +3080,7 @@ void helloCommand(client *c) {
addReplyBulkCString(c,"mode");
if (g_pserver->sentinel_mode) addReplyBulkCString(c,"sentinel");
if (g_pserver->cluster_enabled) addReplyBulkCString(c,"cluster");
else if (g_pserver->cluster_enabled) addReplyBulkCString(c,"cluster");
else addReplyBulkCString(c,"standalone");
if (!g_pserver->sentinel_mode) {

3
src/rdb.cpp
View File

@ -2457,7 +2457,8 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi) {
if (cksum == 0) {
serverLog(LL_WARNING,"RDB file was saved with checksum disabled: no check performed.");
} else if (cksum != expected) {
serverLog(LL_WARNING,"Wrong RDB checksum. Aborting now.");
serverLog(LL_WARNING,"Wrong RDB checksum expected: (%llx) but "
"got (%llx). Aborting now.",expected,cksum);
rdbExitReportCorruptRDB("RDB CRC error");
}
}

411
src/redis-cli-cpphelper.cpp
View File

@ -28,6 +28,10 @@ extern "C" {
#include "redis-cli.h"
static dict *clusterManagerGetLinkStatus(void);
static clusterManagerNode *clusterManagerNodeMasterRandom();
/* Used by clusterManagerFixSlotsCoverage */
struct dict *clusterManagerUncoveredSlots = NULL;
/* The Cluster Manager global structure */
struct clusterManager cluster_manager;
@ -60,18 +64,6 @@ extern "C" void dictListDestructor(void *privdata, void *val)
listRelease((list*)val);
}
/* Used by clusterManagerFixSlotsCoverage */
dict *clusterManagerUncoveredSlots = NULL;
/* Info about a cluster internal link. */
typedef struct clusterManagerLink {
sds node_name;
sds node_addr;
int connected;
int handshaking;
} clusterManagerLink;
static dictType clusterManagerDictType = {
dictSdsHash, /* hash function */
NULL, /* key dup */
@ -116,6 +108,201 @@ extern "C" void freeClusterManager(void) {
dictRelease(clusterManagerUncoveredSlots);
}
static int clusterManagerFixSlotsCoverage(char *all_slots) {
dictIterator *iter = nullptr;
int force_fix = config.cluster_manager_command.flags &
CLUSTER_MANAGER_CMD_FLAG_FIX_WITH_UNREACHABLE_MASTERS;
if (cluster_manager.unreachable_masters > 0 && !force_fix) {
clusterManagerLogWarn("*** Fixing slots coverage with %d unreachable masters is dangerous: redis-cli will assume that slots about masters that are not reachable are not covered, and will try to reassign them to the reachable nodes. This can cause data loss and is rarely what you want to do. If you really want to proceed use the --cluster-fix-with-unreachable-masters option.\n", cluster_manager.unreachable_masters);
exit(1);
}
int i, fixed = 0;
list *none = NULL, *single = NULL, *multi = NULL;
clusterManagerLogInfo(">>> Fixing slots coverage...\n");
for (i = 0; i < CLUSTER_MANAGER_SLOTS; i++) {
int covered = all_slots[i];
if (!covered) {
sds slot = sdsfromlonglong((long long) i);
list *slot_nodes = listCreate();
sds slot_nodes_str = sdsempty();
listIter li;
listNode *ln;
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *n = (clusterManagerNode*) ln->value;
if (n->flags & CLUSTER_MANAGER_FLAG_SLAVE || n->replicate)
continue;
redisReply *reply = (redisReply*)CLUSTER_MANAGER_COMMAND(n,
"CLUSTER GETKEYSINSLOT %d %d", i, 1);
if (!clusterManagerCheckRedisReply(n, reply, NULL)) {
fixed = -1;
if (reply) freeReplyObject(reply);
goto cleanup;
}
assert(reply->type == REDIS_REPLY_ARRAY);
if (reply->elements > 0) {
listAddNodeTail(slot_nodes, n);
if (listLength(slot_nodes) > 1)
slot_nodes_str = sdscat(slot_nodes_str, ", ");
slot_nodes_str = sdscatfmt(slot_nodes_str,
"%s:%u", n->ip, n->port);
}
freeReplyObject(reply);
}
sdsfree(slot_nodes_str);
dictAdd(clusterManagerUncoveredSlots, slot, slot_nodes);
}
}
/* For every slot, take action depending on the actual condition:
* 1) No node has keys for this slot.
* 2) A single node has keys for this slot.
* 3) Multiple nodes have keys for this slot. */
none = listCreate();
single = listCreate();
multi = listCreate();
iter = dictGetIterator(clusterManagerUncoveredSlots);
dictEntry *entry;
while ((entry = dictNext(iter)) != NULL) {
sds slot = (sds) dictGetKey(entry);
list *nodes = (list *) dictGetVal(entry);
switch (listLength(nodes)){
case 0: listAddNodeTail(none, slot); break;
case 1: listAddNodeTail(single, slot); break;
default: listAddNodeTail(multi, slot); break;
}
}
dictReleaseIterator(iter);
/* Handle case "1": keys in no node. */
if (listLength(none) > 0) {
printf("The following uncovered slots have no keys "
"across the cluster:\n");
clusterManagerPrintSlotsList(none);
if (confirmWithYes("Fix these slots by covering with a random node?")){
listIter li;
listNode *ln;
listRewind(none, &li);
while ((ln = listNext(&li)) != NULL) {
sds slot = (sds)ln->value;
int s = atoi(slot);
clusterManagerNode *n = clusterManagerNodeMasterRandom();
clusterManagerLogInfo(">>> Covering slot %s with %s:%d\n",
slot, n->ip, n->port);
if (!clusterManagerSetSlotOwner(n, s, 0)) {
fixed = -1;
goto cleanup;
}
/* Since CLUSTER ADDSLOTS succeeded, we also update the slot
* info into the node struct, in order to keep it synced */
n->slots[s] = 1;
fixed++;
}
}
}
/* Handle case "2": keys only in one node. */
if (listLength(single) > 0) {
printf("The following uncovered slots have keys in just one node:\n");
clusterManagerPrintSlotsList(single);
if (confirmWithYes("Fix these slots by covering with those nodes?")){
listIter li;
listNode *ln;
listRewind(single, &li);
while ((ln = listNext(&li)) != NULL) {
sds slot = (sds)ln->value;
int s = atoi(slot);
dictEntry *entry = dictFind(clusterManagerUncoveredSlots, slot);
assert(entry != NULL);
list *nodes = (list *) dictGetVal(entry);
listNode *fn = listFirst(nodes);
assert(fn != NULL);
clusterManagerNode *n = (clusterManagerNode*) fn->value;
clusterManagerLogInfo(">>> Covering slot %s with %s:%d\n",
slot, n->ip, n->port);
if (!clusterManagerSetSlotOwner(n, s, 0)) {
fixed = -1;
goto cleanup;
}
/* Since CLUSTER ADDSLOTS succeeded, we also update the slot
* info into the node struct, in order to keep it synced */
n->slots[atoi(slot)] = 1;
fixed++;
}
}
}
/* Handle case "3": keys in multiple nodes. */
if (listLength(multi) > 0) {
printf("The following uncovered slots have keys in multiple nodes:\n");
clusterManagerPrintSlotsList(multi);
if (confirmWithYes("Fix these slots by moving keys "
"into a single node?")) {
listIter li;
listNode *ln;
listRewind(multi, &li);
while ((ln = listNext(&li)) != NULL) {
sds slot = (sds)ln->value;
dictEntry *entry = dictFind(clusterManagerUncoveredSlots, slot);
assert(entry != NULL);
list *nodes = (list *) dictGetVal(entry);
int s = atoi(slot);
clusterManagerNode *target =
clusterManagerGetNodeWithMostKeysInSlot(nodes, s, NULL);
if (target == NULL) {
fixed = -1;
goto cleanup;
}
clusterManagerLogInfo(">>> Covering slot %s moving keys "
"to %s:%d\n", slot,
target->ip, target->port);
if (!clusterManagerSetSlotOwner(target, s, 1)) {
fixed = -1;
goto cleanup;
}
/* Since CLUSTER ADDSLOTS succeeded, we also update the slot
* info into the node struct, in order to keep it synced */
target->slots[atoi(slot)] = 1;
listIter nli;
listNode *nln;
listRewind(nodes, &nli);
while ((nln = listNext(&nli)) != NULL) {
clusterManagerNode *src = (clusterManagerNode*) nln->value;
if (src == target) continue;
/* Assign the slot to target node in the source node. */
if (!clusterManagerSetSlot(src, target, s, "NODE", NULL))
fixed = -1;
if (fixed < 0) goto cleanup;
/* Set the source node in 'importing' state
* (even if we will actually migrate keys away)
* in order to avoid receiving redirections
* for MIGRATE. */
if (!clusterManagerSetSlot(src, target, s,
"IMPORTING", NULL)) fixed = -1;
if (fixed < 0) goto cleanup;
int opts = CLUSTER_MANAGER_OPT_VERBOSE |
CLUSTER_MANAGER_OPT_COLD;
if (!clusterManagerMoveSlot(src, target, s, opts, NULL)) {
fixed = -1;
goto cleanup;
}
if (!clusterManagerClearSlotStatus(src, s))
fixed = -1;
if (fixed < 0) goto cleanup;
}
fixed++;
}
}
}
cleanup:
if (none) listRelease(none);
if (single) listRelease(single);
if (multi) listRelease(multi);
return fixed;
}
/* Return the anti-affinity score, which is a measure of the amount of
* violations of anti-affinity in the current cluster layout, that is, how
* badly the masters and slaves are distributed in the different IP
@ -362,7 +549,7 @@ static clusterManagerNode *clusterManagerNodeMasterRandom() {
listNode *ln;
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *n = (clusterManagerNode*)ln->value;
clusterManagerNode *n = (clusterManagerNode*) ln->value;
if (n->flags & CLUSTER_MANAGER_FLAG_SLAVE) continue;
master_count++;
}
@ -371,7 +558,7 @@ static clusterManagerNode *clusterManagerNodeMasterRandom() {
idx = rand() % master_count;
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *n = (clusterManagerNode*)ln->value;
clusterManagerNode *n = (clusterManagerNode*) ln->value;
if (n->flags & CLUSTER_MANAGER_FLAG_SLAVE) continue;
if (!idx--) {
return n;
@ -381,202 +568,6 @@ static clusterManagerNode *clusterManagerNodeMasterRandom() {
return NULL;
}
static int clusterManagerFixSlotsCoverage(char *all_slots) {
int i, fixed = 0;
dictIterator *iter = nullptr;
dictEntry *entry = nullptr;
list *none = NULL, *single = NULL, *multi = NULL;
clusterManagerLogInfo(">>> Fixing slots coverage...\n");
printf("List of not covered slots: \n");
int uncovered_count = 0;
sds log = sdsempty();
for (i = 0; i < CLUSTER_MANAGER_SLOTS; i++) {
int covered = all_slots[i];
if (!covered) {
sds key = sdsfromlonglong((long long) i);
if (uncovered_count++ > 0) printf(",");
printf("%s", (char *) key);
list *slot_nodes = listCreate();
sds slot_nodes_str = sdsempty();
listIter li;
listNode *ln;
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *n = (clusterManagerNode*)ln->value;
if (n->flags & CLUSTER_MANAGER_FLAG_SLAVE || n->replicate)
continue;
redisReply *reply = (redisReply*)CLUSTER_MANAGER_COMMAND(n,
"CLUSTER GETKEYSINSLOT %d %d", i, 1);
if (!clusterManagerCheckRedisReply(n, reply, NULL)) {
fixed = -1;
if (reply) freeReplyObject(reply);
goto cleanup;
}
assert(reply->type == REDIS_REPLY_ARRAY);
if (reply->elements > 0) {
listAddNodeTail(slot_nodes, n);
if (listLength(slot_nodes) > 1)
slot_nodes_str = sdscat(slot_nodes_str, ", ");
slot_nodes_str = sdscatfmt(slot_nodes_str,
"%s:%u", n->ip, n->port);
}
freeReplyObject(reply);
}
log = sdscatfmt(log, "\nSlot %S has keys in %u nodes: %S",
key, listLength(slot_nodes), slot_nodes_str);
sdsfree(slot_nodes_str);
dictAdd(clusterManagerUncoveredSlots, key, slot_nodes);
}
}
printf("\n%s\n", log);
/* For every slot, take action depending on the actual condition:
* 1) No node has keys for this slot.
* 2) A single node has keys for this slot.
* 3) Multiple nodes have keys for this slot. */
none = listCreate();
single = listCreate();
multi = listCreate();
iter = dictGetIterator(clusterManagerUncoveredSlots);
while ((entry = dictNext(iter)) != NULL) {
sds slot = (sds) dictGetKey(entry);
list *nodes = (list *) dictGetVal(entry);
switch (listLength(nodes)){
case 0: listAddNodeTail(none, slot); break;
case 1: listAddNodeTail(single, slot); break;
default: listAddNodeTail(multi, slot); break;
}
}
dictReleaseIterator(iter);
/* Handle case "1": keys in no node. */
if (listLength(none) > 0) {
printf("The following uncovered slots have no keys "
"across the cluster:\n");
clusterManagerPrintSlotsList(none);
if (confirmWithYes("Fix these slots by covering with a random node?")){
listIter li;
listNode *ln;
listRewind(none, &li);
while ((ln = listNext(&li)) != NULL) {
sds slot = (sds)ln->value;
int s = atoi(slot);
clusterManagerNode *n = clusterManagerNodeMasterRandom();
clusterManagerLogInfo(">>> Covering slot %s with %s:%d\n",
slot, n->ip, n->port);
if (!clusterManagerSetSlotOwner(n, s, 0)) {
fixed = -1;
goto cleanup;
}
/* Since CLUSTER ADDSLOTS succeeded, we also update the slot
* info into the node struct, in order to keep it synced */
n->slots[s] = 1;
fixed++;
}
}
}
/* Handle case "2": keys only in one node. */
if (listLength(single) > 0) {
printf("The following uncovered slots have keys in just one node:\n");
clusterManagerPrintSlotsList(single);
if (confirmWithYes("Fix these slots by covering with those nodes?")){
listIter li;
listNode *ln;
listRewind(single, &li);
while ((ln = listNext(&li)) != NULL) {
sds slot = (sds)ln->value;
int s = atoi(slot);
dictEntry *entry = dictFind(clusterManagerUncoveredSlots, slot);
assert(entry != NULL);
list *nodes = (list *) dictGetVal(entry);
listNode *fn = listFirst(nodes);
assert(fn != NULL);
clusterManagerNode *n = (clusterManagerNode*)fn->value;
clusterManagerLogInfo(">>> Covering slot %s with %s:%d\n",
slot, n->ip, n->port);
if (!clusterManagerSetSlotOwner(n, s, 0)) {
fixed = -1;
goto cleanup;
}
/* Since CLUSTER ADDSLOTS succeeded, we also update the slot
* info into the node struct, in order to keep it synced */
n->slots[atoi(slot)] = 1;
fixed++;
}
}
}
/* Handle case "3": keys in multiple nodes. */
if (listLength(multi) > 0) {
printf("The following uncovered slots have keys in multiple nodes:\n");
clusterManagerPrintSlotsList(multi);
if (confirmWithYes("Fix these slots by moving keys "
"into a single node?")) {
listIter li;
listNode *ln;
listRewind(multi, &li);
while ((ln = listNext(&li)) != NULL) {
sds slot = (sds)ln->value;
dictEntry *entry = dictFind(clusterManagerUncoveredSlots, slot);
assert(entry != NULL);
list *nodes = (list *) dictGetVal(entry);
int s = atoi(slot);
clusterManagerNode *target =
clusterManagerGetNodeWithMostKeysInSlot(nodes, s, NULL);
if (target == NULL) {
fixed = -1;
goto cleanup;
}
clusterManagerLogInfo(">>> Covering slot %s moving keys "
"to %s:%d\n", slot,
target->ip, target->port);
if (!clusterManagerSetSlotOwner(target, s, 1)) {
fixed = -1;
goto cleanup;
}
/* Since CLUSTER ADDSLOTS succeeded, we also update the slot
* info into the node struct, in order to keep it synced */
target->slots[atoi(slot)] = 1;
listIter nli;
listNode *nln;
listRewind(nodes, &nli);
while ((nln = listNext(&nli)) != NULL) {
clusterManagerNode *src = (clusterManagerNode*)nln->value;
if (src == target) continue;
/* Assign the slot to target node in the source node. */
if (!clusterManagerSetSlot(src, target, s, "NODE", NULL))
fixed = -1;
if (fixed < 0) goto cleanup;
/* Set the source node in 'importing' state
* (even if we will actually migrate keys away)
* in order to avoid receiving redirections
* for MIGRATE. */
if (!clusterManagerSetSlot(src, target, s,
"IMPORTING", NULL)) fixed = -1;
if (fixed < 0) goto cleanup;
int opts = CLUSTER_MANAGER_OPT_VERBOSE |
CLUSTER_MANAGER_OPT_COLD;
if (!clusterManagerMoveSlot(src, target, s, opts, NULL)) {
fixed = -1;
goto cleanup;
}
if (!clusterManagerClearSlotStatus(src, s))
fixed = -1;
if (fixed < 0) goto cleanup;
}
fixed++;
}
}
}
cleanup:
sdsfree(log);
if (none) listRelease(none);
if (single) listRelease(single);
if (multi) listRelease(multi);
return fixed;
}
extern "C" int clusterManagerCheckCluster(int quiet) {
listNode *ln = listFirst(cluster_manager.nodes);
if (!ln) return 0;

64
src/redis-cli.c
View File

@ -509,7 +509,8 @@ static char *hintsCallback(const char *buf, int *color, int *bold) {
for (i = 0; i < helpEntriesLen; i++) {
if (!(helpEntries[i].type & CLI_HELP_COMMAND)) continue;
if (strcasecmp(argv[0],helpEntries[i].full) == 0)
if (strcasecmp(argv[0],helpEntries[i].full) == 0 ||
strcasecmp(buf,helpEntries[i].full) == 0)
{
*color = 90;
*bold = 0;
@ -1410,6 +1411,9 @@ static int parseOptions(int argc, char **argv) {
} else if (!strcmp(argv[i],"--cluster-search-multiple-owners")) {
config.cluster_manager_command.flags |=
CLUSTER_MANAGER_CMD_FLAG_CHECK_OWNERS;
} else if (!strcmp(argv[i],"--cluster-fix-with-unreachable-masters")) {
config.cluster_manager_command.flags |=
CLUSTER_MANAGER_CMD_FLAG_FIX_WITH_UNREACHABLE_MASTERS;
#ifdef USE_OPENSSL
} else if (!strcmp(argv[i],"--tls")) {
config.tls = 1;
@ -2021,7 +2025,7 @@ clusterManagerCommandDef clusterManagerCommands[] = {
"search-multiple-owners"},
{"info", clusterManagerCommandInfo, -1, "host:port", NULL},
{"fix", clusterManagerCommandFix, -1, "host:port",
"search-multiple-owners"},
"search-multiple-owners,fix-with-unreachable-masters"},
{"reshard", clusterManagerCommandReshard, -1, "host:port",
"from <arg>,to <arg>,slots <arg>,yes,timeout <arg>,pipeline <arg>,"
"replace"},
@ -3571,7 +3575,9 @@ static int clusterManagerLoadInfoFromNode(clusterManagerNode *node, int opts) {
if (friend->flags & (CLUSTER_MANAGER_FLAG_NOADDR |
CLUSTER_MANAGER_FLAG_DISCONNECT |
CLUSTER_MANAGER_FLAG_FAIL))
{
goto invalid_friend;
}
listAddNodeTail(cluster_manager.nodes, friend);
} else {
clusterManagerLogErr("[ERR] Unable to load info for "
@ -3581,6 +3587,8 @@ static int clusterManagerLoadInfoFromNode(clusterManagerNode *node, int opts) {
}
continue;
invalid_friend:
if (!(friend->flags & CLUSTER_MANAGER_FLAG_SLAVE))
cluster_manager.unreachable_masters++;
freeClusterManagerNode(friend);
}
listRelease(node->friends);
@ -3771,17 +3779,18 @@ int clusterManagerGetCoveredSlots(char *all_slots) {
}
void clusterManagerPrintSlotsList(list *slots) {
clusterManagerNode n = {0};
listIter li;
listNode *ln;
listRewind(slots, &li);
sds first = NULL;
while ((ln = listNext(&li)) != NULL) {
sds slot = ln->value;
if (!first) first = slot;
else printf(", ");
printf("%s", slot);
int slot = atoi(ln->value);
if (slot >= 0 && slot < CLUSTER_MANAGER_SLOTS)
n.slots[slot] = 1;
}
printf("\n");
sds nodeslist = clusterManagerNodeSlotsString(&n);
printf("%s\n", nodeslist);
sdsfree(nodeslist);
}
/* Return the node, among 'nodes' with the greatest number of keys
@ -3846,24 +3855,38 @@ static clusterManagerNode *clusterManagerNodeWithLeastReplicas() {
* more nodes. This function fixes this condition by migrating keys where
* it seems more sensible. */
int clusterManagerFixOpenSlot(int slot) {
int force_fix = config.cluster_manager_command.flags &
CLUSTER_MANAGER_CMD_FLAG_FIX_WITH_UNREACHABLE_MASTERS;
if (cluster_manager.unreachable_masters > 0 && !force_fix) {
clusterManagerLogWarn("*** Fixing open slots with %d unreachable masters is dangerous: redis-cli will assume that slots about masters that are not reachable are not covered, and will try to reassign them to the reachable nodes. This can cause data loss and is rarely what you want to do. If you really want to proceed use the --cluster-fix-with-unreachable-masters option.\n", cluster_manager.unreachable_masters);
exit(1);
}
clusterManagerLogInfo(">>> Fixing open slot %d\n", slot);
/* Try to obtain the current slot owner, according to the current
* nodes configuration. */
int success = 1;
list *owners = listCreate();
list *owners = listCreate(); /* List of nodes claiming some ownership.
it could be stating in the configuration
to have the node ownership, or just
holding keys for such slot. */
list *migrating = listCreate();
list *importing = listCreate();
sds migrating_str = sdsempty();
sds importing_str = sdsempty();
clusterManagerNode *owner = NULL;
clusterManagerNode *owner = NULL; /* The obvious slot owner if any. */
/* Iterate all the nodes, looking for potential owners of this slot. */
listIter li;
listNode *ln;
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *n = ln->value;
if (n->flags & CLUSTER_MANAGER_FLAG_SLAVE) continue;
if (n->slots[slot]) listAddNodeTail(owners, n);
else {
if (n->slots[slot]) {
listAddNodeTail(owners, n);
} else {
redisReply *r = CLUSTER_MANAGER_COMMAND(n,
"CLUSTER COUNTKEYSINSLOT %d", slot);
success = clusterManagerCheckRedisReply(n, r, NULL);
@ -3877,7 +3900,14 @@ int clusterManagerFixOpenSlot(int slot) {
if (!success) goto cleanup;
}
}
/* If we have only a single potential owner for this slot,
* set it as "owner". */
if (listLength(owners) == 1) owner = listFirst(owners)->value;
/* Scan the list of nodes again, in order to populate the
* list of nodes in importing or migrating state for
* this slot. */
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *n = ln->value;
@ -3909,6 +3939,7 @@ int clusterManagerFixOpenSlot(int slot) {
}
}
}
/* If the node is neither migrating nor importing and it's not
* the owner, then is added to the importing list in case
* it has keys in the slot. */
@ -3933,11 +3964,12 @@ int clusterManagerFixOpenSlot(int slot) {
printf("Set as migrating in: %s\n", migrating_str);
if (sdslen(importing_str) > 0)
printf("Set as importing in: %s\n", importing_str);
/* If there is no slot owner, set as owner the node with the biggest
* number of keys, among the set of migrating / importing nodes. */
if (owner == NULL) {
clusterManagerLogInfo(">>> Nobody claims ownership, "
"selecting an owner...\n");
clusterManagerLogInfo(">>> No single clear owner for the slot, "
"selecting an owner by # of keys...\n");
owner = clusterManagerGetNodeWithMostKeysInSlot(cluster_manager.nodes,
slot, NULL);
// If we still don't have an owner, we can't fix it.
@ -3968,6 +4000,7 @@ int clusterManagerFixOpenSlot(int slot) {
clusterManagerRemoveNodeFromList(migrating, owner);
clusterManagerRemoveNodeFromList(importing, owner);
}
/* If there are multiple owners of the slot, we need to fix it
* so that a single node is the owner and all the other nodes
* are in importing state. Later the fix can be handled by one
@ -4000,6 +4033,7 @@ int clusterManagerFixOpenSlot(int slot) {
}
}
int move_opts = CLUSTER_MANAGER_OPT_VERBOSE;
/* Case 1: The slot is in migrating state in one node, and in
* importing state in 1 node. That's trivial to address. */
if (listLength(migrating) == 1 && listLength(importing) == 1) {
@ -4011,6 +4045,7 @@ int clusterManagerFixOpenSlot(int slot) {
move_opts |= CLUSTER_MANAGER_OPT_UPDATE;
success = clusterManagerMoveSlot(src, dst, slot, move_opts, NULL);
}
/* Case 2: There are multiple nodes that claim the slot as importing,
* they probably got keys about the slot after a restart so opened
* the slot. In this case we just move all the keys to the owner
@ -4041,6 +4076,7 @@ int clusterManagerFixOpenSlot(int slot) {
if (!success) goto cleanup;
}
}
/* Case 3: The slot is in migrating state in one node but multiple
* other nodes claim to be in importing state and don't have any key in
* the slot. We search for the importing node having the same ID as

19
src/redis-cli.h
View File

@ -66,6 +66,7 @@ extern "C" {
#define CLUSTER_MANAGER_CMD_FLAG_COPY 1 << 7
#define CLUSTER_MANAGER_CMD_FLAG_COLOR 1 << 8
#define CLUSTER_MANAGER_CMD_FLAG_CHECK_OWNERS 1 << 9
#define CLUSTER_MANAGER_CMD_FLAG_FIX_WITH_UNREACHABLE_MASTERS 1 << 10
#define CLUSTER_MANAGER_OPT_GETFRIENDS 1 << 0
#define CLUSTER_MANAGER_OPT_COLD 1 << 1
@ -180,11 +181,12 @@ extern struct config {
int resp3;
} config;
/* The Cluster Manager global structure */
extern struct clusterManager {
struct clusterManager {
list *nodes; /* List of nodes in the configuration. */
list *errors;
} cluster_manager;
int unreachable_masters; /* Masters we are not able to reach. */
};
extern struct clusterManager cluster_manager;
typedef struct clusterManagerNode {
redisContext *context;
@ -196,7 +198,7 @@ typedef struct clusterManagerNode {
time_t ping_recv;
int flags;
list *flags_str; /* Flags string representations */
sds replicate; /* Master ID if node is a replica */
sds replicate; /* Master ID if node is a slave */
int dirty; /* Node has changes that can be flushed */
uint8_t slots[CLUSTER_MANAGER_SLOTS];
int slots_count;
@ -226,6 +228,15 @@ typedef struct clusterManagerReshardTableItem {
int slot;
} clusterManagerReshardTableItem;
/* Info about a cluster internal link. */
typedef struct clusterManagerLink {
sds node_name;
sds node_addr;
int connected;
int handshaking;
} clusterManagerLink;
typedef struct typeinfo {
char *name;
char *sizecmd;

70
src/replication.cpp
View File

@ -46,6 +46,7 @@
#include <unordered_map>
#include <string>
long long adjustMeaningfulReplOffset();
void replicationDiscardCachedMaster(redisMaster *mi);
void replicationResurrectCachedMaster(redisMaster *mi, connection *conn);
void replicationSendAck(redisMaster *mi);
@ -3249,6 +3250,9 @@ void replicationCacheMaster(redisMaster *mi, client *c) {
* pending outputs to the master. */
sdsclear(mi->master->querybuf);
sdsclear(mi->master->pending_querybuf);
/* Adjust reploff and read_reploff to the last meaningful offset we executed.
* this is the offset the replica will use for future PSYNC. */
mi->master->reploff = adjustMeaningfulReplOffset();
mi->master->read_reploff = mi->master->reploff;
if (c->flags & CLIENT_MULTI) discardTransaction(c);
listEmpty(c->reply);
@ -3274,9 +3278,41 @@ void replicationCacheMaster(redisMaster *mi, client *c) {
replicationHandleMasterDisconnection(mi);
}
/* This function is called when a master is turend into a replica, in order to
/* If the "meaningful" offset, that is the offset without the final PINGs
* in the stream, is different than the last offset, use it instead:
* often when the master is no longer reachable, replicas will never
* receive the PINGs, however the master will end with an incremented
* offset because of the PINGs and will not be able to incrementally
* PSYNC with the new master.
* This function trims the replication backlog when needed, and returns
* the offset to be used for future partial sync. */
long long adjustMeaningfulReplOffset() {
if (g_pserver->master_repl_offset > g_pserver->master_repl_meaningful_offset) {
long long delta = g_pserver->master_repl_offset -
g_pserver->master_repl_meaningful_offset;
serverLog(LL_NOTICE,
"Using the meaningful offset %lld instead of %lld to exclude "
"the final PINGs (%lld bytes difference)",
g_pserver->master_repl_meaningful_offset,
g_pserver->master_repl_offset,
delta);
g_pserver->master_repl_offset = g_pserver->master_repl_meaningful_offset;
if (g_pserver->repl_backlog_histlen <= delta) {
g_pserver->repl_backlog_histlen = 0;
g_pserver->repl_backlog_idx = 0;
} else {
g_pserver->repl_backlog_histlen -= delta;
g_pserver->repl_backlog_idx =
(g_pserver->repl_backlog_idx + (g_pserver->repl_backlog_size - delta)) %
g_pserver->repl_backlog_size;
}
}
return g_pserver->master_repl_offset;
}
/* This function is called when a master is turend into a slave, in order to
* create from scratch a cached master for the new client, that will allow
* to PSYNC with the replica that was promoted as the new master after a
* to PSYNC with the slave that was promoted as the new master after a
* failover.
*
* Assuming this instance was previously the master instance of the new master,
@ -3299,35 +3335,7 @@ void replicationCacheMasterUsingMyself(redisMaster *mi) {
* by replicationCreateMasterClient(). We'll later set the created
* master as server.cached_master, so the replica will use such
* offset for PSYNC. */
mi->master_initial_offset = g_pserver->master_repl_offset;
/* However if the "meaningful" offset, that is the offset without
* the final PINGs in the stream, is different, use this instead:
* often when the master is no longer reachable, replicas will never
* receive the PINGs, however the master will end with an incremented
* offset because of the PINGs and will not be able to incrementally
* PSYNC with the new master. */
if (g_pserver->master_repl_offset > g_pserver->master_repl_meaningful_offset) {
long long delta = g_pserver->master_repl_offset -
g_pserver->master_repl_meaningful_offset;
serverLog(LL_NOTICE,
"Using the meaningful offset %lld instead of %lld to exclude "
"the final PINGs (%lld bytes difference)",
g_pserver->master_repl_meaningful_offset,
g_pserver->master_repl_offset,
delta);
mi->master_initial_offset = g_pserver->master_repl_meaningful_offset;
g_pserver->master_repl_offset = g_pserver->master_repl_meaningful_offset;
if (g_pserver->repl_backlog_histlen <= delta) {
g_pserver->repl_backlog_histlen = 0;
g_pserver->repl_backlog_idx = 0;
} else {
g_pserver->repl_backlog_histlen -= delta;
g_pserver->repl_backlog_idx =
(g_pserver->repl_backlog_idx + (g_pserver->repl_backlog_size - delta)) %
g_pserver->repl_backlog_size;
}
}
mi->master_initial_offset = adjustMeaningfulReplOffset();
/* The master client we create can be set to any DBID, because
* the new master will start its replication stream with SELECT. */

2
src/sds.c
View File

@ -106,11 +106,11 @@ sds sdsnewlen(const void *init, ssize_t initlen) {
unsigned char *fp; /* flags pointer. */
sh = s_malloc(hdrlen+initlen+1, MALLOC_SHARED);
if (sh == NULL) return NULL;
if (init==SDS_NOINIT)
init = NULL;
else if (!init)
memset(sh, 0, hdrlen+initlen+1);
if (sh == NULL) return NULL;
s = (char*)sh+hdrlen;
fp = ((unsigned char*)s)-1;
switch(type) {

2
src/sentinel.cpp
View File

@ -869,8 +869,8 @@ void sentinelCollectTerminatedScripts(void) {
}
listDelNode(sentinel.scripts_queue,ln);
sentinelReleaseScriptJob(sj);
sentinel.running_scripts--;
}
sentinel.running_scripts--;
}
}

61
src/server.cpp
View File

@ -1047,8 +1047,8 @@ struct redisCommand redisCommandTable[] = {
"write fast @hash",
0,NULL,0,0,0,0,0,0},
{"lcs",lcsCommand,-4,
"write use-memory @string",
{"stralgo",stralgoCommand,-2,
"read-only @string",
0,lcsGetKeys,0,0,0,0,0,0}
};
@ -1264,11 +1264,15 @@ int dictEncObjKeyCompare(void *privdata, const void *key1,
o2->encoding == OBJ_ENCODING_INT)
return ptrFromObj(o1) == ptrFromObj(o2);
o1 = getDecodedObject(o1);
o2 = getDecodedObject(o2);
/* Due to OBJ_STATIC_REFCOUNT, we avoid calling getDecodedObject() without
* good reasons, because it would incrRefCount() the object, which
* is invalid. So we check to make sure dictFind() works with static
* objects as well. */
if (o1->getrefcount() != OBJ_STATIC_REFCOUNT) o1 = getDecodedObject(o1);
if (o2->getrefcount() != OBJ_STATIC_REFCOUNT) o2 = getDecodedObject(o2);
cmp = dictSdsKeyCompare(privdata,ptrFromObj(o1),ptrFromObj(o2));
decrRefCount(o1);
decrRefCount(o2);
if (o1->getrefcount() != OBJ_STATIC_REFCOUNT) decrRefCount(o1);
if (o2->getrefcount() != OBJ_STATIC_REFCOUNT) decrRefCount(o2);
return cmp;
}
@ -2190,21 +2194,6 @@ void beforeSleep(struct aeEventLoop *eventLoop) {
if (g_pserver->active_expire_enabled && (listLength(g_pserver->masters) == 0 || g_pserver->fActiveReplica))
activeExpireCycle(ACTIVE_EXPIRE_CYCLE_FAST);
/* Send all the slaves an ACK request if at least one client blocked
* during the previous event loop iteration. */
if (g_pserver->get_ack_from_slaves) {
robj *argv[3];
argv[0] = createStringObject("REPLCONF",8);
argv[1] = createStringObject("GETACK",6);
argv[2] = createStringObject("*",1); /* Not used argument. */
replicationFeedSlaves(g_pserver->slaves, g_pserver->replicaseldb, argv, 3);
decrRefCount(argv[0]);
decrRefCount(argv[1]);
decrRefCount(argv[2]);
g_pserver->get_ack_from_slaves = 0;
}
/* Unblock all the clients blocked for synchronous replication
* in WAIT. */
if (listLength(g_pserver->clients_waiting_acks))
@ -2220,6 +2209,24 @@ void beforeSleep(struct aeEventLoop *eventLoop) {
processUnblockedClients(IDX_EVENT_LOOP_MAIN);
}
/* Send all the slaves an ACK request if at least one client blocked
* during the previous event loop iteration. Note that we do this after
* processUnblockedClients(), so if there are multiple pipelined WAITs
* and the just unblocked WAIT gets blocked again, we don't have to wait
* a server cron cycle in absence of other event loop events. See #6623. */
if (g_pserver->get_ack_from_slaves) {
robj *argv[3];
argv[0] = createStringObject("REPLCONF",8);
argv[1] = createStringObject("GETACK",6);
argv[2] = createStringObject("*",1); /* Not used argument. */
replicationFeedSlaves(g_pserver->slaves, g_pserver->replicaseldb, argv, 3);
decrRefCount(argv[0]);
decrRefCount(argv[1]);
decrRefCount(argv[2]);
g_pserver->get_ack_from_slaves = 0;
}
/* Send the invalidation messages to clients participating to the
* client side caching protocol in broadcasting (BCAST) mode. */
trackingBroadcastInvalidationMessages();
@ -3117,6 +3124,7 @@ void initServer(void) {
scriptingInit(1);
slowlogInit();
latencyMonitorInit();
crc64_init();
}
/* Some steps in server initialization need to be done last (after modules
@ -3308,8 +3316,13 @@ struct redisCommand *lookupCommandOrOriginal(sds name) {
* + PROPAGATE_AOF (propagate into the AOF file if is enabled)
* + PROPAGATE_REPL (propagate into the replication link)
*
* This should not be used inside commands implementation. Use instead
* alsoPropagate(), preventCommandPropagation(), forceCommandPropagation().
* This should not be used inside commands implementation since it will not
* wrap the resulting commands in MULTI/EXEC. Use instead alsoPropagate(),
* preventCommandPropagation(), forceCommandPropagation().
*
* However for functions that need to (also) propagate out of the context of a
* command execution, for example when serving a blocked client, you
* want to use propagate().
*/
void propagate(struct redisCommand *cmd, int dbid, robj **argv, int argc,
int flags)
@ -4307,7 +4320,7 @@ sds genRedisInfoString(const char *section) {
size_t zmalloc_used = zmalloc_used_memory();
size_t total_system_mem = cserver.system_memory_size;
const char *evict_policy = evictPolicyToString();
long long memory_lua = (long long)lua_gc(g_pserver->lua,LUA_GCCOUNT,0)*1024;
long long memory_lua = g_pserver->lua ? (long long)lua_gc(g_pserver->lua,LUA_GCCOUNT,0)*1024 : 0;
struct redisMemOverhead *mh = getMemoryOverheadData();
/* Peak memory is updated from time to time by serverCron() so it

19
src/server.h
View File

@ -427,8 +427,10 @@ public:
#define CLIENT_TRACKING_OPTOUT (1ULL<<35) /* Tracking in opt-out mode. */
#define CLIENT_TRACKING_CACHING (1ULL<<36) /* CACHING yes/no was given,
depending on optin/optout mode. */
#define CLIENT_IN_TO_TABLE (1ULL<<37) /* This client is in the timeout table. */
#define CLIENT_FORCE_REPLY (1ULL<<38) /* Should addReply be forced to write the text? */
#define CLIENT_TRACKING_NOLOOP (1ULL<<37) /* Don't send invalidation messages
about writes performed by myself.*/
#define CLIENT_IN_TO_TABLE (1ULL<<38) /* This client is in the timeout table. */
#define CLIENT_FORCE_REPLY (1ULL<<39) /* Should addReply be forced to write the text? */
/* Client block type (btype field in client structure)
* if CLIENT_BLOCKED flag is set. */
@ -796,7 +798,7 @@ public:
void SetFExpires(bool fExpires);
void setrefcount(unsigned ref);
unsigned getrefcount(std::memory_order order) const { return (refcount.load(order) & ~(1U << 31)); }
unsigned getrefcount(std::memory_order order = std::memory_order_relaxed) const { return (refcount.load(order) & ~(1U << 31)); }
void addref() const { refcount.fetch_add(1, std::memory_order_relaxed); }
unsigned release() const { return refcount.fetch_sub(1, std::memory_order_relaxed) & ~(1U << 31); }
} robj;
@ -2153,7 +2155,6 @@ void setDeferredSetLen(client *c, void *node, long length);
void setDeferredAttributeLen(client *c, void *node, long length);
void setDeferredPushLen(client *c, void *node, long length);
void processInputBuffer(client *c, int callFlags);
void processInputBufferAndReplicate(client *c);
void acceptHandler(aeEventLoop *el, int fd, void *privdata, int mask);
void acceptTcpHandler(aeEventLoop *el, int fd, void *privdata, int mask);
void acceptTLSHandler(aeEventLoop *el, int fd, void *privdata, int mask);
@ -2261,7 +2262,7 @@ void addReplyStatusFormat(client *c, const char *fmt, ...);
void enableTracking(client *c, uint64_t redirect_to, uint64_t options, robj **prefix, size_t numprefix);
void disableTracking(client *c);
void trackingRememberKeys(client *c);
void trackingInvalidateKey(robj *keyobj);
void trackingInvalidateKey(client *c, robj *keyobj);
void trackingInvalidateKeysOnFlush(int dbid);
void trackingLimitUsedSlots(void);
uint64_t trackingGetTotalItems(void);
@ -2660,8 +2661,8 @@ void dbAdd(redisDb *db, robj *key, robj *val);
int dbAddRDBLoad(redisDb *db, sds key, robj *val);
void dbOverwrite(redisDb *db, robj *key, robj *val);
int dbMerge(redisDb *db, robj *key, robj *val, int fReplace);
void genericSetKey(redisDb *db, robj *key, robj *val, int keepttl, int signal);
void setKey(redisDb *db, robj *key, robj *val);
void genericSetKey(client *c, redisDb *db, robj *key, robj *val, int keepttl, int signal);
void setKey(client *c, redisDb *db, robj *key, robj *val);
int dbExists(redisDb *db, robj *key);
robj *dbRandomKey(redisDb *db);
int dbSyncDelete(redisDb *db, robj *key);
@ -2677,7 +2678,7 @@ void flushAllDataAndResetRDB(int flags);
long long dbTotalServerKeyCount();
int selectDb(client *c, int id);
void signalModifiedKey(redisDb *db, robj *key);
void signalModifiedKey(client *c, redisDb *db, robj *key);
void signalFlushedDb(int dbid);
unsigned int getKeysInSlot(unsigned int hashslot, robj **keys, unsigned int count);
unsigned int countKeysInSlot(unsigned int hashslot);
@ -2982,7 +2983,7 @@ void xtrimCommand(client *c);
void aclCommand(client *c);
void replicaReplayCommand(client *c);
void hrenameCommand(client *c);
void lcsCommand(client *c);
void stralgoCommand(client *c);
int FBrokenLinkToMaster();
int FActiveMaster(client *c);

4
src/sort.cpp
View File

@ -571,12 +571,12 @@ void sortCommand(client *c) {
}
}
if (outputlen) {
setKey(c->db,storekey,sobj);
setKey(c,c->db,storekey,sobj);
notifyKeyspaceEvent(NOTIFY_LIST,"sortstore",storekey,
c->db->id);
g_pserver->dirty += outputlen;
} else if (dbDelete(c->db,storekey)) {
signalModifiedKey(c->db,storekey);
signalModifiedKey(c,c->db,storekey);
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",storekey,c->db->id);
g_pserver->dirty++;
}

10
src/t_hash.cpp
View File

@ -521,7 +521,7 @@ void hsetnxCommand(client *c) {
} else {
hashTypeSet(o,szFromObj(c->argv[2]),szFromObj(c->argv[3]),HASH_SET_COPY);
addReply(c, shared.cone);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_HASH,"hset",c->argv[1],c->db->id);
g_pserver->dirty++;
}
@ -551,7 +551,7 @@ void hsetCommand(client *c) {
/* HMSET */
addReply(c, shared.ok);
}
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_HASH,"hset",c->argv[1],c->db->id);
g_pserver->dirty++;
}
@ -586,7 +586,7 @@ void hincrbyCommand(client *c) {
newstr = sdsfromlonglong(value);
hashTypeSet(o,szFromObj(c->argv[2]),newstr,HASH_SET_TAKE_VALUE);
addReplyLongLong(c,value);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_HASH,"hincrby",c->argv[1],c->db->id);
g_pserver->dirty++;
}
@ -625,7 +625,7 @@ void hincrbyfloatCommand(client *c) {
newstr = sdsnewlen(buf,len);
hashTypeSet(o,szFromObj(c->argv[2]),newstr,HASH_SET_TAKE_VALUE);
addReplyBulkCBuffer(c,buf,len);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_HASH,"hincrbyfloat",c->argv[1],c->db->id);
g_pserver->dirty++;
@ -721,7 +721,7 @@ void hdelCommand(client *c) {
}
}
if (deleted) {
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_HASH,"hdel",c->argv[1],c->db->id);
if (keyremoved)
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",c->argv[1],

20
src/t_list.cpp
View File

@ -217,7 +217,7 @@ void pushGenericCommand(client *c, int where) {
if (pushed) {
const char *event = (where == LIST_HEAD) ? "lpush" : "rpush";
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_LIST,event,c->argv[1],c->db->id);
}
g_pserver->dirty += pushed;
@ -247,7 +247,7 @@ void pushxGenericCommand(client *c, int where) {
if (pushed) {
const char *event = (where == LIST_HEAD) ? "lpush" : "rpush";
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_LIST,event,c->argv[1],c->db->id);
}
g_pserver->dirty += pushed;
@ -292,7 +292,7 @@ void linsertCommand(client *c) {
listTypeReleaseIterator(iter);
if (inserted) {
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_LIST,"linsert",
c->argv[1],c->db->id);
g_pserver->dirty++;
@ -355,7 +355,7 @@ void lsetCommand(client *c) {
addReply(c,shared.outofrangeerr);
} else {
addReply(c,shared.ok);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_LIST,"lset",c->argv[1],c->db->id);
g_pserver->dirty++;
}
@ -382,7 +382,7 @@ void popGenericCommand(client *c, int where) {
c->argv[1],c->db->id);
dbDelete(c->db,c->argv[1]);
}
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
g_pserver->dirty++;
}
}
@ -482,7 +482,7 @@ void ltrimCommand(client *c) {
dbDelete(c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",c->argv[1],c->db->id);
}
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
g_pserver->dirty++;
addReply(c,shared.ok);
}
@ -519,7 +519,7 @@ void lremCommand(client *c) {
listTypeReleaseIterator(li);
if (removed) {
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_LIST,"lrem",c->argv[1],c->db->id);
}
@ -555,7 +555,7 @@ static void rpoplpushHandlePush(client *c, robj *dstkey, robj *dstobj, robj *val
g_pserver->list_compress_depth);
dbAdd(c->db,dstkey,dstobj);
}
signalModifiedKey(c->db,dstkey);
signalModifiedKey(c,c->db,dstkey);
listTypePush(dstobj,value,LIST_HEAD);
notifyKeyspaceEvent(NOTIFY_LIST,"lpush",dstkey,c->db->id);
/* Always send the pushed value to the client. */
@ -593,7 +593,7 @@ void rpoplpushCommand(client *c) {
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",
touchedkey,c->db->id);
}
signalModifiedKey(c->db,touchedkey);
signalModifiedKey(c,c->db,touchedkey);
decrRefCount(touchedkey);
g_pserver->dirty++;
if (c->cmd->proc == brpoplpushCommand) {
@ -713,7 +713,7 @@ void blockingPopGenericCommand(client *c, int where) {
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",
c->argv[j],c->db->id);
}
signalModifiedKey(c->db,c->argv[j]);
signalModifiedKey(c,c->db,c->argv[j]);
g_pserver->dirty++;
/* Replicate it as an [LR]POP instead of B[LR]POP. */

20
src/t_set.cpp
View File

@ -285,7 +285,7 @@ void saddCommand(client *c) {
if (setTypeAdd(set,szFromObj(c->argv[j]))) added++;
}
if (added) {
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_SET,"sadd",c->argv[1],c->db->id);
}
g_pserver->dirty += added;
@ -310,7 +310,7 @@ void sremCommand(client *c) {
}
}
if (deleted) {
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_SET,"srem",c->argv[1],c->db->id);
if (keyremoved)
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",c->argv[1],
@ -363,8 +363,8 @@ void smoveCommand(client *c) {
dbAdd(c->db,c->argv[2],dstset);
}
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c->db,c->argv[2]);
signalModifiedKey(c,c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[2]);
g_pserver->dirty++;
/* An extra key has changed when ele was successfully added to dstset */
@ -449,7 +449,7 @@ void spopWithCountCommand(client *c) {
/* Propagate this command as an DEL operation */
rewriteClientCommandVector(c,2,shared.del,c->argv[1]);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
g_pserver->dirty++;
return;
}
@ -551,7 +551,7 @@ void spopWithCountCommand(client *c) {
* the alsoPropagate() API. */
decrRefCount(propargv[0]);
preventCommandPropagation(c);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
g_pserver->dirty++;
}
@ -604,7 +604,7 @@ void spopCommand(client *c) {
}
/* Set has been modified */
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
g_pserver->dirty++;
}
@ -813,7 +813,7 @@ void sinterGenericCommand(client *c, robj **setkeys,
zfree(sets);
if (dstkey) {
if (dbDelete(c->db,dstkey)) {
signalModifiedKey(c->db,dstkey);
signalModifiedKey(c,c->db,dstkey);
g_pserver->dirty++;
}
addReply(c,shared.czero);
@ -913,7 +913,7 @@ void sinterGenericCommand(client *c, robj **setkeys,
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",
dstkey,c->db->id);
}
signalModifiedKey(c->db,dstkey);
signalModifiedKey(c,c->db,dstkey);
g_pserver->dirty++;
} else {
setDeferredSetLen(c,replylen,cardinality);
@ -1088,7 +1088,7 @@ void sunionDiffGenericCommand(client *c, robj **setkeys, int setnum,
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",
dstkey,c->db->id);
}
signalModifiedKey(c->db,dstkey);
signalModifiedKey(c,c->db,dstkey);
g_pserver->dirty++;
}
zfree(sets);

270
src/t_stream.cpp
View File

@ -853,7 +853,12 @@ void streamPropagateXCLAIM(client *c, robj *key, streamCG *group, robj *groupnam
argv[11] = createStringObject("JUSTID",6);
argv[12] = createStringObject("LASTID",6);
argv[13] = createObjectFromStreamID(&group->last_id);
alsoPropagate(cserver.xclaimCommand,c->db->id,argv,14,PROPAGATE_AOF|PROPAGATE_REPL);
/* We use progagate() because this code path is not always called from
* the command execution context. Moreover this will just alter the
* consumer group state, and we don't need MULTI/EXEC wrapping because
* there is no message state cross-message atomicity required. */
propagate(cserver.xclaimCommand,c->db->id,argv,14,PROPAGATE_AOF|PROPAGATE_REPL);
decrRefCount(argv[0]);
decrRefCount(argv[3]);
decrRefCount(argv[4]);
@ -880,7 +885,12 @@ void streamPropagateGroupID(client *c, robj *key, streamCG *group, robj *groupna
argv[2] = key;
argv[3] = groupname;
argv[4] = createObjectFromStreamID(&group->last_id);
alsoPropagate(cserver.xgroupCommand,c->db->id,argv,5,PROPAGATE_AOF|PROPAGATE_REPL);
/* We use progagate() because this code path is not always called from
* the command execution context. Moreover this will just alter the
* consumer group state, and we don't need MULTI/EXEC wrapping because
* there is no message state cross-message atomicity required. */
propagate(cserver.xgroupCommand,c->db->id,argv,5,PROPAGATE_AOF|PROPAGATE_REPL);
decrRefCount(argv[0]);
decrRefCount(argv[1]);
decrRefCount(argv[4]);
@ -940,6 +950,8 @@ size_t streamReplyWithRange(client *c, stream *s, streamID *start, streamID *end
streamIterator si;
int64_t numfields;
streamID id;
int propagate_last_id = 0;
int noack = flags & STREAM_RWR_NOACK;
/* If the client is asking for some history, we serve it using a
* different function, so that we return entries *solely* from its
@ -955,12 +967,13 @@ size_t streamReplyWithRange(client *c, stream *s, streamID *start, streamID *end
arraylen_ptr = addReplyDeferredLenAsync(c);
streamIteratorStart(&si,s,start,end,rev);
while(streamIteratorGetID(&si,&id,&numfields)) {
int propagate_last_id = 0;
/* Update the group last_id if needed. */
if (group && streamCompareID(&id,&group->last_id) > 0) {
group->last_id = id;
propagate_last_id = 1;
/* Group last ID should be propagated only if NOACK was
* specified, otherwise the last id will be included
* in the propagation of XCLAIM itself. */
if (noack) propagate_last_id = 1;
}
/* Emit a two elements array for each item. The first is
@ -968,7 +981,7 @@ size_t streamReplyWithRange(client *c, stream *s, streamID *start, streamID *end
addReplyArrayLenAsync(c,2);
addReplyStreamIDAsync(c,&id);
addReplyMapLenAsync(c,numfields);
addReplyArrayLenAsync(c,numfields*2);
/* Emit the field-value pairs. */
while(numfields--) {
@ -988,7 +1001,7 @@ size_t streamReplyWithRange(client *c, stream *s, streamID *start, streamID *end
* XGROUP SETID command. So if we find that there is already
* a NACK for the entry, we need to associate it to the new
* consumer. */
if (group && !(flags & STREAM_RWR_NOACK)) {
if (group && !noack) {
unsigned char buf[sizeof(streamID)];
streamEncodeID(buf,&id);
@ -1025,14 +1038,15 @@ size_t streamReplyWithRange(client *c, stream *s, streamID *start, streamID *end
streamPropagateXCLAIM(c,spi->keyname,group,spi->groupname,idarg,nack);
decrRefCount(idarg);
}
} else {
if (propagate_last_id)
streamPropagateGroupID(c,spi->keyname,group,spi->groupname);
}
arraylen++;
if (count && count == arraylen) break;
}
if (spi && propagate_last_id)
streamPropagateGroupID(c,spi->keyname,group,spi->groupname);
streamIteratorStop(&si);
if (arraylen_ptr) setDeferredArrayLenAsync(c,arraylen_ptr,arraylen);
return arraylen;
@ -1268,7 +1282,7 @@ void xaddCommand(client *c) {
}
addReplyStreamID(c,&id);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STREAM,"xadd",c->argv[1],c->db->id);
g_pserver->dirty++;
@ -2396,7 +2410,7 @@ void xdelCommand(client *c) {
/* Propagate the write if needed. */
if (deleted) {
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STREAM,"xdel",c->argv[1],c->db->id);
g_pserver->dirty += deleted;
}
@ -2473,7 +2487,7 @@ void xtrimCommand(client *c) {
/* Propagate the write if needed. */
if (deleted) {
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STREAM,"xtrim",c->argv[1],c->db->id);
g_pserver->dirty += deleted;
if (approx_maxlen) streamRewriteApproxMaxlen(c,s,maxlen_arg_idx);
@ -2481,16 +2495,204 @@ void xtrimCommand(client *c) {
addReplyLongLong(c,deleted);
}
/* Helper function for xinfoCommand.
* Handles the variants of XINFO STREAM */
void xinfoReplyWithStreamInfo(client *c, stream *s) {
int full = 1;
long long count = 10; /* Default COUNT is 10 so we don't block the server */
robj **optv = c->argv + 3; /* Options start after XINFO STREAM <key> */
int optc = c->argc - 3;
/* Parse options. */
if (optc == 0) {
full = 0;
} else {
/* Valid options are [FULL] or [FULL COUNT <count>] */
if (optc != 1 && optc != 3) {
addReplySubcommandSyntaxError(c);
return;
}
/* First option must be "FULL" */
if (strcasecmp(szFromObj(optv[0]),"full")) {
addReplySubcommandSyntaxError(c);
return;
}
if (optc == 3) {
/* First option must be "FULL" */
if (strcasecmp(szFromObj(optv[1]),"count")) {
addReplySubcommandSyntaxError(c);
return;
}
if (getLongLongFromObjectOrReply(c,optv[2],&count,NULL) == C_ERR)
return;
if (count < 0) count = 10;
}
}
addReplyMapLen(c,full ? 6 : 7);
addReplyBulkCString(c,"length");
addReplyLongLong(c,s->length);
addReplyBulkCString(c,"radix-tree-keys");
addReplyLongLong(c,raxSize(s->prax));
addReplyBulkCString(c,"radix-tree-nodes");
addReplyLongLong(c,s->prax->numnodes);
addReplyBulkCString(c,"last-generated-id");
addReplyStreamID(c,&s->last_id);
if (!full) {
/* XINFO STREAM <key> */
addReplyBulkCString(c,"groups");
addReplyLongLong(c,s->cgroups ? raxSize(s->cgroups) : 0);
/* To emit the first/last entry we use streamReplyWithRange(). */
int emitted;
streamID start, end;
start.ms = start.seq = 0;
end.ms = end.seq = UINT64_MAX;
addReplyBulkCString(c,"first-entry");
emitted = streamReplyWithRange(c,s,&start,&end,1,0,NULL,NULL,
STREAM_RWR_RAWENTRIES,NULL);
if (!emitted) addReplyNull(c);
addReplyBulkCString(c,"last-entry");
emitted = streamReplyWithRange(c,s,&start,&end,1,1,NULL,NULL,
STREAM_RWR_RAWENTRIES,NULL);
if (!emitted) addReplyNull(c);
} else {
/* XINFO STREAM <key> FULL [COUNT <count>] */
/* Stream entries */
addReplyBulkCString(c,"entries");
streamReplyWithRange(c,s,NULL,NULL,count,0,NULL,NULL,0,NULL);
/* Consumer groups */
addReplyBulkCString(c,"groups");
if (s->cgroups == NULL) {
addReplyArrayLen(c,0);
} else {
addReplyArrayLen(c,raxSize(s->cgroups));
raxIterator ri_cgroups;
raxStart(&ri_cgroups,s->cgroups);
raxSeek(&ri_cgroups,"^",NULL,0);
while(raxNext(&ri_cgroups)) {
streamCG *cg = (streamCG*) ri_cgroups.data;
addReplyMapLen(c,5);
/* Name */
addReplyBulkCString(c,"name");
addReplyBulkCBuffer(c,ri_cgroups.key,ri_cgroups.key_len);
/* Last delivered ID */
addReplyBulkCString(c,"last-delivered-id");
addReplyStreamID(c,&cg->last_id);
/* Group PEL count */
addReplyBulkCString(c,"pel-count");
addReplyLongLong(c,raxSize(cg->pel));
/* Group PEL */
addReplyBulkCString(c,"pending");
long long arraylen_cg_pel = 0;
void *arrayptr_cg_pel = addReplyDeferredLen(c);
raxIterator ri_cg_pel;
raxStart(&ri_cg_pel,cg->pel);
raxSeek(&ri_cg_pel,"^",NULL,0);
while(raxNext(&ri_cg_pel) && (!count || arraylen_cg_pel < count)) {
streamNACK *nack = (streamNACK*) ri_cg_pel.data;
addReplyArrayLen(c,4);
/* Entry ID. */
streamID id;
streamDecodeID(ri_cg_pel.key,&id);
addReplyStreamID(c,&id);
/* Consumer name. */
addReplyBulkCBuffer(c,nack->consumer->name,
sdslen(nack->consumer->name));
/* Last delivery. */
addReplyLongLong(c,nack->delivery_time);
/* Number of deliveries. */
addReplyLongLong(c,nack->delivery_count);
arraylen_cg_pel++;
}
setDeferredArrayLen(c,arrayptr_cg_pel,arraylen_cg_pel);
raxStop(&ri_cg_pel);
/* Consumers */
addReplyBulkCString(c,"consumers");
addReplyArrayLen(c,raxSize(cg->consumers));
raxIterator ri_consumers;
raxStart(&ri_consumers,cg->consumers);
raxSeek(&ri_consumers,"^",NULL,0);
while(raxNext(&ri_consumers)) {
streamConsumer *consumer = (streamConsumer*)ri_consumers.data;
addReplyMapLen(c,4);
/* Consumer name */
addReplyBulkCString(c,"name");
addReplyBulkCBuffer(c,consumer->name,sdslen(consumer->name));
/* Seen-time */
addReplyBulkCString(c,"seen-time");
addReplyLongLong(c,consumer->seen_time);
/* Consumer PEL count */
addReplyBulkCString(c,"pel-count");
addReplyLongLong(c,raxSize(consumer->pel));
/* Consumer PEL */
addReplyBulkCString(c,"pending");
long long arraylen_cpel = 0;
void *arrayptr_cpel = addReplyDeferredLen(c);
raxIterator ri_cpel;
raxStart(&ri_cpel,consumer->pel);
raxSeek(&ri_cpel,"^",NULL,0);
while(raxNext(&ri_cpel) && (!count || arraylen_cpel < count)) {
streamNACK *nack = (streamNACK*)ri_cpel.data;
addReplyArrayLen(c,3);
/* Entry ID. */
streamID id;
streamDecodeID(ri_cpel.key,&id);
addReplyStreamID(c,&id);
/* Last delivery. */
addReplyLongLong(c,nack->delivery_time);
/* Number of deliveries. */
addReplyLongLong(c,nack->delivery_count);
arraylen_cpel++;
}
setDeferredArrayLen(c,arrayptr_cpel,arraylen_cpel);
raxStop(&ri_cpel);
}
raxStop(&ri_consumers);
}
raxStop(&ri_cgroups);
}
}
}
/* XINFO CONSUMERS <key> <group>
* XINFO GROUPS <key>
* XINFO STREAM <key>
* XINFO STREAM <key> [FULL [COUNT <count>]]
* XINFO HELP. */
void xinfoCommand(client *c) {
const char *help[] = {
"CONSUMERS <key> <groupname> -- Show consumer groups of group <groupname>.",
"GROUPS <key> -- Show the stream consumer groups.",
"STREAM <key> -- Show information about the stream.",
"HELP -- Print this help.",
"CONSUMERS <key> <groupname> -- Show consumer groups of group <groupname>.",
"GROUPS <key> -- Show the stream consumer groups.",
"STREAM <key> [FULL [COUNT <count>]] -- Show information about the stream.",
" FULL will return the full state of the stream,",
" including all entries, groups, consumers and PELs.",
" It's possible to show only the first stream/PEL entries",
" by using the COUNT modifier (Default is 10)",
"HELP -- Print this help.",
NULL
};
stream *s = NULL;
@ -2570,36 +2772,10 @@ NULL
addReplyStreamID(c,&cg->last_id);
}
raxStop(&ri);
} else if (!strcasecmp(opt,"STREAM") && c->argc == 3) {
/* XINFO STREAM <key> (or the alias XINFO <key>). */
addReplyMapLen(c,7);
addReplyBulkCString(c,"length");
addReplyLongLong(c,s->length);
addReplyBulkCString(c,"radix-tree-keys");
addReplyLongLong(c,raxSize(s->prax));
addReplyBulkCString(c,"radix-tree-nodes");
addReplyLongLong(c,s->prax->numnodes);
addReplyBulkCString(c,"groups");
addReplyLongLong(c,s->cgroups ? raxSize(s->cgroups) : 0);
addReplyBulkCString(c,"last-generated-id");
addReplyStreamID(c,&s->last_id);
/* To emit the first/last entry we us the streamReplyWithRange()
* API. */
int count;
streamID start, end;
start.ms = start.seq = 0;
end.ms = end.seq = UINT64_MAX;
addReplyBulkCString(c,"first-entry");
count = streamReplyWithRange(c,s,&start,&end,1,0,NULL,NULL,
STREAM_RWR_RAWENTRIES,NULL);
if (!count) addReplyNull(c);
addReplyBulkCString(c,"last-entry");
count = streamReplyWithRange(c,s,&start,&end,1,1,NULL,NULL,
STREAM_RWR_RAWENTRIES,NULL);
if (!count) addReplyNull(c);
} else if (!strcasecmp(opt,"STREAM")) {
/* XINFO STREAM <key> [FULL [COUNT <count>]]. */
xinfoReplyWithStreamInfo(c,s);
} else {
addReplySubcommandSyntaxError(c);
}
}

37
src/t_string.cpp
View File

@ -84,7 +84,7 @@ void setGenericCommand(client *c, int flags, robj *key, robj *val, robj *expire,
addReply(c, abort_reply ? abort_reply : shared.null[c->resp]);
return;
}
genericSetKey(c->db,key,val,flags & OBJ_SET_KEEPTTL,1);
genericSetKey(c,c->db,key,val,flags & OBJ_SET_KEEPTTL,1);
g_pserver->dirty++;
if (expire) setExpire(c,c->db,key,nullptr,mstime()+milliseconds);
notifyKeyspaceEvent(NOTIFY_STRING,"set",key,c->db->id);
@ -183,7 +183,7 @@ void getCommand(client *c) {
void getsetCommand(client *c) {
if (getGenericCommand(c) == C_ERR) return;
c->argv[2] = tryObjectEncoding(c->argv[2]);
setKey(c->db,c->argv[1],c->argv[2]);
setKey(c,c->db,c->argv[1],c->argv[2]);
notifyKeyspaceEvent(NOTIFY_STRING,"set",c->argv[1],c->db->id);
g_pserver->dirty++;
}
@ -240,7 +240,7 @@ void setrangeCommand(client *c) {
if (sdslen(value) > 0) {
o->m_ptr = sdsgrowzero((sds)ptrFromObj(o),offset+sdslen(value));
memcpy((char*)ptrFromObj(o)+offset,value,sdslen(value));
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STRING,
"setrange",c->argv[1],c->db->id);
g_pserver->dirty++;
@ -329,7 +329,7 @@ void msetGenericCommand(client *c, int nx) {
for (j = 1; j < c->argc; j += 2) {
c->argv[j+1] = tryObjectEncoding(c->argv[j+1]);
setKey(c->db,c->argv[j],c->argv[j+1]);
setKey(c,c->db,c->argv[j],c->argv[j+1]);
notifyKeyspaceEvent(NOTIFY_STRING,"set",c->argv[j],c->db->id);
}
g_pserver->dirty += (c->argc-1)/2;
@ -374,7 +374,7 @@ void incrDecrCommand(client *c, long long incr) {
dbAdd(c->db,c->argv[1],newObj);
}
}
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STRING,"incrby",c->argv[1],c->db->id);
g_pserver->dirty++;
addReply(c,shared.colon);
@ -424,7 +424,7 @@ void incrbyfloatCommand(client *c) {
dbOverwrite(c->db,c->argv[1],newObj);
else
dbAdd(c->db,c->argv[1],newObj);
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STRING,"incrbyfloat",c->argv[1],c->db->id);
g_pserver->dirty++;
addReplyBulk(c,newObj);
@ -468,7 +468,7 @@ void appendCommand(client *c) {
o->m_ptr = sdscatlen((sds)ptrFromObj(o),ptrFromObj(append),sdslen((sds)ptrFromObj(append)));
totlen = sdslen((sds)ptrFromObj(o));
}
signalModifiedKey(c->db,c->argv[1]);
signalModifiedKey(c,c->db,c->argv[1]);
notifyKeyspaceEvent(NOTIFY_STRING,"append",c->argv[1],c->db->id);
g_pserver->dirty++;
addReplyLongLong(c,totlen);
@ -481,11 +481,24 @@ void strlenCommand(client *c) {
addReplyLongLong(c,stringObjectLen(o));
}
/* LCS -- Longest common subsequence.
/* STRALGO -- Implement complex algorithms on strings.
*
* LCS [IDX] [MINMATCHLEN <len>]
* STRINGS <string> <string> | KEYS <keya> <keyb> */
void lcsCommand(client *c) {
* STRALGO <algorithm> ... arguments ... */
void stralgoLCS(client *c); /* This implements the LCS algorithm. */
void stralgoCommand(client *c) {
/* Select the algorithm. */
if (!strcasecmp(szFromObj(c->argv[1]),"lcs")) {
stralgoLCS(c);
} else {
addReply(c,shared.syntaxerr);
}
}
/* STRALGO <algo> [IDX] [MINMATCHLEN <len>] [WITHMATCHLEN]
* STRINGS <string> <string> | KEYS <keya> <keyb>
*/
void stralgoLCS(client *c) {
uint32_t i, j;
long long minmatchlen = 0;
const char *a = NULL;
@ -493,7 +506,7 @@ void lcsCommand(client *c) {
int getlen = 0, getidx = 0, withmatchlen = 0;
robj_roptr obja, objb;
for (j = 1; j < (uint32_t)c->argc; j++) {
for (j = 2; j < (uint32_t)c->argc; j++) {
char *opt = szFromObj(c->argv[j]);
int moreargs = (c->argc-1) - j;

21
src/t_zset.cpp
View File

@ -186,7 +186,8 @@ zskiplistNode *zslInsert(zskiplist *zsl, double score, sds ele) {
return x;
}
/* Internal function used by zslDelete, zslDeleteByScore and zslDeleteByRank */
/* Internal function used by zslDelete, zslDeleteRangeByScore and
* zslDeleteRangeByRank. */
void zslDeleteNode(zskiplist *zsl, zskiplistNode *x, zskiplistNode **update) {
int i;
for (i = 0; i < zsl->level; i++) {
@ -1300,14 +1301,14 @@ int zsetScore(robj_roptr zobj, sds member, double *score) {
* none could be set if we re-added an element using the same score it used
* to have, or in the case a zero increment is used).
*
* The function returns 0 on erorr, currently only when the increment
* The function returns 0 on error, currently only when the increment
* produces a NAN condition, or when the 'score' value is NAN since the
* start.
*
* The commad as a side effect of adding a new element may convert the sorted
* The command as a side effect of adding a new element may convert the sorted
* set internal encoding from ziplist to hashtable+skiplist.
*
* Memory managemnet of 'ele':
* Memory management of 'ele':
*
* The function does not take ownership of the 'ele' SDS string, but copies
* it if needed. */
@ -1645,7 +1646,7 @@ reply_to_client:
cleanup:
zfree(scores);
if (added || updated) {
signalModifiedKey(c->db,key);
signalModifiedKey(c,c->db,key);
notifyKeyspaceEvent(NOTIFY_ZSET,
incr ? "zincr" : "zadd", key, c->db->id);
}
@ -1680,7 +1681,7 @@ void zremCommand(client *c) {
notifyKeyspaceEvent(NOTIFY_ZSET,"zrem",key,c->db->id);
if (keyremoved)
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",key,c->db->id);
signalModifiedKey(c->db,key);
signalModifiedKey(c,c->db,key);
g_pserver->dirty += deleted;
}
addReplyLongLong(c,deleted);
@ -1778,7 +1779,7 @@ void zremrangeGenericCommand(client *c, int rangetype) {
/* Step 4: Notifications and reply. */
if (deleted) {
const char *event[3] = {"zremrangebyrank","zremrangebyscore","zremrangebylex"};
signalModifiedKey(c->db,key);
signalModifiedKey(c,c->db,key);
notifyKeyspaceEvent(NOTIFY_ZSET,event[rangetype],key,c->db->id);
if (keyremoved)
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",key,c->db->id);
@ -2382,7 +2383,7 @@ void zunionInterGenericCommand(client *c, robj *dstkey, int op) {
zsetConvertToZiplistIfNeeded(dstobj,maxelelen);
dbAdd(c->db,dstkey,dstobj);
addReplyLongLong(c,zsetLength(dstobj));
signalModifiedKey(c->db,dstkey);
signalModifiedKey(c,c->db,dstkey);
notifyKeyspaceEvent(NOTIFY_ZSET,
(op == SET_OP_UNION) ? "zunionstore" : "zinterstore",
dstkey,c->db->id);
@ -2391,7 +2392,7 @@ void zunionInterGenericCommand(client *c, robj *dstkey, int op) {
decrRefCount(dstobj);
addReply(c,shared.czero);
if (touched) {
signalModifiedKey(c->db,dstkey);
signalModifiedKey(c,c->db,dstkey);
notifyKeyspaceEvent(NOTIFY_GENERIC,"del",dstkey,c->db->id);
g_pserver->dirty++;
}
@ -3215,7 +3216,7 @@ void genericZpopCommand(client *c, robj **keyv, int keyc, int where, int emitkey
if (arraylen == 0) { /* Do this only for the first iteration. */
const char *events[2] = {"zpopmin","zpopmax"};
notifyKeyspaceEvent(NOTIFY_ZSET,events[where],key,c->db->id);
signalModifiedKey(c->db,key);
signalModifiedKey(c,c->db,key);
}
addReplyBulkCBufferAsync(c,ele,sdslen(ele));

2
src/timeout.cpp
View File

@ -168,7 +168,7 @@ int getTimeoutFromObjectOrReply(client *c, robj *object, mstime_t *timeout, int
if (unit == UNIT_SECONDS) {
if (getLongDoubleFromObjectOrReply(c,object,&ftval,
"timeout is not an float or out of range") != C_OK)
"timeout is not a float or out of range") != C_OK)
return C_ERR;
tval = (long long) (ftval * 1000.0);
} else {

4
src/tls.cpp
View File

@ -160,7 +160,7 @@ int tlsConfigure(redisTLSContextConfig *ctx_config) {
#endif
#ifdef SSL_OP_NO_CLIENT_RENEGOTIATION
SSL_CTX_set_options(ssl->ctx, SSL_OP_NO_CLIENT_RENEGOTIATION);
SSL_CTX_set_options(ctx, SSL_OP_NO_CLIENT_RENEGOTIATION);
#endif
if (ctx_config->prefer_server_ciphers)
@ -168,7 +168,9 @@ int tlsConfigure(redisTLSContextConfig *ctx_config) {
SSL_CTX_set_mode(ctx, SSL_MODE_ENABLE_PARTIAL_WRITE|SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER);
SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER|SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL);
#if defined(SSL_CTX_set_ecdh_auto)
SSL_CTX_set_ecdh_auto(ctx, 1);
#endif
if (SSL_CTX_use_certificate_file(ctx, ctx_config->cert_file, SSL_FILETYPE_PEM) <= 0) {
ERR_error_string_n(ERR_get_error(), errbuf, sizeof(errbuf));

137
src/tracking.cpp
View File

@ -94,7 +94,8 @@ void disableTracking(client *c) {
g_pserver->tracking_clients--;
c->flags &= ~(CLIENT_TRACKING|CLIENT_TRACKING_BROKEN_REDIR|
CLIENT_TRACKING_BCAST|CLIENT_TRACKING_OPTIN|
CLIENT_TRACKING_OPTOUT|CLIENT_TRACKING_CACHING);
CLIENT_TRACKING_OPTOUT|CLIENT_TRACKING_CACHING|
CLIENT_TRACKING_NOLOOP);
}
}
@ -129,14 +130,19 @@ void enableTracking(client *c, uint64_t redirect_to, uint64_t options, robj **pr
if (!(c->flags & CLIENT_TRACKING)) g_pserver->tracking_clients++;
c->flags |= CLIENT_TRACKING;
c->flags &= ~(CLIENT_TRACKING_BROKEN_REDIR|CLIENT_TRACKING_BCAST|
CLIENT_TRACKING_OPTIN|CLIENT_TRACKING_OPTOUT);
CLIENT_TRACKING_OPTIN|CLIENT_TRACKING_OPTOUT|
CLIENT_TRACKING_NOLOOP);
c->client_tracking_redirection = redirect_to;
/* This may be the first client we ever enable. Crete the tracking
* table if it does not exist. */
if (TrackingTable == NULL) {
TrackingTable = raxNew();
PrefixTable = raxNew();
TrackingChannelName = createStringObject("__redis__:invalidate",20);
}
/* For broadcasting, set the list of prefixes in the client. */
if (options & CLIENT_TRACKING_BCAST) {
c->flags |= CLIENT_TRACKING_BCAST;
if (numprefix == 0) enableBcastTrackingForPrefix(c,"",0);
@ -145,7 +151,10 @@ void enableTracking(client *c, uint64_t redirect_to, uint64_t options, robj **pr
enableBcastTrackingForPrefix(c,sdsprefix,sdslen(sdsprefix));
}
}
c->flags |= options & (CLIENT_TRACKING_OPTIN|CLIENT_TRACKING_OPTOUT);
/* Set the remaining flags that don't need any special handling. */
c->flags |= options & (CLIENT_TRACKING_OPTIN|CLIENT_TRACKING_OPTOUT|
CLIENT_TRACKING_NOLOOP);
}
/* This function is called after the execution of a readonly command in the
@ -245,7 +254,7 @@ void sendTrackingMessage(client *c, const char *keyname, size_t keylen, int prot
* matches one or more prefixes in the prefix table. Later when we
* return to the event loop, we'll send invalidation messages to the
* clients subscribed to each prefix. */
void trackingRememberKeyToBroadcast(char *keyname, size_t keylen) {
void trackingRememberKeyToBroadcast(client *c, char *keyname, size_t keylen) {
raxIterator ri;
raxStart(&ri,PrefixTable);
raxSeek(&ri,"^",NULL,0);
@ -254,7 +263,11 @@ void trackingRememberKeyToBroadcast(char *keyname, size_t keylen) {
if (ri.key_len != 0 && memcmp(ri.key,keyname,ri.key_len) != 0)
continue;
bcastState *bs = (bcastState*)ri.data;
raxTryInsert(bs->keys,(unsigned char*)keyname,keylen,NULL,NULL);
/* We insert the client pointer as associated value in the radix
* tree. This way we know who was the client that did the last
* change to the key, and can avoid sending the notification in the
* case the client is in NOLOOP mode. */
raxTryInsert(bs->keys,(unsigned char*)keyname,keylen,c,NULL);
}
raxStop(&ri);
}
@ -262,13 +275,17 @@ void trackingRememberKeyToBroadcast(char *keyname, size_t keylen) {
/* This function is called from signalModifiedKey() or other places in Redis
* when a key changes value. In the context of keys tracking, our task here is
* to send a notification to every client that may have keys about such caching
* slot. */
void trackingInvalidateKey(robj *keyobj) {
* slot.
*
* Note that 'c' may be NULL in case the operation was performed outside the
* context of a client modifying the database (for instance when we delete a
* key because of expire). */
void trackingInvalidateKey(client *c, robj *keyobj) {
if (TrackingTable == NULL) return;
sds sdskey = szFromObj(keyobj);
if (raxSize(PrefixTable) > 0)
trackingRememberKeyToBroadcast(sdskey,sdslen(sdskey));
trackingRememberKeyToBroadcast(c,sdskey,sdslen(sdskey));
rax *ids = (rax*)raxFind(TrackingTable,(unsigned char*)sdskey,sdslen(sdskey));
if (ids == raxNotFound) return;
@ -279,19 +296,28 @@ void trackingInvalidateKey(robj *keyobj) {
while(raxNext(&ri)) {
uint64_t id;
memcpy(&id,ri.key,sizeof(id));
client *c = lookupClientByID(id);
client *target = lookupClientByID(id);
/* Note that if the client is in BCAST mode, we don't want to
* send invalidation messages that were pending in the case
* previously the client was not in BCAST mode. This can happen if
* TRACKING is enabled normally, and then the client switches to
* BCAST mode. */
if (c == NULL ||
!(c->flags & CLIENT_TRACKING)||
c->flags & CLIENT_TRACKING_BCAST)
if (target == NULL ||
!(target->flags & CLIENT_TRACKING)||
target->flags & CLIENT_TRACKING_BCAST)
{
continue;
}
sendTrackingMessage(c,sdskey,sdslen(sdskey),0);
/* If the client enabled the NOLOOP mode, don't send notifications
* about keys changed by the client itself. */
if (target->flags & CLIENT_TRACKING_NOLOOP &&
target == c)
{
continue;
}
sendTrackingMessage(target,sdskey,sdslen(sdskey),0);
}
raxStop(&ri);
@ -383,6 +409,54 @@ void trackingLimitUsedSlots(void) {
timeout_counter++;
}
/* Generate Redis protocol for an array containing all the key names
* in the 'keys' radix tree. If the client is not NULL, the list will not
* include keys that were modified the last time by this client, in order
* to implement the NOLOOP option.
*
* If the resultin array would be empty, NULL is returned instead. */
sds trackingBuildBroadcastReply(client *c, rax *keys) {
raxIterator ri;
uint64_t count;
if (c == NULL) {
count = raxSize(keys);
} else {
count = 0;
raxStart(&ri,keys);
raxSeek(&ri,"^",NULL,0);
while(raxNext(&ri)) {
if (ri.data != c) count++;
}
raxStop(&ri);
if (count == 0) return NULL;
}
/* Create the array reply with the list of keys once, then send
* it to all the clients subscribed to this prefix. */
char buf[32];
size_t len = ll2string(buf,sizeof(buf),count);
sds proto = sdsempty();
proto = sdsMakeRoomFor(proto,count*15);
proto = sdscatlen(proto,"*",1);
proto = sdscatlen(proto,buf,len);
proto = sdscatlen(proto,"\r\n",2);
raxStart(&ri,keys);
raxSeek(&ri,"^",NULL,0);
while(raxNext(&ri)) {
if (c && ri.data == c) continue;
len = ll2string(buf,sizeof(buf),ri.key_len);
proto = sdscatlen(proto,"$",1);
proto = sdscatlen(proto,buf,len);
proto = sdscatlen(proto,"\r\n",2);
proto = sdscatlen(proto,ri.key,ri.key_len);
proto = sdscatlen(proto,"\r\n",2);
}
raxStop(&ri);
return proto;
}
/* This function will run the prefixes of clients in BCAST mode and
* keys that were modified about each prefix, and will send the
* notifications to each client in each prefix. */
@ -394,29 +468,15 @@ void trackingBroadcastInvalidationMessages(void) {
raxStart(&ri,PrefixTable);
raxSeek(&ri,"^",NULL,0);
/* For each prefix... */
while(raxNext(&ri)) {
bcastState *bs = (bcastState*)ri.data;
if (raxSize(bs->keys)) {
/* Create the array reply with the list of keys once, then send
* it to all the clients subscribed to this prefix. */
char buf[32];
size_t len = ll2string(buf,sizeof(buf),raxSize(bs->keys));
sds proto = sdsempty();
proto = sdsMakeRoomFor(proto,raxSize(bs->keys)*15);
proto = sdscatlen(proto,"*",1);
proto = sdscatlen(proto,buf,len);
proto = sdscatlen(proto,"\r\n",2);
raxStart(&ri2,bs->keys);
raxSeek(&ri2,"^",NULL,0);
while(raxNext(&ri2)) {
len = ll2string(buf,sizeof(buf),ri2.key_len);
proto = sdscatlen(proto,"$",1);
proto = sdscatlen(proto,buf,len);
proto = sdscatlen(proto,"\r\n",2);
proto = sdscatlen(proto,ri2.key,ri2.key_len);
proto = sdscatlen(proto,"\r\n",2);
}
raxStop(&ri2);
/* Generate the common protocol for all the clients that are
* not using the NOLOOP option. */
sds proto = trackingBuildBroadcastReply(NULL,bs->keys);
/* Send this array of keys to every client in the list. */
raxStart(&ri2,bs->clients);
@ -424,7 +484,16 @@ void trackingBroadcastInvalidationMessages(void) {
while(raxNext(&ri2)) {
client *c;
memcpy(&c,ri2.key,sizeof(c));
sendTrackingMessage(c,proto,sdslen(proto),1);
if (c->flags & CLIENT_TRACKING_NOLOOP) {
/* This client may have certain keys excluded. */
sds adhoc = trackingBuildBroadcastReply(c,bs->keys);
if (adhoc) {
sendTrackingMessage(c,adhoc,sdslen(adhoc),1);
sdsfree(adhoc);
}
} else {
sendTrackingMessage(c,proto,sdslen(proto),1);
}
}
raxStop(&ri2);

38
src/util.c
View File

@ -42,7 +42,7 @@
#include <time.h>
#include "util.h"
#include "sha1.h"
#include "sha256.h"
/* Glob-style pattern matching. */
int stringmatchlen(const char *pattern, int patternLen,
@ -622,7 +622,7 @@ int ld2string(char *buf, size_t len, long double value, ld2string_mode mode) {
void getRandomBytes(unsigned char *p, size_t len) {
/* Global state. */
static int seed_initialized = 0;
static unsigned char seed[20]; /* The SHA1 seed, from /dev/urandom. */
static unsigned char seed[64]; /* 512 bit internal block size. */
static uint64_t counter = 0; /* The counter we hash with the seed. */
if (!seed_initialized) {
@ -647,14 +647,34 @@ void getRandomBytes(unsigned char *p, size_t len) {
}
while(len) {
unsigned char digest[20];
SHA1_CTX ctx;
unsigned int copylen = len > 20 ? 20 : len;
/* This implements SHA256-HMAC. */
unsigned char digest[SHA256_BLOCK_SIZE];
unsigned char kxor[64];
unsigned int copylen =
len > SHA256_BLOCK_SIZE ? SHA256_BLOCK_SIZE : len;
SHA1Init(&ctx);
SHA1Update(&ctx, seed, sizeof(seed));
SHA1Update(&ctx, (unsigned char*)&counter,sizeof(counter));
SHA1Final(digest, &ctx);
/* IKEY: key xored with 0x36. */
memcpy(kxor,seed,sizeof(kxor));
for (unsigned int i = 0; i < sizeof(kxor); i++) kxor[i] ^= 0x36;
/* Obtain HASH(IKEY||MESSAGE). */
SHA256_CTX ctx;
sha256_init(&ctx);
sha256_update(&ctx,kxor,sizeof(kxor));
sha256_update(&ctx,(unsigned char*)&counter,sizeof(counter));
sha256_final(&ctx,digest);
/* OKEY: key xored with 0x5c. */
memcpy(kxor,seed,sizeof(kxor));
for (unsigned int i = 0; i < sizeof(kxor); i++) kxor[i] ^= 0x5C;
/* Obtain HASH(OKEY || HASH(IKEY||MESSAGE)). */
sha256_init(&ctx);
sha256_update(&ctx,kxor,sizeof(kxor));
sha256_update(&ctx,digest,SHA256_BLOCK_SIZE);
sha256_final(&ctx,digest);
/* Increment the counter for the next iteration. */
counter++;
memcpy(p,digest,copylen);

4
src/zmalloc.h
View File

@ -88,11 +88,13 @@ extern "C" {
#ifdef __cplusplus
void *zmalloc(size_t size, enum MALLOC_CLASS mclass = MALLOC_LOCAL);
void *zcalloc(size_t size, enum MALLOC_CLASS mclass = MALLOC_LOCAL);
void *zrealloc(void *ptr, size_t size, enum MALLOC_CLASS mclass = MALLOC_LOCAL);
#else
void *zmalloc(size_t size, enum MALLOC_CLASS mclass);
#endif
void *zcalloc(size_t size, enum MALLOC_CLASS mclass);
void *zrealloc(void *ptr, size_t size, enum MALLOC_CLASS mclass);
#endif
void zfree(const void *ptr);
char *zstrdup(const char *s);
size_t zmalloc_used_memory(void);

44
tests/cluster/tests/15-cluster-slots.tcl Normal file
View File

@ -0,0 +1,44 @@
source "../tests/includes/init-tests.tcl"
proc cluster_allocate_mixedSlots {n} {
set slot 16383
while {$slot >= 0} {
set node [expr {$slot % $n}]
lappend slots_$node $slot
incr slot -1
}
for {set j 0} {$j < $n} {incr j} {
R $j cluster addslots {*}[set slots_${j}]
}
}
proc create_cluster_with_mixedSlot {masters slaves} {
cluster_allocate_mixedSlots $masters
if {$slaves} {
cluster_allocate_slaves $masters $slaves
}
assert_cluster_state ok
}
test "Create a 5 nodes cluster" {
create_cluster_with_mixedSlot 5 15
}
test "Cluster is up" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 0
}
test "Instance #5 is a slave" {
assert {[RI 5 role] eq {slave}}
}
test "client do not break when cluster slot" {
R 0 config set client-output-buffer-limit "normal 33554432 16777216 60"
if { [catch {R 0 cluster slots}] } {
fail "output overflow when cluster slots"
}
}

1
tests/integration/psync2-pingoff.tcl
View File

@ -20,6 +20,7 @@ start_server {} {
$R(1) replicaof $R_host(0) $R_port(0)
$R(0) set foo bar
wait_for_condition 50 1000 {
[status $R(1) master_link_status] == "up" &&
[$R(0) dbsize] == 1 && [$R(1) dbsize] == 1
} else {
fail "Replicas not replicating from master"

5
tests/integration/psync2-reg.tcl
View File

@ -28,7 +28,10 @@ start_server {} {
$R(2) slaveof $R_host(0) $R_port(0)
$R(0) set foo bar
wait_for_condition 50 1000 {
[$R(1) dbsize] == 1 && [$R(2) dbsize] == 1
[status $R(1) master_link_status] == "up" &&
[status $R(2) master_link_status] == "up" &&
[$R(1) dbsize] == 1 &&
[$R(2) dbsize] == 1
} else {
fail "Replicas not replicating from master"
}

156
tests/integration/psync2.tcl
View File

@ -44,6 +44,7 @@ start_server {} {
set used [list $master_id]
test "PSYNC2: \[NEW LAYOUT\] Set #$master_id as master" {
$R($master_id) slaveof no one
$R($master_id) config set repl-ping-replica-period 1 ;# increse the chance that random ping will cause issues
if {$counter_value == 0} {
$R($master_id) set x $counter_value
}
@ -66,6 +67,16 @@ start_server {} {
lappend used $slave_id
}
# Wait for replicas to sync. so next loop won't get -LOADING error
wait_for_condition 50 1000 {
[status $R([expr {($master_id+1)%5}]) master_link_status] == "up" &&
[status $R([expr {($master_id+2)%5}]) master_link_status] == "up" &&
[status $R([expr {($master_id+3)%5}]) master_link_status] == "up" &&
[status $R([expr {($master_id+4)%5}]) master_link_status] == "up"
} else {
fail "Replica not reconnecting"
}
# 3) Increment the counter and wait for all the instances
# to converge.
test "PSYNC2: cluster is consistent after failover" {
@ -114,23 +125,20 @@ start_server {} {
}
}
# wait for all the slaves to be in sync with the master
set master_ofs [status $R($master_id) master_repl_offset]
# wait for all the slaves to be in sync with the master, due to pings, we have to re-sample the master constantly too
wait_for_condition 500 100 {
$master_ofs == [status $R(0) master_repl_offset] &&
$master_ofs == [status $R(1) master_repl_offset] &&
$master_ofs == [status $R(2) master_repl_offset] &&
$master_ofs == [status $R(3) master_repl_offset] &&
$master_ofs == [status $R(4) master_repl_offset]
[status $R($master_id) master_repl_offset] == [status $R(0) master_repl_offset] &&
[status $R($master_id) master_repl_offset] == [status $R(1) master_repl_offset] &&
[status $R($master_id) master_repl_offset] == [status $R(2) master_repl_offset] &&
[status $R($master_id) master_repl_offset] == [status $R(3) master_repl_offset] &&
[status $R($master_id) master_repl_offset] == [status $R(4) master_repl_offset]
} else {
if {$debug_msg} {
for {set j 0} {$j < 5} {incr j} {
puts "$j: sync_full: [status $R($j) sync_full]"
puts "$j: id1 : [status $R($j) master_replid]:[status $R($j) master_repl_offset]"
puts "$j: id2 : [status $R($j) master_replid2]:[status $R($j) second_repl_offset]"
puts "$j: backlog : firstbyte=[status $R($j) repl_backlog_first_byte_offset] len=[status $R($j) repl_backlog_histlen]"
puts "---"
}
for {set j 0} {$j < 5} {incr j} {
puts "$j: sync_full: [status $R($j) sync_full]"
puts "$j: id1 : [status $R($j) master_replid]:[status $R($j) master_repl_offset]"
puts "$j: id2 : [status $R($j) master_replid2]:[status $R($j) second_repl_offset]"
puts "$j: backlog : firstbyte=[status $R($j) repl_backlog_first_byte_offset] len=[status $R($j) repl_backlog_histlen]"
puts "---"
}
fail "Slaves are not in sync with the master after too long time."
}
@ -175,9 +183,14 @@ start_server {} {
$R($j) slaveof $master_host $master_port
}
# Wait for slaves to sync
wait_for_condition 50 2000 {
[status $R($master_id) connected_slaves] == 4
# Wait for replicas to sync. it is not enough to just wait for connected_slaves==4
# since we might do the check before the master realized that they're disconnected
wait_for_condition 50 1000 {
[status $R($master_id) connected_slaves] == 4 &&
[status $R([expr {($master_id+1)%5}]) master_link_status] == "up" &&
[status $R([expr {($master_id+2)%5}]) master_link_status] == "up" &&
[status $R([expr {($master_id+3)%5}]) master_link_status] == "up" &&
[status $R([expr {($master_id+4)%5}]) master_link_status] == "up"
} else {
fail "Replica not reconnecting"
}
@ -188,6 +201,7 @@ start_server {} {
set slave_id [expr {($master_id+1)%5}]
set sync_count [status $R($master_id) sync_full]
set sync_partial [status $R($master_id) sync_partial_ok]
set sync_partial_err [status $R($master_id) sync_partial_err]
catch {
$R($slave_id) config rewrite
$R($slave_id) debug restart
@ -197,7 +211,11 @@ start_server {} {
wait_for_condition 50 2000 {
[status $R($master_id) sync_partial_ok] == $sync_partial + 1
} else {
fail "Replica not reconnecting"
puts "prev sync_full: $sync_count"
puts "prev sync_partial_ok: $sync_partial"
puts "prev sync_partial_err: $sync_partial_err"
puts [$R($master_id) info stats]
fail "Replica didn't partial sync"
}
set new_sync_count [status $R($master_id) sync_full]
assert {$sync_count == $new_sync_count}
@ -271,3 +289,103 @@ start_server {} {
}
}}}}}
start_server {tags {"psync2"}} {
start_server {} {
start_server {} {
start_server {} {
start_server {} {
test {pings at the end of replication stream are ignored for psync} {
set master [srv -4 client]
set master_host [srv -4 host]
set master_port [srv -4 port]
set replica1 [srv -3 client]
set replica2 [srv -2 client]
set replica3 [srv -1 client]
set replica4 [srv -0 client]
$replica1 replicaof $master_host $master_port
$replica2 replicaof $master_host $master_port
$replica3 replicaof $master_host $master_port
$replica4 replicaof $master_host $master_port
wait_for_condition 50 1000 {
[status $master connected_slaves] == 4
} else {
fail "replicas didn't connect"
}
$master incr x
wait_for_condition 50 1000 {
[$replica1 get x] == 1 && [$replica2 get x] == 1 &&
[$replica3 get x] == 1 && [$replica4 get x] == 1
} else {
fail "replicas didn't get incr"
}
# disconnect replica1 and replica2
# and wait for the master to send a ping to replica3 and replica4
$replica1 replicaof no one
$replica2 replicaof 127.0.0.1 1 ;# we can't promote it to master since that will cycle the replication id
$master config set repl-ping-replica-period 1
after 1500
# make everyone sync from the replica1 that didn't get the last ping from the old master
# replica4 will keep syncing from the old master which now syncs from replica1
# and replica2 will re-connect to the old master (which went back in time)
set new_master_host [srv -3 host]
set new_master_port [srv -3 port]
$replica3 replicaof $new_master_host $new_master_port
$master replicaof $new_master_host $new_master_port
$replica2 replicaof $master_host $master_port
wait_for_condition 50 1000 {
[status $replica2 master_link_status] == "up" &&
[status $replica3 master_link_status] == "up" &&
[status $replica4 master_link_status] == "up" &&
[status $master master_link_status] == "up"
} else {
fail "replicas didn't connect"
}
# make sure replication is still alive and kicking
$replica1 incr x
wait_for_condition 50 1000 {
[$replica2 get x] == 2 &&
[$replica3 get x] == 2 &&
[$replica4 get x] == 2 &&
[$master get x] == 2
} else {
fail "replicas didn't get incr"
}
# make sure there are full syncs other than the initial ones
assert_equal [status $master sync_full] 4
assert_equal [status $replica1 sync_full] 0
assert_equal [status $replica2 sync_full] 0
assert_equal [status $replica3 sync_full] 0
assert_equal [status $replica4 sync_full] 0
# force psync
$master client kill type master
$replica2 client kill type master
$replica3 client kill type master
$replica4 client kill type master
# make sure replication is still alive and kicking
$replica1 incr x
wait_for_condition 50 1000 {
[$replica2 get x] == 3 &&
[$replica3 get x] == 3 &&
[$replica4 get x] == 3 &&
[$master get x] == 3
} else {
fail "replicas didn't get incr"
}
# make sure there are full syncs other than the initial ones
assert_equal [status $master sync_full] 4
assert_equal [status $replica1 sync_full] 0
assert_equal [status $replica2 sync_full] 0
assert_equal [status $replica3 sync_full] 0
assert_equal [status $replica4 sync_full] 0
}
}}}}}

51
tests/test_helper.tcl
View File

@ -87,6 +87,7 @@ set ::skiptests {}
set ::allowtags {}
set ::only_tests {}
set ::single_tests {}
set ::run_solo_tests {}
set ::skip_till ""
set ::external 0; # If "1" this means, we are running against external instance
set ::file ""; # If set, runs only the tests in this comma separated list
@ -110,13 +111,27 @@ set ::tlsdir "tests/tls"
set ::client 0
set ::numclients 16
proc execute_tests name {
# This function is called by one of the test clients when it receives
# a "run" command from the server, with a filename as data.
# It will run the specified test source file and signal it to the
# test server when finished.
proc execute_test_file name {
set path "tests/$name.tcl"
set ::curfile $path
source $path
send_data_packet $::test_server_fd done "$name"
}
# This function is called by one of the test clients when it receives
# a "run_code" command from the server, with a verbatim test source code
# as argument, and an associated name.
# It will run the specified code and signal it to the test server when
# finished.
proc execute_test_code {name code} {
eval $code
send_data_packet $::test_server_fd done "$name"
}
# Setup a list to hold a stack of server configs. When calls to start_server
# are nested, use "srv 0 pid" to get the pid of the inner server. To access
# outer servers, use "srv -1 pid" etcetera.
@ -193,6 +208,18 @@ proc s {args} {
status [srv $level "client"] [lindex $args 0]
}
# Test wrapped into run_solo are sent back from the client to the
# test server, so that the test server will send them again to
# clients once the clients are idle.
proc run_solo {name code} {
if {$::numclients == 1 || $::loop || $::external} {
# run_solo is not supported in these scenarios, just run the code.
eval $code
return
}
send_data_packet $::test_server_fd run_solo [list $name $code]
}
proc cleanup {} {
if {$::dont_clean} {
return
@ -342,6 +369,8 @@ proc read_from_test_client fd {
} elseif {$status eq {server-killed}} {
set ::active_servers [lsearch -all -inline -not -exact $::active_servers $data]
set ::active_clients_task($fd) "(KILLED SERVER) pid:$data"
} elseif {$status eq {run_solo}} {
lappend ::run_solo_tests $data
} else {
if {!$::quiet} {
puts "\[$status\]: $data"
@ -374,6 +403,13 @@ proc force_kill_all_servers {} {
}
}
proc lpop {listVar {count 1}} {
upvar 1 $listVar l
set ele [lindex $l 0]
set l [lrange $l 1 end]
set ele
}
# A new client is idle. Remove it from the list of active clients and
# if there are still test units to run, launch them.
proc signal_idle_client fd {
@ -394,6 +430,14 @@ proc signal_idle_client fd {
if {$::loop && $::next_test == [llength $::all_tests]} {
set ::next_test 0
}
} elseif {[llength $::run_solo_tests] != 0 && [llength $::active_clients] == 0} {
if {!$::quiet} {
puts [colorstr bold-white "Testing solo test"]
set ::active_clients_task($fd) "ASSIGNED: $fd solo test"
}
set ::clients_start_time($fd) [clock seconds]
send_data_packet $fd run_code [lpop ::run_solo_tests]
lappend ::active_clients $fd
} else {
lappend ::idle_clients $fd
set ::active_clients_task($fd) "SLEEPING, no more units to assign"
@ -437,7 +481,10 @@ proc test_client_main server_port {
set payload [read $::test_server_fd $bytes]
foreach {cmd data} $payload break
if {$cmd eq {run}} {
execute_tests $data
execute_test_file $data
} elseif {$cmd eq {run_code}} {
foreach {name code} $data break
execute_test_code $name $code
} else {
error "Unknown test client command: $cmd"
}

2
tests/unit/memefficiency.tcl
View File

@ -36,6 +36,7 @@ start_server {tags {"memefficiency"}} {
}
}
run_solo {defrag} {
start_server {tags {"defrag"}} {
if {[string match {*jemalloc*} [s mem_allocator]]} {
test "Active defrag" {
@ -328,3 +329,4 @@ start_server {tags {"defrag"}} {
} {1}
}
}
} ;# run_solo

45
tests/unit/tracking.tcl
View File

@ -7,6 +7,9 @@ start_server {tags {"tracking"}} {
$rd1 subscribe __redis__:invalidate
$rd1 read ; # Consume the SUBSCRIBE reply.
# Create another client as well in order to test NOLOOP
set rd2 [redis_deferring_client]
test {Clients are able to enable tracking and redirect it} {
r CLIENT TRACKING on REDIRECT $redir
} {*OK}
@ -62,5 +65,47 @@ start_server {tags {"tracking"}} {
assert {$keys eq {c:1234}}
}
test {Tracking NOLOOP mode in standard mode works} {
r CLIENT TRACKING off
r CLIENT TRACKING on REDIRECT $redir NOLOOP
r MGET otherkey1 loopkey otherkey2
$rd2 SET otherkey1 1; # We should get this
r SET loopkey 1 ; # We should not get this
$rd2 SET otherkey2 1; # We should get this
# Because of the internals, we know we are going to receive
# two separated notifications for the two different prefixes.
set keys1 [lsort [lindex [$rd1 read] 2]]
set keys2 [lsort [lindex [$rd1 read] 2]]
set keys [lsort [list {*}$keys1 {*}$keys2]]
assert {$keys eq {otherkey1 otherkey2}}
}
test {Tracking NOLOOP mode in BCAST mode works} {
r CLIENT TRACKING off
r CLIENT TRACKING on BCAST REDIRECT $redir NOLOOP
$rd2 SET otherkey1 1; # We should get this
r SET loopkey 1 ; # We should not get this
$rd2 SET otherkey2 1; # We should get this
# Because of the internals, we know we are going to receive
# two separated notifications for the two different prefixes.
set keys1 [lsort [lindex [$rd1 read] 2]]
set keys2 [lsort [lindex [$rd1 read] 2]]
set keys [lsort [list {*}$keys1 {*}$keys2]]
assert {$keys eq {otherkey1 otherkey2}}
}
test {Tracking gets notification of expired keys} {
r CLIENT TRACKING off
r CLIENT TRACKING on BCAST REDIRECT $redir NOLOOP
r SET mykey myval px 1
r SET mykeyotherkey myval ; # We should not get it
after 1000
# Because of the internals, we know we are going to receive
# two separated notifications for the two different prefixes.
set keys1 [lsort [lindex [$rd1 read] 2]]
set keys [lsort [list {*}$keys1]]
assert {$keys eq {mykey}}
}
$rd1 close
}

34
tests/unit/type/stream-cgroups.tcl
View File

@ -294,6 +294,40 @@ start_server {
assert {[lindex $reply 0 3] == 2}
}
test {XINFO FULL output} {
r del x
r XADD x 100 a 1
r XADD x 101 b 1
r XADD x 102 c 1
r XADD x 103 e 1
r XADD x 104 f 1
r XGROUP CREATE x g1 0
r XGROUP CREATE x g2 0
r XREADGROUP GROUP g1 Alice COUNT 1 STREAMS x >
r XREADGROUP GROUP g1 Bob COUNT 1 STREAMS x >
r XREADGROUP GROUP g1 Bob NOACK COUNT 1 STREAMS x >
r XREADGROUP GROUP g2 Charlie COUNT 4 STREAMS x >
r XDEL x 103
set reply [r XINFO STREAM x FULL]
assert_equal [llength $reply] 12
assert_equal [lindex $reply 1] 4 ;# stream length
assert_equal [lindex $reply 9] "{100-0 {a 1}} {101-0 {b 1}} {102-0 {c 1}} {104-0 {f 1}}" ;# entries
assert_equal [lindex $reply 11 0 1] "g1" ;# first group name
assert_equal [lindex $reply 11 0 7 0 0] "100-0" ;# first entry in group's PEL
assert_equal [lindex $reply 11 0 9 0 1] "Alice" ;# first consumer
assert_equal [lindex $reply 11 0 9 0 7 0 0] "100-0" ;# first entry in first consumer's PEL
assert_equal [lindex $reply 11 1 1] "g2" ;# second group name
assert_equal [lindex $reply 11 1 9 0 1] "Charlie" ;# first consumer
assert_equal [lindex $reply 11 1 9 0 7 0 0] "100-0" ;# first entry in first consumer's PEL
assert_equal [lindex $reply 11 1 9 0 7 1 0] "101-0" ;# second entry in first consumer's PEL
set reply [r XINFO STREAM x FULL COUNT 1]
assert_equal [llength $reply] 12
assert_equal [lindex $reply 1] 4
assert_equal [lindex $reply 9] "{100-0 {a 1}}"
}
start_server {} {
set master [srv -1 client]
set master_host [srv -1 host]

2
tests/unit/type/stream.tcl
View File

@ -414,7 +414,7 @@ start_server {tags {"stream"} overrides {appendonly yes stream-node-max-entries
}
}
start_server {tags {"xsetid"}} {
start_server {tags {"stream xsetid"}} {
test {XADD can CREATE an empty stream} {
r XADD mystream MAXLEN 0 * a b
assert {[dict get [r xinfo stream mystream] length] == 0}

16
tests/unit/type/string.tcl
View File

@ -424,29 +424,29 @@ start_server {tags {"string"}} {
set rna2 {ATTAAAGGTTTATACCTTCCCAGGTAACAAACCAACCAACTTTCGATCTCTTGTAGATCTGTTCTCTAAACGAACTTTAAAATCTGTGTGGCTGTCACTCGGCTGCATGCTTAGTGCACTCACGCAGTATAATTAATAACTAATTACTGTCGTTGACAGGACACGAGTAACTCGTCTATCTTCTGCAGGCTGCTTACGGTTTCGTCCGTGTTGCAGCCGATCATCAGCACATCTAGGTTT}
set rnalcs {ACCTTCCCAGGTAACAAACCAACCAACTTTCGATCTCTTGTAGATCTGTTCTCTAAACGAACTTTAAAATCTGTGTGGCTGTCACTCGGCTGCATGCTTAGTGCACTCACGCAGTATAATTAATAACTAATTACTGTCGTTGACAGGACACGAGTAACTCGTCTATCTTCTGCAGGCTGCTTACGGTTTCGTCCGTGTTGCAGCCGATCATCAGCACATCTAGGTTT}
test {LCS string output with STRINGS option} {
r LCS STRINGS $rna1 $rna2
test {STRALGO LCS string output with STRINGS option} {
r STRALGO LCS STRINGS $rna1 $rna2
} $rnalcs
test {LCS len} {
r LCS LEN STRINGS $rna1 $rna2
test {STRALGO LCS len} {
r STRALGO LCS LEN STRINGS $rna1 $rna2
} [string length $rnalcs]
test {LCS with KEYS option} {
r set virus1 $rna1
r set virus2 $rna2
r LCS KEYS virus1 virus2
r STRALGO LCS KEYS virus1 virus2
} $rnalcs
test {LCS indexes} {
dict get [r LCS IDX KEYS virus1 virus2] matches
dict get [r STRALGO LCS IDX KEYS virus1 virus2] matches
} {{{238 238} {239 239}} {{236 236} {238 238}} {{229 230} {236 237}} {{224 224} {235 235}} {{1 222} {13 234}}}
test {LCS indexes with match len} {
dict get [r LCS IDX KEYS virus1 virus2 WITHMATCHLEN] matches
dict get [r STRALGO LCS IDX KEYS virus1 virus2 WITHMATCHLEN] matches
} {{{238 238} {239 239} 1} {{236 236} {238 238} 1} {{229 230} {236 237} 2} {{224 224} {235 235} 1} {{1 222} {13 234} 222}}
test {LCS indexes with match len and minimum match len} {
dict get [r LCS IDX KEYS virus1 virus2 WITHMATCHLEN MINMATCHLEN 5] matches
dict get [r STRALGO LCS IDX KEYS virus1 virus2 WITHMATCHLEN MINMATCHLEN 5] matches
} {{{1 222} {13 234} 222}}
}

14
utils/create-cluster/create-cluster
View File

@ -1,6 +1,7 @@
#!/bin/bash
# Settings
BIN_PATH="../../src/"
CLUSTER_HOST=127.0.0.1
PORT=30000
TIMEOUT=2000
@ -25,7 +26,7 @@ then
while [ $((PORT < ENDPORT)) != "0" ]; do
PORT=$((PORT+1))
echo "Starting $PORT"
../../src/keydb-server --port $PORT --protected-mode $PROTECTED_MODE --cluster-enabled yes --cluster-config-file nodes-${PORT}.conf --cluster-node-timeout $TIMEOUT --appendonly yes --appendfilename appendonly-${PORT}.aof --dbfilename dump-${PORT}.rdb --logfile ${PORT}.log --daemonize yes ${ADDITIONAL_OPTIONS}
$BIN_PATH/keydb-server --port $PORT --protected-mode $PROTECTED_MODE --cluster-enabled yes --cluster-config-file nodes-${PORT}.conf --cluster-node-timeout $TIMEOUT --appendonly yes --appendfilename appendonly-${PORT}.aof --dbfilename dump-${PORT}.rdb --logfile ${PORT}.log --daemonize yes ${ADDITIONAL_OPTIONS}
done
exit 0
fi
@ -37,7 +38,7 @@ then
PORT=$((PORT+1))
HOSTS="$HOSTS $CLUSTER_HOST:$PORT"
done
../../src/keydb-cli --cluster create $HOSTS --cluster-replicas $REPLICAS
$BIN_PATH/keydb-cli --cluster create $HOSTS --cluster-replicas $REPLICAS
exit 0
fi
@ -46,7 +47,7 @@ then
while [ $((PORT < ENDPORT)) != "0" ]; do
PORT=$((PORT+1))
echo "Stopping $PORT"
../../src/keydb-cli -p $PORT shutdown nosave
$BIN_PATH/keydb-cli -p $PORT shutdown nosave
done
exit 0
fi
@ -57,7 +58,7 @@ then
while [ 1 ]; do
clear
date
../../src/keydb-cli -p $PORT cluster nodes | head -30
$BIN_PATH/keydb-cli -p $PORT cluster nodes | head -30
sleep 1
done
exit 0
@ -81,7 +82,7 @@ if [ "$1" == "call" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
PORT=$((PORT+1))
../../src/keydb-cli -p $PORT $2 $3 $4 $5 $6 $7 $8 $9
$BIN_PATH/keydb-cli -p $PORT $2 $3 $4 $5 $6 $7 $8 $9
done
exit 0
fi
@ -101,7 +102,7 @@ then
exit 0
fi
echo "Usage: $0 [start|create|stop|watch|tail|clean]"
echo "Usage: $0 [start|create|stop|watch|tail|clean|call]"
echo "start -- Launch Redis Cluster instances."
echo "create -- Create a cluster using keydb-cli --cluster create."
echo "stop -- Stop Redis Cluster instances."
@ -110,3 +111,4 @@ echo "tail <id> -- Run tail -f of instance at base port + ID."
echo "tailall -- Run tail -f for all the log files at once."
echo "clean -- Remove all instances data, logs, configs."
echo "clean-logs -- Remove just instances logs."
echo "call <cmd> -- Call a command (up to 7 arguments) on all nodes."