10909 Commits

Author SHA1 Message Date
John Sully
211b0e5f80 Avoid posting unnecessary async tasks
Former-commit-id: 1661d2a05943f96992be195f5dc32dd9a67c0e68
2021-04-12 03:44:29 +00:00
John Sully
ea2bbfb5c1 Fix issue where GC is not free'd until a BGSAVE
Former-commit-id: 38523e6b508cf5f4b40c178dfe98554abea8f6bd
2021-04-12 03:44:15 +00:00
John Sully
14250faddd Make prefetch more aggressive
Former-commit-id: 25a5cfefcf7fa3451e92500f5d310290c4b6bbde
2021-04-12 03:42:05 +00:00
John Sully
1b121723e2 Don't prefetch when lock contention is low, it increases latency
Former-commit-id: 0d21614e0e5aba28acd364231823d51a3073081f
2021-04-12 03:41:53 +00:00
John Sully
e6782d8f1b Reduce P99 latency with async rehash
Former-commit-id: 7ea4c26fd82c0cdfa377183083f38a43336e480b
2021-04-12 03:39:13 +00:00
John Sully
df66c598e9 Make prefetch configurable
Former-commit-id: 3b660347d70cc25d57119080bd43fb4671e36488
2021-04-12 03:38:57 +00:00
Wang Yuan
a0e19e3cf1
Fix wrong check for aof fsync and handle aof fsync errno (#8751)
The bio aof fsync fd may be closed by main thread (AOFRW done handler)
and even possibly reused for another socket, pipe, or file.
This can can an EBADF or EINVAL fsync error, which will lead to -MISCONF errors failing all writes.
We just ignore these errno because aof fsync did not really fail.

We handle errno when fsyncing aof in bio, so we could know the real reason
when users get -MISCONF Errors writing to the AOF file error

Issue created with #8419
2021-04-11 08:14:31 +03:00
yjph
cd03e293c3
Fix the display of make install (#8667) 2021-04-10 21:25:53 +03:00
John Sully
d58a415e29 DO not set the timethread as high priority, it can starve the server threads
Former-commit-id: faeac65371af9d6b6effe0886bcbdefaec24ad6d
2021-04-09 01:06:24 +00:00
Wen Hui
d96f47cf06
use getPositiveLongFromObjectOrReply for positive check of args (#8750)
just a cleanup
2021-04-07 10:28:53 +03:00
Yang Bodong
4c14e8668c
Fix out of range confusing error messages (XAUTOCLAIM, RPOP count) (#8746)
Fix out of range error messages to be clearer (avoid mentioning 9223372036854775807)
* Fix XAUTOCLAIM COUNT option confusing error msg
* Fix other RPOP and alike error message to mention positive
2021-04-07 10:01:28 +03:00
Oran Agra
4de1031d8a
update help.h (#8747) 2021-04-06 12:42:18 +03:00
yjph
073598ed8f
Update the location information in some URLs (#8595) 2021-04-06 12:29:02 +03:00
Bonsai
808f3004f0
Fix: server will crash if rdbload or rdbsave method is not provided in module (#8670)
With this fix, module data type registration will fail if the load or save callbacks are not defined, or the optional aux load and save callbacks are not either both defined or both missing.
2021-04-06 12:09:36 +03:00
Huang Zhw
3b74b55084
Fix "default" and overwritten / reset users will not have pubsub channels permissions by default. (#8723)
Background:
Redis 6.2 added ACL control for pubsub channels (#7993), which were supposed
to be permissive by default to retain compatibility with redis 6.0 ACL. 
But due to a bug, only newly created users got this `acl-pubsub-default` applied,
while overwritten (updated) users got reset to `resetchannels` (denied).

Since the "default" user exists before loading the config file,
any ACL change to it, results in an update / overwrite.

So when a "default" user is loaded from config file or include ACL
file with no channels related rules, the user will not have any
permissions to any channels. But other users will have default
permissions to any channels.

When upgraded from 6.0 with config rewrite, this will lead to
"default" user channels permissions lost.
When users are loaded from include file, then call "acl load", users
will also lost channels permissions.

Similarly, the `reset` ACL rule, would have reset the user to be denied
access to any channels, ignoring `acl-pubsub-default` and breaking
compatibility with redis 6.0.

The implication of this fix is that it regains compatibility with redis 6.0,
but breaks compatibility with redis 6.2.0 and 2.0.1. e.g. after the upgrade,
the default user will regain access to pubsub channels.

Other changes:
Additionally this commit rename server.acl_pubusub_default to
server.acl_pubsub_default and fix typo in acl tests.
2021-04-05 23:13:20 +03:00
Huang Zhw
a3da3e592d
redis-cli --bigkeys / memkeys, report detailed error on dbsize failure (#8740)
When DBSIZE failed (e.g. on AUTH error), the printed error didn't reflect the reason.
2021-04-05 08:30:41 +03:00
Sokolov Yura
1cab962098
Add cluster-allow-replica-migration option. (#5285)
Previously (and by default after commit) when master loose its last slot
(due to migration, for example), its replicas will migrate to new last slot
holder.

There are cases where this is not desired:
* Consolidation that results with removed nodes (including the replica, eventually).
* Manually configured cluster topologies, which the admin wishes to preserve.

Needlessly migrating a replica triggers a full synchronization and can have a negative impact, so
we prefer to be able to avoid it where possible.

This commit adds 'cluster-allow-replica-migration' configuration option that is
enabled by default to preserve existed behavior. When disabled, replicas will
not be auto-migrated.

Fixes #4896

Co-authored-by: Oran Agra <oran@redislabs.com>
2021-04-04 09:43:24 +03:00
Wang Yuan
1eb85249e7
Handle remaining fsync errors (#8419)
In `aof.c`, we call fsync when stop aof, and now print a log to let user know that if fail.
In `cluster.c`, we now return error, the calling function already handles these write errors.
In `redis-cli.c`, users hope to save rdb, we now print a message if fsync failed.
In `rio.c`, we now treat fsync errors like we do for write errors. 
In `server.c`, we try to fsync aof file when shutdown redis, we only can print one log if fail.
In `bio.c`, if failing to fsync aof file, we will set `aof_bio_fsync_status` to error , and reject writing just like last writing aof error,  moreover also set INFO command field `aof_last_write_status` to error.
2021-04-01 12:45:15 +03:00
Valentino Geron
44d8b039e8
Fix XAUTOCLAIM response to return the next available id as the cursor (#8725)
This command used to return the last scanned entry id as the cursor,
instead of the next one to be scanned.
so in the next call, the user could / should have sent `(cursor` and not
just `cursor` if he wanted to avoid scanning the same record twice.

Scanning the record twice would look odd if someone is checking what
exactly was scanned, but it also has a side effect of incrementing the
delivery count twice.
2021-04-01 12:13:55 +03:00
guybe7
843f769b96
zsetAdd: Fix wrong reply in case of INCR and GT/LT (#8717)
If GT/LT fails the operation we need to reply with
nill (like failure due to NX).

Other changes:
Add the missing $encoding suffix to many zset tests

Note: there's a behavior change just in case of INCR + GT/LT that fails.
The old code was replying with the wrong (rejected) score, and now it'll reply with nil.

Note that that's anyway a corner case so this "behavior change" shouldn't have too much affect.
Using GT/LT with INCR has a predictable result even before we run the command
(INCR GT will only only / always fail if the increment is negative).
2021-04-01 09:33:53 +03:00
Wen Hui
d5935bb0a4
generalize config file check for sentinel (#8730)
The implications of this change is just that in the past when a config file was missing,
in some cases it was exiting before printing the sever startup prints and sometimes after,
and now it'll always exit before printing them.
2021-04-01 09:01:05 +03:00
wuYin
e5d50b236c
reuse existing range comparators in the zset (#8714)
There are 2 common range comparators for skiplist: zslValueGteMin and
zslValueLteMax, but they're not being reused in zslDeleteRangeByScore

This is a small change to make code cleaner.
2021-04-01 08:50:23 +03:00
Dvir Volk
07f39ae47b
Added macros for RM_log logging levels (#4246)
Added macros for RM_log logging levels, to avoid typos and the need
to memorize the level strings by heart
2021-04-01 08:44:57 +03:00
yoav-steinberg
636aa8de76
Always use prev node if there's some room in it setDeferredReply (#8729)
this is a followup PR for #8699
instead of copying the deferred reply data to the previous node only if it has room for the entire thing,
we can now split the new payload, put part of it into the spare space in the prev node,
and the rest may fit into the next node.
2021-04-01 00:07:19 +03:00
guybe7
68ba4d43ac
Initialize RedisModuleDigest with zeroes (#8731)
In case the module's digest function doesn't modify
'md' it'll contain garbage and result in wrong
DEBUG DIGEST
2021-03-31 23:56:04 +03:00
John Sully
57b242a5ba Fix issue #300
Former-commit-id: bee7f398c9c3b4c01b687d26194913bf215dca36
2021-03-30 23:44:34 +00:00
John Sully
ec13e045f7 Fix issue #300
Former-commit-id: e9551c9e8d196f37e3742dfc7df824e164181d60
2021-03-30 23:30:41 +00:00
guybe7
2c120af61e
XAUTOCLAIM: JUSTID should prevent incrementing delivery_count (#8724)
To align with XCLAIM and the XAUTOCLAIM docs
2021-03-30 23:48:01 +03:00
John Sully
32926467aa Fix crash in RDB save
Former-commit-id: ef1e122fcb039d25f60752ca27c9feb5e8cfbb31
2021-03-30 20:44:22 +00:00
Jérôme Loyet
91f4f41665
Add replica-announced config option (#8653)
The 'sentinel replicas <master>' command will ignore replicas with
`replica-announced` set to no.

The goal of disabling the config setting replica-announced is to allow ghost
replicas. The replica is in the cluster, synchronize with its master, can be
promoted to master and is not exposed to sentinel clients. This way, it is
acting as a live backup or living ghost.

In addition, to prevent the replica to be promoted as master, set
replica-priority to 0.
2021-03-30 23:40:22 +03:00
Viktor Söderqvist
5629dbe715
Add support for plaintext clients in TLS cluster (#8587)
The cluster bus is established over TLS or non-TLS depending on the configuration tls-cluster. The client ports distributed in the cluster and sent to clients are assumed to be TLS or non-TLS also depending on tls-cluster.

The cluster bus is now extended to also contain the non-TLS port of clients in a TLS cluster, when available. The non-TLS port of a cluster node, when available, is sent to clients connected without TLS in responses to CLUSTER SLOTS, CLUSTER NODES, CLUSTER SLAVES and MOVED and ASK redirects, instead of the TLS port.

The user was able to override the client port by defining cluster-announce-port. Now cluster-announce-tls-port is added, so the user can define an alternative announce port for both TLS and non-TLS clients.

Fixes #8134
2021-03-30 23:11:32 +03:00
yoav-steinberg
8cbd858d45
Minor client output buffer optimizations (#8699)
* Avoid checking output limits if buffer didn't grow.
* Use previouse node in case it has room in deferred output node.
2021-03-30 23:06:29 +03:00
JunhuaY
28375ff63e
re-fix config rewrite for empty save directive (#8722)
the bug was also discussed in #8716, and was solved in #8719, but incompletely:
when the server is started, and the save option is default, if you issue the " config set save "" "
to change the save option, and then issue the “config rewrite” command, the " save "" " won't be saved.
2021-03-30 22:49:06 +03:00
Hanif Ariffin
2c3d767431 Fix typo in error string
Former-commit-id: 0ac80086bcef36b2abd2eb67f0d9830e3172ab17
2021-03-30 13:52:47 -04:00
VivekSainiEQ
6a6f847289 Added logic back to only acquire/release GIL if modules are enabled, without causing deadlocks
Former-commit-id: 9ab36ddc36e1d12e41d2eca917ee24a44a82df52
2021-03-30 13:46:03 -04:00
VivekSainiEQ
fcc528db50 now acquire and release GIL every time you go to/wake from sleep
Former-commit-id: a6bc34c09b58f8c87da1611cd89638245d428a96
2021-03-30 13:46:03 -04:00
VivekSainiEQ
c48144356e Created and initialized seperate thread variables for modules
Former-commit-id: 3bb6b16c4a8f692b46040b72a51bef57fa03f1e6
2021-03-30 13:46:03 -04:00
Yossi Gottlieb
65311a3360
Fix config rewrite with an empty "save" parameter. (#8719) 2021-03-29 18:53:20 +03:00
Meir Shpilraien (Spielrein)
036963a7da
Restore old client 'processCommandAndResetClient' to fix false dead client indicator (#8715)
'processCommandAndResetClient' returns 1 if client is dead. It does it
by checking if serve.current_client is NULL. On script timeout, Redis will re-enter
'processCommandAndResetClient' and when finish we will set server.current_client
to NULL. This will cause later to falsely return 1 and think that the client that
sent the timed-out script is dead (Redis to stop reading from the client buffer).
2021-03-29 13:34:16 +03:00
John Sully
07530bc0fb Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable
Former-commit-id: 2910cee32ba6e4ef4b79b83bec2980c582a9310c
2021-03-29 00:49:34 +00:00
Steve
54fd3d4024
Fix redis-cli handling of rediss:// URL scheme (#8705) 2021-03-29 00:25:30 +03:00
John Sully
bb629c6f1e Enable LTO
Former-commit-id: 7241bfc25525e8b27193fb0d4f9eae2e5e98e8d6
2021-03-28 18:27:14 +00:00
John Sully
526761b6f9 Make some asserts debug only for perf
Former-commit-id: c22740df4f7c1468fbe19fd97dfaec20e011b6f1
2021-03-28 18:27:00 +00:00
John Sully
5a28256b80 Excessive rehashing adds latency
Former-commit-id: 9db2277b0fea032abcde089f617fe7e4f5ce12e3
2021-03-28 17:59:02 +00:00
John Sully
73bc5aec95 Fix thread safety issues with the cache prefetch logic
Former-commit-id: a80a128bb64b81115c095d6dd91896ff73048b3d
2021-03-28 17:58:43 +00:00
Huang Zhw
a79bb86856
Remove redundant defrag dictEntry in activeDefragSdsListAndDict. (#8713)
In activeDefragSdsListAndDict when defrag list key sucess, the key
in dict is also replaced. Then the corresponding dictEntry is
defraged. But the dictEntry will be defraged in next step by defrag
the dict values too.
2021-03-28 15:16:30 +03:00
John Sully
c356822897 Prefetch keys even in pure RAM scenarios
Former-commit-id: 54c0f11db0b35f34c9dfc3c8b4b76b7b7921bacd
2021-03-26 23:48:24 +00:00
John Sully
1a2803a1ac Eliminate unnecessary lookup in ensure when there is no snapshot
Former-commit-id: 04540531b95ebadf6d60c90c6681c11060eaf907
2021-03-26 23:44:42 +00:00
Huang Zhw
e138698e54
make processCommand check publish channel permissions. (#8534)
Add publish channel permissions check in processCommand.

processCommand didn't check publish channel permissions, so we can
queue a publish command in a transaction. But when exec the transaction,
it will fail with -NOPERM.

We also union keys/commands/channels permissions check togegher in
ACLCheckAllPerm. Remove pubsubCheckACLPermissionsOrReply in 
publishCommand/subscribeCommand/psubscribeCommand. Always 
check permissions in processCommand/execCommand/
luaRedisGenericCommand.
2021-03-26 14:10:01 +03:00
John Sully
931a5cce44 Fix issue where timethread wakes too slowly
Former-commit-id: 60f875f5eb7cb541e537f0f440ce8d5aeac57862
2021-03-26 02:54:33 +00:00