1274 Commits

Author SHA1 Message Date
John Sully
15d9f45b4e Fix thread safety issues with the cache prefetch logic
Former-commit-id: 4892122fc02109d98684a350bd732a0b08a8c7b4
2021-03-28 20:21:44 +00:00
John Sully
73bc5aec95 Fix thread safety issues with the cache prefetch logic
Former-commit-id: a80a128bb64b81115c095d6dd91896ff73048b3d
2021-03-28 17:58:43 +00:00
Huang Zhw
e138698e54
make processCommand check publish channel permissions. (#8534)
Add publish channel permissions check in processCommand.

processCommand didn't check publish channel permissions, so we can
queue a publish command in a transaction. But when exec the transaction,
it will fail with -NOPERM.

We also union keys/commands/channels permissions check togegher in
ACLCheckAllPerm. Remove pubsubCheckACLPermissionsOrReply in 
publishCommand/subscribeCommand/psubscribeCommand. Always 
check permissions in processCommand/execCommand/
luaRedisGenericCommand.
2021-03-26 14:10:01 +03:00
Oran Agra
497351ad07
Fix SLOWLOG for blocked commands (#8632)
* SLOWLOG didn't record anything for blocked commands because the client
  was reset and argv was already empty. there was a fix for this issue
  specifically for modules, now it works for all blocked clients.
* The original command argv (before being re-written) was also reset
  before adding the slowlog on behalf of the blocked command.
* Latency monitor is now updated regardless of the slowlog flags of the
  command or its execution (their purpose is to hide sensitive info from
  the slowlog, not hide the fact the latency happened).
* Latency monitor now uses real_cmd rather than c->cmd (which may be
  different if the command got re-written, e.g. GEOADD)

Changes:
* Unify shared code between slowlog insertion in call() and
  updateStatsOnUnblock(), hopefully prevent future bugs from happening
  due to the later being overlooked.
* Reset CLIENT_PREVENT_LOGGING in resetClient rather than after command
  processing.
* Add a test for SLOWLOG and BLPOP

Notes:
- real_cmd == c->lastcmd, except inside MULTI and Lua.
- blocked commands never happen in these cases (MULTI / Lua)
- real_cmd == c->cmd, except for when the command is rewritten (e.g.
  GEOADD)
- blocked commands (currently) are never rewritten
- other than the command's CLIENT_PREVENT_LOGGING, and the
  execution flag CLIENT_PREVENT_LOGGING, other cases that we want to
  avoid slowlog are on AOF loading (specifically CMD_CALL_SLOWLOG will
  be off when executed from execCommand that runs from an AOF)
2021-03-25 10:20:27 +02:00
yoav-steinberg
d026647f4f
Avoid evaluating log arguments when log filtered by level. (#8685) 2021-03-24 08:22:12 +02:00
John Sully
98ca9892c3 Perform GET command inline
Former-commit-id: 5623936d99e334ab103f3dc1541b145c125d0ee8
2021-03-23 03:44:20 +00:00
Yossi Gottlieb
c4ef1efdb7
Add support for reading encrypted keyfiles. (#8644) 2021-03-22 13:27:46 +02:00
Yossi Gottlieb
c3df27d1ea
Fix slowdown due to child reporting CoW. (#8645)
Reading CoW from /proc/<pid>/smaps can be slow with large processes on
some platforms.

This measures the time it takes to read CoW info and limits the duty
cycle of future updates to roughly 1/100.

As current_cow_size no longer represnets a current, fixed interval value
there is also a new current_cow_size_age field that provides information
about the age of the size value, in seconds.
2021-03-22 13:25:58 +02:00
VivekSainiEQ
8714a26cd9 Removed hasModuleGIL boolean and added fix from PR #292
Former-commit-id: 68d213f4c9c1c3161929a5e20ca4f2b27665c8fd
2021-03-19 20:10:24 +00:00
VivekSainiEQ
189967e7af added lock releasing w/ hasModuleGIL, changed module serverTL, and moved module_blocking_pipe to global scope to fix issue #276
Former-commit-id: 7d9a2ce827a2f8d48e4682b3cc95460cc82f9778
2021-03-19 20:08:28 +00:00
Madelyn Olson
e1d98bca5a
Redact slowlog entries for config with sensitive data. (#8584)
Redact config set requirepass/masterauth/masteruser from slowlog in addition to showing ACL commands without sensitive values.
2021-03-15 22:00:29 -07:00
John Sully
25f56c59b0 Remove snapshot consolidation because its not properly thread safe
Former-commit-id: dcb86d2b92e3ecdb9f914b6f5de9e184cdd23036
2021-03-16 02:37:49 +00:00
John Sully
6b579db919 Fix TSAN errors in tests
Former-commit-id: 30207d2ab34f175b94f430f97581191343d23f1e
2021-03-16 02:29:24 +00:00
Huang Zhw
84d056d0f7
Fix typo and outdated comments. (#8640) 2021-03-14 09:41:43 +02:00
guybe7
3d0b427c30
Fix some issues with modules and MULTI/EXEC (#8617)
Bug 1:
When a module ctx is freed moduleHandlePropagationAfterCommandCallback
is called and handles propagation. We want to prevent it from propagating
commands that were not replicated by the same context. Example:
1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3)
2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte
3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx.
4. moduleHandlePropagationAfterCommandCallback is called, calling
   alsoPropagates EXEC (Note: EXEC is still not written to socket),
   setting server.in_trnsaction = 0
5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now
   we have nested MULTI calls, which is no good) and then cmd3

We must prevent RM_Call(cmd2) from resetting server.in_transaction.
REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose.

Bug 2:
Fix issues with nested RM_Call where some have '!' and some don't.
Example:
1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!')
2. module2.bar internally calls RM_Call of INCR with '!'
3. at the end of module1.foo we call RM_ReplicateVerbatim

We want the replica/AOF to see only module1.foo and not the INCR from module2.bar

Introduced a global replication_allowed flag inside RM_Call to determine
whether we need to replicate or not (even if '!' was specified)

Other changes:
Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec
just for better readability
2021-03-10 18:02:17 +02:00
guybe7
61a73de64d
Cleanup ZADD_* flags (#8559)
Have a clear separation between in and out flags

Other changes:

delete dead code in RM_ZsetIncrby: if zsetAdd returned error (happens only if
the result of the operation is NAN or if score is NAN) we return immediately so
there is no way that zsetAdd succeeded and returned NAN in the out-flags
2021-03-10 16:09:43 +02:00
Huang Zhw
367ba4125f
Remove some dead code (#8605) 2021-03-05 09:54:34 -08:00
John Sully
1bf6ca0645 Load perf fixes with a storage provider set
Former-commit-id: 861b19de00c75c9167cc25031292284ad1c21893
2021-03-04 07:41:06 +00:00
Yossi Gottlieb
ea1b48bd12
Improve SSL cleanup handling. (#8589)
This solves the problem of /dev/random and /dev/urandom open file
descriptors leaking to childs with some versions of OpenSSL.
2021-03-03 10:08:06 +02:00
Yossi Gottlieb
0bad00d049
Fix errors when loading RDB with missing modules. (#8579)
Fixes #8574
2021-03-02 09:39:37 +02:00
John Sully
20089d7256 Drastically improve perf when loading an RDB with a storage provider
Former-commit-id: 0133b42d54676e8fac2c5cb006cc87988dced268
2021-03-02 04:16:20 +00:00
YaacovHazan
a031d268b1
Make port, tls-port and bind configurations modifiable (#8510)
Add ability to modify port, tls-port and bind configurations by CONFIG SET command.

To simplify the code and make it cleaner, a new structure
added, socketFds, which contains the file descriptors array and its counter,
and used for TCP, TLS and Cluster sockets file descriptors.
2021-03-01 16:04:44 +02:00
Viktor Söderqvist
6122f1c450
Shared reusable client for RM_Call() (#8516)
A single client pointer is added in the server struct. This is
initialized by the first RM_Call() and reused for every subsequent
RM_Call() except if it's already in use, which means that it's not
used for (recursive) module calls to modules. For these, a new
"fake" client is created each time.

Other changes:
* Avoid allocating a dict iterator in pubsubUnsubscribeAllChannels
  when not needed
2021-02-28 14:11:18 +02:00
John Sully
bfbdf23b1a Eliminate needless lock
Former-commit-id: 60f972d463f202edb33ff9a25bc2bd3e2558105c
2021-02-26 05:40:56 +00:00
Madelyn Olson
4a474843fb
Moved requirepass and querybuf length to generic configs (#8557)
Moved additional configs to generic infrastructure.
2021-02-25 21:00:27 -08:00
christian
75539ced8e Offload updating cached time to dedicated thread
Former-commit-id: 9bfc8a43952481b5b54a7b051d44b8bece4a18dd
2021-02-26 00:50:22 +00:00
Yossi Gottlieb
d828f90c26
Fix allowed length for REPLCONF ip-address. (#8517)
Originally this was limited to IPv6 address length, but effectively it
has been used for host names and now that Sentinel accepts that as well
we need to be able to store full hostnames.

Fixes #8507
2021-02-21 11:22:36 +02:00
John Sully
7fb41ba1bd Implement maxstorage config
Former-commit-id: 79e07859dfec14edf2a39646013e1a82db033d4f
2021-02-19 05:17:41 +00:00
Harkrishn Patro
303465af35
Remove redundant pubsub list to store the patterns. (#8472)
Remove redundant pubsub list to store the patterns.
2021-02-17 14:13:50 -08:00
uriyage
fd052d2a86
Adds INFO fields to track fork child progress (#8414)
* Adding current_save_keys_total and current_save_keys_processed info fields.
  Present in replication, BGSAVE and AOFRW.
* Changing RM_SendChildCOWInfo() to RM_SendChildHeartbeat(double progress)
* Adding new info field current_fork_perc. Present in Replication, BGSAVE, AOFRW,
  and module forks.
2021-02-16 16:06:51 +02:00
Yossi Gottlieb
141ac8df59
Escape unsafe field name characters in INFO. (#8492)
Fixes #8489
2021-02-15 17:08:53 +02:00
John Sully
f3853b6485 Fix deadlock in storage prefetch
Former-commit-id: 9e6162de08248f7726b97b73aee2f23808ff4e7b
2021-02-12 19:38:57 +00:00
John Sully
bb767ad61d Enable prefetch with storage providers
Former-commit-id: 21f9c52c446ad1b460a340c3d873839b2355728f
2021-02-11 00:50:21 +00:00
Madelyn Olson
899c85ae67
Moved most static strings into the shared structure (#8411)
Moved most static strings into the shared structure
2021-02-09 11:52:28 -08:00
John Sully
e5343f47c2 Merge branch 'unstable' into keydbpro
Former-commit-id: e2140793f2bf565972ada799af73bf4457e2718d
2021-02-08 18:17:09 +00:00
John Sully
bef72e5f6a Implement rehash during spinlock
Former-commit-id: f68a26381a35b27948046d46c2c7bcfbdc21143d
2021-02-07 19:11:05 -05:00
John Sully
4c9f912c3b Fix cluster test failure
Former-commit-id: 230fde39882766426d892fd30d0c8349f5cca912
2021-02-07 23:38:25 +00:00
christianEQ
511f01bd14 fixed up backup stuff
Former-commit-id: 8a23162c24dc13a4268928c5defd85b0b5249d39
2021-02-04 22:36:32 +00:00
christianEQ
cca8d186aa Merge remote-tracking branch 'opensource/unstable' into keydbpro
Former-commit-id: 5bad058733de2c217340bb9ee48f02b07d754808
2021-02-03 18:10:27 +00:00
christianEQ
406265263b added severity levels for afterErrorReply
Former-commit-id: fe0f07199353abf6668cd66cd2e21751db5c21d9
2021-01-31 00:12:49 -05:00
filipe oliveira
f0c5052aa8
Enabled background and reply time tracking on blocked on keys/blocked on background work clients (#7491)
This commit enables tracking time of the background tasks and on replies,
opening the door for properly tracking commands that rely on blocking / background
 work via the slowlog, latency history, and commandstats. 

Some notes:
- The time spent blocked waiting for key changes, or blocked on synchronous
  replication is not accounted for. 

- **This commit does not affect latency tracking of commands that are non-blocking
  or do not have background work.** ( meaning that it all stays the same with exception to
  `BZPOPMIN`,`BZPOPMAX`,`BRPOP`,`BLPOP`, etc... and module's commands that rely
  on background threads ). 

-  Specifically for latency history command we've added a new event class named
  `command-unblocking` that will enable latency monitoring on commands that spawn
  background threads to do the work.

- For blocking commands we're now considering the total time of a command as the
  time spent on call() + the time spent on replying when unblocked.

- For Modules commands that rely on background threads we're now considering the
  total time of a command as the time spent on call (main thread) + the time spent on
  the background thread ( if marked within `RedisModule_MeasureTimeStart()` and
  `RedisModule_MeasureTimeEnd()` ) + the time spent on replying (main thread)

To test for this feature we've added a `unit/moduleapi/blockonbackground` test that relies on
a module that blocks the client and sleeps on the background for a given time. 
- check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time
- check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time even in timeout
- check blocked command with multiple calls RedisModule_MeasureTimeStart()  is tracking the total background time
- check blocked command without calling RedisModule_MeasureTimeStart() is not reporting background time
2021-01-29 15:38:30 +02:00
Yang Bodong
b9a0500f16
Add HRANDFIELD and ZRANDMEMBER. improvements to SRANDMEMBER (#8297)
New commands:
`HRANDFIELD [<count> [WITHVALUES]]`
`ZRANDMEMBER [<count> [WITHSCORES]]`
Algorithms are similar to the one in SRANDMEMBER.

Both return a simple bulk response when no arguments are given, and an array otherwise.
In case values/scores are requested, RESP2 returns a long array, and RESP3 a nested array.
note: in all 3 commands, the only option that also provides random order is the one with negative count.

Changes to SRANDMEMBER
* Optimization when count is 1, we can use the more efficient algorithm of non-unique random
* optimization: work with sds strings rather than robj

Other changes:
* zzlGetScore: when zset needs to convert string to double, we use safer memcpy (in
  case the buffer is too small)
* Solve a "bug" in SRANDMEMBER test: it intended to test a positive count (case 3 or
  case 4) and by accident used a negative count

Co-authored-by: xinluton <xinluton@qq.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-01-29 10:47:28 +02:00
christianEQ
011b69e9a6 killMainThread -> killServerThreads
Former-commit-id: 0fea781f8f0218c5475890a395c49bd7e4faa03f
2021-01-28 23:57:19 +00:00
Allen Farris
0d18a1e85f
implement FAILOVER command (#8315)
Implement FAILOVER command, which coordinates failover
between the server and one of its replicas.
2021-01-28 13:18:05 -08:00
Yossi Gottlieb
4bb5ccbefb
Add proc-title-template option. (#8397)
Make it possible to customize the process title, i.e. include custom
strings, immutable configuration like port, tls-port, unix socket name,
etc.
2021-01-28 18:17:39 +02:00
Z. Liu
17b34c7309
Add 'set-proc-title' config so that this mechanism can be disabled (#3623)
if option `set-proc-title' is no, then do nothing for proc title.

The reason has been explained long ago, see following:

We update redis to 2.8.8, then found there are some side effect when
redis always change the process title.

We run several slave instance on one computer, and all these salves
listen on unix socket only, then ps will show:

  1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server *:0
  1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server *:0

for redis 2.6 the output of ps is like following:

  1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server /etc/redis/a.conf
  1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server /etc/redis/b.conf

Later is more informational in our case. The situation
is worse when we manage the config and process running
state by salt. Salt check the process by running "ps |
grep SIG" (for Gentoo System) to check the running
state, where SIG is the string to search for when
looking for the service process with ps. Previously, we
define sig as "/usr/sbin/redis-server
/etc/redis/a.conf". Since the ps output is identical for
our case, so we have no way to check the state of
specified redis instance.

So, for our case, we prefer the old behavior, i.e, do
not change the process title for the main redis process.
Or add an option such as "set-proc-title [yes|no]" to
control this behavior.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-01-28 11:12:39 +02:00
Raghav Muddur
0367a80819
GETEX, GETDEL and SET PXAT/EXAT (#8327)
This commit introduces two new command and two options for an existing command

GETEX <key> [PERSIST][EX seconds][PX milliseconds] [EXAT seconds-timestamp]
[PXAT milliseconds-timestamp]

The getexCommand() function implements extended options and variants of the GET
command. Unlike GET command this command is not read-only. Only one of the options
can be used at a given time.

1. PERSIST removes any TTL associated with the key.
2. EX Set expiry TTL in seconds.
3. PX Set expiry TTL in milliseconds.
4. EXAT Same like EX instead of specifying the number of seconds representing the
    TTL (time to live), it takes an absolute Unix timestamp
5. PXAT Same like PX instead of specifying the number of milliseconds representing the
    TTL (time to live), it takes an absolute Unix timestamp

Command would return either the bulk string, error or nil.

GETDEL <key>
Would delete the key after getting.

SET key value [NX] [XX] [KEEPTTL] [GET] [EX <seconds>] [PX <milliseconds>]
[EXAT <seconds-timestamp>][PXAT <milliseconds-timestamp>]

Two new options added here are EXAT and PXAT

Key implementation notes
- `SET` with `PX/EX/EXAT/PXAT` is always translated to `PXAT` in `AOF`. When relative time is
  specified (`PX/EX`), replication will always use `PX`.
- `setexCommand` and `psetexCommand` would no longer need translation in `feedAppendOnlyFile`
  as they are modified to invoke `setGenericCommand ` with appropriate flags which will take care of
  correct AOF translation.
- `GETEX` without any optional argument behaves like `GET`.
- `GETEX` command is never propagated, It is either propagated as `PEXPIRE[AT], or PERSIST`.
- `GETDEL` command is propagated as `DEL`
- Combined the validation for `SET` and `GETEX` arguments. 
- Test cases to validate AOF/Replication propagation
2021-01-27 19:47:26 +02:00
Wang Yuan
ad7d4c6b70
Implement rdb-only replication (#8303)
In some scenarios, such as remote backup, we only want to get remote
redis server db snapshot. Currently, redis-cli acts as a replica and
sends SYNC to redis, but redis still accumulates replication buffer
in the replica client output buffer, that may result in using vast
memory, or failing to transfer RDB because of client-output-buffer-limit.
In this commit, we add 'replconf rdb-only 0|1', redis doesn't send
incremental replication buffer to them if they send 'replconf rdb-only 1',
so we can reduce used memory and improve success of getting RDB.
2021-01-27 17:13:10 +02:00
christianEQ
8f132802d8 removed unimplemented killIOThreads function declaration
Former-commit-id: f30f8eba2f0a5cb247478c2104a4f748659a6cf2
2021-01-26 22:27:22 +00:00
christianEQ
396f8316e8 fixed server.cpp mentions
Former-commit-id: 7ed2296fc5d10a1c41020558b4bc445953531bdd
2021-01-26 22:22:19 +00:00