1216 Commits

Author SHA1 Message Date
John Sully
c5f6cb1ba5 Add multi-master-no-forward command to reduce bus traffic with multi-master
Former-commit-id: d99d06b1250a51ea4bc54f678f451acbb7901e33
2020-07-12 19:25:19 +00:00
John Sully
17661f2382 Implement storage key cache, and writeback memory model
Former-commit-id: 732bd9c153459f1174475ad67de36c399ddbe359
2020-07-11 21:23:48 +00:00
Yossi Gottlieb
3e6f2b1a45
TLS: Session caching configuration support. (#7420)
* TLS: Session caching configuration support.
* TLS: Remove redundant config initialization.
2020-07-10 11:33:47 +03:00
Oran Agra
5977a94842
RESTORE ABSTTL won't store expired keys into the db (#7472)
Similarly to EXPIREAT with TTL in the past, which implicitly deletes the
key and return success, RESTORE should not store key that are already
expired into the db.
When used together with REPLACE it should emit a DEL to keyspace
notification and replication stream.
2020-07-10 10:02:37 +03:00
John Sully
52123f9529 Merge branch 'keydbpro' into PRO_RELEASE_6
Former-commit-id: 243dcb3853cc965109cb24a940229db7844cdd11
2020-07-10 04:11:57 +00:00
John Sully
f03b75b005 MVCC scan support filtering by type on the async thread
Former-commit-id: 14f8c0ff686b93976eead5fa6bf526c2eecb5ae0
2020-07-10 03:43:56 +00:00
John Sully
34482220af Fix issue where SCAN misses elements while snapshot is in flight
Former-commit-id: ce005d748ebf0e116d674a96f74d698d17394010
2020-07-10 01:43:51 +00:00
John Sully
d3b72484e4 Initial scan implementation (not tested)
Former-commit-id: 0f3911d56e6878d91b6e3cacc7d637934ffe6099
2020-07-09 12:57:35 -04:00
Muhammad Zahalqa
ca929bb9c1 fixes for robj_sharedptr
1. fix cases where null pointer might be accessed
2. make assignmnet op safe
3. make operator bool explicit (safe bool idiom)
4. make comparison operators symetric

fix robj_sharedptr use in rdb.cpp


Former-commit-id: ede524c0647c0875f1071978f26ff785c8d1183e
2020-06-24 23:18:27 -04:00
Salvatore Sanfilippo
6bbbdd26f4
Merge pull request #7390 from oranagra/exec_fails_abort
EXEC always fails with EXECABORT and multi-state is cleared
2020-06-23 13:12:52 +02:00
Oran Agra
65a3307bc9 EXEC always fails with EXECABORT and multi-state is cleared
In order to support the use of multi-exec in pipeline, it is important that
MULTI and EXEC are never rejected and it is easy for the client to know if the
connection is still in multi state.

It was easy to make sure MULTI and DISCARD never fail (done by previous
commits) since these only change the client state and don't do any actual
change in the server, but EXEC is a different story.

Since in the past, it was possible for clients to handle some EXEC errors and
retry the EXEC, we now can't affort to return any error on EXEC other than
EXECABORT, which now carries with it the real reason for the abort too.

Other fixes in this commit:
- Some checks that where performed at the time of queuing need to be re-
  validated when EXEC runs, for instance if the transaction contains writes
  commands, it needs to be aborted. there was one check that was already done
  in execCommand (-READONLY), but other checks where missing: -OOM, -MISCONF,
  -NOREPLICAS, -MASTERDOWN
- When a command is rejected by processCommand it was rejected with addReply,
  which was not recognized as an error in case the bad command came from the
  master. this will enable to count or MONITOR these errors in the future.
- make it easier for tests to create additional (non deferred) clients.
- add tests for the fixes of this commit.
2020-06-23 12:01:33 +03:00
antirez
afb8b3c030 LPOS: implement the final design. 2020-06-12 12:08:06 +02:00
Paul Spooren
31c9cd8604 LRANK: Add command (the command will be renamed LPOS).
The `LRANK` command returns the index (position) of a given element
within a list. Using the `direction` argument it is possible to specify
going from head to tail (acending, 1) or from tail to head (decending,
-1). Only the first found index is returend. The complexity is O(N).

When using lists as a queue it can be of interest at what position a
given element is, for instance to monitor a job processing through a
work queue. This came up within the Python `rq` project which is based
on Redis[0].

[0]: https://github.com/rq/rq/issues/1197

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-06-12 12:08:06 +02:00
antirez
e63a5ba122 LPOS: implement the final design. 2020-06-10 12:49:15 +02:00
Paul Spooren
a7936ef96d LRANK: Add command (the command will be renamed LPOS).
The `LRANK` command returns the index (position) of a given element
within a list. Using the `direction` argument it is possible to specify
going from head to tail (acending, 1) or from tail to head (decending,
-1). Only the first found index is returend. The complexity is O(N).

When using lists as a queue it can be of interest at what position a
given element is, for instance to monitor a job processing through a
work queue. This came up within the Python `rq` project which is based
on Redis[0].

[0]: https://github.com/rq/rq/issues/1197

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-06-10 12:07:40 +02:00
John Sully
ed2d611b6d Fix lock inversion
Former-commit-id: 430cf88623fce8f0675ea19f73d98f6cafe9070f
2020-06-09 20:58:03 -04:00
John Sully
c83fe382ff COW pointer is not thread safe so we need a global expire lock
Former-commit-id: ac12eef75301564d920e622a08e4f586486fd395
2020-06-09 19:58:42 -04:00
John Sully
327f922e04 We need to search snapshots before deciding to not save a key
Former-commit-id: 357b09c500eecfe61970e121644a7200ac2387b9
2020-06-05 00:39:58 -04:00
John Sully
f184eb6c3a Consolidate snapshots should not consider keys in bulk storage
Former-commit-id: e02a8f503d45daf63150c60a8a8c3751b9b8581e
2020-06-04 00:26:51 -04:00
John Sully
44c1e6d5e7 Merge branch 'unstable' into keydbpro
Former-commit-id: 08a36155e3db9918048e87c3d691b7317787c9ab
2020-06-01 17:41:37 -04:00
John Sully
e380087cdb Unify beforeSleep handling
Former-commit-id: 1cb48c7bf6a7e91e728a677902a7bfc64fe80dd6
2020-05-29 17:44:55 -04:00
John Sully
cfe9f8f3bc Merge tag '6.0.4' into unstable
Redis 6.0.4.


Former-commit-id: 9c31ac7925edba187e527f506e5e992946bd38a6
2020-05-29 00:57:07 -04:00
antirez
cc549b46a4 Replication: showLatestBacklog() refactored out. 2020-05-28 10:09:51 +02:00
antirez
2112a57028 Remove the meaningful offset feature.
After a closer look, the Redis core devleopers all believe that this was
too fragile, caused many bugs that we didn't expect and that were very
hard to track. Better to find an alternative solution that is simpler.
2020-05-28 10:09:51 +02:00
antirez
6a16a636bf Replication: showLatestBacklog() refactored out. 2020-05-28 10:08:16 +02:00
antirez
22472fe5a1 Remove the meaningful offset feature.
After a closer look, the Redis core devleopers all believe that this was
too fragile, caused many bugs that we didn't expect and that were very
hard to track. Better to find an alternative solution that is simpler.
2020-05-27 12:06:33 +02:00
John Sully
d50c0c5ba3 Merge branch 'unstable' into keydbpro
Former-commit-id: b0636273806ce323627ce31c1ad7f86ceb39696d
2020-05-26 01:34:19 -04:00
John Sully
2bb0be3a96 Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable
Former-commit-id: 1449c8b9fe647288da7de0ff10cc68ada4be8f92
2020-05-26 01:29:08 -04:00
John Sully
d7fa406a0e Fix client tracking, also ensure tracking tests are enabled
Former-commit-id: 1938af27f50f9686dc98b4839fb439cc03b4a250
2020-05-26 01:28:52 -04:00
antirez
e3f864b5fd Make disconnectSlaves() synchronous in the base case.
Otherwise we run into that:

Backtrace:
src/redis-server 127.0.0.1:21322(logStackTrace+0x45)[0x479035]
src/redis-server 127.0.0.1:21322(sigsegvHandler+0xb9)[0x4797f9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fd373c5e390]
src/redis-server 127.0.0.1:21322(_serverAssert+0x6a)[0x47660a]
src/redis-server 127.0.0.1:21322(freeReplicationBacklog+0x42)[0x451282]
src/redis-server 127.0.0.1:21322[0x4552d4]
src/redis-server 127.0.0.1:21322[0x4c5593]
src/redis-server 127.0.0.1:21322(aeProcessEvents+0x2e6)[0x42e786]
src/redis-server 127.0.0.1:21322(aeMain+0x1d)[0x42eb0d]
src/redis-server 127.0.0.1:21322(main+0x4c5)[0x42b145]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd3738a3830]
src/redis-server 127.0.0.1:21322(_start+0x29)[0x42b409]

Since we disconnect all the replicas and free the replication backlog in
certain replication paths, and the code that will free the replication
backlog expects that no replica is connected.

However we still need to free the replicas asynchronously in certain
cases, as documented in the top comment of disconnectSlaves().
2020-05-25 12:08:01 +02:00
Maxime de Roucy
b9b4b4cfde acl: @replication command group
containing mandatory commands for replication (used by masteruser user)


Former-commit-id: 123bc125cfe7b30fdeef833759a33ccc854b5c51
2020-05-25 02:35:22 -04:00
John Sully
fcfff165da Enable TLS connections
Former-commit-id: d05da0fabdfeb7eadce6546c7c1d85739b2794d7
2020-05-24 19:57:16 -04:00
John Sully
15bf66b922 Fix test failures
Former-commit-id: c0b2263478d79f33d04ed409a68caa312884fa4c
2020-05-24 18:33:15 -04:00
John Sully
cece963cf3 Merge branch 'unstable' into keydbpro
Former-commit-id: a830cf85df236885558c5571c0bf23cfb23e3655
2020-05-24 14:41:53 -04:00
John Sully
23e1601276 Hold the AE lock until we process the repl backlog after a command
Former-commit-id: 4893eedacbf5f9210d02361ed07b9beee0c9f655
2020-05-24 11:19:23 -04:00
John Sully
b892d3deff Add standardized way to post client dependent functions
Former-commit-id: fb50163b47cf660911dc7f67809c15ba1394613a
2020-05-24 02:46:39 -04:00
John Sully
51976e2be8 Incorrect memory order on weak ordered architectures
Former-commit-id: 90162da1a6c025d148a0d1761ae4d5fbc9bfc0ef
2020-05-24 01:57:18 -04:00
John Sully
2d783a3cbf Merge tag '6.0.2' into unstable
Redis 6.0.2


Former-commit-id: a010e4a4b2cc2bcad1cb14604b7ebc596c35b05e
2020-05-22 16:45:18 -04:00
John Sully
67b7d512cb Merge commit 'cb683a84f7a9d530bec629f2b656e7d0842a0f75' into unstable
Former-commit-id: 817a11ff110772893eda0675912bbb0cfc1fca74
2020-05-22 15:56:35 -04:00
antirez
adc5df1bc3 Make disconnectSlaves() synchronous in the base case.
Otherwise we run into that:

Backtrace:
src/redis-server 127.0.0.1:21322(logStackTrace+0x45)[0x479035]
src/redis-server 127.0.0.1:21322(sigsegvHandler+0xb9)[0x4797f9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fd373c5e390]
src/redis-server 127.0.0.1:21322(_serverAssert+0x6a)[0x47660a]
src/redis-server 127.0.0.1:21322(freeReplicationBacklog+0x42)[0x451282]
src/redis-server 127.0.0.1:21322[0x4552d4]
src/redis-server 127.0.0.1:21322[0x4c5593]
src/redis-server 127.0.0.1:21322(aeProcessEvents+0x2e6)[0x42e786]
src/redis-server 127.0.0.1:21322(aeMain+0x1d)[0x42eb0d]
src/redis-server 127.0.0.1:21322(main+0x4c5)[0x42b145]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd3738a3830]
src/redis-server 127.0.0.1:21322(_start+0x29)[0x42b409]

Since we disconnect all the replicas and free the replication backlog in
certain replication paths, and the code that will free the replication
backlog expects that no replica is connected.

However we still need to free the replicas asynchronously in certain
cases, as documented in the top comment of disconnectSlaves().
2020-05-22 19:29:09 +02:00
John Sully
27eb239f1a Fix bad merge in CI.yml
Former-commit-id: 6311d709c39b3bacaeab77b18033010f1b548f81
2020-05-21 22:09:06 -04:00
John Sully
ad1eac9856 Merge commit 'bec68bff2978de88b23140f41b6a943162333b11' into unstable
Former-commit-id: 8ab77465dbb3c0b1f859f24cdbb461937516eb6a
2020-05-21 19:37:42 -04:00
John Sully
3384af024d Merge commit 'cbcd07777dc569618a34f59e5fd0de53178f4f1d' into unstable
Former-commit-id: 49784c8122e827bd9de86bbae4d88c313400e86e
2020-05-21 18:48:49 -04:00
John Sully
3324d4dc0f Merge commit '5719b3054a534e62c25ae97680ecd4f7238ba484' into unstable
Former-commit-id: 3e03f308b564cd94f4a6407c80792d080e0f83c5
2020-05-21 17:55:09 -04:00
John Sully
b6500a08dc Merge commit '026cc11b056f063631d990f1a9db45b4e583974e' into unstable
Former-commit-id: 95cecb0229af0278cf614ffd746ba829ae7c897c
2020-05-21 17:45:15 -04:00
John Sully
c4db71f971 Merge commit '024c380b9da02bc4112822c0f5f9ac1388b4205b' into unstable
Former-commit-id: 7676f5b15f24a044257250b8891d23b14642da48
2020-05-21 17:36:53 -04:00
antirez
80c906bd30 Cache master without checking of deferred close flags.
The context is issue #7205: since the introduction of threaded I/O we close
clients asynchronously by default from readQueryFromClient(). So we
should no longer prevent the caching of the master client, to later
PSYNC incrementally, if such flags are set. However we also don't want
the master client to be cached with such flags (would be closed
immediately after being restored). And yet we want a way to understand
if a master was closed because of a protocol error, and in that case
prevent the caching.
2020-05-15 22:23:24 +02:00
antirez
1a7cd2c0e2 Cache master without checking of deferred close flags.
The context is issue #7205: since the introduction of threaded I/O we close
clients asynchronously by default from readQueryFromClient(). So we
should no longer prevent the caching of the master client, to later
PSYNC incrementally, if such flags are set. However we also don't want
the master client to be cached with such flags (would be closed
immediately after being restored). And yet we want a way to understand
if a master was closed because of a protocol error, and in that case
prevent the caching.
2020-05-15 10:19:13 +02:00
antirez
74249be4a2 Track events processed while blocked globally.
Related to #7234.
2020-05-14 11:29:43 +02:00
antirez
bc4667acbc Track events processed while blocked globally.
Related to #7234.
2020-05-14 10:06:27 +02:00