989 Commits

Author SHA1 Message Date
John Sully
b336d269ca Remove addReply*Async methods since we already know if its async or not. This is just a source of bugs
Former-commit-id: df22cdf6e91a1b9c390b69c4209c719ecf1e44f1
2020-10-24 02:18:03 +00:00
John Sully
b5074e9b1e Fix replica buffer overflows
Former-commit-id: 738c782f02517744662991091beb3f724661317e
2020-10-16 06:47:40 +00:00
John Sully
a9f4c37604 Reduce async write copies
Former-commit-id: ed369d722335ed4105748ef2bed5d64f3f32c433
2020-10-15 23:10:17 +00:00
John Sully
a543a502b8 Mac build break fix
Former-commit-id: 5f3543921b4123c2216d9294c5eb7bfed007cbf5
2020-09-30 20:17:02 +00:00
John Sully
4f18a247e3 Merge tag '6.0.8' into unstable
Former-commit-id: 4c7e4b91a6bb2034636856b608b8c386d07f5541
2020-09-30 19:47:55 +00:00
John Sully
ad4c2390a8 Merge branch 'unstable' into keydbpro
Former-commit-id: d1986d9fcdfd56f2c30ee95edef6d5260e0aa777
2020-09-24 22:02:30 +00:00
John Sully
d0885f945e Support read only replicas attaching to active replicas (Bug #229)
Former-commit-id: 676644f2db1634f97dd557a6e321234914a5e5d3
2020-09-23 04:16:08 +00:00
John Sully
f333cf98c9 Dramatically improve the performance of subkey expires
Former-commit-id: 368f67f42217c5fd2cfb3cb3643984917793e994
2020-09-20 23:30:21 +00:00
John Sully
d89fbd4175 Avoid memory overhead in situations it won't be used (active rep only)
Former-commit-id: 8f7d977f9068904eca783fefbb24df4d8abd4214
2020-09-19 21:44:59 +00:00
John Sully
2c6fbac6ea Allow reads during an active replica load, optionally allow writes as an experimental feature
Former-commit-id: 655bb6e21c5e814980a7b8b3ac1481ef142c1845
2020-09-17 00:13:03 +00:00
John Sully
4eecb1825f Implement replica-quorum config
Former-commit-id: ab4cdd2ed4d3ee9752737d20662178d73d16b4c2
2020-09-16 03:26:31 +00:00
Yossi Gottlieb
bf6c002005 TLS: Session caching configuration support. (#7420)
* TLS: Session caching configuration support.
* TLS: Remove redundant config initialization.


Former-commit-id: d3834c50699bc4f31f381d6d03d4c1b022380895
2020-09-16 02:10:29 +00:00
Roi Lipman
25e211bcf0 RM_ThreadSafeContextTryLock a non-blocking method for acquiring GIL (#7738)
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 042189fd8707544139337b3ddcf38b5c5fea1bf0)
2020-09-10 14:09:00 +03:00
Yossi Gottlieb
635704d424 Tests: validate CONFIG REWRITE for all params. (#7764)
This is a catch-all test to confirm that that rewrite produces a valid
output for all parameters and that this process does not introduce
undesired configuration changes.

(cherry picked from commit a8b7268911803f4ac827ab22e8e5b10134664832)
2020-09-10 14:09:00 +03:00
John Sully
b6d8a5938d Dramatically improve perf by blocking commands
Former-commit-id: e47584b286c41cf0783fe014ac8b6ec187564ade
2020-09-07 00:49:53 +00:00
Yossi Gottlieb
f6d04d01b9 Add oom-score-adj configuration option to control Linux OOM killer. (#1690)
Add Linux kernel OOM killer control option.

This adds the ability to control the Linux OOM killer oom_score_adj
parameter for all Redis processes, depending on the process role (i.e.
master, replica, background child).

A oom-score-adj global boolean flag control this feature. In addition,
specific values can be configured using oom-score-adj-values if
additional tuning is required.

(cherry picked from commit 2530dc0ebd8be8d792f4673073401377cd5bdc42)
2020-09-01 09:27:58 +03:00
Arun Ranganathan
e81bac32fd Show threading configuration in INFO output (#7446)
Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit f6cad30bb69b2ad35bb0a870077fac2d4605d727)
2020-09-01 09:27:58 +03:00
Meir Shpilraien (Spielrein)
2257f38b68 This PR introduces a new loaded keyspace event (#7536)
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
(cherry picked from commit 8d826393191399e132bd9e56fb51ed83223cc5ca)
2020-09-01 09:27:58 +03:00
Oran Agra
2165a78d10 Fix rejectCommand trims newline in shared error objects, hung clients (#7714)
65a3307bc (released in 6.0.6) has a side effect, when processCommand
rejects a command with pre-made shared object error string, it trims the
newlines from the end of the string. if that string is later used with
addReply, the newline will be missing, breaking the protocol, and
leaving the client hung.

It seems that the only scenario which this happens is when replying with
-LOADING to some command, and later using that reply from the CONFIG
SET command (still during loading). this will result in hung client.

Refactoring the code in order to avoid trimming these newlines from
shared string objects, and do the newline trimming only in other cases
where it's needed.

Co-authored-by: Guy Benoish <guy.benoish@redislabs.com>
(cherry picked from commit 9fcd9e191e6f54276688fb7c74e1d5c3c4be9a75)
2020-09-01 09:27:58 +03:00
杨博东
113d5ae872 Fix flock cluster config may cause failure to restart after kill -9 (#7674)
After fork, the child process(redis-aof-rewrite) will get the fd opened
by the parent process(redis), when redis killed by kill -9, it will not
graceful exit(call prepareForShutdown()), so redis-aof-rewrite thread may still
alive, the fd(lock) will still be held by redis-aof-rewrite thread, and
redis restart will fail to get lock, means fail to start.

This issue was causing failures in the cluster tests in github actions.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit cbaf3c5bbafd43e009a2d6b38dd0e9fc450a3e12)
2020-09-01 09:27:58 +03:00
Jiayuan Chen
096285ab64 Add optional tls verification (#7502)
Adds an `optional` value to the previously boolean `tls-auth-clients` configuration keyword.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
(cherry picked from commit f31260b0445f5649449da41555e1272a40ae4af7)
2020-09-01 09:27:58 +03:00
grishaf
395d339234 Fix prepareForShutdown function declaration (#7566)
(cherry picked from commit 4126ca466fb9fbf0d84a66e31c9f59f28e2fbff6)
2020-09-01 09:27:58 +03:00
John Sully
d1f45b8ebf Implement use-fork config (fails with diskless repl)
Former-commit-id: f2d5c2bca22e9fd506db123c47b7f60cdded7e2c
2020-08-24 03:17:59 +00:00
John Sully
035cde502a Fast cleanup of snapshots without leaving them forever
Former-commit-id: fdd83b2b49244ed2988b080892ee5cffe9fd2684
2020-08-17 00:33:37 +00:00
John Sully
b71f16471b Allow garbage collection of generic data
Former-commit-id: feadb7fb1845027422bcfca43dbcb6097409b8dc
2020-08-17 00:32:48 +00:00
John Sully
6ba6652fcf Don't try and consolidate snapshots with a depth of 1
Former-commit-id: 26c298bd9bc4e2c6981de5c20284120ea54580c3
2020-08-16 00:26:05 +00:00
John Sully
71506d7e0a Don't free snapshot objects in a critical path (under the AE lock)
Former-commit-id: d0da3d3cb74334cc8a2d14f4bdaef7935181700a
2020-08-16 00:13:19 +00:00
John Sully
34937b0ad5 Rehash efficiency
Former-commit-id: fab383156626ec683881101c22eb2f6c2cea4c5d
2020-08-15 23:05:56 +00:00
John Sully
07c019fd3d Prevent unnecessary copy when overwriting a value from a snapshot
Former-commit-id: 654a7bc6ea82f4ac45a1c1a25c794e1c27c0d902
2020-08-15 22:59:01 +00:00
John Sully
db193a1ef1 Merge branch 'unstable' into keydbpro
Former-commit-id: ae482585f0dc470efd73833f74111c2f87a172c5
2020-08-15 21:29:00 +00:00
John Sully
e87dee8dc7 Add build flag to disable MVCC tstamps
Former-commit-id: f17d178d03f44abcdaddd851a313dd3f7ec87ed5
2020-08-10 06:10:24 +00:00
John Sully
649745924b RocksDB Read Performance Improvements
Former-commit-id: 80cca4869888e048e10e11f1f20796c482c3e5b3
2020-08-09 23:36:20 +00:00
Yossi Gottlieb
7a536c2912 TLS: Session caching configuration support. (#7420)
* TLS: Session caching configuration support.
* TLS: Remove redundant config initialization.

(cherry picked from commit 3e6f2b1a45176ac3d81b95cb6025f30d7aaa1393)
2020-07-20 21:08:26 +03:00
Oran Agra
95ba01b538 RESTORE ABSTTL won't store expired keys into the db (#7472)
Similarly to EXPIREAT with TTL in the past, which implicitly deletes the
key and return success, RESTORE should not store key that are already
expired into the db.
When used together with REPLACE it should emit a DEL to keyspace
notification and replication stream.

(cherry picked from commit 5977a94842a25140520297fe4bfda15e0e4de711)
2020-07-20 21:08:26 +03:00
Oran Agra
05e483cbb3 EXEC always fails with EXECABORT and multi-state is cleared
In order to support the use of multi-exec in pipeline, it is important that
MULTI and EXEC are never rejected and it is easy for the client to know if the
connection is still in multi state.

It was easy to make sure MULTI and DISCARD never fail (done by previous
commits) since these only change the client state and don't do any actual
change in the server, but EXEC is a different story.

Since in the past, it was possible for clients to handle some EXEC errors and
retry the EXEC, we now can't affort to return any error on EXEC other than
EXECABORT, which now carries with it the real reason for the abort too.

Other fixes in this commit:
- Some checks that where performed at the time of queuing need to be re-
  validated when EXEC runs, for instance if the transaction contains writes
  commands, it needs to be aborted. there was one check that was already done
  in execCommand (-READONLY), but other checks where missing: -OOM, -MISCONF,
  -NOREPLICAS, -MASTERDOWN
- When a command is rejected by processCommand it was rejected with addReply,
  which was not recognized as an error in case the bad command came from the
  master. this will enable to count or MONITOR these errors in the future.
- make it easier for tests to create additional (non deferred) clients.
- add tests for the fixes of this commit.

(cherry picked from commit 65a3307bc95aadbc91d85cdf9dfbe1b3493222ca)
2020-07-20 21:08:26 +03:00
John Sully
655bc912e6 Add new storage-provider-options config
Former-commit-id: 195a28beecc6094f959ddafef7fe33f5b55e4047
2020-07-14 04:24:46 +00:00
John Sully
782e675072 Remove unnecessary work from critical path (Latency Fixes)
Former-commit-id: 096a90deb7afe489875d3186f3f8f43e41fea329
2020-07-13 16:03:52 +00:00
John Sully
91a803815d Merge branch 'flash_cache' into keydbpro
Former-commit-id: 2a721ef645921d62b39f1374c0a3f5c92b00fae5
2020-07-13 03:53:34 +00:00
John Sully
6a57593467 Merge branch 'PRO_RELEASE_6' into keydbpro
Former-commit-id: bffe010ea5279bee869bc61cc6d933979e10bbea
2020-07-13 03:32:14 +00:00
John Sully
1ad2d96697 Merge branch 'unstable' into keydbpro
Former-commit-id: 0dafbc254a0efd5ee302d5c58fb2ca0a85110104
2020-07-13 03:31:47 +00:00
John Sully
cfcb5ac5c7 Add the KEYDB.MEXISTS command, see issue #203
Former-commit-id: 5619f515285b08d9c443425de1f3092ae3058d40
2020-07-12 21:42:11 +00:00
John Sully
c5f6cb1ba5 Add multi-master-no-forward command to reduce bus traffic with multi-master
Former-commit-id: d99d06b1250a51ea4bc54f678f451acbb7901e33
2020-07-12 19:25:19 +00:00
John Sully
17661f2382 Implement storage key cache, and writeback memory model
Former-commit-id: 732bd9c153459f1174475ad67de36c399ddbe359
2020-07-11 21:23:48 +00:00
John Sully
52123f9529 Merge branch 'keydbpro' into PRO_RELEASE_6
Former-commit-id: 243dcb3853cc965109cb24a940229db7844cdd11
2020-07-10 04:11:57 +00:00
John Sully
f03b75b005 MVCC scan support filtering by type on the async thread
Former-commit-id: 14f8c0ff686b93976eead5fa6bf526c2eecb5ae0
2020-07-10 03:43:56 +00:00
John Sully
34482220af Fix issue where SCAN misses elements while snapshot is in flight
Former-commit-id: ce005d748ebf0e116d674a96f74d698d17394010
2020-07-10 01:43:51 +00:00
John Sully
d3b72484e4 Initial scan implementation (not tested)
Former-commit-id: 0f3911d56e6878d91b6e3cacc7d637934ffe6099
2020-07-09 12:57:35 -04:00
Muhammad Zahalqa
ca929bb9c1 fixes for robj_sharedptr
1. fix cases where null pointer might be accessed
2. make assignmnet op safe
3. make operator bool explicit (safe bool idiom)
4. make comparison operators symetric

fix robj_sharedptr use in rdb.cpp


Former-commit-id: ede524c0647c0875f1071978f26ff785c8d1183e
2020-06-24 23:18:27 -04:00
antirez
afb8b3c030 LPOS: implement the final design. 2020-06-12 12:08:06 +02:00
Paul Spooren
31c9cd8604 LRANK: Add command (the command will be renamed LPOS).
The `LRANK` command returns the index (position) of a given element
within a list. Using the `direction` argument it is possible to specify
going from head to tail (acending, 1) or from tail to head (decending,
-1). Only the first found index is returend. The complexity is O(N).

When using lists as a queue it can be of interest at what position a
given element is, for instance to monitor a job processing through a
work queue. This came up within the Python `rq` project which is based
on Redis[0].

[0]: https://github.com/rq/rq/issues/1197

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-06-12 12:08:06 +02:00