21038 Commits

Author SHA1 Message Date
John Sully
67003ec714 Disarm not release D'oh
Former-commit-id: 3e33fbcd351a719126d30405c4dc9209ad381954
2020-10-27 06:41:33 +00:00
John Sully
588fa164c3 Disarm not release D'oh
Former-commit-id: 3e33fbcd351a719126d30405c4dc9209ad381954
2020-10-27 06:41:33 +00:00
WuYunlong
2184dffaf3 Speedup cluster failover. (#7948)
This commit deals with manual failover as well as non-manual failover.

We did tests with manual failover as follows:
1, Setup redis cluster which holds 16 partions, each having only
   1 corresponding replica.
2, Write a batch of data to redis cluster and make sure the redis is doing
   a active expire in serverCron.
3, Do a manual failover sequentially to each partions with a time interval
   of 3 minutes.
4, Collect logs and do some computaiton work.

The result:
case    avgTime    maxTime    minTime
C1      95.8ms	   227ms      25ms
C2      47.9ms     96ms       12ms
C3      12.6ms     27ms       7ms

Explanation
case C1: All nodes use the version before optimization
case C2: Masters use the elder version while replicas use the optimized version
case C3: All nodes use the optimized version
failover time: The time between when replica got a `manual failover request` and
               when it `won the failover election`.
avgTime: average failover time
maxTime: maximum failover time
minTime: mimimum failover time
ms: millisecond

Co-authored-by: chendq8 <c.d_q@163.com>
2020-10-27 08:13:59 +02:00
WuYunlong
7fa56dd773
Speedup cluster failover. (#7948)
This commit deals with manual failover as well as non-manual failover.

We did tests with manual failover as follows:
1, Setup redis cluster which holds 16 partions, each having only
   1 corresponding replica.
2, Write a batch of data to redis cluster and make sure the redis is doing
   a active expire in serverCron.
3, Do a manual failover sequentially to each partions with a time interval
   of 3 minutes.
4, Collect logs and do some computaiton work.

The result:
case    avgTime    maxTime    minTime
C1      95.8ms	   227ms      25ms
C2      47.9ms     96ms       12ms
C3      12.6ms     27ms       7ms

Explanation
case C1: All nodes use the version before optimization
case C2: Masters use the elder version while replicas use the optimized version
case C3: All nodes use the optimized version
failover time: The time between when replica got a `manual failover request` and
               when it `won the failover election`.
avgTime: average failover time
maxTime: maximum failover time
minTime: mimimum failover time
ms: millisecond

Co-authored-by: chendq8 <c.d_q@163.com>
2020-10-27 08:13:59 +02:00
John Sully
2ae013910b Prevent crashes on shutdown due to lock being held
Former-commit-id: 6a74f524e558100dfb3e54779020cd3407706e08
2020-10-27 04:53:19 +00:00
John Sully
b226b318ff Prevent crashes on shutdown due to lock being held
Former-commit-id: 6a74f524e558100dfb3e54779020cd3407706e08
2020-10-27 04:53:19 +00:00
John Sully
366ef70f6e Ensure MVCC timestamp is incremented after the real time is updated
Former-commit-id: 9c5b59ac010fa1e948d5e51d1cd408c7e743d66d
2020-10-27 04:52:35 +00:00
John Sully
cba89cf9f9 Ensure MVCC timestamp is incremented after the real time is updated
Former-commit-id: 9c5b59ac010fa1e948d5e51d1cd408c7e743d66d
2020-10-27 04:52:35 +00:00
Madelyn Olson
b9a519190d Only supress implitic fallthrough on GCC 7 2020-10-26 21:46:50 -07:00
Madelyn Olson
dac26729a9 Only supress implitic fallthrough on GCC 7 2020-10-26 21:46:50 -07:00
John Sully
a3323d2b5b Fix multithreaded test failure
Former-commit-id: 1840601f8efb27174efa0a66f78de8c490b5bba4
2020-10-27 01:54:13 +00:00
John Sully
0e44e11e3a Fix multithreaded test failure
Former-commit-id: 1840601f8efb27174efa0a66f78de8c490b5bba4
2020-10-27 01:54:13 +00:00
Yossi Gottlieb
05a086617d Fix wrong zmalloc_size() assumption. (#7963)
When using a system with no malloc_usable_size(), zmalloc_size() assumed
that the heap allocator always returns blocks that are long-padded.

This may not always be the case, and will result with zmalloc_size()
returning a size that is bigger than allocated. At least in one case
this leads to out of bound write, process crash and a potential security
vulnerability.

Effectively this does not affect the vast majority of users, who use
jemalloc or glibc.

This problem along with a (different) fix was reported by Drew DeVault.
2020-10-26 14:49:08 +02:00
Yossi Gottlieb
9824fe3e39
Fix wrong zmalloc_size() assumption. (#7963)
When using a system with no malloc_usable_size(), zmalloc_size() assumed
that the heap allocator always returns blocks that are long-padded.

This may not always be the case, and will result with zmalloc_size()
returning a size that is bigger than allocated. At least in one case
this leads to out of bound write, process crash and a potential security
vulnerability.

Effectively this does not affect the vast majority of users, who use
jemalloc or glibc.

This problem along with a (different) fix was reported by Drew DeVault.
2020-10-26 14:49:08 +02:00
filipe oliveira
f307c73ea6 redis-benchmark: add tests, --version, a minor bug fixes (#7947)
- add test suite coverage for redis-benchmark
- add --version (similar to what redis-cli has)
- fix bug sending more requests than intended when pipeline > 1.
- when done sending requests, avoid freeing client in the write handler, in theory before
  responses are received (probably dead code since the read handler will call clientDone first)

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-10-26 08:04:59 +02:00
filipe oliveira
01acfa71ca
redis-benchmark: add tests, --version, a minor bug fixes (#7947)
- add test suite coverage for redis-benchmark
- add --version (similar to what redis-cli has)
- fix bug sending more requests than intended when pipeline > 1.
- when done sending requests, avoid freeing client in the write handler, in theory before
  responses are received (probably dead code since the read handler will call clientDone first)

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-10-26 08:04:59 +02:00
John Sully
418b6206ea Avoid unnecessary memory fences
Former-commit-id: 88962af8b2fe835fb5d542013062cec8c771c6c6
2020-10-26 02:28:38 +00:00
John Sully
c6ebf961f4 Avoid unnecessary memory fences
Former-commit-id: 88962af8b2fe835fb5d542013062cec8c771c6c6
2020-10-26 02:28:38 +00:00
Itamar Haber
79594252e5 Adds command introspection to Sentinel (#7940)
Adds the `COMMAND` command to Sentinel.
2020-10-26 00:37:58 +02:00
Itamar Haber
d2af0f25be
Adds command introspection to Sentinel (#7940)
Adds the `COMMAND` command to Sentinel.
2020-10-26 00:37:58 +02:00
David CARLIER
c1ff07dfbb cpu affinity: DragonFlyBSD support (#7956) 2020-10-25 14:14:05 +02:00
David CARLIER
27f4c212f3
cpu affinity: DragonFlyBSD support (#7956) 2020-10-25 14:14:05 +02:00
John Sully
19a4d2d358 Remove addReply*Async methods since we already know if its async or not. This is just a source of bugs
Former-commit-id: df22cdf6e91a1b9c390b69c4209c719ecf1e44f1
2020-10-24 02:18:03 +00:00
John Sully
b336d269ca Remove addReply*Async methods since we already know if its async or not. This is just a source of bugs
Former-commit-id: df22cdf6e91a1b9c390b69c4209c719ecf1e44f1
2020-10-24 02:18:03 +00:00
John Sully
7a9fbad132 Allow the locker to be used even without a client
Former-commit-id: 041446005b03121adf7ac061bd0fd2ec70d9418e
2020-10-23 20:21:49 +00:00
John Sully
95606ffb10 Allow the locker to be used even without a client
Former-commit-id: 041446005b03121adf7ac061bd0fd2ec70d9418e
2020-10-23 20:21:49 +00:00
John Sully
0691608ba0 Fix multithread test failures
Former-commit-id: 7c39a9b0e193f5c0b327ff21cd49210037142642
2020-10-23 15:57:39 +00:00
John Sully
c88a19bc5e Fix multithread test failures
Former-commit-id: 7c39a9b0e193f5c0b327ff21cd49210037142642
2020-10-23 15:57:39 +00:00
WuYunlong
51f36283df Update rdb_last_bgsave_time_sec in INFO on diskless replication (#7917)
`info Persistence` will include correct (updated) rdb_last_bgsave_time_sec
For diskless bgsave (sockets) too (like a few other persistence info fields).

Refactor code to reduce duplicate code.
2020-10-23 15:26:30 +03:00
WuYunlong
e05a7df7f9
Update rdb_last_bgsave_time_sec in INFO on diskless replication (#7917)
`info Persistence` will include correct (updated) rdb_last_bgsave_time_sec
For diskless bgsave (sockets) too (like a few other persistence info fields).

Refactor code to reduce duplicate code.
2020-10-23 15:26:30 +03:00
Wen Hui
f61435fe8b do not add save parameter during config rewrite in sentinel mode (#7945)
Previous code would have added default redis save parameters
to the config file on rewrite, which would have been silently ignored
when the config file is loaded.

The new code avoids adding this, and also actively removes these lines
If added by a previous config rewrite.
2020-10-22 19:47:32 +03:00
Wen Hui
0f370f9b66
do not add save parameter during config rewrite in sentinel mode (#7945)
Previous code would have added default redis save parameters
to the config file on rewrite, which would have been silently ignored
when the config file is loaded.

The new code avoids adding this, and also actively removes these lines
If added by a previous config rewrite.
2020-10-22 19:47:32 +03:00
Qu Chen
44bcbed2ee WATCH no longer ignores keys which have expired for MULTI/EXEC. (#7920)
This wrong behavior was backed by a test, and also documentation, and dates back to 2010.
But it makes no sense to anyone involved so it was decided to change that.

Note that 3b031b1 (invalidate watch on expire on access) was released in 6.0 RC2
and 62a3ec8 released in in 6.0.0 GA (invalidate watch when key is evicted).
both of which do similar changes.
2020-10-22 12:57:45 +03:00
Qu Chen
556acefe75
WATCH no longer ignores keys which have expired for MULTI/EXEC. (#7920)
This wrong behavior was backed by a test, and also documentation, and dates back to 2010.
But it makes no sense to anyone involved so it was decided to change that.

Note that 20eeddf (invalidate watch on expire on access) was released in 6.0 RC2
and 2d1968f released in in 6.0.0 GA (invalidate watch when key is evicted).
both of which do similar changes.
2020-10-22 12:57:45 +03:00
John Sully
5fb0c880a8 Remove leak warning with ASAN in the CLI
Former-commit-id: 05cd92f7661ba1a96cdcac278720b7da0b6ba60e
2020-10-21 21:16:08 +00:00
John Sully
dcd6d809aa Remove leak warning with ASAN in the CLI
Former-commit-id: 05cd92f7661ba1a96cdcac278720b7da0b6ba60e
2020-10-21 21:16:08 +00:00
filipe oliveira
128bc0f495 Fixed bug concerning redis-benchmark non clustered benchmark forcing always the same hash tag {tag} (#7931)
Adding the ":{tag}" only if --cluster is used, so that when used against
a proxy it generates traffic to all shards.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-10-20 19:52:05 +03:00
filipe oliveira
6cf23d6610
Fixed bug concerning redis-benchmark non clustered benchmark forcing always the same hash tag {tag} (#7931)
Adding the ":{tag}" only if --cluster is used, so that when used against
a proxy it generates traffic to all shards.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-10-20 19:52:05 +03:00
Oran Agra
cc231e4882 fix 32bit build warnings (#7926) 2020-10-20 09:12:24 +03:00
Oran Agra
a425e1d26d
fix 32bit build warnings (#7926) 2020-10-20 09:12:24 +03:00
Wen Hui
ee8cb472a7 fix double fclose in aofrewrite (#7919)
minor fix for a bug which happen on error handling code
and doesn't look like it could have caused any real harm
(fd number wouldn't have been reused yet)
2020-10-19 15:32:18 +03:00
Wen Hui
04a0af9085
fix double fclose in aofrewrite (#7919)
minor fix for a bug which happen on error handling code
and doesn't look like it could have caused any real harm
(fd number wouldn't have been reused yet)
2020-10-19 15:32:18 +03:00
Wen Hui
4db692c059 Support ACL for Sentinel Mode (#7888)
This commit implements ACL for Sentinel mode, main work of this PR includes:

- Update Sentinel command table in order to better support ACLs.
- Fix couple of things which currently blocks the support for ACL on sentinel mode.
- Provide "sentinel sentinel-user" and "sentinel sentinel-pass " configuration in order to let sentinel authenticate with a specific user in other sentinels.
- requirepass is kept just for compatibility with old config files

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-10-19 07:33:55 +03:00
Wen Hui
0047702aab
Support ACL for Sentinel Mode (#7888)
This commit implements ACL for Sentinel mode, main work of this PR includes:

- Update Sentinel command table in order to better support ACLs.
- Fix couple of things which currently blocks the support for ACL on sentinel mode.
- Provide "sentinel sentinel-user" and "sentinel sentinel-pass " configuration in order to let sentinel authenticate with a specific user in other sentinels.
- requirepass is kept just for compatibility with old config files

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-10-19 07:33:55 +03:00
Oran Agra
0de424cab9 INFO report peak memory before eviction (#7894)
In some cases one command added a very big bulk of memory, and this
would be "resolved" by the eviction before the next command.

Seeing an unexplained mass eviction we would wish to
know the highest momentary usage too.

Tracking it in call() and beforeSleep() adds some hooks in AOF and RDB
loading.

The fix in clientsCronTrackExpansiveClients is related to #7874
2020-10-18 16:56:43 +03:00
Oran Agra
457b7073b5
INFO report peak memory before eviction (#7894)
In some cases one command added a very big bulk of memory, and this
would be "resolved" by the eviction before the next command.

Seeing an unexplained mass eviction we would wish to
know the highest momentary usage too.

Tracking it in call() and beforeSleep() adds some hooks in AOF and RDB
loading.

The fix in clientsCronTrackExpansiveClients is related to #7874
2020-10-18 16:56:43 +03:00
Wen Hui
38f55cc3aa support NOMKSTREAM option in xadd command (#7910)
introduces a NOMKSTREAM option for xadd command, this would be useful for some
use cases when we do not want to create new stream by default:

XADD key [MAXLEN [~|=] <count>] [NOMKSTREAM] <ID or *> [field value] [field value]
2020-10-18 10:15:43 +03:00
Wen Hui
f328194d12
support NOMKSTREAM option in xadd command (#7910)
introduces a NOMKSTREAM option for xadd command, this would be useful for some
use cases when we do not want to create new stream by default:

XADD key [MAXLEN [~|=] <count>] [NOMKSTREAM] <ID or *> [field value] [field value]
2020-10-18 10:15:43 +03:00
Pierre Jambet
1cfb63b699 t_set.c comment update for srandmemberWithCountCommand (#7922)
Reference the correct "case", case 4, in the comment explaining the need
for case 3, when the number of request items is too close to the
cardinality of the set. Case 4 is indeed the "natural approach"
referenced earlier in that sentence.
2020-10-18 08:14:45 +03:00
Pierre Jambet
f6010e106b
t_set.c comment update for srandmemberWithCountCommand (#7922)
Reference the correct "case", case 4, in the comment explaining the need
for case 3, when the number of request items is too close to the
cardinality of the set. Case 4 is indeed the "natural approach"
referenced earlier in that sentence.
2020-10-18 08:14:45 +03:00