14038 Commits

Author SHA1 Message Date
Oran Agra
30775bc3e3
solve race in replication-2 test - again (#8491)
this should make it timing independent and also faster in most cases
2021-02-15 12:50:23 +02:00
Viktor Söderqvist
0bc8c9c8f9
Modules: In RM_HashSet, add COUNT_ALL flag and set errno (#8446)
The added flag affects the return value of RM_HashSet() to include
the number of inserted fields, in addition to updated and deleted
fields.

errno is set on errors, tests are added and documentation updated.
2021-02-15 11:40:05 +02:00
filipe oliveira
efccd6353b
redis-benchmark: Fix broken protocol when used with -a or --dbnum (#8486)
Fix the pointers to the slot hash tags in the case of prefixed commands usage
i.e. AUTH / SELECT

It adjusts the pointers to the slot hash tags in the case of prefixed commands
usage as soon as we get the 1st reply (same like we already did for the random
strings within the command )
2021-02-14 14:42:41 +02:00
Yossi Gottlieb
8c42d1257f
Fix errors with sentinel leaked fds test. (#8482)
* Don't run test script on non-Linux.
* Verify that reported fds do indeed exist also in parent, to avoid
  false negatives on some systems (namely CentOS).

Co-authored-by: Andy Pan <panjf2000@gmail.com>
2021-02-11 15:25:01 +02:00
Yossi Gottlieb
94bc26e652
Fix duplicate replicas issue. (#8481)
We need to store replicas referenced by their announced address (IP or
address). Before that, if hostnames were used and the IP address
changed, duplicate entries would have been created.
2021-02-11 11:50:47 +02:00
Yossi Gottlieb
29ac9aea5d
Fix Sentinel configuration rewrite. (#8480)
The `resolve-hostnames` and `announce-hostnames` parameters were not
specified correctly according to the new convention introduced by
1aad55b66.
2021-02-10 18:38:10 +02:00
filipe oliveira
b5ca1e9e53
Removed time sensitive checks from block on background tests. Fixed uninitialized variable (#8479)
- removes time sensitive checks from block on background tests during leak checks.
- fix uninitialized variable on RedisModuleBlockedClient() when calling
  RM_BlockedClientMeasureTimeEnd() without RM_BlockedClientMeasureTimeStart()
2021-02-10 08:59:07 +02:00
Madelyn Olson
899c85ae67
Moved most static strings into the shared structure (#8411)
Moved most static strings into the shared structure
2021-02-09 11:52:28 -08:00
Itamar Haber
4554a49d32
Adds code of conduct (#8471) 2021-02-09 14:38:09 +02:00
WuYunlong
203f357c32
Cleanup in redis-cli and tests: release memory on exit, change dup test name (#8475)
1. Rename 18-cluster-nodes-slots.tcl to 19-cluster-nodes-slots.tcl.
  it was conflicting with another test prefixed by 18
2. Release memory on exit in redis-cli.c.
3. Fix freeConvertedSds indentation.
2021-02-09 12:36:09 +02:00
John Sully
6ecabe823f Fix polarity issue in async rehash
Former-commit-id: dfea168499f0b7829607faec49c491d5072511bc
2021-02-09 05:10:22 +00:00
Huang Zw
8f9958dc24
Fix typo and some out of date comments (#8449)
Fix typo and some out of date comments
2021-02-08 09:29:32 -08:00
Yossi Gottlieb
dbcc0a85d0
Fix and cleanup Sentinel leaked fds test. (#8469)
* For consistency, use tclsh for the script as well
* Ignore leaked fds that originate from grandparent process, since we
  only care about fds redis-sentinel itself is responsible for
* Check every test iteration to catch problems early
* Some cleanups, e.g. parameterization of file name, etc.
2021-02-08 17:02:46 +02:00
filipe oliveira
b2351ea0dc
[fix] Increasing block on background timeout time to avoid test failure (#8470)
The test failed from time to time on Github actions.
We think it's possible that on the module's blocking timeout
time tracking test, the timeout is happening prior we issue the
RedisModule_BlockedClientMeasureTimeStart(bc) on the
background thread. If that is the case one possible solution
is to increase the timeout.
Increasing to 200ms to 500ms to see if nightly stops failing.
2021-02-08 16:24:00 +02:00
sundb
1f12be3072
Fix random probability check in ziplistRandomPairsUnique (#8467)
When (remaining == (total_size - index)), element will definitely be random to.
But when rand() == RAND_MAX, the element will miss, this will trigger assert
in serverAssert(ziplistRandomPairsUnique(zsetobj->ptr, count, keys, vals) == count).
2021-02-08 12:41:16 +02:00
Andy Pan
88272cf7ac
Fix typos in comments (#8466) 2021-02-08 12:09:39 +02:00
John Sully
bb733b0aab Fix bug in fastlock stack metadata
Former-commit-id: 241d1bd7e1ed64885adbb07653a5c7e8ef882607
2021-02-07 19:11:05 -05:00
John Sully
7d588e637a Disable multithreaded rehash when active defrag is enabled. The two are not compatible
Former-commit-id: 56addcd17262ffbaefea26c0097cde53c616ece2
2021-02-07 19:11:05 -05:00
John Sully
3970f4a5a7 The server should recalibrate periodically. Also reduce log noise
Former-commit-id: 8d8e1810d49da6aa921d2327cb4ea250c2b5b234
2021-02-07 19:11:05 -05:00
John Sully
8ce03686ed Ensure the C lock implementation behaves the same as the ASM
Former-commit-id: b5ddc11c46d2eabd28fae4c69927c356dd18bf6e
2021-02-07 19:11:05 -05:00
John Sully
7c700f1987 Ensure rehash completes even when we're in a long running task
Former-commit-id: f107746e90f7a8ff3c7094145ee1ad438911e8c2
2021-02-07 19:11:05 -05:00
John Sully
bef72e5f6a Implement rehash during spinlock
Former-commit-id: f68a26381a35b27948046d46c2c7bcfbdc21143d
2021-02-07 19:11:05 -05:00
John Sully
5ab1095022 Allow multiple threads to rehash simultaneously
Former-commit-id: 5a2cc90786dfd1bfd341dbf5713bcde01f0cfff3
2021-02-07 19:11:05 -05:00
John Sully
5d161ca2f2 Fix races in free
Former-commit-id: 3881d5c1f22f022855c0abcf7b0c2070e204c0a3
2021-02-07 19:11:05 -05:00
John Sully
c6c1084dd8 Initial implementation
Former-commit-id: 958f2c00c8efc15dc91fdeec2ff2e2ae2016c124
2021-02-07 19:11:05 -05:00
John Sully
e55ca3603c Fix memory leak in mvccRestore
Former-commit-id: 165333b0fc648c79e66f04d9c8c4a1d0059fe57a
2021-02-07 23:38:49 +00:00
Oran Agra
62b1f32062
Optimize HRANDFIELD and ZRANDMEMBER case 4 when ziplist encoded (#8444)
It is inefficient to repeatedly pick a single random element from a
ziplist.
For CASE4, which is when the user requested a low number of unique
random picks from the collectoin, we used thta pattern.

Now we use a different algorithm that picks unique elements from a
ziplist, and guarentee no duplicate but doesn't provide random order
(which is only needed in the non-unique random picks case)

Unrelated changes:
* change ziplist count and indexes variables to unsigned
* solve compilation warnings about uninitialized vars in gcc 10.2

Co-authored-by: xinluton <xinluton@qq.com>
2021-02-07 16:55:11 +02:00
Oran Agra
02ab14cc2e
solve race in replication-2 test (#8461)
use SIGSTOP instead of DEBUG SLEEP, reduces the test
time by some 2 seconds and avoids failures on slow machines
2021-02-07 16:22:30 +02:00
Oran Agra
018f7b7378
Update CI on Ubuntu to tcl8.6 (since 20.04 is now used) (#8460)
Github started shifting some repositoreis to use ubuntu 20.04 by default
tcl8.5 is missing in these, but 8.6 exists in both 20.04 and 18.04
2021-02-07 15:41:49 +02:00
Yossi Gottlieb
5b8350aaaa
Add --dump-logs tests option. (#8459)
Dump the entire server log if a test failed, to easy troubleshooting
with no access to log files.
2021-02-07 12:37:24 +02:00
Yossi Gottlieb
be83bb13a8
Add --insecure option to command line tools. (#8416)
Disable certificate validation, making it possible to connect to servers
without configuring full trust chain.

The use of this option is insecure and makes the connection vulnerable
to man in the middle attacks.
2021-02-07 12:36:56 +02:00
Viktor Söderqvist
aea6e71ef8
RM_ZsetRem: Delete key if empty (#8453)
Without this fix, RM_ZsetRem can leave empty sorted sets which are
not allowed to exist.

Removing from a sorted set while iterating seems to work (while
inserting causes failed assetions). RM_ZsetRangeEndReached is
modified to return 1 if the key doesn't exist, to terminate
iteration when the last element has been removed.
2021-02-05 19:54:01 +02:00
filipe oliveira
b3bdcd2278
Fix compiler warning on implicit declaration of ‘nanosleep’ . Removed unused variable (#8454) 2021-02-05 19:51:31 +02:00
sundb
18ac41973b
RAND* commands: fix risk of OOM panic in hash and zset, use fair random in hash, and add tests for even distribution to all (#8429)
Changes to HRANDFIELD and ZRANDMEMBER:
* Fix risk of OOM panic when client query a very big negative count (avoid allocating huge temporary buffer).
* Fix uneven random distribution in HRANDFIELD with negative count (wasn't using dictGetFairRandomKey).
* Add tests to check an even random distribution (HRANDFIELD, SRANDMEMBER, ZRANDMEMBER).

Co-authored-by: Oran Agra <oran@redislabs.com>
2021-02-05 15:56:20 +02:00
Yang Bodong
b7b23a0ff5
Fix GEOSEARCH tcl test error (#8451)
Issue with new test due to longitude wraparound.
2021-02-04 19:39:07 +02:00
Yang Bodong
ded1655d49
GEOSEARCH bybox bug fixes and new fuzzy tester (#8445)
Fix errors of GEOSEARCH bybox search due to:
1. projection of the box to a trapezoid (when the meter box is converted to long / lat it's no longer a box).
2. width and height mismatch

Changes:
- New GEOSEARCH point in rectangle algorithm
- Fix GEOSEARCH bybox width and height mismatch bug
- Add GEOSEARCH bybox testing to the existing "GEOADD + GEORANGE randomized test"
- Add new fuzzy test to stress test the bybox corners and edges
- Add some tests for edge cases of the bybox algorithm

Co-authored-by: Oran Agra <oran@redislabs.com>
2021-02-04 18:08:35 +02:00
Yossi Gottlieb
52fb306535
Fix 32-bit test modules build. (#8448) 2021-02-04 11:37:28 +02:00
Yossi Gottlieb
de6f3ad017
Fix FreeBSD tests and CI Daily issues. (#8438)
* Add bash temporarily to allow sentinel fd leaks test to run.
* Use vmactions-freebsd rdist sync to work around bind permission denied
  and slow execution issues.
* Upgrade to tcl8.6 to be aligned with latest Ubuntu envs.
* Concat all command executions to avoid ignoring failures.
* Skip intensive fuzzer on FreeBSD. For some yet unknown reason, generate_fuzzy_traffic_on_key causes TCL to significantly bloat on FreeBSD resulting with out of memory.
2021-02-03 17:35:28 +02:00
Jonah H. Harris
a3718cde06
Optimizing sorted GEORADIUS COUNT with partial sorting. (#8326)
This commit provides an optimization, in terms of time, for all GEORADIUS*
and GEOSEARCH* searches which utilize the default, sorted, COUNT clause.
This is commonly used for nearest-neighbor (top-K points closest to a given lat/lon)
searches. While the previous implementation appends all matching points to the
geoPoint array and performs pruning after-the-fact via a full sort and [0, count)-based
for-loop, this PR sorts only the required number of elements.

This optimization provides a 5-20% improvement in runtime depending on the
density of points of interest (POI) as well as the radius searched.
No performance degradation has been observed.
2021-02-02 10:57:12 +02:00
Huang Zw
9760475a39
Cleanup: addReplyAggregateLen and addReplyBulkLen remove redundant check (#8431)
addReplyLongLongWithPrefix, has a check against negative length, and the code
flow removed in this commit bypasses the check.
addReplyAggregateLen has an assertion for negative length, but addReplyBulkLen
does not, so this commit fixes theoretical case of access violation (probably
unreachable though)
2021-02-02 10:54:19 +02:00
zhaozhao.zz
f2a5fe3678
XINFO should use lookupKeyReadOrReply (#8436)
This bug would have let users observe logically expired keys on replicas and
during CLIENT PAUSE WRITE.
2021-02-02 10:51:19 +02:00
John Sully
26898b43a1 Module test reliability
Former-commit-id: cc827e157a8cdd0224eaa76d6a70e0cccd95f544
2021-02-01 22:21:49 +00:00
John Sully
23495c22a5 Simplify the handling of CreateTimer
Former-commit-id: 7b4e25008a352bd425582a3e60b26894826af626
2021-02-01 15:51:22 -05:00
VivekSainiEQ
55659b1569 Modified RM_CreateTimer to prevent more that one function being posted at a time
Former-commit-id: f66e5a9c3d6925c3c6d383fbc43fbfbd56721dad
2021-02-01 15:51:22 -05:00
VivekSainiEQ
ad11549020 Changed RM_CreateTimer to only call aePostFunction is existing aePostFunction isn't in flight
Former-commit-id: 9954f5b4a48286d07fb876fd9579801365b6c237
2021-02-01 15:51:22 -05:00
VivekSainiEQ
73e8825266 Now post entire timer installation process as one function to make atomic with respect to global locks
Former-commit-id: 53936661c88bd7eac88308afc75c510134a8e044
2021-02-01 15:51:22 -05:00
VivekSainiEQ
e3eec7e37c Updated header file to remove fSynchronous flag
Former-commit-id: e2552ff8a92ea5adf7cea070b48afc573003254d
2021-02-01 15:51:22 -05:00
VivekSainiEQ
eb83f3d47b Removed more uses of fSynchronous and the use of condition variable and mutex on the control struct.
Former-commit-id: 6ab08cc3e1429178b26b55ed7aa8ba85240eb766
2021-02-01 15:51:22 -05:00
VivekSainiEQ
363293ef61 Added fix for scenario where module thread waiting for s_mutexModule in acquireGIL can deadlock with module thread waiting for s_mutex in releaseGIL
Former-commit-id: 3205373bb378f895824cc1936a6bae663b1abdcc
2021-02-01 15:51:22 -05:00
VivekSainiEQ
cba4dcffdf removed synchronous calls to aePostFunction and changed scope of g_fModuleThread in order to prevent module related deadlocks, issue #214
Former-commit-id: 3b8d1f7076e4ab2082cd0768abc7b0b6ed4f951a
2021-02-01 15:51:22 -05:00