27289 Commits

Author SHA1 Message Date
YaacovHazan
4af7a28728 stabilize tests that involved with load handlers (#8967)
When test stop 'load handler' by killing the process that generating the load,
some commands that already in the input buffer, still might be processed by the server.
This may cause some instability in tests, that count on that no more commands
processed after we stop the `load handler'

In this commit, new proc 'wait_load_handlers_disconnected' added, to verify that no more
cammands from any 'load handler' prossesed, by checking that the clients who
genreate the load is disconnceted.

Also, replacing check of dbsize with wait_for_ofs_sync before comparing debug digest, as
it would fail in case the last key the workload wrote was an overridden key (not a new one).

Affected tests
Race fix:
- failover command to specific replica works
- Connect multiple replicas at the same time (issue #141), master diskless=$mdl, replica diskless=$sdl
- AOF rewrite during write load: RDB preamble=$rdbpre

Cleanup and speedup:
- Test replication with blocking lists and sorted sets operations
- Test replication with parallel clients writing in different DBs
- Test replication partial resync: $descr (diskless: $mdl, $sdl, reconnect: $reconnect

(cherry picked from commit 32a2584e079a1b3c2d1e6649e38239381a73a459)
2021-07-21 21:06:49 +03:00
YaacovHazan
ff27217639 stabilize tests that involved with load handlers (#8967)
When test stop 'load handler' by killing the process that generating the load,
some commands that already in the input buffer, still might be processed by the server.
This may cause some instability in tests, that count on that no more commands
processed after we stop the `load handler'

In this commit, new proc 'wait_load_handlers_disconnected' added, to verify that no more
cammands from any 'load handler' prossesed, by checking that the clients who
genreate the load is disconnceted.

Also, replacing check of dbsize with wait_for_ofs_sync before comparing debug digest, as
it would fail in case the last key the workload wrote was an overridden key (not a new one).

Affected tests
Race fix:
- failover command to specific replica works
- Connect multiple replicas at the same time (issue #141), master diskless=$mdl, replica diskless=$sdl
- AOF rewrite during write load: RDB preamble=$rdbpre

Cleanup and speedup:
- Test replication with blocking lists and sorted sets operations
- Test replication with parallel clients writing in different DBs
- Test replication partial resync: $descr (diskless: $mdl, $sdl, reconnect: $reconnect

(cherry picked from commit 32a2584e079a1b3c2d1e6649e38239381a73a459)
2021-07-21 21:06:49 +03:00
perryitay
6ae3673b5e Fail EXEC command in case a watched key is expired (#9194)
There are two issues fixed in this commit: 
1. we want to fail the EXEC command in case there is a watched key that's logically
   expired but not yet deleted by active expire or lazy expire.
2. we saw that currently cache time is update in every `call()` (including nested calls),
   this time is being also being use for the isKeyExpired comparison, we want to update
   the cache time only in the first call (execCommand)

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit ac8b1df8850cc80fbf9ce8c2fbde0c1d3a1b4e91)
2021-07-21 21:06:49 +03:00
perryitay
3f4f9b6331 Fail EXEC command in case a watched key is expired (#9194)
There are two issues fixed in this commit: 
1. we want to fail the EXEC command in case there is a watched key that's logically
   expired but not yet deleted by active expire or lazy expire.
2. we saw that currently cache time is update in every `call()` (including nested calls),
   this time is being also being use for the isKeyExpired comparison, we want to update
   the cache time only in the first call (execCommand)

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit ac8b1df8850cc80fbf9ce8c2fbde0c1d3a1b4e91)
2021-07-21 21:06:49 +03:00
Huang Zhw
970302c00c Do not install a file event to send data to rewrite child when parent stop sending diff to child in aof rewrite. (#8767)
In aof rewrite, when parent stop sending data to child, if there is
new rewrite data, aofChildWriteDiffData write event will be installed.
Then this event is issued and deletes the file event without do anyting.
This will happen over and over again until aof rewrite finish.

This bug used to waste a few system calls per excessive wake-up
(epoll_ctl and epoll_wait) per cycle, each cycle triggered by receiving
a write command from a client.

(cherry picked from commit cb961d8c8e10ff3b619f1579a03336a15e9e6f45)
2021-07-21 21:06:49 +03:00
Huang Zhw
5b8e395174 Do not install a file event to send data to rewrite child when parent stop sending diff to child in aof rewrite. (#8767)
In aof rewrite, when parent stop sending data to child, if there is
new rewrite data, aofChildWriteDiffData write event will be installed.
Then this event is issued and deletes the file event without do anyting.
This will happen over and over again until aof rewrite finish.

This bug used to waste a few system calls per excessive wake-up
(epoll_ctl and epoll_wait) per cycle, each cycle triggered by receiving
a write command from a client.

(cherry picked from commit cb961d8c8e10ff3b619f1579a03336a15e9e6f45)
2021-07-21 21:06:49 +03:00
Omer Shadmi
3687f210a1 Avoid exiting to allow diskless loading to recover from RDB short read on module AUX data (#9199)
Currently a replica is able to recover from a short read (when diskless loading
is enabled) and avoid crashing/exiting, replying to the master and then the rdb
could be sent again by the master for another load attempt by the replica.
There were a few scenarios that were not behaving similarly, such as when
there is no end-of-file marker, or when module aux data failed to load, which
should be allowed to occur due to a short read.

(cherry picked from commit f06d782f5abcb30efb0117841232828ed3e129bf)
2021-07-21 21:06:49 +03:00
Omer Shadmi
9914b676b3 Avoid exiting to allow diskless loading to recover from RDB short read on module AUX data (#9199)
Currently a replica is able to recover from a short read (when diskless loading
is enabled) and avoid crashing/exiting, replying to the master and then the rdb
could be sent again by the master for another load attempt by the replica.
There were a few scenarios that were not behaving similarly, such as when
there is no end-of-file marker, or when module aux data failed to load, which
should be allowed to occur due to a short read.

(cherry picked from commit f06d782f5abcb30efb0117841232828ed3e129bf)
2021-07-21 21:06:49 +03:00
Oran Agra
f1ff72c7a3 Fix race in client side tracking (#9116)
The `Tracking gets notification of expired keys` test in tracking.tcl
used to hung in valgrind CI quite a lot.

It turns out the reason is that with valgrind and a busy machine, the
server cron active expire cycle could easily run in the same event loop
as the command that created `mykey`, so that when they key got expired,
there were two change events to broadcast, one that set the key and one
that expired it, but since we used raxTryInsert, the client that was
associated with the "last" change was the one that created the key, so
the NOLOOP filtered that event.

This commit adds a test that reproduces the problem by using lazy expire
in a multi-exec which makes sure the key expires in the same event loop
as the one that added it.

(cherry picked from commit 9b564b525d8ce88295ec14ffdc3bede7e5f5c33e)
2021-07-21 21:06:49 +03:00
Oran Agra
abd44c8393 Fix race in client side tracking (#9116)
The `Tracking gets notification of expired keys` test in tracking.tcl
used to hung in valgrind CI quite a lot.

It turns out the reason is that with valgrind and a busy machine, the
server cron active expire cycle could easily run in the same event loop
as the command that created `mykey`, so that when they key got expired,
there were two change events to broadcast, one that set the key and one
that expired it, but since we used raxTryInsert, the client that was
associated with the "last" change was the one that created the key, so
the NOLOOP filtered that event.

This commit adds a test that reproduces the problem by using lazy expire
in a multi-exec which makes sure the key expires in the same event loop
as the one that added it.

(cherry picked from commit 9b564b525d8ce88295ec14ffdc3bede7e5f5c33e)
2021-07-21 21:06:49 +03:00
Maxim Galushka
9562821f12 redis-cli: support for REDIS_REPLY_SET in CSV and RAW output. (#7338)
Fixes #6792. Added support of REDIS_REPLY_SET in raw and csv output of `./redis-cli`

Test:

run commands to test:
  ./redis-cli -3 --csv COMMAND
  ./redis-cli -3 --raw COMMAND

Now they are returning resuts, were failing with: "Unknown reply type: 10" before the change.

(cherry picked from commit 96bb078577ce2b0d093c873faae5d3ecca26a1de)
2021-07-21 21:06:49 +03:00
Maxim Galushka
f70916f8b0 redis-cli: support for REDIS_REPLY_SET in CSV and RAW output. (#7338)
Fixes #6792. Added support of REDIS_REPLY_SET in raw and csv output of `./redis-cli`

Test:

run commands to test:
  ./redis-cli -3 --csv COMMAND
  ./redis-cli -3 --raw COMMAND

Now they are returning resuts, were failing with: "Unknown reply type: 10" before the change.

(cherry picked from commit 96bb078577ce2b0d093c873faae5d3ecca26a1de)
2021-07-21 21:06:49 +03:00
malavan
69f3e7c802 remove duplicate function
Former-commit-id: 81cf9979ed1d86d0a690e5cd27ed018b125b1d58
2021-07-21 16:00:24 +00:00
malavan
4291d2ddd1 remove duplicate function
Former-commit-id: 81cf9979ed1d86d0a690e5cd27ed018b125b1d58
2021-07-21 16:00:24 +00:00
malavan
99a165e0f6 add stacktrace message
Former-commit-id: 5be1620629ce8de7f419bc001f18d22d6eb4fa8a
2021-07-19 23:17:18 +00:00
malavan
a269f35002 add stacktrace message
Former-commit-id: 5be1620629ce8de7f419bc001f18d22d6eb4fa8a
2021-07-19 23:17:18 +00:00
John Sully
ea19d31df8 Merge branch 'keydbpro' into keydbpro_collab
Former-commit-id: e4e5c6696c6d831924f314a198b266b10d831e14
2021-07-19 20:51:27 +00:00
John Sully
26750791b3 Merge branch 'keydbpro' into keydbpro_collab
Former-commit-id: e4e5c6696c6d831924f314a198b266b10d831e14
2021-07-19 20:51:27 +00:00
John Sully
bacaa204cf Disable async rehash during load as it interferes with eviction
Former-commit-id: 54b4f39e9d634bf53b04cd94433b051b14323bc6
2021-07-19 18:17:54 +00:00
John Sully
795da380a2 Disable async rehash during load as it interferes with eviction
Former-commit-id: 54b4f39e9d634bf53b04cd94433b051b14323bc6
2021-07-19 18:17:54 +00:00
John Sully
345ec75a36 We need to free in order since the first big async free is likely the largest, so don't set the hipri bit
Former-commit-id: 76a9cefa94e0f446e12a690909cbda15d03ca211
2021-07-19 18:01:39 +00:00
John Sully
2609b87a95 We need to free in order since the first big async free is likely the largest, so don't set the hipri bit
Former-commit-id: 76a9cefa94e0f446e12a690909cbda15d03ca211
2021-07-19 18:01:39 +00:00
malavan
16f361abf6 dd comment and set eip unused
Former-commit-id: 685b87e2ef4f67cd599b12e38b08f5b70b7a6c58
2021-07-19 17:32:38 +00:00
malavan
fe2292f111 dd comment and set eip unused
Former-commit-id: 685b87e2ef4f67cd599b12e38b08f5b70b7a6c58
2021-07-19 17:32:38 +00:00
malavan
70cb3267de dd comment and set eip unused
Former-commit-id: 72429c21067318696897c26783f654eb7bc6040f
2021-07-19 17:31:24 +00:00
malavan
4bc552fba2 dd comment and set eip unused
Former-commit-id: 72429c21067318696897c26783f654eb7bc6040f
2021-07-19 17:31:24 +00:00
malavan
9e411426f2 dd comment and set eip unused
Former-commit-id: 7707fe15a9c9774820b3afb9837a50e73c40a5a8
2021-07-19 17:29:43 +00:00
malavan
7d4b413958 dd comment and set eip unused
Former-commit-id: 7707fe15a9c9774820b3afb9837a50e73c40a5a8
2021-07-19 17:29:43 +00:00
malavan
abf2114117 add backtrace for alpine using libunwind
Former-commit-id: c583ffff412b9df705b7c96e68e3908b645c7d43
2021-07-19 17:20:52 +00:00
malavan
c49a46ca8a add backtrace for alpine using libunwind
Former-commit-id: c583ffff412b9df705b7c96e68e3908b645c7d43
2021-07-19 17:20:52 +00:00
malavan
fedf45675c Merge branch 'add_ci' of https://gitlab.eqalpha.com/keydb-dev/KeyDB-Pro into add_ci
Former-commit-id: 82438d8332e4f23c67df17f888b87aff04a46363
2021-07-19 17:01:17 +00:00
malavan
cf0a4a5da7 Merge branch 'add_ci' of https://gitlab.eqalpha.com/keydb-dev/KeyDB-Pro into add_ci
Former-commit-id: 82438d8332e4f23c67df17f888b87aff04a46363
2021-07-19 17:01:17 +00:00
malavan
697f17f91d split up runtests
Former-commit-id: 93e36a064033dc55fe281fe113425b33cf42b2c3
2021-07-19 16:59:03 +00:00
malavan
41fba215df split up runtests
Former-commit-id: 93e36a064033dc55fe281fe113425b33cf42b2c3
2021-07-19 16:59:03 +00:00
John Sully
88f5bf1d90 We need to periodically flush the GC or we'll end up blocking with a huge backlog at the end of load
Former-commit-id: 29c0bf79ad1a810e808790de2f7db24f3cc603e8
2021-07-19 16:50:48 +00:00
John Sully
a3cf05574f We need to periodically flush the GC or we'll end up blocking with a huge backlog at the end of load
Former-commit-id: 29c0bf79ad1a810e808790de2f7db24f3cc603e8
2021-07-19 16:50:48 +00:00
Malavan Sotheeswaran
105fc01d39 Merge branch 'keydbpro' into 'add_ci'
# Conflicts:
#   .gitlab-ci.yml

Former-commit-id: 828924d67d622483fccf570649fc81aa3d78673b
2021-07-19 16:16:36 +00:00
Malavan Sotheeswaran
57c97e0b5b Merge branch 'keydbpro' into 'add_ci'
# Conflicts:
#   .gitlab-ci.yml

Former-commit-id: 828924d67d622483fccf570649fc81aa3d78673b
2021-07-19 16:16:36 +00:00
malavan
e11d7f25d1 change make test to multithreaded
Former-commit-id: 8f2bae851f2aa7c2600a487733a82850d6552882
2021-07-19 15:52:33 +00:00
malavan
88b781b007 change make test to multithreaded
Former-commit-id: 8f2bae851f2aa7c2600a487733a82850d6552882
2021-07-19 15:52:33 +00:00
John Sully
d3793efb33 Info command should show how many keys are cached in RAM vs storage provider
Former-commit-id: 08597bee69bc16ca7c3d5ff31020472774c6eec9
2021-07-19 15:11:33 +00:00
John Sully
fc7221739c Info command should show how many keys are cached in RAM vs storage provider
Former-commit-id: 08597bee69bc16ca7c3d5ff31020472774c6eec9
2021-07-19 15:11:33 +00:00
John Sully
614860ce3c StorageCache dtor leaks
Former-commit-id: 0262c4dc76a320141b8a4454df2f6baab4f74ab3
2021-07-19 15:10:48 +00:00
John Sully
bd74913b8e StorageCache dtor leaks
Former-commit-id: 0262c4dc76a320141b8a4454df2f6baab4f74ab3
2021-07-19 15:10:48 +00:00
John Sully
4000334b1f Do not resize replica buffer past the max client limit
Former-commit-id: ba116500ca4fd53e4e40f04fc33981e60bb21ab7
2021-07-18 20:48:08 +00:00
John Sully
48b30ec34a Do not resize replica buffer past the max client limit
Former-commit-id: ba116500ca4fd53e4e40f04fc33981e60bb21ab7
2021-07-18 20:48:08 +00:00
John Sully
5f72ce9317 Return the ring buffer to its original size if we temporarily resized it
Former-commit-id: a12ce4a0d105bf7d6ccff95f7dc0044c4676b0a7
2021-07-18 20:45:32 +00:00
John Sully
29ef9708fa Return the ring buffer to its original size if we temporarily resized it
Former-commit-id: a12ce4a0d105bf7d6ccff95f7dc0044c4676b0a7
2021-07-18 20:45:32 +00:00
John Sully
0b220bddc7 Do not update batch variables when not in a batch
Former-commit-id: ad1e0286cf9b2d9de33c65e8e798a05ead3f7d5a
2021-07-18 20:28:42 +00:00
John Sully
653f7c96a0 Do not update batch variables when not in a batch
Former-commit-id: ad1e0286cf9b2d9de33c65e8e798a05ead3f7d5a
2021-07-18 20:28:42 +00:00