gvsafronov/futriix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,287 Commits 1 Branch 0 Tags
Commit Graph

1322 Commits

Author SHA1 Message Date
John Sully
4650e051bd Use the concurrentqueue for multithread load instead of mutex and cvs 
Former-commit-id: d5a59113dbfedaf7b62a650cff58a2e8ec01826f
 2021-10-04 07:33:03 +00:00
jsully
7cf8433745 Merge branch 'improve_overwrite_performance' into 'keydbpro' 
Improve overwrite performance

See merge request external-collab/keydb-pro-6!4

Former-commit-id: 7e59e889f28e30fca409dfb951c31f8cb47effbd
 2021-09-28 18:13:12 +00:00
John Sully
617519e021 Revert "Don't zero out potentially long buffers" - It has no benefit above the noise floor and potentially is the cause of a multithread slowdown 
This reverts commit 711c08c62491d0670b329fa7760c6938689910f1 [formerly 91e76ab4a00546278100baf6dfe61c13e802b40e].


Former-commit-id: f651cf67d731a10c3a52e30261c8d8b9d291576c
 2021-09-28 18:08:15 +00:00
malavan
d502437cf3 Merge branch 'async_commands' of https://gitlab.eqalpha.com/keydb-dev/KeyDB-Pro into async_commands 
Former-commit-id: a19112ec839da4684aeb1a9d1b41906f4c698944
 2021-09-23 17:48:33 +00:00
John Sully
154c9d1f79 Rework the tuning to be better for mixed read/write workloads 
Former-commit-id: a4fdd3e3cb41160b20e92e1b1f4f4ebd2ee86a4a
 2021-09-23 17:46:56 +00:00
John Sully
3294e4f788 Move the snapshot stale threshold to a named constant 
Former-commit-id: 46d2aaf17abb0fb021aa6d7b393ffc143493d339
 2021-09-23 17:45:49 +00:00
malavan
4297a3fb32 prev_error_count needs to be thread local 
Former-commit-id: 4abede1c40db1dfb4235b9aa1e3177678c5304b1
 2021-09-23 17:41:28 +00:00
John Sully
af1c51dc53 Make async commands runtime configurable 
Former-commit-id: 23bafb5f7c3c9805581f4bf09d9d5136898c2883
 2021-09-21 20:49:44 +00:00
John Sully
59855278e0 Initial implementation of async commands 
Former-commit-id: 0d3a4ce7e60c42b2d81962f13100bd1789bf64a8
 2021-09-21 20:43:16 +00:00
malavan
51fe42b10e improve overwrite key performance 
Former-commit-id: 56f9d5528385ea78074a308c6d3987b920d6cc35
 2021-09-14 17:06:04 +00:00
John Sully
711c08c624 Don't zero out potentially long buffers 
Former-commit-id: 91e76ab4a00546278100baf6dfe61c13e802b40e
 2021-09-10 00:40:08 +00:00
John Sully
5eda1c5b42 Don't zero out potentially long buffers 
Former-commit-id: 99eb47e129a3cd555914d4816febb02f0058253a
 2021-09-10 00:38:49 +00:00
jsully
91ec5dffa2 Merge branch 'async_mget' into 'keydbpro' 
add async mget

See merge request keydb-dev/KeyDB-Pro!38

Former-commit-id: e315b39f4779d530f5ab93e1f15915211e6706e2
 2021-09-08 19:08:23 +00:00
malavan
26aa8f0828 updates from comments 
Former-commit-id: 852885f09e7df1d9570408546baffa8545707335
 2021-09-01 21:00:27 +00:00
malavan
3239f678d5 refactor asyncCommand 
Former-commit-id: 6af5775e01872f130bc18791fdb4c0b22507b37f
 2021-09-01 20:18:41 +00:00
John Sully
40d5a2db44 Don't be in tracking mode during load as processChangesAsync works outside the normal system 
Former-commit-id: 8d31ce6eafea1cea2f9f4ea25e44306efef28fa3
 2021-09-01 04:15:28 +00:00
malavan
ee879d04ee add method to get client args as list 
Former-commit-id: 576b8cd2153c48c0ca4dfc9ed7d12f77d6f76f7e
 2021-08-30 21:09:22 +00:00
malavan
59ba007812 add stats for storage provider reads 
Former-commit-id: 612393ffba43cda31443605de924dc8dcf77ef52
 2021-08-26 19:45:33 +00:00
malavan
d658891bf0 refactor of asyncCommand to include snapshot creation 
Former-commit-id: c0908362162b5f2834b90cd9ce84fd1ee6768834
 2021-08-26 13:10:59 +00:00
malavan
65a29f9322 refactor async to offload common code to client 
Former-commit-id: 9a7547bfaa0ceff76e604262913fb11a64c627d8
 2021-08-25 20:06:06 +00:00
MalavanEQAlpha
e0af0bcb95 Merge pull request #313 from MalavanEQAlpha/fixissue295 
Resolve Issue #295 by speeding up keyIsExpired and setting timeout on subkey expires.

Former-commit-id: 9e5e6c2f86548b18ae27b4e1ac20c72517392b07
 2021-08-18 17:34:18 -04:00
christianEQ
8bccf9bf12 fixed overly strict assert for 32bit 
Former-commit-id: ce4891b33d65038bb0543eb2d3526c5310fee59b
 2021-08-13 11:05:12 -04:00
malavan
67ffa578d7 Merge remote-tracking branch 'opensource/unstable' into keydbpro 
Former-commit-id: cf40fa5da29ba9574d20d706d0a41ca7001df95a
 2021-08-11 20:41:54 +00:00
VivekSainiEQ
2598493a62 Prevent invalid mvcc timestamps from causing critical errors 
Former-commit-id: 6f2dbb00119b1d0a1f5a2543d2c6af05f83ef5de
 2021-08-11 15:06:14 -04:00
John Sully
a560bcc4a2 Merge branch 'keydbpro_collab' into multithread_load 
Former-commit-id: 8016c20f1f9a648e658c816e2f6777c5718d5e19
 2021-08-09 20:20:34 +00:00
Christian Legge
9e562f1fe5 Add REPLPING command for use during replication (#329) 
* added replping command for initiating replication

* backwards compatibility for replping (retry if not recognized)

* don't allow ping during loading (load balancer fix)

* changed replping warning to notice

Former-commit-id: d7f6bc16145206e96ffeb9941398d564c3dba6a9
 2021-07-29 15:50:30 -04:00
John Sully
73560dc7b2 Initialize el so we can detect if it is null 
Former-commit-id: ec0f833ea17c668971893aa8f198d22da2e1d289
 2021-07-23 19:31:22 +00:00
Huang Zhw
835d15b536 On 32 bit platform, the bit position of GETBIT/SETBIT/BITFIELD/BITCOUNT,BITPOS may overflow (see CVE-2021-32761) (#9191) 
GETBIT, SETBIT may access wrong address because of wrap.
BITCOUNT and BITPOS may return wrapped results.
BITFIELD may access the wrong address but also allocate insufficient memory and segfault (see CVE-2021-32761).

This commit uses `uint64_t` or `long long` instead of `size_t`.
related https://github.com/redis/redis/pull/8096

At 32bit platform:
> setbit bit 4294967295 1
(integer) 0
> config set proto-max-bulk-len 536870913
OK
> append bit "\xFF"
(integer) 536870913
> getbit bit 4294967296
(integer) 0

When the bit index is larger than 4294967295, size_t can't hold bit index. In the past,  `proto-max-bulk-len` is limit to 536870912, so there is no problem.

After this commit, bit position is stored in `uint64_t` or `long long`. So when `proto-max-bulk-len > 536870912`, 32bit platforms can still be correct.

For 64bit platform, this problem still exists. The major reason is bit pos 8 times of byte pos. When proto-max-bulk-len is very larger, bit pos may overflow.
But at 64bit platform, we don't have so long string. So this bug may never happen.

Additionally this commit add a test cost `512MB` memory which is tag as `large-memory`. Make freebsd ci and valgrind ci ignore this test.

(cherry picked from commit 71d452876ebf8456afaadd6b3c27988abadd1148)
 2021-07-21 21:06:49 +03:00
Yossi Gottlieb
79fa5618f1 Fix CLIENT UNBLOCK crashing modules. (#9167) 
Modules that use background threads with thread safe contexts are likely
to use RM_BlockClient() without a timeout function, because they do not
set up a timeout.

Before this commit, `CLIENT UNBLOCK` would result with a crash as the
`NULL` timeout callback is called. Beyond just crashing, this is also
logically wrong as it may throw the module into an unexpected client
state.

This commits makes `CLIENT UNBLOCK` on such clients behave the same as
any other client that is not in a blocked state and therefore cannot be
unblocked.

(cherry picked from commit aa139e2f02292d668370afde8c91575363c2d611)
 2021-07-21 21:06:49 +03:00
Oran Agra
6cd84b64f0 Test infra, handle RESP3 attributes and big-numbers and bools (#9235) 
- promote the code in DEBUG PROTOCOL to addReplyBigNum
- DEBUG PROTOCOL ATTRIB skips the attribute when client is RESP2
- networking.c addReply for push and attributes generate assertion when
  called on a RESP2 client, anything else would produce a broken
  protocol that clients can't handle.

(cherry picked from commit 6a5bac309e868deef749c36949723b415de2496f)
 2021-07-21 21:06:49 +03:00
perryitay
3f4f9b6331 Fail EXEC command in case a watched key is expired (#9194) 
There are two issues fixed in this commit: 
1. we want to fail the EXEC command in case there is a watched key that's logically
   expired but not yet deleted by active expire or lazy expire.
2. we saw that currently cache time is update in every `call()` (including nested calls),
   this time is being also being use for the isKeyExpired comparison, we want to update
   the cache time only in the first call (execCommand)

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit ac8b1df8850cc80fbf9ce8c2fbde0c1d3a1b4e91)
 2021-07-21 21:06:49 +03:00
John Sully
26750791b3 Merge branch 'keydbpro' into keydbpro_collab 
Former-commit-id: e4e5c6696c6d831924f314a198b266b10d831e14
 2021-07-19 20:51:27 +00:00
John Sully
fc7221739c Info command should show how many keys are cached in RAM vs storage provider 
Former-commit-id: 08597bee69bc16ca7c3d5ff31020472774c6eec9
 2021-07-19 15:11:33 +00:00
John Sully
29ef9708fa Return the ring buffer to its original size if we temporarily resized it 
Former-commit-id: a12ce4a0d105bf7d6ccff95f7dc0044c4676b0a7
 2021-07-18 20:45:32 +00:00
John Sully
ca920fed38 Merge branch 'keydbpro' into keydbpro_collab 
Former-commit-id: 8eec3e948ffd204bb2d6170ad3ca42fa8a2c6d8b
 2021-07-09 05:25:04 +00:00
John Sully
6bb2075d3c Merge branch 'unstable' into keydbpro 
Former-commit-id: 81ded8a35daa5100cac7299a7d0b5f43ee7ac74f
 2021-07-09 04:41:47 +00:00
John Sully
da2aceabcf Merge tag '6.2.3' into unstable 
Former-commit-id: 1895dbb7680fa9aadf6040912e89c733abc8c706
 2021-07-09 04:40:31 +00:00
John Sully
46c1969997 Merge branch 'repl_backlog_rework' into keydbpro_collab 
Former-commit-id: 0983e195757d92d6cbcb63436a1332a6b80e275c
 2021-06-30 06:36:45 +00:00
John Sully
d55bcf23bd Merge branch 'keydbpro' into keydbpro_collab 
Former-commit-id: ecc69952dfd1f145e1aff12bca56a4b4e102d669
 2021-06-25 06:21:58 +00:00
vivek
e68f9cfea1 Primitive implementation of bypassing client buffer, stats are all messed up and print statements everywhere 
Former-commit-id: 59b2ae8ff451f8a5ac2f3baf3c7b509f6872895e
 2021-06-25 03:10:56 +00:00
John Sully
53d7d09f07 Remove fPendingReplicaWrite flag which can instead be calculated on demand 
Former-commit-id: ae26afd13f955eb230b5c2cab20ec90db9b714ad
 2021-06-25 01:54:38 +00:00
VivekSainiEQ
4ac475ea20 More code cleanup 
Former-commit-id: 8e9962b9b7b9093399451bf93d30e5b5d26e3d33
 2021-06-16 19:41:55 +00:00
VivekSainiEQ
0b08909024 Merged in 6.2.2 
Former-commit-id: 3fb9bb5b4fdfb8e09430c7bef6bf9421065adf0f
 2021-06-10 20:11:14 +00:00
VivekSainiEQ
f7d8a0950d Cleaned up code a bit, need to rewrite some comments to reflect new behaviour 
Former-commit-id: 850ec766cd71614ce9e61c12414545cd212d3878
 2021-06-08 23:10:53 +00:00
VivekSainiEQ
9e344f2577 Seems to pass multithreaded test cases, thank the lord 
Former-commit-id: 6cbf70cfff5735f3d4ef2e980945b4b1a1f85971
 2021-06-04 20:09:47 +00:00
VivekSainiEQ
09c4939287 Removed unused variables 
Former-commit-id: 48663bc480f7279a94c68aeebdd9721ca64f7038
 2021-06-03 21:47:33 +00:00
VivekSainiEQ
615fbeb10f Sync works single threaded properly, passes all but one testcase (which hangs) 
Former-commit-id: 9a6ca3a5d906b9d87fe70652d218decbb2775ac1
 2021-06-02 23:41:36 +00:00
John Sully
6fdd8449ff Merge branch 'keydbpro_collab' into multithread_load 
Former-commit-id: b580a5561220bc6dc7bc073135f3045ba0cdda51
 2021-06-02 04:24:49 +00:00
John Sully
075ce8db9e Merge branch 'merge_6.2.2' into keydbpro_collab 
Former-commit-id: 54fe037e4e215b1e5bdb42b567c6df04a69ba150
 2021-06-02 02:05:44 +00:00
John Sully
1d45cc0476 Reduce lock contention when loading to a storage provider 
Former-commit-id: 58bc777f2215918043325753b6e2bf89dc3108f7
 2021-06-01 20:01:41 +00:00