1006 Commits

Author SHA1 Message Date
John Sully
721e572275 Merge remote-tracking branch 'collab/keydbpro' into multithread_load
Former-commit-id: a09454b3a4b295b2d04bdb7d742db7f9a3e17be7
2021-05-31 01:16:01 +00:00
John Sully
10383a8234 Add endurance testing to better detect threading bugs
Former-commit-id: 945e428aa110968479fdcdfc2d5c5308a99eadc3
2021-05-06 00:09:07 +00:00
John Sully
f41b79cea6 Merge branch 'keydbpro' into keydbpro_collab
Former-commit-id: 446daf8f4c33005096372ad9e29ae9b3d465c64e
2021-05-03 22:58:56 +00:00
jsully
fe9f276f95 Merge branch 'cluster_bugfix' into 'keydbpro'
Set repl_down_since in freeClientAsync

See merge request keydb-dev/KeyDB-Pro!12

Former-commit-id: bd7591fe50b1e5565da2fa521438b88058d4da07
2021-05-03 21:41:05 +00:00
christian
12044bab09 Set repl_down_since in freeClientAsync
Former-commit-id: fb4bd2984651837fadd1dbbbb56327e8452b4a2a
2021-05-03 21:41:04 +00:00
John Sully
0c7589580d Optimize freeMemory by lazy freeing objects
Former-commit-id: cca31ed5ee2f42975f0051cfabf1e88720b3d678
2021-04-21 01:20:13 +00:00
John Sully
211b0e5f80 Avoid posting unnecessary async tasks
Former-commit-id: 1661d2a05943f96992be195f5dc32dd9a67c0e68
2021-04-12 03:44:29 +00:00
John Sully
1b121723e2 Don't prefetch when lock contention is low, it increases latency
Former-commit-id: 0d21614e0e5aba28acd364231823d51a3073081f
2021-04-12 03:41:53 +00:00
John Sully
df66c598e9 Make prefetch configurable
Former-commit-id: 3b660347d70cc25d57119080bd43fb4671e36488
2021-04-12 03:38:57 +00:00
John Sully
c070f6ece2 Don't prefetch when lock contention is low, it increases latency
Former-commit-id: 9b2629f6a20368cec8e55f0d006f3a67c8b770b7
2021-04-12 03:23:49 +00:00
John Sully
9d0001ae7c Make prefetch configurable
Former-commit-id: 16996c048042bd3799c8051645bbe7c54137d54c
2021-04-08 19:52:38 +00:00
John Sully
5d924bf16a Avoid posting unnecessary async tasks
Former-commit-id: 08f63fdfd6c78388bba093ec2edda7d77fc9173e
2021-04-07 17:41:20 +00:00
jsully
59a9e93abb Merge branch 'flash_read_perf' into 'keydbpro'
Flash read perf

See merge request external-collab/keydb-pro-6!1

Former-commit-id: f6d7277f307b78f079dd99bf532cd26a7b62018d
2021-04-07 06:12:55 +00:00
Cloud User
6684f150d6 Make high pri time thread configurable
Former-commit-id: 7e94207765d2166f46792aea0919786f3f30d7b3
2021-04-06 22:39:20 +00:00
jsully
c6cea9d84f Merge branch 'keydbpro' into 'flash_read_perf'
# Conflicts:
#   src/db.cpp
#   src/networking.cpp

Former-commit-id: e16c846b2b9e70f20981172287b19e585f81d73d
2021-04-01 17:25:31 +00:00
John Sully
57b242a5ba Fix issue #300
Former-commit-id: bee7f398c9c3b4c01b687d26194913bf215dca36
2021-03-30 23:44:34 +00:00
John Sully
98f27709c6 Fix crash in RDB save
Former-commit-id: b032809b3e978fe571b791179d32ecdc9c067045
2021-03-30 20:49:33 +00:00
John Sully
32926467aa Fix crash in RDB save
Former-commit-id: ef1e122fcb039d25f60752ca27c9feb5e8cfbb31
2021-03-30 20:44:22 +00:00
John Sully
15d9f45b4e Fix thread safety issues with the cache prefetch logic
Former-commit-id: 4892122fc02109d98684a350bd732a0b08a8c7b4
2021-03-28 20:21:44 +00:00
John Sully
73bc5aec95 Fix thread safety issues with the cache prefetch logic
Former-commit-id: a80a128bb64b81115c095d6dd91896ff73048b3d
2021-03-28 17:58:43 +00:00
John Sully
98ca9892c3 Perform GET command inline
Former-commit-id: 5623936d99e334ab103f3dc1541b145c125d0ee8
2021-03-23 03:44:20 +00:00
John Sully
25f56c59b0 Remove snapshot consolidation because its not properly thread safe
Former-commit-id: dcb86d2b92e3ecdb9f914b6f5de9e184cdd23036
2021-03-16 02:37:49 +00:00
John Sully
6b579db919 Fix TSAN errors in tests
Former-commit-id: 30207d2ab34f175b94f430f97581191343d23f1e
2021-03-16 02:29:24 +00:00
John Sully
1bf6ca0645 Load perf fixes with a storage provider set
Former-commit-id: 861b19de00c75c9167cc25031292284ad1c21893
2021-03-04 07:41:06 +00:00
John Sully
20089d7256 Drastically improve perf when loading an RDB with a storage provider
Former-commit-id: 0133b42d54676e8fac2c5cb006cc87988dced268
2021-03-02 04:16:20 +00:00
John Sully
bfbdf23b1a Eliminate needless lock
Former-commit-id: 60f972d463f202edb33ff9a25bc2bd3e2558105c
2021-02-26 05:40:56 +00:00
christian
75539ced8e Offload updating cached time to dedicated thread
Former-commit-id: 9bfc8a43952481b5b54a7b051d44b8bece4a18dd
2021-02-26 00:50:22 +00:00
John Sully
7fb41ba1bd Implement maxstorage config
Former-commit-id: 79e07859dfec14edf2a39646013e1a82db033d4f
2021-02-19 05:17:41 +00:00
John Sully
f3853b6485 Fix deadlock in storage prefetch
Former-commit-id: 9e6162de08248f7726b97b73aee2f23808ff4e7b
2021-02-12 19:38:57 +00:00
John Sully
bb767ad61d Enable prefetch with storage providers
Former-commit-id: 21f9c52c446ad1b460a340c3d873839b2355728f
2021-02-11 00:50:21 +00:00
John Sully
e5343f47c2 Merge branch 'unstable' into keydbpro
Former-commit-id: e2140793f2bf565972ada799af73bf4457e2718d
2021-02-08 18:17:09 +00:00
John Sully
bef72e5f6a Implement rehash during spinlock
Former-commit-id: f68a26381a35b27948046d46c2c7bcfbdc21143d
2021-02-07 19:11:05 -05:00
John Sully
4c9f912c3b Fix cluster test failure
Former-commit-id: 230fde39882766426d892fd30d0c8349f5cca912
2021-02-07 23:38:25 +00:00
christianEQ
511f01bd14 fixed up backup stuff
Former-commit-id: 8a23162c24dc13a4268928c5defd85b0b5249d39
2021-02-04 22:36:32 +00:00
christianEQ
cca8d186aa Merge remote-tracking branch 'opensource/unstable' into keydbpro
Former-commit-id: 5bad058733de2c217340bb9ee48f02b07d754808
2021-02-03 18:10:27 +00:00
christianEQ
406265263b added severity levels for afterErrorReply
Former-commit-id: fe0f07199353abf6668cd66cd2e21751db5c21d9
2021-01-31 00:12:49 -05:00
christianEQ
011b69e9a6 killMainThread -> killServerThreads
Former-commit-id: 0fea781f8f0218c5475890a395c49bd7e4faa03f
2021-01-28 23:57:19 +00:00
christianEQ
8f132802d8 removed unimplemented killIOThreads function declaration
Former-commit-id: f30f8eba2f0a5cb247478c2104a4f748659a6cf2
2021-01-26 22:27:22 +00:00
christianEQ
396f8316e8 fixed server.cpp mentions
Former-commit-id: 7ed2296fc5d10a1c41020558b4bc445953531bdd
2021-01-26 22:22:19 +00:00
christianEQ
358debebfa Merge tag 'tags/6.0.10' into redismerge_2021-01-20
Former-commit-id: dadce055f897cee83946c2d3e5cbb76341b94230
2021-01-26 21:43:09 +00:00
christianEQ
247c916f69 changed loading process config types to be consistent across environments
Former-commit-id: d2867580aba70bbfa703578cdb40093a4e1a7147
2021-01-14 21:12:36 -05:00
christianEQ
81536a31c8 added config option for loading callback interval keys
Former-commit-id: 9f7ed0b2327924563668105ead09f52833902d0b
2021-01-14 13:06:57 -05:00
Yang Bodong
f464cf2380 Swapdb should make transaction fail if there is any client watching keys (#8239)
This PR not only fixes the problem that swapdb does not make the
transaction fail, but also optimizes the FLUSHALL and FLUSHDB command to
set the CLIENT_DIRTY_CAS flag to avoid unnecessary traversal of clients.

FLUSHDB was changed to first iterate on all watched keys, and then on the
clients watching each key.
Instead of iterating though all clients, and for each iterate on watched keys.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 10f94b0ab12f9315939dcccf39d64b9388c0c7fa)
2021-01-12 16:25:37 +02:00
George Prekas
8ff0226361 Add check for the MADV_FREE/fork arm64 Linux kernel bug (#8224)
Older arm64 Linux kernels have a bug that could lead to data corruption during
background save under the following scenario:

1) jemalloc uses MADV_FREE on a page,
2) jemalloc reuses and writes the page,
3) Redis forks the background save process, and
4) Linux performs page reclamation.

Under these conditions, Linux will reclaim the page wrongfully and the
background save process will read zeros when it tries to read the page.

The bug has been fixed in Linux with commit:
ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is
preserved across pte_wrprotect()")

This Commit adds an ignore-warnings config, when not found, redis will
print a warning and exit on startup (default behavior).

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit b02780c41dbc5b28d265b5cf141c03c1a7383ef9)

in 6.0 this warning is ignored by default in order to avoid adding
regression, specifically for deployments that don't need persistence or
replication
2021-01-12 16:25:37 +02:00
Wang Yuan
a60ed4a599 Backup keys to slots map and restore when fail to sync if diskless-load type is swapdb in cluster mode (#8108)
When replica diskless-load type is swapdb in cluster mode, we didn't backup
keys to slots map, so we will lose keys to slots map if fail to sync.
Now we backup keys to slots map at first, and restore it properly when fail.

This commit includes a refactory/cleanup of the backups mechanism (moving it to db.c and re-structuring it a bit).

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit b55a827ea2e19bd6cd48f216e8e6caa34299f9b9)
2021-01-12 16:25:37 +02:00
Oran Agra
3ac666b3e6 Fix oom-score-adj-values range, abs options, and bug when used in config file (#8046)
Fix: When oom-score-adj-values is provided in the config file after
oom-score-adj yes, it'll take an immediate action, before
readOOMScoreAdj was acquired, resulting in an error (out of range score
due to uninitialized value. delay the reaction the real call is made by
main().

Since the values are clamped to -1000..1000, and they're
applied as an offset from the value at startup (which may be -1000), we
need to allow the offsets to reach to +2000 so that a value of +1000 is
achievable in case the value at startup was -1000.

Adding an option for absolute values rather than relative ones.

(cherry picked from commit 61954951edbda670bfbae8be0147daa64df95f26)
2021-01-12 16:25:37 +02:00
guybe7
4b37eb13dc EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica (#8016)
The bug was introduced by #5021 which only attempted avoid EXIST on an
already expired key from returning 1 on a replica.

Before that commit, dbExists was used instead of
lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)

Other than that, this commit fixes OBJECT to also come empty handed on
expired keys in replica.

And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
the key regardless of it's expired state)

(cherry picked from commit f8ae991717f10c837c1a76b2954dae56ecb0e6bc)
2021-01-12 16:25:37 +02:00
John Sully
828cf80c94 Merge branch 'repl_perf' into unstable
Former-commit-id: bc8cc1adb301d51e926f2552ce1eb28165ed569b
2020-12-24 02:22:57 +00:00
John Sully
95181438dd Merge branch 'repl_perf' into keydbpro
Former-commit-id: c4431a6be7a1ddc05abdd4510630fac2a2a6664e
2020-12-10 18:45:52 +00:00
John Sully
706ba4e1dc Merge branch 'unstable' into keydbpro
Former-commit-id: b7a1e16c0f04e8aeb3764e3681fa14fc0f97f6a3
2020-12-10 02:37:28 +00:00