12903 Commits

Author SHA1 Message Date
Yossi Gottlieb
f84c181eac Fix crashes with io-threads-do-reads enabled. (#8230)
Normally IO threads should simply read data from the socket into the
buffer and attempt to parse it.

If a protocol error is detected, a reply is generated which may result
with installing a write handler which is not thread safe. This fix
delays that until the client is processed back in the main thread.

Fixes #8220

(cherry picked from commit e7047ec2fcc20e150c0c8cf29addf582638d7e80)
2021-01-12 16:25:37 +02:00
George Prekas
8ff0226361 Add check for the MADV_FREE/fork arm64 Linux kernel bug (#8224)
Older arm64 Linux kernels have a bug that could lead to data corruption during
background save under the following scenario:

1) jemalloc uses MADV_FREE on a page,
2) jemalloc reuses and writes the page,
3) Redis forks the background save process, and
4) Linux performs page reclamation.

Under these conditions, Linux will reclaim the page wrongfully and the
background save process will read zeros when it tries to read the page.

The bug has been fixed in Linux with commit:
ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is
preserved across pte_wrprotect()")

This Commit adds an ignore-warnings config, when not found, redis will
print a warning and exit on startup (default behavior).

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit b02780c41dbc5b28d265b5cf141c03c1a7383ef9)

in 6.0 this warning is ignored by default in order to avoid adding
regression, specifically for deployments that don't need persistence or
replication
2021-01-12 16:25:37 +02:00
Qu Chen
d756b11fa2 Not over-allocate client query buffer when reading large objects. (#5954)
In response to large client query buffer optimization introduced in 1898e6c. The calculation of the amount of
remaining bytes we need to write to the query buffer was calculated wrong, as a result we are unnecessarily
growing the client query buffer by sdslen(c->querybuf) always. This fix corrects that behavior.

Please note the previous behavior prior to the before-mentioned change was correctly calculating the remaining
additional bytes, and this change makes that calculate to be consistent.

Useful context, the argument of size `ll` starts at qb_pos (which is now the beginning of the sds), but much of it
may have already been read from the socket, so we only need to grow the sds for the remainder of it.

(cherry picked from commit 11b3325e9999721d35ec64afac7b917664f6291b)
2021-01-12 16:25:37 +02:00
Oran Agra
ec74ae7ec1 Handle output buffer limits for Module blocked clients (#8141)
Module blocked clients cache the response in a temporary client,
the reply list in this client would be affected by the recent fix
in #7202, but when the reply is later copied into the real client,
it would have bypassed all the checks for output buffer limit, which
would have resulted in both: responding with a partial response to
the client, and also not disconnecting it at all.

(cherry picked from commit 48efc25f749c3620f9245786582ac76cb40e9bf4)
2021-01-12 16:25:37 +02:00
Yossi Gottlieb
785851a736 Fix setproctitle related crashes. (#8150)
Makes spt_init more careful with assumptions about what memory regions
may be overwritten. It will now only consider a contiguous block of argv
and envp elements and mind any gaps.

(cherry picked from commit ec02c761aa16175eb599b2fc9d0b2792ffe8a66c)
2021-01-12 16:25:37 +02:00
Yossi Gottlieb
2a2d5a7a1f Fix use-after-free issue in spt_copyenv. (#8088)
Seems to have gone unnoticed for a long time, because at least with
glibc it will only be triggered if setenv() was called before spt_init,
which Redis doesn't.

Fixes #8064.

(cherry picked from commit 7e5a6313f0add995c723351532d994118e3e8a6d)
2021-01-12 16:25:37 +02:00
Wang Yuan
a60ed4a599 Backup keys to slots map and restore when fail to sync if diskless-load type is swapdb in cluster mode (#8108)
When replica diskless-load type is swapdb in cluster mode, we didn't backup
keys to slots map, so we will lose keys to slots map if fail to sync.
Now we backup keys to slots map at first, and restore it properly when fail.

This commit includes a refactory/cleanup of the backups mechanism (moving it to db.c and re-structuring it a bit).

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit b55a827ea2e19bd6cd48f216e8e6caa34299f9b9)
2021-01-12 16:25:37 +02:00
Wang Yuan
5ed6662145 Reset average ttl when empty databases (#8106)
On FLUSHDB or full sync, reset old average TTL stat.
This Stat is incrementally collected by the master over time when it searches for expired keys. 

(cherry picked from commit c85bf2352d2a3dd04872124d33066403b114a7e7)
2021-01-12 16:25:37 +02:00
Oran Agra
f885e364ba Fix bug with module GIL being released prematurely (#8061)
This is hopefully usually harmles.
The server.ready_keys will usually be empty so the code after releasing
the GIL will soon be done.
The only case where it'll actually process things is when a module
releases a client (or module) blocked on a key, by triggering this NOT
from within a command (e.g. a timer event).

This bug was introduced in redis 6.0.9, see #7903

(cherry picked from commit e6fa47380a5274119ed37c7a5ea7455d4b7dbdcc)
2021-01-12 16:25:37 +02:00
Oran Agra
3ac666b3e6 Fix oom-score-adj-values range, abs options, and bug when used in config file (#8046)
Fix: When oom-score-adj-values is provided in the config file after
oom-score-adj yes, it'll take an immediate action, before
readOOMScoreAdj was acquired, resulting in an error (out of range score
due to uninitialized value. delay the reaction the real call is made by
main().

Since the values are clamped to -1000..1000, and they're
applied as an offset from the value at startup (which may be -1000), we
need to allow the offsets to reach to +2000 so that a value of +1000 is
achievable in case the value at startup was -1000.

Adding an option for absolute values rather than relative ones.

(cherry picked from commit 61954951edbda670bfbae8be0147daa64df95f26)
2021-01-12 16:25:37 +02:00
guybe7
4b37eb13dc EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica (#8016)
The bug was introduced by #5021 which only attempted avoid EXIST on an
already expired key from returning 1 on a replica.

Before that commit, dbExists was used instead of
lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)

Other than that, this commit fixes OBJECT to also come empty handed on
expired keys in replica.

And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
the key regardless of it's expired state)

(cherry picked from commit f8ae991717f10c837c1a76b2954dae56ecb0e6bc)
2021-01-12 16:25:37 +02:00
Wang Yuan
3a13c654e6 Disable rehash when redis has child process (#8007)
In redisFork(), we don't set child pid, so updateDictResizePolicy()
doesn't take effect, that isn't friendly for copy-on-write.

The bug was introduced this in redis 6.0: 56258c6

(cherry picked from commit 89c78a980807aa1de0a6d0ccde6042450333a783)
2021-01-12 16:25:37 +02:00
John Sully
cbc331be74 Fix intermittent module test failure, resizing the repl backlog is not good enough to clear the backlog
Former-commit-id: d50e4bb8193a349eb3295a53e3111e764676ddcf
2021-01-12 00:29:07 +00:00
benschermel
e0703f0627 update perf chart, build info, job post link
Former-commit-id: 4485ca74f6d177803f7ca8d2a2b75dcba8e6817b
2021-01-11 11:40:00 -05:00
benschermel
812959637b created ascii logo image
Former-commit-id: 7f72b214e58164a9731508da6f95241874174210
2021-01-11 11:40:00 -05:00
John Sully
828cf80c94 Merge branch 'repl_perf' into unstable
Former-commit-id: bc8cc1adb301d51e926f2552ce1eb28165ed569b
2020-12-24 02:22:57 +00:00
John Sully
761dd95d2a Don't write in the lock
Former-commit-id: 93b7364aaecbe578be4b75c18c840c4b396c7d4e
2020-12-17 06:52:59 +00:00
John Sully
07a52ceb67 Merge branch 'unstable' into keydbpro
Former-commit-id: 53a062a6b796ccbd43560fed1b05367ba759da96
2020-12-11 18:39:30 +00:00
John Sully
513da06a6f Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable
Former-commit-id: d03c2de9017b73b5d339fe76dd99292f27dfcd52
2020-12-11 03:59:50 +00:00
John Sully
59a1d5085f don't try and free a client that can't be freed
Former-commit-id: 7d3c5f1e64a79f47a103ce97c6991aa473fb697a
2020-12-11 03:59:39 +00:00
John Sully
51ef4343c4 Prevent use after free
Former-commit-id: 81573de393eef083d2209117b270d088a7b3f819
2020-12-11 03:59:26 +00:00
John Sully
fb5b1e8e7d Merge branch 'unstable' into keydbpro
Former-commit-id: e59c3945d3ed0b05a3e42713b091b1b5c96a9db2
2020-12-11 00:49:54 +00:00
John Sully
69f375c45b Remove AE_ASSERT and use the proper assert infra
Former-commit-id: a80dd5d0010d87bd7dbfb7e2ccbca98a8511db69
2020-12-11 00:24:27 +00:00
John Sully
367067c98a Prevent test crash due to unimplemented save for nested hash
Former-commit-id: 7e0077f293c5f91f240ab401e5f167e487afbfe0
2020-12-11 00:23:44 +00:00
John Sully
ffd99133ef Fix crash freeing nested hash with list
Former-commit-id: 1eb035e6ad23048df8103300c65fd219297ceff7
2020-12-11 00:23:14 +00:00
John Sully
2a18091d73 Fix test timeouts
Former-commit-id: 6f3feee67678b594fbe9fd592f9f823a7a07b770
2020-12-10 20:09:19 +00:00
John Sully
95181438dd Merge branch 'repl_perf' into keydbpro
Former-commit-id: c4431a6be7a1ddc05abdd4510630fac2a2a6664e
2020-12-10 18:45:52 +00:00
John Sully
706ba4e1dc Merge branch 'unstable' into keydbpro
Former-commit-id: b7a1e16c0f04e8aeb3764e3681fa14fc0f97f6a3
2020-12-10 02:37:28 +00:00
Kajaruban Surendran
8586dfb15c Addressing PR comments
Former-commit-id: bdd14ac517fb1921dd6c398ad10346c74324740d
2020-12-04 14:54:06 -05:00
Kajaruban Surendran
d46ed5d770 Addressing PR comments
Former-commit-id: 63e5d00bdfdb054353c7f3fb7a5890d98e7a7f92
2020-12-04 14:54:06 -05:00
Kajaruban Surendran
5886c1b985 Handling of aeResizeSetSize's return code for current thread
Former-commit-id: c25fb3d74fb9e7adb6ad7ef730355e325e982cd2
2020-12-04 14:54:06 -05:00
Kajaruban Surendran
123aec9bed check the aeResizeSetSize return code
Former-commit-id: 572d41325b9fd5322ab0e41f88e384d69f52f41d
2020-12-04 14:54:06 -05:00
Kajaruban Surendran
024739dee7 Adding thread index to the error message
Former-commit-id: e02120cdf7ca5adec58aa8e84aece89b01d6b751
2020-12-04 14:54:06 -05:00
Kajaruban Surendran
98f0c6b599 Call aePostFunction to change the setsize of each threads by that thread
Former-commit-id: 7b1221a74d06616149436fd44d67a2ad83048e44
2020-12-04 14:54:06 -05:00
Kajaruban Surendran
63d79cc297 Calling initServerThread function cserver.cthreads(server-threads) times than MAX_EVENT_LOOPS(constant)
Former-commit-id: d6b4f994ad776efab6d647947fe1543359ec9401
2020-12-04 14:52:28 -05:00
christianEQ
71b4a7b6d3 Issue: #204 Allocate 8 MB to thread stack
Former-commit-id: 66a41dafc47a20251f5f6776625780dfa26ee505
2020-12-01 14:50:03 -05:00
John Sully
300eca8ff6 Issue: #260 Don't use C++ static_assert in C files
Former-commit-id: f19b9c155ef5b957f7cd9a3017c54aab742ffb2f
2020-11-30 23:01:20 +00:00
John Sully
ecad9741bc code coverage reports should test multithreaded
Former-commit-id: 5d016fdb6b11404d0eddbab651ed17180526a991
2020-11-30 22:59:00 +00:00
VivekSainiEQ
52561f1729 Added module data type load/save tests
Former-commit-id: cae9924fd9eefcd88cef1c964f0bc8bce7dd4242
2020-11-30 17:47:04 -05:00
VivekSainiEQ
3d15a7b65a Removed use of module datatypes, now should work if tests/modules is not built
Former-commit-id: 089f28e8ba91ed2b875048efa0b28ceb32010541
2020-11-30 17:47:04 -05:00
VivekSainiEQ
54b36868ba Added tests for saving various data types to disk and loading them back, and for loading data types from redis to maintain compatibility
Former-commit-id: dcb44d3a09a4021f05079bedbac690e33ec7f39e
2020-11-30 17:47:04 -05:00
VivekSainiEQ
6c16fe69e7 Added remainder of runtest scripts to make lcov command. Code coverage goes from 57.9% to 72.1%
Former-commit-id: f99cba0b92d921b1a2d350f2c7545138485c2a74
2020-11-30 17:47:04 -05:00
VivekSainiEQ
6632bafced Added KeyDB.info to .gitignore file
Former-commit-id: e10dc0dace6b152418aad2bd3ee212a471bca9e8
2020-11-30 17:47:04 -05:00
Kajaruban Surendran
fd03008bbf Configurable option for MOTD
Former-commit-id: 49a89d636ba698dbd0858d5059d3d6387c8c1fc7
2020-11-30 10:42:48 -05:00
VivekSainiEQ
f99824faf4 Added references to KeyDB.info instead of redis.info
Former-commit-id: b94f91b9b25b5476d62bdbe2f054cc61691de9f5
2020-11-25 18:49:20 -05:00
VivekSainiEQ
7ad88d0bac Properly escaped $ in lcov command
Former-commit-id: 06a17c8511cfa8e07962401cc01045878525dc27
2020-11-25 18:49:20 -05:00
VivekSainiEQ
2ab3385733 Updated 'make lcov' command to exclude external dependancies and generate final code coverage percentage
Former-commit-id: eebd70aaa6bec0362ea90148174263fa6ffcc169
2020-11-25 18:49:20 -05:00
VivekSainiEQ
86ca4300c8 Added test for issue #257
Former-commit-id: 779c775df9e8be36064689404e3ff51bead8a364
2020-11-25 17:21:37 -05:00
VivekSainiEQ
47a18f5131 Made NULL pointer check explicit, issue #257
Former-commit-id: 1d4d2925c2a11bb7fdffe92c17a315c2df011a54
2020-11-25 17:21:37 -05:00
VivekSainiEQ
d4a2ccd50b Added NULL check to function objFromAllocPtr, now NULL input will map to NULL output even if active replica is enabled
Former-commit-id: b67db7c2d5396892cc0c91c3c822a5b9a547badf
2020-11-25 17:21:37 -05:00