21065 Commits

Author SHA1 Message Date
Wen Hui
f5cf1e46a4
[Sentinel] more specific config errors for duplicate known-replica and known-sentinel (#8332)
Previously invalid configuration errors were not very specific and in some cases hard to understand.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2021-01-17 15:43:21 +02:00
Yang Bodong
8df851176b Add lazyfree-lazy-user-flush config to control default behavior of FLUSH[ALL|DB], SCRIPT FLUSH (#8258)
* Adds ASYNC and SYNC arguments to SCRIPT FLUSH
* Adds SYNC argument to FLUSHDB and FLUSHALL
* Adds new config to control the default behavior of FLUSHDB, FLUSHALL and SCRIPT FLUASH.

the new behavior is as follows:
* FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the
  value of lazyfree-lazy-user-flush.
* FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner.
* FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
2021-01-15 15:32:58 +02:00
Yang Bodong
294f93af97
Add lazyfree-lazy-user-flush config to control default behavior of FLUSH[ALL|DB], SCRIPT FLUSH (#8258)
* Adds ASYNC and SYNC arguments to SCRIPT FLUSH
* Adds SYNC argument to FLUSHDB and FLUSHALL
* Adds new config to control the default behavior of FLUSHDB, FLUSHALL and SCRIPT FLUASH.

the new behavior is as follows:
* FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the
  value of lazyfree-lazy-user-flush.
* FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner.
* FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
2021-01-15 15:32:58 +02:00
Viktor Söderqvist
c694535f5f Rename non-API RM-prefixed functions to hide them from API docs
The prefix is changed from `RM_` to `module` on the following
internal functions, to prevent them from appearing in the API docs:

    RM_LogRaw -> moduleLogRaw
    RM_FreeCallReplyRec -> moduleFreeCallReplyRec
    RM_ZsetAddFlagsToCoreFlags -> moduleZsetAddFlagsToCoreFlags
    RM_ZsetAddFlagsFromCoreFlags -> moduleZsetAddFlagsFromCoreFlags
2021-01-15 13:33:56 +02:00
Viktor Söderqvist
fcb3dfe56d Rename non-API RM-prefixed functions to hide them from API docs
The prefix is changed from `RM_` to `module` on the following
internal functions, to prevent them from appearing in the API docs:

    RM_LogRaw -> moduleLogRaw
    RM_FreeCallReplyRec -> moduleFreeCallReplyRec
    RM_ZsetAddFlagsToCoreFlags -> moduleZsetAddFlagsToCoreFlags
    RM_ZsetAddFlagsFromCoreFlags -> moduleZsetAddFlagsFromCoreFlags
2021-01-15 13:33:56 +02:00
Viktor Söderqvist
3ec23f0b6e Modules API reference formatting fixes
Fixes markdown formatting errors and some functions not showing
up in the generated documentation at all.

Ruby script (gendoc.rb) fixes:

* Modified automatic instertion of backquotes:
  * Don't add backquotes around names which are already preceded by a
    backquote. Fixes for example \`RedisModule_Reply\*\` which turning
    into \`\`RedisModule_Reply\`\*\` messes up the formatting.
  * Add backquotes around types such as RedisModuleString (in addition
    to function names `RedisModule_[A-z()]*` and macro names
    `REDISMODULE_[A-z]*`).
  * Require 4 spaces indentation for disabling automatic backquotes, i.e.
    code blocks. Fixes continuations of list items (indented 2 spaces).
* More permissive extraction of doc comments:
  * Allow doc comments starting with `/**`.
  * Make space before `*` on each line optional.
  * Make space after `/*` and `/**` optional (needed when appearing on
    its own line).

Markdown fixes in module.c:

* Fix code blocks not indented enough (4 spaces needed).
* Add black line before code blocks and lists where missing (needed).
* Enclose special markdown characters `_*^<>` in backticks to prevent them
  from messing up formatting.
* Lists with `1)` changed to `1.` for proper markdown lists.
* Remove excessive indentation which causes text to be unintentionally
  rendered as code blocks.
* Other minor formatting fixes.

Other fixes in module.c:

* Remove blank lines between doc comment and function definition. A blank
  line here makes the Ruby script exclude the function in docs.
2021-01-15 13:33:56 +02:00
Viktor Söderqvist
ebf20b83b2 Modules API reference formatting fixes
Fixes markdown formatting errors and some functions not showing
up in the generated documentation at all.

Ruby script (gendoc.rb) fixes:

* Modified automatic instertion of backquotes:
  * Don't add backquotes around names which are already preceded by a
    backquote. Fixes for example \`RedisModule_Reply\*\` which turning
    into \`\`RedisModule_Reply\`\*\` messes up the formatting.
  * Add backquotes around types such as RedisModuleString (in addition
    to function names `RedisModule_[A-z()]*` and macro names
    `REDISMODULE_[A-z]*`).
  * Require 4 spaces indentation for disabling automatic backquotes, i.e.
    code blocks. Fixes continuations of list items (indented 2 spaces).
* More permissive extraction of doc comments:
  * Allow doc comments starting with `/**`.
  * Make space before `*` on each line optional.
  * Make space after `/*` and `/**` optional (needed when appearing on
    its own line).

Markdown fixes in module.c:

* Fix code blocks not indented enough (4 spaces needed).
* Add black line before code blocks and lists where missing (needed).
* Enclose special markdown characters `_*^<>` in backticks to prevent them
  from messing up formatting.
* Lists with `1)` changed to `1.` for proper markdown lists.
* Remove excessive indentation which causes text to be unintentionally
  rendered as code blocks.
* Other minor formatting fixes.

Other fixes in module.c:

* Remove blank lines between doc comment and function definition. A blank
  line here makes the Ruby script exclude the function in docs.
2021-01-15 13:33:56 +02:00
christianEQ
ab80e23152 changed loading process config types to be consistent across environments
Former-commit-id: d2867580aba70bbfa703578cdb40093a4e1a7147
2021-01-14 21:12:36 -05:00
christianEQ
247c916f69 changed loading process config types to be consistent across environments
Former-commit-id: d2867580aba70bbfa703578cdb40093a4e1a7147
2021-01-14 21:12:36 -05:00
christianEQ
464a08226d renamed loaded key counter to be more clear
Former-commit-id: 8dc238efae8f6ba6b4ddc9b169e7195bcb43920b
2021-01-14 13:06:57 -05:00
christianEQ
588a37583e renamed loaded key counter to be more clear
Former-commit-id: 8dc238efae8f6ba6b4ddc9b169e7195bcb43920b
2021-01-14 13:06:57 -05:00
christianEQ
047e1174d9 fixed signedness warning
Former-commit-id: 00043f95099803f6b06a208ac3b19e2cef9abf4d
2021-01-14 13:06:57 -05:00
christianEQ
6f97805577 fixed signedness warning
Former-commit-id: 00043f95099803f6b06a208ac3b19e2cef9abf4d
2021-01-14 13:06:57 -05:00
christianEQ
9262739be2 added section to keydb.conf for new options, tweaked default
Former-commit-id: 9b81f3f263de8cffab20ef4d20c2ac4935f63a10
2021-01-14 13:06:57 -05:00
christianEQ
e68cee23a1 added section to keydb.conf for new options, tweaked default
Former-commit-id: 9b81f3f263de8cffab20ef4d20c2ac4935f63a10
2021-01-14 13:06:57 -05:00
christianEQ
327c44f8eb ping replicas during key load
Former-commit-id: 2747d1a1a99d3415d5486a472fe0e39135b79bc5
2021-01-14 13:06:57 -05:00
christianEQ
d4ad045535 ping replicas during key load
Former-commit-id: 2747d1a1a99d3415d5486a472fe0e39135b79bc5
2021-01-14 13:06:57 -05:00
christianEQ
28a0cdc572 added config option for loading callback interval keys
Former-commit-id: 9f7ed0b2327924563668105ead09f52833902d0b
2021-01-14 13:06:57 -05:00
christianEQ
81536a31c8 added config option for loading callback interval keys
Former-commit-id: 9f7ed0b2327924563668105ead09f52833902d0b
2021-01-14 13:06:57 -05:00
christianEQ
fc3388acfe configurized loading_process_events_interval_bytes
Former-commit-id: c176ea73664a182af9caacda9d489fb2b344b2de
2021-01-14 13:06:57 -05:00
christianEQ
4fa25cbc57 configurized loading_process_events_interval_bytes
Former-commit-id: c176ea73664a182af9caacda9d489fb2b344b2de
2021-01-14 13:06:57 -05:00
charsyam
b2526c5040 Replace many calling zrealloc to one zmalloc in sentinelResetMasterAndChangeAddress (#8319)
Replace many calling zrealloc to one zmalloc in sentinelResetMasterAndChangeAddress
2021-01-13 16:44:04 -08:00
charsyam
108afbc0bb
Replace many calling zrealloc to one zmalloc in sentinelResetMasterAndChangeAddress (#8319)
Replace many calling zrealloc to one zmalloc in sentinelResetMasterAndChangeAddress
2021-01-13 16:44:04 -08:00
Wang Yuan
b3477eddf9 Optimize performance of clusterGenNodesDescription for large clusters (#8182)
Optimize the performance of clusterGenNodesDescription by only checking slot ownership of each slot once, instead of checking each slot for each node.
2021-01-13 12:36:03 -08:00
Wang Yuan
9cb9f98d2f
Optimize performance of clusterGenNodesDescription for large clusters (#8182)
Optimize the performance of clusterGenNodesDescription by only checking slot ownership of each slot once, instead of checking each slot for each node.
2021-01-13 12:36:03 -08:00
houzj.fnst
8da27781f4 remove some unnecessary checks (#7431)
Remove several checks that always evaluate to true.
2021-01-13 12:00:55 -08:00
houzj.fnst
f5577fdbd8
remove some unnecessary checks (#7431)
Remove several checks that always evaluate to true.
2021-01-13 12:00:55 -08:00
sundb
60423a3e5a Fix use of lookupKeyRead and lookupKeyWrite in zrangeGenericCommand, zunionInterDiffGenericCommand (#8316)
* Change zunionInterDiffGenericCommand to use lookupKeyRead if dstkey is null
* Change zrangeGenericCommand to use lookupKey Write if dstkey isn't null

ZRANGESTORE and UNION, ZINTER, ZDIFF are all new commands (6.2 RC1 and RC2).
In redis 6.0 the ZRANGE was using lookupKeyRead, and ZUNIONSTORE / ZINTERSTORE were using lookupKeyWrite.
So there bugs are introduced in 6.2 and will be resolved before it is released.

the implications of this bug are also not big:
The sole difference between LookupKeyRead and LookupKeyWrite is for command executed on a replica, which are not received from its master client. (for the master, and for the master client on the replica, these two functions behave the same)!
2021-01-13 11:58:12 +02:00
sundb
8a81ed1b5a
Fix use of lookupKeyRead and lookupKeyWrite in zrangeGenericCommand, zunionInterDiffGenericCommand (#8316)
* Change zunionInterDiffGenericCommand to use lookupKeyRead if dstkey is null
* Change zrangeGenericCommand to use lookupKey Write if dstkey isn't null

ZRANGESTORE and UNION, ZINTER, ZDIFF are all new commands (6.2 RC1 and RC2).
In redis 6.0 the ZRANGE was using lookupKeyRead, and ZUNIONSTORE / ZINTERSTORE were using lookupKeyWrite.
So there bugs are introduced in 6.2 and will be resolved before it is released.

the implications of this bug are also not big:
The sole difference between LookupKeyRead and LookupKeyWrite is for command executed on a replica, which are not received from its master client. (for the master, and for the master client on the replica, these two functions behave the same)!
2021-01-13 11:58:12 +02:00
sundb
3e8c7315b7 Fix units in log message of copy-on-write (#8320) 2021-01-13 11:38:02 +02:00
sundb
593cde16bc
Fix units in log message of copy-on-write (#8320) 2021-01-13 11:38:02 +02:00
Oran Agra
93fbd5d13e Redis 6.0.10 2021-01-12 16:25:37 +02:00
Oran Agra
8d70d498d5 Redis 6.0.10 2021-01-12 16:25:37 +02:00
Bob Li
5041095cad Fix overflow of rdbWriteRaw return value (#8306)
Saving string of more than 2GB to the RDB file, can result in corrupt RDB, or failure in rdbSave.
S

(cherry picked from commit 3fb4197a742d064236ae4fdccf9dc00ed3b538d3)
2021-01-12 16:25:37 +02:00
Bob Li
203feb6ea2 Fix overflow of rdbWriteRaw return value (#8306)
Saving string of more than 2GB to the RDB file, can result in corrupt RDB, or failure in rdbSave.
S

(cherry picked from commit 542455ce2afbaffc065afbfdf6578399b8faecda)
2021-01-12 16:25:37 +02:00
guybe7
2d537b376c Modules: Fix an integer sign bug in moduleTimerHandler (#8131)
bug was introduced in 9cbdc8dcdb

(cherry picked from commit de339c29635091c5ce41b7a061b5aff723952a8b)
2021-01-12 16:25:37 +02:00
guybe7
08c3f0bca6 Modules: Fix an integer sign bug in moduleTimerHandler (#8131)
bug was introduced in 1a91a2700b

(cherry picked from commit 2f41a3856845265ffc6cc3a35524883a8690cff7)
2021-01-12 16:25:37 +02:00
Egor Seredin
85bcc60165 Allow '\0' inside of result of sdscatvprintf, and efficiency improvements (#6260)
This will allow to use: RedisModule_CreateStringPrintf(ctx, "%s %c %s", "string1", 0, "string2");

On large string, the previous code would incrementally retry to double the output buffer.
now it uses the the return value of snprintf and grows to the right size in one step.

and also avoids an excessive strlen in sdscat at the end.

(cherry picked from commit 1ad4e18394eac68ab8bae1ef5a2920086ce0f9ba)
2021-01-12 16:25:37 +02:00
Egor Seredin
3bf02ea7e8 Allow '\0' inside of result of sdscatvprintf, and efficiency improvements (#6260)
This will allow to use: RedisModule_CreateStringPrintf(ctx, "%s %c %s", "string1", 0, "string2");

On large string, the previous code would incrementally retry to double the output buffer.
now it uses the the return value of snprintf and grows to the right size in one step.

and also avoids an excessive strlen in sdscat at the end.

(cherry picked from commit f4ca3d8757d6abb3536610ddb7b9ab3ad39e81df)
2021-01-12 16:25:37 +02:00
guybe7
c01b72f30d Modules: Improve timer accuracy (#7987)
The bug occurs when 'callback' re-registers itself to a point
in the future and the execution time in non-negligible:
'now' refers to time BEFORE callback was executed and is used
to calculate 'next_period'.
We must get the actual current time when calculating 'next_period'

(cherry picked from commit 9cbdc8dcdbaf96869251dd9728c0876adf1b2492)
2021-01-12 16:25:37 +02:00
guybe7
acb4cd2135 Modules: Improve timer accuracy (#7987)
The bug occurs when 'callback' re-registers itself to a point
in the future and the execution time in non-negligible:
'now' refers to time BEFORE callback was executed and is used
to calculate 'next_period'.
We must get the actual current time when calculating 'next_period'

(cherry picked from commit 1a91a2700b24211e90c695d3fdbbfe7e8d75dbe4)
2021-01-12 16:25:37 +02:00
Meir Shpilraien (Spielrein)
58c00320e3 Moved RMAPI_FUNC_SUPPORTED location such that it will be visible to modules (#8037)
The RMAPI_FUNC_SUPPORTED was defined in the wrong place on redismodule.h
and was not visible to modules.

(cherry picked from commit 560d2dc0081bc35b81b6a64d25c4077fa2e69ad9)
2021-01-12 16:25:37 +02:00
Meir Shpilraien (Spielrein)
fb86ecb802 Moved RMAPI_FUNC_SUPPORTED location such that it will be visible to modules (#8037)
The RMAPI_FUNC_SUPPORTED was defined in the wrong place on redismodule.h
and was not visible to modules.

(cherry picked from commit 97d647a13920c230a27685992a23679b44b73550)
2021-01-12 16:25:37 +02:00
David CARLIER
010223a7f4 raspberry build fix. (#8095)
__ILP32__ is 32 bits ABI and does not imply x86, this patch resolves this. 

(cherry picked from commit 6e44f74dbc1e18383e53c6e86aa72e327c8f47c8)
2021-01-12 16:25:37 +02:00
David CARLIER
2c5865cd90 raspberry build fix. (#8095)
__ILP32__ is 32 bits ABI and does not imply x86, this patch resolves this. 

(cherry picked from commit 0719388cfb1a79160204314beb1de1f9c29a3684)
2021-01-12 16:25:37 +02:00
Yossi Gottlieb
1167967014 Fix crash log output on ARM. (#8020)
(cherry picked from commit 43c07a877cd67ede008c51d227723cfc6c7fd896)
2021-01-12 16:25:37 +02:00
Yossi Gottlieb
ebbcd05e03 Fix crash log output on ARM. (#8020)
(cherry picked from commit 7e4325cbc9af96d55f0b61c7648233a4473e8c93)
2021-01-12 16:25:37 +02:00
Oran Agra
df32c5a812 Fix cluster access to unaligned memory (SIGBUS on old ARM) (#7958)
Turns out this was broken since version 4.0 when we added sds size
classes.
The cluster code uses sds for the receive buffer, and then casts it to a
struct and accesses a 64 bit variable.
This commit replaces the use of sds with a simple reallocated buffer.

(cherry picked from commit b71d06c269878887eed85b63a731c3d4ad7a8b12)
2021-01-12 16:25:37 +02:00
Oran Agra
de4f04a272 Fix cluster access to unaligned memory (SIGBUS on old ARM) (#7958)
Turns out this was broken since version 4.0 when we added sds size
classes.
The cluster code uses sds for the receive buffer, and then casts it to a
struct and accesses a 64 bit variable.
This commit replaces the use of sds with a simple reallocated buffer.

(cherry picked from commit 380f6048e0bbc762f12fa50b57b73cf29049f967)
2021-01-12 16:25:37 +02:00
Oran Agra
39ac9ba73f prevent client tracking from causing feedback loop in performEvictions (#8100)
When client tracking is enabled signalModifiedKey can increase memory usage,
this can cause the loop in performEvictions to keep running since it was measuring
the memory usage impact of signalModifiedKey.

The section that measures the memory impact of the eviction should be just on dbDelete,
excluding keyspace notification, client tracking, and propagation to AOF and replicas.

This resolves part of the problem described in #8069
p.s. fix took 1 minute, test took about 3 hours to write.

(cherry picked from commit b640e2944e42759412ac67228cf64c43dfbed9c3)
2021-01-12 16:25:37 +02:00