27332 Commits

Author SHA1 Message Date
Yossi Gottlieb
cfccfbd6f4 Module API: Fail ineffective auth calls.
The client pointed to by the module context may in some cases be a fake
client. RM_Authenticate*() calls in this case would be ineffective but
appear to succeed, and this change fails them to make it easier to catch
such cases.
2020-09-30 17:06:57 +03:00
Yossi Gottlieb
3bd9d0cc85 TLS: Do not require CA config if not used. (#7862)
The tls-ca-cert or tls-ca-cert-dir configuration parameters are only
used when Redis needs to authenticate peer certificates, in one of these
scenarios:

1. Incoming clients or replicas, with `tls-auth-clients` enabled.
2. A replica authenticating the master's peer certificate.
3. Cluster nodes authenticating other nodes when establishing the bus
   protocol connection.
2020-09-29 20:48:21 +03:00
Yossi Gottlieb
1591e3479d
TLS: Do not require CA config if not used. (#7862)
The tls-ca-cert or tls-ca-cert-dir configuration parameters are only
used when Redis needs to authenticate peer certificates, in one of these
scenarios:

1. Incoming clients or replicas, with `tls-auth-clients` enabled.
2. A replica authenticating the master's peer certificate.
3. Cluster nodes authenticating other nodes when establishing the bus
   protocol connection.
2020-09-29 20:48:21 +03:00
YoongHM
0ab96fcc63 Fix compilation warnings in Lua and jemalloc dependencies (#7785)
- The argument `u` in for `ar` is ignored (and generates warnings since `D` became the default.
  All it does is avoid updating unchanged objects (shouldn't have any impact on our build)
- Enable `LUA_USE_MKSTEMP` to force the use of `mkstemp()` instead of `tmpname()` (which is dead
  code in redis anyway).
- Remove unused variable `c` in `f_parser()`
- Removed misleadingly indented space in `luaL_loadfile()` and ``addfield()`

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-29 17:10:54 +03:00
YoongHM
448c435b1b
Fix compilation warnings in Lua and jemalloc dependencies (#7785)
- The argument `u` in for `ar` is ignored (and generates warnings since `D` became the default.
  All it does is avoid updating unchanged objects (shouldn't have any impact on our build)
- Enable `LUA_USE_MKSTEMP` to force the use of `mkstemp()` instead of `tmpname()` (which is dead
  code in redis anyway).
- Remove unused variable `c` in `f_parser()`
- Removed misleadingly indented space in `luaL_loadfile()` and ``addfield()`

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-29 17:10:54 +03:00
Oran Agra
c11bda25fd warning: comparison between signed and unsigned integer in 32bit build (#7838) 2020-09-29 17:03:47 +03:00
Oran Agra
ef33252c43
warning: comparison between signed and unsigned integer in 32bit build (#7838) 2020-09-29 17:03:47 +03:00
David CARLIER
d535a5061c Add support for Haiku OS (#7435) 2020-09-29 15:52:13 +03:00
David CARLIER
f971a5d8ed
Add support for Haiku OS (#7435) 2020-09-29 15:52:13 +03:00
Gavrie Philipson
ce5efb444b Fix typo in module API docs (#7861) 2020-09-29 13:10:08 +03:00
Gavrie Philipson
49383a4af7
Fix typo in module API docs (#7861) 2020-09-29 13:10:08 +03:00
David CARLIER
520c3b26c3 getting rss size implementation for netbsd (#7293) 2020-09-29 08:49:35 +03:00
David CARLIER
ce8bfc56ad
getting rss size implementation for netbsd (#7293) 2020-09-29 08:49:35 +03:00
John Sully
c0df1ac173 Test RDB merge on load with active replication
Former-commit-id: 28183f4b66fc4c865048080b61e599eeb1d2293b
2020-09-29 03:26:06 +00:00
John Sully
3c0556093b Test RDB merge on load with active replication
Former-commit-id: 28183f4b66fc4c865048080b61e599eeb1d2293b
2020-09-29 03:26:06 +00:00
Ben Schermel
a0ca031b8c update deb changelog
Former-commit-id: 63c47a09e900e28f018cfb2c6d92901bd9e61e61
2020-09-28 23:42:54 +00:00
Ben Schermel
a472d76dd1 update deb changelog
Former-commit-id: 63c47a09e900e28f018cfb2c6d92901bd9e61e61
2020-09-28 23:42:54 +00:00
Ben Schermel
f338a3024b update deb changelog
Former-commit-id: 087af94c229469da06dc9e26d3a65c1e9a5d141b
2020-09-28 20:51:31 +00:00
Ben Schermel
be7b877b8b update deb changelog
Former-commit-id: 087af94c229469da06dc9e26d3a65c1e9a5d141b
2020-09-28 20:51:31 +00:00
John Sully
3fbb5b5efa Drop min-clients-per-thread to a more reasonable number
Former-commit-id: bf3a3225fccfb63f7b5e03ef0b362b788b97026d
2020-09-28 19:19:47 +00:00
John Sully
0dd31eeed5 Drop min-clients-per-thread to a more reasonable number
Former-commit-id: bf3a3225fccfb63f7b5e03ef0b362b788b97026d
2020-09-28 19:19:47 +00:00
John Sully
59e86a111b Drop min-clients-per-thread to a more reasonable number
Former-commit-id: a0abc1eddd071f984950ad8918fad0259c495184
2020-09-28 19:17:36 +00:00
John Sully
a4aa190f20 Drop min-clients-per-thread to a more reasonable number
Former-commit-id: a0abc1eddd071f984950ad8918fad0259c495184
2020-09-28 19:17:36 +00:00
John Sully
bdd58afb8f bump version 6.0.16
Former-commit-id: dab9296d7cede9d692e65403b5fa1fa2663fdc6c
2020-09-28 19:14:17 +00:00
John Sully
5f01bf9c91 bump version 6.0.16
Former-commit-id: dab9296d7cede9d692e65403b5fa1fa2663fdc6c
2020-09-28 19:14:17 +00:00
John Sully
0f9bb23dc5 Merge branch 'unstable' into RELEASE_6
Former-commit-id: a6f5a391a140aa9a28464a00ea4148f387bb8d98
2020-09-28 18:55:24 +00:00
John Sully
8b1dad9d92 Merge branch 'unstable' into RELEASE_6
Former-commit-id: a6f5a391a140aa9a28464a00ea4148f387bb8d98
2020-09-28 18:55:24 +00:00
WuYunlong
66c889702b Normalize sds test mechanism together with some compile warnings. (#7854) 2020-09-28 11:27:26 +03:00
WuYunlong
c2e5546071
Normalize sds test mechanism together with some compile warnings. (#7854) 2020-09-28 11:27:26 +03:00
Oran Agra
d89ae2d7ab Fix new obuf-limits tests to work with TLS (#7848)
Also stabilize new shutdown tests on slow machines (valgrind)
2020-09-27 17:13:33 +03:00
Oran Agra
8aa083bd28
Fix new obuf-limits tests to work with TLS (#7848)
Also stabilize new shutdown tests on slow machines (valgrind)
2020-09-27 17:13:33 +03:00
caozb
01694608cb ignore slaveof no one in redis.conf (#7842)
when slaveof config is "no one", reset any pre-existing config and resume.

also solve a memory leak if slaveof appears twice.
and fail loading if port number is out of range or not an integer.

Co-authored-by: caozhengbin <caozb@yidingyun.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-27 15:40:07 +03:00
caozb
a295770e32
ignore slaveof no one in redis.conf (#7842)
when slaveof config is "no one", reset any pre-existing config and resume.

also solve a memory leak if slaveof appears twice.
and fail loading if port number is out of range or not an integer.

Co-authored-by: caozhengbin <caozb@yidingyun.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-27 15:40:07 +03:00
Wang Yuan
9bdef76f8e Don't support Gopher if enable io threads to read queries (#7851)
There's currently an issue with IO threads and gopher (issuing lookupKey from within the thread).
simply fix is to just not support it for now.
2020-09-27 12:35:16 +03:00
Wang Yuan
c9f00bcce2
Don't support Gopher if enable io threads to read queries (#7851)
There's currently an issue with IO threads and gopher (issuing lookupKey from within the thread).
simply fix is to just not support it for now.
2020-09-27 12:35:16 +03:00
Wang Yuan
917043fa43 Set 'loading' and 'shutdown_asap' to volatile sig_atomic_t type (#7845)
We may access and modify these two variables in signal handler function,
to guarantee them async-signal-safe, so we should set them to volatile
sig_atomic_t type.

It doesn't look like this could have caused any real issue, and it seems that
signals are handled in main thread on most platforms. But we want to follow C
and POSIX standard in signal handler function.
2020-09-25 16:25:47 +03:00
Wang Yuan
f1863a1fe7
Set 'loading' and 'shutdown_asap' to volatile sig_atomic_t type (#7845)
We may access and modify these two variables in signal handler function,
to guarantee them async-signal-safe, so we should set them to volatile
sig_atomic_t type.

It doesn't look like this could have caused any real issue, and it seems that
signals are handled in main thread on most platforms. But we want to follow C
and POSIX standard in signal handler function.
2020-09-25 16:25:47 +03:00
Uri Shachar
8dbe91f031 Fix config rewrite file handling to make it really atomic (#7824)
Make sure we handle short writes correctly, sync to disk after writing  and use
rename to make sure the replacement is actually atomic.
In any case of failure old configuration will remain in place.

Also, add some additional logging to make it easier to diagnose rewrite problems.
2020-09-25 12:55:45 +03:00
Uri Shachar
c30bd02c9d
Fix config rewrite file handling to make it really atomic (#7824)
Make sure we handle short writes correctly, sync to disk after writing  and use
rename to make sure the replacement is actually atomic.
In any case of failure old configuration will remain in place.

Also, add some additional logging to make it easier to diagnose rewrite problems.
2020-09-25 12:55:45 +03:00
WuYunlong
d119448881 Add fsync to readSyncBulkPayload(). (#7839)
We should sync temp DB file before renaming as rdb_fsync_range does not use
flag `SYNC_FILE_RANGE_WAIT_AFTER`.

Refer to `Linux Programmer's Manual`:
SYNC_FILE_RANGE_WAIT_AFTER
    Wait upon write-out of all pages in the range after performing any write.
2020-09-25 08:08:06 +03:00
WuYunlong
0d62caab21
Add fsync to readSyncBulkPayload(). (#7839)
We should sync temp DB file before renaming as rdb_fsync_range does not use
flag `SYNC_FILE_RANGE_WAIT_AFTER`.

Refer to `Linux Programmer's Manual`:
SYNC_FILE_RANGE_WAIT_AFTER
    Wait upon write-out of all pages in the range after performing any write.
2020-09-25 08:08:06 +03:00
John Sully
cb6bc67a87 Merge branch 'unstable' into keydbpro
Former-commit-id: d1986d9fcdfd56f2c30ee95edef6d5260e0aa777
2020-09-24 22:02:30 +00:00
John Sully
ad4c2390a8 Merge branch 'unstable' into keydbpro
Former-commit-id: d1986d9fcdfd56f2c30ee95edef6d5260e0aa777
2020-09-24 22:02:30 +00:00
Wen Hui
c67656fa35 rdb.c: handle fclose error case differently to avoid double fclose (#7307)
When fclose would fail, the previous implementation would have attempted to do fclose again
this can in theory lead to segfault.

other changes:
check for non-zero return value as failure rather than a specific error code.
this doesn't fix a real bug, just a minor cleanup.
2020-09-24 18:17:53 +03:00
Wen Hui
323029baa6
rdb.c: handle fclose error case differently to avoid double fclose (#7307)
When fclose would fail, the previous implementation would have attempted to do fclose again
this can in theory lead to segfault.

other changes:
check for non-zero return value as failure rather than a specific error code.
this doesn't fix a real bug, just a minor cleanup.
2020-09-24 18:17:53 +03:00
Wang Yuan
3085577c09 Don't write replies if close the client ASAP (#7202)
Before this commit, we would have continued to add replies to the reply buffer even if client
output buffer limit is reached, so the used memory would keep increasing over the configured limit.
What's more, we shouldn’t write any reply to the client if it is set 'CLIENT_CLOSE_ASAP' flag
because that doesn't conform to its definition and we will close all clients flagged with
'CLIENT_CLOSE_ASAP' in ‘beforeSleep’.

Because of code execution order, before this, we may firstly write to part of the replies to
the socket before disconnecting it, but in fact, we may can’t send the full replies to clients
since OS socket buffer is limited. But this unexpected behavior makes some commands work well,
for instance ACL DELUSER, if the client deletes the current user, we need to send reply to client
and close the connection, but before, we close the client firstly and write the reply to reply
buffer. secondly, we shouldn't do this despite the fact it works well in most cases.

We add a flag 'CLIENT_CLOSE_AFTER_COMMAND' to mark clients, this flag means we will close the
client after executing commands and send all entire replies, so that we can write replies to
reply buffer during executing commands, send replies to clients, and close them later.

We also fix some implicit problems. If client output buffer limit is enforced in 'multi/exec',
all commands will be executed completely in redis and clients will not read any reply instead of
partial replies. Even more, if the client executes 'ACL deluser' the using user in 'multi/exec',
it will not read the replies after 'ACL deluser' just like before executing 'client kill' itself
in 'multi/exec'.

We added some tests for output buffer limit breach during multi-exec and using a pipeline of
many small commands rather than one with big response.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-24 16:01:41 +03:00
Wang Yuan
57709c4bc6
Don't write replies if close the client ASAP (#7202)
Before this commit, we would have continued to add replies to the reply buffer even if client
output buffer limit is reached, so the used memory would keep increasing over the configured limit.
What's more, we shouldn’t write any reply to the client if it is set 'CLIENT_CLOSE_ASAP' flag
because that doesn't conform to its definition and we will close all clients flagged with
'CLIENT_CLOSE_ASAP' in ‘beforeSleep’.

Because of code execution order, before this, we may firstly write to part of the replies to
the socket before disconnecting it, but in fact, we may can’t send the full replies to clients
since OS socket buffer is limited. But this unexpected behavior makes some commands work well,
for instance ACL DELUSER, if the client deletes the current user, we need to send reply to client
and close the connection, but before, we close the client firstly and write the reply to reply
buffer. secondly, we shouldn't do this despite the fact it works well in most cases.

We add a flag 'CLIENT_CLOSE_AFTER_COMMAND' to mark clients, this flag means we will close the
client after executing commands and send all entire replies, so that we can write replies to
reply buffer during executing commands, send replies to clients, and close them later.

We also fix some implicit problems. If client output buffer limit is enforced in 'multi/exec',
all commands will be executed completely in redis and clients will not read any reply instead of
partial replies. Even more, if the client executes 'ACL deluser' the using user in 'multi/exec',
it will not read the replies after 'ACL deluser' just like before executing 'client kill' itself
in 'multi/exec'.

We added some tests for output buffer limit breach during multi-exec and using a pipeline of
many small commands rather than one with big response.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-24 16:01:41 +03:00
Guy Korland
04945e0e6d Fix RedisModule_HashGet examples (#6697) 2020-09-24 12:45:30 +03:00
Guy Korland
b464afb9e2
Fix RedisModule_HashGet examples (#6697) 2020-09-24 12:45:30 +03:00
valentinogeron
7c45f38436 Stream: Inconsistency between master and replica some XREADGROUP case (#7526)
XREADGROUP auto-creates the consumer inside the consumer group the
first time it saw it.
When XREADGROUP is being used with NOACK option, the message will not
be added into the client's PEL and XGROUP SETID would be propagated.
When the replica gets the XGROUP SETID it will only update the last delivered
id of the group, but will not create the consumer.

So, in this commit XGROUP CREATECONSUMER is being added.
Command pattern: XGROUP CREATECONSUMER <key> <group> <consumer>.

When NOACK option is being used, createconsumer command would be
propagated as well.

In case of AOFREWRITE, consumer with an empty PEL would be saved with
XGROUP CREATECONSUMER whereas consumer with pending entries would be
saved with XCLAIM
2020-09-24 12:02:40 +03:00