979 Commits

Author SHA1 Message Date
Oran Agra
8704be6947 TLS: Implement support for write barrier. 2019-10-07 21:06:30 +03:00
Oran Agra
6fd5ff8c98 diskless replication rdb transfer uses pipe, and writes to sockets form the parent process.
misc:
- handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents
- fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed)
- add key-load-delay config for testing
- trim connShutdown which is no longer needed
- rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs
- don't detect rdb child exited (don't call wait3) until we detect the pipe is closed
- Cleanup bad optimization from rio.c, add another one
2019-10-07 21:06:30 +03:00
Yossi Gottlieb
10ffeb03e4 TLS: Connections refactoring and TLS support.
* Introduce a connection abstraction layer for all socket operations and
integrate it across the code base.
* Provide an optional TLS connections implementation based on OpenSSL.
* Pull a newer version of hiredis with TLS support.
* Tests, redis-cli updates for TLS support.
2019-10-07 21:06:13 +03:00
antirez
59982e85eb Modules: add RM_Replicate() Tcl test file & fix the module. 2019-10-03 18:44:50 +02:00
antirez
d6d4fea080 Modules: RM_Replicate() test with threads. 2019-10-03 13:23:53 +02:00
antirez
d859b42899 Modules: RM_Replicate() test module: initial implementation. 2019-10-03 13:06:13 +02:00
Salvatore Sanfilippo
b5f075f8ae Merge pull request #6270 from oranagra/modules_info
Extend modules API to allow modules report to redis INFO
2019-10-01 18:02:33 +02:00
Oran Agra
bd49c4679b Merge remote-tracking branch 'antirez/unstable' into modules_info 2019-09-30 20:47:35 +03:00
antirez
6118989790 ACLs: change hashed passwords opcode to also remove them.
Related to PR #6405
2019-09-30 18:28:45 +02:00
Madelyn Olson
364c8601e3 Allowed passing in of password hash and fixed config rewrite 2019-09-30 17:57:49 +02:00
Salvatore Sanfilippo
5c5761bcd6 Merge branch 'unstable' into modules_fork 2019-09-27 11:24:06 +02:00
Oran Agra
2add4524c5 Fix lastbgsave_status, when new child signal handler get intended kill
And add a test for that.
2019-09-26 15:16:34 +03:00
Salvatore Sanfilippo
f11b71b9b8 Merge pull request #6235 from oranagra/module_rdb_load_errors
Allow modules to handle RDB loading errors.
2019-09-26 11:52:42 +02:00
valentino
bd32b8b2a8 DISCARD should not fail during OOM
discard command should not fail during OOM, otherwise client MULTI state
will not be cleared.
2019-09-22 09:22:53 +03:00
Oran Agra
8319ffb34d Module INFO, support default section for simple modules 2019-08-18 10:01:57 +03:00
Oran Agra
d4c24a3067 Module INFO, add support for dict fields, rename API to have common prefix 2019-08-18 09:41:45 +03:00
Oran Agra
037ac27e41 Add test for module diskless short reads 2019-07-30 15:20:03 +03:00
Oran Agra
b2fce80695 Extend modules API to allow modules report to redis INFO
this implements #6012
2019-07-24 12:58:15 +03:00
Oran Agra
ff0780e8e6 Implement module api for aux data in rdb
Other changes:
* fix memory leak in error handling of rdb loading of type OBJ_MODULE
2019-07-22 21:15:33 +03:00
Oran Agra
73a945c73c prevent diskless replica from terminating on short read
now that replica can read rdb directly from the socket, it should avoid exiting
on short read and instead try to re-sync.

this commit tries to have minimal effects on non-diskless rdb reading.
and includes a test that tries to trigger this scenario on various read cases.
2019-07-17 16:46:22 +02:00
Oran Agra
e70fbad802 Module API for Forking
* create module API for forking child processes.
* refactor duplicate code around creating and tracking forks by AOF and RDB.
* child processes listen to SIGUSR1 and dies exitFromChild in order to
  eliminate a valgrind warning of unhandled signal.
* note that BGSAVE error reply has changed.

valgrind error is:
  Process terminating with default action of signal 10 (SIGUSR1)
2019-07-17 16:40:24 +03:00
Salvatore Sanfilippo
89e00762e7 Merge pull request #5779 from madolson/dev-unstable-geo-bugfix
Fixed a rounding bug in geo.tcl
2019-07-15 23:41:57 +02:00
Oran Agra
29754ebe22 diskless replication on slave side (don't store rdb to file), plus some other related fixes
The implementation of the diskless replication was currently diskless only on the master side.
The slave side was still storing the received rdb file to the disk before loading it back in and parsing it.

This commit adds two modes to load rdb directly from socket:
1) when-empty
2) using "swapdb"
the third mode of using diskless slave by flushdb is risky and currently not included.

other changes:
--------------
distinguish between aof configuration and state so that we can re-enable aof only when sync eventually
succeeds (and not when exiting from readSyncBulkPayload after a failed attempt)
also a CONFIG GET and INFO during rdb loading would have lied

When loading rdb from the network, don't kill the server on short read (that can be a network error)

Fix rdb check when performed on preamble AOF

tests:
run replication tests for diskless slave too
make replication test a bit more aggressive
Add test for diskless load swapdb
2019-07-08 15:37:48 +03:00
Angus Pearson
4bbaf621a1 Implement SCAN cursor [TYPE type] modifier suggested in issue #6107.
Add tests to check basic functionality of this optional keyword, and also tested with
a module (redisgraph). Checked quickly with valgrind, no issues.

Copies name the type name canonicalisation code from `typeCommand`, perhaps this would
be better factored out to prevent the two diverging and both needing to be edited to
add new `OBJ_*` types, but this is a little fiddly with C strings.

The [redis-doc](https://github.com/antirez/redis-doc/blob/master/commands.json) repo
will need to be updated with this new arg if accepted.

A quirk to be aware of here is that the GEO commands are backed by zsets not their own
type, so they're not distinguishable from other zsets.

Additionally, for sparse types this has the same behaviour as `MATCH` in that it may
return many empty results before giving something, even for large `COUNT`s.
2019-05-22 16:39:04 +01:00
antirez
78978eb5e1 Test: fix slowlog test false positive.
In fast systems "SLOWLOG RESET" is fast enough to don't be logged even
when the time limit is "1" sometimes. Leading to false positives such
as:

[err]: SLOWLOG - can be disabled in tests/unit/slowlog.tcl
Expected '1' to be equal to '0'
2019-05-14 16:55:52 +02:00
antirez
5606036fb4 Fix test false positive introduced by threaded I/O.
Now clients that are ready to be terminated asynchronously are processed
more often in beforeSleep() instead of being processed in serverCron().
This means that the test will not be able to catch the moment the client
was terminated, also note that the 'omem' figure now changes in big
steps, because of the new client output buffers layout.

So we have to change the test range in order to accomodate for that.
Yet the test is useful enough to be worth taking, even if its precision
is reduced by this commit. Probably if we get more problems, a thing
that makes sense is just to check that the limit is < 200k. That's more
than enough actually.
2019-05-13 17:27:10 +02:00
zhaozhao.zz
1e45354eb6 test cases: skiptill -> skip-till 2019-05-10 16:27:25 +08:00
Oran Agra
c76bb465f2 make replication tests more stable on slow machines
solving few replication related tests race conditions which fail on slow machines

bugfix in slave buffers test: since the test is executed twice, each time with
a different commands count, the threshold for the delta can't be a constant.
2019-05-05 08:25:01 +03:00
Salvatore Sanfilippo
f55e492701 Merge pull request #5961 from yossigo/modules-tests
Modules tests
2019-04-11 07:56:23 -07:00
antirez
1a505a3ba9 ACL: Fix memory leak in ACLResetSubcommandsForCommand().
This commit fixes bug reported at #5998. Thanks to @tomcat1102.
2019-04-08 18:08:37 +02:00
antirez
b364f3fc21 ACL: regression test for #5998. 2019-04-08 18:06:50 +02:00
Yossi Gottlieb
a631f66710 Add runtest-moduleapi with commandfilter coverage. 2019-03-24 12:03:03 +02:00
Yossi Gottlieb
2d4635b483 fix: missing initialization. 2019-03-24 12:00:33 +02:00
Yossi Gottlieb
898677d59e CommandFilter API: REDISMODULE_CMDFILTER_NOSELF.
Add a flag to automatically protect filters from being called
recursively by their own module.
2019-03-21 19:53:12 +02:00
Yossi Gottlieb
c675d44488 CommandFilter API: Add unregister option.
A filter handle is returned and can be used to unregister a filter.  In
the future it can also be used to further configure or manipulate the
filter.

Filters are now automatically unregistered when a module unloads.
2019-03-21 14:44:49 +02:00
Yossi Gottlieb
06a6d70ab5 CommandFilter API: hellofilter and tests. 2019-03-18 23:07:28 +02:00
Yossi Gottlieb
95881cec60 Add command filter Module API tests. 2019-03-18 19:34:52 +02:00
antirez
ca291b0951 HyperLogLog: speedup fuzz test. 2019-03-15 17:13:19 +01:00
antirez
a47cd50a0d HyperLogLog: dense/sparse repr parsing fuzz test. 2019-03-15 13:52:29 +01:00
Salvatore Sanfilippo
cb44f7ece0 Merge pull request #5907 from spjwebster/xclaim-increment-delivery-count
Increment delivery counter on XCLAIM unless RETRYCOUNT specified
2019-03-13 11:55:46 +01:00
Steve Webster
c652f706cb Only increment delivery count if JUSTID option is omitted 2019-03-12 20:27:53 +00:00
Salvatore Sanfilippo
685d10894b Merge pull request #2774 from rouzier/blocking-list-commands-support-milliseconds-floating
Added millisecond resolution for blpop command && friends
2019-03-12 18:10:28 +01:00
Steve Webster
8fd63c220a Increment delivery counter on XCLAIM unless RETRYCOUNT specified
The XCLAIM docs state the XCLAIM increments the delivery counter for
messages. This PR makes the code match the documentation - which seems
like the desired behaviour - whilst still allowing RETRYCOUNT to be
specified manually.

My understanding of the way streamPropagateXCLAIM() works is that this
change will safely propagate to replicas since retry count is pulled
directly from the streamNACK struct.

Fixes #5194
2019-03-08 17:09:11 +00:00
antirez
1db503c865 Acl: Test: check command rules synthesis. 2019-01-30 12:01:37 +01:00
antirez
c4f8e91a6c ACL: Test: check subcommands (test fails). 2019-01-28 18:29:22 +01:00
antirez
0f7c574f9f ACL: Test: nopass user flag, commands/groups +/- rules. 2019-01-28 12:40:07 +01:00
antirez
3cf816c703 ACL: Test: check default behavior and keys ACLs. 2019-01-28 12:33:18 +01:00
antirez
9246923436 ACL: Test: add the acl tests as default unit. 2019-01-28 12:12:04 +01:00
antirez
a791d5a7a1 ACL: Test: check passwords validity and removal. 2019-01-28 12:06:25 +01:00
antirez
f94c5c77b0 ACL: Test: enabled/disabled user. 2019-01-28 11:44:10 +01:00