336 Commits

Author SHA1 Message Date
Oran Agra
6fd5ff8c98 diskless replication rdb transfer uses pipe, and writes to sockets form the parent process.
misc:
- handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents
- fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed)
- add key-load-delay config for testing
- trim connShutdown which is no longer needed
- rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs
- don't detect rdb child exited (don't call wait3) until we detect the pipe is closed
- Cleanup bad optimization from rio.c, add another one
2019-10-07 21:06:30 +03:00
Yossi Gottlieb
10ffeb03e4 TLS: Connections refactoring and TLS support.
* Introduce a connection abstraction layer for all socket operations and
integrate it across the code base.
* Provide an optional TLS connections implementation based on OpenSSL.
* Pull a newer version of hiredis with TLS support.
* Tests, redis-cli updates for TLS support.
2019-10-07 21:06:13 +03:00
antirez
2e1e170aad RESP3: fix cases of NULL reported instead of empty aggregate. 2019-09-02 12:50:47 +02:00
antirez
e87768e8ca Replication: clarify why repl_put_online_on_ack exists at all. 2019-08-05 17:38:15 +02:00
Salvatore Sanfilippo
3e66c43e84 Merge pull request #6266 from madolson/dev-unstable-hide-auth-and-hello
Hide HELLO and AUTH from slowlog and monitor
2019-07-31 11:12:46 +02:00
Salvatore Sanfilippo
8e2a99eb4b Merge pull request #6264 from oranagra/modules_api_aux_rdb
Implement module api for aux data in rdb
2019-07-30 11:50:44 +02:00
antirez
83b8fbf580 Client side caching: config option for table fill rate. 2019-07-24 11:35:01 +02:00
antirez
1d54931685 Client side caching: show tracking slots usage in INFO. 2019-07-23 11:02:14 +02:00
antirez
79a73bb95a Client side caching: implement full slot limit function. 2019-07-23 10:57:22 +02:00
Madelyn Olson
4520e87800 Hide HELLO and AUTH from slowlog and monitor 2019-07-22 22:53:15 -07:00
Oran Agra
ff0780e8e6 Implement module api for aux data in rdb
Other changes:
* fix memory leak in error handling of rdb loading of type OBJ_MODULE
2019-07-22 21:15:33 +03:00
antirez
f2f3d91e78 Client side caching: split invalidation into key / slot. 2019-07-22 18:59:53 +02:00
zhaozhao.zz
8d5e3b7893 Client side caching: implement trackingInvalidateKeysOnFlush() 2019-07-17 20:33:52 +08:00
Oran Agra
29754ebe22 diskless replication on slave side (don't store rdb to file), plus some other related fixes
The implementation of the diskless replication was currently diskless only on the master side.
The slave side was still storing the received rdb file to the disk before loading it back in and parsing it.

This commit adds two modes to load rdb directly from socket:
1) when-empty
2) using "swapdb"
the third mode of using diskless slave by flushdb is risky and currently not included.

other changes:
--------------
distinguish between aof configuration and state so that we can re-enable aof only when sync eventually
succeeds (and not when exiting from readSyncBulkPayload after a failed attempt)
also a CONFIG GET and INFO during rdb loading would have lied

When loading rdb from the network, don't kill the server on short read (that can be a network error)

Fix rdb check when performed on preamble AOF

tests:
run replication tests for diskless slave too
make replication test a bit more aggressive
Add test for diskless load swapdb
2019-07-08 15:37:48 +03:00
Salvatore Sanfilippo
9c1932beeb Merge pull request #6116 from AngusP/scan-types
SCAN: New Feature `SCAN cursor [TYPE type]` modifier suggested in issue #6107
2019-07-08 12:53:34 +02:00
Angus Pearson
00c9b4f15d Change typeNameCanonicalize -> getObjectTypeName, and other style changes 2019-07-08 11:04:37 +01:00
antirez
f3a8b5001d Client side caching: RESP2 support. 2019-07-05 12:24:28 +02:00
antirez
4b5027845e Client side caching: implement trackingInvalidateKey(). 2019-07-03 19:16:20 +02:00
antirez
a4e44726d0 Client side caching: hook inside call() for tracking. 2019-07-03 12:42:16 +02:00
antirez
b54789d472 Client side caching: enable tracking mode. 2019-06-30 06:19:08 -04:00
antirez
7f9de752de Client side caching: fields and flags for tracking mode. 2019-06-29 20:08:41 -04:00
Angus Pearson
49edda41b9 Spelling cannonical -> canonical 2019-06-13 17:49:33 +01:00
Angus Pearson
5ca48db2e2 Add char* typeNameCanonicalize(robj*) to remove duplicate code between SCAN and TYPE commands,
and to keep OBJ_* enum to string canonicalization in one place.
2019-06-10 17:41:44 +01:00
Salvatore Sanfilippo
d7dd023bfd Merge pull request #6053 from soloestoy/enhance-aof-fsync-everysec
aof: enhance AOF_FSYNC_EVERYSEC, more details in #5985
2019-05-10 18:06:40 +02:00
antirez
50b09bc408 Threaded IO: ability to disable reads from threaded path. 2019-05-06 18:02:51 +02:00
antirez
1347c13551 Threaded IO: parsing WIP 2: refactoring to parse from thread. 2019-05-06 18:02:51 +02:00
antirez
b9b29641cf Threaded IO: read side WIP 3. 2019-05-06 18:02:51 +02:00
antirez
7bae8afef8 Threaded IO: read side WIP. 2019-05-06 18:02:51 +02:00
antirez
dc4d13e751 Threaded IO: make num of I/O threads configurable. 2019-05-06 18:02:51 +02:00
Ubuntu
0353d13321 Threaded IO: stop threads when no longer needed + C11 in Makefile.
Now threads are stopped even when the connections drop immediately to
zero, not allowing the networking code to detect the condition and stop
the threads. serverCron() will handle that.
2019-05-06 18:02:51 +02:00
antirez
62261aa905 Threaded IO: implement handleClientsWithPendingWritesUsingThreads().
This is just an experiment for now, there are a couple of race
conditions, mostly harmless for the performance gain experiment that
this commit represents so far.

The general idea here is to take Redis single threaded and instead
fan-out on expansive kernel calls: write(2) in this case, but the same
concept could be easily implemented for read(2) and protcol parsing.

However just threading writes like in this commit, is enough to evaluate
if the approach is sounding.
2019-05-06 18:02:51 +02:00
zhaozhao.zz
7ff5c6011d aof: enhance AOF_FSYNC_EVERYSEC, more details in #5985 2019-04-29 14:38:28 +08:00
Oran Agra
48d14e5aa7 slave corrupts replication stream when module blocked client uses large reply (or POSTPONED_ARRAY)
when redis appends the blocked client reply list to the real client, it didn't
bother to check if it is in fact the master client. so a slave executing that
module command will send replies to the master, causing the master to send the
slave error responses, which will mess up the replication offset
(slave will advance it's replication offset, and the master does not)
2019-03-24 14:17:37 +02:00
Salvatore Sanfilippo
d415aa89c0 Merge pull request #5944 from yossigo/command-filtering
Command Filtering API
2019-03-22 17:43:49 +01:00
Salvatore Sanfilippo
e19f00f8b2 Merge pull request #5945 from dvirsky/miss_notification
Added keyspace miss notifications support
2019-03-22 17:41:00 +01:00
Dvir Volk
51a54dfde3 remove extra linebreak 2019-03-21 12:48:37 +02:00
Dvir Volk
50befc42ad added special flag for keyspace miss notifications 2019-03-21 11:47:14 +02:00
Yossi Gottlieb
bb6e8ba682 Initial command filter experiment. 2019-03-18 13:50:34 +02:00
Yossi Gottlieb
dd405d4026 Add RedisModule_GetKeyNameFromIO(). 2019-03-15 10:23:27 +02:00
Salvatore Sanfilippo
06339c7f54 Merge pull request #5834 from guybe7/trim_module_sds
Trim SDS free space of retained module strings
2019-03-14 12:41:31 +01:00
antirez
4e3f2ab269 Merge branch 'sharedapi' into unstable 2019-03-14 11:24:48 +01:00
Yuan Zhou
e5149be9de server.h: remove dead code
hashTypeTryObjectEncoding() is not used now
2019-03-07 18:38:27 +08:00
Salvatore Sanfilippo
e2a7df2d6c Merge pull request #5877 from vattezhang/unstable_sentinel_cmd
fix: fix sentinel command table and new flags format
2019-02-27 15:45:03 +01:00
vattezhang
034741f7e8 fix: fix sentinel command table and new flags format 2019-02-27 21:35:58 +08:00
antirez
8b01d81fd4 Merge branch 'gopher' into unstable 2019-02-25 18:16:58 +01:00
antirez
505815da4b RESP3: allow HELLO to be used with version = 2. 2019-02-25 16:41:00 +01:00
antirez
c8b67fea44 Gopher: initial request handling. 2019-02-21 23:13:08 +01:00
antirez
20c7d1b334 Gopher: config setting to turn support on/off. 2019-02-21 17:28:53 +01:00
zhaozhao.zz
3eb2f4ca14 ACL: show categories in COMMAND reply
Adding another new filed categories at the end of
command reply, it's easy to read and distinguish
flags and categories, also compatible with old format.
2019-02-14 00:13:01 +08:00
Guy Benoish
9cd3b12cdf Trim SDS free space of retained module strings
In some cases processMultibulkBuffer uses sdsMakeRoomFor to
expand the querybuf, but later in some cases it uses that query
buffer as is for an argv element (see "Optimization"), which means
that the sds in argv may have a lot of wasted space, and then in case
modules keep that argv RedisString inside their data structure, this
space waste will remain for long (until restarted from rdb).
2019-02-12 14:21:21 +01:00