Merge branch 'redis_merge' into unstable

Former-commit-id: ab1524ff1a8290199bc5a88e2bd9623566866d6f

.github/workflows/ci.yml

@@ -0,0 +1,28 @@
+name: CI
+
+on: [push, pull_request]
+
+jobs:
+  build-ubuntu:
+    strategy:
+      matrix:
+        platform: [ubuntu-latest, ubuntu-16.04]
+    runs-on: ${{ matrix.platform }}
+    steps:
+    - uses: actions/checkout@v1
+    - name: make
+      run: make
+    - name: test
+      run: |
+        sudo apt-get install tcl8.5
+        make test
+
+  build-macos-latest:
+    strategy:
+      matrix:
+        platform: [macos-latest, macOS-10.14]
+    runs-on: ${{ matrix.platform }}
+    steps:
+    - uses: actions/checkout@v1
+    - name: make
+      run: make

.gitignore

@@ -1,6 +1,8 @@
 .*.swp
 core
 *.o
+*.xo
+*.so
 *.d
 *.log
 dump.rdb
@@ -37,3 +39,4 @@ deps/lua/src/liblua.a
 .prerequisites
 *.dSYM
 Makefile.dep
+.vscode/*

00-RELEASENOTES

@@ -1,16 +1,130 @@
-Hello! This file is just a placeholder, since this is the "unstable" branch
+Redis 6.0 release notes
-of Redis, the place where all the development happens.
+=======================
 
-There is no release notes for this branch, it gets forked into another branch
+--------------------------------------------------------------------------------
-every time there is a partial feature freeze in order to eventually create
+Upgrade urgency levels:
-a new stable release.
 
-Usually "unstable" is stable enough for you to use it in development environments
+LOW:      No need to upgrade unless there are new features you want to use.
-however you should never use it in production environments. It is possible
+MODERATE: Program an upgrade of the server, but it's not urgent.
-to download the latest stable release here:
+HIGH:     There is a critical bug that may affect a subset of users. Upgrade!
+CRITICAL: There is a critical bug affecting MOST USERS. Upgrade ASAP.
+SECURITY: There are security fixes in the release.
+--------------------------------------------------------------------------------
 
-    http://download.redis.io/releases/redis-stable.tar.gz
+================================================================================
+Redis 6.0 RC1   Released Thu Dec 19 09:58:24 CEST 2019
+================================================================================
 
-More information is available at http://redis.io
+Upgrade urgency LOW: This is the first RC of Redis 6.
 
-Happy hacking!
+Introduction to the Redis 6 release
+===================================
+
+Redis 6 improves Redis in a number of key areas and is one of the largest
+Redis releases in the history of the project, so here we'll list only
+the biggest features in this release:
+
+* The modules system now has a number of new APIs that allow module authors
+  to make things otherwise not possible in the past. It is possible to
+  store arbitrary module private data in RDB files, to hook on different
+  server events, capture and rewrite commands executions, block clients on
+  keys, and so forth. 
+* The Redis active expire cycle was rewritten for much faster eviction of keys
+  that are already expired. Now the effort is tunable.
+* Redis now supports SSL on all channels.
+* ACL support, you can define users that can run only certain commands and/or
+  can only access only certain keys patterns.
+* Redis now supports a new protocol called RESP3, which returns more
+  semantical replies: new clients using this protocol can understand just
+  from the reply what type to return to the calling program.
+* There is server-side support for client-side caching of key values. This
+  feature is still experimental and will get more changes during the next
+  release candidates, but you can already test it and read about it here:
+  https://redis.io/topics/client-side-caching
+* Redis can now optionally use threads to handle I/O, allowing to serve
+  2 times as much operations per second in a single instance when
+  pipelining cannot be used.
+* Diskless replication is now supported even on replicas: a replica is now
+  able, under certain conditions the user can configure, to load the RDB
+  in the first synchronization directly from the socket to the memory.
+* Redis-benchmark now supports a Redis Cluster mode.
+* SRANDMEMBER and similar commands have a better distribution.
+* Redis-cli improvements.
+* Systemd support rewritten.
+* A Redis Cluster proxy was released here:
+  https://github.com/artix75/redis-cluster-proxy
+* A Disque module for Redis was released here:
+  https://github.com/antirez/disque-module
+
+Thanks to all the users and developers who made this release possible.
+We'll follow up with more RC releases, until the code looks production ready
+and we don't get reports of serious issues for a while.
+
+A special thank you for the amount of work put into this release
+(in decreasing number of commits, only listing contributors with more
+than a single commit) by:
+
+   685  antirez
+    81  zhaozhao.zz
+    76  Oran Agra
+    51  artix
+    28  Madelyn Olson
+    27  Yossi Gottlieb
+    15  David Carlier
+    14  Guy Benoish
+    14  Guy Korland
+    13  Itamar Haber
+     9  Angus Pearson
+     8  WuYunlong
+     8  yongman
+     7  vattezhang
+     7  Chris Lamb
+     5  Dvir Volk
+     5  meir@redislabs.com
+     5  chendianqiang
+     5  John Sully
+     4  dejun.xdj
+     4  Daniel Dai
+     4  Johannes Truschnigg
+     4  swilly22
+     3  Bruce Merry
+     3  filipecosta90
+     3  youjiali1995
+     2  James Rouzier
+     2  Andrey Bugaevskiy
+     2  Brad Solomon
+     2  Hamid Alaei
+     2  Michael Chaten
+     2  Steve Webster
+     2  Wander Hillen
+     2  Weiliang Li
+     2  Yuan Zhou
+     2  charsyam
+     2  hujie
+     2  jem
+     2  shenlongxing
+     2  valentino
+     2  zhudacai 00228490
+     2  喜欢兰花山丘
+
+Migrating from 5.0 to 6.0
+=========================
+
+Redis 6.0 is mostly a strict superset of 5.0, you should not have any problem
+upgrading your application from 5.0 to 6.0. However this is a list of small
+non-backward compatible changes introduced in the 6.0 release:
+
+* Nothing found yet.
+
+--------------------------------------------------------------------------------
+
+Credits: For each release, a list of changes with the relative author is
+provided. Where not specified the implementation and design is done by
+Salvatore Sanfilippo. Thanks to Redis Labs for making all this possible.
+Also many thanks to all the other contributors and the amazing community
+we have.
+
+Commit messages may contain additional credits.
+
+Enjoy,
+Salvatore

README.md

@@ -251,8 +251,49 @@ $ docker run -it --rm -v /path-to-dump-binaries:/keydb_bin eqalpha/keydb-build-b
 ```
 Please note that you will need libcurl4-openssl-dev in order to run keydb. With flash version you may need libnuma-dev and libtool installed in order to run the binaries. Keep this in mind especially when running in a container. For a copy of all our Dockerfiles, please see them on [docs]( https://docs.keydb.dev/docs/dockerfiles/).
 
+<<<<<<< HEAD
 Code contributions
 -----------------
+=======
+One of the most important functions inside this file is `replicationFeedSlaves()` that writes commands to the clients representing replica instances connected
+to our master, so that the replicas can get the writes performed by the clients:
+this way their data set will remain synchronized with the one in the master.
+
+This file also implements both the `SYNC` and `PSYNC` commands that are
+used in order to perform the first synchronization between masters and
+replicas, or to continue the replication after a disconnection.
+
+Other C files
+---
+
+* `t_hash.c`, `t_list.c`, `t_set.c`, `t_string.c`, `t_zset.c` and `t_stream.c` contains the implementation of the Redis data types. They implement both an API to access a given data type, and the client commands implementations for these data types.
+* `ae.c` implements the Redis event loop, it's a self contained library which is simple to read and understand.
+* `sds.c` is the Redis string library, check http://github.com/antirez/sds for more information.
+* `anet.c` is a library to use POSIX networking in a simpler way compared to the raw interface exposed by the kernel.
+* `dict.c` is an implementation of a non-blocking hash table which rehashes incrementally.
+* `scripting.c` implements Lua scripting. It is completely self contained from the rest of the Redis implementation and is simple enough to understand if you are familar with the Lua API.
+* `cluster.c` implements the Redis Cluster. Probably a good read only after being very familiar with the rest of the Redis code base. If you want to read `cluster.c` make sure to read the [Redis Cluster specification][3].
+
+[3]: http://redis.io/topics/cluster-spec
+
+Anatomy of a Redis command
+---
+
+All the Redis commands are defined in the following way:
+
+    void foobarCommand(client *c) {
+        printf("%s",c->argv[1]->ptr); /* Do something with the argument. */
+        addReply(c,shared.ok); /* Reply something to the client. */
+    }
+
+The command is then referenced inside `server.c` in the command table:
+
+    {"foobar",foobarCommand,2,"rtF",0,NULL,0,0,0,0,0},
+
+In the above example `2` is the number of arguments the command takes,
+while `"rtF"` are the command flags, as documented in the command table
+top comment inside `server.c`.
+>>>>>>> redis/6.0
 
 Note: by contributing code to the KeyDB project in any form, including sending
 a pull request via Github, a code fragment or patch via private email or

TLS.md

@@ -0,0 +1,106 @@
+TLS Support -- Work In Progress
+===============================
+
+This is a brief note to capture current thoughts/ideas and track pending action
+items.
+
+Getting Started
+---------------
+
+### Building
+
+To build with TLS support you'll need OpenSSL development libraries (e.g.
+libssl-dev on Debian/Ubuntu).
+
+Run `make BUILD_TLS=yes`.
+
+### Tests
+
+To run Redis test suite with TLS, you'll need TLS support for TCL (i.e.
+`tcl-tls` package on Debian/Ubuntu).
+
+1. Run `./utils/gen-test-certs.sh` to generate a root CA and a server
+   certificate.
+
+2. Run `./runtest --tls` or `./runtest-cluster --tls` to run Redis and Redis
+   Cluster tests in TLS mode.
+
+### Running manually
+
+To manually run a Redis server with TLS mode (assuming `gen-test-certs.sh` was
+invoked so sample certificates/keys are available):
+
+    ./src/redis-server --tls-port 6379 --port 0 \
+        --tls-cert-file ./tests/tls/redis.crt \
+        --tls-key-file ./tests/tls/redis.key \
+        --tls-ca-cert-file ./tests/tls/ca.crt
+
+To connect to this Redis server with `redis-cli`:
+
+    ./src/redis-cli --tls \
+        --cert ./tests/tls/redis.crt \
+        --key ./tests/tls/redis.key \
+        --cacert ./tests/tls/ca.crt
+
+This will disable TCP and enable TLS on port 6379. It's also possible to have
+both TCP and TLS available, but you'll need to assign different ports.
+
+To make a Replica connect to the master using TLS, use `--tls-replication yes`,
+and to make Redis Cluster use TLS across nodes use `--tls-cluster yes`.
+
+Connections
+-----------
+
+All socket operations now go through a connection abstraction layer that hides
+I/O and read/write event handling from the caller.
+
+**Multi-threading I/O is not currently supported for TLS**, as a TLS connection
+needs to do its own manipulation of AE events which is not thread safe. The
+solution is probably to manage independent AE loops for I/O threads and longer
+term association of connections with threads. This may potentially improve
+overall performance as well.
+
+Sync IO for TLS is currently implemented in a hackish way, i.e. making the
+socket blocking and configuring socket-level timeout.  This means the timeout
+value may not be so accurate, and there would be a lot of syscall overhead.
+However I believe that getting rid of syncio completely in favor of pure async
+work is probably a better move than trying to fix that. For replication it would
+probably not be so hard. For cluster keys migration it might be more difficult,
+but there are probably other good reasons to improve that part anyway.
+
+To-Do List
+==========
+
+Additional TLS Features
+-----------------------
+
+1. Add metrics to INFO?
+2. Add session caching support. Check if/how it's handled by clients to assess
+   how useful/important it is.
+
+redis-benchmark
+---------------
+
+The current implementation is a mix of using hiredis for parsing and basic
+networking (establishing connections), but directly manipulating sockets for
+most actions.
+
+This will need to be cleaned up for proper TLS support. The best approach is
+probably to migrate to hiredis async mode.
+
+redis-cli
+---------
+
+1. Add support for TLS in --slave and --rdb modes.
+
+Others
+------
+
+Consider the implications of allowing TLS to be configured on a separate port,
+making Redis listening on multiple ports.
+
+This impacts many things, like
+1. Startup banner port notification
+2. Proctitle
+3. How slaves announce themselves
+4. Cluster bus port calculation

deps/Makefile

@@ -41,9 +41,13 @@ distclean:
 
 .PHONY: distclean
 
+ifeq ($(BUILD_TLS),yes)
+    HIREDIS_MAKE_FLAGS = USE_SSL=1
+endif
+
 hiredis: .make-prerequisites
 	@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
-	cd hiredis && $(MAKE) static
+	cd hiredis && $(MAKE) static $(HIREDIS_MAKE_FLAGS)
 
 .PHONY: hiredis
 

deps/hiredis/.gitignore

@@ -5,3 +5,4 @@
 /*.dylib
 /*.a
 /*.pc
+*.dSYM

deps/hiredis/.travis.yml

@@ -26,20 +26,72 @@ addons:
     - libc6-dev-i386
     - libc6-dbg:i386
     - gcc-multilib
+    - g++-multilib
     - valgrind
 
 env:
-    - CFLAGS="-Werror"
+  - BITS="32"
-    - PRE="valgrind --track-origins=yes --leak-check=full"
+  - BITS="64"
-    - TARGET="32bit" TARGET_VARS="32bit-vars" CFLAGS="-Werror"
+
-    - TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
+script:
+  - EXTRA_CMAKE_OPTS="-DENABLE_EXAMPLES:BOOL=ON -DHIREDIS_SSL:BOOL=ON";
+    if [ "$TRAVIS_OS_NAME" == "osx" ]; then
+      if [ "$BITS" == "32" ]; then
+        CFLAGS="-m32 -Werror";
+        CXXFLAGS="-m32 -Werror";
+        LDFLAGS="-m32";
+        EXTRA_CMAKE_OPTS=;
+      else
+        CFLAGS="-Werror";
+        CXXFLAGS="-Werror";
+      fi;
+    else
+      TEST_PREFIX="valgrind --track-origins=yes --leak-check=full";
+      if [ "$BITS" == "32" ]; then
+        CFLAGS="-m32 -Werror";
+        CXXFLAGS="-m32 -Werror";
+        LDFLAGS="-m32";
+        EXTRA_CMAKE_OPTS=;
+      else
+        CFLAGS="-Werror";
+        CXXFLAGS="-Werror";
+      fi;
+    fi;
+    export CFLAGS CXXFLAGS LDFLAGS TEST_PREFIX EXTRA_CMAKE_OPTS
+  - mkdir build/ && cd build/
+  - cmake .. ${EXTRA_CMAKE_OPTS}
+  - make VERBOSE=1
+  - ctest -V
 
 matrix:
-  exclude:
+  include:
-    - os: osx
+    # Windows MinGW cross compile on Linux
-      env: PRE="valgrind --track-origins=yes --leak-check=full"
+    - os: linux
+      dist: xenial
+      compiler: mingw
+      addons:
+        apt:
+          packages:
+            - ninja-build
+            - gcc-mingw-w64-x86-64
+            - g++-mingw-w64-x86-64
+      script:
+        - mkdir build && cd build
+        - CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_BUILD_WITH_INSTALL_RPATH=on
+        - ninja -v
 
-    - os: osx
+    # Windows MSVC 2017
-      env: TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
+    - os: windows
-
+      compiler: msvc
-script: make $TARGET CFLAGS="$CFLAGS" && make check PRE="$PRE" && make $TARGET_VARS hiredis-example
+      env:
+        - MATRIX_EVAL="CC=cl.exe && CXX=cl.exe"
+      before_install:
+        - eval "${MATRIX_EVAL}"
+      install:
+        - choco install ninja
+      script:
+        - mkdir build && cd build
+        - cmd.exe /C '"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" amd64 &&
+          cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release &&
+          ninja -v'
+        - ctest -V

deps/hiredis/CHANGELOG.md

@@ -12,6 +12,16 @@
   compare to other values, casting might be necessary or can be removed, if
   casting was applied before.
 
+### 0.x.x (unreleased)
+**BREAKING CHANGES**:
+
+* Change `redisReply.len` to `size_t`, as it denotes the the size of a string
+
+User code should compare this to `size_t` values as well.
+If it was used to compare to other values, casting might be necessary or can be removed, if casting was applied before.
+
+* `redisReplyObjectFunctions.createArray` now takes `size_t` for its length parameter.
+
 ### 0.14.0 (2018-09-25)
 
 * Make string2ll static to fix conflict with Redis (Tom Lee [c3188b])
@@ -50,8 +60,9 @@
 * Import latest upstream sds. This breaks applications that are linked against the old hiredis v0.13
 * Fix warnings, when compiled with -Wshadow
 * Make hiredis compile in Cygwin on Windows, now CI-tested
-
+* Bulk and multi-bulk lengths less than -1 or greater than `LLONG_MAX` are now
-**BREAKING CHANGES**:
+  protocol errors. This is consistent with the RESP specification. On 32-bit
+  platforms, the upper bound is lowered to `SIZE_MAX`.
 
 * Remove backwards compatibility macro's
 

deps/hiredis/CMakeLists.txt

@@ -0,0 +1,90 @@
+CMAKE_MINIMUM_REQUIRED(VERSION 3.4.0)
+INCLUDE(GNUInstallDirs)
+PROJECT(hiredis)
+
+OPTION(ENABLE_SSL "Build hiredis_ssl for SSL support" OFF)
+
+MACRO(getVersionBit name)
+  SET(VERSION_REGEX "^#define ${name} (.+)$")
+  FILE(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/hiredis.h"
+    VERSION_BIT REGEX ${VERSION_REGEX})
+  STRING(REGEX REPLACE ${VERSION_REGEX} "\\1" ${name} "${VERSION_BIT}")
+ENDMACRO(getVersionBit)
+
+getVersionBit(HIREDIS_MAJOR)
+getVersionBit(HIREDIS_MINOR)
+getVersionBit(HIREDIS_PATCH)
+getVersionBit(HIREDIS_SONAME)
+SET(VERSION "${HIREDIS_MAJOR}.${HIREDIS_MINOR}.${HIREDIS_PATCH}")
+MESSAGE("Detected version: ${VERSION}")
+
+PROJECT(hiredis VERSION "${VERSION}")
+
+SET(ENABLE_EXAMPLES OFF CACHE BOOL "Enable building hiredis examples")
+
+ADD_LIBRARY(hiredis SHARED
+    async.c
+    dict.c
+    hiredis.c
+    net.c
+    read.c
+    sds.c
+    sockcompat.c)
+
+SET_TARGET_PROPERTIES(hiredis
+    PROPERTIES
+    VERSION "${HIREDIS_SONAME}")
+IF(WIN32 OR MINGW)
+    TARGET_LINK_LIBRARIES(hiredis PRIVATE ws2_32)
+ENDIF()
+TARGET_INCLUDE_DIRECTORIES(hiredis PUBLIC .)
+
+CONFIGURE_FILE(hiredis.pc.in hiredis.pc @ONLY)
+
+INSTALL(TARGETS hiredis
+    DESTINATION "${CMAKE_INSTALL_LIBDIR}")
+
+INSTALL(FILES hiredis.h read.h sds.h async.h
+    DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
+    
+INSTALL(DIRECTORY adapters
+    DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
+    
+INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/hiredis.pc
+    DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
+
+IF(ENABLE_SSL)
+    IF (NOT OPENSSL_ROOT_DIR)
+        IF (APPLE)
+            SET(OPENSSL_ROOT_DIR "/usr/local/opt/openssl")
+        ENDIF()
+    ENDIF()
+    FIND_PACKAGE(OpenSSL REQUIRED)
+    ADD_LIBRARY(hiredis_ssl SHARED
+        ssl.c)
+    TARGET_INCLUDE_DIRECTORIES(hiredis_ssl PRIVATE "${OPENSSL_INCLUDE_DIR}")
+    TARGET_LINK_LIBRARIES(hiredis_ssl PRIVATE ${OPENSSL_LIBRARIES})
+    CONFIGURE_FILE(hiredis_ssl.pc.in hiredis_ssl.pc @ONLY)
+
+    INSTALL(TARGETS hiredis_ssl
+        DESTINATION "${CMAKE_INSTALL_LIBDIR}")
+
+    INSTALL(FILES hiredis_ssl.h
+        DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
+    
+    INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/hiredis_ssl.pc
+        DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
+ENDIF()
+
+IF(NOT (WIN32 OR MINGW))
+    ENABLE_TESTING()
+    ADD_EXECUTABLE(hiredis-test test.c)
+    TARGET_LINK_LIBRARIES(hiredis-test hiredis)
+    ADD_TEST(NAME hiredis-test
+        COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/test.sh)
+ENDIF()
+
+# Add examples
+IF(ENABLE_EXAMPLES)
+  ADD_SUBDIRECTORY(examples)
+ENDIF(ENABLE_EXAMPLES)

deps/hiredis/Makefile

@@ -3,11 +3,17 @@
 # Copyright (C) 2010-2011 Pieter Noordhuis <pcnoordhuis at gmail dot com>
 # This file is released under the BSD license, see the COPYING file
 
-OBJ=net.o hiredis.o sds.o async.o read.o
+OBJ=net.o hiredis.o sds.o async.o read.o sockcompat.o
+SSL_OBJ=ssl.o
 EXAMPLES=hiredis-example hiredis-example-libevent hiredis-example-libev hiredis-example-glib
+ifeq ($(USE_SSL),1)
+EXAMPLES+=hiredis-example-ssl hiredis-example-libevent-ssl
+endif
 TESTS=hiredis-test
 LIBNAME=libhiredis
+SSL_LIBNAME=libhiredis_ssl
 PKGCONFNAME=hiredis.pc
+SSL_PKGCONFNAME=hiredis_ssl.pc
 
 HIREDIS_MAJOR=$(shell grep HIREDIS_MAJOR hiredis.h | awk '{print $$3}')
 HIREDIS_MINOR=$(shell grep HIREDIS_MINOR hiredis.h | awk '{print $$3}')
@@ -39,7 +45,7 @@ export REDIS_TEST_CONFIG
 CC:=$(shell sh -c 'type $${CC%% *} >/dev/null 2>/dev/null && echo $(CC) || echo gcc')
 CXX:=$(shell sh -c 'type $${CXX%% *} >/dev/null 2>/dev/null && echo $(CXX) || echo g++')
 OPTIMIZATION?=-O3
-WARNINGS=-Wall -W -Wstrict-prototypes -Wwrite-strings
+WARNINGS=-Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers
 DEBUG_FLAGS?= -g -ggdb
 REAL_CFLAGS=$(OPTIMIZATION) -fPIC $(CPPFLAGS) $(CFLAGS) $(WARNINGS) $(DEBUG_FLAGS)
 REAL_LDFLAGS=$(LDFLAGS)
@@ -49,12 +55,30 @@ STLIBSUFFIX=a
 DYLIB_MINOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_SONAME)
 DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR)
 DYLIBNAME=$(LIBNAME).$(DYLIBSUFFIX)
-DYLIB_MAKE_CMD=$(CC) -shared -Wl,-soname,$(DYLIB_MINOR_NAME) -o $(DYLIBNAME) $(LDFLAGS)
+SSL_DYLIBNAME=$(SSL_LIBNAME).$(DYLIBSUFFIX)
+DYLIB_MAKE_CMD=$(CC) -shared -Wl,-soname,$(DYLIB_MINOR_NAME)
 STLIBNAME=$(LIBNAME).$(STLIBSUFFIX)
-STLIB_MAKE_CMD=ar rcs $(STLIBNAME)
+SSL_STLIBNAME=$(SSL_LIBNAME).$(STLIBSUFFIX)
+STLIB_MAKE_CMD=$(AR) rcs
 
 # Platform-specific overrides
 uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
+
+USE_SSL?=0
+
+# This is required for test.c only
+ifeq ($(USE_SSL),1)
+  CFLAGS+=-DHIREDIS_TEST_SSL
+endif
+
+ifeq ($(uname_S),Linux)
+  SSL_LDFLAGS=-lssl -lcrypto
+else
+  OPENSSL_PREFIX?=/usr/local/opt/openssl
+  CFLAGS+=-I$(OPENSSL_PREFIX)/include
+  SSL_LDFLAGS+=-L$(OPENSSL_PREFIX)/lib -lssl -lcrypto
+endif
+
 ifeq ($(uname_S),SunOS)
   REAL_LDFLAGS+= -ldl -lnsl -lsocket
   DYLIB_MAKE_CMD=$(CC) -G -o $(DYLIBNAME) -h $(DYLIB_MINOR_NAME) $(LDFLAGS)
@@ -66,40 +90,61 @@ ifeq ($(uname_S),Darwin)
 endif
 
 all: $(DYLIBNAME) $(STLIBNAME) hiredis-test $(PKGCONFNAME)
+ifeq ($(USE_SSL),1)
+all: $(SSL_DYLIBNAME) $(SSL_STLIBNAME) $(SSL_PKGCONFNAME)
+endif
 
 # Deps (use make dep to generate this)
 async.o: async.c fmacros.h async.h hiredis.h read.h sds.h net.h dict.c dict.h
 dict.o: dict.c fmacros.h dict.h
-hiredis.o: hiredis.c fmacros.h hiredis.h read.h sds.h net.h
+hiredis.o: hiredis.c fmacros.h hiredis.h read.h sds.h net.h win32.h
-net.o: net.c fmacros.h net.h hiredis.h read.h sds.h
+net.o: net.c fmacros.h net.h hiredis.h read.h sds.h sockcompat.h win32.h
 read.o: read.c fmacros.h read.h sds.h
 sds.o: sds.c sds.h
+sockcompat.o: sockcompat.c sockcompat.h
+ssl.o: ssl.c hiredis.h
 test.o: test.c fmacros.h hiredis.h read.h sds.h
 
 $(DYLIBNAME): $(OBJ)
-	$(DYLIB_MAKE_CMD) $(OBJ)
+	$(DYLIB_MAKE_CMD) -o $(DYLIBNAME) $(OBJ) $(REAL_LDFLAGS)
 
 $(STLIBNAME): $(OBJ)
-	$(STLIB_MAKE_CMD) $(OBJ)
+	$(STLIB_MAKE_CMD) $(STLIBNAME) $(OBJ)
+
+$(SSL_DYLIBNAME): $(SSL_OBJ)
+	$(DYLIB_MAKE_CMD) -o $(SSL_DYLIBNAME) $(SSL_OBJ) $(REAL_LDFLAGS) $(SSL_LDFLAGS)
+
+$(SSL_STLIBNAME): $(SSL_OBJ)
+	$(STLIB_MAKE_CMD) $(SSL_STLIBNAME) $(SSL_OBJ)
 
 dynamic: $(DYLIBNAME)
 static: $(STLIBNAME)
+ifeq ($(USE_SSL),1)
+dynamic: $(SSL_DYLIBNAME)
+static: $(SSL_STLIBNAME)
+endif
 
 # Binaries:
 hiredis-example-libevent: examples/example-libevent.c adapters/libevent.h $(STLIBNAME)
-	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -levent $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -levent $(STLIBNAME) $(REAL_LDFLAGS)
+
+hiredis-example-libevent-ssl: examples/example-libevent-ssl.c adapters/libevent.h $(STLIBNAME) $(SSL_STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -levent $(STLIBNAME) $(SSL_STLIBNAME) $(REAL_LDFLAGS) $(SSL_LDFLAGS)
 
 hiredis-example-libev: examples/example-libev.c adapters/libev.h $(STLIBNAME)
-	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -lev $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -lev $(STLIBNAME) $(REAL_LDFLAGS)
 
 hiredis-example-glib: examples/example-glib.c adapters/glib.h $(STLIBNAME)
-	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< $(shell pkg-config --cflags --libs glib-2.0) $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(shell pkg-config --cflags --libs glib-2.0) $(STLIBNAME) $(REAL_LDFLAGS)
 
 hiredis-example-ivykis: examples/example-ivykis.c adapters/ivykis.h $(STLIBNAME)
-	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -livykis $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -livykis $(STLIBNAME) $(REAL_LDFLAGS)
 
 hiredis-example-macosx: examples/example-macosx.c adapters/macosx.h $(STLIBNAME)
-	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -framework CoreFoundation $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -framework CoreFoundation $(STLIBNAME) $(REAL_LDFLAGS)
+
+hiredis-example-ssl: examples/example-ssl.c $(STLIBNAME) $(SSL_STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(SSL_STLIBNAME) $(REAL_LDFLAGS) $(SSL_LDFLAGS)
 
 ifndef AE_DIR
 hiredis-example-ae:
@@ -116,7 +161,7 @@ hiredis-example-libuv:
 	@false
 else
 hiredis-example-libuv: examples/example-libuv.c adapters/libuv.h $(STLIBNAME)
-	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(LIBUV_DIR)/include $< $(LIBUV_DIR)/.libs/libuv.a -lpthread -lrt $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) -I. -I$(LIBUV_DIR)/include $< $(LIBUV_DIR)/.libs/libuv.a -lpthread -lrt $(STLIBNAME) $(REAL_LDFLAGS)
 endif
 
 ifeq ($(and $(QT_MOC),$(QT_INCLUDE_DIR),$(QT_LIBRARY_DIR)),)
@@ -133,32 +178,33 @@ hiredis-example-qt: examples/example-qt.cpp adapters/qt.h $(STLIBNAME)
 endif
 
 hiredis-example: examples/example.c $(STLIBNAME)
-	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(REAL_LDFLAGS)
 
 examples: $(EXAMPLES)
 
-hiredis-test: test.o $(STLIBNAME)
+TEST_LIBS = $(STLIBNAME)
+ifeq ($(USE_SSL),1)
+    TEST_LIBS += $(SSL_STLIBNAME) -lssl -lcrypto -lpthread
+endif
+hiredis-test: test.o $(TEST_LIBS)
 
 hiredis-%: %.o $(STLIBNAME)
-	$(CC) $(REAL_CFLAGS) -o $@ $(REAL_LDFLAGS) $< $(STLIBNAME)
+	$(CC) $(REAL_CFLAGS) -o $@ $< $(TEST_LIBS) $(REAL_LDFLAGS)
 
 test: hiredis-test
 	./hiredis-test
 
 check: hiredis-test
-	@echo "$$REDIS_TEST_CONFIG" | $(REDIS_SERVER) -
+	TEST_SSL=$(USE_SSL) ./test.sh
-	$(PRE) ./hiredis-test -h 127.0.0.1 -p $(REDIS_PORT) -s /tmp/hiredis-test-redis.sock || \
-			( kill `cat /tmp/hiredis-test-redis.pid` && false )
-	kill `cat /tmp/hiredis-test-redis.pid`
 
 .c.o:
 	$(CC) -std=c99 -pedantic -c $(REAL_CFLAGS) $<
 
 clean:
-	rm -rf $(DYLIBNAME) $(STLIBNAME) $(TESTS) $(PKGCONFNAME) examples/hiredis-example* *.o *.gcda *.gcno *.gcov
+	rm -rf $(DYLIBNAME) $(STLIBNAME) $(SSL_DYLIBNAME) $(SSL_STLIBNAME) $(TESTS) $(PKGCONFNAME) examples/hiredis-example* *.o *.gcda *.gcno *.gcov
 
 dep:
-	$(CC) -MM *.c
+	$(CC) $(CPPFLAGS) $(CFLAGS) -MM *.c
 
 INSTALL?= cp -pPR
 
@@ -175,6 +221,20 @@ $(PKGCONFNAME): hiredis.h
 	@echo Libs: -L\$${libdir} -lhiredis >> $@
 	@echo Cflags: -I\$${includedir} -D_FILE_OFFSET_BITS=64 >> $@
 
+$(SSL_PKGCONFNAME): hiredis.h
+	@echo "Generating $@ for pkgconfig..."
+	@echo prefix=$(PREFIX) > $@
+	@echo exec_prefix=\$${prefix} >> $@
+	@echo libdir=$(PREFIX)/$(LIBRARY_PATH) >> $@
+	@echo includedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
+	@echo >> $@
+	@echo Name: hiredis_ssl >> $@
+	@echo Description: SSL Support for hiredis. >> $@
+	@echo Version: $(HIREDIS_MAJOR).$(HIREDIS_MINOR).$(HIREDIS_PATCH) >> $@
+	@echo Requires: hiredis >> $@
+	@echo Libs: -L\$${libdir} -lhiredis_ssl >> $@
+	@echo Libs.private: -lssl -lcrypto >> $@
+
 install: $(DYLIBNAME) $(STLIBNAME) $(PKGCONFNAME)
 	mkdir -p $(INSTALL_INCLUDE_PATH) $(INSTALL_INCLUDE_PATH)/adapters $(INSTALL_LIBRARY_PATH)
 	$(INSTALL) hiredis.h async.h read.h sds.h $(INSTALL_INCLUDE_PATH)

deps/hiredis/README.md

@@ -286,6 +286,7 @@ return `REDIS_ERR`. The function to set the disconnect callback has the followin
 ```c
 int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
 ```
+`ac->data` may be used to pass user data to this callback, the same can be done for redisConnectCallback.
 ### Sending commands and their callbacks
 
 In an asynchronous context, commands are automatically pipelined due to the nature of an event loop.

deps/hiredis/adapters/libevent.h

@@ -34,48 +34,113 @@
 #include "../hiredis.h"
 #include "../async.h"
 
+#define REDIS_LIBEVENT_DELETED 0x01
+#define REDIS_LIBEVENT_ENTERED 0x02
+
 typedef struct redisLibeventEvents {
     redisAsyncContext *context;
-    struct event *rev, *wev;
+    struct event *ev;
+    struct event_base *base;
+    struct timeval tv;
+    short flags;
+    short state;
 } redisLibeventEvents;
 
-static void redisLibeventReadEvent(int fd, short event, void *arg) {
+static void redisLibeventDestroy(redisLibeventEvents *e) {
-    ((void)fd); ((void)event);
+    free(e);
-    redisLibeventEvents *e = (redisLibeventEvents*)arg;
-    redisAsyncHandleRead(e->context);
 }
 
-static void redisLibeventWriteEvent(int fd, short event, void *arg) {
+static void redisLibeventHandler(int fd, short event, void *arg) {
-    ((void)fd); ((void)event);
+    ((void)fd);
     redisLibeventEvents *e = (redisLibeventEvents*)arg;
+    e->state |= REDIS_LIBEVENT_ENTERED;
+
+    #define CHECK_DELETED() if (e->state & REDIS_LIBEVENT_DELETED) {\
+        redisLibeventDestroy(e);\
+        return; \
+    }
+
+    if ((event & EV_TIMEOUT) && (e->state & REDIS_LIBEVENT_DELETED) == 0) {
+        redisAsyncHandleTimeout(e->context);
+        CHECK_DELETED();
+    }
+
+    if ((event & EV_READ) && e->context && (e->state & REDIS_LIBEVENT_DELETED) == 0) {
+        redisAsyncHandleRead(e->context);
+        CHECK_DELETED();
+    }
+
+    if ((event & EV_WRITE) && e->context && (e->state & REDIS_LIBEVENT_DELETED) == 0) {
         redisAsyncHandleWrite(e->context);
+        CHECK_DELETED();
+    }
+
+    e->state &= ~REDIS_LIBEVENT_ENTERED;
+    #undef CHECK_DELETED
+}
+
+static void redisLibeventUpdate(void *privdata, short flag, int isRemove) {
+    redisLibeventEvents *e = (redisLibeventEvents *)privdata;
+    const struct timeval *tv = e->tv.tv_sec || e->tv.tv_usec ? &e->tv : NULL;
+
+    if (isRemove) {
+        if ((e->flags & flag) == 0) {
+            return;
+        } else {
+            e->flags &= ~flag;
+        }
+    } else {
+        if (e->flags & flag) {
+            return;
+        } else {
+            e->flags |= flag;
+        }
+    }
+
+    event_del(e->ev);
+    event_assign(e->ev, e->base, e->context->c.fd, e->flags | EV_PERSIST,
+                 redisLibeventHandler, privdata);
+    event_add(e->ev, tv);
 }
 
 static void redisLibeventAddRead(void *privdata) {
-    redisLibeventEvents *e = (redisLibeventEvents*)privdata;
+    redisLibeventUpdate(privdata, EV_READ, 0);
-    event_add(e->rev,NULL);
 }
 
 static void redisLibeventDelRead(void *privdata) {
-    redisLibeventEvents *e = (redisLibeventEvents*)privdata;
+    redisLibeventUpdate(privdata, EV_READ, 1);
-    event_del(e->rev);
 }
 
 static void redisLibeventAddWrite(void *privdata) {
-    redisLibeventEvents *e = (redisLibeventEvents*)privdata;
+    redisLibeventUpdate(privdata, EV_WRITE, 0);
-    event_add(e->wev,NULL);
 }
 
 static void redisLibeventDelWrite(void *privdata) {
-    redisLibeventEvents *e = (redisLibeventEvents*)privdata;
+    redisLibeventUpdate(privdata, EV_WRITE, 1);
-    event_del(e->wev);
 }
 
 static void redisLibeventCleanup(void *privdata) {
     redisLibeventEvents *e = (redisLibeventEvents*)privdata;
-    event_free(e->rev);
+    if (!e) {
-    event_free(e->wev);
+        return;
-    free(e);
+    }
+    event_del(e->ev);
+    event_free(e->ev);
+    e->ev = NULL;
+
+    if (e->state & REDIS_LIBEVENT_ENTERED) {
+        e->state |= REDIS_LIBEVENT_DELETED;
+    } else {
+        redisLibeventDestroy(e);
+    }
+}
+
+static void redisLibeventSetTimeout(void *privdata, struct timeval tv) {
+    redisLibeventEvents *e = (redisLibeventEvents *)privdata;
+    short flags = e->flags;
+    e->flags = 0;
+    e->tv = tv;
+    redisLibeventUpdate(e, flags, 0);
 }
 
 static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
@@ -87,7 +152,7 @@ static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
         return REDIS_ERR;
 
     /* Create container for context and r/w events */
-    e = (redisLibeventEvents*)malloc(sizeof(*e));
+    e = (redisLibeventEvents*)calloc(1, sizeof(*e));
     e->context = ac;
 
     /* Register functions to start/stop listening for events */
@@ -96,13 +161,12 @@ static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
     ac->ev.addWrite = redisLibeventAddWrite;
     ac->ev.delWrite = redisLibeventDelWrite;
     ac->ev.cleanup = redisLibeventCleanup;
+    ac->ev.scheduleTimer = redisLibeventSetTimeout;
     ac->ev.data = e;
 
     /* Initialize and install read/write events */
-    e->rev = event_new(base, c->fd, EV_READ, redisLibeventReadEvent, e);
+    e->ev = event_new(base, c->fd, EV_READ | EV_WRITE, redisLibeventHandler, e);
-    e->wev = event_new(base, c->fd, EV_WRITE, redisLibeventWriteEvent, e);
+    e->base = base;
-    event_add(e->rev, NULL);
-    event_add(e->wev, NULL);
     return REDIS_OK;
 }
 #endif

deps/hiredis/appveyor.yml

@@ -5,8 +5,9 @@ environment:
           CC: gcc
         - CYG_BASH: C:\cygwin\bin\bash
           CC: gcc
-          TARGET: 32bit
+          CFLAGS: -m32
-          TARGET_VARS: 32bit-vars
+          CXXFLAGS: -m32
+          LDFLAGS: -m32
 
 clone_depth: 1
 
@@ -20,4 +21,4 @@ install:
 
 build_script:
     - 'echo building...'
-    - '%CYG_BASH% -lc "cd $APPVEYOR_BUILD_FOLDER; exec 0</dev/null; make LDFLAGS=$LDFLAGS CC=$CC $TARGET CFLAGS=$CFLAGS && make LDFLAGS=$LDFLAGS CC=$CC $TARGET_VARS hiredis-example"'
+    - '%CYG_BASH% -lc "cd $APPVEYOR_BUILD_FOLDER; exec 0</dev/null; mkdir build && cd build && cmake .. -G \"Unix Makefiles\" && make VERBOSE=1"'

deps/hiredis/async.c

@@ -32,7 +32,9 @@
 #include "fmacros.h"
 #include <stdlib.h>
 #include <string.h>
+#ifndef _MSC_VER
 #include <strings.h>
+#endif
 #include <assert.h>
 #include <ctype.h>
 #include <errno.h>
@@ -40,22 +42,9 @@
 #include "net.h"
 #include "dict.c"
 #include "sds.h"
+#include "win32.h"
 
-#define _EL_ADD_READ(ctx) do { \
+#include "async_private.h"
-        if ((ctx)->ev.addRead) (ctx)->ev.addRead((ctx)->ev.data); \
-    } while(0)
-#define _EL_DEL_READ(ctx) do { \
-        if ((ctx)->ev.delRead) (ctx)->ev.delRead((ctx)->ev.data); \
-    } while(0)
-#define _EL_ADD_WRITE(ctx) do { \
-        if ((ctx)->ev.addWrite) (ctx)->ev.addWrite((ctx)->ev.data); \
-    } while(0)
-#define _EL_DEL_WRITE(ctx) do { \
-        if ((ctx)->ev.delWrite) (ctx)->ev.delWrite((ctx)->ev.data); \
-    } while(0)
-#define _EL_CLEANUP(ctx) do { \
-        if ((ctx)->ev.cleanup) (ctx)->ev.cleanup((ctx)->ev.data); \
-    } while(0);
 
 /* Forward declaration of function in hiredis.c */
 int __redisAppendCommand(redisContext *c, const char *cmd, size_t len);
@@ -126,6 +115,7 @@ static redisAsyncContext *redisAsyncInitialize(redisContext *c) {
     ac->ev.addWrite = NULL;
     ac->ev.delWrite = NULL;
     ac->ev.cleanup = NULL;
+    ac->ev.scheduleTimer = NULL;
 
     ac->onConnect = NULL;
     ac->onDisconnect = NULL;
@@ -150,56 +140,52 @@ static void __redisAsyncCopyError(redisAsyncContext *ac) {
     ac->errstr = c->errstr;
 }
 
-redisAsyncContext *redisAsyncConnect(const char *ip, int port) {
+redisAsyncContext *redisAsyncConnectWithOptions(const redisOptions *options) {
+    redisOptions myOptions = *options;
     redisContext *c;
     redisAsyncContext *ac;
 
-    c = redisConnectNonBlock(ip,port);
+    myOptions.options |= REDIS_OPT_NONBLOCK;
-    if (c == NULL)
+    c = redisConnectWithOptions(&myOptions);
+    if (c == NULL) {
         return NULL;
-
+    }
     ac = redisAsyncInitialize(c);
     if (ac == NULL) {
         redisFree(c);
         return NULL;
     }
-
     __redisAsyncCopyError(ac);
     return ac;
 }
 
+redisAsyncContext *redisAsyncConnect(const char *ip, int port) {
+    redisOptions options = {0};
+    REDIS_OPTIONS_SET_TCP(&options, ip, port);
+    return redisAsyncConnectWithOptions(&options);
+}
+
 redisAsyncContext *redisAsyncConnectBind(const char *ip, int port,
                                          const char *source_addr) {
-    redisContext *c = redisConnectBindNonBlock(ip,port,source_addr);
+    redisOptions options = {0};
-    redisAsyncContext *ac = redisAsyncInitialize(c);
+    REDIS_OPTIONS_SET_TCP(&options, ip, port);
-    __redisAsyncCopyError(ac);
+    options.endpoint.tcp.source_addr = source_addr;
-    return ac;
+    return redisAsyncConnectWithOptions(&options);
 }
 
 redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
                                                   const char *source_addr) {
-    redisContext *c = redisConnectBindNonBlockWithReuse(ip,port,source_addr);
+    redisOptions options = {0};
-    redisAsyncContext *ac = redisAsyncInitialize(c);
+    REDIS_OPTIONS_SET_TCP(&options, ip, port);
-    __redisAsyncCopyError(ac);
+    options.options |= REDIS_OPT_REUSEADDR;
-    return ac;
+    options.endpoint.tcp.source_addr = source_addr;
+    return redisAsyncConnectWithOptions(&options);
 }
 
 redisAsyncContext *redisAsyncConnectUnix(const char *path) {
-    redisContext *c;
+    redisOptions options = {0};
-    redisAsyncContext *ac;
+    REDIS_OPTIONS_SET_UNIX(&options, path);
-
+    return redisAsyncConnectWithOptions(&options);
-    c = redisConnectUnixNonBlock(path);
-    if (c == NULL)
-        return NULL;
-
-    ac = redisAsyncInitialize(c);
-    if (ac == NULL) {
-        redisFree(c);
-        return NULL;
-    }
-
-    __redisAsyncCopyError(ac);
-    return ac;
 }
 
 int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn) {
@@ -328,7 +314,7 @@ void redisAsyncFree(redisAsyncContext *ac) {
 }
 
 /* Helper function to make the disconnect happen and clean up. */
-static void __redisAsyncDisconnect(redisAsyncContext *ac) {
+void __redisAsyncDisconnect(redisAsyncContext *ac) {
     redisContext *c = &(ac->c);
 
     /* Make sure error is accessible if there is any */
@@ -344,10 +330,16 @@ static void __redisAsyncDisconnect(redisAsyncContext *ac) {
         c->flags |= REDIS_DISCONNECTING;
     }
 
+    /* cleanup event library on disconnect.
+     * this is safe to call multiple times */
+    _EL_CLEANUP(ac);
+
     /* For non-clean disconnects, __redisAsyncFree() will execute pending
      * callbacks with a NULL-reply. */
+    if (!(c->flags & REDIS_NO_AUTO_FREE)) {
       __redisAsyncFree(ac);
     }
+}
 
 /* Tries to do a clean disconnect from Redis, meaning it stops new commands
  * from being issued, but tries to flush the output buffer and execute
@@ -358,6 +350,9 @@ static void __redisAsyncDisconnect(redisAsyncContext *ac) {
 void redisAsyncDisconnect(redisAsyncContext *ac) {
     redisContext *c = &(ac->c);
     c->flags |= REDIS_DISCONNECTING;
+
+    /** unset the auto-free flag here, because disconnect undoes this */
+    c->flags &= ~REDIS_NO_AUTO_FREE;
     if (!(c->flags & REDIS_IN_CALLBACK) && ac->replies.head == NULL)
         __redisAsyncDisconnect(ac);
 }
@@ -524,6 +519,18 @@ static int __redisAsyncHandleConnect(redisAsyncContext *ac) {
     }
 }
 
+void redisAsyncRead(redisAsyncContext *ac) {
+    redisContext *c = &(ac->c);
+
+    if (redisBufferRead(c) == REDIS_ERR) {
+        __redisAsyncDisconnect(ac);
+    } else {
+        /* Always re-schedule reads */
+        _EL_ADD_READ(ac);
+        redisProcessCallbacks(ac);
+    }
+}
+
 /* This function should be called when the socket is readable.
  * It processes all replies that can be read and executes their callbacks.
  */
@@ -539,28 +546,13 @@ void redisAsyncHandleRead(redisAsyncContext *ac) {
             return;
     }
 
-    if (redisBufferRead(c) == REDIS_ERR) {
+    c->funcs->async_read(ac);
-        __redisAsyncDisconnect(ac);
-    } else {
-        /* Always re-schedule reads */
-        _EL_ADD_READ(ac);
-        redisProcessCallbacks(ac);
-    }
 }
 
-void redisAsyncHandleWrite(redisAsyncContext *ac) {
+void redisAsyncWrite(redisAsyncContext *ac) {
     redisContext *c = &(ac->c);
     int done = 0;
 
-    if (!(c->flags & REDIS_CONNECTED)) {
-        /* Abort connect was not successful. */
-        if (__redisAsyncHandleConnect(ac) != REDIS_OK)
-            return;
-        /* Try again later when the context is still not connected. */
-        if (!(c->flags & REDIS_CONNECTED))
-            return;
-    }
-
     if (redisBufferWrite(c,&done) == REDIS_ERR) {
         __redisAsyncDisconnect(ac);
     } else {
@@ -575,6 +567,51 @@ void redisAsyncHandleWrite(redisAsyncContext *ac) {
     }
 }
 
+void redisAsyncHandleWrite(redisAsyncContext *ac) {
+    redisContext *c = &(ac->c);
+
+    if (!(c->flags & REDIS_CONNECTED)) {
+        /* Abort connect was not successful. */
+        if (__redisAsyncHandleConnect(ac) != REDIS_OK)
+            return;
+        /* Try again later when the context is still not connected. */
+        if (!(c->flags & REDIS_CONNECTED))
+            return;
+    }
+
+    c->funcs->async_write(ac);
+}
+
+void __redisSetError(redisContext *c, int type, const char *str);
+
+void redisAsyncHandleTimeout(redisAsyncContext *ac) {
+    redisContext *c = &(ac->c);
+    redisCallback cb;
+
+    if ((c->flags & REDIS_CONNECTED) && ac->replies.head == NULL) {
+        /* Nothing to do - just an idle timeout */
+        return;
+    }
+
+    if (!c->err) {
+        __redisSetError(c, REDIS_ERR_TIMEOUT, "Timeout");
+    }
+
+    if (!(c->flags & REDIS_CONNECTED) && ac->onConnect) {
+        ac->onConnect(ac, REDIS_ERR);
+    }
+
+    while (__redisShiftCallback(&ac->replies, &cb) == REDIS_OK) {
+        __redisRunCallback(ac, &cb, NULL);
+    }
+
+    /**
+     * TODO: Don't automatically sever the connection,
+     * rather, allow to ignore <x> responses before the queue is clear
+     */
+    __redisAsyncDisconnect(ac);
+}
+
 /* Sets a pointer to the first argument and its length starting at p. Returns
  * the number of bytes to skip to get to the following argument. */
 static const char *nextArgument(const char *start, const char **str, size_t *len) {
@@ -714,3 +751,16 @@ int redisAsyncFormattedCommand(redisAsyncContext *ac, redisCallbackFn *fn, void
     int status = __redisAsyncCommand(ac,fn,privdata,cmd,len);
     return status;
 }
+
+void redisAsyncSetTimeout(redisAsyncContext *ac, struct timeval tv) {
+    if (!ac->c.timeout) {
+        ac->c.timeout = calloc(1, sizeof(tv));
+    }
+
+    if (tv.tv_sec == ac->c.timeout->tv_sec &&
+        tv.tv_usec == ac->c.timeout->tv_usec) {
+        return;
+    }
+
+    *ac->c.timeout = tv;
+}

deps/hiredis/async.h

@@ -57,6 +57,7 @@ typedef struct redisCallbackList {
 /* Connection callback prototypes */
 typedef void (redisDisconnectCallback)(const struct redisAsyncContext*, int status);
 typedef void (redisConnectCallback)(const struct redisAsyncContext*, int status);
+typedef void(redisTimerCallback)(void *timer, void *privdata);
 
 /* Context for an async connection to Redis */
 typedef struct redisAsyncContext {
@@ -81,6 +82,7 @@ typedef struct redisAsyncContext {
         void (*addWrite)(void *privdata);
         void (*delWrite)(void *privdata);
         void (*cleanup)(void *privdata);
+        void (*scheduleTimer)(void *privdata, struct timeval tv);
     } ev;
 
     /* Called when either the connection is terminated due to an error or per
@@ -106,6 +108,7 @@ typedef struct redisAsyncContext {
 } redisAsyncContext;
 
 /* Functions that proxy to hiredis */
+redisAsyncContext *redisAsyncConnectWithOptions(const redisOptions *options);
 redisAsyncContext *redisAsyncConnect(const char *ip, int port);
 redisAsyncContext *redisAsyncConnectBind(const char *ip, int port, const char *source_addr);
 redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
@@ -113,12 +116,17 @@ redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
 redisAsyncContext *redisAsyncConnectUnix(const char *path);
 int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn);
 int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
+
+void redisAsyncSetTimeout(redisAsyncContext *ac, struct timeval tv);
 void redisAsyncDisconnect(redisAsyncContext *ac);
 void redisAsyncFree(redisAsyncContext *ac);
 
 /* Handle read/write events */
 void redisAsyncHandleRead(redisAsyncContext *ac);
 void redisAsyncHandleWrite(redisAsyncContext *ac);
+void redisAsyncHandleTimeout(redisAsyncContext *ac);
+void redisAsyncRead(redisAsyncContext *ac);
+void redisAsyncWrite(redisAsyncContext *ac);
 
 /* Command functions for an async context. Write the command to the
  * output buffer and register the provided callback. */

deps/hiredis/async_private.h

@@ -0,0 +1,72 @@
+/*
+ * Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __HIREDIS_ASYNC_PRIVATE_H
+#define __HIREDIS_ASYNC_PRIVATE_H
+
+#define _EL_ADD_READ(ctx)                                         \
+    do {                                                          \
+        refreshTimeout(ctx);                                      \
+        if ((ctx)->ev.addRead) (ctx)->ev.addRead((ctx)->ev.data); \
+    } while (0)
+#define _EL_DEL_READ(ctx) do { \
+        if ((ctx)->ev.delRead) (ctx)->ev.delRead((ctx)->ev.data); \
+    } while(0)
+#define _EL_ADD_WRITE(ctx)                                          \
+    do {                                                            \
+        refreshTimeout(ctx);                                        \
+        if ((ctx)->ev.addWrite) (ctx)->ev.addWrite((ctx)->ev.data); \
+    } while (0)
+#define _EL_DEL_WRITE(ctx) do { \
+        if ((ctx)->ev.delWrite) (ctx)->ev.delWrite((ctx)->ev.data); \
+    } while(0)
+#define _EL_CLEANUP(ctx) do { \
+        if ((ctx)->ev.cleanup) (ctx)->ev.cleanup((ctx)->ev.data); \
+        ctx->ev.cleanup = NULL; \
+    } while(0);
+
+static inline void refreshTimeout(redisAsyncContext *ctx) {
+    if (ctx->c.timeout && ctx->ev.scheduleTimer &&
+        (ctx->c.timeout->tv_sec || ctx->c.timeout->tv_usec)) {
+        ctx->ev.scheduleTimer(ctx->ev.data, *ctx->c.timeout);
+    // } else {
+    //     printf("Not scheduling timer.. (tmo=%p)\n", ctx->c.timeout);
+    //     if (ctx->c.timeout){
+    //         printf("tv_sec: %u. tv_usec: %u\n", ctx->c.timeout->tv_sec,
+    //                ctx->c.timeout->tv_usec);
+    //     }
+    }
+}
+
+void __redisAsyncDisconnect(redisAsyncContext *ac);
+void redisProcessCallbacks(redisAsyncContext *ac);
+
+#endif  /* __HIREDIS_ASYNC_PRIVATE_H */

deps/hiredis/examples/CMakeLists.txt

@@ -0,0 +1,46 @@
+INCLUDE(FindPkgConfig)
+# Check for GLib
+
+PKG_CHECK_MODULES(GLIB2 glib-2.0)
+if (GLIB2_FOUND)
+    INCLUDE_DIRECTORIES(${GLIB2_INCLUDE_DIRS})
+    LINK_DIRECTORIES(${GLIB2_LIBRARY_DIRS})
+    ADD_EXECUTABLE(example-glib example-glib.c)
+    TARGET_LINK_LIBRARIES(example-glib hiredis ${GLIB2_LIBRARIES})
+ENDIF(GLIB2_FOUND)
+
+FIND_PATH(LIBEV ev.h
+    HINTS /usr/local /usr/opt/local
+    ENV LIBEV_INCLUDE_DIR)
+
+if (LIBEV)
+    # Just compile and link with libev
+    ADD_EXECUTABLE(example-libev example-libev.c)
+    TARGET_LINK_LIBRARIES(example-libev hiredis ev)
+ENDIF()
+
+FIND_PATH(LIBEVENT event.h)
+if (LIBEVENT)
+    ADD_EXECUTABLE(example-libevent example-libevent)
+    TARGET_LINK_LIBRARIES(example-libevent hiredis event)
+ENDIF()
+
+FIND_PATH(LIBUV uv.h)
+IF (LIBUV)
+    ADD_EXECUTABLE(example-libuv example-libuv.c)
+    TARGET_LINK_LIBRARIES(example-libuv hiredis uv)
+ENDIF()
+
+IF (APPLE)
+    FIND_LIBRARY(CF CoreFoundation)
+    ADD_EXECUTABLE(example-macosx example-macosx.c)
+    TARGET_LINK_LIBRARIES(example-macosx hiredis ${CF})
+ENDIF()
+
+IF (ENABLE_SSL)
+    ADD_EXECUTABLE(example-ssl example-ssl.c)
+    TARGET_LINK_LIBRARIES(example-ssl hiredis hiredis_ssl)
+ENDIF()
+
+ADD_EXECUTABLE(example example.c)
+TARGET_LINK_LIBRARIES(example hiredis)

deps/hiredis/examples/example-libevent-ssl.c

@@ -0,0 +1,73 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <signal.h>
+
+#include <hiredis.h>
+#include <hiredis_ssl.h>
+#include <async.h>
+#include <adapters/libevent.h>
+
+void getCallback(redisAsyncContext *c, void *r, void *privdata) {
+    redisReply *reply = r;
+    if (reply == NULL) return;
+    printf("argv[%s]: %s\n", (char*)privdata, reply->str);
+
+    /* Disconnect after receiving the reply to GET */
+    redisAsyncDisconnect(c);
+}
+
+void connectCallback(const redisAsyncContext *c, int status) {
+    if (status != REDIS_OK) {
+        printf("Error: %s\n", c->errstr);
+        return;
+    }
+    printf("Connected...\n");
+}
+
+void disconnectCallback(const redisAsyncContext *c, int status) {
+    if (status != REDIS_OK) {
+        printf("Error: %s\n", c->errstr);
+        return;
+    }
+    printf("Disconnected...\n");
+}
+
+int main (int argc, char **argv) {
+    signal(SIGPIPE, SIG_IGN);
+    struct event_base *base = event_base_new();
+    if (argc < 5) {
+        fprintf(stderr,
+                "Usage: %s <key> <host> <port> <cert> <certKey> [ca]\n", argv[0]);
+        exit(1);
+    }
+
+    const char *value = argv[1];
+    size_t nvalue = strlen(value);
+
+    const char *hostname = argv[2];
+    int port = atoi(argv[3]);
+
+    const char *cert = argv[4];
+    const char *certKey = argv[5];
+    const char *caCert = argc > 5 ? argv[6] : NULL;
+
+    redisAsyncContext *c = redisAsyncConnect(hostname, port);
+    if (c->err) {
+        /* Let *c leak for now... */
+        printf("Error: %s\n", c->errstr);
+        return 1;
+    }
+    if (redisSecureConnection(&c->c, caCert, cert, certKey, "sni") != REDIS_OK) {
+        printf("SSL Error!\n");
+        exit(1);
+    }
+
+    redisLibeventAttach(c,base);
+    redisAsyncSetConnectCallback(c,connectCallback);
+    redisAsyncSetDisconnectCallback(c,disconnectCallback);
+    redisAsyncCommand(c, NULL, NULL, "SET key %b", value, nvalue);
+    redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
+    event_base_dispatch(base);
+    return 0;
+}

deps/hiredis/examples/example-libevent.c

@@ -9,7 +9,12 @@
 
 void getCallback(redisAsyncContext *c, void *r, void *privdata) {
     redisReply *reply = r;
-    if (reply == NULL) return;
+    if (reply == NULL) {
+        if (c->errstr) {
+            printf("errstr: %s\n", c->errstr);
+        }
+        return;
+    }
     printf("argv[%s]: %s\n", (char*)privdata, reply->str);
 
     /* Disconnect after receiving the reply to GET */
@@ -35,8 +40,14 @@ void disconnectCallback(const redisAsyncContext *c, int status) {
 int main (int argc, char **argv) {
     signal(SIGPIPE, SIG_IGN);
     struct event_base *base = event_base_new();
+    redisOptions options = {0};
+    REDIS_OPTIONS_SET_TCP(&options, "127.0.0.1", 6379);
+    struct timeval tv = {0};
+    tv.tv_sec = 1;
+    options.timeout = &tv;
 
-    redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
+
+    redisAsyncContext *c = redisAsyncConnectWithOptions(&options);
     if (c->err) {
         /* Let *c leak for now... */
         printf("Error: %s\n", c->errstr);

deps/hiredis/examples/example-ssl.c

@@ -0,0 +1,97 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <hiredis.h>
+#include <hiredis_ssl.h>
+
+int main(int argc, char **argv) {
+    unsigned int j;
+    redisContext *c;
+    redisReply *reply;
+    if (argc < 4) {
+        printf("Usage: %s <host> <port> <cert> <key> [ca]\n", argv[0]);
+        exit(1);
+    }
+    const char *hostname = (argc > 1) ? argv[1] : "127.0.0.1";
+    int port = atoi(argv[2]);
+    const char *cert = argv[3];
+    const char *key = argv[4];
+    const char *ca = argc > 4 ? argv[5] : NULL;
+
+    struct timeval tv = { 1, 500000 }; // 1.5 seconds
+    redisOptions options = {0};
+    REDIS_OPTIONS_SET_TCP(&options, hostname, port);
+    options.timeout = &tv;
+    c = redisConnectWithOptions(&options);
+
+    if (c == NULL || c->err) {
+        if (c) {
+            printf("Connection error: %s\n", c->errstr);
+            redisFree(c);
+        } else {
+            printf("Connection error: can't allocate redis context\n");
+        }
+        exit(1);
+    }
+
+    if (redisSecureConnection(c, ca, cert, key, "sni") != REDIS_OK) {
+        printf("Couldn't initialize SSL!\n");
+        printf("Error: %s\n", c->errstr);
+        redisFree(c);
+        exit(1);
+    }
+
+    /* PING server */
+    reply = redisCommand(c,"PING");
+    printf("PING: %s\n", reply->str);
+    freeReplyObject(reply);
+
+    /* Set a key */
+    reply = redisCommand(c,"SET %s %s", "foo", "hello world");
+    printf("SET: %s\n", reply->str);
+    freeReplyObject(reply);
+
+    /* Set a key using binary safe API */
+    reply = redisCommand(c,"SET %b %b", "bar", (size_t) 3, "hello", (size_t) 5);
+    printf("SET (binary API): %s\n", reply->str);
+    freeReplyObject(reply);
+
+    /* Try a GET and two INCR */
+    reply = redisCommand(c,"GET foo");
+    printf("GET foo: %s\n", reply->str);
+    freeReplyObject(reply);
+
+    reply = redisCommand(c,"INCR counter");
+    printf("INCR counter: %lld\n", reply->integer);
+    freeReplyObject(reply);
+    /* again ... */
+    reply = redisCommand(c,"INCR counter");
+    printf("INCR counter: %lld\n", reply->integer);
+    freeReplyObject(reply);
+
+    /* Create a list of numbers, from 0 to 9 */
+    reply = redisCommand(c,"DEL mylist");
+    freeReplyObject(reply);
+    for (j = 0; j < 10; j++) {
+        char buf[64];
+
+        snprintf(buf,64,"%u",j);
+        reply = redisCommand(c,"LPUSH mylist element-%s", buf);
+        freeReplyObject(reply);
+    }
+
+    /* Let's check what we have inside the list */
+    reply = redisCommand(c,"LRANGE mylist 0 -1");
+    if (reply->type == REDIS_REPLY_ARRAY) {
+        for (j = 0; j < reply->elements; j++) {
+            printf("%u) %s\n", j, reply->element[j]->str);
+        }
+    }
+    freeReplyObject(reply);
+
+    /* Disconnects and frees the context */
+    redisFree(c);
+
+    return 0;
+}

deps/hiredis/examples/example.c

@@ -5,14 +5,27 @@
 #include <hiredis.h>
 
 int main(int argc, char **argv) {
-    unsigned int j;
+    unsigned int j, isunix = 0;
     redisContext *c;
     redisReply *reply;
     const char *hostname = (argc > 1) ? argv[1] : "127.0.0.1";
+
+    if (argc > 2) {
+        if (*argv[2] == 'u' || *argv[2] == 'U') {
+            isunix = 1;
+            /* in this case, host is the path to the unix socket */
+            printf("Will connect to unix socket @%s\n", hostname);
+        }
+    }
+
     int port = (argc > 2) ? atoi(argv[2]) : 6379;
 
     struct timeval timeout = { 1, 500000 }; // 1.5 seconds
+    if (isunix) {
+        c = redisConnectUnixWithTimeout(hostname, timeout);
+    } else {
         c = redisConnectWithTimeout(hostname, port, timeout);
+    }
     if (c == NULL || c->err) {
         if (c) {
             printf("Connection error: %s\n", c->errstr);

deps/hiredis/hiredis.c

@@ -34,7 +34,6 @@
 #include "fmacros.h"
 #include <string.h>
 #include <stdlib.h>
-#include <unistd.h>
 #include <assert.h>
 #include <errno.h>
 #include <ctype.h>
@@ -42,10 +41,20 @@
 #include "hiredis.h"
 #include "net.h"
 #include "sds.h"
+#include "async.h"
+#include "win32.h"
+
+static redisContextFuncs redisContextDefaultFuncs = {
+    .free_privdata = NULL,
+    .async_read = redisAsyncRead,
+    .async_write = redisAsyncWrite,
+    .read = redisNetRead,
+    .write = redisNetWrite
+};
 
 static redisReply *createReplyObject(int type);
 static void *createStringObject(const redisReadTask *task, char *str, size_t len);
-static void *createArrayObject(const redisReadTask *task, int elements);
+static void *createArrayObject(const redisReadTask *task, size_t elements);
 static void *createIntegerObject(const redisReadTask *task, long long value);
 static void *createDoubleObject(const redisReadTask *task, double value, char *str, size_t len);
 static void *createNilObject(const redisReadTask *task);
@@ -112,21 +121,34 @@ static void *createStringObject(const redisReadTask *task, char *str, size_t len
     if (r == NULL)
         return NULL;
 
+    assert(task->type == REDIS_REPLY_ERROR  ||
+           task->type == REDIS_REPLY_STATUS ||
+           task->type == REDIS_REPLY_STRING ||
+           task->type == REDIS_REPLY_VERB);
+
+    /* Copy string value */
+    if (task->type == REDIS_REPLY_VERB) {
+        buf = malloc(len-4+1); /* Skip 4 bytes of verbatim type header. */
+        if (buf == NULL) {
+            freeReplyObject(r);
+            return NULL;
+        }
+        memcpy(r->vtype,str,3);
+        r->vtype[3] = '\0';
+        memcpy(buf,str+4,len-4);
+        buf[len-4] = '\0';
+        r->len = len-4;
+    } else {
         buf = malloc(len+1);
         if (buf == NULL) {
             freeReplyObject(r);
             return NULL;
         }
-
-    assert(task->type == REDIS_REPLY_ERROR  ||
-           task->type == REDIS_REPLY_STATUS ||
-           task->type == REDIS_REPLY_STRING);
-
-    /* Copy string value */
         memcpy(buf,str,len);
         buf[len] = '\0';
-    r->str = buf;
         r->len = len;
+    }
+    r->str = buf;
 
     if (task->parent) {
         parent = task->parent->obj;
@@ -138,7 +160,7 @@ static void *createStringObject(const redisReadTask *task, char *str, size_t len
     return r;
 }
 
-static void *createArrayObject(const redisReadTask *task, int elements) {
+static void *createArrayObject(const redisReadTask *task, size_t elements) {
     redisReply *r, *parent;
 
     r = createReplyObject(task->type);
@@ -649,29 +671,30 @@ redisReader *redisReaderCreate(void) {
     return redisReaderCreateWithFunctions(&defaultFunctions);
 }
 
-static redisContext *redisContextInit(void) {
+static redisContext *redisContextInit(const redisOptions *options) {
     redisContext *c;
 
-    c = calloc(1,sizeof(redisContext));
+    c = calloc(1, sizeof(*c));
     if (c == NULL)
         return NULL;
 
+    c->funcs = &redisContextDefaultFuncs;
     c->obuf = sdsempty();
     c->reader = redisReaderCreate();
+    c->fd = REDIS_INVALID_FD;
 
     if (c->obuf == NULL || c->reader == NULL) {
         redisFree(c);
         return NULL;
     }
-
+    (void)options; /* options are used in other functions */
     return c;
 }
 
 void redisFree(redisContext *c) {
     if (c == NULL)
         return;
-    if (c->fd > 0)
+    redisNetClose(c);
-        close(c->fd);
 
     sdsfree(c->obuf);
     redisReaderFree(c->reader);
@@ -680,12 +703,16 @@ void redisFree(redisContext *c) {
     free(c->unix_sock.path);
     free(c->timeout);
     free(c->saddr);
+    if (c->funcs->free_privdata) {
+        c->funcs->free_privdata(c->privdata);
+    }
+    memset(c, 0xff, sizeof(*c));
     free(c);
 }
 
-int redisFreeKeepFd(redisContext *c) {
+redisFD redisFreeKeepFd(redisContext *c) {
-    int fd = c->fd;
+    redisFD fd = c->fd;
-    c->fd = -1;
+    c->fd = REDIS_INVALID_FD;
     redisFree(c);
     return fd;
 }
@@ -694,10 +721,13 @@ int redisReconnect(redisContext *c) {
     c->err = 0;
     memset(c->errstr, '\0', strlen(c->errstr));
 
-    if (c->fd > 0) {
+    if (c->privdata && c->funcs->free_privdata) {
-        close(c->fd);
+        c->funcs->free_privdata(c->privdata);
+        c->privdata = NULL;
     }
 
+    redisNetClose(c);
+
     sdsfree(c->obuf);
     redisReaderFree(c->reader);
 
@@ -718,112 +748,107 @@ int redisReconnect(redisContext *c) {
     return REDIS_ERR;
 }
 
+redisContext *redisConnectWithOptions(const redisOptions *options) {
+    redisContext *c = redisContextInit(options);
+    if (c == NULL) {
+        return NULL;
+    }
+    if (!(options->options & REDIS_OPT_NONBLOCK)) {
+        c->flags |= REDIS_BLOCK;
+    }
+    if (options->options & REDIS_OPT_REUSEADDR) {
+        c->flags |= REDIS_REUSEADDR;
+    }
+    if (options->options & REDIS_OPT_NOAUTOFREE) {
+      c->flags |= REDIS_NO_AUTO_FREE;
+    }
+
+    if (options->type == REDIS_CONN_TCP) {
+        redisContextConnectBindTcp(c, options->endpoint.tcp.ip,
+                                   options->endpoint.tcp.port, options->timeout,
+                                   options->endpoint.tcp.source_addr);
+    } else if (options->type == REDIS_CONN_UNIX) {
+        redisContextConnectUnix(c, options->endpoint.unix_socket,
+                                options->timeout);
+    } else if (options->type == REDIS_CONN_USERFD) {
+        c->fd = options->endpoint.fd;
+        c->flags |= REDIS_CONNECTED;
+    } else {
+        // Unknown type - FIXME - FREE
+        return NULL;
+    }
+    if (options->timeout != NULL && (c->flags & REDIS_BLOCK) && c->fd != REDIS_INVALID_FD) {
+        redisContextSetTimeout(c, *options->timeout);
+    }
+    return c;
+}
+
 /* Connect to a Redis instance. On error the field error in the returned
  * context will be set to the return value of the error function.
  * When no set of reply functions is given, the default set will be used. */
 redisContext *redisConnect(const char *ip, int port) {
-    redisContext *c;
+    redisOptions options = {0};
-
+    REDIS_OPTIONS_SET_TCP(&options, ip, port);
-    c = redisContextInit();
+    return redisConnectWithOptions(&options);
-    if (c == NULL)
-        return NULL;
-
-    c->flags |= REDIS_BLOCK;
-    redisContextConnectTcp(c,ip,port,NULL);
-    return c;
 }
 
 redisContext *redisConnectWithTimeout(const char *ip, int port, const struct timeval tv) {
-    redisContext *c;
+    redisOptions options = {0};
-
+    REDIS_OPTIONS_SET_TCP(&options, ip, port);
-    c = redisContextInit();
+    options.timeout = &tv;
-    if (c == NULL)
+    return redisConnectWithOptions(&options);
-        return NULL;
-
-    c->flags |= REDIS_BLOCK;
-    redisContextConnectTcp(c,ip,port,&tv);
-    return c;
 }
 
 redisContext *redisConnectNonBlock(const char *ip, int port) {
-    redisContext *c;
+    redisOptions options = {0};
-
+    REDIS_OPTIONS_SET_TCP(&options, ip, port);
-    c = redisContextInit();
+    options.options |= REDIS_OPT_NONBLOCK;
-    if (c == NULL)
+    return redisConnectWithOptions(&options);
-        return NULL;
-
-    c->flags &= ~REDIS_BLOCK;
-    redisContextConnectTcp(c,ip,port,NULL);
-    return c;
 }
 
 redisContext *redisConnectBindNonBlock(const char *ip, int port,
                                        const char *source_addr) {
-    redisContext *c = redisContextInit();
+    redisOptions options = {0};
-    if (c == NULL)
+    REDIS_OPTIONS_SET_TCP(&options, ip, port);
-        return NULL;
+    options.endpoint.tcp.source_addr = source_addr;
-    c->flags &= ~REDIS_BLOCK;
+    options.options |= REDIS_OPT_NONBLOCK;
-    redisContextConnectBindTcp(c,ip,port,NULL,source_addr);
+    return redisConnectWithOptions(&options);
-    return c;
 }
 
 redisContext *redisConnectBindNonBlockWithReuse(const char *ip, int port,
                                                 const char *source_addr) {
-    redisContext *c = redisContextInit();
+    redisOptions options = {0};
-    if (c == NULL)
+    REDIS_OPTIONS_SET_TCP(&options, ip, port);
-        return NULL;
+    options.endpoint.tcp.source_addr = source_addr;
-    c->flags &= ~REDIS_BLOCK;
+    options.options |= REDIS_OPT_NONBLOCK|REDIS_OPT_REUSEADDR;
-    c->flags |= REDIS_REUSEADDR;
+    return redisConnectWithOptions(&options);
-    redisContextConnectBindTcp(c,ip,port,NULL,source_addr);
-    return c;
 }
 
 redisContext *redisConnectUnix(const char *path) {
-    redisContext *c;
+    redisOptions options = {0};
-
+    REDIS_OPTIONS_SET_UNIX(&options, path);
-    c = redisContextInit();
+    return redisConnectWithOptions(&options);
-    if (c == NULL)
-        return NULL;
-
-    c->flags |= REDIS_BLOCK;
-    redisContextConnectUnix(c,path,NULL);
-    return c;
 }
 
 redisContext *redisConnectUnixWithTimeout(const char *path, const struct timeval tv) {
-    redisContext *c;
+    redisOptions options = {0};
-
+    REDIS_OPTIONS_SET_UNIX(&options, path);
-    c = redisContextInit();
+    options.timeout = &tv;
-    if (c == NULL)
+    return redisConnectWithOptions(&options);
-        return NULL;
-
-    c->flags |= REDIS_BLOCK;
-    redisContextConnectUnix(c,path,&tv);
-    return c;
 }
 
 redisContext *redisConnectUnixNonBlock(const char *path) {
-    redisContext *c;
+    redisOptions options = {0};
-
+    REDIS_OPTIONS_SET_UNIX(&options, path);
-    c = redisContextInit();
+    options.options |= REDIS_OPT_NONBLOCK;
-    if (c == NULL)
+    return redisConnectWithOptions(&options);
-        return NULL;
-
-    c->flags &= ~REDIS_BLOCK;
-    redisContextConnectUnix(c,path,NULL);
-    return c;
 }
 
-redisContext *redisConnectFd(int fd) {
+redisContext *redisConnectFd(redisFD fd) {
-    redisContext *c;
+    redisOptions options = {0};
-
+    options.type = REDIS_CONN_USERFD;
-    c = redisContextInit();
+    options.endpoint.fd = fd;
-    if (c == NULL)
+    return redisConnectWithOptions(&options);
-        return NULL;
-
-    c->fd = fd;
-    c->flags |= REDIS_BLOCK | REDIS_CONNECTED;
-    return c;
 }
 
 /* Set read/write timeout on a blocking socket. */
@@ -853,22 +878,15 @@ int redisBufferRead(redisContext *c) {
     if (c->err)
         return REDIS_ERR;
 
-    nread = read(c->fd,buf,sizeof(buf));
+    nread = c->funcs->read(c, buf, sizeof(buf));
-    if (nread == -1) {
+    if (nread > 0) {
-        if ((errno == EAGAIN && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
-            /* Try again later */
-        } else {
-            __redisSetError(c,REDIS_ERR_IO,NULL);
-            return REDIS_ERR;
-        }
-    } else if (nread == 0) {
-        __redisSetError(c,REDIS_ERR_EOF,"Server closed the connection");
-        return REDIS_ERR;
-    } else {
         if (redisReaderFeed(c->reader, buf, nread) != REDIS_OK) {
             __redisSetError(c, c->reader->err, c->reader->errstr);
             return REDIS_ERR;
+        } else {
         }
+    } else if (nread < 0) {
+        return REDIS_ERR;
     }
     return REDIS_OK;
 }
@@ -883,21 +901,15 @@ int redisBufferRead(redisContext *c) {
  * c->errstr to hold the appropriate error string.
  */
 int redisBufferWrite(redisContext *c, int *done) {
-    int nwritten;
 
     /* Return early when the context has seen an error. */
     if (c->err)
         return REDIS_ERR;
 
     if (sdslen(c->obuf) > 0) {
-        nwritten = write(c->fd,c->obuf,sdslen(c->obuf));
+        int nwritten = c->funcs->write(c);
-        if (nwritten == -1) {
+        if (nwritten < 0) {
-            if ((errno == EAGAIN && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
-                /* Try again later */
-            } else {
-                __redisSetError(c,REDIS_ERR_IO,NULL);
             return REDIS_ERR;
-            }
         } else if (nwritten > 0) {
             if (nwritten == (signed)sdslen(c->obuf)) {
                 sdsfree(c->obuf);

deps/hiredis/hiredis.h

@@ -35,7 +35,11 @@
 #define __HIREDIS_H
 #include "read.h"
 #include <stdarg.h> /* for va_list */
+#ifndef _MSC_VER
 #include <sys/time.h> /* for struct timeval */
+#else
+struct timeval; /* forward declaration */
+#endif
 #include <stdint.h> /* uintXX_t, etc */
 #include "sds.h" /* for sds */
 
@@ -74,6 +78,12 @@
 /* Flag that is set when we should set SO_REUSEADDR before calling bind() */
 #define REDIS_REUSEADDR 0x80
 
+/**
+ * Flag that indicates the user does not want the context to
+ * be automatically freed upon error
+ */
+#define REDIS_NO_AUTO_FREE 0x200
+
 #define REDIS_KEEPALIVE_INTERVAL 15 /* seconds */
 
 /* number of times we retry to connect in the case of EADDRNOTAVAIL and
@@ -92,6 +102,8 @@ typedef struct redisReply {
     size_t len; /* Length of string */
     char *str; /* Used for REDIS_REPLY_ERROR, REDIS_REPLY_STRING
                   and REDIS_REPLY_DOUBLE (in additionl to dval). */
+    char vtype[4]; /* Used for REDIS_REPLY_VERB, contains the null
+                      terminated 3 character content type, such as "txt". */
     size_t elements; /* number of elements, for REDIS_REPLY_ARRAY */
     struct redisReply **element; /* elements vector for REDIS_REPLY_ARRAY */
 } redisReply;
@@ -111,14 +123,93 @@ void redisFreeSdsCommand(sds cmd);
 
 enum redisConnectionType {
     REDIS_CONN_TCP,
-    REDIS_CONN_UNIX
+    REDIS_CONN_UNIX,
+    REDIS_CONN_USERFD
 };
 
+struct redisSsl;
+
+#define REDIS_OPT_NONBLOCK 0x01
+#define REDIS_OPT_REUSEADDR 0x02
+
+/**
+ * Don't automatically free the async object on a connection failure,
+ * or other implicit conditions. Only free on an explicit call to disconnect() or free()
+ */
+#define REDIS_OPT_NOAUTOFREE 0x04
+
+/* In Unix systems a file descriptor is a regular signed int, with -1
+ * representing an invalid descriptor. In Windows it is a SOCKET
+ * (32- or 64-bit unsigned integer depending on the architecture), where
+ * all bits set (~0) is INVALID_SOCKET.  */
+#ifndef _WIN32
+typedef int redisFD;
+#define REDIS_INVALID_FD -1
+#else
+#ifdef _WIN64
+typedef unsigned long long redisFD; /* SOCKET = 64-bit UINT_PTR */
+#else
+typedef unsigned long redisFD;      /* SOCKET = 32-bit UINT_PTR */
+#endif
+#define REDIS_INVALID_FD ((redisFD)(~0)) /* INVALID_SOCKET */
+#endif
+
+typedef struct {
+    /*
+     * the type of connection to use. This also indicates which
+     * `endpoint` member field to use
+     */
+    int type;
+    /* bit field of REDIS_OPT_xxx */
+    int options;
+    /* timeout value. if NULL, no timeout is used */
+    const struct timeval *timeout;
+    union {
+        /** use this field for tcp/ip connections */
+        struct {
+            const char *source_addr;
+            const char *ip;
+            int port;
+        } tcp;
+        /** use this field for unix domain sockets */
+        const char *unix_socket;
+        /**
+         * use this field to have hiredis operate an already-open
+         * file descriptor */
+        redisFD fd;
+    } endpoint;
+} redisOptions;
+
+/**
+ * Helper macros to initialize options to their specified fields.
+ */
+#define REDIS_OPTIONS_SET_TCP(opts, ip_, port_) \
+    (opts)->type = REDIS_CONN_TCP; \
+    (opts)->endpoint.tcp.ip = ip_; \
+    (opts)->endpoint.tcp.port = port_;
+
+#define REDIS_OPTIONS_SET_UNIX(opts, path) \
+    (opts)->type = REDIS_CONN_UNIX;        \
+    (opts)->endpoint.unix_socket = path;
+
+struct redisAsyncContext;
+struct redisContext;
+
+typedef struct redisContextFuncs {
+    void (*free_privdata)(void *);
+    void (*async_read)(struct redisAsyncContext *);
+    void (*async_write)(struct redisAsyncContext *);
+    int (*read)(struct redisContext *, char *, size_t);
+    int (*write)(struct redisContext *);
+} redisContextFuncs;
+
 /* Context for a connection to Redis */
 typedef struct redisContext {
+    const redisContextFuncs *funcs;   /* Function table */
+
     int err; /* Error flags, 0 when there is no error */
     char errstr[128]; /* String representation of error when applicable */
-    int fd;
+    redisFD fd;
     int flags;
     char *obuf; /* Write buffer */
     redisReader *reader; /* Protocol reader */
@@ -139,8 +230,12 @@ typedef struct redisContext {
     /* For non-blocking connect */
     struct sockadr *saddr;
     size_t addrlen;
+
+    /* Additional private data for hiredis addons such as SSL */
+    void *privdata;
 } redisContext;
 
+redisContext *redisConnectWithOptions(const redisOptions *options);
 redisContext *redisConnect(const char *ip, int port);
 redisContext *redisConnectWithTimeout(const char *ip, int port, const struct timeval tv);
 redisContext *redisConnectNonBlock(const char *ip, int port);
@@ -151,7 +246,7 @@ redisContext *redisConnectBindNonBlockWithReuse(const char *ip, int port,
 redisContext *redisConnectUnix(const char *path);
 redisContext *redisConnectUnixWithTimeout(const char *path, const struct timeval tv);
 redisContext *redisConnectUnixNonBlock(const char *path);
-redisContext *redisConnectFd(int fd);
+redisContext *redisConnectFd(redisFD fd);
 
 /**
  * Reconnect the given context using the saved information.
@@ -167,7 +262,7 @@ int redisReconnect(redisContext *c);
 int redisSetTimeout(redisContext *c, const struct timeval tv);
 int redisEnableKeepAlive(redisContext *c);
 void redisFree(redisContext *c);
-int redisFreeKeepFd(redisContext *c);
+redisFD redisFreeKeepFd(redisContext *c);
 int redisBufferRead(redisContext *c);
 int redisBufferWrite(redisContext *c, int *done);
 

deps/hiredis/hiredis.pc.in

@@ -0,0 +1,11 @@
+prefix=@CMAKE_INSTALL_PREFIX@
+exec_prefix=${prefix}
+libdir=${exec_prefix}/lib
+includedir=${prefix}/include
+pkgincludedir=${includedir}/hiredis
+
+Name: hiredis
+Description: Minimalistic C client library for Redis.
+Version: @PROJECT_VERSION@
+Libs: -L${libdir} -lhiredis
+Cflags: -I${pkgincludedir} -D_FILE_OFFSET_BITS=64

deps/hiredis/hiredis_ssl.h

@@ -0,0 +1,53 @@
+
+/*
+ * Copyright (c) 2019, Redis Labs
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __HIREDIS_SSL_H
+#define __HIREDIS_SSL_H
+
+/* This is the underlying struct for SSL in ssl.h, which is not included to
+ * keep build dependencies short here.
+ */
+struct ssl_st;
+
+/**
+ * Secure the connection using SSL. This should be done before any command is
+ * executed on the connection.
+ */
+int redisSecureConnection(redisContext *c, const char *capath, const char *certpath,
+                          const char *keypath, const char *servername);
+
+/**
+ * Initiate SSL/TLS negotiation on a provided context.
+ */
+
+int redisInitiateSSL(redisContext *c, struct ssl_st *ssl);
+
+#endif  /* __HIREDIS_SSL_H */

deps/hiredis/hiredis_ssl.pc.in

@@ -0,0 +1,12 @@
+prefix=@CMAKE_INSTALL_PREFIX@
+exec_prefix=${prefix}
+libdir=${exec_prefix}/lib
+includedir=${prefix}/include
+pkgincludedir=${includedir}/hiredis
+
+Name: hiredis_ssl
+Description: SSL Support for hiredis.
+Version: @PROJECT_VERSION@
+Requires: hiredis
+Libs: -L${libdir} -lhiredis_ssl
+Libs.private: -lssl -lcrypto

deps/hiredis/net.c

@@ -34,36 +34,64 @@
 
 #include "fmacros.h"
 #include <sys/types.h>
-#include <sys/socket.h>
-#include <sys/select.h>
-#include <sys/un.h>
-#include <netinet/in.h>
-#include <netinet/tcp.h>
-#include <arpa/inet.h>
-#include <unistd.h>
 #include <fcntl.h>
 #include <string.h>
-#include <netdb.h>
 #include <errno.h>
 #include <stdarg.h>
 #include <stdio.h>
-#include <poll.h>
 #include <limits.h>
 #include <stdlib.h>
 
 #include "net.h"
 #include "sds.h"
+#include "sockcompat.h"
+#include "win32.h"
 
 /* Defined in hiredis.c */
 void __redisSetError(redisContext *c, int type, const char *str);
 
-static void redisContextCloseFd(redisContext *c) {
+void redisNetClose(redisContext *c) {
-    if (c && c->fd >= 0) {
+    if (c && c->fd != REDIS_INVALID_FD) {
         close(c->fd);
-        c->fd = -1;
+        c->fd = REDIS_INVALID_FD;
     }
 }
 
+int redisNetRead(redisContext *c, char *buf, size_t bufcap) {
+    int nread = recv(c->fd, buf, bufcap, 0);
+    if (nread == -1) {
+        if ((errno == EWOULDBLOCK && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
+            /* Try again later */
+            return 0;
+        } else if(errno == ETIMEDOUT && (c->flags & REDIS_BLOCK)) {
+            /* especially in windows */
+            __redisSetError(c, REDIS_ERR_TIMEOUT, "recv timeout");
+            return -1;
+        } else {
+            __redisSetError(c, REDIS_ERR_IO, NULL);
+            return -1;
+        }
+    } else if (nread == 0) {
+        __redisSetError(c, REDIS_ERR_EOF, "Server closed the connection");
+        return -1;
+    } else {
+        return nread;
+    }
+}
+
+int redisNetWrite(redisContext *c) {
+    int nwritten = send(c->fd, c->obuf, sdslen(c->obuf), 0);
+    if (nwritten < 0) {
+        if ((errno == EWOULDBLOCK && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
+            /* Try again later */
+        } else {
+            __redisSetError(c, REDIS_ERR_IO, NULL);
+            return -1;
+        }
+    }
+    return nwritten;
+}
+
 static void __redisSetErrorFromErrno(redisContext *c, int type, const char *prefix) {
     int errorno = errno;  /* snprintf() may change errno */
     char buf[128] = { 0 };
@@ -79,15 +107,15 @@ static int redisSetReuseAddr(redisContext *c) {
     int on = 1;
     if (setsockopt(c->fd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) == -1) {
         __redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
-        redisContextCloseFd(c);
+        redisNetClose(c);
         return REDIS_ERR;
     }
     return REDIS_OK;
 }
 
 static int redisCreateSocket(redisContext *c, int type) {
-    int s;
+    redisFD s;
-    if ((s = socket(type, SOCK_STREAM, 0)) == -1) {
+    if ((s = socket(type, SOCK_STREAM, 0)) == REDIS_INVALID_FD) {
         __redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
         return REDIS_ERR;
     }
@@ -101,6 +129,7 @@ static int redisCreateSocket(redisContext *c, int type) {
 }
 
 static int redisSetBlocking(redisContext *c, int blocking) {
+#ifndef _WIN32
     int flags;
 
     /* Set the socket nonblocking.
@@ -108,7 +137,7 @@ static int redisSetBlocking(redisContext *c, int blocking) {
      * interrupted by a signal. */
     if ((flags = fcntl(c->fd, F_GETFL)) == -1) {
         __redisSetErrorFromErrno(c,REDIS_ERR_IO,"fcntl(F_GETFL)");
-        redisContextCloseFd(c);
+        redisNetClose(c);
         return REDIS_ERR;
     }
 
@@ -119,15 +148,23 @@ static int redisSetBlocking(redisContext *c, int blocking) {
 
     if (fcntl(c->fd, F_SETFL, flags) == -1) {
         __redisSetErrorFromErrno(c,REDIS_ERR_IO,"fcntl(F_SETFL)");
-        redisContextCloseFd(c);
+        redisNetClose(c);
         return REDIS_ERR;
     }
+#else
+    u_long mode = blocking ? 0 : 1;
+    if (ioctl(c->fd, FIONBIO, &mode) == -1) {
+        __redisSetErrorFromErrno(c, REDIS_ERR_IO, "ioctl(FIONBIO)");
+        redisNetClose(c);
+        return REDIS_ERR;
+    }
+#endif /* _WIN32 */
     return REDIS_OK;
 }
 
 int redisKeepAlive(redisContext *c, int interval) {
     int val = 1;
-    int fd = c->fd;
+    redisFD fd = c->fd;
 
     if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &val, sizeof(val)) == -1){
         __redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
@@ -170,7 +207,7 @@ static int redisSetTcpNoDelay(redisContext *c) {
     int yes = 1;
     if (setsockopt(c->fd, IPPROTO_TCP, TCP_NODELAY, &yes, sizeof(yes)) == -1) {
         __redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(TCP_NODELAY)");
-        redisContextCloseFd(c);
+        redisNetClose(c);
         return REDIS_ERR;
     }
     return REDIS_OK;
@@ -212,12 +249,12 @@ static int redisContextWaitReady(redisContext *c, long msec) {
 
         if ((res = poll(wfd, 1, msec)) == -1) {
             __redisSetErrorFromErrno(c, REDIS_ERR_IO, "poll(2)");
-            redisContextCloseFd(c);
+            redisNetClose(c);
             return REDIS_ERR;
         } else if (res == 0) {
             errno = ETIMEDOUT;
             __redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
-            redisContextCloseFd(c);
+            redisNetClose(c);
             return REDIS_ERR;
         }
 
@@ -230,7 +267,7 @@ static int redisContextWaitReady(redisContext *c, long msec) {
     }
 
     __redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
-    redisContextCloseFd(c);
+    redisNetClose(c);
     return REDIS_ERR;
 }
 
@@ -277,11 +314,18 @@ int redisCheckSocketError(redisContext *c) {
 }
 
 int redisContextSetTimeout(redisContext *c, const struct timeval tv) {
-    if (setsockopt(c->fd,SOL_SOCKET,SO_RCVTIMEO,&tv,sizeof(tv)) == -1) {
+    const void *to_ptr = &tv;
+    size_t to_sz = sizeof(tv);
+#ifdef _WIN32
+    DWORD timeout_msec = tv.tv_sec * 1000 + tv.tv_usec / 1000;
+    to_ptr = &timeout_msec;
+    to_sz = sizeof(timeout_msec);
+#endif
+    if (setsockopt(c->fd,SOL_SOCKET,SO_RCVTIMEO,to_ptr,to_sz) == -1) {
         __redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(SO_RCVTIMEO)");
         return REDIS_ERR;
     }
-    if (setsockopt(c->fd,SOL_SOCKET,SO_SNDTIMEO,&tv,sizeof(tv)) == -1) {
+    if (setsockopt(c->fd,SOL_SOCKET,SO_SNDTIMEO,to_ptr,to_sz) == -1) {
         __redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(SO_SNDTIMEO)");
         return REDIS_ERR;
     }
@@ -291,7 +335,8 @@ int redisContextSetTimeout(redisContext *c, const struct timeval tv) {
 static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
                                    const struct timeval *timeout,
                                    const char *source_addr) {
-    int s, rv, n;
+    redisFD s;
+    int rv, n;
     char _port[6];  /* strlen("65535"); */
     struct addrinfo hints, *servinfo, *bservinfo, *p, *b;
     int blocking = (c->flags & REDIS_BLOCK);
@@ -360,7 +405,7 @@ static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
     }
     for (p = servinfo; p != NULL; p = p->ai_next) {
 addrretry:
-        if ((s = socket(p->ai_family,p->ai_socktype,p->ai_protocol)) == -1)
+        if ((s = socket(p->ai_family,p->ai_socktype,p->ai_protocol)) == REDIS_INVALID_FD)
             continue;
 
         c->fd = s;
@@ -401,16 +446,14 @@ addrretry:
         }
 
         /* For repeat connection */
-        if (c->saddr) {
         free(c->saddr);
-        }
         c->saddr = malloc(p->ai_addrlen);
         memcpy(c->saddr, p->ai_addr, p->ai_addrlen);
         c->addrlen = p->ai_addrlen;
 
         if (connect(s,p->ai_addr,p->ai_addrlen) == -1) {
             if (errno == EHOSTUNREACH) {
-                redisContextCloseFd(c);
+                redisNetClose(c);
                 continue;
             } else if (errno == EINPROGRESS) {
                 if (blocking) {
@@ -424,7 +467,7 @@ addrretry:
                 if (++reuses >= REDIS_CONNECT_RETRIES) {
                     goto error;
                 } else {
-                    redisContextCloseFd(c);
+                    redisNetClose(c);
                     goto addrretry;
                 }
             } else {
@@ -471,8 +514,9 @@ int redisContextConnectBindTcp(redisContext *c, const char *addr, int port,
 }
 
 int redisContextConnectUnix(redisContext *c, const char *path, const struct timeval *timeout) {
+#ifndef _WIN32
     int blocking = (c->flags & REDIS_BLOCK);
-    struct sockaddr_un sa;
+    struct sockaddr_un *sa;
     long timeout_msec = -1;
 
     if (redisCreateSocket(c,AF_UNIX) < 0)
@@ -499,9 +543,11 @@ int redisContextConnectUnix(redisContext *c, const char *path, const struct time
     if (redisContextTimeoutMsec(c,&timeout_msec) != REDIS_OK)
         return REDIS_ERR;
 
-    sa.sun_family = AF_UNIX;
+    sa = (struct sockaddr_un*)(c->saddr = malloc(sizeof(struct sockaddr_un)));
-    strncpy(sa.sun_path,path,sizeof(sa.sun_path)-1);
+    c->addrlen = sizeof(struct sockaddr_un);
-    if (connect(c->fd, (struct sockaddr*)&sa, sizeof(sa)) == -1) {
+    sa->sun_family = AF_UNIX;
+    strncpy(sa->sun_path, path, sizeof(sa->sun_path) - 1);
+    if (connect(c->fd, (struct sockaddr*)sa, sizeof(*sa)) == -1) {
         if (errno == EINPROGRESS && !blocking) {
             /* This is ok. */
         } else {
@@ -516,4 +562,10 @@ int redisContextConnectUnix(redisContext *c, const char *path, const struct time
 
     c->flags |= REDIS_CONNECTED;
     return REDIS_OK;
+#else
+    /* We currently do not support Unix sockets for Windows. */
+    /* TODO(m): https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ */
+    errno = EPROTONOSUPPORT;
+    return REDIS_ERR;
+#endif /* _WIN32 */
 }

deps/hiredis/net.h

@@ -37,6 +37,10 @@
 
 #include "hiredis.h"
 
+void redisNetClose(redisContext *c);
+int redisNetRead(redisContext *c, char *buf, size_t bufcap);
+int redisNetWrite(redisContext *c);
+
 int redisCheckSocketError(redisContext *c);
 int redisContextSetTimeout(redisContext *c, const struct timeval tv);
 int redisContextConnectTcp(redisContext *c, const char *addr, int port, const struct timeval *timeout);

deps/hiredis/read.c

@@ -31,10 +31,10 @@
 
 #include "fmacros.h"
 #include <string.h>
-#include <strings.h>
 #include <stdlib.h>
 #ifndef _MSC_VER
 #include <unistd.h>
+#include <strings.h>
 #endif
 #include <assert.h>
 #include <errno.h>
@@ -44,6 +44,7 @@
 
 #include "read.h"
 #include "sds.h"
+#include "win32.h"
 
 static void __redisReaderSetError(redisReader *r, int type, const char *str) {
     size_t len;
@@ -294,9 +295,9 @@ static int processLineItem(redisReader *r) {
                 buf[len] = '\0';
 
                 if (strcasecmp(buf,",inf") == 0) {
-                    d = 1.0/0.0; /* Positive infinite. */
+                    d = INFINITY; /* Positive infinite. */
                 } else if (strcasecmp(buf,",-inf") == 0) {
-                    d = -1.0/0.0; /* Nevative infinite. */
+                    d = -INFINITY; /* Nevative infinite. */
                 } else {
                     d = strtod((char*)buf,&eptr);
                     if (buf[0] == '\0' || eptr[0] != '\0' || isnan(d)) {
@@ -379,10 +380,18 @@ static int processBulkItem(redisReader *r) {
             /* Only continue when the buffer contains the entire bulk item. */
             bytelen += len+2; /* include \r\n */
             if (r->pos+bytelen <= r->len) {
+                if ((cur->type == REDIS_REPLY_VERB && len < 4) ||
+                    (cur->type == REDIS_REPLY_VERB && s[5] != ':'))
+                {
+                    __redisReaderSetError(r,REDIS_ERR_PROTOCOL,
+                            "Verbatim string 4 bytes of content type are "
+                            "missing or incorrectly encoded.");
+                    return REDIS_ERR;
+                }
                 if (r->fn && r->fn->createString)
                     obj = r->fn->createString(cur,s+2,len);
                 else
-                    obj = (void*)REDIS_REPLY_STRING;
+                    obj = (void*)(long)cur->type;
                 success = 1;
             }
         }
@@ -430,7 +439,7 @@ static int processAggregateItem(redisReader *r) {
 
         root = (r->ridx == 0);
 
-        if (elements < -1 || elements > INT_MAX) {
+        if (elements < -1 || (LLONG_MAX > SIZE_MAX && elements > SIZE_MAX)) {
             __redisReaderSetError(r,REDIS_ERR_PROTOCOL,
                     "Multi-bulk length out of range");
             return REDIS_ERR;
@@ -523,6 +532,9 @@ static int processItem(redisReader *r) {
             case '#':
                 cur->type = REDIS_REPLY_BOOL;
                 break;
+            case '=':
+                cur->type = REDIS_REPLY_VERB;
+                break;
             default:
                 __redisReaderSetErrorProtocolByte(r,*p);
                 return REDIS_ERR;
@@ -543,6 +555,7 @@ static int processItem(redisReader *r) {
     case REDIS_REPLY_BOOL:
         return processLineItem(r);
     case REDIS_REPLY_STRING:
+    case REDIS_REPLY_VERB:
         return processBulkItem(r);
     case REDIS_REPLY_ARRAY:
     case REDIS_REPLY_MAP:
@@ -657,8 +670,11 @@ int redisReaderGetReply(redisReader *r, void **reply) {
 
     /* Emit a reply when there is one. */
     if (r->ridx == -1) {
-        if (reply != NULL)
+        if (reply != NULL) {
             *reply = r->reply;
+        } else if (r->reply != NULL && r->fn && r->fn->freeObject) {
+            r->fn->freeObject(r->reply);
+        }
         r->reply = NULL;
     }
     return REDIS_OK;

deps/hiredis/read.h

@@ -45,6 +45,7 @@
 #define REDIS_ERR_EOF 3 /* End of file */
 #define REDIS_ERR_PROTOCOL 4 /* Protocol error */
 #define REDIS_ERR_OOM 5 /* Out of memory */
+#define REDIS_ERR_TIMEOUT 6 /* Timed out */
 #define REDIS_ERR_OTHER 2 /* Everything else... */
 
 #define REDIS_REPLY_STRING 1
@@ -55,12 +56,12 @@
 #define REDIS_REPLY_ERROR 6
 #define REDIS_REPLY_DOUBLE 7
 #define REDIS_REPLY_BOOL 8
-#define REDIS_REPLY_VERB 9
 #define REDIS_REPLY_MAP 9
 #define REDIS_REPLY_SET 10
 #define REDIS_REPLY_ATTR 11
 #define REDIS_REPLY_PUSH 12
 #define REDIS_REPLY_BIGNUM 13
+#define REDIS_REPLY_VERB 14
 
 #define REDIS_READER_MAX_BUF (1024*16)  /* Default max unused reader buffer. */
 
@@ -79,7 +80,7 @@ typedef struct redisReadTask {
 
 typedef struct redisReplyObjectFunctions {
     void *(*createString)(const redisReadTask*, char*, size_t);
-    void *(*createArray)(const redisReadTask*, int);
+    void *(*createArray)(const redisReadTask*, size_t);
     void *(*createInteger)(const redisReadTask*, long long);
     void *(*createDouble)(const redisReadTask*, double, char*, size_t);
     void *(*createNil)(const redisReadTask*);

deps/hiredis/sds.h

@@ -34,6 +34,9 @@
 #define __SDS_H
 
 #define SDS_MAX_PREALLOC (1024*1024)
+#ifdef _MSC_VER
+#define __attribute__(x)
+#endif
 
 #include <sys/types.h>
 #include <stdarg.h>
@@ -143,20 +146,20 @@ static inline void sdssetlen(sds s, size_t newlen) {
         case SDS_TYPE_5:
             {
                 unsigned char *fp = ((unsigned char*)s)-1;
-                *fp = SDS_TYPE_5 | (newlen << SDS_TYPE_BITS);
+                *fp = (unsigned char)(SDS_TYPE_5 | (newlen << SDS_TYPE_BITS));
             }
             break;
         case SDS_TYPE_8:
-            SDS_HDR(8,s)->len = newlen;
+            SDS_HDR(8,s)->len = (uint8_t)newlen;
             break;
         case SDS_TYPE_16:
-            SDS_HDR(16,s)->len = newlen;
+            SDS_HDR(16,s)->len = (uint16_t)newlen;
             break;
         case SDS_TYPE_32:
-            SDS_HDR(32,s)->len = newlen;
+            SDS_HDR(32,s)->len = (uint32_t)newlen;
             break;
         case SDS_TYPE_64:
-            SDS_HDR(64,s)->len = newlen;
+            SDS_HDR(64,s)->len = (uint64_t)newlen;
             break;
     }
 }
@@ -167,21 +170,21 @@ static inline void sdsinclen(sds s, size_t inc) {
         case SDS_TYPE_5:
             {
                 unsigned char *fp = ((unsigned char*)s)-1;
-                unsigned char newlen = SDS_TYPE_5_LEN(flags)+inc;
+                unsigned char newlen = SDS_TYPE_5_LEN(flags)+(unsigned char)inc;
                 *fp = SDS_TYPE_5 | (newlen << SDS_TYPE_BITS);
             }
             break;
         case SDS_TYPE_8:
-            SDS_HDR(8,s)->len += inc;
+            SDS_HDR(8,s)->len += (uint8_t)inc;
             break;
         case SDS_TYPE_16:
-            SDS_HDR(16,s)->len += inc;
+            SDS_HDR(16,s)->len += (uint16_t)inc;
             break;
         case SDS_TYPE_32:
-            SDS_HDR(32,s)->len += inc;
+            SDS_HDR(32,s)->len += (uint32_t)inc;
             break;
         case SDS_TYPE_64:
-            SDS_HDR(64,s)->len += inc;
+            SDS_HDR(64,s)->len += (uint64_t)inc;
             break;
     }
 }
@@ -211,16 +214,16 @@ static inline void sdssetalloc(sds s, size_t newlen) {
             /* Nothing to do, this type has no total allocation info. */
             break;
         case SDS_TYPE_8:
-            SDS_HDR(8,s)->alloc = newlen;
+            SDS_HDR(8,s)->alloc = (uint8_t)newlen;
             break;
         case SDS_TYPE_16:
-            SDS_HDR(16,s)->alloc = newlen;
+            SDS_HDR(16,s)->alloc = (uint16_t)newlen;
             break;
         case SDS_TYPE_32:
-            SDS_HDR(32,s)->alloc = newlen;
+            SDS_HDR(32,s)->alloc = (uint32_t)newlen;
             break;
         case SDS_TYPE_64:
-            SDS_HDR(64,s)->alloc = newlen;
+            SDS_HDR(64,s)->alloc = (uint64_t)newlen;
             break;
     }
 }

deps/hiredis/sockcompat.c

@@ -0,0 +1,248 @@
+/*
+ * Copyright (c) 2019, Marcus Geelnard <m at bitsnbites dot eu>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#define REDIS_SOCKCOMPAT_IMPLEMENTATION
+#include "sockcompat.h"
+
+#ifdef _WIN32
+static int _wsaErrorToErrno(int err) {
+    switch (err) {
+        case WSAEWOULDBLOCK:
+            return EWOULDBLOCK;
+        case WSAEINPROGRESS:
+            return EINPROGRESS;
+        case WSAEALREADY:
+            return EALREADY;
+        case WSAENOTSOCK:
+            return ENOTSOCK;
+        case WSAEDESTADDRREQ:
+            return EDESTADDRREQ;
+        case WSAEMSGSIZE:
+            return EMSGSIZE;
+        case WSAEPROTOTYPE:
+            return EPROTOTYPE;
+        case WSAENOPROTOOPT:
+            return ENOPROTOOPT;
+        case WSAEPROTONOSUPPORT:
+            return EPROTONOSUPPORT;
+        case WSAEOPNOTSUPP:
+            return EOPNOTSUPP;
+        case WSAEAFNOSUPPORT:
+            return EAFNOSUPPORT;
+        case WSAEADDRINUSE:
+            return EADDRINUSE;
+        case WSAEADDRNOTAVAIL:
+            return EADDRNOTAVAIL;
+        case WSAENETDOWN:
+            return ENETDOWN;
+        case WSAENETUNREACH:
+            return ENETUNREACH;
+        case WSAENETRESET:
+            return ENETRESET;
+        case WSAECONNABORTED:
+            return ECONNABORTED;
+        case WSAECONNRESET:
+            return ECONNRESET;
+        case WSAENOBUFS:
+            return ENOBUFS;
+        case WSAEISCONN:
+            return EISCONN;
+        case WSAENOTCONN:
+            return ENOTCONN;
+        case WSAETIMEDOUT:
+            return ETIMEDOUT;
+        case WSAECONNREFUSED:
+            return ECONNREFUSED;
+        case WSAELOOP:
+            return ELOOP;
+        case WSAENAMETOOLONG:
+            return ENAMETOOLONG;
+        case WSAEHOSTUNREACH:
+            return EHOSTUNREACH;
+        case WSAENOTEMPTY:
+            return ENOTEMPTY;
+        default:
+            /* We just return a generic I/O error if we could not find a relevant error. */
+            return EIO;
+    }
+}
+
+static void _updateErrno(int success) {
+    errno = success ? 0 : _wsaErrorToErrno(WSAGetLastError());
+}
+
+static int _initWinsock() {
+    static int s_initialized = 0;
+    if (!s_initialized) {
+        static WSADATA wsadata;
+        int err = WSAStartup(MAKEWORD(2,2), &wsadata);
+        if (err != 0) {
+            errno = _wsaErrorToErrno(err);
+            return 0;
+        }
+        s_initialized = 1;
+    }
+    return 1;
+}
+
+int win32_getaddrinfo(const char *node, const char *service, const struct addrinfo *hints, struct addrinfo **res) {
+    /* Note: This function is likely to be called before other functions, so run init here. */
+    if (!_initWinsock()) {
+        return EAI_FAIL;
+    }
+
+    switch (getaddrinfo(node, service, hints, res)) {
+        case 0:                     return 0;
+        case WSATRY_AGAIN:          return EAI_AGAIN;
+        case WSAEINVAL:             return EAI_BADFLAGS;
+        case WSAEAFNOSUPPORT:       return EAI_FAMILY;
+        case WSA_NOT_ENOUGH_MEMORY: return EAI_MEMORY;
+        case WSAHOST_NOT_FOUND:     return EAI_NONAME;
+        case WSATYPE_NOT_FOUND:     return EAI_SERVICE;
+        case WSAESOCKTNOSUPPORT:    return EAI_SOCKTYPE;
+        default:                    return EAI_FAIL;     /* Including WSANO_RECOVERY */
+    }
+}
+
+const char *win32_gai_strerror(int errcode) {
+    switch (errcode) {
+        case 0:            errcode = 0;                     break;
+        case EAI_AGAIN:    errcode = WSATRY_AGAIN;          break;
+        case EAI_BADFLAGS: errcode = WSAEINVAL;             break;
+        case EAI_FAMILY:   errcode = WSAEAFNOSUPPORT;       break;
+        case EAI_MEMORY:   errcode = WSA_NOT_ENOUGH_MEMORY; break;
+        case EAI_NONAME:   errcode = WSAHOST_NOT_FOUND;     break;
+        case EAI_SERVICE:  errcode = WSATYPE_NOT_FOUND;     break;
+        case EAI_SOCKTYPE: errcode = WSAESOCKTNOSUPPORT;    break;
+        default:           errcode = WSANO_RECOVERY;        break; /* Including EAI_FAIL */
+    }
+    return gai_strerror(errcode);
+}
+
+void win32_freeaddrinfo(struct addrinfo *res) {
+    freeaddrinfo(res);
+}
+
+SOCKET win32_socket(int domain, int type, int protocol) {
+    SOCKET s;
+
+    /* Note: This function is likely to be called before other functions, so run init here. */
+    if (!_initWinsock()) {
+        return INVALID_SOCKET;
+    }
+
+    _updateErrno((s = socket(domain, type, protocol)) != INVALID_SOCKET);
+    return s;
+}
+
+int win32_ioctl(SOCKET fd, unsigned long request, unsigned long *argp) {
+    int ret = ioctlsocket(fd, (long)request, argp);
+    _updateErrno(ret != SOCKET_ERROR);
+    return ret != SOCKET_ERROR ? ret : -1;
+}
+
+int win32_bind(SOCKET sockfd, const struct sockaddr *addr, socklen_t addrlen) {
+    int ret = bind(sockfd, addr, addrlen);
+    _updateErrno(ret != SOCKET_ERROR);
+    return ret != SOCKET_ERROR ? ret : -1;
+}
+
+int win32_connect(SOCKET sockfd, const struct sockaddr *addr, socklen_t addrlen) {
+    int ret = connect(sockfd, addr, addrlen);
+    _updateErrno(ret != SOCKET_ERROR);
+
+    /* For Winsock connect(), the WSAEWOULDBLOCK error means the same thing as
+     * EINPROGRESS for POSIX connect(), so we do that translation to keep POSIX
+     * logic consistent. */
+    if (errno == EWOULDBLOCK) {
+        errno = EINPROGRESS;
+    }
+
+    return ret != SOCKET_ERROR ? ret : -1;
+}
+
+int win32_getsockopt(SOCKET sockfd, int level, int optname, void *optval, socklen_t *optlen) {
+    int ret = 0;
+    if ((level == SOL_SOCKET) && ((optname == SO_RCVTIMEO) || (optname == SO_SNDTIMEO))) {
+        if (*optlen >= sizeof (struct timeval)) {
+            struct timeval *tv = optval;
+            DWORD timeout = 0;
+            socklen_t dwlen = 0;
+            ret = getsockopt(sockfd, level, optname, (char *)&timeout, &dwlen);
+            tv->tv_sec = timeout / 1000;
+            tv->tv_usec = (timeout * 1000) % 1000000;
+        } else {
+            ret = WSAEFAULT;
+        }
+        *optlen = sizeof (struct timeval);
+    } else {
+        ret = getsockopt(sockfd, level, optname, (char*)optval, optlen);
+    }
+    _updateErrno(ret != SOCKET_ERROR);
+    return ret != SOCKET_ERROR ? ret : -1;
+}
+
+int win32_setsockopt(SOCKET sockfd, int level, int optname, const void *optval, socklen_t optlen) {
+    int ret = 0;
+    if ((level == SOL_SOCKET) && ((optname == SO_RCVTIMEO) || (optname == SO_SNDTIMEO))) {
+        struct timeval *tv = optval;
+        DWORD timeout = tv->tv_sec * 1000 + tv->tv_usec / 1000;
+        ret = setsockopt(sockfd, level, optname, (const char*)&timeout, sizeof(DWORD));
+    } else {
+        ret = setsockopt(sockfd, level, optname, (const char*)optval, optlen);
+    }
+    _updateErrno(ret != SOCKET_ERROR);
+    return ret != SOCKET_ERROR ? ret : -1;
+}
+
+int win32_close(SOCKET fd) {
+    int ret = closesocket(fd);
+    _updateErrno(ret != SOCKET_ERROR);
+    return ret != SOCKET_ERROR ? ret : -1;
+}
+
+ssize_t win32_recv(SOCKET sockfd, void *buf, size_t len, int flags) {
+    int ret = recv(sockfd, (char*)buf, (int)len, flags);
+    _updateErrno(ret != SOCKET_ERROR);
+    return ret != SOCKET_ERROR ? ret : -1;
+}
+
+ssize_t win32_send(SOCKET sockfd, const void *buf, size_t len, int flags) {
+    int ret = send(sockfd, (const char*)buf, (int)len, flags);
+    _updateErrno(ret != SOCKET_ERROR);
+    return ret != SOCKET_ERROR ? ret : -1;
+}
+
+int win32_poll(struct pollfd *fds, nfds_t nfds, int timeout) {
+    int ret = WSAPoll(fds, nfds, timeout);
+    _updateErrno(ret != SOCKET_ERROR);
+    return ret != SOCKET_ERROR ? ret : -1;
+}
+#endif /* _WIN32 */

deps/hiredis/sockcompat.h

@@ -0,0 +1,91 @@
+/*
+ * Copyright (c) 2019, Marcus Geelnard <m at bitsnbites dot eu>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __SOCKCOMPAT_H
+#define __SOCKCOMPAT_H
+
+#ifndef _WIN32
+/* For POSIX systems we use the standard BSD socket API. */
+#include <unistd.h>
+#include <sys/socket.h>
+#include <sys/select.h>
+#include <sys/un.h>
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+#include <arpa/inet.h>
+#include <netdb.h>
+#include <poll.h>
+#else
+/* For Windows we use winsock. */
+#undef _WIN32_WINNT
+#define _WIN32_WINNT 0x0600 /* To get WSAPoll etc. */
+#include <winsock2.h>
+#include <ws2tcpip.h>
+#include <stddef.h>
+
+#ifdef _MSC_VER
+typedef signed long ssize_t;
+#endif
+
+/* Emulate the parts of the BSD socket API that we need (override the winsock signatures). */
+int win32_getaddrinfo(const char *node, const char *service, const struct addrinfo *hints, struct addrinfo **res);
+const char *win32_gai_strerror(int errcode);
+void win32_freeaddrinfo(struct addrinfo *res);
+SOCKET win32_socket(int domain, int type, int protocol);
+int win32_ioctl(SOCKET fd, unsigned long request, unsigned long *argp);
+int win32_bind(SOCKET sockfd, const struct sockaddr *addr, socklen_t addrlen);
+int win32_connect(SOCKET sockfd, const struct sockaddr *addr, socklen_t addrlen);
+int win32_getsockopt(SOCKET sockfd, int level, int optname, void *optval, socklen_t *optlen);
+int win32_setsockopt(SOCKET sockfd, int level, int optname, const void *optval, socklen_t optlen);
+int win32_close(SOCKET fd);
+ssize_t win32_recv(SOCKET sockfd, void *buf, size_t len, int flags);
+ssize_t win32_send(SOCKET sockfd, const void *buf, size_t len, int flags);
+typedef ULONG nfds_t;
+int win32_poll(struct pollfd *fds, nfds_t nfds, int timeout);
+
+#ifndef REDIS_SOCKCOMPAT_IMPLEMENTATION
+#define getaddrinfo(node, service, hints, res) win32_getaddrinfo(node, service, hints, res)
+#undef gai_strerror
+#define gai_strerror(errcode) win32_gai_strerror(errcode)
+#define freeaddrinfo(res) win32_freeaddrinfo(res)
+#define socket(domain, type, protocol) win32_socket(domain, type, protocol)
+#define ioctl(fd, request, argp) win32_ioctl(fd, request, argp)
+#define bind(sockfd, addr, addrlen) win32_bind(sockfd, addr, addrlen)
+#define connect(sockfd, addr, addrlen) win32_connect(sockfd, addr, addrlen)
+#define getsockopt(sockfd, level, optname, optval, optlen) win32_getsockopt(sockfd, level, optname, optval, optlen)
+#define setsockopt(sockfd, level, optname, optval, optlen) win32_setsockopt(sockfd, level, optname, optval, optlen)
+#define close(fd) win32_close(fd)
+#define recv(sockfd, buf, len, flags) win32_recv(sockfd, buf, len, flags)
+#define send(sockfd, buf, len, flags) win32_send(sockfd, buf, len, flags)
+#define poll(fds, nfds, timeout) win32_poll(fds, nfds, timeout)
+#endif /* REDIS_SOCKCOMPAT_IMPLEMENTATION */
+#endif /* _WIN32 */
+
+#endif /* __SOCKCOMPAT_H */

deps/hiredis/ssl.c

@@ -0,0 +1,448 @@
+/*
+ * Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2019, Redis Labs
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "hiredis.h"
+#include "async.h"
+
+#include <assert.h>
+#include <pthread.h>
+#include <errno.h>
+#include <string.h>
+
+#include <openssl/ssl.h>
+#include <openssl/err.h>
+
+#include "async_private.h"
+
+void __redisSetError(redisContext *c, int type, const char *str);
+
+/* The SSL context is attached to SSL/TLS connections as a privdata. */
+typedef struct redisSSLContext {
+    /**
+     * OpenSSL SSL_CTX; It is optional and will not be set when using
+     * user-supplied SSL.
+     */
+    SSL_CTX *ssl_ctx;
+
+    /**
+     * OpenSSL SSL object.
+     */
+    SSL *ssl;
+
+    /**
+     * SSL_write() requires to be called again with the same arguments it was
+     * previously called with in the event of an SSL_read/SSL_write situation
+     */
+    size_t lastLen;
+
+    /** Whether the SSL layer requires read (possibly before a write) */
+    int wantRead;
+
+    /**
+     * Whether a write was requested prior to a read. If set, the write()
+     * should resume whenever a read takes place, if possible
+     */
+    int pendingWrite;
+} redisSSLContext;
+
+/* Forward declaration */
+redisContextFuncs redisContextSSLFuncs;
+
+#ifdef HIREDIS_SSL_TRACE
+/**
+ * Callback used for debugging
+ */
+static void sslLogCallback(const SSL *ssl, int where, int ret) {
+    const char *retstr = "";
+    int should_log = 1;
+    /* Ignore low-level SSL stuff */
+
+    if (where & SSL_CB_ALERT) {
+        should_log = 1;
+    }
+    if (where == SSL_CB_HANDSHAKE_START || where == SSL_CB_HANDSHAKE_DONE) {
+        should_log = 1;
+    }
+    if ((where & SSL_CB_EXIT) && ret == 0) {
+        should_log = 1;
+    }
+
+    if (!should_log) {
+        return;
+    }
+
+    retstr = SSL_alert_type_string(ret);
+    printf("ST(0x%x). %s. R(0x%x)%s\n", where, SSL_state_string_long(ssl), ret, retstr);
+
+    if (where == SSL_CB_HANDSHAKE_DONE) {
+        printf("Using SSL version %s. Cipher=%s\n", SSL_get_version(ssl), SSL_get_cipher_name(ssl));
+    }
+}
+#endif
+
+/**
+ * OpenSSL global initialization and locking handling callbacks.
+ * Note that this is only required for OpenSSL < 1.1.0.
+ */
+
+#if OPENSSL_VERSION_NUMBER < 0x10100000L
+#define HIREDIS_USE_CRYPTO_LOCKS
+#endif
+
+#ifdef HIREDIS_USE_CRYPTO_LOCKS
+typedef pthread_mutex_t sslLockType;
+static void sslLockInit(sslLockType *l) {
+    pthread_mutex_init(l, NULL);
+}
+static void sslLockAcquire(sslLockType *l) {
+    pthread_mutex_lock(l);
+}
+static void sslLockRelease(sslLockType *l) {
+    pthread_mutex_unlock(l);
+}
+static pthread_mutex_t *ossl_locks;
+
+static void opensslDoLock(int mode, int lkid, const char *f, int line) {
+    sslLockType *l = ossl_locks + lkid;
+
+    if (mode & CRYPTO_LOCK) {
+        sslLockAcquire(l);
+    } else {
+        sslLockRelease(l);
+    }
+
+    (void)f;
+    (void)line;
+}
+
+static void initOpensslLocks(void) {
+    unsigned ii, nlocks;
+    if (CRYPTO_get_locking_callback() != NULL) {
+        /* Someone already set the callback before us. Don't destroy it! */
+        return;
+    }
+    nlocks = CRYPTO_num_locks();
+    ossl_locks = malloc(sizeof(*ossl_locks) * nlocks);
+    for (ii = 0; ii < nlocks; ii++) {
+        sslLockInit(ossl_locks + ii);
+    }
+    CRYPTO_set_locking_callback(opensslDoLock);
+}
+#endif /* HIREDIS_USE_CRYPTO_LOCKS */
+
+/**
+ * SSL Connection initialization.
+ */
+
+static int redisSSLConnect(redisContext *c, SSL_CTX *ssl_ctx, SSL *ssl) {
+    if (c->privdata) {
+        __redisSetError(c, REDIS_ERR_OTHER, "redisContext was already associated");
+        return REDIS_ERR;
+    }
+    c->privdata = calloc(1, sizeof(redisSSLContext));
+
+    c->funcs = &redisContextSSLFuncs;
+    redisSSLContext *rssl = c->privdata;
+
+    rssl->ssl_ctx = ssl_ctx;
+    rssl->ssl = ssl;
+
+    SSL_set_mode(rssl->ssl, SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER);
+    SSL_set_fd(rssl->ssl, c->fd);
+    SSL_set_connect_state(rssl->ssl);
+
+    ERR_clear_error();
+    int rv = SSL_connect(rssl->ssl);
+    if (rv == 1) {
+        return REDIS_OK;
+    }
+
+    rv = SSL_get_error(rssl->ssl, rv);
+    if (((c->flags & REDIS_BLOCK) == 0) &&
+        (rv == SSL_ERROR_WANT_READ || rv == SSL_ERROR_WANT_WRITE)) {
+        return REDIS_OK;
+    }
+
+    if (c->err == 0) {
+        char err[512];
+        if (rv == SSL_ERROR_SYSCALL)
+            snprintf(err,sizeof(err)-1,"SSL_connect failed: %s",strerror(errno));
+        else {
+            unsigned long e = ERR_peek_last_error();
+            snprintf(err,sizeof(err)-1,"SSL_connect failed: %s",
+                    ERR_reason_error_string(e));
+        }
+        __redisSetError(c, REDIS_ERR_IO, err);
+    }
+    return REDIS_ERR;
+}
+
+int redisInitiateSSL(redisContext *c, SSL *ssl) {
+    return redisSSLConnect(c, NULL, ssl);
+}
+
+int redisSecureConnection(redisContext *c, const char *capath,
+                          const char *certpath, const char *keypath, const char *servername) {
+
+    SSL_CTX *ssl_ctx = NULL;
+    SSL *ssl = NULL;
+
+    /* Initialize global OpenSSL stuff */
+    static int isInit = 0;
+    if (!isInit) {
+        isInit = 1;
+        SSL_library_init();
+#ifdef HIREDIS_USE_CRYPTO_LOCKS
+        initOpensslLocks();
+#endif
+    }
+
+    ssl_ctx = SSL_CTX_new(SSLv23_client_method());
+    if (!ssl_ctx) {
+        __redisSetError(c, REDIS_ERR_OTHER, "Failed to create SSL_CTX");
+        goto error;
+    }
+
+#ifdef HIREDIS_SSL_TRACE
+    SSL_CTX_set_info_callback(ssl_ctx, sslLogCallback);
+#endif
+    SSL_CTX_set_options(ssl_ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3);
+    SSL_CTX_set_verify(ssl_ctx, SSL_VERIFY_PEER, NULL);
+    if ((certpath != NULL && keypath == NULL) || (keypath != NULL && certpath == NULL)) {
+        __redisSetError(c, REDIS_ERR_OTHER, "certpath and keypath must be specified together");
+        goto error;
+    }
+
+    if (capath) {
+        if (!SSL_CTX_load_verify_locations(ssl_ctx, capath, NULL)) {
+            __redisSetError(c, REDIS_ERR_OTHER, "Invalid CA certificate");
+            goto error;
+        }
+    }
+    if (certpath) {
+        if (!SSL_CTX_use_certificate_chain_file(ssl_ctx, certpath)) {
+            __redisSetError(c, REDIS_ERR_OTHER, "Invalid client certificate");
+            goto error;
+        }
+        if (!SSL_CTX_use_PrivateKey_file(ssl_ctx, keypath, SSL_FILETYPE_PEM)) {
+            __redisSetError(c, REDIS_ERR_OTHER, "Invalid client key");
+            goto error;
+        }
+    }
+
+    ssl = SSL_new(ssl_ctx);
+    if (!ssl) {
+        __redisSetError(c, REDIS_ERR_OTHER, "Couldn't create new SSL instance");
+        goto error;
+    }
+    if (servername) {
+        if (!SSL_set_tlsext_host_name(ssl, servername)) {
+            __redisSetError(c, REDIS_ERR_OTHER, "Couldn't set server name indication");
+            goto error;
+        }
+    }
+
+    return redisSSLConnect(c, ssl_ctx, ssl);
+
+error:
+    if (ssl) SSL_free(ssl);
+    if (ssl_ctx) SSL_CTX_free(ssl_ctx);
+    return REDIS_ERR;
+}
+
+static int maybeCheckWant(redisSSLContext *rssl, int rv) {
+    /**
+     * If the error is WANT_READ or WANT_WRITE, the appropriate flags are set
+     * and true is returned. False is returned otherwise
+     */
+    if (rv == SSL_ERROR_WANT_READ) {
+        rssl->wantRead = 1;
+        return 1;
+    } else if (rv == SSL_ERROR_WANT_WRITE) {
+        rssl->pendingWrite = 1;
+        return 1;
+    } else {
+        return 0;
+    }
+}
+
+/**
+ * Implementation of redisContextFuncs for SSL connections.
+ */
+
+static void redisSSLFreeContext(void *privdata){
+    redisSSLContext *rsc = privdata;
+
+    if (!rsc) return;
+    if (rsc->ssl) {
+        SSL_free(rsc->ssl);
+        rsc->ssl = NULL;
+    }
+    if (rsc->ssl_ctx) {
+        SSL_CTX_free(rsc->ssl_ctx);
+        rsc->ssl_ctx = NULL;
+    }
+    free(rsc);
+}
+
+static int redisSSLRead(redisContext *c, char *buf, size_t bufcap) {
+    redisSSLContext *rssl = c->privdata;
+
+    int nread = SSL_read(rssl->ssl, buf, bufcap);
+    if (nread > 0) {
+        return nread;
+    } else if (nread == 0) {
+        __redisSetError(c, REDIS_ERR_EOF, "Server closed the connection");
+        return -1;
+    } else {
+        int err = SSL_get_error(rssl->ssl, nread);
+        if (c->flags & REDIS_BLOCK) {
+            /**
+             * In blocking mode, we should never end up in a situation where
+             * we get an error without it being an actual error, except
+             * in the case of EINTR, which can be spuriously received from
+             * debuggers or whatever.
+             */
+            if (errno == EINTR) {
+                return 0;
+            } else {
+                const char *msg = NULL;
+                if (errno == EAGAIN) {
+                    msg = "Resource temporarily unavailable";
+                }
+                __redisSetError(c, REDIS_ERR_IO, msg);
+                return -1;
+            }
+        }
+
+        /**
+         * We can very well get an EWOULDBLOCK/EAGAIN, however
+         */
+        if (maybeCheckWant(rssl, err)) {
+            return 0;
+        } else {
+            __redisSetError(c, REDIS_ERR_IO, NULL);
+            return -1;
+        }
+    }
+}
+
+static int redisSSLWrite(redisContext *c) {
+    redisSSLContext *rssl = c->privdata;
+
+    size_t len = rssl->lastLen ? rssl->lastLen : sdslen(c->obuf);
+    int rv = SSL_write(rssl->ssl, c->obuf, len);
+
+    if (rv > 0) {
+        rssl->lastLen = 0;
+    } else if (rv < 0) {
+        rssl->lastLen = len;
+
+        int err = SSL_get_error(rssl->ssl, rv);
+        if ((c->flags & REDIS_BLOCK) == 0 && maybeCheckWant(rssl, err)) {
+            return 0;
+        } else {
+            __redisSetError(c, REDIS_ERR_IO, NULL);
+            return -1;
+        }
+    }
+    return rv;
+}
+
+static void redisSSLAsyncRead(redisAsyncContext *ac) {
+    int rv;
+    redisSSLContext *rssl = ac->c.privdata;
+    redisContext *c = &ac->c;
+
+    rssl->wantRead = 0;
+
+    if (rssl->pendingWrite) {
+        int done;
+
+        /* This is probably just a write event */
+        rssl->pendingWrite = 0;
+        rv = redisBufferWrite(c, &done);
+        if (rv == REDIS_ERR) {
+            __redisAsyncDisconnect(ac);
+            return;
+        } else if (!done) {
+            _EL_ADD_WRITE(ac);
+        }
+    }
+
+    rv = redisBufferRead(c);
+    if (rv == REDIS_ERR) {
+        __redisAsyncDisconnect(ac);
+    } else {
+        _EL_ADD_READ(ac);
+        redisProcessCallbacks(ac);
+    }
+}
+
+static void redisSSLAsyncWrite(redisAsyncContext *ac) {
+    int rv, done = 0;
+    redisSSLContext *rssl = ac->c.privdata;
+    redisContext *c = &ac->c;
+
+    rssl->pendingWrite = 0;
+    rv = redisBufferWrite(c, &done);
+    if (rv == REDIS_ERR) {
+        __redisAsyncDisconnect(ac);
+        return;
+    }
+
+    if (!done) {
+        if (rssl->wantRead) {
+            /* Need to read-before-write */
+            rssl->pendingWrite = 1;
+            _EL_DEL_WRITE(ac);
+        } else {
+            /* No extra reads needed, just need to write more */
+            _EL_ADD_WRITE(ac);
+        }
+    } else {
+        /* Already done! */
+        _EL_DEL_WRITE(ac);
+    }
+
+    /* Always reschedule a read */
+    _EL_ADD_READ(ac);
+}
+
+redisContextFuncs redisContextSSLFuncs = {
+    .free_privdata = redisSSLFreeContext,
+    .async_read = redisSSLAsyncRead,
+    .async_write = redisSSLAsyncWrite,
+    .read = redisSSLRead,
+    .write = redisSSLWrite
+};
+

deps/hiredis/test.c

@@ -13,12 +13,16 @@
 #include <limits.h>
 
 #include "hiredis.h"
+#ifdef HIREDIS_TEST_SSL
+#include "hiredis_ssl.h"
+#endif
 #include "net.h"
 
 enum connection_type {
     CONN_TCP,
     CONN_UNIX,
-    CONN_FD
+    CONN_FD,
+    CONN_SSL
 };
 
 struct config {
@@ -33,6 +37,14 @@ struct config {
     struct {
         const char *path;
     } unix_sock;
+
+    struct {
+        const char *host;
+        int port;
+        const char *ca_cert;
+        const char *cert;
+        const char *key;
+    } ssl;
 };
 
 /* The following lines make up our testing "framework" :) */
@@ -93,11 +105,27 @@ static int disconnect(redisContext *c, int keep_fd) {
     return -1;
 }
 
+static void do_ssl_handshake(redisContext *c, struct config config) {
+#ifdef HIREDIS_TEST_SSL
+    redisSecureConnection(c, config.ssl.ca_cert, config.ssl.cert, config.ssl.key, NULL);
+    if (c->err) {
+        printf("SSL error: %s\n", c->errstr);
+        redisFree(c);
+        exit(1);
+    }
+#else
+    (void) c;
+    (void) config;
+#endif
+}
+
 static redisContext *do_connect(struct config config) {
     redisContext *c = NULL;
 
     if (config.type == CONN_TCP) {
         c = redisConnect(config.tcp.host, config.tcp.port);
+    } else if (config.type == CONN_SSL) {
+        c = redisConnect(config.ssl.host, config.ssl.port);
     } else if (config.type == CONN_UNIX) {
         c = redisConnectUnix(config.unix_sock.path);
     } else if (config.type == CONN_FD) {
@@ -121,9 +149,21 @@ static redisContext *do_connect(struct config config) {
         exit(1);
     }
 
+    if (config.type == CONN_SSL) {
+        do_ssl_handshake(c, config);
+    }
+
     return select_database(c);
 }
 
+static void do_reconnect(redisContext *c, struct config config) {
+    redisReconnect(c);
+
+    if (config.type == CONN_SSL) {
+        do_ssl_handshake(c, config);
+    }
+}
+
 static void test_format_commands(void) {
     char *cmd;
     int len;
@@ -360,7 +400,8 @@ static void test_reply_reader(void) {
     freeReplyObject(reply);
     redisReaderFree(reader);
 
-    test("Set error when array > INT_MAX: ");
+#if LLONG_MAX > SIZE_MAX
+    test("Set error when array > SIZE_MAX: ");
     reader = redisReaderCreate();
     redisReaderFeed(reader, "*9223372036854775807\r\n+asdf\r\n",29);
     ret = redisReaderGetReply(reader,&reply);
@@ -369,7 +410,6 @@ static void test_reply_reader(void) {
     freeReplyObject(reply);
     redisReaderFree(reader);
 
-#if LLONG_MAX > SIZE_MAX
     test("Set error when bulk > SIZE_MAX: ");
     reader = redisReaderCreate();
     redisReaderFeed(reader, "$9223372036854775807\r\nasdf\r\n",28);
@@ -434,22 +474,23 @@ static void test_free_null(void) {
     test_cond(reply == NULL);
 }
 
+#define HIREDIS_BAD_DOMAIN "idontexist-noreally.com"
 static void test_blocking_connection_errors(void) {
     redisContext *c;
     struct addrinfo hints = {.ai_family = AF_INET};
     struct addrinfo *ai_tmp = NULL;
-    const char *bad_domain = "idontexist.com";
 
-    int rv = getaddrinfo(bad_domain, "6379", &hints, &ai_tmp);
+    int rv = getaddrinfo(HIREDIS_BAD_DOMAIN, "6379", &hints, &ai_tmp);
     if (rv != 0) {
         // Address does *not* exist
         test("Returns error when host cannot be resolved: ");
         // First see if this domain name *actually* resolves to NXDOMAIN
-        c = redisConnect("dontexist.com", 6379);
+        c = redisConnect(HIREDIS_BAD_DOMAIN, 6379);
         test_cond(
             c->err == REDIS_ERR_OTHER &&
             (strcmp(c->errstr, "Name or service not known") == 0 ||
-             strcmp(c->errstr, "Can't resolve: sadkfjaskfjsa.com") == 0 ||
+             strcmp(c->errstr, "Can't resolve: " HIREDIS_BAD_DOMAIN) == 0 ||
+             strcmp(c->errstr, "Name does not resolve") == 0 ||
              strcmp(c->errstr,
                     "nodename nor servname provided, or not known") == 0 ||
              strcmp(c->errstr, "No address associated with hostname") == 0 ||
@@ -574,7 +615,8 @@ static void test_blocking_connection_timeouts(struct config config) {
 
     c = do_connect(config);
     test("Does not return a reply when the command times out: ");
-    s = write(c->fd, cmd, strlen(cmd));
+    redisAppendFormattedCommand(c, cmd, strlen(cmd));
+    s = c->funcs->write(c);
     tv.tv_sec = 0;
     tv.tv_usec = 10000;
     redisSetTimeout(c, tv);
@@ -583,7 +625,7 @@ static void test_blocking_connection_timeouts(struct config config) {
     freeReplyObject(reply);
 
     test("Reconnect properly reconnects after a timeout: ");
-    redisReconnect(c);
+    do_reconnect(c, config);
     reply = redisCommand(c, "PING");
     test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
     freeReplyObject(reply);
@@ -591,7 +633,7 @@ static void test_blocking_connection_timeouts(struct config config) {
     test("Reconnect properly uses owned parameters: ");
     config.tcp.host = "foo";
     config.unix_sock.path = "foo";
-    redisReconnect(c);
+    do_reconnect(c, config);
     reply = redisCommand(c, "PING");
     test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
     freeReplyObject(reply);
@@ -894,6 +936,23 @@ int main(int argc, char **argv) {
             throughput = 0;
         } else if (argc >= 1 && !strcmp(argv[0],"--skip-inherit-fd")) {
             test_inherit_fd = 0;
+#ifdef HIREDIS_TEST_SSL
+        } else if (argc >= 2 && !strcmp(argv[0],"--ssl-port")) {
+            argv++; argc--;
+            cfg.ssl.port = atoi(argv[0]);
+        } else if (argc >= 2 && !strcmp(argv[0],"--ssl-host")) {
+            argv++; argc--;
+            cfg.ssl.host = argv[0];
+        } else if (argc >= 2 && !strcmp(argv[0],"--ssl-ca-cert")) {
+            argv++; argc--;
+            cfg.ssl.ca_cert  = argv[0];
+        } else if (argc >= 2 && !strcmp(argv[0],"--ssl-cert")) {
+            argv++; argc--;
+            cfg.ssl.cert = argv[0];
+        } else if (argc >= 2 && !strcmp(argv[0],"--ssl-key")) {
+            argv++; argc--;
+            cfg.ssl.key = argv[0];
+#endif
         } else {
             fprintf(stderr, "Invalid argument: %s\n", argv[0]);
             exit(1);
@@ -922,6 +981,20 @@ int main(int argc, char **argv) {
     test_blocking_io_errors(cfg);
     if (throughput) test_throughput(cfg);
 
+#ifdef HIREDIS_TEST_SSL
+    if (cfg.ssl.port && cfg.ssl.host) {
+        printf("\nTesting against SSL connection (%s:%d):\n", cfg.ssl.host, cfg.ssl.port);
+        cfg.type = CONN_SSL;
+
+        test_blocking_connection(cfg);
+        test_blocking_connection_timeouts(cfg);
+        test_blocking_io_errors(cfg);
+        test_invalid_timeout_errors(cfg);
+        test_append_formatted_commands(cfg);
+        if (throughput) test_throughput(cfg);
+    }
+#endif
+
     if (test_inherit_fd) {
         printf("\nTesting against inherited fd (%s):\n", cfg.unix_sock.path);
         cfg.type = CONN_FD;

deps/hiredis/test.sh

@@ -0,0 +1,70 @@
+#!/bin/sh -ue
+
+REDIS_SERVER=${REDIS_SERVER:-redis-server}
+REDIS_PORT=${REDIS_PORT:-56379}
+REDIS_SSL_PORT=${REDIS_SSL_PORT:-56443}
+TEST_SSL=${TEST_SSL:-0}
+SSL_TEST_ARGS=
+
+tmpdir=$(mktemp -d)
+PID_FILE=${tmpdir}/hiredis-test-redis.pid
+SOCK_FILE=${tmpdir}/hiredis-test-redis.sock
+
+if [ "$TEST_SSL" = "1" ]; then
+    SSL_CA_CERT=${tmpdir}/ca.crt
+    SSL_CA_KEY=${tmpdir}/ca.key
+    SSL_CERT=${tmpdir}/redis.crt
+    SSL_KEY=${tmpdir}/redis.key
+
+    openssl genrsa -out ${tmpdir}/ca.key 4096
+    openssl req \
+        -x509 -new -nodes -sha256 \
+        -key ${SSL_CA_KEY} \
+        -days 3650 \
+        -subj '/CN=Hiredis Test CA' \
+        -out ${SSL_CA_CERT}
+    openssl genrsa -out ${SSL_KEY} 2048
+    openssl req \
+        -new -sha256 \
+        -key ${SSL_KEY} \
+        -subj '/CN=Hiredis Test Cert' | \
+        openssl x509 \
+            -req -sha256 \
+            -CA ${SSL_CA_CERT} \
+            -CAkey ${SSL_CA_KEY} \
+            -CAserial ${tmpdir}/ca.txt \
+            -CAcreateserial \
+            -days 365 \
+            -out ${SSL_CERT}
+
+    SSL_TEST_ARGS="--ssl-host 127.0.0.1 --ssl-port ${REDIS_SSL_PORT} --ssl-ca-cert ${SSL_CA_CERT} --ssl-cert ${SSL_CERT} --ssl-key ${SSL_KEY}"
+fi
+
+cleanup() {
+  set +e
+  kill $(cat ${PID_FILE})
+  rm -rf ${tmpdir}
+}
+trap cleanup INT TERM EXIT
+
+cat > ${tmpdir}/redis.conf <<EOF
+daemonize yes
+pidfile ${PID_FILE}
+port ${REDIS_PORT}
+bind 127.0.0.1
+unixsocket ${SOCK_FILE}
+EOF
+
+if [ "$TEST_SSL" = "1" ]; then
+    cat >> ${tmpdir}/redis.conf <<EOF
+tls-port ${REDIS_SSL_PORT}
+tls-ca-cert-file ${SSL_CA_CERT}
+tls-cert-file ${SSL_CERT}
+tls-key-file ${SSL_KEY}
+EOF
+fi
+
+cat ${tmpdir}/redis.conf
+${REDIS_SERVER} ${tmpdir}/redis.conf
+
+${TEST_PREFIX:-} ./hiredis-test -h 127.0.0.1 -p ${REDIS_PORT} -s ${SOCK_FILE} ${SSL_TEST_ARGS}

deps/hiredis/win32.h

@@ -2,10 +2,20 @@
 #define _WIN32_HELPER_INCLUDE
 #ifdef _MSC_VER
 
+#include <winsock2.h> /* for struct timeval */
+
 #ifndef inline
 #define inline __inline
 #endif
 
+#ifndef strcasecmp
+#define strcasecmp stricmp
+#endif
+
+#ifndef strncasecmp
+#define strncasecmp strnicmp
+#endif
+
 #ifndef va_copy
 #define va_copy(d,s) ((d) = (s))
 #endif
@@ -37,6 +47,10 @@ __inline int c99_snprintf(char* str, size_t size, const char* format, ...)
     return count;
 }
 #endif
+#endif /* _MSC_VER */
 
-#endif
+#ifdef _WIN32
-#endif
+#define strerror_r(errno,buf,len) strerror_s(buf,len,errno)
+#endif /* _WIN32 */
+
+#endif /* _WIN32_HELPER_INCLUDE */

deps/jemalloc/src/background_thread.c

@@ -787,7 +787,13 @@ background_thread_stats_read(tsdn_t *tsdn, background_thread_stats_t *stats) {
 	nstime_init(&stats->run_interval, 0);
 	for (unsigned i = 0; i < max_background_threads; i++) {
 		background_thread_info_t *info = &background_thread_info[i];
-		malloc_mutex_lock(tsdn, &info->mtx);
+		if (malloc_mutex_trylock(tsdn, &info->mtx)) {
+			/*
+			 * Each background thread run may take a long time;
+			 * avoid waiting on the stats if the thread is active.
+			 */
+			continue;
+		}
 		if (info->state != background_thread_stopped) {
 			num_runs += info->tot_n_runs;
 			nstime_add(&stats->run_interval, &info->tot_sleep_time);

keydb.conf

@@ -129,6 +129,76 @@ timeout 0
 # KeyDB default starting with Redis 3.2.1.
 tcp-keepalive 300
 
+################################# TLS/SSL #####################################
+
+# By default, TLS/SSL is disabled. To enable it, the "tls-port" configuration
+# directive can be used to define TLS-listening ports. To enable TLS on the
+# default port, use:
+#
+# port 0
+# tls-port 6379
+
+# Configure a X.509 certificate and private key to use for authenticating the
+# server to connected clients, masters or cluster peers.  These files should be
+# PEM formatted.
+#
+# tls-cert-file redis.crt tls-key-file redis.key
+
+# Configure a DH parameters file to enable Diffie-Hellman (DH) key exchange:
+#
+# tls-dh-params-file redis.dh
+
+# Configure a CA certificate(s) bundle or directory to authenticate TLS/SSL
+# clients and peers.  Redis requires an explicit configuration of at least one
+# of these, and will not implicitly use the system wide configuration.
+#
+# tls-ca-cert-file ca.crt
+# tls-ca-cert-dir /etc/ssl/certs
+
+# If TLS/SSL clients are required to authenticate using a client side
+# certificate, use this directive.
+#
+# Note: this applies to all incoming clients, including replicas.
+#
+# tls-auth-clients yes
+
+# If TLS/SSL should be used when connecting as a replica to a master, enable
+# this configuration directive:
+#
+# tls-replication yes
+
+# If TLS/SSL should be used for the Redis Cluster bus, enable this configuration
+# directive.
+#
+# NOTE: If TLS/SSL is enabled for Cluster Bus, mutual authentication is always
+# enforced.
+#
+# tls-cluster yes
+
+# Explicitly specify TLS versions to support. Allowed values are case insensitive
+# and include "TLSv1", "TLSv1.1", "TLSv1.2", "TLSv1.3" (OpenSSL >= 1.1.1) or
+# "default" which is currently >= TLSv1.1.
+#
+# tls-protocols TLSv1.2
+
+# Configure allowed ciphers.  See the ciphers(1ssl) manpage for more information
+# about the syntax of this string.
+#
+# Note: this configuration applies only to <= TLSv1.2.
+#
+# tls-ciphers DEFAULT:!MEDIUM
+
+# Configure allowed TLSv1.3 ciphersuites.  See the ciphers(1ssl) manpage for more
+# information about the syntax of this string, and specifically for TLSv1.3
+# ciphersuites.
+#
+# tls-ciphersuites TLS_CHACHA20_POLY1305_SHA256
+
+# When choosing a cipher, use the server's preference instead of the client
+# preference. By default, the server follows the client's preference.
+#
+# tls-prefer-server-ciphers yes
+
 ################################# GENERAL #####################################
 
 # By default KeyDB does not run as a daemon. Use 'yes' if you need it.
@@ -336,13 +406,11 @@ replica-read-only yes
 
 # Replication SYNC strategy: disk or socket.
 #
-# -------------------------------------------------------
+# New replicas and reconnecting replicas that are not able to continue the
-# WARNING: DISKLESS REPLICATION IS EXPERIMENTAL CURRENTLY
+# replication process just receiving differences, need to do what is called a
-# -------------------------------------------------------
+# "full synchronization". An RDB file is transmitted from the master to the
+# replicas.
 #
-# New replicas and reconnecting replicas that are not able to continue the replication
-# process just receiving differences, need to do what is called a "full
-# synchronization". An RDB file is transmitted from the master to the replicas.
 # The transmission can happen in two different ways:
 #
 # 1) Disk-backed: The KeyDB master creates a new process that writes the RDB
@@ -352,14 +420,14 @@ replica-read-only yes
 #              RDB file to replica sockets, without touching the disk at all.
 #
 # With disk-backed replication, while the RDB file is generated, more replicas
-# can be queued and served with the RDB file as soon as the current child producing
+# can be queued and served with the RDB file as soon as the current child
-# the RDB file finishes its work. With diskless replication instead once
+# producing the RDB file finishes its work. With diskless replication instead
-# the transfer starts, new replicas arriving will be queued and a new transfer
+# once the transfer starts, new replicas arriving will be queued and a new
-# will start when the current one terminates.
+# transfer will start when the current one terminates.
 #
 # When diskless replication is used, the master waits a configurable amount of
-# time (in seconds) before starting the transfer in the hope that multiple replicas
+# time (in seconds) before starting the transfer in the hope that multiple
-# will arrive and the transfer can be parallelized.
+# replicas will arrive and the transfer can be parallelized.
 #
 # With slow disks and fast (large bandwidth) networks, diskless replication
 # works better.
@@ -370,16 +438,42 @@ repl-diskless-sync no
 # to the replicas.
 #
 # This is important since once the transfer starts, it is not possible to serve
-# new replicas arriving, that will be queued for the next RDB transfer, so the server
+# new replicas arriving, that will be queued for the next RDB transfer, so the
-# waits a delay in order to let more replicas arrive.
+# server waits a delay in order to let more replicas arrive.
 #
 # The delay is specified in seconds, and by default is 5 seconds. To disable
 # it entirely just set it to 0 seconds and the transfer will start ASAP.
 repl-diskless-sync-delay 5
 
-# Replicas send PINGs to server in a predefined interval. It's possible to change
+# -----------------------------------------------------------------------------
-# this interval with the repl_ping_replica_period option. The default value is 10
+# WARNING: RDB diskless load is experimental. Since in this setup the replica
-# seconds.
+# does not immediately store an RDB on disk, it may cause data loss during
+# failovers. RDB diskless load + Redis modules not handling I/O reads may also
+# cause Redis to abort in case of I/O errors during the initial synchronization
+# stage with the master. Use only if your do what you are doing.
+# -----------------------------------------------------------------------------
+#
+# Replica can load the RDB it reads from the replication link directly from the
+# socket, or store the RDB to a file and read that file after it was completely
+# recived from the master.
+#
+# In many cases the disk is slower than the network, and storing and loading
+# the RDB file may increase replication time (and even increase the master's
+# Copy on Write memory and salve buffers).
+# However, parsing the RDB file directly from the socket may mean that we have
+# to flush the contents of the current database before the full rdb was
+# received. For this reason we have the following options:
+#
+# "disabled"    - Don't use diskless load (store the rdb file to the disk first)
+# "on-empty-db" - Use diskless load only when it is completely safe.
+# "swapdb"      - Keep a copy of the current db contents in RAM while parsing
+#                 the data directly from the socket. note that this requires
+#                 sufficient memory, if you don't have it, you risk an OOM kill.
+repl-diskless-load disabled
+
+# Replicas send PINGs to server in a predefined interval. It's possible to
+# change this interval with the repl_ping_replica_period option. The default
+# value is 10 seconds.
 #
 # repl-ping-replica-period 10
 
@@ -411,10 +505,10 @@ repl-diskless-sync-delay 5
 repl-disable-tcp-nodelay no
 
 # Set the replication backlog size. The backlog is a buffer that accumulates
-# replica data when replicas are disconnected for some time, so that when a replica
+# replica data when replicas are disconnected for some time, so that when a
-# wants to reconnect again, often a full resync is not needed, but a partial
+# replica wants to reconnect again, often a full resync is not needed, but a
-# resync is enough, just passing the portion of data the replica missed while
+# partial resync is enough, just passing the portion of data the replica
-# disconnected.
+# missed while disconnected.
 #
 # The bigger the replication backlog, the longer the time the replica can be
 # disconnected and later be able to perform a partial resynchronization.
@@ -436,13 +530,13 @@ repl-disable-tcp-nodelay no
 #
 # repl-backlog-ttl 3600
 
-# The replica priority is an integer number published by KeyDB in the INFO output.
+# The replica priority is an integer number published by KeyDB in the INFO
-# It is used by KeyDB Sentinel in order to select a replica to promote into a
+# output. It is used by Redis Sentinel in order to select a replica to promote
-# master if the master is no longer working correctly.
+# into a master if the master is no longer working correctly.
 #
 # A replica with a low priority number is considered better for promotion, so
-# for instance if there are three replicas with priority 10, 100, 25 Sentinel will
+# for instance if there are three replicas with priority 10, 100, 25 Sentinel
-# pick the one with priority 10, that is the lowest.
+# will pick the one with priority 10, that is the lowest.
 #
 # However a special priority of 0 marks the replica as not able to perform the
 # role of master, so a replica with priority of 0 will never be selected by
@@ -502,6 +596,39 @@ replica-priority 100
 # replica-announce-ip 5.5.5.5
 # replica-announce-port 1234
 
+############################### KEYS TRACKING #################################
+
+# Redis implements server assisted support for client side caching of values.
+# This is implemented using an invalidation table that remembers, using
+# 16 millions of slots, what clients may have certain subsets of keys. In turn
+# this is used in order to send invalidation messages to clients. Please
+# to understand more about the feature check this page:
+#
+#   https://redis.io/topics/client-side-caching
+#
+# When tracking is enabled for a client, all the read only queries are assumed
+# to be cached: this will force Redis to store information in the invalidation
+# table. When keys are modified, such information is flushed away, and
+# invalidation messages are sent to the clients. However if the workload is
+# heavily dominated by reads, Redis could use more and more memory in order
+# to track the keys fetched by many clients.
+#
+# For this reason it is possible to configure a maximum fill value for the
+# invalidation table. By default it is set to 10%, and once this limit is
+# reached, Redis will start to evict caching slots in the invalidation table
+# even if keys are not modified, just to reclaim memory: this will in turn
+# force the clients to invalidate the cached values. Basically the table
+# maximum fill rate is a trade off between the memory you want to spend server
+# side to track information about who cached what, and the ability of clients
+# to retain cached objects in memory.
+#
+# If you set the value to 0, it means there are no limits, and all the 16
+# millions of caching slots can be used at the same time. In the "stats"
+# INFO section, you can find information about the amount of caching slots
+# used at every given moment.
+#
+# tracking-table-max-fill 10
+
 ################################## SECURITY ###################################
 
 # Warning: since KeyDB is pretty fast an outside user can try up to
@@ -684,13 +811,13 @@ replica-priority 100
 # maxmemory <bytes>
 
 # MAXMEMORY POLICY: how KeyDB will select what to remove when maxmemory
-# is reached. You can select among five behaviors:
+# is reached. You can select one from the following behaviors:
 #
-# volatile-lru -> Evict using approximated LRU among the keys with an expire set.
+# volatile-lru -> Evict using approximated LRU, only keys with an expire set.
 # allkeys-lru -> Evict any key using approximated LRU.
-# volatile-lfu -> Evict using approximated LFU among the keys with an expire set.
+# volatile-lfu -> Evict using approximated LFU, only keys with an expire set.
 # allkeys-lfu -> Evict any key using approximated LFU.
-# volatile-random -> Remove a random key among the ones with an expire set.
+# volatile-random -> Remove a random key having an expire set.
 # allkeys-random -> Remove a random key, any key.
 # volatile-ttl -> Remove the key with the nearest expire time (minor TTL)
 # noeviction -> Don't evict anything, just return an error on write operations.
@@ -731,20 +858,37 @@ replica-priority 100
 # DEL commands to the replica as keys evict in the master side.
 #
 # This behavior ensures that masters and replicas stay consistent, and is usually
-# what you want, however if your replica is writable, or you want the replica to have
+# what you want, however if your replica is writable, or you want the replica
-# a different memory setting, and you are sure all the writes performed to the
+# to have a different memory setting, and you are sure all the writes performed
-# replica are idempotent, then you may change this default (but be sure to understand
+# to the replica are idempotent, then you may change this default (but be sure
-# what you are doing).
+# to understand what you are doing).
 #
 # Note that since the replica by default does not evict, it may end using more
 # memory than the one set via maxmemory (there are certain buffers that may
-# be larger on the replica, or data structures may sometimes take more memory and so
+# be larger on the replica, or data structures may sometimes take more memory
-# forth). So make sure you monitor your replicas and make sure they have enough
+# and so forth). So make sure you monitor your replicas and make sure they
-# memory to never hit a real out-of-memory condition before the master hits
+# have enough memory to never hit a real out-of-memory condition before the
-# the configured maxmemory setting.
+# master hits the configured maxmemory setting.
 #
 # replica-ignore-maxmemory yes
 
+# Redis reclaims expired keys in two ways: upon access when those keys are
+# found to be expired, and also in background, in what is called the
+# "active expire key". The key space is slowly and interactively scanned
+# looking for expired keys to reclaim, so that it is possible to free memory
+# of keys that are expired and will never be accessed again in a short time.
+#
+# The default effort of the expire cycle will try to avoid having more than
+# ten percent of expired keys still in memory, and will try to avoid consuming
+# more than 25% of total memory and to add latency to the system. However
+# it is possible to increase the expire "effort" that is normally set to
+# "1", to a greater value, up to the value "10". At its maximum value the
+# system will use more CPU, longer cycles (and technically may introduce
+# more latency), and will tollerate less already expired keys still present
+# in the system. It's a tradeoff betweeen memory, CPU and latecy.
+#
+# active-expire-effort 1
+
 ############################# LAZY FREEING ####################################
 
 # KeyDB has two primitives to delete keys. One is called DEL and is a blocking
@@ -1050,6 +1194,22 @@ lua-time-limit 5000
 #
 # cluster-replica-no-failover no
 
+# This option, when set to yes, allows nodes to serve read traffic while the
+# the cluster is in a down state, as long as it believes it owns the slots. 
+#
+# This is useful for two cases.  The first case is for when an application 
+# doesn't require consistency of data during node failures or network partitions.
+# One example of this is a cache, where as long as the node has the data it
+# should be able to serve it. 
+#
+# The second use case is for configurations that don't meet the recommended  
+# three shards but want to enable cluster mode and scale later. A 
+# master outage in a 1 or 2 shard configuration causes a read/write outage to the
+# entire cluster without this option set, with it set there is only a write outage.
+# Without a quorum of masters, slot ownership will not change automatically. 
+#
+# cluster-allow-reads-when-down no
+
 # In order to setup your cluster make sure to read the documentation
 # available at http://redis.io web site.
 
@@ -1220,7 +1380,7 @@ notify-keyspace-events ""
 # to server Gopher pages MAKE SURE TO SET A PASSWORD to the instance.
 # Once a password is set:
 #
-#   1. The Gopher server (when enabled, not by default) will kill serve
+#   1. The Gopher server (when enabled, not by default) will still serve
 #      content via Gopher.
 #   2. However other commands cannot be called before the client will
 #      authenticate.
@@ -1479,10 +1639,6 @@ rdb-save-incremental-fsync yes
 
 ########################### ACTIVE DEFRAGMENTATION #######################
 #
-# WARNING THIS FEATURE IS EXPERIMENTAL. However it was stress tested
-# even in production and manually tested by multiple engineers for some
-# time.
-#
 # What is active defragmentation?
 # -------------------------------
 #
@@ -1522,7 +1678,7 @@ rdb-save-incremental-fsync yes
 # a good idea to leave the defaults untouched.
 
 # Enabled active defragmentation
-# activedefrag yes
+# activedefrag no
 
 # Minimum amount of fragmentation waste to start active defrag
 # active-defrag-ignore-bytes 100mb
@@ -1533,11 +1689,13 @@ rdb-save-incremental-fsync yes
 # Maximum percentage of fragmentation at which we use maximum effort
 # active-defrag-threshold-upper 100
 
-# Minimal effort for defrag in CPU percentage
+# Minimal effort for defrag in CPU percentage, to be used when the lower
-# active-defrag-cycle-min 5
+# threshold is reached
+# active-defrag-cycle-min 1
 
-# Maximal effort for defrag in CPU percentage
+# Maximal effort for defrag in CPU percentage, to be used when the upper
-# active-defrag-cycle-max 75
+# threshold is reached
+# active-defrag-cycle-max 25
 
 # Maximum number of set/hash/zset/list fields that will be processed from
 # the main dictionary scan

runtest-moduleapi

@@ -13,4 +13,16 @@ then
 fi
 
 make -C tests/modules && \
-$TCLSH tests/test_helper.tcl --single unit/moduleapi/commandfilter "${@}"
+$TCLSH tests/test_helper.tcl \
+--single unit/moduleapi/commandfilter \
+--single unit/moduleapi/fork \
+--single unit/moduleapi/testrdb \
+--single unit/moduleapi/infotest \
+--single unit/moduleapi/propagate \
+--single unit/moduleapi/hooks \
+--single unit/moduleapi/misc \
+--single unit/moduleapi/blockonkeys \
+--single unit/moduleapi/scan \
+--single unit/moduleapi/datatype \
+--single unit/moduleapi/auth \
+"${@}"

src/Makefile

@@ -21,7 +21,7 @@ NODEPS:=clean distclean
 
 # Default settings
 STD=-std=c11 -pedantic -DREDIS_STATIC=''
-CXX_STD=-std=c++14 -pedantic -fno-rtti -D__STDC_FORMAT_MACROS  
+CXX_STD=-std=c++2a -pedantic -fno-rtti -D__STDC_FORMAT_MACROS  
 ifneq (,$(findstring clang,$(CC)))
 ifneq (,$(findstring FreeBSD,$(uname_S)))
   STD+=-Wno-c11-extensions
@@ -33,6 +33,7 @@ OPT=$(OPTIMIZATION)
 PREFIX?=/usr/local
 INSTALL_BIN=$(PREFIX)/bin
 INSTALL=install
+PKG_CONFIG?=pkg-config
 
 # Default allocator defaults to Jemalloc if it's not an ARM
 MALLOC=libc
@@ -108,6 +109,15 @@ FINAL_LDFLAGS=$(LDFLAGS) $(REDIS_LDFLAGS) $(DEBUG)
 FINAL_LIBS=-lm
 DEBUG=-g -ggdb
 
+# Linux ARM needs -latomic at linking time
+ifneq (,$(filter aarch64 armv,$(uname_M)))
+        FINAL_LIBS+=-latomic
+else
+ifneq (,$(findstring armv,$(uname_M)))
+        FINAL_LIBS+=-latomic
+endif
+endif
+
 ifeq ($(uname_S),SunOS)
 	# SunOS
         ifneq ($(@@),32bit)
@@ -126,6 +136,8 @@ else
 ifeq ($(uname_S),Darwin)
 	# Darwin
 	FINAL_LIBS+= -ldl
+	OPENSSL_CFLAGS=-I/usr/local/opt/openssl/include
+	OPENSSL_LDFLAGS=-L/usr/local/opt/openssl/lib
 else
 ifeq ($(uname_S),AIX)
         # AIX
@@ -166,6 +178,30 @@ endif
 FINAL_CFLAGS+= -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src
 FINAL_CXXFLAGS+= -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src 
 
+# Determine systemd support and/or build preference (defaulting to auto-detection)
+BUILD_WITH_SYSTEMD=no
+# If 'USE_SYSTEMD' in the environment is neither "no" nor "yes", try to
+# auto-detect libsystemd's presence and link accordingly.
+ifneq ($(USE_SYSTEMD),no)
+	LIBSYSTEMD_PKGCONFIG := $(shell $(PKG_CONFIG) --exists libsystemd && echo $$?)
+# If libsystemd cannot be detected, continue building without support for it
+# (unless a later check tells us otherwise)
+ifeq ($(LIBSYSTEMD_PKGCONFIG),0)
+	BUILD_WITH_SYSTEMD=yes
+endif
+endif
+ifeq ($(USE_SYSTEMD),yes)
+ifneq ($(LIBSYSTEMD_PKGCONFIG),0)
+$(error USE_SYSTEMD is set to "$(USE_SYSTEMD)", but $(PKG_CONFIG) cannot find libsystemd)
+endif
+# Force building with libsystemd
+	BUILD_WITH_SYSTEMD=yes
+endif
+ifeq ($(BUILD_WITH_SYSTEMD),yes)
+	FINAL_LIBS+=$(shell $(PKG_CONFIG) --libs libsystemd)
+	FINAL_CFLAGS+= -DHAVE_LIBSYSTEMD
+endif
+
 ifeq ($(MALLOC),tcmalloc)
 	FINAL_CFLAGS+= -DUSE_TCMALLOC
 	FINAL_CXXFLAGS+= -DUSE_TCMALLOC
@@ -192,6 +228,12 @@ ifeq ($(MALLOC),memkind)
 	FINAL_LIBS := ../deps/memkind/src/.libs/libmemkind.a -lnuma $(FINAL_LIBS)
 endif
 
+ifeq ($(BUILD_TLS),yes)
+    FINAL_CFLAGS+=-DUSE_OPENSSL $(OPENSSL_CFLAGS)
+    FINAL_LDFLAGS+=$(OPENSSL_LDFLAGS)
+    FINAL_LIBS += ../deps/hiredis/libhiredis_ssl.a -lssl -lcrypto
+endif
+
 REDIS_CC=$(QUIET_CC)$(CC) $(FINAL_CFLAGS)
 REDIS_CXX=$(QUIET_CC)$(CXX) $(FINAL_CXXFLAGS)
 KEYDB_AS=$(QUIET_CC) as --64 -g
@@ -213,7 +255,7 @@ endif
 
 REDIS_SERVER_NAME=keydb-server
 REDIS_SENTINEL_NAME=keydb-sentinel
-REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o acl.o storage.o rdb-s3.o fastlock.o new.o tracking.o cron.o $(ASM_OBJ)
+REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o acl.o storage.o rdb-s3.o fastlock.o new.o tracking.o cron.o connection.o tls.o sha256.o $(ASM_OBJ)
 REDIS_CLI_NAME=keydb-cli
 REDIS_CLI_OBJ=anet.o adlist.o dict.o redis-cli.o redis-cli-cpphelper.o zmalloc.o release.o anet.o ae.o crc64.o siphash.o crc16.o storage-lite.o fastlock.o new.o $(ASM_OBJ)
 REDIS_BENCHMARK_NAME=keydb-benchmark
@@ -294,7 +336,6 @@ $(REDIS_BENCHMARK_NAME): $(REDIS_BENCHMARK_OBJ)
 dict-benchmark: dict.cpp zmalloc.cpp sds.c siphash.c
 	$(REDIS_CC) $(FINAL_CFLAGS) $^ -D DICT_BENCHMARK_MAIN -o $@ $(FINAL_LIBS)
 
-
 DEP = $(REDIS_SERVER_OBJ:%.o=%.d) $(REDIS_CLI_OBJ:%.o=%.d) $(REDIS_BENCHMARK_OBJ:%.o=%.d)
 -include $(DEP)
 
@@ -302,10 +343,10 @@ DEP = $(REDIS_SERVER_OBJ:%.o=%.d) $(REDIS_CLI_OBJ:%.o=%.d) $(REDIS_BENCHMARK_OBJ
 # building it should complete before building any other object. Instead of
 # depending on a single artifact, build all dependencies first.
 %.o: %.c .make-prerequisites
-	$(REDIS_CC) -MMD -c $<
+	$(REDIS_CC) -MMD -o $@ -c $<
 
 %.o: %.cpp .make-prerequisites
-	$(REDIS_CXX) -MMD -c $<
+	$(REDIS_CXX) -MMD -o $@ -c $<
 
 %.o: %.asm .make-prerequisites
 	$(KEYDB_AS) $< -o $@

src/acl.cpp

@@ -28,6 +28,9 @@
  */
 
 #include "server.h"
+extern "C" {
+#include "sha256.h"
+}
 #include <fcntl.h>
 
 /* =============================================================================
@@ -93,6 +96,9 @@ void ACLResetSubcommandsForCommand(user *u, unsigned long id);
 void ACLResetSubcommands(user *u);
 void ACLAddAllowedSubcommand(user *u, unsigned long id, const char *sub);
 
+/* The length of the string representation of a hashed password. */
+#define HASH_PASSWORD_LEN SHA256_BLOCK_SIZE*2
+
 /* =============================================================================
  * Helper functions for the rest of the ACL implementation
  * ==========================================================================*/
@@ -139,6 +145,25 @@ int time_independent_strcmp(char *a, char *b) {
     return diff; /* If zero strings are the same. */
 }
 
+/* Given an SDS string, returns the SHA256 hex representation as a
+ * new SDS string. */
+sds ACLHashPassword(unsigned char *cleartext, size_t len) {
+    SHA256_CTX ctx;
+    unsigned char hash[SHA256_BLOCK_SIZE];
+    char hex[HASH_PASSWORD_LEN];
+    const char *cset = "0123456789abcdef";
+
+    sha256_init(&ctx);
+    sha256_update(&ctx,(unsigned char*)cleartext,len);
+    sha256_final(&ctx,hash);
+
+    for (int j = 0; j < SHA256_BLOCK_SIZE; j++) {
+        hex[j*2] = cset[((hash[j]&0xF0)>>4)];
+        hex[j*2+1] = cset[(hash[j]&0xF)];
+    }
+    return sdsnewlen(hex,HASH_PASSWORD_LEN);
+}
+
 /* =============================================================================
  * Low level ACL API
  * ==========================================================================*/
@@ -160,12 +185,12 @@ int ACLListMatchSds(void *a, void *b) {
     return sdscmp((sds)a,(sds)b) == 0;
 }
 
-/* Method to free list elements from ACL users password/ptterns lists. */
+/* Method to free list elements from ACL users password/patterns lists. */
 void ACLListFreeSds(const void *item) {
     sdsfree((sds)item);
 }
 
-/* Method to duplicate list elements from ACL users password/ptterns lists. */
+/* Method to duplicate list elements from ACL users password/patterns lists. */
 void *ACLListDupSds(void *item) {
     return sdsdup((sds)item);
 }
@@ -502,7 +527,7 @@ sds ACLDescribeUser(user *u) {
     listRewind(u->passwords,&li);
     while((ln = listNext(&li))) {
         sds thispass = (sds)listNodeValue(ln);
-        res = sdscatlen(res,">",1);
+        res = sdscatlen(res,"#",1);
         res = sdscatsds(res,thispass);
         res = sdscatlen(res," ",1);
     }
@@ -629,7 +654,14 @@ void ACLAddAllowedSubcommand(user *u, unsigned long id, const char *sub) {
  * ><password>  Add this password to the list of valid password for the user.
  *              For example >mypass will add "mypass" to the list.
  *              This directive clears the "nopass" flag (see later).
+ * #<hash>      Add this password hash to the list of valid hashes for
+ *              the user. This is useful if you have previously computed
+ *              the hash, and don't want to store it in plaintext.
+ *              This directive clears the "nopass" flag (see later).
  * <<password>  Remove this password from the list of valid passwords.
+ * !<hash>      Remove this hashed password from the list of valid passwords.
+ *              This is useful when you want to remove a password just by
+ *              hash without knowing its plaintext version at all.
  * nopass       All the set passwords of the user are removed, and the user
  *              is flagged as requiring no password: it means that every
  *              password will work against this user. If this directive is
@@ -665,6 +697,7 @@ void ACLAddAllowedSubcommand(user *u, unsigned long id, const char *sub) {
  * EEXIST: You are adding a key pattern after "*" was already added. This is
  *         almost surely an error on the user side.
  * ENODEV: The password you are trying to remove from the user does not exist.
+ * EBADMSG: The hash you are trying to add is not a valid hash. 
  */
 int ACLSetUser(user *u, const char *op, ssize_t oplen) {
     if (oplen == -1) oplen = strlen(op);
@@ -700,14 +733,48 @@ int ACLSetUser(user *u, const char *op, ssize_t oplen) {
     } else if (!strcasecmp(op,"resetpass")) {
         u->flags &= ~USER_FLAG_NOPASS;
         listEmpty(u->passwords);
-    } else if (op[0] == '>') {
+    } else if (op[0] == '>' || op[0] == '#') {
-        sds newpass = sdsnewlen(op+1,oplen-1);
+        sds newpass;
+        if (op[0] == '>') {
+            newpass = ACLHashPassword((unsigned char*)op+1,oplen-1);
+        } else {
+            if (oplen != HASH_PASSWORD_LEN + 1) {
+                errno = EBADMSG;
+                return C_ERR;
+            }
+
+            /* Password hashes can only be characters that represent
+             * hexadecimal values, which are numbers and lowercase
+             * characters 'a' through 'f'.
+             */
+            for(int i = 1; i < HASH_PASSWORD_LEN + 1; i++) {
+                char c = op[i];
+                if ((c < 'a' || c > 'f') && (c < '0' || c > '9')) {
+                    errno = EBADMSG;
+                    return C_ERR;
+                }
+            }
+            newpass = sdsnewlen(op+1,oplen-1);
+        }
+
         listNode *ln = listSearchKey(u->passwords,newpass);
         /* Avoid re-adding the same password multiple times. */
-        if (ln == NULL) listAddNodeTail(u->passwords,newpass);
+        if (ln == NULL)
+            listAddNodeTail(u->passwords,newpass);
+        else
+            sdsfree(newpass);
         u->flags &= ~USER_FLAG_NOPASS;
-    } else if (op[0] == '<') {
+    } else if (op[0] == '<' || op[0] == '!') {
-        sds delpass = sdsnewlen(op+1,oplen-1);
+        sds delpass;
+        if (op[0] == '<') {
+            delpass = ACLHashPassword((unsigned char*)op+1,oplen-1);
+        } else {
+            if (oplen != HASH_PASSWORD_LEN + 1) {
+                errno = EBADMSG;
+                return C_ERR;
+            }
+            delpass = sdsnewlen(op+1,oplen-1);
+        }
         listNode *ln = listSearchKey(u->passwords,delpass);
         sdsfree(delpass);
         if (ln) {
@@ -724,7 +791,10 @@ int ACLSetUser(user *u, const char *op, ssize_t oplen) {
         sds newpat = sdsnewlen(op+1,oplen-1);
         listNode *ln = listSearchKey(u->patterns,newpat);
         /* Avoid re-adding the same pattern multiple times. */
-        if (ln == NULL) listAddNodeTail(u->patterns,newpat);
+        if (ln == NULL)
+            listAddNodeTail(u->patterns,newpat);
+        else
+            sdsfree(newpat);
         u->flags &= ~USER_FLAG_ALLKEYS;
     } else if (op[0] == '+' && op[1] != '@') {
         if (strchr(op,'|') == NULL) {
@@ -822,6 +892,9 @@ const char *ACLSetUserStringError(void) {
     else if (errno == ENODEV)
         errmsg = "The password you are trying to remove from the user does "
                  "not exist";
+    else if (errno == EBADMSG)
+        errmsg = "The password hash must be exactly 64 characters and contain "
+                 "only lowercase hexadecimal characters";
     return errmsg;
 }
 
@@ -879,11 +952,15 @@ int ACLCheckUserCredentials(robj *username, robj *password) {
     listIter li;
     listNode *ln;
     listRewind(u->passwords,&li);
+    sds hashed = ACLHashPassword((unsigned char*)szFromObj(password),sdslen(szFromObj(password)));
     while((ln = listNext(&li))) {
         sds thispass = (sds)listNodeValue(ln);
-        if (!time_independent_strcmp(szFromObj(password), thispass))
+        if (!time_independent_strcmp(hashed, thispass)) {
+            sdsfree(hashed);
             return C_OK;
         }
+    }
+    sdsfree(hashed);
 
     /* If we reached this point, no password matched. */
     errno = EINVAL;
@@ -900,6 +977,7 @@ int ACLAuthenticateUser(client *c, robj *username, robj *password) {
     if (ACLCheckUserCredentials(username,password) == C_OK) {
         c->authenticated = 1;
         c->puser = ACLGetUserByName((sds)ptrFromObj(username),sdslen((sds)ptrFromObj(username)));
+        moduleNotifyUserChanged(c);
         return C_OK;
     } else {
         return C_ERR;

src/adlist.h

@@ -70,7 +70,7 @@ typedef struct list {
 #define listSetMatchMethod(l,m) ((l)->match = (m))
 
 #define listGetDupMethod(l) ((l)->dup)
-#define listGetFree(l) ((l)->free)
+#define listGetFreeMethod(l) ((l)->free)
 #define listGetMatchMethod(l) ((l)->match)
 
 /* Prototypes */

src/ae.cpp

@@ -324,6 +324,7 @@ aeEventLoop *aeCreateEventLoop(int setsize) {
     eventLoop->maxfd = -1;
     eventLoop->beforesleep = NULL;
     eventLoop->aftersleep = NULL;
+    eventLoop->flags = 0;
     if (aeApiCreate(eventLoop) == -1) goto err;
     /* Events with mask == AE_NONE are not set. So let's initialize the
      * vector with it. */
@@ -357,6 +358,14 @@ int aeGetSetSize(aeEventLoop *eventLoop) {
     return eventLoop->setsize;
 }
 
+/* Tells the next iteration/s of the event processing to set timeout of 0. */
+void aeSetDontWait(aeEventLoop *eventLoop, int noWait) {
+    if (noWait)
+        eventLoop->flags |= AE_DONT_WAIT;
+    else
+        eventLoop->flags &= ~AE_DONT_WAIT;
+}
+
 /* Resize the maximum set size of the event loop.
  * If the requested set size is smaller than the current set size, but
  * there is already a file descriptor in use that is >= the requested
@@ -750,6 +759,11 @@ int aeProcessEvents(aeEventLoop *eventLoop, int flags)
             }
         }
 
+        if (eventLoop->flags & AE_DONT_WAIT) {
+            tv.tv_sec = tv.tv_usec = 0;
+            tvp = &tv;
+        }
+
         /* Call the multiplexing API, will return only on timeout or when
          * some event fires. */
         numevents = aeApiPoll(eventLoop, tvp);

src/ae.h

@@ -124,6 +124,7 @@ typedef struct aeEventLoop {
     int fdCmdWrite;
     int fdCmdRead;
     int cevents;
+    int flags;
 } aeEventLoop;
 
 /* Prototypes */
@@ -157,6 +158,7 @@ void aeSetBeforeSleepProc(aeEventLoop *eventLoop, aeBeforeSleepProc *beforesleep
 void aeSetAfterSleepProc(aeEventLoop *eventLoop, aeBeforeSleepProc *aftersleep, int flags);
 int aeGetSetSize(aeEventLoop *eventLoop);
 int aeResizeSetSize(aeEventLoop *eventLoop, int setsize);
+void aeSetDontWait(aeEventLoop *eventLoop, int noWait);
 
 void aeAcquireLock();
 int aeTryAcquireLock(int fWeak);

src/ae_epoll.cpp

@@ -125,8 +125,8 @@ static int aeApiPoll(aeEventLoop *eventLoop, struct timeval *tvp) {
 
             if (e->events & EPOLLIN) mask |= AE_READABLE;
             if (e->events & EPOLLOUT) mask |= AE_WRITABLE;
-            if (e->events & EPOLLERR) mask |= AE_WRITABLE;
+            if (e->events & EPOLLERR) mask |= AE_WRITABLE|AE_READABLE;
-            if (e->events & EPOLLHUP) mask |= AE_WRITABLE;
+            if (e->events & EPOLLHUP) mask |= AE_WRITABLE|AE_READABLE;
             eventLoop->fired[j].fd = e->data.fd;
             eventLoop->fired[j].mask = mask;
         }

src/aelocker.h

@@ -11,6 +11,11 @@ public:
 
     void arm(client *c) // if a client is passed, then the client is already locked
     {
+        if (m_fArmed)
+            return;
+        
+        serverAssertDebug(!GlobalLocksAcquired());
+
         if (c != nullptr)
         {
             serverAssert(!m_fArmed);

src/anet.c

@@ -193,6 +193,20 @@ int anetSendTimeout(char *err, int fd, long long ms) {
     return ANET_OK;
 }
 
+/* Set the socket receive timeout (SO_RCVTIMEO socket option) to the specified
+ * number of milliseconds, or disable it if the 'ms' argument is zero. */
+int anetRecvTimeout(char *err, int fd, long long ms) {
+    struct timeval tv;
+
+    tv.tv_sec = ms/1000;
+    tv.tv_usec = (ms%1000)*1000;
+    if (setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv)) == -1) {
+        anetSetError(err, "setsockopt SO_RCVTIMEO: %s", strerror(errno));
+        return ANET_ERR;
+    }
+    return ANET_OK;
+}
+
 /* anetGenericResolve() is called by anetResolve() and anetResolveIP() to
  * do the actual work. It resolves the hostname "host" and set the string
  * representation of the IP address into the buffer pointed by "ipbuf".
@@ -276,8 +290,8 @@ static int anetCreateSocket(char *err, int domain) {
 #define ANET_CONNECT_NONBLOCK 1
 #define ANET_CONNECT_BE_BINDING 2 /* Best effort binding. */
 #define ANET_CONNECT_REUSEPORT 4
-static int anetTcpGenericConnect(char *err, char *addr, int port,
+static int anetTcpGenericConnect(char *err, const char *addr, int port,
-                                 char *source_addr, int flags)
+                                 const char *source_addr, int flags)
 {
     int s = ANET_ERR, rv;
     char portstr[6];  /* strlen("65535") + 1; */
@@ -359,31 +373,31 @@ end:
     }
 }
 
-int anetTcpConnect(char *err, char *addr, int port)
+int anetTcpConnect(char *err, const char *addr, int port)
 {
     return anetTcpGenericConnect(err,addr,port,NULL,ANET_CONNECT_NONE);
 }
 
-int anetTcpNonBlockConnect(char *err, char *addr, int port)
+int anetTcpNonBlockConnect(char *err, const char *addr, int port)
 {
     return anetTcpGenericConnect(err,addr,port,NULL,ANET_CONNECT_NONBLOCK);
 }
 
-int anetTcpNonBlockBindConnect(char *err, char *addr, int port,
+int anetTcpNonBlockBindConnect(char *err, const char *addr, int port,
-                               char *source_addr)
+                               const char *source_addr)
 {
     return anetTcpGenericConnect(err,addr,port,source_addr,
             ANET_CONNECT_NONBLOCK);
 }
 
-int anetTcpNonBlockBestEffortBindConnect(char *err, char *addr, int port,
+int anetTcpNonBlockBestEffortBindConnect(char *err, const char *addr, int port,
-                                         char *source_addr)
+                                         const char *source_addr)
 {
     return anetTcpGenericConnect(err,addr,port,source_addr,
             ANET_CONNECT_NONBLOCK|ANET_CONNECT_BE_BINDING);
 }
 
-int anetUnixGenericConnect(char *err, char *path, int flags)
+int anetUnixGenericConnect(char *err, const char *path, int flags)
 {
     int s;
     struct sockaddr_un sa;
@@ -411,12 +425,12 @@ int anetUnixGenericConnect(char *err, char *path, int flags)
     return s;
 }
 
-int anetUnixConnect(char *err, char *path)
+int anetUnixConnect(char *err, const char *path)
 {
     return anetUnixGenericConnect(err,path,ANET_CONNECT_NONE);
 }
 
-int anetUnixNonBlockConnect(char *err, char *path)
+int anetUnixNonBlockConnect(char *err, const char *path)
 {
     return anetUnixGenericConnect(err,path,ANET_CONNECT_NONBLOCK);
 }

src/anet.h

@@ -53,12 +53,12 @@ extern "C" {
 #undef ip_len
 #endif
 
-int anetTcpConnect(char *err, char *addr, int port);
+int anetTcpConnect(char *err, const char *addr, int port);
-int anetTcpNonBlockConnect(char *err, char *addr, int port);
+int anetTcpNonBlockConnect(char *err, const char *addr, int port);
-int anetTcpNonBlockBindConnect(char *err, char *addr, int port, char *source_addr);
+int anetTcpNonBlockBindConnect(char *err, const char *addr, int port, const char *source_addr);
-int anetTcpNonBlockBestEffortBindConnect(char *err, char *addr, int port, char *source_addr);
+int anetTcpNonBlockBestEffortBindConnect(char *err, const char *addr, int port, const char *source_addr);
-int anetUnixConnect(char *err, char *path);
+int anetUnixConnect(char *err, const char *path);
-int anetUnixNonBlockConnect(char *err, char *path);
+int anetUnixNonBlockConnect(char *err, const char *path);
 int anetRead(int fd, char *buf, int count);
 int anetResolve(char *err, char *host, char *ipbuf, size_t ipbuf_len);
 int anetResolveIP(char *err, char *host, char *ipbuf, size_t ipbuf_len);
@@ -74,6 +74,7 @@ int anetEnableTcpNoDelay(char *err, int fd);
 int anetDisableTcpNoDelay(char *err, int fd);
 int anetTcpKeepAlive(char *err, int fd);
 int anetSendTimeout(char *err, int fd, long long ms);
+int anetRecvTimeout(char *err, int fd, long long ms);
 int anetPeerToString(int fd, char *ip, size_t ip_len, int *port);
 int anetKeepAlive(char *err, int fd, int interval);
 int anetSockName(int fd, char *ip, size_t ip_len, int *port);

src/aof.cpp

@@ -271,9 +271,9 @@ int startAppendOnly(void) {
             strerror(errno));
         return C_ERR;
     }
-    if (g_pserver->rdb_child_pid != -1) {
+    if (hasActiveChildProcess() && g_pserver->aof_child_pid == -1) {
         g_pserver->aof_rewrite_scheduled = 1;
-        serverLog(LL_WARNING,"AOF was enabled but there is already a child process saving an RDB file on disk. An AOF background was scheduled to start when possible.");
+        serverLog(LL_WARNING,"AOF was enabled but there is already another background operation. An AOF background was scheduled to start when possible.");
     } else {
         /* If there is a pending AOF rewrite, we need to switch it off and
          * start a new one: the old one cannot be reused because it is not
@@ -310,9 +310,7 @@ ssize_t aofWrite(int fd, const char *buf, size_t len) {
         nwritten = write(fd, buf, len);
 
         if (nwritten < 0) {
-            if (errno == EINTR) {
+            if (errno == EINTR) continue;
-                continue;
-            }
             return totwritten ? totwritten : -1;
         }
 
@@ -394,6 +392,10 @@ void flushAppendOnlyFile(int force) {
      * there is much to do about the whole server stopping for power problems
      * or alike */
 
+    if (g_pserver->aof_flush_sleep && sdslen(g_pserver->aof_buf)) {
+        usleep(g_pserver->aof_flush_sleep);
+    }
+
     latencyStartMonitor(latency);
     nwritten = aofWrite(g_pserver->aof_fd,g_pserver->aof_buf,sdslen(g_pserver->aof_buf));
     latencyEndMonitor(latency);
@@ -404,7 +406,7 @@ void flushAppendOnlyFile(int force) {
      * useful for graphing / monitoring purposes. */
     if (sync_in_progress) {
         latencyAddSampleIfNeeded("aof-write-pending-fsync",latency);
-    } else if (g_pserver->aof_child_pid != -1 || g_pserver->rdb_child_pid != -1) {
+    } else if (hasActiveChildProcess()) {
         latencyAddSampleIfNeeded("aof-write-active-child",latency);
     } else {
         latencyAddSampleIfNeeded("aof-write-alone",latency);
@@ -500,8 +502,7 @@ void flushAppendOnlyFile(int force) {
 try_fsync:
     /* Don't fsync if no-appendfsync-on-rewrite is set to yes and there are
      * children doing I/O in the background. */
-    if (g_pserver->aof_no_fsync_on_rewrite &&
+    if (g_pserver->aof_no_fsync_on_rewrite && hasActiveChildProcess())
-        (g_pserver->aof_child_pid != -1 || g_pserver->rdb_child_pid != -1))
         return;
 
     /* Perform the fsync if needed. */
@@ -658,11 +659,12 @@ void feedAppendOnlyFile(struct redisCommand *cmd, int dictid, robj **argv, int a
 
 /* In Redis commands are always executed in the context of a client, so in
  * order to load the append only file we need to create a fake client. */
-client *createFakeClient(void) {
+struct client *createAOFClient(void) {
-    client *c = (client*)zmalloc(sizeof(*c), MALLOC_LOCAL);
+    struct client *c =(client*) zmalloc(sizeof(*c), MALLOC_LOCAL);
 
     selectDb(c,0);
-    c->fd = -1;
+    c->id = CLIENT_ID_AOF; /* So modules can identify it's the AOF client. */
+    c->conn = NULL;
     c->iel = IDX_EVENT_LOOP_MAIN;
     c->name = NULL;
     c->querybuf = sdsempty();
@@ -742,8 +744,8 @@ int loadAppendOnlyFile(char *filename) {
      * to the same file we're about to read. */
     g_pserver->aof_state = AOF_OFF;
 
-    fakeClient = createFakeClient();
+    fakeClient = createAOFClient();
-    startLoading(fp);
+    startLoadingFile(fp, filename, RDBFLAGS_AOF_PREAMBLE);
 
     /* Check if this AOF file has an RDB preamble. In that case we need to
      * load the RDB file and later continue loading the AOF tail. */
@@ -754,12 +756,12 @@ int loadAppendOnlyFile(char *filename) {
     } else {
         /* RDB preamble. Pass loading the RDB functions. */
         rio rdb;
+        rdbSaveInfo rsi = RDB_SAVE_INFO_INIT;
 
         serverLog(LL_NOTICE,"Reading RDB preamble from AOF file...");
         if (fseek(fp,0,SEEK_SET) == -1) goto readerr;
         rioInitWithFile(&rdb,fp);
-        rdbSaveInfo rsi = RDB_SAVE_INFO_INIT;
+        if (rdbLoadRio(&rdb,RDBFLAGS_AOF_PREAMBLE,&rsi) != C_OK) {
-        if (rdbLoadRio(&rdb,&rsi,1) != C_OK) {
             serverLog(LL_WARNING,"Error reading the RDB preamble of the AOF file, AOF loading aborted");
             goto readerr;
         } else {
@@ -780,6 +782,7 @@ int loadAppendOnlyFile(char *filename) {
         if (!(loops++ % 1000)) {
             loadingProgress(ftello(fp));
             processEventsWhileBlocked(serverTL - g_pserver->rgthreadvar);
+            processModuleLoadingProgressEvent(1);
         }
 
         if (fgets(buf,sizeof(buf),fp) == NULL) {
@@ -793,18 +796,24 @@ int loadAppendOnlyFile(char *filename) {
         argc = atoi(buf+1);
         if (argc < 1) goto fmterr;
 
+        /* Load the next command in the AOF as our fake client
+         * argv. */
         argv = (robj**)zmalloc(sizeof(robj*)*argc, MALLOC_LOCAL);
         fakeClient->argc = argc;
         fakeClient->argv = argv;
 
         for (j = 0; j < argc; j++) {
-            if (fgets(buf,sizeof(buf),fp) == NULL) {
+            /* Parse the argument len. */
+            if (fgets(buf,sizeof(buf),fp) == NULL ||
+                buf[0] != '$')
+            {
                 fakeClient->argc = j; /* Free up to j-1. */
                 freeFakeClientArgv(fakeClient);
                 goto readerr;
             }
-            if (buf[0] != '$') goto fmterr;
             len = strtol(buf+1,NULL,10);
+
+            /* Read it into a string object. */
             argsds = sdsnewlen(SDS_NOINIT,len);
             if (len && fread(argsds,len,1,fp) == 0) {
                 sdsfree(argsds);
@@ -813,10 +822,12 @@ int loadAppendOnlyFile(char *filename) {
                 goto readerr;
             }
             argv[j] = createObject(OBJ_STRING,argsds);
+
+            /* Discard CRLF. */
             if (fread(buf,2,1,fp) == 0) {
                 fakeClient->argc = j+1; /* Free up to j. */
                 freeFakeClientArgv(fakeClient);
-                goto readerr; /* discard CRLF */
+                goto readerr;
             }
         }
 
@@ -853,6 +864,8 @@ int loadAppendOnlyFile(char *filename) {
         freeFakeClientArgv(fakeClient);
         fakeClient->cmd = NULL;
         if (g_pserver->aof_load_truncated) valid_up_to = ftello(fp);
+        if (g_pserver->key_load_delay)
+            usleep(g_pserver->key_load_delay);
     }
 
     /* This point can only be reached when EOF is reached without errors.
@@ -870,7 +883,7 @@ loaded_ok: /* DB loaded, cleanup and return C_OK to the caller. */
     fclose(fp);
     freeFakeClient(fakeClient);
     g_pserver->aof_state = old_aof_state;
-    stopLoading();
+    stopLoading(1);
     aofUpdateCurrentSize();
     g_pserver->aof_rewrite_base_size = g_pserver->aof_current_size;
     g_pserver->aof_fsync_offset = g_pserver->aof_current_size;
@@ -879,6 +892,7 @@ loaded_ok: /* DB loaded, cleanup and return C_OK to the caller. */
 readerr: /* Read error. If feof(fp) is true, fall through to unexpected EOF. */
     if (!feof(fp)) {
         if (fakeClient) freeFakeClient(fakeClient); /* avoid valgrind warning */
+        fclose(fp);
         serverLog(LL_WARNING,"Unrecoverable error reading the append only file: %s", strerror(errno));
         exit(1);
     }
@@ -909,11 +923,13 @@ uxeof: /* Unexpected AOF end of file. */
         }
     }
     if (fakeClient) freeFakeClient(fakeClient); /* avoid valgrind warning */
-    serverLog(LL_WARNING,"Unexpected end of file reading the append only file. You can: 1) Make a backup of your AOF file, then use ./keydb-check-aof --fix <filename>. 2) Alternatively you can set the 'aof-load-truncated' configuration option to yes and restart the g_pserver->");
+    fclose(fp);
+    serverLog(LL_WARNING,"Unexpected end of file reading the append only file. You can: 1) Make a backup of your AOF file, then use ./keydb-check-aof --fix <filename>. 2) Alternatively you can set the 'aof-load-truncated' configuration option to yes and restart the server.");
     exit(1);
 
 fmterr: /* Format error. */
     if (fakeClient) freeFakeClient(fakeClient); /* avoid valgrind warning */
+    fclose(fp);
     serverLog(LL_WARNING,"Bad file format reading the append only file: make a backup of your AOF file, then use ./keydb-check-aof --fix <filename>");
     exit(1);
 }
@@ -1421,9 +1437,11 @@ int rewriteAppendOnlyFile(char *filename) {
     if (g_pserver->aof_rewrite_incremental_fsync)
         rioSetAutoSync(&aof,REDIS_AUTOSYNC_BYTES);
 
+    startSaving(RDBFLAGS_AOF_PREAMBLE);
+
     if (g_pserver->aof_use_rdb_preamble) {
         int error;
-        if (rdbSaveRio(&aof,&error,RDB_SAVE_AOF_PREAMBLE,NULL) == C_ERR) {
+        if (rdbSaveRio(&aof,&error,RDBFLAGS_AOF_PREAMBLE,NULL) == C_ERR) {
             errno = error;
             goto werr;
         }
@@ -1485,15 +1503,18 @@ int rewriteAppendOnlyFile(char *filename) {
     if (rename(tmpfile,filename) == -1) {
         serverLog(LL_WARNING,"Error moving temp append only file on the final destination: %s", strerror(errno));
         unlink(tmpfile);
+        stopSaving(0);
         return C_ERR;
     }
     serverLog(LL_NOTICE,"SYNC append only file rewrite performed");
+    stopSaving(1);
     return C_OK;
 
 werr:
     serverLog(LL_WARNING,"Write error writing append only file on disk: %s", strerror(errno));
     fclose(fp);
     unlink(tmpfile);
+    stopSaving(0);
     return C_ERR;
 }
 
@@ -1599,39 +1620,24 @@ void aofClosePipes(void) {
  */
 int rewriteAppendOnlyFileBackground(void) {
     pid_t childpid;
-    long long start;
 
-    if (g_pserver->aof_child_pid != -1 || g_pserver->rdb_child_pid != -1) return C_ERR;
+    if (hasActiveChildProcess()) return C_ERR;
     if (aofCreatePipes() != C_OK) return C_ERR;
     openChildInfoPipe();
-    start = ustime();
+    if ((childpid = redisFork()) == 0) {
-    if ((childpid = fork()) == 0) {
         char tmpfile[256];
 
         /* Child */
-        closeListeningSockets(0);
         redisSetProcTitle("keydb-aof-rewrite");
         snprintf(tmpfile,256,"temp-rewriteaof-bg-%d.aof", (int) getpid());
         if (rewriteAppendOnlyFile(tmpfile) == C_OK) {
-            size_t private_dirty = zmalloc_get_private_dirty(-1);
+            sendChildCOWInfo(CHILD_INFO_TYPE_AOF, "AOF rewrite");
-
-            if (private_dirty) {
-                serverLog(LL_NOTICE,
-                    "AOF rewrite: %zu MB of memory used by copy-on-write",
-                    private_dirty/(1024*1024));
-            }
-
-            g_pserver->child_info_data.cow_size = private_dirty;
-            sendChildInfo(CHILD_INFO_TYPE_AOF);
             exitFromChild(0);
         } else {
             exitFromChild(1);
         }
     } else {
         /* Parent */
-        g_pserver->stat_fork_time = ustime()-start;
-        g_pserver->stat_fork_rate = (double) zmalloc_used_memory() * 1000000 / g_pserver->stat_fork_time / (1024*1024*1024); /* GB per second. */
-        latencyAddSampleIfNeeded("fork",g_pserver->stat_fork_time/1000);
         if (childpid == -1) {
             closeChildInfoPipe();
             serverLog(LL_WARNING,
@@ -1645,7 +1651,6 @@ int rewriteAppendOnlyFileBackground(void) {
         g_pserver->aof_rewrite_scheduled = 0;
         g_pserver->aof_rewrite_time_start = time(NULL);
         g_pserver->aof_child_pid = childpid;
-        updateDictResizePolicy();
         /* We set appendseldb to -1 in order to force the next call to the
          * feedAppendOnlyFile() to issue a SELECT command, so the differences
          * accumulated by the parent into g_pserver->aof_rewrite_buf will start
@@ -1660,13 +1665,14 @@ int rewriteAppendOnlyFileBackground(void) {
 void bgrewriteaofCommand(client *c) {
     if (g_pserver->aof_child_pid != -1) {
         addReplyError(c,"Background append only file rewriting already in progress");
-    } else if (g_pserver->rdb_child_pid != -1) {
+    } else if (hasActiveChildProcess()) {
         g_pserver->aof_rewrite_scheduled = 1;
         addReplyStatus(c,"Background append only file rewriting scheduled");
     } else if (rewriteAppendOnlyFileBackground() == C_OK) {
         addReplyStatus(c,"Background append only file rewriting started");
     } else {
-        addReply(c,shared.err);
+        addReplyError(c,"Can't execute an AOF background rewriting. "
+                        "Please check the server logs for more information.");
     }
 }
 
@@ -1805,6 +1811,7 @@ void backgroundRewriteDoneHandler(int exitcode, int bysignal) {
             g_pserver->aof_selected_db = -1; /* Make sure SELECT is re-issued */
             aofUpdateCurrentSize();
             g_pserver->aof_rewrite_base_size = g_pserver->aof_current_size;
+            g_pserver->aof_fsync_offset = g_pserver->aof_current_size;
 
             /* Clear regular AOF buffer since its contents was just written to
              * the new AOF from the background rewrite buffer. */

src/bio.h

@@ -1,3 +1,4 @@
+#pragma once
 /*
  * Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
  * All rights reserved.

src/blocked.cpp

@@ -92,7 +92,7 @@ int getTimeoutFromObjectOrReply(client *c, robj *object, mstime_t *timeout, int
     }
 
     if (tval < 0) {
-        addReplyError(c,"timeout is negative");
+        addReplyErrorAsync(c,"timeout is negative");
         return C_ERR;
     }
 
@@ -189,6 +189,7 @@ void unblockClient(client *c) {
     } else if (c->btype == BLOCKED_WAIT) {
         unblockClientWaitingReplicas(c);
     } else if (c->btype == BLOCKED_MODULE) {
+        if (moduleClientIsBlockedOnKeys(c)) unblockClientWaitingData(c);
         unblockClientFromModule(c);
     } else {
         serverPanic("Unknown btype in unblockClient().");
@@ -209,9 +210,9 @@ void replyToBlockedClientTimedOut(client *c) {
     if (c->btype == BLOCKED_LIST ||
         c->btype == BLOCKED_ZSET ||
         c->btype == BLOCKED_STREAM) {
-        addReplyNullArray(c);
+        addReplyNullArrayAsync(c);
     } else if (c->btype == BLOCKED_WAIT) {
-        addReplyLongLong(c,replicationCountAcksByOffset(c->bpop.reploffset));
+        addReplyLongLongAsync(c,replicationCountAcksByOffset(c->bpop.reploffset));
     } else if (c->btype == BLOCKED_MODULE) {
         moduleBlockedClientTimedOut(c);
     } else {
@@ -247,6 +248,253 @@ void disconnectAllBlockedClients(void) {
     }
 }
 
+/* Helper function for handleClientsBlockedOnKeys(). This function is called
+ * when there may be clients blocked on a list key, and there may be new
+ * data to fetch (the key is ready). */
+void serveClientsBlockedOnListKey(robj *o, readyList *rl) {
+    /* We serve clients in the same order they blocked for
+     * this key, from the first blocked to the last. */
+    dictEntry *de = dictFind(rl->db->blocking_keys,rl->key);
+    if (de) {
+        list *clients = (list*)dictGetVal(de);
+        int numclients = listLength(clients);
+
+        while(numclients--) {
+            listNode *clientnode = listFirst(clients);
+            client *receiver = (client*)clientnode->value;
+            std::unique_lock<decltype(receiver->lock)> lock(receiver->lock);
+
+            if (receiver->btype != BLOCKED_LIST) {
+                /* Put at the tail, so that at the next call
+                 * we'll not run into it again. */
+                listDelNode(clients,clientnode);
+                listAddNodeTail(clients,receiver);
+                continue;
+            }
+
+            robj *dstkey = receiver->bpop.target;
+            int where = (receiver->lastcmd &&
+                         receiver->lastcmd->proc == blpopCommand) ?
+                         LIST_HEAD : LIST_TAIL;
+            robj *value = listTypePop(o,where);
+
+            if (value) {
+                /* Protect receiver->bpop.target, that will be
+                 * freed by the next unblockClient()
+                 * call. */
+                if (dstkey) incrRefCount(dstkey);
+                unblockClient(receiver);
+
+                if (serveClientBlockedOnList(receiver,
+                    rl->key,dstkey,rl->db,value,
+                    where) == C_ERR)
+                {
+                    /* If we failed serving the client we need
+                     * to also undo the POP operation. */
+                    listTypePush(o,value,where);
+                }
+
+                if (dstkey) decrRefCount(dstkey);
+                decrRefCount(value);
+            } else {
+                break;
+            }
+        }
+    }
+
+    if (listTypeLength(o) == 0) {
+        dbDelete(rl->db,rl->key);
+        notifyKeyspaceEvent(NOTIFY_GENERIC,"del",rl->key,rl->db->id);
+    }
+    /* We don't call signalModifiedKey() as it was already called
+     * when an element was pushed on the list. */
+}
+
+/* Helper function for handleClientsBlockedOnKeys(). This function is called
+ * when there may be clients blocked on a sorted set key, and there may be new
+ * data to fetch (the key is ready). */
+void serveClientsBlockedOnSortedSetKey(robj *o, readyList *rl) {
+    /* We serve clients in the same order they blocked for
+     * this key, from the first blocked to the last. */
+    dictEntry *de = dictFind(rl->db->blocking_keys,rl->key);
+    if (de) {
+        list *clients = (list*)dictGetVal(de);
+        int numclients = listLength(clients);
+        unsigned long zcard = zsetLength(o);
+
+        while(numclients-- && zcard) {
+            listNode *clientnode = listFirst(clients);
+            client *receiver = (client*)clientnode->value;
+            std::unique_lock<decltype(receiver->lock)> lock(receiver->lock);
+
+            if (receiver->btype != BLOCKED_ZSET) {
+                /* Put at the tail, so that at the next call
+                 * we'll not run into it again. */
+                listDelNode(clients,clientnode);
+                listAddNodeTail(clients,receiver);
+                continue;
+            }
+
+            int where = (receiver->lastcmd &&
+                         receiver->lastcmd->proc == bzpopminCommand)
+                         ? ZSET_MIN : ZSET_MAX;
+            unblockClient(receiver);
+            genericZpopCommand(receiver,&rl->key,1,where,1,NULL);
+            zcard--;
+
+            /* Replicate the command. */
+            robj *argv[2];
+            struct redisCommand *cmd = where == ZSET_MIN ?
+                                       cserver.zpopminCommand :
+                                       cserver.zpopmaxCommand;
+            argv[0] = createStringObject(cmd->name,strlen(cmd->name));
+            argv[1] = rl->key;
+            incrRefCount(rl->key);
+            propagate(cmd,receiver->db->id,
+                      argv,2,PROPAGATE_AOF|PROPAGATE_REPL);
+            decrRefCount(argv[0]);
+            decrRefCount(argv[1]);
+        }
+    }
+}
+
+/* Helper function for handleClientsBlockedOnKeys(). This function is called
+ * when there may be clients blocked on a stream key, and there may be new
+ * data to fetch (the key is ready). */
+void serveClientsBlockedOnStreamKey(robj *o, readyList *rl) {
+    dictEntry *de = dictFind(rl->db->blocking_keys,rl->key);
+    stream *s = (stream*)ptrFromObj(o);
+
+    /* We need to provide the new data arrived on the stream
+     * to all the clients that are waiting for an offset smaller
+     * than the current top item. */
+    if (de) {
+        list *clients = (list*)dictGetVal(de);
+        listNode *ln;
+        listIter li;
+        listRewind(clients,&li);
+
+        while((ln = listNext(&li))) {
+            client *receiver = (client*)listNodeValue(ln);
+            if (receiver->btype != BLOCKED_STREAM) continue;
+            std::unique_lock<decltype(receiver->lock)> lock(receiver->lock);
+            streamID *gt = (streamID*)dictFetchValue(receiver->bpop.keys,
+                                          rl->key);
+
+            /* If we blocked in the context of a consumer
+             * group, we need to resolve the group and update the
+             * last ID the client is blocked for: this is needed
+             * because serving other clients in the same consumer
+             * group will alter the "last ID" of the consumer
+             * group, and clients blocked in a consumer group are
+             * always blocked for the ">" ID: we need to deliver
+             * only new messages and avoid unblocking the client
+             * otherwise. */
+            streamCG *group = NULL;
+            if (receiver->bpop.xread_group) {
+                group = streamLookupCG(s,
+                        szFromObj(receiver->bpop.xread_group));
+                /* If the group was not found, send an error
+                 * to the consumer. */
+                if (!group) {
+                    addReplyErrorAsync(receiver,
+                        "-NOGROUP the consumer group this client "
+                        "was blocked on no longer exists");
+                    unblockClient(receiver);
+                    continue;
+                } else {
+                    *gt = group->last_id;
+                }
+            }
+
+            if (streamCompareID(&s->last_id, gt) > 0) {
+                streamID start = *gt;
+                streamIncrID(&start);
+
+                /* Lookup the consumer for the group, if any. */
+                streamConsumer *consumer = NULL;
+                int noack = 0;
+
+                if (group) {
+                    consumer = streamLookupConsumer(group,
+                               szFromObj(receiver->bpop.xread_consumer),
+                               1);
+                    noack = receiver->bpop.xread_group_noack;
+                }
+
+                /* Emit the two elements sub-array consisting of
+                 * the name of the stream and the data we
+                 * extracted from it. Wrapped in a single-item
+                 * array, since we have just one key. */
+                if (receiver->resp == 2) {
+                    addReplyArrayLenAsync(receiver,1);
+                    addReplyArrayLenAsync(receiver,2);
+                } else {
+                    addReplyMapLenAsync(receiver,1);
+                }
+                addReplyBulkAsync(receiver,rl->key);
+
+                streamPropInfo pi = {
+                    rl->key,
+                    receiver->bpop.xread_group
+                };
+                streamReplyWithRange(receiver,s,&start,NULL,
+                                     receiver->bpop.xread_count,
+                                     0, group, consumer, noack, &pi);
+
+                /* Note that after we unblock the client, 'gt'
+                 * and other receiver->bpop stuff are no longer
+                 * valid, so we must do the setup above before
+                 * this call. */
+                unblockClient(receiver);
+            }
+        }
+    }
+}
+
+/* Helper function for handleClientsBlockedOnKeys(). This function is called
+ * in order to check if we can serve clients blocked by modules using
+ * RM_BlockClientOnKeys(), when the corresponding key was signaled as ready:
+ * our goal here is to call the RedisModuleBlockedClient reply() callback to
+ * see if the key is really able to serve the client, and in that case,
+ * unblock it. */
+void serveClientsBlockedOnKeyByModule(readyList *rl) {
+    dictEntry *de;
+
+    /* We serve clients in the same order they blocked for
+     * this key, from the first blocked to the last. */
+    de = dictFind(rl->db->blocking_keys,rl->key);
+    if (de) {
+        list *clients = (list*)dictGetVal(de);
+        int numclients = listLength(clients);
+
+        while(numclients--) {
+            listNode *clientnode = listFirst(clients);
+            client *receiver = (client*)clientnode->value;
+
+            /* Put at the tail, so that at the next call
+             * we'll not run into it again: clients here may not be
+             * ready to be served, so they'll remain in the list
+             * sometimes. We want also be able to skip clients that are
+             * not blocked for the MODULE type safely. */
+            listDelNode(clients,clientnode);
+            listAddNodeTail(clients,receiver);
+
+            if (receiver->btype != BLOCKED_MODULE) continue;
+
+            /* Note that if *this* client cannot be served by this key,
+             * it does not mean that another client that is next into the
+             * list cannot be served as well: they may be blocked by
+             * different modules with different triggers to consider if a key
+             * is ready or not. This means we can't exit the loop but need
+             * to continue after the first failure. */
+            if (!moduleTryServeClientBlockedOnKey(receiver, rl->key)) continue;
+
+            moduleUnblockClient(receiver);
+        }
+    }
+}
+
 /* This function should be called by Redis every time a single command,
  * a MULTI/EXEC block, or a Lua script, terminated its execution after
  * being called by a client. It handles serving clients blocked in
@@ -288,215 +536,32 @@ void handleClientsBlockedOnKeys(void) {
              * we can safely call signalKeyAsReady() against this key. */
             dictDelete(rl->db->ready_keys,rl->key);
 
+            /* Even if we are not inside call(), increment the call depth
+             * in order to make sure that keys are expired against a fixed
+             * reference time, and not against the wallclock time. This
+             * way we can lookup an object multiple times (BRPOPLPUSH does
+             * that) without the risk of it being freed in the second
+             * lookup, invalidating the first one.
+             * See https://github.com/antirez/redis/pull/6554. */
+            serverTL->fixed_time_expire++;
+            updateCachedTime(0);
+
             /* Serve clients blocked on list key. */
             robj *o = lookupKeyWrite(rl->db,rl->key);
-            if (o != NULL && o->type == OBJ_LIST) {
-                dictEntry *de;
 
-                /* We serve clients in the same order they blocked for
+            if (o != NULL) {
-                 * this key, from the first blocked to the last. */
+                if (o->type == OBJ_LIST)
-                de = dictFind(rl->db->blocking_keys,rl->key);
+                    serveClientsBlockedOnListKey(o,rl);
-                if (de) {
+                else if (o->type == OBJ_ZSET)
-                    list *clients = (list*)dictGetVal(de);
+                    serveClientsBlockedOnSortedSetKey(o,rl);
-                    int numclients = listLength(clients);
+                else if (o->type == OBJ_STREAM)
-
+                    serveClientsBlockedOnStreamKey(o,rl);
-                    while(numclients--) {
+                /* We want to serve clients blocked on module keys
-                        listNode *clientnode = listFirst(clients);
+                 * regardless of the object type: we don't know what the
-                        client *receiver = (client*)clientnode->value;
+                 * module is trying to accomplish right now. */
-                        std::unique_lock<decltype(client::lock)> lock(receiver->lock);
+                serveClientsBlockedOnKeyByModule(rl);
-
-                        if (receiver->btype != BLOCKED_LIST) {
-                            /* Put at the tail, so that at the next call
-                             * we'll not run into it again. */
-                            listDelNode(clients,clientnode);
-                            listAddNodeTail(clients,receiver);
-                            continue;
-                        }
-
-                        robj *dstkey = receiver->bpop.target;
-                        int where = (receiver->lastcmd &&
-                                     receiver->lastcmd->proc == blpopCommand) ?
-                                     LIST_HEAD : LIST_TAIL;
-                        robj *value = listTypePop(o,where);
-
-                        if (value) {
-                            /* Protect receiver->bpop.target, that will be
-                             * freed by the next unblockClient()
-                             * call. */
-                            if (dstkey) incrRefCount(dstkey);
-                            fastlock_lock(&receiver->lock);
-                            unblockClient(receiver);
-
-                            if (serveClientBlockedOnList(receiver,
-                                rl->key,dstkey,rl->db,value,
-                                where) == C_ERR)
-                            {
-                                /* If we failed serving the client we need
-                                 * to also undo the POP operation. */
-                                listTypePush(o,value,where);
-                            }
-
-                            if (dstkey) decrRefCount(dstkey);
-                            fastlock_unlock(&receiver->lock);
-                            decrRefCount(value);
-                        } else {
-                            break;
-                        }
-                    }
-                }
-
-                if (listTypeLength(o) == 0) {
-                    dbDelete(rl->db,rl->key);
-                    notifyKeyspaceEvent(NOTIFY_GENERIC,"del",rl->key,rl->db->id);
-                }
-                /* We don't call signalModifiedKey() as it was already called
-                 * when an element was pushed on the list. */
-            }
-
-            /* Serve clients blocked on sorted set key. */
-            else if (o != NULL && o->type == OBJ_ZSET) {
-                dictEntry *de;
-
-                /* We serve clients in the same order they blocked for
-                 * this key, from the first blocked to the last. */
-                de = dictFind(rl->db->blocking_keys,rl->key);
-                if (de) {
-                    list *clients = (list*)dictGetVal(de);
-                    int numclients = listLength(clients);
-                    unsigned long zcard = zsetLength(o);
-
-                    while(numclients-- && zcard) {
-                        listNode *clientnode = listFirst(clients);
-                        client *receiver = (client*)clientnode->value;
-
-                        if (receiver->btype != BLOCKED_ZSET) {
-                            /* Put at the tail, so that at the next call
-                             * we'll not run into it again. */
-                            listDelNode(clients,clientnode);
-                            listAddNodeTail(clients,receiver);
-                            continue;
-                        }
-
-                        fastlock_lock(&receiver->lock);
-                        int where = (receiver->lastcmd &&
-                                     receiver->lastcmd->proc == bzpopminCommand)
-                                     ? ZSET_MIN : ZSET_MAX;
-                        unblockClient(receiver);
-                        genericZpopCommand(receiver,&rl->key,1,where,1,NULL);
-                        zcard--;
-
-                        /* Replicate the command. */
-                        robj *argv[2];
-                        struct redisCommand *cmd = where == ZSET_MIN ?
-                                                   cserver.zpopminCommand :
-                                                   cserver.zpopmaxCommand;
-                        argv[0] = createStringObject(cmd->name,strlen(cmd->name));
-                        argv[1] = rl->key;
-                        incrRefCount(rl->key);
-                        propagate(cmd,receiver->db->id,
-                                  argv,2,PROPAGATE_AOF|PROPAGATE_REPL);
-                        fastlock_unlock(&receiver->lock);
-                        decrRefCount(argv[0]);
-                        decrRefCount(argv[1]);
-                    }
-                }
-            }
-
-            /* Serve clients blocked on stream key. */
-            else if (o != NULL && o->type == OBJ_STREAM) {
-                dictEntry *de = dictFind(rl->db->blocking_keys,rl->key);
-                stream *s = (stream*)ptrFromObj(o);
-
-                /* We need to provide the new data arrived on the stream
-                 * to all the clients that are waiting for an offset smaller
-                 * than the current top item. */
-                if (de) {
-                    list *clients = (list*)dictGetVal(de);
-                    listNode *ln;
-                    listIter li;
-                    listRewind(clients,&li);
-
-                    while((ln = listNext(&li))) {
-                        client *receiver = (client*)listNodeValue(ln);
-                        if (receiver->btype != BLOCKED_STREAM) continue;
-                        streamID *gt = (streamID*)dictFetchValue(receiver->bpop.keys,
-                                                      rl->key);
-
-                        /* If we blocked in the context of a consumer
-                         * group, we need to resolve the group and update the
-                         * last ID the client is blocked for: this is needed
-                         * because serving other clients in the same consumer
-                         * group will alter the "last ID" of the consumer
-                         * group, and clients blocked in a consumer group are
-                         * always blocked for the ">" ID: we need to deliver
-                         * only new messages and avoid unblocking the client
-                         * otherwise. */
-                        streamCG *group = NULL;
-                        if (receiver->bpop.xread_group) {
-                            group = streamLookupCG(s,
-                                    szFromObj(receiver->bpop.xread_group));
-                            /* If the group was not found, send an error
-                             * to the consumer. */
-                            if (!group) {
-                                fastlock_lock(&receiver->lock);
-                                addReplyErrorAsync(receiver,
-                                    "-NOGROUP the consumer group this client "
-                                    "was blocked on no longer exists");
-                                unblockClient(receiver);
-                                fastlock_unlock(&receiver->lock);
-                                continue;
-                            } else {
-                                *gt = group->last_id;
-                            }
-                        }
-
-                        if (streamCompareID(&s->last_id, gt) > 0) {
-                            streamID start = *gt;
-                            start.seq++; /* Can't overflow, it's an uint64_t */
-
-                            /* Lookup the consumer for the group, if any. */
-                            streamConsumer *consumer = NULL;
-                            int noack = 0;
-
-                            if (group) {
-                                consumer = streamLookupConsumer(group,
-                                           szFromObj(receiver->bpop.xread_consumer),
-                                           1);
-                                noack = receiver->bpop.xread_group_noack;
-                            }
-
-                            fastlock_lock(&receiver->lock);
-
-                            /* Emit the two elements sub-array consisting of
-                             * the name of the stream and the data we
-                             * extracted from it. Wrapped in a single-item
-                             * array, since we have just one key. */
-                            if (receiver->resp == 2) {
-                                addReplyArrayLenAsync(receiver,1);
-                                addReplyArrayLenAsync(receiver,2);
-                            } else {
-                                addReplyMapLenAsync(receiver,1);
-                            }
-                            addReplyBulkAsync(receiver,rl->key);
-
-                            streamPropInfo pi = {
-                                rl->key,
-                                receiver->bpop.xread_group
-                            };
-                            streamReplyWithRange(receiver,s,&start,NULL,
-                                                 receiver->bpop.xread_count,
-                                                 0, group, consumer, noack, &pi);
-
-                            /* Note that after we unblock the client, 'gt'
-                             * and other receiver->bpop stuff are no longer
-                             * valid, so we must do the setup above before
-                             * this call. */
-                            unblockClient(receiver);
-                            fastlock_unlock(&receiver->lock);
-                        }
-                    }
-                }
             }
+            serverTL->fixed_time_expire--;
 
             /* Free this item. */
             decrRefCount(rl->key);
@@ -621,7 +686,7 @@ void unblockClientWaitingData(client *c) {
  * the same key again and again in the list in case of multiple pushes
  * made by a script or in the context of MULTI/EXEC.
  *
- * The list will be finally processed by handleClientsBlockedOnLists() */
+ * The list will be finally processed by handleClientsBlockedOnKeys() */
 void signalKeyAsReady(redisDb *db, robj *key) {
     readyList *rl;
 

src/childinfo.cpp

@@ -80,6 +80,8 @@ void receiveChildInfo(void) {
             g_pserver->stat_rdb_cow_bytes = g_pserver->child_info_data.cow_size;
         } else if (g_pserver->child_info_data.process_type == CHILD_INFO_TYPE_AOF) {
             g_pserver->stat_aof_cow_bytes = g_pserver->child_info_data.cow_size;
+        } else if (g_pserver->child_info_data.process_type == CHILD_INFO_TYPE_MODULE) {
+            g_pserver->stat_module_cow_bytes = g_pserver->child_info_data.cow_size;
         }
     }
 }

src/cluster.cpp

@@ -49,7 +49,7 @@ clusterNode *myself = NULL;
 clusterNode *createClusterNode(char *nodename, int flags);
 int clusterAddNode(clusterNode *node);
 void clusterAcceptHandler(aeEventLoop *el, int fd, void *privdata, int mask);
-void clusterReadHandler(aeEventLoop *el, int fd, void *privdata, int mask);
+void clusterReadHandler(connection *conn);
 void clusterSendPing(clusterLink *link, int type);
 void clusterSendFail(char *nodename);
 void clusterSendFailoverAuthIfNeeded(clusterNode *node, clusterMsg *request);
@@ -146,6 +146,7 @@ int clusterLoadConfig(char *filename) {
         /* Handle the special "vars" line. Don't pretend it is the last
          * line even if it actually is when generated by Redis. */
         if (strcasecmp(argv[0],"vars") == 0) {
+            if (!(argc % 2)) goto fmterr;
             for (j = 1; j < argc; j += 2) {
                 if (strcasecmp(argv[j],"currentEpoch") == 0) {
                     g_pserver->cluster->currentEpoch =
@@ -489,7 +490,8 @@ void clusterInit(void) {
     /* Port sanity check II
      * The other handshake port check is triggered too late to stop
      * us from trying to use a too-high cluster port number. */
-    if (g_pserver->port > (65535-CLUSTER_PORT_INCR)) {
+    int port = g_pserver->tls_cluster ? g_pserver->tls_port : g_pserver->port;
+    if (port > (65535-CLUSTER_PORT_INCR)) {
         serverLog(LL_WARNING, "Redis port number too high. "
                    "Cluster communication port is 10,000 port "
                    "numbers higher than your Redis port. "
@@ -497,8 +499,7 @@ void clusterInit(void) {
                    "lower than 55535.");
         exit(1);
     }
-
+    if (listenToPort(port+CLUSTER_PORT_INCR,
-    if (listenToPort(g_pserver->port+CLUSTER_PORT_INCR,
         g_pserver->cfd,&g_pserver->cfd_count, 0 /*fReusePort*/, 0 /*fFirstListen*/) == C_ERR)
     {
         exit(1);
@@ -520,8 +521,8 @@ void clusterInit(void) {
 
     /* Set myself->port / cport to my listening ports, we'll just need to
      * discover the IP address via MEET messages. */
-    myself->port = g_pserver->port;
+    myself->port = port;
-    myself->cport = g_pserver->port+CLUSTER_PORT_INCR;
+    myself->cport = port+CLUSTER_PORT_INCR;
     if (g_pserver->cluster_announce_port)
         myself->port = g_pserver->cluster_announce_port;
     if (g_pserver->cluster_announce_bus_port)
@@ -609,7 +610,7 @@ clusterLink *createClusterLink(clusterNode *node) {
     link->sndbuf = sdsempty();
     link->rcvbuf = sdsempty();
     link->node = node;
-    link->fd = -1;
+    link->conn = NULL;
     return link;
 }
 
@@ -628,23 +629,45 @@ void freeClusterLink(clusterLink *link) {
         });
         return;
     }
-    if (link->fd != -1) {
+    if (link->conn) {
-        aeDeleteFileEvent(g_pserver->rgthreadvar[IDX_EVENT_LOOP_MAIN].el, link->fd, AE_READABLE|AE_WRITABLE);
+        connClose(link->conn);
+        link->conn = NULL;
     }
     sdsfree(link->sndbuf);
     sdsfree(link->rcvbuf);
     if (link->node)
         link->node->link = NULL;
-    close(link->fd);
     zfree(link);
 }
 
+static void clusterConnAcceptHandler(connection *conn) {
+    clusterLink *link;
+
+    if (connGetState(conn) != CONN_STATE_CONNECTED) {
+        serverLog(LL_VERBOSE,
+                "Error accepting cluster node connection: %s", connGetLastError(conn));
+        connClose(conn);
+        return;
+    }
+
+    /* Create a link object we use to handle the connection.
+     * It gets passed to the readable handler when data is available.
+     * Initiallly the link->node pointer is set to NULL as we don't know
+     * which node is, but the right node is references once we know the
+     * node identity. */
+    link = createClusterLink(NULL);
+    link->conn = conn;
+    connSetPrivateData(conn, link);
+
+    /* Register read handler */
+    connSetReadHandler(conn, clusterReadHandler);
+}
+
 #define MAX_CLUSTER_ACCEPTS_PER_CALL 1000
 void clusterAcceptHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
     int cport, cfd;
     int max = MAX_CLUSTER_ACCEPTS_PER_CALL;
     char cip[NET_IP_STR_LEN];
-    clusterLink *link;
     UNUSED(el);
     UNUSED(mask);
     UNUSED(privdata);
@@ -661,19 +684,24 @@ void clusterAcceptHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
                     "Error accepting cluster node: %s", serverTL->neterr);
             return;
         }
-        anetNonBlock(NULL,cfd);
+
-        anetEnableTcpNoDelay(NULL,cfd);
+        connection *conn = g_pserver->tls_cluster ? connCreateAcceptedTLS(cfd,1) : connCreateAcceptedSocket(cfd);
+        connNonBlock(conn);
+        connEnableTcpNoDelay(conn);
 
         /* Use non-blocking I/O for cluster messages. */
-        serverLog(LL_VERBOSE,"Accepted cluster node %s:%d", cip, cport);
+        serverLog(LL_VERBOSE,"Accepting cluster node connection from %s:%d", cip, cport);
-        /* Create a link object we use to handle the connection.
+
-         * It gets passed to the readable handler when data is available.
+        /* Accept the connection now.  connAccept() may call our handler directly
-         * Initiallly the link->node pointer is set to NULL as we don't know
+         * or schedule it for later depending on connection implementation.
-         * which node is, but the right node is references once we know the
+         */
-         * node identity. */
+        if (connAccept(conn, clusterConnAcceptHandler) == C_ERR) {
-        link = createClusterLink(NULL);
+            serverLog(LL_VERBOSE,
-        link->fd = cfd;
+                    "Error accepting cluster node connection: %s",
-        aeCreateFileEvent(g_pserver->rgthreadvar[IDX_EVENT_LOOP_MAIN].el,cfd,AE_READABLE,clusterReadHandler,link);
+                    connGetLastError(conn));
+            connClose(conn);
+            return;
+        }
     }
 }
 
@@ -1474,7 +1502,7 @@ void nodeIp2String(char *buf, clusterLink *link, char *announced_ip) {
         memcpy(buf,announced_ip,NET_IP_STR_LEN);
         buf[NET_IP_STR_LEN-1] = '\0'; /* We are not sure the input is sane. */
     } else {
-        anetPeerToString(link->fd, buf, NET_IP_STR_LEN, NULL);
+        connPeerToString(link->conn, buf, NET_IP_STR_LEN, NULL);
     }
 }
 
@@ -1782,7 +1810,7 @@ int clusterProcessPacket(clusterLink *link) {
         {
             char ip[NET_IP_STR_LEN];
 
-            if (anetSockName(link->fd,ip,sizeof(ip),NULL) != -1 &&
+            if (connSockName(link->conn,ip,sizeof(ip),NULL) != -1 &&
                 strcmp(ip,myself->ip))
             {
                 memcpy(myself->ip,ip,NET_IP_STR_LEN);
@@ -2149,12 +2177,10 @@ void handleLinkIOError(clusterLink *link) {
 /* Send data. This is handled using a trivial send buffer that gets
  * consumed by write(). We don't try to optimize this for speed too much
  * as this is a very low traffic channel. */
-void clusterWriteHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
+void clusterWriteHandler(connection *conn) {
-    serverAssert(ielFromEventLoop(el) == IDX_EVENT_LOOP_MAIN);
+    serverAssert(ielFromEventLoop(serverTL->el) == IDX_EVENT_LOOP_MAIN);
-    clusterLink *link = (clusterLink*) privdata;
+    clusterLink *link = (clusterLink*)connGetPrivateData(conn);
     ssize_t nwritten;
-    UNUSED(el);
-    UNUSED(mask);
 
     // We're about to release the lock, so the link's sndbuf needs to be owned fully by us
     //  allocate a new one in case anyone tries to write while we're waiting
@@ -2162,12 +2188,12 @@ void clusterWriteHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
     link->sndbuf = sdsempty();
 
     aeReleaseLock();
-    nwritten = write(fd, sndbuf, sdslen(sndbuf));
+    nwritten = connWrite(conn, link->sndbuf, sdslen(link->sndbuf));
     aeAcquireLock();
 
     if (nwritten <= 0) {
         serverLog(LL_DEBUG,"I/O error writing to node link: %s",
-            (nwritten == -1) ? strerror(errno) : "short write");
+            (nwritten == -1) ? connGetLastError(conn) : "short write");
         sdsfree(sndbuf);
         handleLinkIOError(link);
         return;
@@ -2178,20 +2204,63 @@ void clusterWriteHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
     sdsfree(link->sndbuf);
     link->sndbuf = sndbuf;
     if (sdslen(link->sndbuf) == 0)
-        aeDeleteFileEvent(el, link->fd, AE_WRITABLE);
+        connSetWriteHandler(link->conn, NULL);
+}
+
+/* A connect handler that gets called when a connection to another node
+ * gets established.
+ */
+void clusterLinkConnectHandler(connection *conn) {
+    clusterLink *link = (clusterLink*)connGetPrivateData(conn);
+    clusterNode *node = link->node;
+
+    /* Check if connection succeeded */
+    if (connGetState(conn) != CONN_STATE_CONNECTED) {
+        serverLog(LL_VERBOSE, "Connection with Node %.40s at %s:%d failed: %s",
+                node->name, node->ip, node->cport,
+                connGetLastError(conn));
+        freeClusterLink(link);
+        return;
+    }
+
+    /* Register a read handler from now on */
+    connSetReadHandler(conn, clusterReadHandler);
+
+    /* Queue a PING in the new connection ASAP: this is crucial
+     * to avoid false positives in failure detection.
+     *
+     * If the node is flagged as MEET, we send a MEET message instead
+     * of a PING one, to force the receiver to add us in its node
+     * table. */
+    mstime_t old_ping_sent = node->ping_sent;
+    clusterSendPing(link, node->flags & CLUSTER_NODE_MEET ?
+            CLUSTERMSG_TYPE_MEET : CLUSTERMSG_TYPE_PING);
+    if (old_ping_sent) {
+        /* If there was an active ping before the link was
+         * disconnected, we want to restore the ping time, otherwise
+         * replaced by the clusterSendPing() call. */
+        node->ping_sent = old_ping_sent;
+    }
+    /* We can clear the flag after the first packet is sent.
+     * If we'll never receive a PONG, we'll never send new packets
+     * to this node. Instead after the PONG is received and we
+     * are no longer in meet/handshake status, we want to send
+     * normal PING packets. */
+    node->flags &= ~CLUSTER_NODE_MEET;
+
+    serverLog(LL_DEBUG,"Connecting with Node %.40s at %s:%d",
+            node->name, node->ip, node->cport);
 }
 
 /* Read data. Try to read the first field of the header first to check the
  * full length of the packet. When a whole packet is in memory this function
  * will call the function to process the packet. And so forth. */
-void clusterReadHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
+void clusterReadHandler(connection *conn) {
-    char buf[sizeof(clusterMsg)];
+    clusterMsg buf[1];
     ssize_t nread;
     clusterMsg *hdr;
-    clusterLink *link = (clusterLink*) privdata;
+    clusterLink *link = (clusterLink*)connGetPrivateData(conn);
     unsigned int readlen, rcvbuflen;
-    UNUSED(el);
-    UNUSED(mask);
 
     while(1) { /* Read as long as there is data to read. */
         rcvbuflen = sdslen(link->rcvbuf);
@@ -2219,13 +2288,13 @@ void clusterReadHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
             if (readlen > sizeof(buf)) readlen = sizeof(buf);
         }
 
-        nread = read(fd,buf,readlen);
+        nread = connRead(conn,buf,readlen);
-        if (nread == -1 && errno == EAGAIN) return; /* No more data ready. */
+        if (nread == -1 && (connGetState(conn) == CONN_STATE_CONNECTED)) return; /* No more data ready. */
 
         if (nread <= 0) {
             /* I/O error... */
             serverLog(LL_DEBUG,"I/O error reading from node link: %s",
-                (nread == 0) ? "connection closed" : strerror(errno));
+                (nread == 0) ? "connection closed" : connGetLastError(conn));
             handleLinkIOError(link);
             return;
         } else {
@@ -2255,8 +2324,7 @@ void clusterReadHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
 void clusterSendMessage(clusterLink *link, unsigned char *msg, size_t msglen) {
     serverAssert(GlobalLocksAcquired());
     if (sdslen(link->sndbuf) == 0 && msglen != 0)
-        aeCreateRemoteFileEvent(g_pserver->rgthreadvar[IDX_EVENT_LOOP_MAIN].el,link->fd,AE_WRITABLE|AE_BARRIER,
+        connSetWriteHandlerWithBarrier(link->conn, clusterWriteHandler, 1);
-                    clusterWriteHandler,link,false);
 
     link->sndbuf = sdscatlen(link->sndbuf, msg, msglen);
 
@@ -2322,11 +2390,12 @@ void clusterBuildMessageHdr(clusterMsg *hdr, int type) {
     }
 
     /* Handle cluster-announce-port as well. */
+    int port = g_pserver->tls_cluster ? g_pserver->tls_port : g_pserver->port;
     int announced_port = g_pserver->cluster_announce_port ?
-                         g_pserver->cluster_announce_port : g_pserver->port;
+                         g_pserver->cluster_announce_port : port;
     int announced_cport = g_pserver->cluster_announce_bus_port ?
                           g_pserver->cluster_announce_bus_port :
-                          (g_pserver->port + CLUSTER_PORT_INCR);
+                          (port + CLUSTER_PORT_INCR);
 
     memcpy(hdr->myslots,master->slots,sizeof(hdr->myslots));
     memset(hdr->slaveof,0,CLUSTER_NAMELEN);
@@ -2563,7 +2632,8 @@ void clusterBroadcastPong(int target) {
  *
  * If link is NULL, then the message is broadcasted to the whole cluster. */
 void clusterSendPublish(clusterLink *link, robj *channel, robj *message) {
-    unsigned char buf[sizeof(clusterMsg)], *payload;
+    unsigned char *payload;
+    clusterMsg buf[1];
     clusterMsg *hdr = (clusterMsg*) buf;
     uint32_t totlen;
     uint32_t channel_len, message_len;
@@ -2583,7 +2653,7 @@ void clusterSendPublish(clusterLink *link, robj *channel, robj *message) {
 
     /* Try to use the local buffer if possible */
     if (totlen < sizeof(buf)) {
-        payload = buf;
+        payload = (unsigned char*)buf;
     } else {
         payload = (unsigned char*)zmalloc(totlen, MALLOC_LOCAL);
         memcpy(payload,hdr,sizeof(*hdr));
@@ -2600,7 +2670,7 @@ void clusterSendPublish(clusterLink *link, robj *channel, robj *message) {
 
     decrRefCount(channel);
     decrRefCount(message);
-    if (payload != buf) zfree(payload);
+    if (payload != (unsigned char*)buf) zfree(payload);
 }
 
 /* Send a FAIL message to all the nodes we are able to contact.
@@ -2609,7 +2679,7 @@ void clusterSendPublish(clusterLink *link, robj *channel, robj *message) {
  * we switch the node state to CLUSTER_NODE_FAIL and ask all the other
  * nodes to do the same ASAP. */
 void clusterSendFail(char *nodename) {
-    unsigned char buf[sizeof(clusterMsg)];
+    clusterMsg buf[1];
     clusterMsg *hdr = (clusterMsg*) buf;
 
     clusterBuildMessageHdr(hdr,CLUSTERMSG_TYPE_FAIL);
@@ -2621,7 +2691,7 @@ void clusterSendFail(char *nodename) {
  * slots configuration. The node name, slots bitmap, and configEpoch info
  * are included. */
 void clusterSendUpdate(clusterLink *link, clusterNode *node) {
-    unsigned char buf[sizeof(clusterMsg)];
+    clusterMsg buf[1];
     clusterMsg *hdr = (clusterMsg*) buf;
 
     if (link == NULL) return;
@@ -2629,7 +2699,7 @@ void clusterSendUpdate(clusterLink *link, clusterNode *node) {
     memcpy(hdr->data.update.nodecfg.nodename,node->name,CLUSTER_NAMELEN);
     hdr->data.update.nodecfg.configEpoch = htonu64(node->configEpoch);
     memcpy(hdr->data.update.nodecfg.slots,node->slots,sizeof(node->slots));
-    clusterSendMessage(link,buf,ntohl(hdr->totlen));
+    clusterSendMessage(link,(unsigned char*)buf,ntohl(hdr->totlen));
 }
 
 /* Send a MODULE message.
@@ -2637,7 +2707,8 @@ void clusterSendUpdate(clusterLink *link, clusterNode *node) {
  * If link is NULL, then the message is broadcasted to the whole cluster. */
 void clusterSendModule(clusterLink *link, uint64_t module_id, uint8_t type,
                        unsigned char *payload, uint32_t len) {
-    unsigned char buf[sizeof(clusterMsg)], *heapbuf;
+    unsigned char *heapbuf;
+    clusterMsg buf[1];
     clusterMsg *hdr = (clusterMsg*) buf;
     uint32_t totlen;
 
@@ -2652,7 +2723,7 @@ void clusterSendModule(clusterLink *link, uint64_t module_id, uint8_t type,
 
     /* Try to use the local buffer if possible */
     if (totlen < sizeof(buf)) {
-        heapbuf = buf;
+        heapbuf = (unsigned char*)buf;
     } else {
         heapbuf = (unsigned char*)zmalloc(totlen, MALLOC_LOCAL);
         memcpy(heapbuf,hdr,sizeof(*hdr));
@@ -2665,7 +2736,7 @@ void clusterSendModule(clusterLink *link, uint64_t module_id, uint8_t type,
     else
         clusterBroadcastMessage(heapbuf,totlen);
 
-    if (heapbuf != buf) zfree(heapbuf);
+    if (heapbuf != (unsigned char*)buf) zfree(heapbuf);
 }
 
 /* This function gets a cluster node ID string as target, the same way the nodes
@@ -2709,7 +2780,7 @@ void clusterPropagatePublish(robj *channel, robj *message) {
  * Note that we send the failover request to everybody, master and slave nodes,
  * but only the masters are supposed to reply to our query. */
 void clusterRequestFailoverAuth(void) {
-    unsigned char buf[sizeof(clusterMsg)];
+    clusterMsg buf[1];
     clusterMsg *hdr = (clusterMsg*) buf;
     uint32_t totlen;
 
@@ -2725,7 +2796,7 @@ void clusterRequestFailoverAuth(void) {
 
 /* Send a FAILOVER_AUTH_ACK message to the specified node. */
 void clusterSendFailoverAuth(clusterNode *node) {
-    unsigned char buf[sizeof(clusterMsg)];
+    clusterMsg buf[1];
     clusterMsg *hdr = (clusterMsg*) buf;
     uint32_t totlen;
 
@@ -2733,12 +2804,12 @@ void clusterSendFailoverAuth(clusterNode *node) {
     clusterBuildMessageHdr(hdr,CLUSTERMSG_TYPE_FAILOVER_AUTH_ACK);
     totlen = sizeof(clusterMsg)-sizeof(union clusterMsgData);
     hdr->totlen = htonl(totlen);
-    clusterSendMessage(node->link,buf,totlen);
+    clusterSendMessage(node->link,(unsigned char*)buf,totlen);
 }
 
 /* Send a MFSTART message to the specified node. */
 void clusterSendMFStart(clusterNode *node) {
-    unsigned char buf[sizeof(clusterMsg)];
+    clusterMsg buf[1];
     clusterMsg *hdr = (clusterMsg*) buf;
     uint32_t totlen;
 
@@ -2746,7 +2817,7 @@ void clusterSendMFStart(clusterNode *node) {
     clusterBuildMessageHdr(hdr,CLUSTERMSG_TYPE_MFSTART);
     totlen = sizeof(clusterMsg)-sizeof(union clusterMsgData);
     hdr->totlen = htonl(totlen);
-    clusterSendMessage(node->link,buf,totlen);
+    clusterSendMessage(node->link,(unsigned char*)buf,totlen);
 }
 
 /* Vote for the node asking for our vote if there are the conditions. */
@@ -3429,13 +3500,11 @@ void clusterCron(void) {
         }
 
         if (node->link == NULL) {
-            int fd;
+            clusterLink *link = createClusterLink(node);
-            mstime_t old_ping_sent;
+            link->conn = g_pserver->tls_cluster ? connCreateTLS() : connCreateSocket();
-            clusterLink *link;
+            connSetPrivateData(link->conn, link);
-
+            if (connConnect(link->conn, node->ip, node->cport, NET_FIRST_BIND_ADDR,
-            fd = anetTcpNonBlockBindConnect(serverTL->neterr, node->ip,
+                        clusterLinkConnectHandler) == -1) {
-                node->cport, NET_FIRST_BIND_ADDR);
-            if (fd == -1) {
                 /* We got a synchronous error from connect before
                  * clusterSendPing() had a chance to be called.
                  * If node->ping_sent is zero, failure detection can't work,
@@ -3445,37 +3514,11 @@ void clusterCron(void) {
                 serverLog(LL_DEBUG, "Unable to connect to "
                     "Cluster Node [%s]:%d -> %s", node->ip,
                     node->cport, serverTL->neterr);
+
+                freeClusterLink(link);
                 continue;
             }
-            link = createClusterLink(node);
-            link->fd = fd;
             node->link = link;
-            aeCreateFileEvent(g_pserver->rgthreadvar[IDX_EVENT_LOOP_MAIN].el,link->fd,AE_READABLE,
-                    clusterReadHandler,link);
-            /* Queue a PING in the new connection ASAP: this is crucial
-             * to avoid false positives in failure detection.
-             *
-             * If the node is flagged as MEET, we send a MEET message instead
-             * of a PING one, to force the receiver to add us in its node
-             * table. */
-            old_ping_sent = node->ping_sent;
-            clusterSendPing(link, node->flags & CLUSTER_NODE_MEET ?
-                    CLUSTERMSG_TYPE_MEET : CLUSTERMSG_TYPE_PING);
-            if (old_ping_sent) {
-                /* If there was an active ping before the link was
-                 * disconnected, we want to restore the ping time, otherwise
-                 * replaced by the clusterSendPing() call. */
-                node->ping_sent = old_ping_sent;
-            }
-            /* We can clear the flag after the first packet is sent.
-             * If we'll never receive a PONG, we'll never send new packets
-             * to this node. Instead after the PONG is received and we
-             * are no longer in meet/handshake status, we want to send
-             * normal PING packets. */
-            node->flags &= ~CLUSTER_NODE_MEET;
-
-            serverLog(LL_DEBUG,"Connecting with Node %.40s at %s:%d",
-                    node->name, node->ip, node->cport);
         }
     }
     dictReleaseIterator(di);
@@ -4317,12 +4360,9 @@ NULL
         }
     } else if (!strcasecmp(szFromObj(c->argv[1]),"nodes") && c->argc == 2) {
         /* CLUSTER NODES */
-        robj *o;
+        sds nodes = clusterGenNodesDescription(0);
-        sds ci = clusterGenNodesDescription(0);
+        addReplyVerbatim(c,nodes,sdslen(nodes),"txt");
-
+        sdsfree(nodes);
-        o = createObject(OBJ_STRING,ci);
-        addReplyBulk(c,o);
-        decrRefCount(o);
     } else if (!strcasecmp(szFromObj(c->argv[1]),"myid") && c->argc == 2) {
         /* CLUSTER MYID */
         addReplyBulkCBuffer(c,myself->name, CLUSTER_NAMELEN);
@@ -4564,10 +4604,8 @@ NULL
             "cluster_stats_messages_received:%lld\r\n", tot_msg_received);
 
         /* Produce the reply protocol. */
-        addReplySds(c,sdscatprintf(sdsempty(),"$%lu\r\n",
+        addReplyVerbatim(c,info,sdslen(info),"txt");
-            (unsigned long)sdslen(info)));
+        sdsfree(info);
-        addReplySds(c,info);
-        addReply(c,shared.crlf);
     } else if (!strcasecmp(szFromObj(c->argv[1]),"saveconfig") && c->argc == 2) {
         int retval = clusterSaveConfig(1);
 
@@ -4898,7 +4936,7 @@ int verifyDumpPayload(unsigned char *p, size_t len) {
  * DUMP is actually not used by Redis Cluster but it is the obvious
  * complement of RESTORE and can be useful for different applications. */
 void dumpCommand(client *c) {
-    robj_roptr o, dumpobj;
+    robj_roptr o;
     rio payload;
 
     /* Check if the key is here. */
@@ -4911,9 +4949,7 @@ void dumpCommand(client *c) {
     createDumpPayload(&payload,o,c->argv[1]);
 
     /* Transfer to the client */
-    dumpobj = createObject(OBJ_STRING,payload.io.buffer.ptr);
+    addReplyBulkSds(c,payload.io.buffer.ptr);
-    addReplyBulk(c,dumpobj);
-    decrRefCount(dumpobj);
     return;
 }
 
@@ -4996,7 +5032,7 @@ void restoreCommand(client *c) {
         if (!absttl) ttl+=mstime();
         setExpire(c,c->db,c->argv[1],nullptr,ttl);
     }
-    objectSetLRUOrLFU(obj,lfu_freq,lru_idle,lru_clock);
+    objectSetLRUOrLFU(obj,lfu_freq,lru_idle,lru_clock,1000);
     signalModifiedKey(c->db,c->argv[1]);
     addReply(c,shared.ok);
     g_pserver->dirty++;
@@ -5012,7 +5048,7 @@ void restoreCommand(client *c) {
 #define MIGRATE_SOCKET_CACHE_TTL 10 /* close cached sockets after 10 sec. */
 
 typedef struct migrateCachedSocket {
-    int fd;
+    connection *conn;
     long last_dbid;
     time_t last_use_time;
 } migrateCachedSocket;
@@ -5029,7 +5065,7 @@ typedef struct migrateCachedSocket {
  * should be called so that the connection will be created from scratch
  * the next time. */
 migrateCachedSocket* migrateGetSocket(client *c, robj *host, robj *port, long timeout) {
-    int fd;
+    connection *conn;
     sds name = sdsempty();
     migrateCachedSocket *cs;
 
@@ -5049,34 +5085,26 @@ migrateCachedSocket* migrateGetSocket(client *c, robj *host, robj *port, long ti
         /* Too many items, drop one at random. */
         dictEntry *de = dictGetRandomKey(g_pserver->migrate_cached_sockets);
         cs = (migrateCachedSocket*)dictGetVal(de);
-        close(cs->fd);
+        connClose(cs->conn);
         zfree(cs);
         dictDelete(g_pserver->migrate_cached_sockets,dictGetKey(de));
     }
 
     /* Create the socket */
-    fd = anetTcpNonBlockConnect(serverTL->neterr,szFromObj(c->argv[1]),
+    conn = g_pserver->tls_cluster ? connCreateTLS() : connCreateSocket();
-                                atoi(szFromObj(c->argv[2])));
+    if (connBlockingConnect(conn, szFromObj(c->argv[1]), atoi(szFromObj(c->argv[2])), timeout)
-    if (fd == -1) {
+            != C_OK) {
-        sdsfree(name);
-        addReplyErrorFormat(c,"Can't connect to target node: %s",
-            serverTL->neterr);
-        return NULL;
-    }
-    anetEnableTcpNoDelay(serverTL->neterr,fd);
-
-    /* Check if it connects within the specified timeout. */
-    if ((aeWait(fd,AE_WRITABLE,timeout) & AE_WRITABLE) == 0) {
-        sdsfree(name);
         addReplySds(c,
             sdsnew("-IOERR error or timeout connecting to the client\r\n"));
-        close(fd);
+        connClose(conn);
+        sdsfree(name);
         return NULL;
     }
+    connEnableTcpNoDelay(conn);
 
     /* Add to the cache and return it to the caller. */
     cs = (migrateCachedSocket*)zmalloc(sizeof(*cs), MALLOC_LOCAL);
-    cs->fd = fd;
+    cs->conn = conn;
     cs->last_dbid = -1;
     cs->last_use_time = g_pserver->unixtime;
     dictAdd(g_pserver->migrate_cached_sockets,name,cs);
@@ -5097,7 +5125,7 @@ void migrateCloseSocket(robj *host, robj *port) {
         return;
     }
 
-    close(cs->fd);
+    connClose(cs->conn);
     zfree(cs);
     dictDelete(g_pserver->migrate_cached_sockets,name);
     sdsfree(name);
@@ -5111,7 +5139,7 @@ void migrateCloseTimedoutSockets(void) {
         migrateCachedSocket *cs = (migrateCachedSocket*)dictGetVal(de);
 
         if ((g_pserver->unixtime - cs->last_use_time) > MIGRATE_SOCKET_CACHE_TTL) {
-            close(cs->fd);
+            connClose(cs->conn);
             zfree(cs);
             dictDelete(g_pserver->migrate_cached_sockets,dictGetKey(de));
         }
@@ -5303,7 +5331,7 @@ try_again:
 
         while ((towrite = sdslen(buf)-pos) > 0) {
             towrite = (towrite > (64*1024) ? (64*1024) : towrite);
-            nwritten = syncWrite(cs->fd,buf+pos,towrite,timeout);
+            nwritten = connSyncWrite(cs->conn,buf+pos,towrite,timeout);
             if (nwritten != (signed)towrite) {
                 write_error = 1;
                 goto socket_err;
@@ -5314,11 +5342,11 @@ try_again:
 
 
     /* Read the AUTH reply if needed. */
-    if (password && syncReadLine(cs->fd, buf0, sizeof(buf0), timeout) <= 0)
+    if (password && connSyncReadLine(cs->conn, buf0, sizeof(buf0), timeout) <= 0)
         goto socket_err;
 
     /* Read the SELECT reply if needed. */
-    if (select && syncReadLine(cs->fd, buf1, sizeof(buf1), timeout) <= 0)
+    if (select && connSyncReadLine(cs->conn, buf1, sizeof(buf1), timeout) <= 0)
         goto socket_err;
 
     /* Allocate the new argument vector that will replace the current command,
@@ -5329,7 +5357,7 @@ try_again:
 
     /* Read the RESTORE replies. */
     for (j = 0; j < num_keys; j++) {
-        if (syncReadLine(cs->fd, buf2, sizeof(buf2), timeout) <= 0) {
+        if (connSyncReadLine(cs->conn, buf2, sizeof(buf2), timeout) <= 0) {
             socket_error = 1;
             break;
         }
@@ -5519,8 +5547,8 @@ void readwriteCommand(client *c) {
  * already "down" but it is fragile to rely on the update of the global state,
  * so we also handle it here.
  *
- * CLUSTER_REDIR_DOWN_STATE if the cluster is down but the user attempts to
+ * CLUSTER_REDIR_DOWN_STATE and CLUSTER_REDIR_DOWN_RO_STATE if the cluster is 
- * execute a command that addresses one or more keys. */
+ * down but the user attempts to execute a command that addresses one or more keys. */
 clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, int argc, int *hashslot, int *error_code) {
     clusterNode *n = NULL;
     robj *firstkey = NULL;
@@ -5643,10 +5671,27 @@ clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, in
      * without redirections or errors in all the cases. */
     if (n == NULL) return myself;
 
-    /* Cluster is globally down but we got keys? We can't serve the request. */
+    /* Cluster is globally down but we got keys? We only serve the request
+     * if it is a read command and when allow_reads_when_down is enabled. */
     if (g_pserver->cluster->state != CLUSTER_OK) {
+        if (!g_pserver->cluster_allow_reads_when_down) {
+            /* The cluster is configured to block commands when the
+             * cluster is down. */
             if (error_code) *error_code = CLUSTER_REDIR_DOWN_STATE;
             return NULL;
+        } else if (!(cmd->flags & CMD_READONLY) && !(cmd->proc == evalCommand)
+                && !(cmd->proc == evalShaCommand))
+        {
+            /* The cluster is configured to allow read only commands
+             * but this command is neither readonly, nor EVAL or
+             * EVALSHA. */
+            if (error_code) *error_code = CLUSTER_REDIR_DOWN_RO_STATE;
+            return NULL;
+        } else {
+            /* Fall through and allow the command to be executed:
+             * this happens when g_pserver->cluster_allow_reads_when_down is
+             * true and the command is a readonly command or EVAL / EVALSHA. */
+        }
     }
 
     /* Return the hashslot by reference. */
@@ -5715,6 +5760,8 @@ void clusterRedirectClient(client *c, clusterNode *n, int hashslot, int error_co
         addReplySds(c,sdsnew("-TRYAGAIN Multiple keys request during rehashing of slot\r\n"));
     } else if (error_code == CLUSTER_REDIR_DOWN_STATE) {
         addReplySds(c,sdsnew("-CLUSTERDOWN The cluster is down\r\n"));
+    } else if (error_code == CLUSTER_REDIR_DOWN_RO_STATE) {
+        addReplySds(c,sdsnew("-CLUSTERDOWN The cluster is down and only accepts read commands\r\n"));
     } else if (error_code == CLUSTER_REDIR_DOWN_UNBOUND) {
         addReplySds(c,sdsnew("-CLUSTERDOWN Hash slot not served\r\n"));
     } else if (error_code == CLUSTER_REDIR_MOVED ||
@@ -5750,7 +5797,10 @@ int clusterRedirectBlockedClientIfNeeded(client *c) {
         dictEntry *de;
         dictIterator *di;
 
-        /* If the cluster is down, unblock the client with the right error. */
+        /* If the cluster is down, unblock the client with the right error.
+         * If the cluster is configured to allow reads on cluster down, we
+         * still want to emit this error since a write will be required
+         * to unblock them which may never come.  */
         if (g_pserver->cluster->state == CLUSTER_FAIL) {
             clusterRedirectClient(c,NULL,0,CLUSTER_REDIR_DOWN_STATE);
             return 1;

src/cluster.h

@@ -17,15 +17,10 @@ extern "C" {
 
 /* The following defines are amount of time, sometimes expressed as
  * multiplicators of the node timeout value (when ending with MULT). */
-#define CLUSTER_DEFAULT_NODE_TIMEOUT 15000
-#define CLUSTER_DEFAULT_SLAVE_VALIDITY 10 /* Slave max data age factor. */
-#define CLUSTER_DEFAULT_REQUIRE_FULL_COVERAGE 1
-#define CLUSTER_DEFAULT_SLAVE_NO_FAILOVER 0 /* Failover by default. */
 #define CLUSTER_FAIL_REPORT_VALIDITY_MULT 2 /* Fail report validity. */
 #define CLUSTER_FAIL_UNDO_TIME_MULT 2 /* Undo fail if master is back. */
 #define CLUSTER_FAIL_UNDO_TIME_ADD 10 /* Some additional time. */
 #define CLUSTER_FAILOVER_DELAY 5 /* Seconds */
-#define CLUSTER_DEFAULT_MIGRATION_BARRIER 1
 #define CLUSTER_MF_TIMEOUT 5000 /* Milliseconds to do a manual failover. */
 #define CLUSTER_MF_PAUSE_MULT 2 /* Master pause manual failover mult. */
 #define CLUSTER_SLAVE_MIGRATION_DELAY 5000 /* Delay for slave migration. */
@@ -38,13 +33,14 @@ extern "C" {
 #define CLUSTER_REDIR_MOVED 4         /* -MOVED redirection required. */
 #define CLUSTER_REDIR_DOWN_STATE 5    /* -CLUSTERDOWN, global state. */
 #define CLUSTER_REDIR_DOWN_UNBOUND 6  /* -CLUSTERDOWN, unbound slot. */
+#define CLUSTER_REDIR_DOWN_RO_STATE 7 /* -CLUSTERDOWN, allow reads. */
 
 struct clusterNode;
 
 /* clusterLink encapsulates everything needed to talk with a remote node. */
 typedef struct clusterLink {
     mstime_t ctime;             /* Link creation time */
-    int fd;                     /* TCP socket file descriptor */
+    connection *conn;           /* Connection to remote node */
     sds sndbuf;                 /* Packet send buffer */
     sds rcvbuf;                 /* Packet reception buffer */
     struct clusterNode *node;   /* Node related to this link if any, or NULL */

src/connection.cpp

@@ -0,0 +1,425 @@
+/*
+ * Copyright (c) 2019, Redis Labs
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "server.h"
+#include "connhelpers.h"
+
+/* The connections module provides a lean abstraction of network connections
+ * to avoid direct socket and async event management across the Redis code base.
+ *
+ * It does NOT provide advanced connection features commonly found in similar
+ * libraries such as complete in/out buffer management, throttling, etc. These
+ * functions remain in networking.c.
+ *
+ * The primary goal is to allow transparent handling of TCP and TLS based
+ * connections. To do so, connections have the following properties:
+ *
+ * 1. A connection may live before its corresponding socket exists.  This
+ *    allows various context and configuration setting to be handled before
+ *    establishing the actual connection.
+ * 2. The caller may register/unregister logical read/write handlers to be
+ *    called when the connection has data to read from/can accept writes.
+ *    These logical handlers may or may not correspond to actual AE events,
+ *    depending on the implementation (for TCP they are; for TLS they aren't).
+ */
+
+extern ConnectionType CT_Socket;
+
+/* When a connection is created we must know its type already, but the
+ * underlying socket may or may not exist:
+ *
+ * - For accepted connections, it exists as we do not model the listen/accept
+ *   part; So caller calls connCreateSocket() followed by connAccept().
+ * - For outgoing connections, the socket is created by the connection module
+ *   itself; So caller calls connCreateSocket() followed by connConnect(),
+ *   which registers a connect callback that fires on connected/error state
+ *   (and after any transport level handshake was done).
+ *
+ * NOTE: An earlier version relied on connections being part of other structs
+ * and not independently allocated. This could lead to further optimizations
+ * like using container_of(), etc.  However it was discontinued in favor of
+ * this approach for these reasons:
+ *
+ * 1. In some cases conns are created/handled outside the context of the
+ * containing struct, in which case it gets a bit awkward to copy them.
+ * 2. Future implementations may wish to allocate arbitrary data for the
+ * connection.
+ * 3. The container_of() approach is anyway risky because connections may
+ * be embedded in different structs, not just client.
+ */
+
+connection *connCreateSocket() {
+    connection *conn = (connection*)zcalloc(sizeof(connection), MALLOC_LOCAL);
+    conn->type = &CT_Socket;
+    conn->fd = -1;
+
+    return conn;
+}
+
+/* Create a new socket-type connection that is already associated with
+ * an accepted connection.
+ *
+ * The socket is not read for I/O until connAccept() was called and
+ * invoked the connection-level accept handler.
+ */
+connection *connCreateAcceptedSocket(int fd) {
+    connection *conn = connCreateSocket();
+    conn->fd = fd;
+    conn->state = CONN_STATE_ACCEPTING;
+    return conn;
+}
+
+static int connSocketConnect(connection *conn, const char *addr, int port, const char *src_addr,
+        ConnectionCallbackFunc connect_handler) {
+    int fd = anetTcpNonBlockBestEffortBindConnect(NULL,addr,port,src_addr);
+    if (fd == -1) {
+        conn->state = CONN_STATE_ERROR;
+        conn->last_errno = errno;
+        return C_ERR;
+    }
+
+    conn->fd = fd;
+    conn->state = CONN_STATE_CONNECTING;
+
+    conn->conn_handler = connect_handler;
+    aeCreateFileEvent(serverTL->el, conn->fd, AE_WRITABLE,
+            conn->type->ae_handler, conn);
+
+    return C_OK;
+}
+
+/* Returns true if a write handler is registered */
+int connHasWriteHandler(connection *conn) {
+    return conn->write_handler != NULL;
+}
+
+/* Returns true if a read handler is registered */
+int connHasReadHandler(connection *conn) {
+    return conn->read_handler != NULL;
+}
+
+/* Associate a private data pointer with the connection */
+void connSetPrivateData(connection *conn, void *data) {
+    conn->private_data = data;
+}
+
+/* Get the associated private data pointer */
+void *connGetPrivateData(connection *conn) {
+    return conn->private_data;
+}
+
+/* ------ Pure socket connections ------- */
+
+/* A very incomplete list of implementation-specific calls.  Much of the above shall
+ * move here as we implement additional connection types.
+ */
+
+/* Close the connection and free resources. */
+static void connSocketClose(connection *conn) {
+    if (conn->fd != -1) {
+        aeDeleteFileEvent(serverTL->el,conn->fd,AE_READABLE);
+        aeDeleteFileEvent(serverTL->el,conn->fd,AE_WRITABLE);
+        close(conn->fd);
+        conn->fd = -1;
+    }
+
+    /* If called from within a handler, schedule the close but
+     * keep the connection until the handler returns.
+     */
+    if (conn->flags & CONN_FLAG_IN_HANDLER) {
+        conn->flags |= CONN_FLAG_CLOSE_SCHEDULED;
+        return;
+    }
+
+    zfree(conn);
+}
+
+static int connSocketWrite(connection *conn, const void *data, size_t data_len) {
+    int ret = write(conn->fd, data, data_len);
+    if (ret < 0 && errno != EAGAIN) {
+        conn->last_errno = errno;
+        conn->state = CONN_STATE_ERROR;
+    }
+
+    return ret;
+}
+
+static int connSocketRead(connection *conn, void *buf, size_t buf_len) {
+    int ret = read(conn->fd, buf, buf_len);
+    if (!ret) {
+        conn->state = CONN_STATE_CLOSED;
+    } else if (ret < 0 && errno != EAGAIN) {
+        conn->last_errno = errno;
+        conn->state = CONN_STATE_ERROR;
+    }
+
+    return ret;
+}
+
+static int connSocketAccept(connection *conn, ConnectionCallbackFunc accept_handler) {
+    if (conn->state != CONN_STATE_ACCEPTING) return C_ERR;
+    conn->state = CONN_STATE_CONNECTED;
+    if (!callHandler(conn, accept_handler)) return C_ERR;
+    return C_OK;
+}
+
+/* Register a write handler, to be called when the connection is writable.
+ * If NULL, the existing handler is removed.
+ *
+ * The barrier flag indicates a write barrier is requested, resulting with
+ * CONN_FLAG_WRITE_BARRIER set. This will ensure that the write handler is
+ * always called before and not after the read handler in a single event
+ * loop.
+ */
+static int connSocketSetWriteHandler(connection *conn, ConnectionCallbackFunc func, int barrier, bool fThreadSafe) {
+    if (func == conn->write_handler) return C_OK;
+
+    conn->write_handler = func;
+    if (barrier)
+        conn->flags |= CONN_FLAG_WRITE_BARRIER;
+    else
+        conn->flags &= ~CONN_FLAG_WRITE_BARRIER;
+
+    int flags = AE_WRITABLE;
+    if (fThreadSafe)
+        flags |= AE_WRITE_THREADSAFE;
+
+    if (!conn->write_handler)
+        aeDeleteFileEvent(serverTL->el,conn->fd,AE_WRITABLE);
+    else
+        if (aeCreateFileEvent(serverTL->el,conn->fd,flags,
+                    conn->type->ae_handler,conn) == AE_ERR) return C_ERR;
+    return C_OK;
+}
+
+/* Register a read handler, to be called when the connection is readable.
+ * If NULL, the existing handler is removed.
+ */
+static int connSocketSetReadHandler(connection *conn, ConnectionCallbackFunc func, bool fThreadSafe) {
+    if (func == conn->read_handler) return C_OK;
+
+    int flags = AE_READABLE;
+    if (fThreadSafe)
+        flags |= AE_READ_THREADSAFE;
+
+    conn->read_handler = func;
+    if (!conn->read_handler)
+        aeDeleteFileEvent(serverTL->el,conn->fd,AE_READABLE);
+    else
+        if (aeCreateFileEvent(serverTL->el,conn->fd,
+                    flags,conn->type->ae_handler,conn) == AE_ERR) return C_ERR;
+    return C_OK;
+}
+
+static const char *connSocketGetLastError(connection *conn) {
+    return strerror(conn->last_errno);
+}
+
+static void connSocketEventHandler(struct aeEventLoop *el, int fd, void *clientData, int mask)
+{
+    UNUSED(el);
+    UNUSED(fd);
+    connection *conn = (connection*)clientData;
+
+    if (conn->state == CONN_STATE_CONNECTING &&
+            (mask & AE_WRITABLE) && conn->conn_handler) {
+
+        if (connGetSocketError(conn)) {
+            conn->last_errno = errno;
+            conn->state = CONN_STATE_ERROR;
+        } else {
+            conn->state = CONN_STATE_CONNECTED;
+        }
+
+        if (!conn->write_handler) aeDeleteFileEvent(serverTL->el,conn->fd,AE_WRITABLE);
+
+        if (!callHandler(conn, conn->conn_handler)) return;
+        conn->conn_handler = NULL;
+    }
+
+    /* Normally we execute the readable event first, and the writable
+     * event later. This is useful as sometimes we may be able
+     * to serve the reply of a query immediately after processing the
+     * query.
+     *
+     * However if WRITE_BARRIER is set in the mask, our application is
+     * asking us to do the reverse: never fire the writable event
+     * after the readable. In such a case, we invert the calls.
+     * This is useful when, for instance, we want to do things
+     * in the beforeSleep() hook, like fsync'ing a file to disk,
+     * before replying to a client. */
+    int invert = conn->flags & CONN_FLAG_WRITE_BARRIER;
+
+    int call_write = (mask & AE_WRITABLE) && conn->write_handler;
+    int call_read = (mask & AE_READABLE) && conn->read_handler;
+
+    /* Handle normal I/O flows */
+    if (!invert && call_read) {
+        if (!callHandler(conn, conn->read_handler)) return;
+    }
+    /* Fire the writable event. */
+    if (call_write) {
+        if (!callHandler(conn, conn->write_handler)) return;
+    }
+    /* If we have to invert the call, fire the readable event now
+     * after the writable one. */
+    if (invert && call_read) {
+        if (!callHandler(conn, conn->read_handler)) return;
+    }
+}
+
+static int connSocketBlockingConnect(connection *conn, const char *addr, int port, long long timeout) {
+    int fd = anetTcpNonBlockConnect(NULL,addr,port);
+    if (fd == -1) {
+        conn->state = CONN_STATE_ERROR;
+        conn->last_errno = errno;
+        return C_ERR;
+    }
+
+    if ((aeWait(fd, AE_WRITABLE, timeout) & AE_WRITABLE) == 0) {
+        conn->state = CONN_STATE_ERROR;
+        conn->last_errno = ETIMEDOUT;
+    }
+
+    conn->fd = fd;
+    conn->state = CONN_STATE_CONNECTED;
+    return C_OK;
+}
+
+/* Connection-based versions of syncio.c functions.
+ * NOTE: This should ideally be refactored out in favor of pure async work.
+ */
+
+static ssize_t connSocketSyncWrite(connection *conn, const char *ptr, ssize_t size, long long timeout) {
+    return syncWrite(conn->fd, ptr, size, timeout);
+}
+
+static ssize_t connSocketSyncRead(connection *conn, char *ptr, ssize_t size, long long timeout) {
+    return syncRead(conn->fd, ptr, size, timeout);
+}
+
+static ssize_t connSocketSyncReadLine(connection *conn, char *ptr, ssize_t size, long long timeout) {
+    return syncReadLine(conn->fd, ptr, size, timeout);
+}
+
+
+ConnectionType CT_Socket = {
+    .ae_handler = connSocketEventHandler,
+    .connect = connSocketConnect,
+    .write = connSocketWrite,
+    .read = connSocketRead,
+    .close = connSocketClose,
+    .accept = connSocketAccept,
+    .set_write_handler = connSocketSetWriteHandler,
+    .set_read_handler = connSocketSetReadHandler,
+    .get_last_error = connSocketGetLastError,
+    .blocking_connect = connSocketBlockingConnect,
+    .sync_write = connSocketSyncWrite,
+    .sync_read = connSocketSyncRead,
+    .sync_readline = connSocketSyncReadLine
+};
+
+
+int connGetSocketError(connection *conn) {
+    int sockerr = 0;
+    socklen_t errlen = sizeof(sockerr);
+
+    if (getsockopt(conn->fd, SOL_SOCKET, SO_ERROR, &sockerr, &errlen) == -1)
+        sockerr = errno;
+    return sockerr;
+}
+
+int connPeerToString(connection *conn, char *ip, size_t ip_len, int *port) {
+    return anetPeerToString(conn ? conn->fd : -1, ip, ip_len, port);
+}
+
+int connFormatPeer(connection *conn, char *buf, size_t buf_len) {
+    return anetFormatPeer(conn ? conn->fd : -1, buf, buf_len);
+}
+
+int connSockName(connection *conn, char *ip, size_t ip_len, int *port) {
+    return anetSockName(conn->fd, ip, ip_len, port);
+}
+
+int connBlock(connection *conn) {
+    if (conn->fd == -1) return C_ERR;
+    return anetBlock(NULL, conn->fd);
+}
+
+int connNonBlock(connection *conn) {
+    if (conn->fd == -1) return C_ERR;
+    return anetNonBlock(NULL, conn->fd);
+}
+
+int connEnableTcpNoDelay(connection *conn) {
+    if (conn->fd == -1) return C_ERR;
+    return anetEnableTcpNoDelay(NULL, conn->fd);
+}
+
+int connDisableTcpNoDelay(connection *conn) {
+    if (conn->fd == -1) return C_ERR;
+    return anetDisableTcpNoDelay(NULL, conn->fd);
+}
+
+int connKeepAlive(connection *conn, int interval) {
+    if (conn->fd == -1) return C_ERR;
+    return anetKeepAlive(NULL, conn->fd, interval);
+}
+
+int connSendTimeout(connection *conn, long long ms) {
+    return anetSendTimeout(NULL, conn->fd, ms);
+}
+
+int connRecvTimeout(connection *conn, long long ms) {
+    return anetRecvTimeout(NULL, conn->fd, ms);
+}
+
+int connGetState(connection *conn) {
+    return conn->state;
+}
+
+void connSetThreadAffinity(connection *conn, int cpu) {
+#ifdef HAVE_SO_INCOMING_CPU
+    if (setsockopt(conn->fd, SOL_SOCKET, SO_INCOMING_CPU, &cpu, sizeof(cpu)) != 0)
+    {
+        serverLog(LL_WARNING, "Failed to set socket affinity");
+    }
+#endif
+}
+
+/* Return a text that describes the connection, suitable for inclusion
+ * in CLIENT LIST and similar outputs.
+ *
+ * For sockets, we always return "fd=<fdnum>" to maintain compatibility.
+ */
+const char *connGetInfo(connection *conn, char *buf, size_t buf_len) {
+    snprintf(buf, buf_len-1, "fd=%i", conn->fd);
+    return buf;
+}
+

src/connection.h

@@ -0,0 +1,223 @@
+
+/*
+ * Copyright (c) 2019, Redis Labs
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __REDIS_CONNECTION_H
+#define __REDIS_CONNECTION_H
+
+#define CONN_INFO_LEN   32
+
+struct aeEventLoop;
+typedef struct connection connection;
+
+typedef enum {
+    CONN_STATE_NONE = 0,
+    CONN_STATE_CONNECTING,
+    CONN_STATE_ACCEPTING,
+    CONN_STATE_CONNECTED,
+    CONN_STATE_CLOSED,
+    CONN_STATE_ERROR
+} ConnectionState;
+
+#define CONN_FLAG_IN_HANDLER        (1<<0)      /* A handler execution is in progress */
+#define CONN_FLAG_CLOSE_SCHEDULED   (1<<1)      /* Closed scheduled by a handler */
+#define CONN_FLAG_WRITE_BARRIER     (1<<2)      /* Write barrier requested */
+#define CONN_FLAG_READ_THREADSAFE        (1<<3)
+#define CONN_FLAG_WRITE_THREADSAFE       (1<<4)
+
+typedef void (*ConnectionCallbackFunc)(struct connection *conn);
+
+typedef struct ConnectionType {
+    void (*ae_handler)(struct aeEventLoop *el, int fd, void *clientData, int mask);
+    int (*connect)(struct connection *conn, const char *addr, int port, const char *source_addr, ConnectionCallbackFunc connect_handler);
+    int (*write)(struct connection *conn, const void *data, size_t data_len);
+    int (*read)(struct connection *conn, void *buf, size_t buf_len);
+    void (*close)(struct connection *conn);
+    int (*accept)(struct connection *conn, ConnectionCallbackFunc accept_handler);
+    int (*set_write_handler)(struct connection *conn, ConnectionCallbackFunc handler, int barrier, bool fThreadSafe);
+    int (*set_read_handler)(struct connection *conn, ConnectionCallbackFunc handler, bool fThreadSafe);
+    const char *(*get_last_error)(struct connection *conn);
+    int (*blocking_connect)(struct connection *conn, const char *addr, int port, long long timeout);
+    ssize_t (*sync_write)(struct connection *conn, const char *ptr, ssize_t size, long long timeout);
+    ssize_t (*sync_read)(struct connection *conn, char *ptr, ssize_t size, long long timeout);
+    ssize_t (*sync_readline)(struct connection *conn, char *ptr, ssize_t size, long long timeout);
+} ConnectionType;
+
+struct connection {
+    ConnectionType *type;
+    ConnectionState state;
+    int flags;
+    int last_errno;
+    void *private_data;
+    ConnectionCallbackFunc conn_handler;
+    ConnectionCallbackFunc write_handler;
+    ConnectionCallbackFunc read_handler;
+    int fd;
+};
+
+/* The connection module does not deal with listening and accepting sockets,
+ * so we assume we have a socket when an incoming connection is created.
+ *
+ * The fd supplied should therefore be associated with an already accept()ed
+ * socket.
+ *
+ * connAccept() may directly call accept_handler(), or return and call it
+ * at a later time. This behavior is a bit awkward but aims to reduce the need
+ * to wait for the next event loop, if no additional handshake is required.
+ */
+
+static inline int connAccept(connection *conn, ConnectionCallbackFunc accept_handler) {
+    return conn->type->accept(conn, accept_handler);
+}
+
+/* Establish a connection.  The connect_handler will be called when the connection
+ * is established, or if an error has occured.
+ *
+ * The connection handler will be responsible to set up any read/write handlers
+ * as needed.
+ *
+ * If C_ERR is returned, the operation failed and the connection handler shall
+ * not be expected.
+ */
+static inline int connConnect(connection *conn, const char *addr, int port, const char *src_addr,
+        ConnectionCallbackFunc connect_handler) {
+    return conn->type->connect(conn, addr, port, src_addr, connect_handler);
+}
+
+/* Blocking connect.
+ *
+ * NOTE: This is implemented in order to simplify the transition to the abstract
+ * connections, but should probably be refactored out of cluster.c and replication.c,
+ * in favor of a pure async implementation.
+ */
+static inline int connBlockingConnect(connection *conn, const char *addr, int port, long long timeout) {
+    return conn->type->blocking_connect(conn, addr, port, timeout);
+}
+
+/* Write to connection, behaves the same as write(2).
+ *
+ * Like write(2), a short write is possible. A -1 return indicates an error.
+ *
+ * The caller should NOT rely on errno. Testing for an EAGAIN-like condition, use
+ * connGetState() to see if the connection state is still CONN_STATE_CONNECTED.
+ */
+static inline int connWrite(connection *conn, const void *data, size_t data_len) {
+    return conn->type->write(conn, data, data_len);
+}
+
+/* Read from the connection, behaves the same as read(2).
+ * 
+ * Like read(2), a short read is possible.  A return value of 0 will indicate the
+ * connection was closed, and -1 will indicate an error.
+ *
+ * The caller should NOT rely on errno. Testing for an EAGAIN-like condition, use
+ * connGetState() to see if the connection state is still CONN_STATE_CONNECTED.
+ */
+static inline int connRead(connection *conn, void *buf, size_t buf_len) {
+    return conn->type->read(conn, buf, buf_len);
+}
+
+/* Register a write handler, to be called when the connection is writable.
+ * If NULL, the existing handler is removed.
+ */
+static inline int connSetWriteHandler(connection *conn, ConnectionCallbackFunc func, bool fThreadSafe = false) {
+    return conn->type->set_write_handler(conn, func, 0, fThreadSafe);
+}
+
+/* Register a read handler, to be called when the connection is readable.
+ * If NULL, the existing handler is removed.
+ */
+static inline int connSetReadHandler(connection *conn, ConnectionCallbackFunc func, bool fThreadSafe = false) {
+    return conn->type->set_read_handler(conn, func, fThreadSafe);
+}
+
+/* Set a write handler, and possibly enable a write barrier, this flag is
+ * cleared when write handler is changed or removed.
+ * With barroer enabled, we never fire the event if the read handler already
+ * fired in the same event loop iteration. Useful when you want to persist
+ * things to disk before sending replies, and want to do that in a group fashion. */
+static inline int connSetWriteHandlerWithBarrier(connection *conn, ConnectionCallbackFunc func, int barrier, bool fThreadSafe = false) {
+    return conn->type->set_write_handler(conn, func, barrier, fThreadSafe);
+}
+
+static inline void connClose(connection *conn) {
+    conn->type->close(conn);
+}
+
+/* Returns the last error encountered by the connection, as a string.  If no error,
+ * a NULL is returned.
+ */
+static inline const char *connGetLastError(connection *conn) {
+    return conn->type->get_last_error(conn);
+}
+
+static inline ssize_t connSyncWrite(connection *conn, const char *ptr, ssize_t size, long long timeout) {
+    return conn->type->sync_write(conn, ptr, size, timeout);
+}
+
+static inline ssize_t connSyncRead(connection *conn, char *ptr, ssize_t size, long long timeout) {
+    return conn->type->sync_read(conn, ptr, size, timeout);
+}
+
+static inline ssize_t connSyncReadLine(connection *conn, char *ptr, ssize_t size, long long timeout) {
+    return conn->type->sync_readline(conn, ptr, size, timeout);
+}
+
+connection *connCreateSocket();
+connection *connCreateAcceptedSocket(int fd);
+
+connection *connCreateTLS();
+connection *connCreateAcceptedTLS(int fd, int require_auth);
+
+void connSetPrivateData(connection *conn, void *data);
+void *connGetPrivateData(connection *conn);
+int connGetState(connection *conn);
+int connHasWriteHandler(connection *conn);
+int connHasReadHandler(connection *conn);
+int connGetSocketError(connection *conn);
+void connSetThreadAffinity(connection *conn, int cpu);
+
+/* anet-style wrappers to conns */
+int connBlock(connection *conn);
+int connNonBlock(connection *conn);
+int connEnableTcpNoDelay(connection *conn);
+int connDisableTcpNoDelay(connection *conn);
+int connKeepAlive(connection *conn, int interval);
+int connSendTimeout(connection *conn, long long ms);
+int connRecvTimeout(connection *conn, long long ms);
+int connPeerToString(connection *conn, char *ip, size_t ip_len, int *port);
+int connFormatPeer(connection *conn, char *buf, size_t buf_len);
+int connSockName(connection *conn, char *ip, size_t ip_len, int *port);
+const char *connGetInfo(connection *conn, char *buf, size_t buf_len);
+
+/* Helpers for tls special considerations */
+int tlsHasPendingData();
+void tlsProcessPendingData();
+
+#endif  /* __REDIS_CONNECTION_H */

src/connhelpers.h

@@ -0,0 +1,85 @@
+
+/*
+ * Copyright (c) 2019, Redis Labs
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __REDIS_CONNHELPERS_H
+#define __REDIS_CONNHELPERS_H
+
+#include "connection.h"
+
+/* These are helper functions that are common to different connection
+ * implementations (currently sockets in connection.c and TLS in tls.c).
+ *
+ * Currently helpers implement the mechanisms for invoking connection
+ * handlers, tracking in-handler states and dealing with deferred
+ * destruction (if invoked by a handler).
+ */
+
+/* Called whenever a handler is invoked on a connection and sets the
+ * CONN_FLAG_IN_HANDLER flag to indicate we're in a handler context.
+ *
+ * An attempt to close a connection while CONN_FLAG_IN_HANDLER is
+ * set will result with deferred close, i.e. setting the CONN_FLAG_CLOSE_SCHEDULED
+ * instead of destructing it.
+ */
+static inline void enterHandler(connection *conn) {
+    conn->flags |= CONN_FLAG_IN_HANDLER;
+}
+
+/* Called whenever a handler returns. This unsets the CONN_FLAG_IN_HANDLER
+ * flag and performs actual close/destruction if a deferred close was
+ * scheduled by the handler.
+ */
+static inline int exitHandler(connection *conn) {
+    conn->flags &= ~CONN_FLAG_IN_HANDLER;
+    if (conn->flags & CONN_FLAG_CLOSE_SCHEDULED) {
+        connClose(conn);
+        return 0;
+    }
+    return 1;
+}
+
+/* Helper for connection implementations to call handlers:
+ * 1. Mark the handler in use.
+ * 2. Execute the handler (if set).
+ * 3. Mark the handler as NOT in use and perform deferred close if was
+ * requested by the handler at any time.
+ */
+static inline int callHandler(connection *conn, ConnectionCallbackFunc handler) {
+    conn->flags |= CONN_FLAG_IN_HANDLER;
+    if (handler) handler(conn);
+    conn->flags &= ~CONN_FLAG_IN_HANDLER;
+    if (conn->flags & CONN_FLAG_CLOSE_SCHEDULED) {
+        connClose(conn);
+        return 0;
+    }
+    return 1;
+}
+
+#endif  /* __REDIS_CONNHELPERS_H */

src/cron.cpp

@@ -77,7 +77,7 @@ void executeCronJobExpireHook(const char *key, robj *o)
     serverAssert(o->type == OBJ_CRON);
     cronjob *job = (cronjob*)ptrFromObj(o);
     
-    client *cFake = createClient(-1, IDX_EVENT_LOOP_MAIN);
+    client *cFake = createClient(nullptr, IDX_EVENT_LOOP_MAIN);
     cFake->lock.lock();
     cFake->authenticated = 1;
     cFake->puser = nullptr;

src/db.cpp

@@ -77,10 +77,7 @@ static robj *lookupKey(redisDb *db, robj *key, int flags) {
         /* Update the access time for the ageing algorithm.
          * Don't do it if we have a saving child, as this will trigger
          * a copy on write madness. */
-        if (g_pserver->rdb_child_pid == -1 &&
+        if (!hasActiveChildProcess() && !(flags & LOOKUP_NOTOUCH)){
-            g_pserver->aof_child_pid == -1 &&
-            !(flags & LOOKUP_NOTOUCH))
-        {
             if (g_pserver->maxmemory_policy & MAXMEMORY_FLAG_LFU) {
                 updateLFU(val);
             } else {
@@ -176,13 +173,17 @@ robj_roptr lookupKeyRead(redisDb *db, robj *key) {
  *
  * Returns the linked value object if the key exists or NULL if the key
  * does not exist in the specified DB. */
-robj *lookupKeyWrite(redisDb *db, robj *key) {
+robj *lookupKeyWriteWithFlags(redisDb *db, robj *key, int flags) {
-    robj *o = lookupKey(db,key,LOOKUP_UPDATEMVCC);
+    robj *o = lookupKey(db,key,flags|LOOKUP_UPDATEMVCC);
     if (expireIfNeeded(db,key))
-        o = NULL;
+        o = nullptr;
     return o;
 }
 
+robj *lookupKeyWrite(redisDb *db, robj *key) {
+    return lookupKeyWriteWithFlags(db, key, LOOKUP_NONE);
+}
+
 robj_roptr lookupKeyReadOrReply(client *c, robj *key, robj *reply) {
     robj_roptr o = lookupKeyRead(c->db, key);
     if (!o) addReply(c,reply);
@@ -302,20 +303,28 @@ int dbMerge(redisDb *db, robj *key, robj *val, int fReplace)
  *
  * 1) The ref count of the value object is incremented.
  * 2) clients WATCHing for the destination key notified.
- * 3) The expire time of the key is reset (the key is made persistent).
+ * 3) The expire time of the key is reset (the key is made persistent),
+ *    unless 'keepttl' is true.
  *
  * All the new keys in the database should be created via this interface. */
-void setKey(redisDb *db, robj *key, robj *val) {
+void genericSetKey(redisDb *db, robj *key, robj *val, int keepttl) {
     dictEntry *de = dictFind(db->pdict, ptrFromObj(key));
     if (de == NULL) {
         dbAdd(db,key,val);
     } else {
-        dbOverwriteCore(db,de,key,val,!!g_pserver->fActiveReplica,true);
+        dbOverwriteCore(db,de,key,val,!!g_pserver->fActiveReplica,!keepttl);
     }
     incrRefCount(val);
     signalModifiedKey(db,key);
 }
 
+/* Common case for genericSetKey() where the TTL is not retained. */
+void setKey(redisDb *db, robj *key, robj *val) {
+    genericSetKey(db,key,val,0);
+}
+
+/* Return true if the specified key exists in the specified database.
+ * LRU/LFU info is not updated in any way. */
 int dbExists(redisDb *db, robj *key) {
     return dictFind(db->pdict,ptrFromObj(key)) != NULL;
 }
@@ -441,7 +450,7 @@ robj *dbUnshareStringValue(redisDb *db, robj *key, robj *o) {
  * On success the fuction returns the number of keys removed from the
  * database(s). Otherwise -1 is returned in the specific case the
  * DB number is out of range, and errno is set to EINVAL. */
-long long emptyDb(int dbnum, int flags, void(callback)(void*)) {
+long long emptyDbGeneric(redisDb *dbarray, int dbnum, int flags, void(callback)(void*)) {
     int async = (flags & EMPTYDB_ASYNC);
     long long removed = 0;
 
@@ -450,6 +459,17 @@ long long emptyDb(int dbnum, int flags, void(callback)(void*)) {
         return -1;
     }
 
+    /* Fire the flushdb modules event. */
+    RedisModuleFlushInfoV1 fi = {REDISMODULE_FLUSHINFO_VERSION,!async,dbnum};
+    moduleFireServerEvent(REDISMODULE_EVENT_FLUSHDB,
+                          REDISMODULE_SUBEVENT_FLUSHDB_START,
+                          &fi);
+
+    /* Make sure the WATCHed keys are affected by the FLUSH* commands.
+     * Note that we need to call the function while the keys are still
+     * there. */
+    signalFlushedDb(dbnum);
+
     int startdb, enddb;
     if (dbnum == -1) {
         startdb = 0;
@@ -459,14 +479,12 @@ long long emptyDb(int dbnum, int flags, void(callback)(void*)) {
     }
 
     for (int j = startdb; j <= enddb; j++) {
-        removed += dictSize(g_pserver->db[j].pdict);
+        removed += dictSize(dbarray[j].pdict);
         if (async) {
-            emptyDbAsync(&g_pserver->db[j]);
+            emptyDbAsync(&dbarray[j]);
         } else {
-            dictEmpty(g_pserver->db[j].pdict,callback);
+            dictEmpty(dbarray[j].pdict,callback);
-            delete g_pserver->db[j].setexpire;
+            dbarray[j].setexpire->clear();
-            g_pserver->db[j].setexpire = new (MALLOC_LOCAL) expireset();
-            g_pserver->db[j].expireitr = g_pserver->db[j].setexpire->end();
         }
     }
     if (g_pserver->cluster_enabled) {
@@ -477,9 +495,20 @@ long long emptyDb(int dbnum, int flags, void(callback)(void*)) {
         }
     }
     if (dbnum == -1) flushSlaveKeysWithExpireList();
+
+    /* Also fire the end event. Note that this event will fire almost
+     * immediately after the start event if the flush is asynchronous. */
+    moduleFireServerEvent(REDISMODULE_EVENT_FLUSHDB,
+                          REDISMODULE_SUBEVENT_FLUSHDB_END,
+                          &fi);
+
     return removed;
 }
 
+long long emptyDb(int dbnum, int flags, void(callback)(void*)) {
+    return emptyDbGeneric(g_pserver->db, dbnum, flags, callback);
+}
+
 int selectDb(client *c, int id) {
     if (id < 0 || id >= cserver.dbnum)
         return C_ERR;
@@ -487,6 +516,15 @@ int selectDb(client *c, int id) {
     return C_OK;
 }
 
+long long dbTotalServerKeyCount() {
+    long long total = 0;
+    int j;
+    for (j = 0; j < cserver.dbnum; j++) {
+        total += dictSize(g_pserver->db[j].pdict);
+    }
+    return total;
+}
+
 /*-----------------------------------------------------------------------------
  * Hooks for key space changes.
  *
@@ -498,11 +536,12 @@ int selectDb(client *c, int id) {
 
 void signalModifiedKey(redisDb *db, robj *key) {
     touchWatchedKey(db,key);
-    if (g_pserver->tracking_clients) trackingInvalidateKey(key);
+    trackingInvalidateKey(key);
 }
 
 void signalFlushedDb(int dbid) {
     touchWatchedKeysOnFlush(dbid);
+    trackingInvalidateKeysOnFlush(dbid);
 }
 
 /*-----------------------------------------------------------------------------
@@ -531,28 +570,9 @@ int getFlushCommandFlags(client *c, int *flags) {
     return C_OK;
 }
 
-/* FLUSHDB [ASYNC]
+/* Flushes the whole server data set. */
- *
+void flushAllDataAndResetRDB(int flags) {
- * Flushes the currently SELECTed Redis DB. */
-void flushdbCommand(client *c) {
-    int flags;
-
-    if (getFlushCommandFlags(c,&flags) == C_ERR) return;
-    signalFlushedDb(c->db->id);
-    g_pserver->dirty += emptyDb(c->db->id,flags,NULL);
-    addReply(c,shared.ok);
-}
-
-/* FLUSHALL [ASYNC]
- *
- * Flushes the whole server data set. */
-void flushallCommand(client *c) {
-    int flags;
-
-    if (getFlushCommandFlags(c,&flags) == C_ERR) return;
-    signalFlushedDb(-1);
     g_pserver->dirty += emptyDb(-1,flags,NULL);
-    addReply(c,shared.ok);
     if (g_pserver->rdb_child_pid != -1) killRDBChild();
     if (g_pserver->saveparamslen > 0) {
         /* Normally rdbSave() will reset dirty, but we don't want this here
@@ -564,6 +584,41 @@ void flushallCommand(client *c) {
         g_pserver->dirty = saved_dirty;
     }
     g_pserver->dirty++;
+#if defined(USE_JEMALLOC)
+    /* jemalloc 5 doesn't release pages back to the OS when there's no traffic.
+     * for large databases, flushdb blocks for long anyway, so a bit more won't
+     * harm and this way the flush and purge will be synchroneus. */
+    if (!(flags & EMPTYDB_ASYNC))
+        jemalloc_purge();
+#endif
+}
+
+/* FLUSHDB [ASYNC]
+ *
+ * Flushes the currently SELECTed Redis DB. */
+void flushdbCommand(client *c) {
+    int flags;
+
+    if (getFlushCommandFlags(c,&flags) == C_ERR) return;
+    g_pserver->dirty += emptyDb(c->db->id,flags,NULL);
+    addReply(c,shared.ok);
+#if defined(USE_JEMALLOC)
+    /* jemalloc 5 doesn't release pages back to the OS when there's no traffic.
+     * for large databases, flushdb blocks for long anyway, so a bit more won't
+     * harm and this way the flush and purge will be synchroneus. */
+    if (!(flags & EMPTYDB_ASYNC))
+        jemalloc_purge();
+#endif
+}
+
+/* FLUSHALL [ASYNC]
+ *
+ * Flushes the whole server data set. */
+void flushallCommand(client *c) {
+    int flags;
+    if (getFlushCommandFlags(c,&flags) == C_ERR) return;
+    flushAllDataAndResetRDB(flags);
+    addReply(c,shared.ok);
 }
 
 /* This command implements DEL and LAZYDEL. */
@@ -644,7 +699,7 @@ void keysCommand(client *c) {
     void *replylen = addReplyDeferredLen(c);
 
     di = dictGetSafeIterator(c->db->pdict);
-    allkeys = (pattern[0] == '*' && pattern[1] == '\0');
+    allkeys = (pattern[0] == '*' && plen == 1);
     while((de = dictNext(di)) != NULL) {
         sds key = (sds)dictGetKey(de);
         robj *keyobj;
@@ -1100,6 +1155,13 @@ void moveCommand(client *c) {
     dbAdd(dst,c->argv[1],o);
     if (spexpire != nullptr) setExpire(c,dst,c->argv[1],std::move(*spexpire));
 
+    signalModifiedKey(src,c->argv[1]);
+    signalModifiedKey(dst,c->argv[1]);
+    notifyKeyspaceEvent(NOTIFY_GENERIC,
+                "move_from",c->argv[1],src->id);
+    notifyKeyspaceEvent(NOTIFY_GENERIC,
+                "move_to",c->argv[1],dst->id);
+
     addReply(c,shared.cone);
 }
 
@@ -1129,7 +1191,7 @@ void scanDatabaseForReadyLists(redisDb *db) {
  *
  * Returns C_ERR if at least one of the DB ids are out of range, otherwise
  * C_OK is returned. */
-int dbSwapDatabases(int id1, int id2) {
+int dbSwapDatabases(long id1, long id2) {
     if (id1 < 0 || id1 >= cserver.dbnum ||
         id2 < 0 || id2 >= cserver.dbnum) return C_ERR;
     if (id1 == id2) return C_OK;
@@ -1385,6 +1447,7 @@ void propagateExpire(redisDb *db, robj *key, int lazy) {
 /* Check if the key is expired. Note, this does not check subexpires */
 int keyIsExpired(redisDb *db, robj *key) {
     expireEntry *pexpire = getExpire(db,key);
+    mstime_t now;
 
     if (pexpire == nullptr) return 0; /* No expire for this key */
 
@@ -1409,8 +1472,26 @@ int keyIsExpired(redisDb *db, robj *key) {
      * only the first time it is accessed and not in the middle of the
      * script execution, making propagation to slaves / AOF consistent.
      * See issue #1525 on Github for more information. */
-    mstime_t now = g_pserver->lua_caller ? g_pserver->lua_time_start : mstime();
+    if (g_pserver->lua_caller) {
+        now = g_pserver->lua_time_start;
+    }
+    /* If we are in the middle of a command execution, we still want to use
+     * a reference time that does not change: in that case we just use the
+     * cached time, that we update before each call in the call() function.
+     * This way we avoid that commands such as RPOPLPUSH or similar, that
+     * may re-open the same key multiple times, can invalidate an already
+     * open object in a next call, if the next call will see the key expired,
+     * while the first did not. */
+    else if (serverTL->fixed_time_expire > 0) {
+        now = g_pserver->mstime;
+    }
+    /* For the other cases, we want to use the most fresh time we have. */
+    else {
+        now = mstime();
+    }
 
+    /* The key expired if the current (virtual or real) time is greater
+     * than the expire time of the key. */
     return now > when;
 }
 

src/debug.cpp

@@ -306,6 +306,56 @@ void computeDatasetDigest(unsigned char *final) {
     }
 }
 
+#ifdef USE_JEMALLOC
+void mallctl_int(client *c, robj **argv, int argc) {
+    int ret;
+    /* start with the biggest size (int64), and if that fails, try smaller sizes (int32, bool) */
+    int64_t old = 0, val;
+    if (argc > 1) {
+        long long ll;
+        if (getLongLongFromObjectOrReply(c, argv[1], &ll, NULL) != C_OK)
+            return;
+        val = ll;
+    }
+    size_t sz = sizeof(old);
+    while (sz > 0) {
+        if ((ret=je_mallctl(szFromObj(argv[0]), &old, &sz, argc > 1? &val: NULL, argc > 1?sz: 0))) {
+            if (ret==EINVAL) {
+                /* size might be wrong, try a smaller one */
+                sz /= 2;
+#if BYTE_ORDER == BIG_ENDIAN
+                val <<= 8*sz;
+#endif
+                continue;
+            }
+            addReplyErrorFormat(c,"%s", strerror(ret));
+            return;
+        } else {
+#if BYTE_ORDER == BIG_ENDIAN
+            old >>= 64 - 8*sz;
+#endif
+            addReplyLongLong(c, old);
+            return;
+        }
+    }
+    addReplyErrorFormat(c,"%s", strerror(EINVAL));
+}
+
+void mallctl_string(client *c, robj **argv, int argc) {
+    int ret;
+    char *old;
+    size_t sz = sizeof(old);
+    /* for strings, it seems we need to first get the old value, before overriding it. */
+    if ((ret=je_mallctl(szFromObj(argv[0]), &old, &sz, NULL, 0))) {
+        addReplyErrorFormat(c,"%s", strerror(ret));
+        return;
+    }
+    addReplyBulkCString(c, old);
+    if(argc > 1)
+        je_mallctl(szFromObj(argv[0]), NULL, 0, &argv[1]->m_ptr, sizeof(char*));
+}
+#endif
+
 void debugCommand(client *c) {
     if (c->argc == 2 && !strcasecmp(szFromObj(c->argv[1]),"help")) {
         const char *help[] = {
@@ -328,10 +378,15 @@ void debugCommand(client *c) {
 "SDSLEN <key> -- Show low level SDS string info representing key and value.",
 "SEGFAULT -- Crash the server with sigsegv.",
 "SET-ACTIVE-EXPIRE <0|1> -- Setting it to 0 disables expiring keys in background when they are not accessed (otherwise the Redis behavior). Setting it to 1 reenables back the default.",
+"AOF-FLUSH-SLEEP <microsec> -- Server will sleep before flushing the AOF, this is used for testing",
 "SLEEP <seconds> -- Stop the server for <seconds>. Decimals allowed.",
 "STRUCTSIZE -- Return the size of different Redis core C structures.",
 "ZIPLIST <key> -- Show low level info about the ziplist encoding.",
 "STRINGMATCH-TEST -- Run a fuzz tester against the stringmatchlen() function.",
+#ifdef USE_JEMALLOC
+"MALLCTL <key> [<val>] -- Get or set a malloc tunning integer.",
+"MALLCTL-STR <key> [<val>] -- Get or set a malloc tunning string.",
+#endif
 NULL
         };
         addReplyHelp(c, help);
@@ -371,8 +426,7 @@ NULL
         }
         emptyDb(-1,EMPTYDB_NO_FLAGS,NULL);
         protectClient(c);
-        rdbSaveInfo rsiDft = RDB_SAVE_INFO_INIT;
+        int ret = rdbLoadFile(g_pserver->rdb_filename,NULL,RDBFLAGS_NONE);
-        int ret = rdbLoad(&rsiDft);
         unprotectClient(c);
         if (ret != C_OK) {
             addReplyError(c,"Error trying to load the RDB dump");
@@ -605,6 +659,11 @@ NULL
     {
         g_pserver->active_expire_enabled = atoi(szFromObj(c->argv[2]));
         addReply(c,shared.ok);
+    } else if (!strcasecmp(szFromObj(c->argv[1]),"aof-flush-sleep") &&
+               c->argc == 3)
+    {
+        g_pserver->aof_flush_sleep = atoi(szFromObj(c->argv[2]));
+        addReply(c,shared.ok);
     } else if (!strcasecmp(szFromObj(c->argv[1]),"lua-always-replicate-commands") &&
                c->argc == 3)
     {
@@ -648,7 +707,8 @@ NULL
         g_pserver->db[dbid].setexpire->getstats(buf, sizeof(buf));
         stats = sdscat(stats, buf);
 
-        addReplyBulkSds(c,stats);
+        addReplyVerbatim(c,stats,sdslen(stats),"txt");
+        sdsfree(stats);
     } else if (!strcasecmp(szFromObj(c->argv[1]),"htstats-key") && c->argc == 3) {
         robj *o;
         dict *ht = NULL;
@@ -675,7 +735,7 @@ NULL
         } else {
             char buf[4096];
             dictGetStats(buf,sizeof(buf),ht);
-            addReplyBulkCString(c,buf);
+            addReplyVerbatim(c,buf,strlen(buf),"txt");
         }
     } else if (!strcasecmp(szFromObj(c->argv[1]),"change-repl-id") && c->argc == 2) {
         serverLog(LL_WARNING,"Changing replication IDs after receiving DEBUG change-repl-id");
@@ -688,6 +748,14 @@ NULL
     } else if (!strcasecmp(szFromObj(c->argv[1]), "force-master") && c->argc == 2) {
         c->flags |= CLIENT_MASTER | CLIENT_MASTER_FORCE_REPLY;
         addReply(c, shared.ok);
+#ifdef USE_JEMALLOC
+    } else if(!strcasecmp(szFromObj(c->argv[1]),"mallctl") && c->argc >= 3) {
+        mallctl_int(c, c->argv+2, c->argc-2);
+        return;
+    } else if(!strcasecmp(szFromObj(c->argv[1]),"mallctl-str") && c->argc >= 3) {
+        mallctl_string(c, c->argv+2, c->argc-2);
+        return;
+#endif
     } else {
         addReplySubcommandSyntaxError(c);
         return;
@@ -711,11 +779,12 @@ void _serverAssert(const char *estr, const char *file, int line) {
 
 void _serverAssertPrintClientInfo(const client *c) {
     int j;
+    char conninfo[CONN_INFO_LEN];
 
     bugReportStart();
     serverLog(LL_WARNING,"=== ASSERTION FAILED CLIENT CONTEXT ===");
-    serverLog(LL_WARNING,"client->flags = %llu", static_cast<unsigned long long>(c->flags));
+    serverLog(LL_WARNING,"client->flags = %llu", (unsigned long long) c->flags);
-    serverLog(LL_WARNING,"client->fd = %d", c->fd);
+    serverLog(LL_WARNING,"client->conn = %s", connGetInfo(c->conn, conninfo, sizeof(conninfo)));
     serverLog(LL_WARNING,"client->argc = %d", c->argc);
     for (j=0; j < c->argc; j++) {
         char buf[128];
@@ -1122,6 +1191,33 @@ void logRegisters(ucontext_t *uc) {
         (unsigned long) uc->uc_mcontext.mc_cs
     );
     logStackContent((void**)uc->uc_mcontext.mc_rsp);
+#elif defined(__aarch64__) /* Linux AArch64 */
+    serverLog(LL_WARNING,
+	      "\n"
+	      "X18:%016lx X19:%016lx\nX20:%016lx X21:%016lx\n"
+	      "X22:%016lx X23:%016lx\nX24:%016lx X25:%016lx\n"
+	      "X26:%016lx X27:%016lx\nX28:%016lx X29:%016lx\n"
+	      "X30:%016lx\n"
+	      "pc:%016lx sp:%016lx\npstate:%016lx fault_address:%016lx\n",
+	      (unsigned long) uc->uc_mcontext.regs[18],
+	      (unsigned long) uc->uc_mcontext.regs[19],
+	      (unsigned long) uc->uc_mcontext.regs[20],
+	      (unsigned long) uc->uc_mcontext.regs[21],
+	      (unsigned long) uc->uc_mcontext.regs[22],
+	      (unsigned long) uc->uc_mcontext.regs[23],
+	      (unsigned long) uc->uc_mcontext.regs[24],
+	      (unsigned long) uc->uc_mcontext.regs[25],
+	      (unsigned long) uc->uc_mcontext.regs[26],
+	      (unsigned long) uc->uc_mcontext.regs[27],
+	      (unsigned long) uc->uc_mcontext.regs[28],
+	      (unsigned long) uc->uc_mcontext.regs[29],
+	      (unsigned long) uc->uc_mcontext.regs[30],
+	      (unsigned long) uc->uc_mcontext.pc,
+	      (unsigned long) uc->uc_mcontext.sp,
+	      (unsigned long) uc->uc_mcontext.pstate,
+	      (unsigned long) uc->uc_mcontext.fault_address
+		      );
+	      logStackContent((void**)uc->uc_mcontext.sp);
 #else
     serverLog(LL_WARNING,
         "  Dumping of registers not supported for this OS/arch");
@@ -1405,6 +1501,12 @@ void sigsegvHandler(int sig, siginfo_t *info, void *secret) {
     /* Log dump of processor registers */
     logRegisters(uc);
 
+    /* Log Modules INFO */
+    serverLogRaw(LL_WARNING|LL_RAW, "\n------ MODULES INFO OUTPUT ------\n");
+    infostring = modulesCollectInfo(sdsempty(), NULL, 1, 0);
+    serverLogRaw(LL_WARNING|LL_RAW, infostring);
+    sdsfree(infostring);
+
 #if defined(HAVE_PROC_MAPS)
     /* Test memory */
     serverLogRaw(LL_WARNING|LL_RAW, "\n------ FAST MEMORY TEST ------\n");

src/defrag.cpp

@@ -373,7 +373,7 @@ long activeDefragSdsListAndDict(list *l, dict *d, int dict_val_type) {
             if ((newele = activeDefragStringOb(ele, &defragged)))
                 de->v.val = newele, defragged++;
         } else if (dict_val_type == DEFRAG_SDS_DICT_VAL_VOID_PTR) {
-            void *newptr, *ptr = ln->value;
+            void *newptr, *ptr = dictGetVal(de);
             if ((newptr = activeDefragAlloc(ptr)))
                 ln->value = newptr, defragged++;
         }
@@ -935,10 +935,12 @@ int defragLaterItem(dictEntry *de, unsigned long *cursor, long long endtime) {
     return 0;
 }
 
+/* static variables serving defragLaterStep to continue scanning a key from were we stopped last time. */
+static sds defrag_later_current_key = NULL;
+static unsigned long defrag_later_cursor = 0;
+
 /* returns 0 if no more work needs to be been done, and 1 if time is up and more work is needed. */
 int defragLaterStep(redisDb *db, long long endtime) {
-    static sds current_key = NULL;
-    static unsigned long cursor = 0;
     unsigned int iterations = 0;
     unsigned long long prev_defragged = g_pserver->stat_active_defrag_hits;
     unsigned long long prev_scanned = g_pserver->stat_active_defrag_scanned;
@@ -946,16 +948,15 @@ int defragLaterStep(redisDb *db, long long endtime) {
 
     do {
         /* if we're not continuing a scan from the last call or loop, start a new one */
-        if (!cursor) {
+        if (!defrag_later_cursor) {
             listNode *head = listFirst(db->defrag_later);
 
             /* Move on to next key */
-            if (current_key) {
+            if (defrag_later_current_key) {
-                serverAssert(current_key == head->value);
+                serverAssert(defrag_later_current_key == head->value);
-                sdsfree((sds)head->value);
                 listDelNode(db->defrag_later, head);
-                cursor = 0;
+                defrag_later_cursor = 0;
-                current_key = NULL;
+                defrag_later_current_key = NULL;
             }
 
             /* stop if we reached the last one. */
@@ -964,21 +965,21 @@ int defragLaterStep(redisDb *db, long long endtime) {
                 return 0;
 
             /* start a new key */
-            current_key = (sds)head->value;
+            defrag_later_current_key = (sds)head->value;
-            cursor = 0;
+            defrag_later_cursor = 0;
         }
 
         /* each time we enter this function we need to fetch the key from the dict again (if it still exists) */
-        dictEntry *de = dictFind(db->pdict, current_key);
+        dictEntry *de = dictFind(db->pdict, defrag_later_current_key);
         key_defragged = g_pserver->stat_active_defrag_hits;
         do {
             int quit = 0;
-            if (defragLaterItem(de, &cursor, endtime))
+            if (defragLaterItem(de, &defrag_later_cursor, endtime))
                 quit = 1; /* time is up, we didn't finish all the work */
 
             /* Don't start a new BIG key in this loop, this is because the
              * next key can be a list, and scanLaterList must be done in once cycle */
-            if (!cursor)
+            if (!defrag_later_cursor)
                 quit = 1;
 
             /* Once in 16 scan iterations, 512 pointer reallocations, or 64 fields
@@ -998,7 +999,7 @@ int defragLaterStep(redisDb *db, long long endtime) {
                 prev_defragged = g_pserver->stat_active_defrag_hits;
                 prev_scanned = g_pserver->stat_active_defrag_scanned;
             }
-        } while(cursor);
+        } while(defrag_later_cursor);
         if(key_defragged != g_pserver->stat_active_defrag_hits)
             g_pserver->stat_active_defrag_key_hits++;
         else
@@ -1055,7 +1056,22 @@ void activeDefragCycle(void) {
     mstime_t latency;
     int quit = 0;
 
-    if (g_pserver->aof_child_pid!=-1 || g_pserver->rdb_child_pid!=-1)
+    if (!cserver.active_defrag_enabled) {
+        if (g_pserver->active_defrag_running) {
+            /* if active defrag was disabled mid-run, start from fresh next time. */
+            g_pserver->active_defrag_running = 0;
+            if (db)
+                listEmpty(db->defrag_later);
+            defrag_later_current_key = NULL;
+            defrag_later_cursor = 0;
+            current_db = -1;
+            cursor = 0;
+            db = NULL;
+        }
+        return;
+    }
+
+    if (hasActiveChildProcess())
         return; /* Defragging memory while there's a fork will just do damage. */
 
     /* Once a second, check if we the fragmentation justfies starting a scan

src/evict.cpp

@@ -460,6 +460,7 @@ int getMaxmemoryState(size_t *total, size_t *logical, size_t *tofree, float *lev
  * was freed to return back under the limit, the function returns C_ERR. */
 int freeMemoryIfNeeded(void) {
     serverAssert(GlobalLocksAcquired());
+    int keys_freed = 0;
     /* By default replicas should ignore maxmemory
      * and just be masters exact copies. */
     if (listLength(g_pserver->masters) && g_pserver->repl_slave_ignore_maxmemory) return C_OK;
@@ -483,7 +484,7 @@ int freeMemoryIfNeeded(void) {
 
     latencyStartMonitor(latency);
     while (mem_freed < mem_tofree) {
-        int j, k, i, keys_freed = 0;
+        int j, k, i;
         static unsigned int next_db = 0;
         sds bestkey = NULL;
         int bestdbid;
@@ -627,9 +628,7 @@ int freeMemoryIfNeeded(void) {
                     mem_freed = mem_tofree;
                 }
             }
-        }
+        } else {
-
-        if (!keys_freed) {
             latencyEndMonitor(latency);
             latencyAddSampleIfNeeded("eviction-cycle",latency);
             goto cant_free; /* nothing to free... */

src/geo.cpp

@@ -465,8 +465,8 @@ void georadiusGeneric(client *c, int flags) {
     int storedist = 0; /* 0 for STORE, 1 for STOREDIST. */
 
     /* Look up the requested zset */
-    robj_roptr zobj;
+    robj_roptr zobj = nullptr;
-    if ((zobj = lookupKeyReadOrReply(c, key, shared.null[c->resp])) == nullptr ||
+    if ((zobj = lookupKeyReadOrReply(c, key, shared.emptyarray)) == nullptr ||
         checkType(c, zobj, OBJ_ZSET)) {
         return;
     }
@@ -566,7 +566,7 @@ void georadiusGeneric(client *c, int flags) {
 
     /* If no matching results, the user gets an empty reply. */
     if (ga->used == 0 && storekey == NULL) {
-        addReplyNull(c);
+        addReply(c,shared.emptyarray);
         geoArrayFree(ga);
         return;
     }
@@ -737,7 +737,15 @@ void geohashCommand(client *c) {
             char buf[12];
             int i;
             for (i = 0; i < 11; i++) {
-                int idx = (hash.bits >> (52-((i+1)*5))) & 0x1f;
+                int idx;
+                if (i == 10) {
+                    /* We have just 52 bits, but the API used to output
+                     * an 11 bytes geohash. For compatibility we assume
+                     * zero. */
+                    idx = 0;
+                } else {
+                    idx = (hash.bits >> (52-((i+1)*5))) & 0x1f;
+                }
                 buf[i] = geoalphabet[idx];
             }
             buf[11] = '\0';

src/hyperloglog.cpp

@@ -709,8 +709,7 @@ int hllSparseSet(robj *o, long index, uint8_t count) {
         p += oplen;
         first += span;
     }
-    if (span == 0) return -1; /* Invalid format. */
+    if (span == 0 || p >= end) return -1; /* Invalid format. */
-    if (p >= end) return -1; /* Invalid format. */
 
     next = HLL_SPARSE_IS_XZERO(p) ? p+2 : p+1;
     if (next >= end) next = NULL;
@@ -1253,7 +1252,7 @@ void pfcountCommand(client *c) {
             if (o == nullptr) continue; /* Assume empty HLL for non existing var.*/
             if (isHLLObjectOrReply(c,o) != C_OK) return;
 
-            /* Merge with this HLL with our 'max' HHL by setting max[i]
+            /* Merge with this HLL with our 'max' HLL by setting max[i]
              * to MAX(max[i],hll[i]). */
             if (hllMerge(registers,HLL_REGISTERS,o) == C_ERR) {
                 addReplySds(c,sdsnew(invalid_hll_err));
@@ -1340,7 +1339,7 @@ void pfmergeCommand(client *c) {
         hdr = (hllhdr*)ptrFromObj(o);
         if (hdr->encoding == HLL_DENSE) use_dense = 1;
 
-        /* Merge with this HLL with our 'max' HHL by setting max[i]
+        /* Merge with this HLL with our 'max' HLL by setting max[i]
          * to MAX(max[i],hll[i]). */
         if (hllMerge(max,sizeof(max),o) == C_ERR) {
             addReplySds(c,sdsnew(invalid_hll_err));

src/latency.cpp

@@ -599,7 +599,7 @@ NULL
         event = (char*)dictGetKey(de);
 
         graph = latencyCommandGenSparkeline(event,ts);
-        addReplyBulkCString(c,graph);
+        addReplyVerbatim(c,graph,sdslen(graph),"txt");
         sdsfree(graph);
     } else if (!strcasecmp(szFromObj(c->argv[1]),"latest") && c->argc == 2) {
         /* LATENCY LATEST */
@@ -608,7 +608,7 @@ NULL
         /* LATENCY DOCTOR */
         sds report = createLatencyReport();
 
-        addReplyBulkCBuffer(c,report,sdslen(report));
+        addReplyVerbatim(c,report,sdslen(report),"txt");
         sdsfree(report);
     } else if (!strcasecmp(szFromObj(c->argv[1]),"reset") && c->argc >= 2) {
         /* LATENCY RESET */

src/localtime.c

@@ -52,8 +52,8 @@
 static int is_leap_year(time_t year) {
     if (year % 4) return 0;         /* A year not divisible by 4 is not leap. */
     else if (year % 100) return 1;  /* If div by 4 and not 100 is surely leap. */
-    else if (year % 400) return 0;  /* If div by 100 *and* 400 is not leap. */
+    else if (year % 400) return 0;  /* If div by 100 *and* not by 400 is not leap. */
-    else return 1;                  /* If div by 100 and not by 400 is leap. */
+    else return 1;                  /* If div by 100 and 400 is leap. */
 }
 
 void nolocks_localtime(struct tm *tmp, time_t t, time_t tz, int dst) {

src/lolwut.c

@@ -0,0 +1,188 @@
+/*
+ * Copyright (c) 2018, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ *
+ * ----------------------------------------------------------------------------
+ *
+ * This file implements the LOLWUT command. The command should do something
+ * fun and interesting, and should be replaced by a new implementation at
+ * each new version of Redis.
+ */
+
+#include "server.h"
+#include "lolwut.h"
+#include <math.h>
+
+void lolwut5Command(client *c);
+void lolwut6Command(client *c);
+
+/* The default target for LOLWUT if no matching version was found.
+ * This is what unstable versions of Redis will display. */
+void lolwutUnstableCommand(client *c) {
+    sds rendered = sdsnew("Redis ver. ");
+    rendered = sdscat(rendered,REDIS_VERSION);
+    rendered = sdscatlen(rendered,"\n",1);
+    addReplyVerbatim(c,rendered,sdslen(rendered),"txt");
+    sdsfree(rendered);
+}
+
+/* LOLWUT [VERSION <version>] [... version specific arguments ...] */
+void lolwutCommand(client *c) {
+    char *v = REDIS_VERSION;
+    char verstr[64];
+
+    if (c->argc >= 3 && !strcasecmp(c->argv[1]->ptr,"version")) {
+        long ver;
+        if (getLongFromObjectOrReply(c,c->argv[2],&ver,NULL) != C_OK) return;
+        snprintf(verstr,sizeof(verstr),"%u.0.0",(unsigned int)ver);
+        v = verstr;
+
+        /* Adjust argv/argc to filter the "VERSION ..." option, since the
+         * specific LOLWUT version implementations don't know about it
+         * and expect their arguments. */
+        c->argv += 2;
+        c->argc -= 2;
+    }
+
+    if ((v[0] == '5' && v[1] == '.' && v[2] != '9') ||
+        (v[0] == '4' && v[1] == '.' && v[2] == '9'))
+        lolwut5Command(c);
+    else if ((v[0] == '6' && v[1] == '.' && v[2] != '9') ||
+             (v[0] == '5' && v[1] == '.' && v[2] == '9'))
+        lolwut6Command(c);
+    else
+        lolwutUnstableCommand(c);
+
+    /* Fix back argc/argv in case of VERSION argument. */
+    if (v == verstr) {
+        c->argv -= 2;
+        c->argc += 2;
+    }
+}
+
+/* ========================== LOLWUT Canvase ===============================
+ * Many LOWUT versions will likely print some computer art to the screen.
+ * This is the case with LOLWUT 5 and LOLWUT 6, so here there is a generic
+ * canvas implementation that can be reused.  */
+
+/* Allocate and return a new canvas of the specified size. */
+lwCanvas *lwCreateCanvas(int width, int height, int bgcolor) {
+    lwCanvas *canvas = zmalloc(sizeof(*canvas));
+    canvas->width = width;
+    canvas->height = height;
+    canvas->pixels = zmalloc(width*height);
+    memset(canvas->pixels,bgcolor,width*height);
+    return canvas;
+}
+
+/* Free the canvas created by lwCreateCanvas(). */
+void lwFreeCanvas(lwCanvas *canvas) {
+    zfree(canvas->pixels);
+    zfree(canvas);
+}
+
+/* Set a pixel to the specified color. Color is 0 or 1, where zero means no
+ * dot will be displyed, and 1 means dot will be displayed.
+ * Coordinates are arranged so that left-top corner is 0,0. You can write
+ * out of the size of the canvas without issues. */
+void lwDrawPixel(lwCanvas *canvas, int x, int y, int color) {
+    if (x < 0 || x >= canvas->width ||
+        y < 0 || y >= canvas->height) return;
+    canvas->pixels[x+y*canvas->width] = color;
+}
+
+/* Return the value of the specified pixel on the canvas. */
+int lwGetPixel(lwCanvas *canvas, int x, int y) {
+    if (x < 0 || x >= canvas->width ||
+        y < 0 || y >= canvas->height) return 0;
+    return canvas->pixels[x+y*canvas->width];
+}
+
+/* Draw a line from x1,y1 to x2,y2 using the Bresenham algorithm. */
+void lwDrawLine(lwCanvas *canvas, int x1, int y1, int x2, int y2, int color) {
+    int dx = abs(x2-x1);
+    int dy = abs(y2-y1);
+    int sx = (x1 < x2) ? 1 : -1;
+    int sy = (y1 < y2) ? 1 : -1;
+    int err = dx-dy, e2;
+
+    while(1) {
+        lwDrawPixel(canvas,x1,y1,color);
+        if (x1 == x2 && y1 == y2) break;
+        e2 = err*2;
+        if (e2 > -dy) {
+            err -= dy;
+            x1 += sx;
+        }
+        if (e2 < dx) {
+            err += dx;
+            y1 += sy;
+        }
+    }
+}
+
+/* Draw a square centered at the specified x,y coordinates, with the specified
+ * rotation angle and size. In order to write a rotated square, we use the
+ * trivial fact that the parametric equation:
+ *
+ *  x = sin(k)
+ *  y = cos(k)
+ *
+ * Describes a circle for values going from 0 to 2*PI. So basically if we start
+ * at 45 degrees, that is k = PI/4, with the first point, and then we find
+ * the other three points incrementing K by PI/2 (90 degrees), we'll have the
+ * points of the square. In order to rotate the square, we just start with
+ * k = PI/4 + rotation_angle, and we are done.
+ *
+ * Of course the vanilla equations above will describe the square inside a
+ * circle of radius 1, so in order to draw larger squares we'll have to
+ * multiply the obtained coordinates, and then translate them. However this
+ * is much simpler than implementing the abstract concept of 2D shape and then
+ * performing the rotation/translation transformation, so for LOLWUT it's
+ * a good approach. */
+void lwDrawSquare(lwCanvas *canvas, int x, int y, float size, float angle, int color) {
+    int px[4], py[4];
+
+    /* Adjust the desired size according to the fact that the square inscribed
+     * into a circle of radius 1 has the side of length SQRT(2). This way
+     * size becomes a simple multiplication factor we can use with our
+     * coordinates to magnify them. */
+    size /= 1.4142135623;
+    size = round(size);
+
+    /* Compute the four points. */
+    float k = M_PI/4 + angle;
+    for (int j = 0; j < 4; j++) {
+        px[j] = round(sin(k) * size + x);
+        py[j] = round(cos(k) * size + y);
+        k += M_PI/2;
+    }
+
+    /* Draw the square. */
+    for (int j = 0; j < 4; j++)
+        lwDrawLine(canvas,px[j],py[j],px[(j+1)%4],py[(j+1)%4],color);
+}

src/lolwut.h

@@ -0,0 +1,49 @@
+/*
+ * Copyright (c) 2018-2019, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/* This structure represents our canvas. Drawing functions will take a pointer
+ * to a canvas to write to it. Later the canvas can be rendered to a string
+ * suitable to be printed on the screen, using unicode Braille characters. */
+
+/* This represents a very simple generic canvas in order to draw stuff.
+ * It's up to each LOLWUT versions to translate what they draw to the
+ * screen, depending on the result to accomplish. */
+typedef struct lwCanvas {
+    int width;
+    int height;
+    char *pixels;
+} lwCanvas;
+
+/* Drawing functions implemented inside lolwut.c. */
+lwCanvas *lwCreateCanvas(int width, int height, int bgcolor);
+void lwFreeCanvas(lwCanvas *canvas);
+void lwDrawPixel(lwCanvas *canvas, int x, int y, int color);
+int lwGetPixel(lwCanvas *canvas, int x, int y);
+void lwDrawLine(lwCanvas *canvas, int x1, int y1, int x2, int y2, int color);
+void lwDrawSquare(lwCanvas *canvas, int x, int y, float size, float angle, int color);

src/lolwut5.c

@@ -0,0 +1,177 @@
+/*
+ * Copyright (c) 2018, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ *
+ * ----------------------------------------------------------------------------
+ *
+ * This file implements the LOLWUT command. The command should do something
+ * fun and interesting, and should be replaced by a new implementation at
+ * each new version of Redis.
+ */
+
+#include "server.h"
+#include "lolwut.h"
+#include <math.h>
+
+/* Translate a group of 8 pixels (2x4 vertical rectangle) to the corresponding
+ * braille character. The byte should correspond to the pixels arranged as
+ * follows, where 0 is the least significant bit, and 7 the most significant
+ * bit:
+ *
+ *   0 3
+ *   1 4
+ *   2 5
+ *   6 7
+ *
+ * The corresponding utf8 encoded character is set into the three bytes
+ * pointed by 'output'.
+ */
+#include <stdio.h>
+void lwTranslatePixelsGroup(int byte, char *output) {
+    int code = 0x2800 + byte;
+    /* Convert to unicode. This is in the U0800-UFFFF range, so we need to
+     * emit it like this in three bytes:
+     * 1110xxxx 10xxxxxx 10xxxxxx. */
+    output[0] = 0xE0 | (code >> 12);          /* 1110-xxxx */
+    output[1] = 0x80 | ((code >> 6) & 0x3F);  /* 10-xxxxxx */
+    output[2] = 0x80 | (code & 0x3F);         /* 10-xxxxxx */
+}
+
+/* Schotter, the output of LOLWUT of Redis 5, is a computer graphic art piece
+ * generated by Georg Nees in the 60s. It explores the relationship between
+ * caos and order.
+ *
+ * The function creates the canvas itself, depending on the columns available
+ * in the output display and the number of squares per row and per column
+ * requested by the caller. */
+lwCanvas *lwDrawSchotter(int console_cols, int squares_per_row, int squares_per_col) {
+    /* Calculate the canvas size. */
+    int canvas_width = console_cols*2;
+    int padding = canvas_width > 4 ? 2 : 0;
+    float square_side = (float)(canvas_width-padding*2) / squares_per_row;
+    int canvas_height = square_side * squares_per_col + padding*2;
+    lwCanvas *canvas = lwCreateCanvas(canvas_width, canvas_height, 0);
+
+    for (int y = 0; y < squares_per_col; y++) {
+        for (int x = 0; x < squares_per_row; x++) {
+            int sx = x * square_side + square_side/2 + padding;
+            int sy = y * square_side + square_side/2 + padding;
+            /* Rotate and translate randomly as we go down to lower
+             * rows. */
+            float angle = 0;
+            if (y > 1) {
+                float r1 = (float)rand() / RAND_MAX / squares_per_col * y;
+                float r2 = (float)rand() / RAND_MAX / squares_per_col * y;
+                float r3 = (float)rand() / RAND_MAX / squares_per_col * y;
+                if (rand() % 2) r1 = -r1;
+                if (rand() % 2) r2 = -r2;
+                if (rand() % 2) r3 = -r3;
+                angle = r1;
+                sx += r2*square_side/3;
+                sy += r3*square_side/3;
+            }
+            lwDrawSquare(canvas,sx,sy,square_side,angle,1);
+        }
+    }
+
+    return canvas;
+}
+
+/* Converts the canvas to an SDS string representing the UTF8 characters to
+ * print to the terminal in order to obtain a graphical representaiton of the
+ * logical canvas. The actual returned string will require a terminal that is
+ * width/2 large and height/4 tall in order to hold the whole image without
+ * overflowing or scrolling, since each Barille character is 2x4. */
+static sds renderCanvas(lwCanvas *canvas) {
+    sds text = sdsempty();
+    for (int y = 0; y < canvas->height; y += 4) {
+        for (int x = 0; x < canvas->width; x += 2) {
+            /* We need to emit groups of 8 bits according to a specific
+             * arrangement. See lwTranslatePixelsGroup() for more info. */
+            int byte = 0;
+            if (lwGetPixel(canvas,x,y)) byte |= (1<<0);
+            if (lwGetPixel(canvas,x,y+1)) byte |= (1<<1);
+            if (lwGetPixel(canvas,x,y+2)) byte |= (1<<2);
+            if (lwGetPixel(canvas,x+1,y)) byte |= (1<<3);
+            if (lwGetPixel(canvas,x+1,y+1)) byte |= (1<<4);
+            if (lwGetPixel(canvas,x+1,y+2)) byte |= (1<<5);
+            if (lwGetPixel(canvas,x,y+3)) byte |= (1<<6);
+            if (lwGetPixel(canvas,x+1,y+3)) byte |= (1<<7);
+            char unicode[3];
+            lwTranslatePixelsGroup(byte,unicode);
+            text = sdscatlen(text,unicode,3);
+        }
+        if (y != canvas->height-1) text = sdscatlen(text,"\n",1);
+    }
+    return text;
+}
+
+/* The LOLWUT command:
+ *
+ * LOLWUT [terminal columns] [squares-per-row] [squares-per-col]
+ *
+ * By default the command uses 66 columns, 8 squares per row, 12 squares
+ * per column.
+ */
+void lolwut5Command(client *c) {
+    long cols = 66;
+    long squares_per_row = 8;
+    long squares_per_col = 12;
+
+    /* Parse the optional arguments if any. */
+    if (c->argc > 1 &&
+        getLongFromObjectOrReply(c,c->argv[1],&cols,NULL) != C_OK)
+        return;
+
+    if (c->argc > 2 &&
+        getLongFromObjectOrReply(c,c->argv[2],&squares_per_row,NULL) != C_OK)
+        return;
+
+    if (c->argc > 3 &&
+        getLongFromObjectOrReply(c,c->argv[3],&squares_per_col,NULL) != C_OK)
+        return;
+
+    /* Limits. We want LOLWUT to be always reasonably fast and cheap to execute
+     * so we have maximum number of columns, rows, and output resulution. */
+    if (cols < 1) cols = 1;
+    if (cols > 1000) cols = 1000;
+    if (squares_per_row < 1) squares_per_row = 1;
+    if (squares_per_row > 200) squares_per_row = 200;
+    if (squares_per_col < 1) squares_per_col = 1;
+    if (squares_per_col > 200) squares_per_col = 200;
+
+    /* Generate some computer art and reply. */
+    lwCanvas *canvas = lwDrawSchotter(cols,squares_per_row,squares_per_col);
+    sds rendered = renderCanvas(canvas);
+    rendered = sdscat(rendered,
+        "\nGeorg Nees - schotter, plotter on paper, 1968. Redis ver. ");
+    rendered = sdscat(rendered,REDIS_VERSION);
+    rendered = sdscatlen(rendered,"\n",1);
+    addReplyVerbatim(c,rendered,sdslen(rendered),"txt");
+    sdsfree(rendered);
+    lwFreeCanvas(canvas);
+}

src/lolwut6.c

@@ -0,0 +1,201 @@
+/*
+ * Copyright (c) 2019, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ *
+ * ----------------------------------------------------------------------------
+ *
+ * This file implements the LOLWUT command. The command should do something
+ * fun and interesting, and should be replaced by a new implementation at
+ * each new version of Redis.
+ *
+ * Thanks to Michele Hiki Falcone for the original image that ispired
+ * the image, part of his game, Plaguemon.
+ *
+ * Thanks to the Shhh computer art collective for the help in tuning the
+ * output to have a better artistic effect.
+ */
+
+#include "server.h"
+#include "lolwut.h"
+
+/* Render the canvas using the four gray levels of the standard color
+ * terminal: they match very well to the grayscale display of the gameboy. */
+static sds renderCanvas(lwCanvas *canvas) {
+    sds text = sdsempty();
+    for (int y = 0; y < canvas->height; y++) {
+        for (int x = 0; x < canvas->width; x++) {
+            int color = lwGetPixel(canvas,x,y);
+            char *ce; /* Color escape sequence. */
+
+            /* Note that we set both the foreground and background color.
+             * This way we are able to get a more consistent result among
+             * different terminals implementations. */
+            switch(color) {
+            case 0: ce = "0;30;40m"; break;    /* Black */
+            case 1: ce = "0;90;100m"; break;   /* Gray 1 */
+            case 2: ce = "0;37;47m"; break;    /* Gray 2 */
+            case 3: ce = "0;97;107m"; break;   /* White */
+            default: ce = "0;30;40m"; break;   /* Just for safety. */
+            }
+            text = sdscatprintf(text,"\033[%s \033[0m",ce);
+        }
+        if (y != canvas->height-1) text = sdscatlen(text,"\n",1);
+    }
+    return text;
+}
+
+/* Draw a skyscraper on the canvas, according to the parameters in the
+ * 'skyscraper' structure. Window colors are random and are always one
+ * of the two grays. */
+struct skyscraper {
+    int xoff;       /* X offset. */
+    int width;      /* Pixels width. */
+    int height;     /* Pixels height. */
+    int windows;    /* Draw windows if true. */
+    int color;      /* Color of the skyscraper. */
+};
+
+void generateSkyscraper(lwCanvas *canvas, struct skyscraper *si) {
+    int starty = canvas->height-1;
+    int endy = starty - si->height + 1;
+    for (int y = starty; y >= endy; y--) {
+        for (int x = si->xoff; x < si->xoff+si->width; x++) {
+            /* The roof is four pixels less wide. */
+            if (y == endy && (x <= si->xoff+1 || x >= si->xoff+si->width-2))
+                continue;
+            int color = si->color;
+            /* Alter the color if this is a place where we want to
+             * draw a window. We check that we are in the inner part of the
+             * skyscraper, so that windows are far from the borders. */
+            if (si->windows &&
+                x > si->xoff+1 &&
+                x < si->xoff+si->width-2 &&
+                y > endy+1 &&
+                y < starty-1)
+            {
+                /* Calculate the x,y position relative to the start of
+                 * the window area. */
+                int relx = x - (si->xoff+1);
+                int rely = y - (endy+1);
+
+                /* Note that we want the windows to be two pixels wide
+                 * but just one pixel tall, because terminal "pixels"
+                 * (characters) are not square. */
+                if (relx/2 % 2 && rely % 2) {
+                    do {
+                        color = 1 + rand() % 2;
+                    } while (color == si->color);
+                    /* Except we want adjacent pixels creating the same
+                     * window to be the same color. */
+                    if (relx % 2) color = lwGetPixel(canvas,x-1,y);
+                }
+            }
+            lwDrawPixel(canvas,x,y,color);
+        }
+    }
+}
+
+/* Generate a skyline inspired by the parallax backgrounds of 8 bit games. */
+void generateSkyline(lwCanvas *canvas) {
+    struct skyscraper si;
+
+    /* First draw the background skyscraper without windows, using the
+     * two different grays. We use two passes to make sure that the lighter
+     * ones are always in the background. */
+    for (int color = 2; color >= 1; color--) {
+        si.color = color;
+        for (int offset = -10; offset < canvas->width;) {
+            offset += rand() % 8;
+            si.xoff = offset;
+            si.width = 10 + rand()%9;
+            if (color == 2)
+                si.height = canvas->height/2 + rand()%canvas->height/2;
+            else
+                si.height = canvas->height/2 + rand()%canvas->height/3;
+            si.windows = 0;
+            generateSkyscraper(canvas, &si);
+            if (color == 2)
+                offset += si.width/2;
+            else
+                offset += si.width+1;
+        }
+    }
+
+    /* Now draw the foreground skyscraper with the windows. */
+    si.color = 0;
+    for (int offset = -10; offset < canvas->width;) {
+        offset += rand() % 8;
+        si.xoff = offset;
+        si.width = 5 + rand()%14;
+        if (si.width % 4) si.width += (si.width % 3);
+        si.height = canvas->height/3 + rand()%canvas->height/2;
+        si.windows = 1;
+        generateSkyscraper(canvas, &si);
+        offset += si.width+5;
+    }
+}
+
+/* The LOLWUT 6 command:
+ *
+ * LOLWUT [columns] [rows]
+ *
+ * By default the command uses 80 columns, 40 squares per row
+ * per column.
+ */
+void lolwut6Command(client *c) {
+    long cols = 80;
+    long rows = 20;
+
+    /* Parse the optional arguments if any. */
+    if (c->argc > 1 &&
+        getLongFromObjectOrReply(c,c->argv[1],&cols,NULL) != C_OK)
+        return;
+
+    if (c->argc > 2 &&
+        getLongFromObjectOrReply(c,c->argv[2],&rows,NULL) != C_OK)
+        return;
+
+    /* Limits. We want LOLWUT to be always reasonably fast and cheap to execute
+     * so we have maximum number of columns, rows, and output resulution. */
+    if (cols < 1) cols = 1;
+    if (cols > 1000) cols = 1000;
+    if (rows < 1) rows = 1;
+    if (rows > 1000) rows = 1000;
+
+    /* Generate the city skyline and reply. */
+    lwCanvas *canvas = lwCreateCanvas(cols,rows,3);
+    generateSkyline(canvas);
+    sds rendered = renderCanvas(canvas);
+    rendered = sdscat(rendered,
+        "\nDedicated to the 8 bit game developers of past and present.\n"
+        "Original 8 bit image from Plaguemon by hikikomori. Redis ver. ");
+    rendered = sdscat(rendered,REDIS_VERSION);
+    rendered = sdscatlen(rendered,"\n",1);
+    addReplyVerbatim(c,rendered,sdslen(rendered),"txt");
+    sdsfree(rendered);
+    lwFreeCanvas(canvas);
+}

src/mkreleasehdr.sh

@@ -3,7 +3,7 @@ GIT_SHA1=`(git show-ref --head --hash=8 2> /dev/null || echo 00000000) | head -n
 GIT_DIRTY=`git diff --no-ext-diff 2> /dev/null | wc -l`
 BUILD_ID=`uname -n`"-"`date +%s`
 if [ -n "$SOURCE_DATE_EPOCH" ]; then
-  BUILD_ID=$(date -u -d "@$SOURCE_DATE_EPOCH" +%s 2>/dev/null || date -u -r "$SOURCE_DATE_EPOCH" +%s 2>/dev/null || date -u %s)
+  BUILD_ID=$(date -u -d "@$SOURCE_DATE_EPOCH" +%s 2>/dev/null || date -u -r "$SOURCE_DATE_EPOCH" +%s 2>/dev/null || date -u +%s)
 fi
 test -f release.h || touch release.h
 (cat release.h | grep SHA1 | grep $GIT_SHA1) && \

src/modules/Makefile

@@ -13,7 +13,7 @@ endif
 
 .SUFFIXES: .c .so .xo .o
 
-all: helloworld.so hellotype.so helloblock.so testmodule.so hellocluster.so hellotimer.so hellodict.so
+all: helloworld.so hellotype.so helloblock.so testmodule.so hellocluster.so hellotimer.so hellodict.so hellohook.so helloacl.so
 
 .c.xo:
 	$(CC) -I. $(CFLAGS) $(SHOBJ_CFLAGS) -fPIC -c $< -o $@
@@ -46,6 +46,17 @@ hellotimer.so: hellotimer.xo
 hellodict.xo: ../redismodule.h
 
 hellodict.so: hellodict.xo
+	$(LD) -o $@ $< $(SHOBJ_LDFLAGS) $(LIBS) -lc
+
+hellohook.xo: ../redismodule.h
+
+hellohook.so: hellohook.xo
+	$(LD) -o $@ $< $(SHOBJ_LDFLAGS) $(LIBS) -lc
+
+helloacl.xo: ../redismodule.h
+
+helloacl.so: helloacl.xo
+	$(LD) -o $@ $< $(SHOBJ_LDFLAGS) $(LIBS) -lc
 
 testmodule.xo: ../redismodule.h
 

src/modules/helloacl.c

@@ -0,0 +1,191 @@
+/* ACL API example - An example for performing custom synchronous and
+ * asynchronous password authentication.
+ *
+ * -----------------------------------------------------------------------------
+ *
+ * Copyright 2019 Amazon.com, Inc. or its affiliates.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#define REDISMODULE_EXPERIMENTAL_API
+#include "../redismodule.h"
+#include <pthread.h>
+#include <unistd.h>
+
+// A simple global user
+static RedisModuleUser *global;
+static uint64_t global_auth_client_id = 0;
+
+/* HELLOACL.REVOKE 
+ * Synchronously revoke access from a user. */
+int RevokeCommand_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    REDISMODULE_NOT_USED(argv);
+    REDISMODULE_NOT_USED(argc);
+
+    if (global_auth_client_id) {
+        RedisModule_DeauthenticateAndCloseClient(ctx, global_auth_client_id);
+        return RedisModule_ReplyWithSimpleString(ctx, "OK");
+    } else {
+        return RedisModule_ReplyWithError(ctx, "Global user currently not used");    
+    }
+}
+
+/* HELLOACL.RESET 
+ * Synchronously delete and re-create a module user. */
+int ResetCommand_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    REDISMODULE_NOT_USED(argv);
+    REDISMODULE_NOT_USED(argc);
+
+    RedisModule_FreeModuleUser(global);
+    global = RedisModule_CreateModuleUser("global");
+    RedisModule_SetModuleUserACL(global, "allcommands");
+    RedisModule_SetModuleUserACL(global, "allkeys");
+    RedisModule_SetModuleUserACL(global, "on");
+
+    return RedisModule_ReplyWithSimpleString(ctx, "OK");
+}
+
+/* Callback handler for user changes, use this to notify a module of 
+ * changes to users authenticated by the module */
+void HelloACL_UserChanged(uint64_t client_id, void *privdata) {
+    REDISMODULE_NOT_USED(privdata);
+    REDISMODULE_NOT_USED(client_id);
+    global_auth_client_id = 0;
+}
+
+/* HELLOACL.AUTHGLOBAL 
+ * Synchronously assigns a module user to the current context. */
+int AuthGlobalCommand_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    REDISMODULE_NOT_USED(argv);
+    REDISMODULE_NOT_USED(argc);
+
+    if (global_auth_client_id) {
+        return RedisModule_ReplyWithError(ctx, "Global user currently used");    
+    }
+
+    RedisModule_AuthenticateClientWithUser(ctx, global, HelloACL_UserChanged, NULL, &global_auth_client_id);
+
+    return RedisModule_ReplyWithSimpleString(ctx, "OK");
+}
+
+#define TIMEOUT_TIME 1000
+
+/* Reply callback for auth command HELLOACL.AUTHASYNC */
+int HelloACL_Reply(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    REDISMODULE_NOT_USED(argv);
+    REDISMODULE_NOT_USED(argc);
+    size_t length;
+
+    RedisModuleString *user_string = RedisModule_GetBlockedClientPrivateData(ctx);
+    const char *name = RedisModule_StringPtrLen(user_string, &length);
+
+    if (RedisModule_AuthenticateClientWithACLUser(ctx, name, length, NULL, NULL, NULL) == 
+            REDISMODULE_ERR) {
+        return RedisModule_ReplyWithError(ctx, "Invalid Username or password");    
+    }
+    return RedisModule_ReplyWithSimpleString(ctx, "OK");
+}
+
+/* Timeout callback for auth command HELLOACL.AUTHASYNC */
+int HelloACL_Timeout(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    REDISMODULE_NOT_USED(argv);
+    REDISMODULE_NOT_USED(argc);
+    return RedisModule_ReplyWithSimpleString(ctx, "Request timedout");
+}
+
+/* Private data frees data for HELLOACL.AUTHASYNC command. */
+void HelloACL_FreeData(RedisModuleCtx *ctx, void *privdata) {
+    REDISMODULE_NOT_USED(ctx);
+    RedisModule_FreeString(NULL, privdata);
+}
+
+/* Background authentication can happen here. */
+void *HelloACL_ThreadMain(void *args) {
+    void **targs = args;
+    RedisModuleBlockedClient *bc = targs[0];
+    RedisModuleString *user = targs[1];
+    RedisModule_Free(targs);
+
+    RedisModule_UnblockClient(bc,user);
+    return NULL;
+}
+
+/* HELLOACL.AUTHASYNC 
+ * Asynchronously assigns an ACL user to the current context. */
+int AuthAsyncCommand_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (argc != 2) return RedisModule_WrongArity(ctx);
+
+    pthread_t tid;
+    RedisModuleBlockedClient *bc = RedisModule_BlockClient(ctx, HelloACL_Reply, HelloACL_Timeout, HelloACL_FreeData, TIMEOUT_TIME);
+    
+
+    void **targs = RedisModule_Alloc(sizeof(void*)*2);
+    targs[0] = bc;
+    targs[1] = RedisModule_CreateStringFromString(NULL, argv[1]);
+
+    if (pthread_create(&tid, NULL, HelloACL_ThreadMain, targs) != 0) {
+        RedisModule_AbortBlock(bc);
+        return RedisModule_ReplyWithError(ctx, "-ERR Can't start thread");
+    }
+
+    return REDISMODULE_OK;
+}
+
+/* This function must be present on each Redis module. It is used in order to
+ * register the commands into the Redis server. */
+int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    REDISMODULE_NOT_USED(argv);
+    REDISMODULE_NOT_USED(argc);
+
+    if (RedisModule_Init(ctx,"helloacl",1,REDISMODULE_APIVER_1)
+        == REDISMODULE_ERR) return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"helloacl.reset",
+        ResetCommand_RedisCommand,"",0,0,0) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"helloacl.revoke",
+        RevokeCommand_RedisCommand,"",0,0,0) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"helloacl.authglobal",
+        AuthGlobalCommand_RedisCommand,"no-auth",0,0,0) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"helloacl.authasync",
+        AuthAsyncCommand_RedisCommand,"no-auth",0,0,0) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    global = RedisModule_CreateModuleUser("global");
+    RedisModule_SetModuleUserACL(global, "allcommands");
+    RedisModule_SetModuleUserACL(global, "allkeys");
+    RedisModule_SetModuleUserACL(global, "on");
+
+    global_auth_client_id = 0;
+
+    return REDISMODULE_OK;
+}

src/modules/hellohook.c

@@ -0,0 +1,93 @@
+/* Server hooks API example
+ *
+ * -----------------------------------------------------------------------------
+ *
+ * Copyright (c) 2019, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#define REDISMODULE_EXPERIMENTAL_API
+#include "../redismodule.h"
+#include <stdio.h>
+#include <stdlib.h>
+#include <ctype.h>
+#include <string.h>
+
+/* Client state change callback. */
+void clientChangeCallback(RedisModuleCtx *ctx, RedisModuleEvent e, uint64_t sub, void *data)
+{
+    REDISMODULE_NOT_USED(ctx);
+    REDISMODULE_NOT_USED(e);
+
+    RedisModuleClientInfo *ci = data;
+    printf("Client %s event for client #%llu %s:%d\n",
+        (sub == REDISMODULE_SUBEVENT_CLIENT_CHANGE_CONNECTED) ?
+            "connection" : "disconnection",
+        ci->id,ci->addr,ci->port);
+}
+
+void flushdbCallback(RedisModuleCtx *ctx, RedisModuleEvent e, uint64_t sub, void *data)
+{
+    REDISMODULE_NOT_USED(ctx);
+    REDISMODULE_NOT_USED(e);
+
+    RedisModuleFlushInfo *fi = data;
+    if (sub == REDISMODULE_SUBEVENT_FLUSHDB_START) {
+        if (fi->dbnum != -1) {
+            RedisModuleCallReply *reply;
+            reply = RedisModule_Call(ctx,"DBSIZE","");
+            long long numkeys = RedisModule_CallReplyInteger(reply);
+            printf("FLUSHDB event of database %d started (%lld keys in DB)\n",
+                fi->dbnum, numkeys);
+            RedisModule_FreeCallReply(reply);
+        } else {
+            printf("FLUSHALL event started\n");
+        }
+    } else {
+        if (fi->dbnum != -1) {
+            printf("FLUSHDB event of database %d ended\n",fi->dbnum);
+        } else {
+            printf("FLUSHALL event ended\n");
+        }
+    }
+}
+
+/* This function must be present on each Redis module. It is used in order to
+ * register the commands into the Redis server. */
+int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    REDISMODULE_NOT_USED(argv);
+    REDISMODULE_NOT_USED(argc);
+
+    if (RedisModule_Init(ctx,"hellohook",1,REDISMODULE_APIVER_1)
+        == REDISMODULE_ERR) return REDISMODULE_ERR;
+
+    RedisModule_SubscribeToServerEvent(ctx,
+        RedisModuleEvent_ClientChange, clientChangeCallback);
+    RedisModule_SubscribeToServerEvent(ctx,
+        RedisModuleEvent_FlushDB, flushdbCallback);
+    return REDISMODULE_OK;
+}

src/modules/hellotype.c

@@ -129,6 +129,7 @@ int HelloTypeInsert_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv,
 
     /* Insert the new element. */
     HelloTypeInsert(hto,value);
+    RedisModule_SignalKeyAsReady(ctx,argv[1]);
 
     RedisModule_ReplyWithLongLong(ctx,hto->len);
     RedisModule_ReplicateVerbatim(ctx);
@@ -190,6 +191,77 @@ int HelloTypeLen_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int
     return REDISMODULE_OK;
 }
 
+/* ====================== Example of a blocking command ==================== */
+
+/* Reply callback for blocking command HELLOTYPE.BRANGE, this will get
+ * called when the key we blocked for is ready: we need to check if we
+ * can really serve the client, and reply OK or ERR accordingly. */
+int HelloBlock_Reply(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    REDISMODULE_NOT_USED(argv);
+    REDISMODULE_NOT_USED(argc);
+
+    RedisModuleString *keyname = RedisModule_GetBlockedClientReadyKey(ctx);
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,keyname,REDISMODULE_READ);
+    int type = RedisModule_KeyType(key);
+    if (type != REDISMODULE_KEYTYPE_MODULE ||
+        RedisModule_ModuleTypeGetType(key) != HelloType)
+    {
+        RedisModule_CloseKey(key);
+        return REDISMODULE_ERR;
+    }
+
+    /* In case the key is able to serve our blocked client, let's directly
+     * use our original command implementation to make this example simpler. */
+    RedisModule_CloseKey(key);
+    return HelloTypeRange_RedisCommand(ctx,argv,argc-1);
+}
+
+/* Timeout callback for blocking command HELLOTYPE.BRANGE */
+int HelloBlock_Timeout(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    REDISMODULE_NOT_USED(argv);
+    REDISMODULE_NOT_USED(argc);
+    return RedisModule_ReplyWithSimpleString(ctx,"Request timedout");
+}
+
+/* Private data freeing callback for HELLOTYPE.BRANGE command. */
+void HelloBlock_FreeData(RedisModuleCtx *ctx, void *privdata) {
+    REDISMODULE_NOT_USED(ctx);
+    RedisModule_Free(privdata);
+}
+
+/* HELLOTYPE.BRANGE key first count timeout -- This is a blocking verison of
+ * the RANGE operation, in order to show how to use the API
+ * RedisModule_BlockClientOnKeys(). */
+int HelloTypeBRange_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (argc != 5) return RedisModule_WrongArity(ctx);
+    RedisModule_AutoMemory(ctx); /* Use automatic memory management. */
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    int type = RedisModule_KeyType(key);
+    if (type != REDISMODULE_KEYTYPE_EMPTY &&
+        RedisModule_ModuleTypeGetType(key) != HelloType)
+    {
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    /* Parse the timeout before even trying to serve the client synchronously,
+     * so that we always fail ASAP on syntax errors. */
+    long long timeout;
+    if (RedisModule_StringToLongLong(argv[4],&timeout) != REDISMODULE_OK) {
+        return RedisModule_ReplyWithError(ctx,
+            "ERR invalid timeout parameter");
+    }
+
+    /* Can we serve the reply synchronously? */
+    if (type != REDISMODULE_KEYTYPE_EMPTY) {
+        return HelloTypeRange_RedisCommand(ctx,argv,argc-1);
+    }
+
+    /* Otherwise let's block on the key. */
+    void *privdata = RedisModule_Alloc(100);
+    RedisModule_BlockClientOnKeys(ctx,HelloBlock_Reply,HelloBlock_Timeout,HelloBlock_FreeData,timeout,argv+1,1,privdata);
+    return REDISMODULE_OK;
+}
 
 /* ========================== "hellotype" type methods ======================= */
 
@@ -282,5 +354,9 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
         HelloTypeLen_RedisCommand,"readonly",1,1,1) == REDISMODULE_ERR)
         return REDISMODULE_ERR;
 
+    if (RedisModule_CreateCommand(ctx,"hellotype.brange",
+        HelloTypeBRange_RedisCommand,"readonly",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
     return REDISMODULE_OK;
 }

src/multi.cpp

@@ -108,11 +108,13 @@ void discardCommand(client *c) {
 /* Send a MULTI command to all the slaves and AOF file. Check the execCommand
  * implementation for more information. */
 void execCommandPropagateMulti(client *c) {
-    robj *multistring = createStringObject("MULTI",5);
+    propagate(cserver.multiCommand,c->db->id,&shared.multi,1,
-
+              PROPAGATE_AOF|PROPAGATE_REPL);
-    propagate(cserver.multiCommand,c->db->id,&multistring,1,
+}
+
+void execCommandPropagateExec(client *c) {
+    propagate(cserver.execCommand,c->db->id,&shared.exec,1,
               PROPAGATE_AOF|PROPAGATE_REPL);
-    decrRefCount(multistring);
 }
 
 void execCommand(client *c) {
@@ -177,7 +179,19 @@ void execCommand(client *c) {
             must_propagate = 1;
         }
 
+        int acl_retval = ACLCheckCommandPerm(c);
+        if (acl_retval != ACL_OK) {
+            addReplyErrorFormat(c,
+                "-NOPERM ACLs rules changed between the moment the "
+                "transaction was accumulated and the EXEC call. "
+                "This command is no longer allowed for the "
+                "following reason: %s",
+                (acl_retval == ACL_DENIED_CMD) ?
+                "no permission to execute the command or subcommand" :
+                "no permission to touch the specified keys");
+        } else {
             call(c,g_pserver->loading ? CMD_CALL_NONE : CMD_CALL_FULL);
+        }
 
         /* Commands may alter argc/argv, restore mstate. */
         c->mstate.commands[j].argc = c->argc;

src/networking.cpp

@@ -85,32 +85,28 @@ void linkClient(client *c) {
      * this way removing the client in unlinkClient() will not require
      * a linear scan, but just a constant time operation. */
     c->client_list_node = listLast(g_pserver->clients);
-    if (c->fd != -1) atomicIncr(g_pserver->rgthreadvar[c->iel].cclients, 1);
+    if (c->conn != nullptr) atomicIncr(g_pserver->rgthreadvar[c->iel].cclients, 1);
     uint64_t id = htonu64(c->id);
     raxInsert(g_pserver->clients_index,(unsigned char*)&id,sizeof(id),c,NULL);
 }
 
-client *createClient(int fd, int iel) {
+client *createClient(connection *conn, int iel) {
     client *c = (client*)zmalloc(sizeof(client), MALLOC_LOCAL);
+    serverAssert(conn == nullptr || (iel == (serverTL - g_pserver->rgthreadvar)));
 
     c->iel = iel;
-    /* passing -1 as fd it is possible to create a non connected client.
+    /* passing NULL as conn it is possible to create a non connected client.
      * This is useful since all the commands needs to be executed
      * in the context of a client. When commands are executed in other
      * contexts (for instance a Lua script) we need a non connected client. */
-    if (fd != -1) {
+    if (conn) {
         serverAssert(iel == (serverTL - g_pserver->rgthreadvar));
-        anetNonBlock(NULL,fd);
+        connNonBlock(conn);
-        anetEnableTcpNoDelay(NULL,fd);
+        connEnableTcpNoDelay(conn);
         if (cserver.tcpkeepalive)
-            anetKeepAlive(NULL,fd,cserver.tcpkeepalive);
+            connKeepAlive(conn,cserver.tcpkeepalive);
-        if (aeCreateFileEvent(g_pserver->rgthreadvar[iel].el,fd,AE_READABLE|AE_READ_THREADSAFE,
+        connSetReadHandler(conn, readQueryFromClient, true);
-            readQueryFromClient, c) == AE_ERR)
+        connSetPrivateData(conn, c);
-        {
-            close(fd);
-            zfree(c);
-            return NULL;
-        }
     }
 
     selectDb(c,0);
@@ -120,7 +116,7 @@ client *createClient(int fd, int iel) {
     fastlock_init(&c->lock, "client");
     c->id = client_id;
     c->resp = 2;
-    c->fd = fd;
+    c->conn = conn;
     c->name = NULL;
     c->bufpos = 0;
     c->qb_pos = 0;
@@ -179,9 +175,12 @@ client *createClient(int fd, int iel) {
     c->casyncOpsPending = 0;
     memset(c->uuid, 0, UUID_BINARY_LEN);
 
+    c->auth_callback = NULL;
+    c->auth_callback_privdata = NULL;
+    c->auth_module = NULL;
     listSetFreeMethod(c->pubsub_patterns,decrRefCountVoid);
     listSetMatchMethod(c->pubsub_patterns,listMatchObjects);
-    if (fd != -1) linkClient(c);
+    if (conn) linkClient(c);
     initClientMultiState(c);
     AssertCorrectThread(c);
     return c;
@@ -250,7 +249,7 @@ int prepareClientToWrite(client *c, bool fAsync) {
     fAsync = fAsync && !FCorrectThread(c);  // Not async if we're on the right thread
     serverAssert(FCorrectThread(c) || fAsync);
 	if (FCorrectThread(c)) {
-		serverAssert(c->fd <= 0 || c->lock.fOwnLock());
+		serverAssert(c->conn == nullptr || c->lock.fOwnLock());
 	} else {
 		serverAssert(GlobalLocksAcquired());
 	}
@@ -270,7 +269,7 @@ int prepareClientToWrite(client *c, bool fAsync) {
     if ((c->flags & CLIENT_MASTER) &&
         !(c->flags & CLIENT_MASTER_FORCE_REPLY)) return C_ERR;
 
-    if (c->fd <= 0) return C_ERR; /* Fake client for AOF loading. */
+    if (!c->conn) return C_ERR; /* Fake client for AOF loading. */
 
     /* Schedule the client to write the output buffers to the socket, unless
      * it should already be setup to do so (it has already pending data). */
@@ -636,7 +635,7 @@ void addReplyDoubleCore(client *c, double d, bool fAsync) {
         if (c->resp == 2) {
             addReplyBulkCStringCore(c, d > 0 ? "inf" : "-inf", fAsync);
         } else {
-            addReplyProtoCore(c, d > 0 ? ",inf\r\n" : "-inf\r\n",
+            addReplyProtoCore(c, d > 0 ? ",inf\r\n" : ",-inf\r\n",
                               d > 0 ? 6 : 7, fAsync);
         }
     } else {
@@ -672,7 +671,7 @@ void addReplyHumanLongDouble(client *c, long double d) {
         decrRefCount(o);
     } else {
         char buf[MAX_LONG_DOUBLE_CHARS];
-        int len = ld2string(buf,sizeof(buf),d,1);
+        int len = ld2string(buf,sizeof(buf),d,LD_STR_HUMAN);
         addReplyProto(c,",",1);
         addReplyProto(c,buf,len);
         addReplyProto(c,"\r\n",2);
@@ -816,14 +815,25 @@ void addReplyBool(client *c, int b) {
  * RESP2 had it, so API-wise we have this call, that will emit the correct
  * RESP2 protocol, however for RESP3 the reply will always be just the
  * Null type "_\r\n". */
-void addReplyNullArray(client *c) {
+void addReplyNullArrayCore(client *c, bool fAsync) 
+{
     if (c->resp == 2) {
-        addReplyProto(c,"*-1\r\n",5);
+        addReplyProtoCore(c,"*-1\r\n",5,fAsync);
     } else {
-        addReplyProto(c,"_\r\n",3);
+        addReplyProtoCore(c,"_\r\n",3,fAsync);
     }
 }
 
+void addReplyNullArray(client *c)
+{
+    addReplyNullArrayCore(c, false);
+}
+
+void addReplyNullArrayAsync(client *c)
+{
+    addReplyNullArrayCore(c, true);
+}
+
 /* Create the length prefix of a bulk reply, example: $2234 */
 void addReplyBulkLenCore(client *c, robj_roptr obj, bool fAsync) {
     size_t len = stringObjectLen(obj);
@@ -1003,44 +1013,21 @@ int clientHasPendingReplies(client *c) {
     return (c->bufpos || listLength(c->reply)) && !(c->flags & CLIENT_CLOSE_ASAP);
 }
 
-#define MAX_ACCEPTS_PER_CALL 1000
+void clientAcceptHandler(connection *conn) {
-static void acceptCommonHandler(int fd, int flags, char *ip, int iel) {
+    client *c = (client*)connGetPrivateData(conn);
-    client *c;
+
-    if ((c = createClient(fd, iel)) == NULL) {
+    if (connGetState(conn) != CONN_STATE_CONNECTED) {
         serverLog(LL_WARNING,
-            "Error registering fd event for the new client: %s (fd=%d)",
+                "Error accepting a client connection: %s",
-            strerror(errno),fd);
+                connGetLastError(conn));
-        return;
-    }
-
-#ifdef HAVE_SO_INCOMING_CPU
-    // Set thread affinity
-    if (cserver.fThreadAffinity)
-    {
-        int cpu = iel;
-        if (setsockopt(fd, SOL_SOCKET, SO_INCOMING_CPU, &cpu, sizeof(iel)) != 0)
-        {
-            serverLog(LL_WARNING, "Failed to set socket affinity");
-        }
-    }
-#endif
-
-    /* If maxclient directive is set and this is one client more... close the
-     * connection. Note that we create the client instead to check before
-     * for this condition, since now the socket is already set in non-blocking
-     * mode and we can send an error for free using the Kernel I/O */
-    if (listLength(g_pserver->clients) > g_pserver->maxclients) {
-        const char *err = "-ERR max number of clients reached\r\n";
-
-        /* That's a best effort error message, don't check write errors */
-        if (write(c->fd,err,strlen(err)) == -1) {
-            /* Nothing to do, Just to avoid the warning... */
-        }
-        g_pserver->stat_rejected_conn++;
         freeClient(c);
         return;
     }
 
+    // Set thread affinity
+    if (cserver.fThreadAffinity)
+        connSetThreadAffinity(conn, c->iel);
+
     /* If the server is running in protected mode (the default) and there
      * is no password set, nor a specific interface is bound, we don't accept
      * requests from non loopback interfaces. Instead we try to explain the
@@ -1048,10 +1035,12 @@ static void acceptCommonHandler(int fd, int flags, char *ip, int iel) {
     if (g_pserver->protected_mode &&
         g_pserver->bindaddr_count == 0 &&
         DefaultUser->flags & USER_FLAG_NOPASS &&
-        !(flags & CLIENT_UNIX_SOCKET) &&
+        !(c->flags & CLIENT_UNIX_SOCKET))
-        ip != NULL)
     {
-        if (strcmp(ip,"127.0.0.1") && strcmp(ip,"::1")) {
+        char cip[NET_IP_STR_LEN+1] = { 0 };
+        connPeerToString(conn, cip, sizeof(cip)-1, NULL);
+
+        if (strcmp(cip,"127.0.0.1") && strcmp(cip,"::1")) {
             const char *err =
                 "-DENIED Redis is running in protected mode because protected "
                 "mode is enabled, no bind address was specified, no "
@@ -1073,7 +1062,7 @@ static void acceptCommonHandler(int fd, int flags, char *ip, int iel) {
                 "4) Setup a bind address or an authentication password. "
                 "NOTE: You only need to do one of the above things in order for "
                 "the server to start accepting connections from the outside.\r\n";
-            if (write(c->fd,err,strlen(err)) == -1) {
+            if (connWrite(c->conn,err,strlen(err)) == -1) {
                 /* Nothing to do, Just to avoid the warning... */
             }
             g_pserver->stat_rejected_conn++;
@@ -1083,7 +1072,67 @@ static void acceptCommonHandler(int fd, int flags, char *ip, int iel) {
     }
 
     g_pserver->stat_numconnections++;
+    moduleFireServerEvent(REDISMODULE_EVENT_CLIENT_CHANGE,
+                          REDISMODULE_SUBEVENT_CLIENT_CHANGE_CONNECTED,
+                          c);
+}
+
+#define MAX_ACCEPTS_PER_CALL 1000
+static void acceptCommonHandler(connection *conn, int flags, char *ip, int iel) {
+    client *c;
+    UNUSED(ip);
+    AeLocker locker;
+    locker.arm(nullptr);
+
+    /* Admission control will happen before a client is created and connAccept()
+     * called, because we don't want to even start transport-level negotiation
+     * if rejected.
+     */
+    if (listLength(g_pserver->clients) >= g_pserver->maxclients) {
+        const char *err = "-ERR max number of clients reached\r\n";
+
+        /* That's a best effort error message, don't check write errors.
+         * Note that for TLS connections, no handshake was done yet so nothing is written
+         * and the connection will just drop.
+         */
+        if (connWrite(conn,err,strlen(err)) == -1) {
+            /* Nothing to do, Just to avoid the warning... */
+        }
+        g_pserver->stat_rejected_conn++;
+        connClose(conn);
+        return;
+    }
+
+    /* Create connection and client */
+    if ((c = createClient(conn, iel)) == NULL) {
+        char conninfo[100];
+        serverLog(LL_WARNING,
+            "Error registering fd event for the new client: %s (conn: %s)",
+            connGetLastError(conn),
+            connGetInfo(conn, conninfo, sizeof(conninfo)));
+        connClose(conn); /* May be already closed, just ignore errors */
+        return;
+    }
+
+    /* Last chance to keep flags */
     c->flags |= flags;
+
+    /* Initiate accept.
+     *
+     * Note that connAccept() is free to do two things here:
+     * 1. Call clientAcceptHandler() immediately;
+     * 2. Schedule a future call to clientAcceptHandler().
+     *
+     * Because of that, we must do nothing else afterwards.
+     */
+    if (connAccept(conn, clientAcceptHandler) == C_ERR) {
+        char conninfo[100];
+        serverLog(LL_WARNING,
+                "Error accepting a client connection: %s (conn: %s)",
+                connGetLastError(conn), connGetInfo(conn, conninfo, sizeof(conninfo)));
+        freeClient((client*)connGetPrivateData(conn));
+        return;
+    }
 }
 
 void acceptTcpHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
@@ -1108,7 +1157,7 @@ void acceptTcpHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
             // We always accept on the same thread
         LLocalThread:
             aeAcquireLock();
-            acceptCommonHandler(cfd,0,cip, ielCur);
+            acceptCommonHandler(connCreateAcceptedSocket(cfd),0,cip,ielCur);
             aeReleaseLock();
         }
         else
@@ -1123,19 +1172,43 @@ void acceptTcpHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
             char *szT = (char*)zmalloc(NET_IP_STR_LEN, MALLOC_LOCAL);
             memcpy(szT, cip, NET_IP_STR_LEN);
             aePostFunction(g_pserver->rgthreadvar[iel].el, [cfd, iel, szT]{
-                acceptCommonHandler(cfd,0,szT, iel);
+                acceptCommonHandler(connCreateAcceptedSocket(cfd),0,szT,iel);
                 zfree(szT);
             });
         }
     }
 }
 
+void acceptTLSHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
+    int cport, cfd, max = MAX_ACCEPTS_PER_CALL;
+    char cip[NET_IP_STR_LEN];
+    UNUSED(el);
+    UNUSED(mask);
+    UNUSED(privdata);
+
+    int ielCur = ielFromEventLoop(el);
+    while(max--) {
+        cfd = anetTcpAccept(serverTL->neterr, fd, cip, sizeof(cip), &cport);
+        if (cfd == ANET_ERR) {
+            if (errno != EWOULDBLOCK)
+                serverLog(LL_WARNING,
+                    "Accepting client connection: %s", serverTL->neterr);
+            return;
+        }
+        serverLog(LL_VERBOSE,"Accepted %s:%d", cip, cport);
+        aeAcquireLock();
+        acceptCommonHandler(connCreateAcceptedTLS(cfd, g_pserver->tls_auth_clients),0,cip,ielCur);
+        aeReleaseLock();
+    }
+}
+
 void acceptUnixHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
     int cfd, max = MAX_ACCEPTS_PER_CALL;
     UNUSED(el);
     UNUSED(mask);
     UNUSED(privdata);
 
+    int iel = ielFromEventLoop(el);
     while(max--) {
         cfd = anetUnixAccept(serverTL->neterr, fd);
         if (cfd == ANET_ERR) {
@@ -1144,23 +1217,8 @@ void acceptUnixHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
                     "Accepting client connection: %s", serverTL->neterr);
             return;
         }
-        int ielCur = ielFromEventLoop(el);
         serverLog(LL_VERBOSE,"Accepted connection to %s", g_pserver->unixsocket);
-
+        acceptCommonHandler(connCreateAcceptedSocket(cfd),CLIENT_UNIX_SOCKET,NULL,iel);
-        aeAcquireLock();
-        int ielTarget = rand() % cserver.cthreads;
-        if (ielTarget == ielCur)
-        {
-            acceptCommonHandler(cfd,CLIENT_UNIX_SOCKET,NULL, ielCur);
-        }
-        else
-        {
-            aePostFunction(g_pserver->rgthreadvar[ielTarget].el, [cfd, ielTarget]{
-                acceptCommonHandler(cfd,CLIENT_UNIX_SOCKET,NULL, ielTarget);
-            });
-        }
-        aeReleaseLock();
-        
     }
 }
 
@@ -1199,16 +1257,16 @@ void disconnectSlaves(void) {
 void unlinkClient(client *c) {
     listNode *ln;
     AssertCorrectThread(c);
-    serverAssert(c->fd == -1 || GlobalLocksAcquired());
+    serverAssert(c->conn == nullptr || GlobalLocksAcquired());
-    serverAssert(c->fd == -1 || c->lock.fOwnLock());
+    serverAssert(c->conn == nullptr || c->lock.fOwnLock());
 
     /* If this is marked as current client unset it. */
     if (serverTL && serverTL->current_client == c) serverTL->current_client = NULL;
 
-    /* Certain operations must be done only if the client has an active socket.
+    /* Certain operations must be done only if the client has an active connection.
      * If the client was already unlinked or if it's a "fake client" the
-     * fd is already set to -1. */
+     * conn is already set to NULL. */
-    if (c->fd != -1) {
+    if (c->conn) {
         /* Remove from the list of active clients. */
         if (c->client_list_node) {
             uint64_t id = htonu64(c->id);
@@ -1217,22 +1275,23 @@ void unlinkClient(client *c) {
             c->client_list_node = NULL;
         }
 
-        /* In the case of diskless replication the fork is writing to the
+        /* Check if this is a replica waiting for diskless replication (rdb pipe),
-         * sockets and just closing the fd isn't enough, if we don't also
+         * in which case it needs to be cleaned from that list */
-         * shutdown the socket the fork will continue to write to the replica
+        if (c->flags & CLIENT_SLAVE &&
-         * and the salve will only find out that it was disconnected when
+            c->replstate == SLAVE_STATE_WAIT_BGSAVE_END &&
-         * it will finish reading the rdb. */
+            g_pserver->rdb_pipe_conns)
-        if ((c->flags & CLIENT_SLAVE) &&
+        {
-            (c->replstate == SLAVE_STATE_WAIT_BGSAVE_END)) {
+            int i;
-            shutdown(c->fd, SHUT_RDWR);
+            for (i=0; i < g_pserver->rdb_pipe_numconns; i++) {
+                if (g_pserver->rdb_pipe_conns[i] == c->conn) {
+                    rdbPipeWriteHandlerConnRemoved(c->conn);
+                    g_pserver->rdb_pipe_conns[i] = NULL;
+                    break;
                 }
-
+            }
-        /* Unregister async I/O handlers and close the socket. */
+        }
-        aeDeleteFileEvent(g_pserver->rgthreadvar[c->iel].el,c->fd,AE_READABLE);
+        connClose(c->conn);
-        aeDeleteFileEvent(g_pserver->rgthreadvar[c->iel].el,c->fd,AE_WRITABLE);
+        c->conn = NULL;
-        close(c->fd);
-        c->fd = -1;
-
         atomicDecr(g_pserver->rgthreadvar[c->iel].cclients, 1);
     }
 
@@ -1277,7 +1336,7 @@ void unlinkClient(client *c) {
 
 bool freeClient(client *c) {
     listNode *ln;
-    serverAssert(c->fd == -1 || GlobalLocksAcquired());
+    serverAssert(c->conn == nullptr || GlobalLocksAcquired());
     AssertCorrectThread(c);
     std::unique_lock<decltype(c->lock)> ulock(c->lock);
 
@@ -1288,6 +1347,16 @@ bool freeClient(client *c) {
         return false;
     }
 
+    /* For connected clients, call the disconnection event of modules hooks. */
+    if (c->conn) {
+        moduleFireServerEvent(REDISMODULE_EVENT_CLIENT_CHANGE,
+                              REDISMODULE_SUBEVENT_CLIENT_CHANGE_DISCONNECTED,
+                              c);
+    }
+
+    /* Notify module system that this client auth status changed. */
+    moduleNotifyUserChanged(c);
+
     /* If it is our master that's beging disconnected we should make sure
      * to cache the state to try a partial resynchronization later.
      *
@@ -1355,6 +1424,11 @@ bool freeClient(client *c) {
         if (c->flags & CLIENT_SLAVE && listLength(g_pserver->slaves) == 0)
             g_pserver->repl_no_slaves_since = g_pserver->unixtime;
         refreshGoodSlavesCount();
+        /* Fire the replica change modules event. */
+        if (c->replstate == SLAVE_STATE_ONLINE)
+            moduleFireServerEvent(REDISMODULE_EVENT_REPLICA_CHANGE,
+                                  REDISMODULE_SUBEVENT_REPLICA_CHANGE_OFFLINE,
+                                  NULL);
     }
 
     /* Master/replica cleanup Case 2:
@@ -1437,12 +1511,13 @@ client *lookupClientByID(uint64_t id) {
 
 /* Write data in output buffers to client. Return C_OK if the client
  * is still valid after the call, C_ERR if it was freed because of some
- * error.
+ * error.  If handler_installed is set, it will attempt to clear the
+ * write event.
  *
  * This function is called by threads, but always with handler_installed
  * set to 0. So when handler_installed is set to 0 the function must be
  * thread safe. */
-int writeToClient(int fd, client *c, int handler_installed) {
+int writeToClient(client *c, int handler_installed) {
     ssize_t nwritten = 0, totwritten = 0;
     clientReplyBlock *o;
     AssertCorrectThread(c);
@@ -1451,8 +1526,7 @@ int writeToClient(int fd, client *c, int handler_installed) {
    
     while(clientHasPendingReplies(c)) {
         if (c->bufpos > 0) {
-            nwritten = write(fd,c->buf+c->sentlen,c->bufpos-c->sentlen);
+            nwritten = connWrite(c->conn,c->buf+c->sentlen,c->bufpos-c->sentlen);
-
             if (nwritten <= 0) break;
             c->sentlen += nwritten;
             totwritten += nwritten;
@@ -1471,10 +1545,8 @@ int writeToClient(int fd, client *c, int handler_installed) {
                 continue;
             }
 
-            nwritten = write(fd, o->buf() + c->sentlen, o->used - c->sentlen);
+            nwritten = connWrite(c->conn, o->buf() + c->sentlen, o->used - c->sentlen);
-            if (nwritten <= 0)
+            if (nwritten <= 0) break;
-                break;
-                
             c->sentlen += nwritten;
             totwritten += nwritten;
             
@@ -1509,11 +1581,11 @@ int writeToClient(int fd, client *c, int handler_installed) {
     
     g_pserver->stat_net_output_bytes += totwritten;
     if (nwritten == -1) {
-        if (errno == EAGAIN) {
+        if (connGetState(c->conn) == CONN_STATE_CONNECTED) {
             nwritten = 0;
         } else {
             serverLog(LL_VERBOSE,
-                "Error writing to client: %s", strerror(errno));
+                "Error writing to client: %s", connGetLastError(c->conn));
             freeClientAsync(c);
             
             return C_ERR;
@@ -1528,7 +1600,7 @@ int writeToClient(int fd, client *c, int handler_installed) {
     }
     if (!clientHasPendingReplies(c)) {
         c->sentlen = 0;
-        if (handler_installed) aeDeleteFileEvent(g_pserver->rgthreadvar[c->iel].el,c->fd,AE_WRITABLE);
+        if (handler_installed) connSetWriteHandler(c->conn, NULL);
 
         /* Close connection after entire reply has been sent. */
         if (c->flags & CLIENT_CLOSE_AFTER_REPLY) {
@@ -1540,12 +1612,9 @@ int writeToClient(int fd, client *c, int handler_installed) {
 }
 
 /* Write event handler. Just send data to the client. */
-void sendReplyToClient(aeEventLoop *el, int fd, void *privdata, int mask) {
+void sendReplyToClient(connection *conn) {
-    UNUSED(mask);
+    client *c = (client*)connGetPrivateData(conn);
-    client *c = (client*)privdata;
+    if (writeToClient(c,1) == C_ERR)
-
-    serverAssert(ielFromEventLoop(el) == c->iel);
-    if (writeToClient(fd,c,1) == C_ERR)
     {
         AeLocker ae;
         c->lock.lock();
@@ -1633,7 +1702,7 @@ void ProcessPendingAsyncWrites()
                         std::lock_guard<decltype(c->lock)> lock(c->lock);
                         serverAssert(c->casyncOpsPending > 0);
                         c->casyncOpsPending--;
-                        aeCreateFileEvent(g_pserver->rgthreadvar[c->iel].el, c->fd, AE_WRITABLE|AE_WRITE_THREADSAFE, sendReplyToClient, c);
+                        connSetWriteHandler(c->conn, sendReplyToClient, true);
                     }, false) == AE_ERR
                 )
                 {
@@ -1682,7 +1751,7 @@ int handleClientsWithPendingWrites(int iel) {
         std::unique_lock<decltype(c->lock)> lock(c->lock);
 
         /* Try to write buffers to the client socket. */
-        if (writeToClient(c->fd,c,0) == C_ERR) 
+        if (writeToClient(c,0) == C_ERR) 
         {
             if (c->flags & CLIENT_CLOSE_ASAP)
             {
@@ -1698,7 +1767,7 @@ int handleClientsWithPendingWrites(int iel) {
         /* If after the synchronous writes above we still have data to
          * output to the client, we need to install the writable handler. */
         if (clientHasPendingReplies(c)) {
-            if (aeCreateFileEvent(g_pserver->rgthreadvar[c->iel].el, c->fd, ae_flags, sendReplyToClient, c) == AE_ERR)
+            if (connSetWriteHandlerWithBarrier(c->conn, sendReplyToClient, ae_flags, true) == C_ERR) 
                 freeClientAsync(c);
         }
     }
@@ -1753,8 +1822,8 @@ void resetClient(client *c) {
 void protectClient(client *c) {
     c->flags |= CLIENT_PROTECTED;
     AssertCorrectThread(c);
-    aeDeleteFileEvent(g_pserver->rgthreadvar[c->iel].el,c->fd,AE_READABLE);
+    connSetReadHandler(c->conn,NULL);
-    aeDeleteFileEvent(g_pserver->rgthreadvar[c->iel].el,c->fd,AE_WRITABLE);
+    connSetWriteHandler(c->conn,NULL);
 }
 
 /* This will undo the client protection done by protectClient() */
@@ -1762,7 +1831,7 @@ void unprotectClient(client *c) {
     AssertCorrectThread(c);
     if (c->flags & CLIENT_PROTECTED) {
         c->flags &= ~CLIENT_PROTECTED;
-        aeCreateFileEvent(g_pserver->rgthreadvar[c->iel].el,c->fd,AE_READABLE|AE_READ_THREADSAFE,readQueryFromClient,c);
+        connSetReadHandler(c->conn,readQueryFromClient, true);
         if (clientHasPendingReplies(c)) clientInstallWriteHandler(c);
     }
 }
@@ -1824,12 +1893,8 @@ int processInlineBuffer(client *c) {
 
     /* Create redis objects for all arguments. */
     for (c->argc = 0, j = 0; j < argc; j++) {
-        if (sdslen(argv[j])) {
         c->argv[c->argc] = createObject(OBJ_STRING,argv[j]);
         c->argc++;
-        } else {
-            sdsfree(argv[j]);
-        }
     }
     sds_free(argv);
     return C_OK;
@@ -2143,14 +2208,13 @@ void processInputBufferAndReplicate(client *c) {
     }
 }
 
-void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {
+void readQueryFromClient(connection *conn) {
-    client *c = (client*) privdata;
+    client *c = (client*)connGetPrivateData(conn);
+    serverAssert(conn == c->conn);
     int nread, readlen;
     size_t qblen;
-    UNUSED(el);
+
-    UNUSED(mask);
+    serverAssert(FCorrectThread(c));
-    serverAssert(mask & AE_READ_THREADSAFE);
-    serverAssert(c->iel == ielFromEventLoop(el));
     
     AeLocker aelock;
     AssertCorrectThread(c);
@@ -2179,13 +2243,13 @@ void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {
     if (c->querybuf_peak < qblen) c->querybuf_peak = qblen;
     c->querybuf = sdsMakeRoomFor(c->querybuf, readlen);
 
-    nread = read(fd, c->querybuf+qblen, readlen);
+    nread = connRead(c->conn, c->querybuf+qblen, readlen);
     
     if (nread == -1) {
-        if (errno == EAGAIN) {
+        if (connGetState(conn) == CONN_STATE_CONNECTED) {
             return;
         } else {
-            serverLog(LL_VERBOSE, "Reading from client: %s",strerror(errno));
+            serverLog(LL_VERBOSE, "Reading from client: %s",connGetLastError(c->conn));
             freeClientAsync(c);
             return;
         }
@@ -2266,7 +2330,7 @@ void genClientPeerId(client *client, char *peerid,
         snprintf(peerid,peerid_len,"%s:0",g_pserver->unixsocket);
     } else {
         /* TCP client. */
-        anetFormatPeer(client->fd,peerid,peerid_len);
+        connFormatPeer(client->conn,peerid,peerid_len);
     }
 }
 
@@ -2287,8 +2351,7 @@ char *getClientPeerId(client *c) {
 /* Concatenate a string representing the state of a client in an human
  * readable format, into the sds string 's'. */
 sds catClientInfoString(sds s, client *client) {
-    char flags[16], events[3], *p;
+    char flags[16], events[3], conninfo[CONN_INFO_LEN], *p;
-    int emask;
 
     p = flags;
     if (client->flags & CLIENT_SLAVE) {
@@ -2312,17 +2375,18 @@ sds catClientInfoString(sds s, client *client) {
     if (p == flags) *p++ = 'N';
     *p++ = '\0';
 
-    emask = client->fd == -1 ? 0 : aeGetFileEvents(g_pserver->rgthreadvar[client->iel].el,client->fd);
     p = events;
-    if (emask & AE_READABLE) *p++ = 'r';
+    if (client->conn) {
-    if (emask & AE_WRITABLE) *p++ = 'w';
+        if (connHasReadHandler(client->conn)) *p++ = 'r';
+        if (connHasWriteHandler(client->conn)) *p++ = 'w';
+    }
     *p = '\0';
     return sdscatfmt(s,
-        "id=%U addr=%s fd=%i name=%s age=%I idle=%I flags=%s db=%i sub=%i psub=%i multi=%i qbuf=%U qbuf-free=%U obl=%U oll=%U omem=%U events=%s cmd=%s",
+        "id=%U addr=%s %s name=%s age=%I idle=%I flags=%s db=%i sub=%i psub=%i multi=%i qbuf=%U qbuf-free=%U obl=%U oll=%U omem=%U events=%s cmd=%s user=%s",
         (unsigned long long) client->id,
         getClientPeerId(client),
-        client->fd,
+        connGetInfo(client->conn, conninfo, sizeof(conninfo)),
-        client->name ? (char*)ptrFromObj(client->name) : "",
+        client->name ? (char*)szFromObj(client->name) : "",
         (long long)(g_pserver->unixtime - client->ctime),
         (long long)(g_pserver->unixtime - client->lastinteraction),
         flags,
@@ -2336,7 +2400,8 @@ sds catClientInfoString(sds s, client *client) {
         (unsigned long long) listLength(client->reply),
         (unsigned long long) getClientOutputBufferMemoryUsage(client),
         events,
-        client->lastcmd ? client->lastcmd->name : "NULL");
+        client->lastcmd ? client->lastcmd->name : "NULL",
+        client->puser ? client->puser->name : "(superuser)");
 }
 
 sds getAllClientsInfoString(int type) {
@@ -2374,7 +2439,6 @@ int clientSetNameOrReply(client *c, robj *name) {
     if (len == 0) {
         if (c->name) decrRefCount(c->name);
         c->name = NULL;
-        addReply(c,shared.ok);
         return C_OK;
     }
 
@@ -2402,20 +2466,21 @@ void clientCommand(client *c) {
 
     if (c->argc == 2 && !strcasecmp((const char*)ptrFromObj(c->argv[1]),"help")) {
         const char *help[] = {
-"id                     -- Return the ID of the current connection.",
+"ID                     -- Return the ID of the current connection.",
-"getname                -- Return the name of the current connection.",
+"GETNAME                -- Return the name of the current connection.",
-"kill <ip:port>         -- Kill connection made from <ip:port>.",
+"KILL <ip:port>         -- Kill connection made from <ip:port>.",
-"kill <option> <value> [option value ...] -- Kill connections. Options are:",
+"KILL <option> <value> [option value ...] -- Kill connections. Options are:",
-"     addr <ip:port>                      -- Kill connection made from <ip:port>",
+"     ADDR <ip:port>                      -- Kill connection made from <ip:port>",
-"     type (normal|master|replica|pubsub) -- Kill connections by type.",
+"     TYPE (normal|master|replica|pubsub) -- Kill connections by type.",
-"     skipme (yes|no)   -- Skip killing current connection (default: yes).",
+"     SKIPME (yes|no)   -- Skip killing current connection (default: yes).",
-"list [options ...]     -- Return information about client connections. Options:",
+"LIST [options ...]     -- Return information about client connections. Options:",
-"     type (normal|master|replica|pubsub) -- Return clients of specified type.",
+"     TYPE (normal|master|replica|pubsub) -- Return clients of specified type.",
-"pause <timeout>        -- Suspend all Redis clients for <timout> milliseconds.",
+"PAUSE <timeout>        -- Suspend all Redis clients for <timout> milliseconds.",
-"reply (on|off|skip)    -- Control the replies sent to the current connection.",
+"REPLY (on|off|skip)    -- Control the replies sent to the current connection.",
-"setname <name>         -- Assign the name <name> to the current connection.",
+"SETNAME <name>         -- Assign the name <name> to the current connection.",
-"unblock <clientid> [TIMEOUT|ERROR] -- Unblock the specified blocked client.",
+"UNBLOCK <clientid> [TIMEOUT|ERROR] -- Unblock the specified blocked client.",
-"tracking (on|off) [REDIRECT <id>] -- Enable client keys tracking for client side caching.",
+"TRACKING (on|off) [REDIRECT <id>] -- Enable client keys tracking for client side caching.",
+"GETREDIR               -- Return the client ID we are redirecting to when tracking is enabled.",
 NULL
         };
         addReplyHelp(c, help);
@@ -2437,7 +2502,7 @@ NULL
             return;
         }
         sds o = getAllClientsInfoString(type);
-        addReplyBulkCBuffer(c,o,sdslen(o));
+        addReplyVerbatim(c,o,sdslen(o),"txt");
         sdsfree(o);
     } else if (!strcasecmp((const char*)ptrFromObj(c->argv[1]),"reply") && c->argc == 3) {
         /* CLIENT REPLY ON|OFF|SKIP */
@@ -2633,6 +2698,13 @@ NULL
             return;
         }
         addReply(c,shared.ok);
+    } else if (!strcasecmp(szFromObj(c->argv[1]),"getredir") && c->argc == 2) {
+        /* CLIENT GETREDIR */
+        if (c->flags & CLIENT_TRACKING) {
+            addReplyLongLong(c,c->client_tracking_redirection);
+        } else {
+            addReplyLongLong(c,-1);
+        }
     } else {
         addReplyErrorFormat(c, "Unknown subcommand or wrong number of arguments for '%s'. Try CLIENT HELP", (char*)ptrFromObj(c->argv[1]));
     }
@@ -2898,7 +2970,7 @@ int checkClientOutputBufferLimits(client *c) {
  * called from contexts where the client can't be freed safely, i.e. from the
  * lower level functions pushing data inside the client output buffers. */
 void asyncCloseClientOnOutputBufferLimitReached(client *c) {
-    if (c->fd == -1) return; /* It is unsafe to free fake clients. */
+    if (!c->conn) return; /* It is unsafe to free fake clients. */
     serverAssert(c->reply_bytes < SIZE_MAX-(1024*64));
     if (c->reply_bytes == 0 || c->flags & CLIENT_CLOSE_ASAP) return;
     if (checkClientOutputBufferLimits(c)) {
@@ -2922,23 +2994,33 @@ void flushSlavesOutputBuffers(void) {
     listRewind(g_pserver->slaves,&li);
     while((ln = listNext(&li))) {
         client *replica = (client*)listNodeValue(ln);
-        int events;
 
         if (!FCorrectThread(replica))
             continue;   // we cannot synchronously flush other thread's clients
 
-        /* Note that the following will not flush output buffers of slaves
+        int can_receive_writes = connHasWriteHandler(replica->conn) ||
-         * in STATE_ONLINE but having put_online_on_ack set to true: in this
+                                 (replica->flags & CLIENT_PENDING_WRITE);
-         * case the writable event is never installed, since the purpose
+
-         * of put_online_on_ack is to postpone the moment it is installed.
+        /* We don't want to send the pending data to the replica in a few
-         * This is what we want since slaves in this state should not receive
+         * cases:
-         * writes before the first ACK. */
+         *
-        events = aeGetFileEvents(g_pserver->rgthreadvar[replica->iel].el,replica->fd);
+         * 1. For some reason there is neither the write handler installed
-        if (events & AE_WRITABLE &&
+         *    nor the client is flagged as to have pending writes: for some
-            replica->replstate == SLAVE_STATE_ONLINE &&
+         *    reason this replica may not be set to receive data. This is
+         *    just for the sake of defensive programming.
+         *
+         * 2. The put_online_on_ack flag is true. To know why we don't want
+         *    to send data to the replica in this case, please grep for the
+         *    flag for this flag.
+         *
+         * 3. Obviously if the slave is not ONLINE.
+         */
+        if (replica->replstate == SLAVE_STATE_ONLINE &&
+            can_receive_writes &&
+            !replica->repl_put_online_on_ack &&
             clientHasPendingReplies(replica))
         {
-            writeToClient(replica->fd,replica,0);
+            writeToClient(replica,0);
         }
     }
 }
@@ -3028,6 +3110,7 @@ int processEventsWhileBlocked(int iel) {
         c->lock.unlock();
     }
     aeReleaseLock();
+    serverAssertDebug(!GlobalLocksAcquired());
     while (iterations--) {
         int events = 0;
         events += aeProcessEvents(g_pserver->rgthreadvar[iel].el, AE_FILE_EVENTS|AE_DONT_WAIT);

src/object.cpp

@@ -75,6 +75,12 @@ robj *makeObjectShared(robj *o) {
     return o;
 }
 
+robj *makeObjectShared(const char *rgch, size_t cch)
+{
+    robj *o = createObject(OBJ_STRING,sdsnewlen(rgch, cch));
+    return makeObjectShared(o);
+}
+
 /* Create a string object with encoding OBJ_ENCODING_RAW, that is a plain
  * string object where ptrFromObj(o) points to a proper sds string. */
 robj *createRawStringObject(const char *ptr, size_t len) {
@@ -185,7 +191,7 @@ robj *createStringObjectFromLongLongForValue(long long value) {
  * The 'humanfriendly' option is used for INCRBYFLOAT and HINCRBYFLOAT. */
 robj *createStringObjectFromLongDouble(long double value, int humanfriendly) {
     char buf[MAX_LONG_DOUBLE_CHARS];
-    int len = ld2string(buf,sizeof(buf),value,humanfriendly);
+    int len = ld2string(buf,sizeof(buf),value,humanfriendly? LD_STR_HUMAN: LD_STR_AUTO);
     return createStringObject(buf,len);
 }
 
@@ -477,10 +483,15 @@ robj *tryObjectEncoding(robj *o) {
             incrRefCount(shared.integers[value]);
             return shared.integers[value];
         } else {
-            if (o->encoding == OBJ_ENCODING_RAW) sdsfree(szFromObj(o));
+            if (o->encoding == OBJ_ENCODING_RAW) {
+                sdsfree(szFromObj(o));
                 o->encoding = OBJ_ENCODING_INT;
                 o->m_ptr = (void*) value;
                 return o;
+            } else if (o->encoding == OBJ_ENCODING_EMBSTR) {
+                decrRefCount(o);
+                return createStringObjectFromLongLongForValue(value);
+            }
         }
     }
 
@@ -615,21 +626,13 @@ size_t stringObjectLen(robj_roptr o) {
 
 int getDoubleFromObject(const robj *o, double *target) {
     double value;
-    char *eptr;
 
     if (o == NULL) {
         value = 0;
     } else {
         serverAssertWithInfo(NULL,o,o->type == OBJ_STRING);
         if (sdsEncodedObject(o)) {
-            errno = 0;
+            if (!string2d(szFromObj(o), sdslen(szFromObj(o)), &value))
-            value = strtod(szFromObj(o), &eptr);
-            if (sdslen(szFromObj(o)) == 0 ||
-                isspace(((const char*)szFromObj(o))[0]) ||
-                (size_t)(eptr-(char*)szFromObj(o)) != sdslen(szFromObj(o)) ||
-                (errno == ERANGE &&
-                    (value == HUGE_VAL || value == -HUGE_VAL || value == 0)) ||
-                std::isnan(value))
                 return C_ERR;
         } else if (o->encoding == OBJ_ENCODING_INT) {
             value = (long)ptrFromObj(o);
@@ -876,7 +879,9 @@ size_t objectComputeSize(robj *o, size_t sample_size) {
             d = ((zset*)ptrFromObj(o))->pdict;
             zskiplist *zsl = ((zset*)ptrFromObj(o))->zsl;
             zskiplistNode *znode = zsl->header->level(0)->forward;
-            asize = sizeof(*o)+sizeof(zset)+(sizeof(struct dictEntry*)*dictSlots(d));
+            asize = sizeof(*o)+sizeof(zset)+sizeof(zskiplist)+sizeof(dict)+
+                    (sizeof(struct dictEntry*)*dictSlots(d))+
+                    zmalloc_size(zsl->header);
             while(znode != NULL && samples < sample_size) {
                 elesize += sdsAllocSize(znode->ele);
                 elesize += sizeof(struct dictEntry) + zmalloc_size(znode);
@@ -1247,19 +1252,20 @@ sds getMemoryDoctorReport(void) {
  * The lru_idle and lru_clock args are only relevant if policy
  * is MAXMEMORY_FLAG_LRU.
  * Either or both of them may be <0, in that case, nothing is set. */
-void objectSetLRUOrLFU(robj *val, long long lfu_freq, long long lru_idle,
+int objectSetLRUOrLFU(robj *val, long long lfu_freq, long long lru_idle,
-                       long long lru_clock) {
+                       long long lru_clock, int lru_multiplier) {
     if (g_pserver->maxmemory_policy & MAXMEMORY_FLAG_LFU) {
         if (lfu_freq >= 0) {
             serverAssert(lfu_freq <= 255);
             val->lru = (LFUGetTimeInMinutes()<<8) | lfu_freq;
+            return 1;
         }
     } else if (lru_idle >= 0) {
         /* Provided LRU idle time is in seconds. Scale
          * according to the LRU clock resolution this Redis
          * instance was compiled with (normally 1000 ms, so the
          * below statement will expand to lru_idle*1000/1000. */
-        lru_idle = lru_idle*1000/LRU_CLOCK_RESOLUTION;
+        lru_idle = lru_idle*lru_multiplier/LRU_CLOCK_RESOLUTION;
         long lru_abs = lru_clock - lru_idle; /* Absolute access time. */
         /* If the LRU field underflows (since LRU it is a wrapping
          * clock), the best we can do is to provide a large enough LRU
@@ -1269,7 +1275,9 @@ void objectSetLRUOrLFU(robj *val, long long lfu_freq, long long lru_idle,
         if (lru_abs < 0)
             lru_abs = (lru_clock+(LRU_CLOCK_MAX/2)) % LRU_CLOCK_MAX;
         val->lru = lru_abs;
+        return 1;
     }
+    return 0;
 }
 
 /* ======================= The OBJECT and MEMORY commands =================== */
@@ -1482,30 +1490,20 @@ NULL
 #if defined(USE_JEMALLOC)
         sds info = sdsempty();
         je_malloc_stats_print(inputCatSds, &info, NULL);
-        addReplyBulkSds(c, info);
+        addReplyVerbatim(c,info,sdslen(info),"txt");
+        sdsfree(info);
 #else
         addReplyBulkCString(c,"Stats not supported for the current allocator");
 #endif
     } else if (!strcasecmp(szFromObj(c->argv[1]),"doctor") && c->argc == 2) {
         sds report = getMemoryDoctorReport();
-        addReplyBulkSds(c,report);
+        addReplyVerbatim(c,report,sdslen(report),"txt");
+        sdsfree(report);
     } else if (!strcasecmp(szFromObj(c->argv[1]),"purge") && c->argc == 2) {
-#if defined(USE_JEMALLOC)
+        if (jemalloc_purge() == 0)
-        char tmp[32];
-        unsigned narenas = 0;
-        size_t sz = sizeof(unsigned);
-        if (!je_mallctl("arenas.narenas", &narenas, &sz, NULL, 0)) {
-            sprintf(tmp, "arena.%d.purge", narenas);
-            if (!je_mallctl(tmp, NULL, 0, NULL, 0)) {
             addReply(c, shared.ok);
-                return;
+        else
-            }
-        }
             addReplyError(c, "Error purging dirty pages");
-#else
-        addReply(c, shared.ok);
-        /* Nothing to do for other allocators. */
-#endif
     } else {
         addReplyErrorFormat(c, "Unknown subcommand or wrong number of arguments for '%s'. Try MEMORY HELP", (char*)ptrFromObj(c->argv[1]));
     }

src/rax.c

@@ -1673,6 +1673,7 @@ int raxSeek(raxIterator *it, const char *op, unsigned char *ele, size_t len) {
                  * node, but will be our match, representing the key "f".
                  *
                  * So in that case, we don't seek backward. */
+                it->data = raxGetData(it->node);
             } else {
                 if (gt && !raxIteratorNextStep(it,0)) return 0;
                 if (lt && !raxIteratorPrevStep(it,0)) return 0;
@@ -1791,7 +1792,8 @@ int raxCompare(raxIterator *iter, const char *op, unsigned char *key, size_t key
         if (eq && key_len == iter->key_len) return 1;
         else if (lt) return iter->key_len < key_len;
         else if (gt) return iter->key_len > key_len;
-    } if (cmp > 0) {
+        else return 0; /* Avoid warning, just 'eq' is handled before. */
+    } else if (cmp > 0) {
         return gt ? 1 : 0;
     } else /* (cmp < 0) */ {
         return lt ? 1 : 0;

src/rdb-s3.cpp

@@ -58,22 +58,22 @@ int rdbSaveS3(char *s3bucket, rdbSaveInfo *rsi)
 }
 
 
-int rdbLoadS3Core(int fd, rdbSaveInfo *rsi) 
+int rdbLoadS3Core(int fd, rdbSaveInfo *rsi, int rdbflags) 
 {
     FILE *fp;
     rio rdb;
     int retval;
 
     if ((fp = fdopen(fd, "rb")) == NULL) return C_ERR;
-    startLoading(fp);
+    startLoading(0, rdbflags);
     rioInitWithFile(&rdb,fp);
-    retval = rdbLoadRio(&rdb,rsi,0);
+    retval = rdbLoadRio(&rdb,rdbflags,rsi);
     fclose(fp);
-    stopLoading();
+    stopLoading(retval == C_OK);
     return retval;
 }
 
-int rdbLoadS3(char *s3bucket, rdbSaveInfo *rsi)
+int rdbLoadS3(char *s3bucket, rdbSaveInfo *rsi, int rdbflags)
 {
     int status = EXIT_FAILURE;
     int fd[2];
@@ -100,7 +100,7 @@ int rdbLoadS3(char *s3bucket, rdbSaveInfo *rsi)
     else
     {
         close(fd[1]);
-        if (rdbLoadS3Core(fd[0], rsi) != C_OK)
+        if (rdbLoadS3Core(fd[0], rsi, rdbflags) != C_OK)
         {
             close(fd[0]);
             return C_ERR;

src/rdb.h

@@ -124,8 +124,10 @@
 #define RDB_LOAD_PLAIN  (1<<1)
 #define RDB_LOAD_SDS    (1<<2)
 
-#define RDB_SAVE_NONE 0
+/* flags on the purpose of rdb save or load */
-#define RDB_SAVE_AOF_PREAMBLE (1<<0)
+#define RDBFLAGS_NONE 0
+#define RDBFLAGS_AOF_PREAMBLE (1<<0)
+#define RDBFLAGS_REPLICATION (1<<1)
 
 int rdbSaveType(rio *rdb, unsigned char type);
 int rdbLoadType(rio *rdb);
@@ -138,8 +140,8 @@ uint64_t rdbLoadLen(rio *rdb, int *isencoded);
 int rdbLoadLenByRef(rio *rdb, int *isencoded, uint64_t *lenptr);
 int rdbSaveObjectType(rio *rdb, robj_roptr o);
 int rdbLoadObjectType(rio *rdb);
-int rdbLoad(rdbSaveInfo *rsi);
+int rdbLoad(rdbSaveInfo *rsi, int rdbflags);
-int rdbLoadFile(const char *filename, rdbSaveInfo *rsi);
+int rdbLoadFile(const char *filename, rdbSaveInfo *rsi, int rdbflags);
 int rdbSaveBackground(rdbSaveInfo *rsi);
 int rdbSaveToSlavesSockets(rdbSaveInfo *rsi);
 void rdbRemoveTempFile(pid_t childpid);
@@ -147,12 +149,13 @@ int rdbSave(rdbSaveInfo *rsi);
 int rdbSaveFile(char *filename, rdbSaveInfo *rsi);
 int rdbSaveFp(FILE *pf, rdbSaveInfo *rsi);
 int rdbSaveS3(char *path, rdbSaveInfo *rsi);
-int rdbLoadS3(char *path, rdbSaveInfo *rsi);
+int rdbLoadS3(char *path, rdbSaveInfo *rsi, int rdbflags);
 ssize_t rdbSaveObject(rio *rdb, robj_roptr o, robj *key);
 size_t rdbSavedObjectLen(robj *o);
 robj *rdbLoadObject(int type, rio *rdb, robj *key, uint64_t mvcc_tstamp);
 void backgroundSaveDoneHandler(int exitcode, int bysignal);
 int rdbSaveKeyValuePair(rio *rdb, robj *key, robj *val, long long expiretime);
+ssize_t rdbSaveSingleModuleAux(rio *rdb, int when, moduleType *mt);
 robj *rdbLoadStringObject(rio *rdb);
 ssize_t rdbSaveStringObject(rio *rdb, robj_roptr obj);
 ssize_t rdbSaveRawString(rio *rdb, const unsigned char *s, size_t len);
@@ -161,7 +164,8 @@ int rdbSaveBinaryDoubleValue(rio *rdb, double val);
 int rdbLoadBinaryDoubleValue(rio *rdb, double *val);
 int rdbSaveBinaryFloatValue(rio *rdb, float val);
 int rdbLoadBinaryFloatValue(rio *rdb, float *val);
-int rdbLoadRio(rio *rdb, rdbSaveInfo *rsi, int loading_aof);
+int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi);
+int rdbSaveRio(rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi);
 rdbSaveInfo *rdbPopulateSaveInfo(rdbSaveInfo *rsi);
 
 #endif

src/redis-benchmark.cpp

@@ -106,6 +106,7 @@ static struct config {
     int is_fetching_slots;
     int is_updating_slots;
     int slots_last_update;
+    int enable_tracking;
     /* Thread mutexes to be used as fallbacks by atomicvar.h */
     pthread_mutex_t requests_issued_mutex;
     pthread_mutex_t requests_finished_mutex;
@@ -638,6 +639,14 @@ static client createClient(const char *cmd, size_t len, client from, int thread_
         c->prefix_pending++;
     }
 
+    if (config.enable_tracking) {
+        char *buf = NULL;
+        int len = redisFormatCommand(&buf, "CLIENT TRACKING on");
+        c->obuf = sdscatlen(c->obuf, buf, len);
+        free(buf);
+        c->prefix_pending++;
+    }
+
     /* If a DB number different than zero is selected, prefix our request
      * buffer with the SELECT command, that will be discarded the first
      * time the replies are received, so if the client is reused the
@@ -1356,6 +1365,8 @@ int parseOptions(int argc, const char **argv) {
              } else if (config.num_threads < 0) config.num_threads = 0;
         } else if (!strcmp(argv[i],"--cluster")) {
             config.cluster_mode = 1;
+        } else if (!strcmp(argv[i],"--enable-tracking")) {
+            config.enable_tracking = 1;
         } else if (!strcmp(argv[i],"--help")) {
             exit_status = 0;
             goto usage;
@@ -1386,6 +1397,7 @@ usage:
 " --dbnum <db>       SELECT the specified db number (default 0)\n"
 " --threads <num>    Enable multi-thread mode.\n"
 " --cluster          Enable cluster mode.\n"
+" --enable-tracking  Send CLIENT TRACKING on before starting benchmark.\n"
 " -k <boolean>       1=keep alive 0=reconnect (default 1)\n"
 " -r <keyspacelen>   Use random keys for SET/GET/INCR, random values for SADD\n"
 "  Using this option the benchmark will expand the string __rand_int__\n"
@@ -1512,6 +1524,7 @@ int main(int argc, const char **argv) {
     config.is_fetching_slots = 0;
     config.is_updating_slots = 0;
     config.slots_last_update = 0;
+    config.enable_tracking = 0;
 
     i = parseOptions(argc,argv);
     argc -= i;

src/redis-check-rdb.cpp

@@ -202,7 +202,7 @@ int redis_check_rdb(const char *rdbfilename, FILE *fp) {
     }
 
     expiretime = -1;
-    startLoading(fp);
+    startLoadingFile(fp, rdbfilename, RDBFLAGS_NONE);
     while(1) {
         robj *key, *val;
 
@@ -216,14 +216,16 @@ int redis_check_rdb(const char *rdbfilename, FILE *fp) {
             /* EXPIRETIME: load an expire associated with the next key
              * to load. Note that after loading an expire we need to
              * load the actual type, and continue. */
-            if ((expiretime = rdbLoadTime(&rdb)) == -1) goto eoferr;
+            expiretime = rdbLoadTime(&rdb);
             expiretime *= 1000;
+            if (rioGetReadError(&rdb)) goto eoferr;
             continue; /* Read next opcode. */
         } else if (type == RDB_OPCODE_EXPIRETIME_MS) {
             /* EXPIRETIME_MS: milliseconds precision expire times introduced
              * with RDB v3. Like EXPIRETIME but no with more precision. */
             rdbstate.doing = RDB_CHECK_DOING_READ_EXPIRE;
-            if ((expiretime = rdbLoadMillisecondTime(&rdb, rdbver)) == -1) goto eoferr;
+            expiretime = rdbLoadMillisecondTime(&rdb, rdbver);
+            if (rioGetReadError(&rdb)) goto eoferr;
             continue; /* Read next opcode. */
         } else if (type == RDB_OPCODE_FREQ) {
             /* FREQ: LFU frequency. */
@@ -314,6 +316,7 @@ int redis_check_rdb(const char *rdbfilename, FILE *fp) {
     }
 
     if (closefile) fclose(fp);
+    stopLoading(1);
     return 0;
 
 eoferr: /* unexpected end of file is handled here with a fatal exit */
@@ -324,6 +327,7 @@ eoferr: /* unexpected end of file is handled here with a fatal exit */
     }
 err:
     if (closefile) fclose(fp);
+    stopLoading(0);
     return 1;
 }
 

src/redis-cli.c

@@ -49,6 +49,10 @@
 #include <pwd.h>
 
 #include <hiredis.h>
+#ifdef USE_OPENSSL
+#include <openssl/ssl.h>
+#include <hiredis_ssl.h>
+#endif
 #include <sds.h> /* use sds.h from hiredis, so that only one set of sds functions will be present in the binary */
 #include "adlist.h"
 #include "zmalloc.h"
@@ -60,7 +64,6 @@
 
 #include "redis-cli.h"
 
-
 redisContext *context;
 struct config config;
 
@@ -98,12 +101,6 @@ int g_fInCrash = 0;
 
 uint16_t crc16(const char *buf, int len);
 
-uint64_t dictSdsHash(const void *key);
-int dictSdsKeyCompare(void *privdata, const void *key1,
-        const void *key2);
-void dictSdsDestructor(void *privdata, void *val);
-void dictListDestructor(void *privdata, void *val);
-
 static long long ustime(void) {
     struct timeval tv;
     long long ust;
@@ -209,7 +206,7 @@ static sds percentDecode(const char *pe, size_t len) {
  * URI scheme is based on the the provisional specification[1] excluding support
  * for query parameters. Valid URIs are:
  *   scheme:    "redis://"
- *   authority: [<username> ":"] <password> "@"] [<hostname> [":" <port>]]
+ *   authority: [[<username> ":"] <password> "@"] [<hostname> [":" <port>]]
  *   path:      ["/" [<db>]]
  *
  *  [1]: https://www.iana.org/assignments/uri-schemes/prov/redis */
@@ -551,8 +548,13 @@ static int cliAuth(void) {
     redisReply *reply;
     if (config.auth == NULL) return REDIS_OK;
 
+    if (config.user == NULL)
         reply = redisCommand(context,"AUTH %s",config.auth);
+    else
+        reply = redisCommand(context,"AUTH %s %s",config.user,config.auth);
     if (reply != NULL) {
+        if (reply->type == REDIS_REPLY_ERROR)
+            fprintf(stderr,"Warning: AUTH failed\n");
         freeReplyObject(reply);
         return REDIS_OK;
     }
@@ -574,6 +576,86 @@ static int cliSelect(void) {
     return REDIS_ERR;
 }
 
+/* Wrapper around redisSecureConnection to avoid hiredis_ssl dependencies if
+ * not building with TLS support.
+ */
+static int cliSecureConnection(redisContext *c, const char **err) {
+#ifdef USE_OPENSSL
+    static SSL_CTX *ssl_ctx = NULL;
+
+    if (!ssl_ctx) {
+        ssl_ctx = SSL_CTX_new(SSLv23_client_method());
+        if (!ssl_ctx) {
+            *err = "Failed to create SSL_CTX";
+            goto error;
+        }
+
+        SSL_CTX_set_options(ssl_ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3);
+        SSL_CTX_set_verify(ssl_ctx, SSL_VERIFY_PEER, NULL);
+
+        if (config.cacert || config.cacertdir) {
+            if (!SSL_CTX_load_verify_locations(ssl_ctx, config.cacert, config.cacertdir)) {
+                *err = "Invalid CA Certificate File/Directory";
+                goto error;
+            }
+        } else {
+            if (!SSL_CTX_set_default_verify_paths(ssl_ctx)) {
+                *err = "Failed to use default CA paths";
+                goto error;
+            }
+        }
+
+        if (config.cert && !SSL_CTX_use_certificate_chain_file(ssl_ctx, config.cert)) {
+            *err = "Invalid client certificate";
+            goto error;
+        }
+
+        if (config.key && !SSL_CTX_use_PrivateKey_file(ssl_ctx, config.key, SSL_FILETYPE_PEM)) {
+            *err = "Invalid private key";
+            goto error;
+        }
+    }
+
+    SSL *ssl = SSL_new(ssl_ctx);
+    if (!ssl) {
+        *err = "Failed to create SSL object";
+        return REDIS_ERR;
+    }
+
+    if (config.sni && !SSL_set_tlsext_host_name(ssl, config.sni)) {
+        *err = "Failed to configure SNI";
+        SSL_free(ssl);
+        return REDIS_ERR;
+    }
+
+    return redisInitiateSSL(c, ssl);
+
+error:
+    SSL_CTX_free(ssl_ctx);
+    ssl_ctx = NULL;
+    return REDIS_ERR;
+#else
+    (void) c;
+    (void) err;
+    return REDIS_OK;
+#endif
+}
+
+/* Select RESP3 mode if redis-cli was started with the -3 option.  */
+static int cliSwitchProto(void) {
+    redisReply *reply;
+    if (config.resp3 == 0) return REDIS_OK;
+
+    reply = redisCommand(context,"HELLO 3");
+    if (reply != NULL) {
+        int result = REDIS_OK;
+        if (reply->type == REDIS_REPLY_ERROR) result = REDIS_ERR;
+        freeReplyObject(reply);
+        return result;
+    }
+    return REDIS_ERR;
+}
+
 /* Connect to the server. It is possible to pass certain flags to the function:
  *      CC_FORCE: The connection is performed even if there is already
  *                a connected socket.
@@ -590,6 +672,16 @@ static int cliConnect(int flags) {
             context = redisConnectUnix(config.hostsocket);
         }
 
+        if (!context->err && config.tls) {
+            const char *err = NULL;
+            if (cliSecureConnection(context, &err) == REDIS_ERR && err) {
+                fprintf(stderr, "Could not negotiate a TLS connection: %s\n", err);
+                context = NULL;
+                redisFree(context);
+                return REDIS_ERR;
+            }
+        }
+
         if (context->err) {
             if (!(flags & CC_QUIET)) {
                 fprintf(stderr,"Could not connect to Redis at ");
@@ -605,17 +697,20 @@ static int cliConnect(int flags) {
             return REDIS_ERR;
         }
 
+
         /* Set aggressive KEEP_ALIVE socket option in the Redis context socket
          * in order to prevent timeouts caused by the execution of long
          * commands. At the same time this improves the detection of real
          * errors. */
         anetKeepAlive(NULL, context->fd, REDIS_CLI_KEEPALIVE_INTERVAL);
 
-        /* Do AUTH and select the right DB. */
+        /* Do AUTH, select the right DB, switch to RESP3 if needed. */
         if (cliAuth() != REDIS_OK)
             return REDIS_ERR;
         if (cliSelect() != REDIS_OK)
             return REDIS_ERR;
+        if (cliSwitchProto() != REDIS_OK)
+            return REDIS_ERR;
     }
     return REDIS_OK;
 }
@@ -642,10 +737,17 @@ static sds cliFormatReplyTTY(redisReply *r, char *prefix) {
         out = sdscatprintf(out,"(double) %s\n",r->str);
     break;
     case REDIS_REPLY_STRING:
+    case REDIS_REPLY_VERB:
         /* If you are producing output for the standard output we want
-        * a more interesting output with quoted characters and so forth */
+        * a more interesting output with quoted characters and so forth,
+        * unless it's a verbatim string type. */
+        if (r->type == REDIS_REPLY_STRING) {
             out = sdscatrepr(out,r->str,r->len);
             out = sdscat(out,"\n");
+        } else {
+            out = sdscatlen(out,r->str,r->len);
+            out = sdscat(out,"\n");
+        }
     break;
     case REDIS_REPLY_NIL:
         out = sdscat(out,"(nil)\n");
@@ -784,6 +886,7 @@ static sds cliFormatReplyRaw(redisReply *r) {
         break;
     case REDIS_REPLY_STATUS:
     case REDIS_REPLY_STRING:
+    case REDIS_REPLY_VERB:
         if (r->type == REDIS_REPLY_STATUS && config.eval_ldb) {
             /* The Lua debugger replies with arrays of simple (status)
              * strings. We colorize the output for more fun if this
@@ -803,9 +906,15 @@ static sds cliFormatReplyRaw(redisReply *r) {
             out = sdscatlen(out,r->str,r->len);
         }
         break;
+    case REDIS_REPLY_BOOL:
+        out = sdscat(out,r->integer ? "(true)" : "(false)");
+    break;
     case REDIS_REPLY_INTEGER:
         out = sdscatprintf(out,"%lld",r->integer);
         break;
+    case REDIS_REPLY_DOUBLE:
+        out = sdscatprintf(out,"%s",r->str);
+        break;
     case REDIS_REPLY_ARRAY:
         for (i = 0; i < r->elements; i++) {
             if (i > 0) out = sdscat(out,config.mb_delim);
@@ -814,6 +923,19 @@ static sds cliFormatReplyRaw(redisReply *r) {
             sdsfree(tmp);
         }
         break;
+    case REDIS_REPLY_MAP:
+        for (i = 0; i < r->elements; i += 2) {
+            if (i > 0) out = sdscat(out,config.mb_delim);
+            tmp = cliFormatReplyRaw(r->element[i]);
+            out = sdscatlen(out,tmp,sdslen(tmp));
+            sdsfree(tmp);
+
+            out = sdscatlen(out," ",1);
+            tmp = cliFormatReplyRaw(r->element[i+1]);
+            out = sdscatlen(out,tmp,sdslen(tmp));
+            sdsfree(tmp);
+        }
+        break;
     default:
         fprintf(stderr,"Unknown reply type: %d\n", r->type);
         exit(1);
@@ -836,13 +958,21 @@ static sds cliFormatReplyCSV(redisReply *r) {
     case REDIS_REPLY_INTEGER:
         out = sdscatprintf(out,"%lld",r->integer);
     break;
+    case REDIS_REPLY_DOUBLE:
+        out = sdscatprintf(out,"%s",r->str);
+        break;
     case REDIS_REPLY_STRING:
+    case REDIS_REPLY_VERB:
         out = sdscatrepr(out,r->str,r->len);
     break;
     case REDIS_REPLY_NIL:
-        out = sdscat(out,"NIL");
+        out = sdscat(out,"NULL");
+    break;
+    case REDIS_REPLY_BOOL:
+        out = sdscat(out,r->integer ? "true" : "false");
     break;
     case REDIS_REPLY_ARRAY:
+    case REDIS_REPLY_MAP: /* CSV has no map type, just output flat list. */
         for (i = 0; i < r->elements; i++) {
             sds tmp = cliFormatReplyCSV(r->element[i]);
             out = sdscatlen(out,tmp,sdslen(tmp));
@@ -1036,7 +1166,8 @@ static int cliSendCommand(int argc, char **argv, long repeat) {
             if (!strcasecmp(command,"select") && argc == 2 && config.last_cmd_type != REDIS_REPLY_ERROR) {
                 config.dbnum = atoi(argv[1]);
                 cliRefreshPrompt();
-            } else if (!strcasecmp(command,"auth") && argc == 2) {
+            } else if (!strcasecmp(command,"auth") && (argc == 2 || argc == 3))
+            {
                 cliSelect();
             }
         }
@@ -1068,6 +1199,13 @@ static redisReply *reconnectingRedisCommand(redisContext *c, const char *fmt, ..
 
             redisFree(c);
             c = redisConnect(config.hostip,config.hostport);
+            if (!c->err && config.tls) {
+                const char *err = NULL;
+                if (cliSecureConnection(c, &err) == REDIS_ERR && err) {
+                    fprintf(stderr, "TLS Error: %s\n", err);
+                    exit(1);
+                }
+            }
             usleep(1000000);
         }
 
@@ -1119,8 +1257,12 @@ static int parseOptions(int argc, char **argv) {
             config.dbnum = atoi(argv[++i]);
         } else if (!strcmp(argv[i], "--no-auth-warning")) {
             config.no_auth_warning = 1;
-        } else if (!strcmp(argv[i],"-a") && !lastarg) {
+        } else if ((!strcmp(argv[i],"-a") || !strcmp(argv[i],"--pass"))
+                   && !lastarg)
+        {
             config.auth = argv[++i];
+        } else if (!strcmp(argv[i],"--user") && !lastarg) {
+            config.user = argv[++i];
         } else if (!strcmp(argv[i],"-u") && !lastarg) {
             parseRedisUri(argv[++i]);
         } else if (!strcmp(argv[i],"--raw")) {
@@ -1257,11 +1399,27 @@ static int parseOptions(int argc, char **argv) {
         } else if (!strcmp(argv[i],"--cluster-search-multiple-owners")) {
             config.cluster_manager_command.flags |=
                 CLUSTER_MANAGER_CMD_FLAG_CHECK_OWNERS;
+#ifdef USE_OPENSSL
+        } else if (!strcmp(argv[i],"--tls")) {
+            config.tls = 1;
+        } else if (!strcmp(argv[i],"--sni")) {
+            config.sni = argv[++i];
+        } else if (!strcmp(argv[i],"--cacertdir")) {
+            config.cacertdir = argv[++i];
+        } else if (!strcmp(argv[i],"--cacert")) {
+            config.cacert = argv[++i];
+        } else if (!strcmp(argv[i],"--cert")) {
+            config.cert = argv[++i];
+        } else if (!strcmp(argv[i],"--key")) {
+            config.key = argv[++i];
+#endif
         } else if (!strcmp(argv[i],"-v") || !strcmp(argv[i], "--version")) {
             sds version = cliVersion();
             printf("keydb-cli %s\n", version);
             sdsfree(version);
             exit(0);
+        } else if (!strcmp(argv[i],"-3")) {
+            config.resp3 = 1;
         } else if (CLUSTER_MANAGER_MODE() && argv[i][0] != '-') {
             if (config.cluster_manager_command.argc == 0) {
                 int j = i + 1;
@@ -1337,14 +1495,26 @@ static void usage(void) {
 "                     You can also use the " REDIS_CLI_AUTH_ENV " environment\n"
 "                     variable to pass this password more safely\n"
 "                     (if both are used, this argument takes predecence).\n"
+"  -user <username>   Used to send ACL style 'AUTH username pass'. Needs -a.\n"
+"  -pass <password>   Alias of -a for consistency with the new --user option.\n"
 "  -u <uri>           Server URI.\n"
 "  -r <repeat>        Execute specified command N times.\n"
 "  -i <interval>      When -r is used, waits <interval> seconds per command.\n"
 "                     It is possible to specify sub-second times like -i 0.1.\n"
 "  -n <db>            Database number.\n"
+"  -3                 Start session in RESP3 protocol mode.\n"
 "  -x                 Read last argument from STDIN.\n"
 "  -d <delimiter>     Multi-bulk delimiter in for raw formatting (default: \\n).\n"
 "  -c                 Enable cluster mode (follow -ASK and -MOVED redirections).\n"
+#ifdef USE_OPENSSL
+"  --tls              Establish a secure TLS connection.\n"
+"  --cacert           CA Certificate file to verify with.\n"
+"  --cacertdir        Directory where trusted CA certificates are stored.\n"
+"                     If neither cacert nor cacertdir are specified, the default\n"
+"                     system-wide trusted root certs configuration will apply.\n"
+"  --cert             Client certificate to authenticate with.\n"
+"  --key              Private key file to authenticate with.\n"
+#endif
 "  --raw              Use raw formatting for replies (default when STDOUT is\n"
 "                     not a tty).\n"
 "  --no-raw           Force formatted output even when STDOUT is not a tty.\n"
@@ -1356,7 +1526,9 @@ static void usage(void) {
 "                     --csv is specified, or if you redirect the output to a non\n"
 "                     TTY, it samples the latency for 1 second (you can use\n"
 "                     -i to change the interval), then produces a single output\n"
-"                     and exits.\n"
+"                     and exits.\n",version);
+
+    fprintf(stderr,
 "  --latency-history  Like --latency but tracking latency changes over time.\n"
 "                     Default time interval is 15 sec. Change it using -i.\n"
 "  --latency-dist     Shows latency as a spectrum, requires xterm 256 colors.\n"
@@ -1367,7 +1539,9 @@ static void usage(void) {
 "  --pipe             Transfer raw Redis protocol from stdin to server.\n"
 "  --pipe-timeout <n> In --pipe mode, abort with error if after sending all data.\n"
 "                     no reply is received within <n> seconds.\n"
-"                     Default timeout: %d. Use 0 to wait forever.\n"
+"                     Default timeout: %d. Use 0 to wait forever.\n",
+    REDIS_CLI_DEFAULT_PIPE_TIMEOUT);
+    fprintf(stderr,
 "  --bigkeys          Sample Redis keys looking for keys with many elements (complexity).\n"
 "  --memkeys          Sample Redis keys looking for keys consuming a lot of memory.\n"
 "  --memkeys-samples <n> Sample Redis keys looking for keys consuming a lot of memory.\n"
@@ -1390,8 +1564,7 @@ static void usage(void) {
 "                     line interface.\n"
 "  --help             Output this help and exit.\n"
 "  --version          Output version and exit.\n"
-"\n",
+"\n");
-    version, REDIS_CLI_DEFAULT_PIPE_TIMEOUT);
     /* Using another fprintf call to avoid -Woverlength-strings compile warning */
     fprintf(stderr,
 "Cluster Manager Commands:\n"
@@ -1580,8 +1753,20 @@ static void repl(void) {
                 }
             }
 
-            /* Won't save auth command in history file */
+            /* Won't save auth or acl setuser commands in history file */
-            if (!(argv && argc > 0 && !strcasecmp(argv[0+skipargs], "auth"))) {
+            int dangerous = 0;
+            if (argv && argc > 0) {
+                if (!strcasecmp(argv[skipargs], "auth")) {
+                    dangerous = 1;
+                } else if (skipargs+1 < argc &&
+                           !strcasecmp(argv[skipargs], "acl") &&
+                           !strcasecmp(argv[skipargs+1], "setuser"))
+                {
+                    dangerous = 1;
+                }
+            }
+
+            if (!dangerous) {
                 if (history) linenoiseHistoryAdd(line);
                 if (historyfile) linenoiseHistorySave(historyfile);
             }
@@ -2058,6 +2243,15 @@ cleanup:
 static int clusterManagerNodeConnect(clusterManagerNode *node) {
     if (node->context) redisFree(node->context);
     node->context = redisConnect(node->ip, node->port);
+    if (!node->context->err && config.tls) {
+        const char *err = NULL;
+        if (cliSecureConnection(node->context, &err) == REDIS_ERR && err) {
+            fprintf(stderr,"TLS Error: %s\n", err);
+            redisFree(node->context);
+            node->context = NULL;
+            return 0;
+        }
+    }
     if (node->context->err) {
         fprintf(stderr,"Could not connect to Redis at ");
         fprintf(stderr,"%s:%d: %s\n", node->ip, node->port,
@@ -2072,7 +2266,12 @@ static int clusterManagerNodeConnect(clusterManagerNode *node) {
      * errors. */
     anetKeepAlive(NULL, node->context->fd, REDIS_CLI_KEEPALIVE_INTERVAL);
     if (config.auth) {
-        redisReply *reply = redisCommand(node->context,"AUTH %s",config.auth);
+        redisReply *reply;
+        if (config.user == NULL)
+            reply = redisCommand(node->context,"AUTH %s", config.auth);
+        else
+            reply = redisCommand(node->context,"AUTH %s %s",
+                                 config.user,config.auth);
         int ok = clusterManagerCheckRedisReply(node, reply, NULL);
         if (reply != NULL) freeReplyObject(reply);
         if (!ok) return 0;
@@ -2848,7 +3047,7 @@ static redisReply *clusterManagerMigrateKeysInReply(clusterManagerNode *source,
         redisReply *entry = reply->element[i];
         size_t idx = i + offset;
         assert(entry->type == REDIS_REPLY_STRING);
-        argv[idx] = (char *) sdsnew(entry->str);
+        argv[idx] = (char *) sdsnewlen(entry->str, entry->len);
         argv_len[idx] = entry->len;
         if (dots) dots[i] = '.';
     }
@@ -5826,6 +6025,7 @@ static void pipeMode(void) {
         /* Handle the readable state: we can read replies from the server. */
         if (mask & AE_READABLE) {
             ssize_t nread;
+            int read_error = 0;
 
             /* Read from socket and feed the hiredis reader. */
             do {
@@ -5833,7 +6033,8 @@ static void pipeMode(void) {
                 if (nread == -1 && errno != EAGAIN && errno != EINTR) {
                     fprintf(stderr, "Error reading from the server: %s\n",
                         strerror(errno));
-                    exit(1);
+                    read_error = 1;
+                    break;
                 }
                 if (nread > 0) {
                     redisReaderFeed(reader,ibuf,nread);
@@ -5866,6 +6067,11 @@ static void pipeMode(void) {
                     freeReplyObject(reply);
                 }
             } while(reply);
+
+            /* Abort on read errors. We abort here because it is important
+             * to consume replies even after a read error: this way we can
+             * show a potential problem to the user. */
+            if (read_error) exit(1);
         }
 
         /* Handle the writable state: we can send protocol to the server. */
@@ -6685,6 +6891,7 @@ int main(int argc, char **argv) {
     config.hotkeys = 0;
     config.stdinarg = 0;
     config.auth = NULL;
+    config.user = NULL;
     config.eval = NULL;
     config.eval_ldb = 0;
     config.eval_ldb_end = 0;

src/redis-cli.h

@@ -127,6 +127,12 @@ extern struct config {
     char *hostip;
     int hostport;
     char *hostsocket;
+    int tls;
+    char *sni;
+    char *cacert;
+    char *cacertdir;
+    char *cert;
+    char *key;
     long repeat;
     long interval;
     int dbnum;
@@ -157,6 +163,7 @@ extern struct config {
     int hotkeys;
     int stdinarg; /* get last arg from stdin. (-x option) */
     char *auth;
+    char *user;
     int output; /* output mode, see OUTPUT_* defines */
     sds mb_delim;
     char prompt[128];
@@ -169,6 +176,7 @@ extern struct config {
     int verbose;
     clusterManagerCommand cluster_manager_command;
     int no_auth_warning;
+    int resp3;
 } config;
 
 /* The Cluster Manager global structure */

src/redismodule.h

@@ -22,6 +22,10 @@ extern "C" {
 #define REDISMODULE_READ (1<<0)
 #define REDISMODULE_WRITE (1<<1)
 
+/* RedisModule_OpenKey extra flags for the 'mode' argument.
+ * Avoid touching the LRU/LFU of the key when opened. */
+#define REDISMODULE_OPEN_KEY_NOTOUCH (1<<16)
+
 #define REDISMODULE_LIST_HEAD 0
 #define REDISMODULE_LIST_TAIL 1
 
@@ -33,6 +37,7 @@ extern "C" {
 #define REDISMODULE_KEYTYPE_SET 4
 #define REDISMODULE_KEYTYPE_ZSET 5
 #define REDISMODULE_KEYTYPE_MODULE 6
+#define REDISMODULE_KEYTYPE_STREAM 7
 
 /* Reply types. */
 #define REDISMODULE_REPLY_UNKNOWN -1
@@ -93,8 +98,30 @@ extern "C" {
 #define REDISMODULE_CTX_FLAGS_REPLICATED (1<<12)
 /* Redis is currently loading either from AOF or RDB. */
 #define REDISMODULE_CTX_FLAGS_LOADING (1<<13)
+/* The replica has no link with its master, note that
+ * there is the inverse flag as well:
+ *
+ *  REDISMODULE_CTX_FLAGS_REPLICA_IS_ONLINE
+ *
+ * The two flags are exclusive, one or the other can be set. */
+#define REDISMODULE_CTX_FLAGS_REPLICA_IS_STALE (1<<14)
+/* The replica is trying to connect with the master.
+ * (REPL_STATE_CONNECT and REPL_STATE_CONNECTING states) */
+#define REDISMODULE_CTX_FLAGS_REPLICA_IS_CONNECTING (1<<15)
+/* THe replica is receiving an RDB file from its master. */
+#define REDISMODULE_CTX_FLAGS_REPLICA_IS_TRANSFERRING (1<<16)
+/* The replica is online, receiving updates from its master. */
+#define REDISMODULE_CTX_FLAGS_REPLICA_IS_ONLINE (1<<17)
+/* There is currently some background process active. */
+#define REDISMODULE_CTX_FLAGS_ACTIVE_CHILD (1<<18)
+/* The next EXEC will fail due to dirty CAS (touched keys). */
+#define REDISMODULE_CTX_FLAGS_MULTI_DIRTY (1<<19)
 
-
+/* Keyspace changes notification classes. Every class is associated with a
+ * character for configuration purposes.
+ * NOTE: These have to be in sync with NOTIFY_* in server.h */
+#define REDISMODULE_NOTIFY_KEYSPACE (1<<0)    /* K */
+#define REDISMODULE_NOTIFY_KEYEVENT (1<<1)    /* E */
 #define REDISMODULE_NOTIFY_GENERIC (1<<2)     /* g */
 #define REDISMODULE_NOTIFY_STRING (1<<3)      /* $ */
 #define REDISMODULE_NOTIFY_LIST (1<<4)        /* l */
@@ -133,6 +160,10 @@ extern "C" {
 
 #define REDISMODULE_NOT_USED(V) ((void) V)
 
+/* Bit flags for aux_save_triggers and the aux_load and aux_save callbacks */
+#define REDISMODULE_AUX_BEFORE_RDB (1<<0)
+#define REDISMODULE_AUX_AFTER_RDB (1<<1)
+
 /* This type represents a timer handle, and is returned when a timer is
  * registered and used in order to invalidate a timer. It's just a 64 bit
  * number, because this is how each timer is represented inside the radix tree
@@ -144,6 +175,209 @@ typedef uint64_t RedisModuleTimerID;
 /* Do filter RedisModule_Call() commands initiated by module itself. */
 #define REDISMODULE_CMDFILTER_NOSELF    (1<<0)
 
+/* Declare that the module can handle errors with RedisModule_SetModuleOptions. */
+#define REDISMODULE_OPTIONS_HANDLE_IO_ERRORS    (1<<0)
+/* When set, Redis will not call RedisModule_SignalModifiedKey(), implicitly in
+ * RedisModule_CloseKey, and the module needs to do that when manually when keys
+ * are modified from the user's sperspective, to invalidate WATCH. */
+#define REDISMODULE_OPTION_NO_IMPLICIT_SIGNAL_MODIFIED (1<<1)
+
+/* Server events definitions. */
+#define REDISMODULE_EVENT_REPLICATION_ROLE_CHANGED 0
+#define REDISMODULE_EVENT_PERSISTENCE 1
+#define REDISMODULE_EVENT_FLUSHDB 2
+#define REDISMODULE_EVENT_LOADING 3
+#define REDISMODULE_EVENT_CLIENT_CHANGE 4
+#define REDISMODULE_EVENT_SHUTDOWN 5
+#define REDISMODULE_EVENT_REPLICA_CHANGE 6
+#define REDISMODULE_EVENT_MASTER_LINK_CHANGE 7
+#define REDISMODULE_EVENT_CRON_LOOP 8
+#define REDISMODULE_EVENT_MODULE_CHANGE 9
+#define REDISMODULE_EVENT_LOADING_PROGRESS 10
+
+typedef struct RedisModuleEvent {
+    uint64_t id;        /* REDISMODULE_EVENT_... defines. */
+    uint64_t dataver;   /* Version of the structure we pass as 'data'. */
+} RedisModuleEvent;
+
+struct RedisModuleCtx;
+typedef void (*RedisModuleEventCallback)(struct RedisModuleCtx *ctx, RedisModuleEvent eid, uint64_t subevent, void *data);
+
+static const RedisModuleEvent
+    RedisModuleEvent_ReplicationRoleChanged = {
+        REDISMODULE_EVENT_REPLICATION_ROLE_CHANGED,
+        1
+    },
+    RedisModuleEvent_Persistence = {
+        REDISMODULE_EVENT_PERSISTENCE,
+        1
+    },
+    RedisModuleEvent_FlushDB = {
+        REDISMODULE_EVENT_FLUSHDB,
+        1
+    },
+    RedisModuleEvent_Loading = {
+        REDISMODULE_EVENT_LOADING,
+        1
+    },
+    RedisModuleEvent_ClientChange = {
+        REDISMODULE_EVENT_CLIENT_CHANGE,
+        1
+    },
+    RedisModuleEvent_Shutdown = {
+        REDISMODULE_EVENT_SHUTDOWN,
+        1
+    },
+    RedisModuleEvent_ReplicaChange = {
+        REDISMODULE_EVENT_REPLICA_CHANGE,
+        1
+    },
+    RedisModuleEvent_CronLoop = {
+        REDISMODULE_EVENT_CRON_LOOP,
+        1
+    },
+    RedisModuleEvent_MasterLinkChange = {
+        REDISMODULE_EVENT_MASTER_LINK_CHANGE,
+        1
+    },
+    RedisModuleEvent_ModuleChange = {
+        REDISMODULE_EVENT_MODULE_CHANGE,
+        1
+    },
+    RedisModuleEvent_LoadingProgress = {
+        REDISMODULE_EVENT_LOADING_PROGRESS,
+        1
+    };
+
+/* Those are values that are used for the 'subevent' callback argument. */
+#define REDISMODULE_SUBEVENT_PERSISTENCE_RDB_START 0
+#define REDISMODULE_SUBEVENT_PERSISTENCE_AOF_START 1
+#define REDISMODULE_SUBEVENT_PERSISTENCE_SYNC_RDB_START 2
+#define REDISMODULE_SUBEVENT_PERSISTENCE_ENDED 3
+#define REDISMODULE_SUBEVENT_PERSISTENCE_FAILED 4
+
+#define REDISMODULE_SUBEVENT_LOADING_RDB_START 0
+#define REDISMODULE_SUBEVENT_LOADING_AOF_START 1
+#define REDISMODULE_SUBEVENT_LOADING_REPL_START 2
+#define REDISMODULE_SUBEVENT_LOADING_ENDED 3
+#define REDISMODULE_SUBEVENT_LOADING_FAILED 4
+
+#define REDISMODULE_SUBEVENT_CLIENT_CHANGE_CONNECTED 0
+#define REDISMODULE_SUBEVENT_CLIENT_CHANGE_DISCONNECTED 1
+
+#define REDISMODULE_SUBEVENT_MASTER_LINK_UP 0
+#define REDISMODULE_SUBEVENT_MASTER_LINK_DOWN 1
+
+#define REDISMODULE_SUBEVENT_REPLICA_CHANGE_ONLINE 0
+#define REDISMODULE_SUBEVENT_REPLICA_CHANGE_OFFLINE 1
+
+#define REDISMODULE_EVENT_REPLROLECHANGED_NOW_MASTER 0
+#define REDISMODULE_EVENT_REPLROLECHANGED_NOW_REPLICA 1
+
+#define REDISMODULE_SUBEVENT_FLUSHDB_START 0
+#define REDISMODULE_SUBEVENT_FLUSHDB_END 1
+
+#define REDISMODULE_SUBEVENT_MODULE_LOADED 0
+#define REDISMODULE_SUBEVENT_MODULE_UNLOADED 1
+
+#define REDISMODULE_SUBEVENT_LOADING_PROGRESS_RDB 0
+#define REDISMODULE_SUBEVENT_LOADING_PROGRESS_AOF 1
+
+/* RedisModuleClientInfo flags. */
+#define REDISMODULE_CLIENTINFO_FLAG_SSL (1<<0)
+#define REDISMODULE_CLIENTINFO_FLAG_PUBSUB (1<<1)
+#define REDISMODULE_CLIENTINFO_FLAG_BLOCKED (1<<2)
+#define REDISMODULE_CLIENTINFO_FLAG_TRACKING (1<<3)
+#define REDISMODULE_CLIENTINFO_FLAG_UNIXSOCKET (1<<4)
+#define REDISMODULE_CLIENTINFO_FLAG_MULTI (1<<5)
+
+/* Here we take all the structures that the module pass to the core
+ * and the other way around. Notably the list here contains the structures
+ * used by the hooks API RedisModule_RegisterToServerEvent().
+ *
+ * The structures always start with a 'version' field. This is useful
+ * when we want to pass a reference to the structure to the core APIs,
+ * for the APIs to fill the structure. In that case, the structure 'version'
+ * field is initialized before passing it to the core, so that the core is
+ * able to cast the pointer to the appropriate structure version. In this
+ * way we obtain ABI compatibility.
+ *
+ * Here we'll list all the structure versions in case they evolve over time,
+ * however using a define, we'll make sure to use the last version as the
+ * public name for the module to use. */
+
+#define REDISMODULE_CLIENTINFO_VERSION 1
+typedef struct RedisModuleClientInfo {
+    uint64_t version;       /* Version of this structure for ABI compat. */
+    uint64_t flags;         /* REDISMODULE_CLIENTINFO_FLAG_* */
+    uint64_t id;            /* Client ID. */
+    char addr[46];          /* IPv4 or IPv6 address. */
+    uint16_t port;          /* TCP port. */
+    uint16_t db;            /* Selected DB. */
+} RedisModuleClientInfoV1;
+
+#define RedisModuleClientInfo RedisModuleClientInfoV1
+
+#define REDISMODULE_REPLICATIONINFO_VERSION 1
+typedef struct RedisModuleReplicationInfo {
+    uint64_t version;       /* Not used since this structure is never passed
+                               from the module to the core right now. Here
+                               for future compatibility. */
+    int master;             /* true if master, false if replica */
+    char *masterhost;       /* master instance hostname for NOW_REPLICA */
+    int masterport;         /* master instance port for NOW_REPLICA */
+    char *replid1;          /* Main replication ID */
+    char *replid2;          /* Secondary replication ID */
+    uint64_t repl1_offset;  /* Main replication offset */
+    uint64_t repl2_offset;  /* Offset of replid2 validity */
+} RedisModuleReplicationInfoV1;
+
+#define RedisModuleReplicationInfo RedisModuleReplicationInfoV1
+
+#define REDISMODULE_FLUSHINFO_VERSION 1
+typedef struct RedisModuleFlushInfo {
+    uint64_t version;       /* Not used since this structure is never passed
+                               from the module to the core right now. Here
+                               for future compatibility. */
+    int32_t sync;           /* Synchronous or threaded flush?. */
+    int32_t dbnum;          /* Flushed database number, -1 for ALL. */
+} RedisModuleFlushInfoV1;
+
+#define RedisModuleFlushInfo RedisModuleFlushInfoV1
+
+#define REDISMODULE_MODULE_CHANGE_VERSION 1
+typedef struct RedisModuleModuleChange {
+    uint64_t version;       /* Not used since this structure is never passed
+                               from the module to the core right now. Here
+                               for future compatibility. */
+    const char* module_name;/* Name of module loaded or unloaded. */
+    int32_t module_version; /* Module version. */
+} RedisModuleModuleChangeV1;
+
+#define RedisModuleModuleChange RedisModuleModuleChangeV1
+
+#define REDISMODULE_CRON_LOOP_VERSION 1
+typedef struct RedisModuleCronLoopInfo {
+    uint64_t version;       /* Not used since this structure is never passed
+                               from the module to the core right now. Here
+                               for future compatibility. */
+    int32_t hz;             /* Approximate number of events per second. */
+} RedisModuleCronLoopV1;
+
+#define RedisModuleCronLoop RedisModuleCronLoopV1
+
+#define REDISMODULE_LOADING_PROGRESS_VERSION 1
+typedef struct RedisModuleLoadingProgressInfo {
+    uint64_t version;       /* Not used since this structure is never passed
+                               from the module to the core right now. Here
+                               for future compatibility. */
+    int32_t hz;             /* Approximate number of events per second. */
+    int32_t progress;       /* Approximate progress between 0 and 1024, or -1
+                             * if unknown. */
+} RedisModuleLoadingProgressV1;
+
+#define RedisModuleLoadingProgress RedisModuleLoadingProgressV1
+
 /* ------------------------- End of common defines ------------------------ */
 
 #ifndef REDISMODULE_CORE
@@ -164,12 +398,18 @@ typedef struct RedisModuleDict RedisModuleDict;
 typedef struct RedisModuleDictIter RedisModuleDictIter;
 typedef struct RedisModuleCommandFilterCtx RedisModuleCommandFilterCtx;
 typedef struct RedisModuleCommandFilter RedisModuleCommandFilter;
+typedef struct RedisModuleInfoCtx RedisModuleInfoCtx;
+typedef struct RedisModuleServerInfoData RedisModuleServerInfoData;
+typedef struct RedisModuleScanCursor RedisModuleScanCursor;
+typedef struct RedisModuleUser RedisModuleUser;
 
 typedef int (*RedisModuleCmdFunc)(RedisModuleCtx *ctx, RedisModuleString **argv, int argc);
 typedef void (*RedisModuleDisconnectFunc)(RedisModuleCtx *ctx, RedisModuleBlockedClient *bc);
 typedef int (*RedisModuleNotificationFunc)(RedisModuleCtx *ctx, int type, const char *event, RedisModuleString *key);
 typedef void *(*RedisModuleTypeLoadFunc)(RedisModuleIO *rdb, int encver);
 typedef void (*RedisModuleTypeSaveFunc)(RedisModuleIO *rdb, void *value);
+typedef int (*RedisModuleTypeAuxLoadFunc)(RedisModuleIO *rdb, int encver, int when);
+typedef void (*RedisModuleTypeAuxSaveFunc)(RedisModuleIO *rdb, int when);
 typedef void (*RedisModuleTypeRewriteFunc)(RedisModuleIO *aof, RedisModuleString *key, void *value);
 typedef size_t (*RedisModuleTypeMemUsageFunc)(const void *value);
 typedef void (*RedisModuleTypeDigestFunc)(RedisModuleDigest *digest, void *value);
@@ -177,8 +417,13 @@ typedef void (*RedisModuleTypeFreeFunc)(void *value);
 typedef void (*RedisModuleClusterMessageReceiver)(RedisModuleCtx *ctx, const char *sender_id, uint8_t type, const unsigned char *payload, uint32_t len);
 typedef void (*RedisModuleTimerProc)(RedisModuleCtx *ctx, void *data);
 typedef void (*RedisModuleCommandFilterFunc) (RedisModuleCommandFilterCtx *filter);
+typedef void (*RedisModuleForkDoneHandler) (int exitcode, int bysignal, void *user_data);
+typedef void (*RedisModuleInfoFunc)(RedisModuleInfoCtx *ctx, int for_crash_report);
+typedef void (*RedisModuleScanCB)(RedisModuleCtx *ctx, RedisModuleString *keyname, RedisModuleKey *key, void *privdata);
+typedef void (*RedisModuleScanKeyCB)(RedisModuleKey *key, RedisModuleString *field, RedisModuleString *value, void *privdata);
+typedef void (*RedisModuleUserChangedFunc) (uint64_t client_id, void *privdata);
 
-#define REDISMODULE_TYPE_METHOD_VERSION 1
+#define REDISMODULE_TYPE_METHOD_VERSION 2
 typedef struct RedisModuleTypeMethods {
     uint64_t version;
     RedisModuleTypeLoadFunc rdb_load;
@@ -187,6 +432,9 @@ typedef struct RedisModuleTypeMethods {
     RedisModuleTypeMemUsageFunc mem_usage;
     RedisModuleTypeDigestFunc digest;
     RedisModuleTypeFreeFunc free;
+    RedisModuleTypeAuxLoadFunc aux_load;
+    RedisModuleTypeAuxSaveFunc aux_save;
+    int aux_save_triggers;
 } RedisModuleTypeMethods;
 
 #define REDISMODULE_GET_API(name) \
@@ -223,6 +471,7 @@ size_t REDISMODULE_API_FUNC(RedisModule_CallReplyLength)(RedisModuleCallReply *r
 RedisModuleCallReply *REDISMODULE_API_FUNC(RedisModule_CallReplyArrayElement)(RedisModuleCallReply *reply, size_t idx);
 RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateString)(RedisModuleCtx *ctx, const char *ptr, size_t len);
 RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringFromLongLong)(RedisModuleCtx *ctx, long long ll);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringFromLongDouble)(RedisModuleCtx *ctx, long double ld, int humanfriendly);
 RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringFromString)(RedisModuleCtx *ctx, const RedisModuleString *str);
 RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringPrintf)(RedisModuleCtx *ctx, const char *fmt, ...);
 void REDISMODULE_API_FUNC(RedisModule_FreeString)(RedisModuleCtx *ctx, RedisModuleString *str);
@@ -230,15 +479,21 @@ const char *REDISMODULE_API_FUNC(RedisModule_StringPtrLen)(const RedisModuleStri
 int REDISMODULE_API_FUNC(RedisModule_ReplyWithError)(RedisModuleCtx *ctx, const char *err);
 int REDISMODULE_API_FUNC(RedisModule_ReplyWithSimpleString)(RedisModuleCtx *ctx, const char *msg);
 int REDISMODULE_API_FUNC(RedisModule_ReplyWithArray)(RedisModuleCtx *ctx, long len);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithNullArray)(RedisModuleCtx *ctx);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithEmptyArray)(RedisModuleCtx *ctx);
 void REDISMODULE_API_FUNC(RedisModule_ReplySetArrayLength)(RedisModuleCtx *ctx, long len);
 int REDISMODULE_API_FUNC(RedisModule_ReplyWithStringBuffer)(RedisModuleCtx *ctx, const char *buf, size_t len);
 int REDISMODULE_API_FUNC(RedisModule_ReplyWithCString)(RedisModuleCtx *ctx, const char *buf);
 int REDISMODULE_API_FUNC(RedisModule_ReplyWithString)(RedisModuleCtx *ctx, RedisModuleString *str);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithEmptyString)(RedisModuleCtx *ctx);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithVerbatimString)(RedisModuleCtx *ctx, const char *buf, size_t len);
 int REDISMODULE_API_FUNC(RedisModule_ReplyWithNull)(RedisModuleCtx *ctx);
 int REDISMODULE_API_FUNC(RedisModule_ReplyWithDouble)(RedisModuleCtx *ctx, double d);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithLongDouble)(RedisModuleCtx *ctx, long double d);
 int REDISMODULE_API_FUNC(RedisModule_ReplyWithCallReply)(RedisModuleCtx *ctx, RedisModuleCallReply *reply);
 int REDISMODULE_API_FUNC(RedisModule_StringToLongLong)(const RedisModuleString *str, long long *ll);
 int REDISMODULE_API_FUNC(RedisModule_StringToDouble)(const RedisModuleString *str, double *d);
+int REDISMODULE_API_FUNC(RedisModule_StringToLongDouble)(const RedisModuleString *str, long double *d);
 void REDISMODULE_API_FUNC(RedisModule_AutoMemory)(RedisModuleCtx *ctx);
 int REDISMODULE_API_FUNC(RedisModule_Replicate)(RedisModuleCtx *ctx, const char *cmdname, const char *fmt, ...);
 int REDISMODULE_API_FUNC(RedisModule_ReplicateVerbatim)(RedisModuleCtx *ctx);
@@ -251,6 +506,9 @@ char *REDISMODULE_API_FUNC(RedisModule_StringDMA)(RedisModuleKey *key, size_t *l
 int REDISMODULE_API_FUNC(RedisModule_StringTruncate)(RedisModuleKey *key, size_t newlen);
 mstime_t REDISMODULE_API_FUNC(RedisModule_GetExpire)(RedisModuleKey *key);
 int REDISMODULE_API_FUNC(RedisModule_SetExpire)(RedisModuleKey *key, mstime_t expire);
+void REDISMODULE_API_FUNC(RedisModule_ResetDataset)(int restart_aof, int async);
+unsigned long long REDISMODULE_API_FUNC(RedisModule_DbSize)(RedisModuleCtx *ctx);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_RandomKey)(RedisModuleCtx *ctx);
 int REDISMODULE_API_FUNC(RedisModule_ZsetAdd)(RedisModuleKey *key, double score, RedisModuleString *ele, int *flagsptr);
 int REDISMODULE_API_FUNC(RedisModule_ZsetIncrby)(RedisModuleKey *key, double score, RedisModuleString *ele, int *flagsptr, double *newscore);
 int REDISMODULE_API_FUNC(RedisModule_ZsetScore)(RedisModuleKey *key, RedisModuleString *ele, double *score);
@@ -269,12 +527,19 @@ int REDISMODULE_API_FUNC(RedisModule_HashGet)(RedisModuleKey *key, int flags, ..
 int REDISMODULE_API_FUNC(RedisModule_IsKeysPositionRequest)(RedisModuleCtx *ctx);
 void REDISMODULE_API_FUNC(RedisModule_KeyAtPos)(RedisModuleCtx *ctx, int pos);
 unsigned long long REDISMODULE_API_FUNC(RedisModule_GetClientId)(RedisModuleCtx *ctx);
+int REDISMODULE_API_FUNC(RedisModule_GetClientInfoById)(void *ci, uint64_t id);
+int REDISMODULE_API_FUNC(RedisModule_PublishMessage)(RedisModuleCtx *ctx, RedisModuleString *channel, RedisModuleString *message);
 int REDISMODULE_API_FUNC(RedisModule_GetContextFlags)(RedisModuleCtx *ctx);
+int REDISMODULE_API_FUNC(RedisModule_AvoidReplicaTraffic)();
 void *REDISMODULE_API_FUNC(RedisModule_PoolAlloc)(RedisModuleCtx *ctx, size_t bytes);
 RedisModuleType *REDISMODULE_API_FUNC(RedisModule_CreateDataType)(RedisModuleCtx *ctx, const char *name, int encver, RedisModuleTypeMethods *typemethods);
 int REDISMODULE_API_FUNC(RedisModule_ModuleTypeSetValue)(RedisModuleKey *key, RedisModuleType *mt, void *value);
+int REDISMODULE_API_FUNC(RedisModule_ModuleTypeReplaceValue)(RedisModuleKey *key, RedisModuleType *mt, void *new_value, void **old_value);
 RedisModuleType *REDISMODULE_API_FUNC(RedisModule_ModuleTypeGetType)(RedisModuleKey *key);
 void *REDISMODULE_API_FUNC(RedisModule_ModuleTypeGetValue)(RedisModuleKey *key);
+int REDISMODULE_API_FUNC(RedisModule_IsIOError)(RedisModuleIO *io);
+void REDISMODULE_API_FUNC(RedisModule_SetModuleOptions)(RedisModuleCtx *ctx, int options);
+int REDISMODULE_API_FUNC(RedisModule_SignalModifiedKey)(RedisModuleCtx *ctx, RedisModuleString *keyname);
 void REDISMODULE_API_FUNC(RedisModule_SaveUnsigned)(RedisModuleIO *io, uint64_t value);
 uint64_t REDISMODULE_API_FUNC(RedisModule_LoadUnsigned)(RedisModuleIO *io);
 void REDISMODULE_API_FUNC(RedisModule_SaveSigned)(RedisModuleIO *io, int64_t value);
@@ -288,13 +553,20 @@ void REDISMODULE_API_FUNC(RedisModule_SaveDouble)(RedisModuleIO *io, double valu
 double REDISMODULE_API_FUNC(RedisModule_LoadDouble)(RedisModuleIO *io);
 void REDISMODULE_API_FUNC(RedisModule_SaveFloat)(RedisModuleIO *io, float value);
 float REDISMODULE_API_FUNC(RedisModule_LoadFloat)(RedisModuleIO *io);
+void REDISMODULE_API_FUNC(RedisModule_SaveLongDouble)(RedisModuleIO *io, long double value);
+long double REDISMODULE_API_FUNC(RedisModule_LoadLongDouble)(RedisModuleIO *io);
+void *REDISMODULE_API_FUNC(RedisModule_LoadDataTypeFromString)(const RedisModuleString *str, const RedisModuleType *mt);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_SaveDataTypeToString)(RedisModuleCtx *ctx, void *data, const RedisModuleType *mt);
 void REDISMODULE_API_FUNC(RedisModule_Log)(RedisModuleCtx *ctx, const char *level, const char *fmt, ...);
 void REDISMODULE_API_FUNC(RedisModule_LogIOError)(RedisModuleIO *io, const char *levelstr, const char *fmt, ...);
+void REDISMODULE_API_FUNC(RedisModule__Assert)(const char *estr, const char *file, int line);
+void REDISMODULE_API_FUNC(RedisModule_LatencyAddSample)(const char *event, mstime_t latency);
 int REDISMODULE_API_FUNC(RedisModule_StringAppendBuffer)(RedisModuleCtx *ctx, RedisModuleString *str, const char *buf, size_t len);
 void REDISMODULE_API_FUNC(RedisModule_RetainString)(RedisModuleCtx *ctx, RedisModuleString *str);
 int REDISMODULE_API_FUNC(RedisModule_StringCompare)(RedisModuleString *a, RedisModuleString *b);
 RedisModuleCtx *REDISMODULE_API_FUNC(RedisModule_GetContextFromIO)(RedisModuleIO *io);
 const RedisModuleString *REDISMODULE_API_FUNC(RedisModule_GetKeyNameFromIO)(RedisModuleIO *io);
+const RedisModuleString *REDISMODULE_API_FUNC(RedisModule_GetKeyNameFromModuleKey)(RedisModuleKey *key);
 long long REDISMODULE_API_FUNC(RedisModule_Milliseconds)(void);
 void REDISMODULE_API_FUNC(RedisModule_DigestAddStringBuffer)(RedisModuleDigest *md, unsigned char *ele, size_t len);
 void REDISMODULE_API_FUNC(RedisModule_DigestAddLongLong)(RedisModuleDigest *md, long long ele);
@@ -321,7 +593,35 @@ RedisModuleString *REDISMODULE_API_FUNC(RedisModule_DictNext)(RedisModuleCtx *ct
 RedisModuleString *REDISMODULE_API_FUNC(RedisModule_DictPrev)(RedisModuleCtx *ctx, RedisModuleDictIter *di, void **dataptr);
 int REDISMODULE_API_FUNC(RedisModule_DictCompareC)(RedisModuleDictIter *di, const char *op, void *key, size_t keylen);
 int REDISMODULE_API_FUNC(RedisModule_DictCompare)(RedisModuleDictIter *di, const char *op, RedisModuleString *key);
-
+int REDISMODULE_API_FUNC(RedisModule_RegisterInfoFunc)(RedisModuleCtx *ctx, RedisModuleInfoFunc cb);
+int REDISMODULE_API_FUNC(RedisModule_InfoAddSection)(RedisModuleInfoCtx *ctx, char *name);
+int REDISMODULE_API_FUNC(RedisModule_InfoBeginDictField)(RedisModuleInfoCtx *ctx, char *name);
+int REDISMODULE_API_FUNC(RedisModule_InfoEndDictField)(RedisModuleInfoCtx *ctx);
+int REDISMODULE_API_FUNC(RedisModule_InfoAddFieldString)(RedisModuleInfoCtx *ctx, char *field, RedisModuleString *value);
+int REDISMODULE_API_FUNC(RedisModule_InfoAddFieldCString)(RedisModuleInfoCtx *ctx, char *field, char *value);
+int REDISMODULE_API_FUNC(RedisModule_InfoAddFieldDouble)(RedisModuleInfoCtx *ctx, char *field, double value);
+int REDISMODULE_API_FUNC(RedisModule_InfoAddFieldLongLong)(RedisModuleInfoCtx *ctx, char *field, long long value);
+int REDISMODULE_API_FUNC(RedisModule_InfoAddFieldULongLong)(RedisModuleInfoCtx *ctx, char *field, unsigned long long value);
+RedisModuleServerInfoData *REDISMODULE_API_FUNC(RedisModule_GetServerInfo)(RedisModuleCtx *ctx, const char *section);
+void REDISMODULE_API_FUNC(RedisModule_FreeServerInfo)(RedisModuleCtx *ctx, RedisModuleServerInfoData *data);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_ServerInfoGetField)(RedisModuleCtx *ctx, RedisModuleServerInfoData *data, const char* field);
+const char *REDISMODULE_API_FUNC(RedisModule_ServerInfoGetFieldC)(RedisModuleServerInfoData *data, const char* field);
+long long REDISMODULE_API_FUNC(RedisModule_ServerInfoGetFieldSigned)(RedisModuleServerInfoData *data, const char* field, int *out_err);
+unsigned long long REDISMODULE_API_FUNC(RedisModule_ServerInfoGetFieldUnsigned)(RedisModuleServerInfoData *data, const char* field, int *out_err);
+double REDISMODULE_API_FUNC(RedisModule_ServerInfoGetFieldDouble)(RedisModuleServerInfoData *data, const char* field, int *out_err);
+int REDISMODULE_API_FUNC(RedisModule_SubscribeToServerEvent)(RedisModuleCtx *ctx, RedisModuleEvent event, RedisModuleEventCallback callback);
+int REDISMODULE_API_FUNC(RedisModule_SetLRU)(RedisModuleKey *key, mstime_t lru_idle);
+int REDISMODULE_API_FUNC(RedisModule_GetLRU)(RedisModuleKey *key, mstime_t *lru_idle);
+int REDISMODULE_API_FUNC(RedisModule_SetLFU)(RedisModuleKey *key, long long lfu_freq);
+int REDISMODULE_API_FUNC(RedisModule_GetLFU)(RedisModuleKey *key, long long *lfu_freq);
+RedisModuleBlockedClient *REDISMODULE_API_FUNC(RedisModule_BlockClientOnKeys)(RedisModuleCtx *ctx, RedisModuleCmdFunc reply_callback, RedisModuleCmdFunc timeout_callback, void (*free_privdata)(RedisModuleCtx*,void*), long long timeout_ms, RedisModuleString **keys, int numkeys, void *privdata);
+void REDISMODULE_API_FUNC(RedisModule_SignalKeyAsReady)(RedisModuleCtx *ctx, RedisModuleString *key);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_GetBlockedClientReadyKey)(RedisModuleCtx *ctx);
+RedisModuleScanCursor *REDISMODULE_API_FUNC(RedisModule_ScanCursorCreate)();
+void REDISMODULE_API_FUNC(RedisModule_ScanCursorRestart)(RedisModuleScanCursor *cursor);
+void REDISMODULE_API_FUNC(RedisModule_ScanCursorDestroy)(RedisModuleScanCursor *cursor);
+int REDISMODULE_API_FUNC(RedisModule_Scan)(RedisModuleCtx *ctx, RedisModuleScanCursor *cursor, RedisModuleScanCB fn, void *privdata);
+int REDISMODULE_API_FUNC(RedisModule_ScanKey)(RedisModuleKey *key, RedisModuleScanCursor *cursor, RedisModuleScanKeyCB fn, void *privdata);
 /* Experimental APIs */
 #ifdef REDISMODULE_EXPERIMENTAL_API
 #define REDISMODULE_EXPERIMENTAL_API_VERSION 3
@@ -337,6 +637,8 @@ void REDISMODULE_API_FUNC(RedisModule_FreeThreadSafeContext)(RedisModuleCtx *ctx
 void REDISMODULE_API_FUNC(RedisModule_ThreadSafeContextLock)(RedisModuleCtx *ctx);
 void REDISMODULE_API_FUNC(RedisModule_ThreadSafeContextUnlock)(RedisModuleCtx *ctx);
 int REDISMODULE_API_FUNC(RedisModule_SubscribeToKeyspaceEvents)(RedisModuleCtx *ctx, int types, RedisModuleNotificationFunc cb);
+int REDISMODULE_API_FUNC(RedisModule_NotifyKeyspaceEvent)(RedisModuleCtx *ctx, int type, const char *event, RedisModuleString *key);
+int REDISMODULE_API_FUNC(RedisModule_GetNotifyKeyspaceEvents)();
 int REDISMODULE_API_FUNC(RedisModule_BlockedClientDisconnected)(RedisModuleCtx *ctx);
 void REDISMODULE_API_FUNC(RedisModule_RegisterClusterMessageReceiver)(RedisModuleCtx *ctx, uint8_t type, RedisModuleClusterMessageReceiver callback);
 int REDISMODULE_API_FUNC(RedisModule_SendClusterMessage)(RedisModuleCtx *ctx, char *target_id, uint8_t type, unsigned char *msg, uint32_t len);
@@ -361,8 +663,21 @@ const RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CommandFilterArgGet)(R
 int REDISMODULE_API_FUNC(RedisModule_CommandFilterArgInsert)(RedisModuleCommandFilterCtx *fctx, int pos, RedisModuleString *arg);
 int REDISMODULE_API_FUNC(RedisModule_CommandFilterArgReplace)(RedisModuleCommandFilterCtx *fctx, int pos, RedisModuleString *arg);
 int REDISMODULE_API_FUNC(RedisModule_CommandFilterArgDelete)(RedisModuleCommandFilterCtx *fctx, int pos);
+int REDISMODULE_API_FUNC(RedisModule_Fork)(RedisModuleForkDoneHandler cb, void *user_data);
+int REDISMODULE_API_FUNC(RedisModule_ExitFromChild)(int retcode);
+int REDISMODULE_API_FUNC(RedisModule_KillForkChild)(int child_pid);
+float REDISMODULE_API_FUNC(RedisModule_GetUsedMemoryRatio)();
+size_t REDISMODULE_API_FUNC(RedisModule_MallocSize)(void* ptr);
+RedisModuleUser *REDISMODULE_API_FUNC(RedisModule_CreateModuleUser)(const char *name);
+void REDISMODULE_API_FUNC(RedisModule_FreeModuleUser)(RedisModuleUser *user);
+int REDISMODULE_API_FUNC(RedisModule_SetModuleUserACL)(RedisModuleUser *user, const char* acl);
+int REDISMODULE_API_FUNC(RedisModule_AuthenticateClientWithACLUser)(RedisModuleCtx *ctx, const char *name, size_t len, RedisModuleUserChangedFunc callback, void *privdata, uint64_t *client_id);
+int REDISMODULE_API_FUNC(RedisModule_AuthenticateClientWithUser)(RedisModuleCtx *ctx, RedisModuleUser *user, RedisModuleUserChangedFunc callback, void *privdata, uint64_t *client_id);
+void REDISMODULE_API_FUNC(RedisModule_DeauthenticateAndCloseClient)(RedisModuleCtx *ctx, uint64_t client_id);
 #endif
 
+#define RedisModule_IsAOFClient(id) ((id) == UINT64_MAX)
+
 /* This is included inline inside each Redis module. */
 static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int apiver) __attribute__((unused));
 static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int apiver) {
@@ -381,14 +696,18 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     REDISMODULE_GET_API(ReplyWithError);
     REDISMODULE_GET_API(ReplyWithSimpleString);
     REDISMODULE_GET_API(ReplyWithArray);
+    REDISMODULE_GET_API(ReplyWithNullArray);
+    REDISMODULE_GET_API(ReplyWithEmptyArray);
     REDISMODULE_GET_API(ReplySetArrayLength);
     REDISMODULE_GET_API(ReplyWithStringBuffer);
     REDISMODULE_GET_API(ReplyWithCString);
     REDISMODULE_GET_API(ReplyWithString);
+    REDISMODULE_GET_API(ReplyWithEmptyString);
+    REDISMODULE_GET_API(ReplyWithVerbatimString);
     REDISMODULE_GET_API(ReplyWithNull);
     REDISMODULE_GET_API(ReplyWithCallReply);
     REDISMODULE_GET_API(ReplyWithDouble);
-    REDISMODULE_GET_API(ReplySetArrayLength);
+    REDISMODULE_GET_API(ReplyWithLongDouble);
     REDISMODULE_GET_API(GetSelectedDb);
     REDISMODULE_GET_API(SelectDb);
     REDISMODULE_GET_API(OpenKey);
@@ -399,6 +718,7 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     REDISMODULE_GET_API(ListPop);
     REDISMODULE_GET_API(StringToLongLong);
     REDISMODULE_GET_API(StringToDouble);
+    REDISMODULE_GET_API(StringToLongDouble);
     REDISMODULE_GET_API(Call);
     REDISMODULE_GET_API(CallReplyProto);
     REDISMODULE_GET_API(FreeCallReply);
@@ -410,6 +730,7 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     REDISMODULE_GET_API(CreateStringFromCallReply);
     REDISMODULE_GET_API(CreateString);
     REDISMODULE_GET_API(CreateStringFromLongLong);
+    REDISMODULE_GET_API(CreateStringFromLongDouble);
     REDISMODULE_GET_API(CreateStringFromString);
     REDISMODULE_GET_API(CreateStringPrintf);
     REDISMODULE_GET_API(FreeString);
@@ -424,6 +745,9 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     REDISMODULE_GET_API(StringTruncate);
     REDISMODULE_GET_API(GetExpire);
     REDISMODULE_GET_API(SetExpire);
+    REDISMODULE_GET_API(ResetDataset);
+    REDISMODULE_GET_API(DbSize);
+    REDISMODULE_GET_API(RandomKey);
     REDISMODULE_GET_API(ZsetAdd);
     REDISMODULE_GET_API(ZsetIncrby);
     REDISMODULE_GET_API(ZsetScore);
@@ -443,11 +767,16 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     REDISMODULE_GET_API(KeyAtPos);
     REDISMODULE_GET_API(GetClientId);
     REDISMODULE_GET_API(GetContextFlags);
+    REDISMODULE_GET_API(AvoidReplicaTraffic);
     REDISMODULE_GET_API(PoolAlloc);
     REDISMODULE_GET_API(CreateDataType);
     REDISMODULE_GET_API(ModuleTypeSetValue);
+    REDISMODULE_GET_API(ModuleTypeReplaceValue);
     REDISMODULE_GET_API(ModuleTypeGetType);
     REDISMODULE_GET_API(ModuleTypeGetValue);
+    REDISMODULE_GET_API(IsIOError);
+    REDISMODULE_GET_API(SetModuleOptions);
+    REDISMODULE_GET_API(SignalModifiedKey);
     REDISMODULE_GET_API(SaveUnsigned);
     REDISMODULE_GET_API(LoadUnsigned);
     REDISMODULE_GET_API(SaveSigned);
@@ -460,14 +789,21 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     REDISMODULE_GET_API(LoadDouble);
     REDISMODULE_GET_API(SaveFloat);
     REDISMODULE_GET_API(LoadFloat);
+    REDISMODULE_GET_API(SaveLongDouble);
+    REDISMODULE_GET_API(LoadLongDouble);
+    REDISMODULE_GET_API(SaveDataTypeToString);
+    REDISMODULE_GET_API(LoadDataTypeFromString);
     REDISMODULE_GET_API(EmitAOF);
     REDISMODULE_GET_API(Log);
     REDISMODULE_GET_API(LogIOError);
+    REDISMODULE_GET_API(_Assert);
+    REDISMODULE_GET_API(LatencyAddSample);
     REDISMODULE_GET_API(StringAppendBuffer);
     REDISMODULE_GET_API(RetainString);
     REDISMODULE_GET_API(StringCompare);
     REDISMODULE_GET_API(GetContextFromIO);
     REDISMODULE_GET_API(GetKeyNameFromIO);
+    REDISMODULE_GET_API(GetKeyNameFromModuleKey);
     REDISMODULE_GET_API(Milliseconds);
     REDISMODULE_GET_API(DigestAddStringBuffer);
     REDISMODULE_GET_API(DigestAddLongLong);
@@ -494,6 +830,37 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     REDISMODULE_GET_API(DictPrev);
     REDISMODULE_GET_API(DictCompare);
     REDISMODULE_GET_API(DictCompareC);
+    REDISMODULE_GET_API(RegisterInfoFunc);
+    REDISMODULE_GET_API(InfoAddSection);
+    REDISMODULE_GET_API(InfoBeginDictField);
+    REDISMODULE_GET_API(InfoEndDictField);
+    REDISMODULE_GET_API(InfoAddFieldString);
+    REDISMODULE_GET_API(InfoAddFieldCString);
+    REDISMODULE_GET_API(InfoAddFieldDouble);
+    REDISMODULE_GET_API(InfoAddFieldLongLong);
+    REDISMODULE_GET_API(InfoAddFieldULongLong);
+    REDISMODULE_GET_API(GetServerInfo);
+    REDISMODULE_GET_API(FreeServerInfo);
+    REDISMODULE_GET_API(ServerInfoGetField);
+    REDISMODULE_GET_API(ServerInfoGetFieldC);
+    REDISMODULE_GET_API(ServerInfoGetFieldSigned);
+    REDISMODULE_GET_API(ServerInfoGetFieldUnsigned);
+    REDISMODULE_GET_API(ServerInfoGetFieldDouble);
+    REDISMODULE_GET_API(GetClientInfoById);
+    REDISMODULE_GET_API(PublishMessage);
+    REDISMODULE_GET_API(SubscribeToServerEvent);
+    REDISMODULE_GET_API(SetLRU);
+    REDISMODULE_GET_API(GetLRU);
+    REDISMODULE_GET_API(SetLFU);
+    REDISMODULE_GET_API(GetLFU);
+    REDISMODULE_GET_API(BlockClientOnKeys);
+    REDISMODULE_GET_API(SignalKeyAsReady);
+    REDISMODULE_GET_API(GetBlockedClientReadyKey);
+    REDISMODULE_GET_API(ScanCursorCreate);
+    REDISMODULE_GET_API(ScanCursorRestart);
+    REDISMODULE_GET_API(ScanCursorDestroy);
+    REDISMODULE_GET_API(Scan);
+    REDISMODULE_GET_API(ScanKey);
 
 #ifdef REDISMODULE_EXPERIMENTAL_API
     REDISMODULE_GET_API(GetThreadSafeContext);
@@ -509,6 +876,8 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     REDISMODULE_GET_API(AbortBlock);
     REDISMODULE_GET_API(SetDisconnectCallback);
     REDISMODULE_GET_API(SubscribeToKeyspaceEvents);
+    REDISMODULE_GET_API(NotifyKeyspaceEvent);
+    REDISMODULE_GET_API(GetNotifyKeyspaceEvents);
     REDISMODULE_GET_API(BlockedClientDisconnected);
     REDISMODULE_GET_API(RegisterClusterMessageReceiver);
     REDISMODULE_GET_API(SendClusterMessage);
@@ -532,6 +901,17 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     REDISMODULE_GET_API(CommandFilterArgInsert);
     REDISMODULE_GET_API(CommandFilterArgReplace);
     REDISMODULE_GET_API(CommandFilterArgDelete);
+    REDISMODULE_GET_API(Fork);
+    REDISMODULE_GET_API(ExitFromChild);
+    REDISMODULE_GET_API(KillForkChild);
+    REDISMODULE_GET_API(GetUsedMemoryRatio);
+    REDISMODULE_GET_API(MallocSize);
+    REDISMODULE_GET_API(CreateModuleUser);
+    REDISMODULE_GET_API(FreeModuleUser);
+    REDISMODULE_GET_API(SetModuleUserACL);
+    REDISMODULE_GET_API(DeauthenticateAndCloseClient);
+    REDISMODULE_GET_API(AuthenticateClientWithACLUser);
+    REDISMODULE_GET_API(AuthenticateClientWithUser);
 #endif
 
     if (RedisModule_IsModuleNameBusy && RedisModule_IsModuleNameBusy(name)) return REDISMODULE_ERR;
@@ -539,6 +919,8 @@ static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int
     return REDISMODULE_OK;
 }
 
+#define RedisModule_Assert(_e) ((_e)?(void)0 : (RedisModule__Assert(#_e,__FILE__,__LINE__),exit(1)))
+
 #else
 
 /* Things only defined for the modules core, not exported to modules

src/release.c

@@ -32,6 +32,7 @@
  * files using this functions. */
 
 #include <string.h>
+#include <stdio.h>
 
 #include "release.h"
 #include "version.h"
@@ -50,3 +51,16 @@ uint64_t redisBuildId(void) {
 
     return crc64(0,(unsigned char*)buildid,strlen(buildid));
 }
+
+/* Return a cached value of the build string in order to avoid recomputing
+ * and converting it in hex every time: this string is shown in the INFO
+ * output that should be fast. */
+char *redisBuildIdString(void) {
+    static char buf[32];
+    static int cached = 0;
+    if (!cached) {
+        snprintf(buf,sizeof(buf),"%llx",(unsigned long long) redisBuildId());
+        cached = 1;
+    }
+    return buf;
+}

src/rio.cpp

@@ -92,6 +92,7 @@ static const rio rioBufferIO = {
     rioBufferFlush,
     NULL,           /* update_checksum */
     0,              /* current checksum */
+    0,              /* flags */
     0,              /* bytes read or written */
     0,              /* read/write chunk size */
     { { NULL, 0 } } /* union for io-specific vars */
@@ -145,6 +146,7 @@ static const rio rioFileIO = {
     rioFileFlush,
     NULL,           /* update_checksum */
     0,              /* current checksum */
+    0,              /* flags */
     0,              /* bytes read or written */
     0,              /* read/write chunk size */
     { { NULL, 0 } } /* union for io-specific vars */
@@ -157,81 +159,180 @@ void rioInitWithFile(rio *r, FILE *fp) {
     r->io.file.autosync = 0;
 }
 
-/* ------------------- File descriptors set implementation ------------------- */
+/* ------------------- Connection implementation -------------------
+ * We use this RIO implemetnation when reading an RDB file directly from
+ * the connection to the memory via rdbLoadRio(), thus this implementation
+ * only implements reading from a connection that is, normally,
+ * just a socket. */
+
+static size_t rioConnWrite(rio *r, const void *buf, size_t len) {
+    UNUSED(r);
+    UNUSED(buf);
+    UNUSED(len);
+    return 0; /* Error, this target does not yet support writing. */
+}
+
+/* Returns 1 or 0 for success/failure. */
+static size_t rioConnRead(rio *r, void *buf, size_t len) {
+    size_t avail = sdslen(r->io.conn.buf)-r->io.conn.pos;
+
+    /* If the buffer is too small for the entire request: realloc. */
+    if (sdslen(r->io.conn.buf) + sdsavail(r->io.conn.buf) < len)
+        r->io.conn.buf = sdsMakeRoomFor(r->io.conn.buf, len - sdslen(r->io.conn.buf));
+
+    /* If the remaining unused buffer is not large enough: memmove so that we
+     * can read the rest. */
+    if (len > avail && sdsavail(r->io.conn.buf) < len - avail) {
+        sdsrange(r->io.conn.buf, r->io.conn.pos, -1);
+        r->io.conn.pos = 0;
+    }
+
+    /* If we don't already have all the data in the sds, read more */
+    while (len > sdslen(r->io.conn.buf) - r->io.conn.pos) {
+        size_t buffered = sdslen(r->io.conn.buf) - r->io.conn.pos;
+        size_t toread = len - buffered;
+        /* Read either what's missing, or PROTO_IOBUF_LEN, the bigger of
+         * the two. */
+        if (toread < PROTO_IOBUF_LEN) toread = PROTO_IOBUF_LEN;
+        if (toread > sdsavail(r->io.conn.buf)) toread = sdsavail(r->io.conn.buf);
+        if (r->io.conn.read_limit != 0 &&
+            r->io.conn.read_so_far + buffered + toread > r->io.conn.read_limit)
+        {
+            if (r->io.conn.read_limit >= r->io.conn.read_so_far - buffered)
+                toread = r->io.conn.read_limit - r->io.conn.read_so_far - buffered;
+            else {
+                errno = EOVERFLOW;
+                return 0;
+            }
+        }
+        int retval = connRead(r->io.conn.conn,
+                          (char*)r->io.conn.buf + sdslen(r->io.conn.buf),
+                          toread);
+        if (retval <= 0) {
+            if (errno == EWOULDBLOCK) errno = ETIMEDOUT;
+            return 0;
+        }
+        sdsIncrLen(r->io.conn.buf, retval);
+    }
+
+    memcpy(buf, (char*)r->io.conn.buf + r->io.conn.pos, len);
+    r->io.conn.read_so_far += len;
+    r->io.conn.pos += len;
+    return len;
+}
+
+/* Returns read/write position in file. */
+static off_t rioConnTell(rio *r) {
+    return r->io.conn.read_so_far;
+}
+
+/* Flushes any buffer to target device if applicable. Returns 1 on success
+ * and 0 on failures. */
+static int rioConnFlush(rio *r) {
+    /* Our flush is implemented by the write method, that recognizes a
+     * buffer set to NULL with a count of zero as a flush request. */
+    return rioConnWrite(r,NULL,0);
+}
+
+static const rio rioConnIO = {
+    rioConnRead,
+    rioConnWrite,
+    rioConnTell,
+    rioConnFlush,
+    NULL,           /* update_checksum */
+    0,              /* current checksum */
+    0,              /* flags */
+    0,              /* bytes read or written */
+    0,              /* read/write chunk size */
+    { { NULL, 0 } } /* union for io-specific vars */
+};
+
+/* Create an RIO that implements a buffered read from an fd
+ * read_limit argument stops buffering when the reaching the limit. */
+void rioInitWithConn(rio *r, connection *conn, size_t read_limit) {
+    *r = rioConnIO;
+    r->io.conn.conn = conn;
+    r->io.conn.pos = 0;
+    r->io.conn.read_limit = read_limit;
+    r->io.conn.read_so_far = 0;
+    r->io.conn.buf = sdsnewlen(NULL, PROTO_IOBUF_LEN);
+    sdsclear(r->io.conn.buf);
+}
+
+/* Release the RIO tream. Optionally returns the unread buffered data
+ * when the SDS pointer 'remaining' is passed. */
+void rioFreeConn(rio *r, sds *remaining) {
+    if (remaining && (size_t)r->io.conn.pos < sdslen(r->io.conn.buf)) {
+        if (r->io.conn.pos > 0) sdsrange(r->io.conn.buf, r->io.conn.pos, -1);
+        *remaining = r->io.conn.buf;
+    } else {
+        sdsfree(r->io.conn.buf);
+        if (remaining) *remaining = NULL;
+    }
+    r->io.conn.buf = NULL;
+}
+
+/* ------------------- File descriptor implementation ------------------
+ * This target is used to write the RDB file to pipe, when the master just
+ * streams the data to the replicas without creating an RDB on-disk image
+ * (diskless replication option).
+ * It only implements writes. */
 
 /* Returns 1 or 0 for success/failure.
- * The function returns success as long as we are able to correctly write
- * to at least one file descriptor.
  *
  * When buf is NULL and len is 0, the function performs a flush operation
  * if there is some pending buffer, so this function is also used in order
- * to implement rioFdsetFlush(). */
+ * to implement rioFdFlush(). */
-static size_t rioFdsetWrite(rio *r, const void *buf, size_t len) {
+static size_t rioFdWrite(rio *r, const void *buf, size_t len) {
     ssize_t retval;
-    int j;
     unsigned char *p = (unsigned char*) buf;
     int doflush = (buf == NULL && len == 0);
 
-    /* To start we always append to our buffer. If it gets larger than
+    /* For small writes, we rather keep the data in user-space buffer, and flush
-     * a given size, we actually write to the sockets. */
+     * it only when it grows. however for larger writes, we prefer to flush
+     * any pre-existing buffer, and write the new one directly without reallocs
+     * and memory copying. */
+    if (len > PROTO_IOBUF_LEN) {
+        /* First, flush any pre-existing buffered data. */
+        if (sdslen(r->io.fd.buf)) {
+            if (rioFdWrite(r, NULL, 0) == 0)
+                return 0;
+        }
+        /* Write the new data, keeping 'p' and 'len' from the input. */
+    } else {
         if (len) {
-        r->io.fdset.buf = sdscatlen(r->io.fdset.buf,buf,len);
+            r->io.fd.buf = sdscatlen(r->io.fd.buf,buf,len);
-        len = 0; /* Prevent entering the while below if we don't flush. */
+            if (sdslen(r->io.fd.buf) > PROTO_IOBUF_LEN)
-        if (sdslen(r->io.fdset.buf) > PROTO_IOBUF_LEN) doflush = 1;
+                doflush = 1;
+            if (!doflush)
+                return 1;
+        }
+        /* Flusing the buffered data. set 'p' and 'len' accordintly. */
+        p = (unsigned char*) r->io.fd.buf;
+        len = sdslen(r->io.fd.buf);
     }
 
-    if (doflush) {
-        p = (unsigned char*) r->io.fdset.buf;
-        len = sdslen(r->io.fdset.buf);
-    }
-
-    /* Write in little chunchs so that when there are big writes we
-     * parallelize while the kernel is sending data in background to
-     * the TCP socket. */
-    while(len) {
-        size_t count = len < 1024 ? len : 1024;
-        int broken = 0;
-        for (j = 0; j < r->io.fdset.numfds; j++) {
-            if (r->io.fdset.state[j] != 0) {
-                /* Skip FDs alraedy in error. */
-                broken++;
-                continue;
-            }
-
-            /* Make sure to write 'count' bytes to the socket regardless
-             * of short writes. */
     size_t nwritten = 0;
-            while(nwritten != count) {
+    while(nwritten != len) {
-                retval = write(r->io.fdset.fds[j],p+nwritten,count-nwritten);
+        retval = write(r->io.fd.fd,p+nwritten,len-nwritten);
         if (retval <= 0) {
-                    /* With blocking sockets, which is the sole user of this
+            /* With blocking io, which is the sole user of this
              * rio target, EWOULDBLOCK is returned only because of
              * the SO_SNDTIMEO socket option, so we translate the error
              * into one more recognizable by the user. */
             if (retval == -1 && errno == EWOULDBLOCK) errno = ETIMEDOUT;
-                    break;
+            return 0; /* error. */
         }
         nwritten += retval;
     }
 
-            if (nwritten != count) {
+    r->io.fd.pos += len;
-                /* Mark this FD as broken. */
+    sdsclear(r->io.fd.buf);
-                r->io.fdset.state[j] = errno;
-                if (r->io.fdset.state[j] == 0) r->io.fdset.state[j] = EIO;
-            }
-        }
-        if (broken == r->io.fdset.numfds) return 0; /* All the FDs in error. */
-        p += count;
-        len -= count;
-        r->io.fdset.pos += count;
-    }
-
-    if (doflush) sdsclear(r->io.fdset.buf);
     return 1;
 }
 
 /* Returns 1 or 0 for success/failure. */
-static size_t rioFdsetRead(rio *r, void *buf, size_t len) {
+static size_t rioFdRead(rio *r, void *buf, size_t len) {
     UNUSED(r);
     UNUSED(buf);
     UNUSED(len);
@@ -239,48 +340,41 @@ static size_t rioFdsetRead(rio *r, void *buf, size_t len) {
 }
 
 /* Returns read/write position in file. */
-static off_t rioFdsetTell(rio *r) {
+static off_t rioFdTell(rio *r) {
-    return r->io.fdset.pos;
+    return r->io.fd.pos;
 }
 
 /* Flushes any buffer to target device if applicable. Returns 1 on success
  * and 0 on failures. */
-static int rioFdsetFlush(rio *r) {
+static int rioFdFlush(rio *r) {
     /* Our flush is implemented by the write method, that recognizes a
      * buffer set to NULL with a count of zero as a flush request. */
-    return rioFdsetWrite(r,NULL,0);
+    return rioFdWrite(r,NULL,0);
 }
 
-static const rio rioFdsetIO = {
+static const rio rioFdIO = {
-    rioFdsetRead,
+    rioFdRead,
-    rioFdsetWrite,
+    rioFdWrite,
-    rioFdsetTell,
+    rioFdTell,
-    rioFdsetFlush,
+    rioFdFlush,
     NULL,           /* update_checksum */
     0,              /* current checksum */
+    0,              /* flags */
     0,              /* bytes read or written */
     0,              /* read/write chunk size */
     { { NULL, 0 } } /* union for io-specific vars */
 };
 
-void rioInitWithFdset(rio *r, int *fds, int numfds) {
+void rioInitWithFd(rio *r, int fd) {
-    int j;
+    *r = rioFdIO;
-
+    r->io.fd.fd = fd;
-    *r = rioFdsetIO;
+    r->io.fd.pos = 0;
-    r->io.fdset.fds = (int*)zmalloc(sizeof(int)*numfds, MALLOC_LOCAL);
+    r->io.fd.buf = sdsempty();
-    r->io.fdset.state = (int*)zmalloc(sizeof(int)*numfds, MALLOC_LOCAL);
-    memcpy(r->io.fdset.fds,fds,sizeof(int)*numfds);
-    for (j = 0; j < numfds; j++) r->io.fdset.state[j] = 0;
-    r->io.fdset.numfds = numfds;
-    r->io.fdset.pos = 0;
-    r->io.fdset.buf = sdsempty();
 }
 
 /* release the rio stream. */
-void rioFreeFdset(rio *r) {
+void rioFreeFd(rio *r) {
-    zfree(r->io.fdset.fds);
+    sdsfree(r->io.fd.buf);
-    zfree(r->io.fdset.state);
-    sdsfree(r->io.fdset.buf);
 }
 
 /* ---------------------------- Generic functions ---------------------------- */
@@ -300,7 +394,7 @@ void rioGenericUpdateChecksum(rio *r, const void *buf, size_t len) {
  * disk I/O concentrated in very little time. When we fsync in an explicit
  * way instead the I/O pressure is more distributed across time. */
 void rioSetAutoSync(rio *r, off_t bytes) {
-    serverAssert(r->read == rioFileIO.read);
+    if(r->write != rioFileIO.write) return;
     r->io.file.autosync = bytes;
 }
 

src/rio.h

@@ -1,6 +1,6 @@
 /*
  * Copyright (c) 2009-2012, Pieter Noordhuis <pcnoordhuis at gmail dot com>
- * Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2009-2019, Salvatore Sanfilippo <antirez at gmail dot com>
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -35,6 +35,10 @@
 #include <stdio.h>
 #include <stdint.h>
 #include "sds.h"
+#include "connection.h"
+
+#define RIO_FLAG_READ_ERROR (1<<0)
+#define RIO_FLAG_WRITE_ERROR (1<<1)
 
 #ifdef __cplusplus
 extern "C" {
@@ -55,8 +59,8 @@ struct _rio {
      * computation. */
     void (*update_cksum)(struct _rio *, const void *buf, size_t len);
 
-    /* The current checksum */
+    /* The current checksum and flags (see RIO_FLAG_*) */
-    uint64_t cksum;
+    uint64_t cksum, flags;
 
     /* number of bytes read or written */
     size_t processed_bytes;
@@ -77,14 +81,20 @@ struct _rio {
             off_t buffered; /* Bytes written since last fsync. */
             off_t autosync; /* fsync after 'autosync' bytes written. */
         } file;
-        /* Multiple FDs target (used to write to N sockets). */
+        /* Connection object (used to read from socket) */
         struct {
-            int *fds;       /* File descriptors. */
+            connection *conn;   /* Connection */
-            int *state;     /* Error state of each fd. 0 (if ok) or errno. */
+            off_t pos;    /* pos in buf that was returned */
-            int numfds;
+            sds buf;      /* buffered data */
+            size_t read_limit;  /* don't allow to buffer/read more than that */
+            size_t read_so_far; /* amount of data read from the rio (not buffered) */
+        } conn;
+        /* FD target (used to write to pipe). */
+        struct {
+            int fd;       /* File descriptor. */
             off_t pos;
             sds buf;
-        } fdset;
+        } fd;
     } io;
 };
 
@@ -95,11 +105,14 @@ typedef struct _rio rio;
  * if needed. */
 
 static inline size_t rioWrite(rio *r, const void *buf, size_t len) {
+    if (r->flags & RIO_FLAG_WRITE_ERROR) return 0;
     while (len) {
         size_t bytes_to_write = (r->max_processing_chunk && r->max_processing_chunk < len) ? r->max_processing_chunk : len;
         if (r->update_cksum) r->update_cksum(r,buf,bytes_to_write);
-        if (r->write(r,buf,bytes_to_write) == 0)
+        if (r->write(r,buf,bytes_to_write) == 0) {
+            r->flags |= RIO_FLAG_WRITE_ERROR;
             return 0;
+        }
         buf = (char*)buf + bytes_to_write;
         len -= bytes_to_write;
         r->processed_bytes += bytes_to_write;
@@ -108,10 +121,13 @@ static inline size_t rioWrite(rio *r, const void *buf, size_t len) {
 }
 
 static inline size_t rioRead(rio *r, void *buf, size_t len) {
+    if (r->flags & RIO_FLAG_READ_ERROR) return 0;
     while (len) {
         size_t bytes_to_read = (r->max_processing_chunk && r->max_processing_chunk < len) ? r->max_processing_chunk : len;
-        if (r->read(r,buf,bytes_to_read) == 0)
+        if (r->read(r,buf,bytes_to_read) == 0) {
+            r->flags |= RIO_FLAG_READ_ERROR;
             return 0;
+        }
         if (r->update_cksum) r->update_cksum(r,buf,bytes_to_read);
         buf = (char*)buf + bytes_to_read;
         len -= bytes_to_read;
@@ -128,11 +144,29 @@ static inline int rioFlush(rio *r) {
     return r->flush(r);
 }
 
+/* This function allows to know if there was a read error in any past
+ * operation, since the rio stream was created or since the last call
+ * to rioClearError(). */
+static inline int rioGetReadError(rio *r) {
+    return (r->flags & RIO_FLAG_READ_ERROR) != 0;
+}
+
+/* Like rioGetReadError() but for write errors. */
+static inline int rioGetWriteError(rio *r) {
+    return (r->flags & RIO_FLAG_WRITE_ERROR) != 0;
+}
+
+static inline void rioClearErrors(rio *r) {
+    r->flags &= ~(RIO_FLAG_READ_ERROR|RIO_FLAG_WRITE_ERROR);
+}
+
 void rioInitWithFile(rio *r, FILE *fp);
 void rioInitWithBuffer(rio *r, sds s);
-void rioInitWithFdset(rio *r, int *fds, int numfds);
+void rioInitWithConn(rio *r, connection *conn, size_t read_limit);
+void rioInitWithFd(rio *r, int fd);
 
-void rioFreeFdset(rio *r);
+void rioFreeFd(rio *r);
+void rioFreeConn(rio *r, sds* out_remainingBufferedData);
 
 size_t rioWriteBulkCount(rio *r, char prefix, long count);
 size_t rioWriteBulkString(rio *r, const char *buf, size_t len);

src/scripting.cpp

@@ -45,7 +45,10 @@ char *redisProtocolToLuaType_Int(lua_State *lua, char *reply);
 char *redisProtocolToLuaType_Bulk(lua_State *lua, char *reply);
 char *redisProtocolToLuaType_Status(lua_State *lua, char *reply);
 char *redisProtocolToLuaType_Error(lua_State *lua, char *reply);
-char *redisProtocolToLuaType_MultiBulk(lua_State *lua, char *reply, int atype);
+char *redisProtocolToLuaType_Aggregate(lua_State *lua, char *reply, int atype);
+char *redisProtocolToLuaType_Null(lua_State *lua, char *reply);
+char *redisProtocolToLuaType_Bool(lua_State *lua, char *reply, int tf);
+char *redisProtocolToLuaType_Double(lua_State *lua, char *reply);
 int redis_math_random (lua_State *L);
 int redis_math_randomseed (lua_State *L);
 void ldbInit(void);
@@ -61,7 +64,7 @@ sds ldbCatStackValue(sds s, lua_State *lua, int idx);
 #define LDB_BREAKPOINTS_MAX 64  /* Max number of breakpoints. */
 #define LDB_MAX_LEN_DEFAULT 256 /* Default len limit for replies / var dumps. */
 struct ldbState {
-    int fd;     /* Socket of the debugging client. */
+    connection *conn; /* Connection of the debugging client. */
     int active; /* Are we debugging EVAL right now? */
     int forked; /* Is this a fork()ed debugging session? */
     list *logs; /* List of messages to send to the client. */
@@ -135,9 +138,12 @@ char *redisProtocolToLuaType(lua_State *lua, char* reply) {
     case '$': p = redisProtocolToLuaType_Bulk(lua,reply); break;
     case '+': p = redisProtocolToLuaType_Status(lua,reply); break;
     case '-': p = redisProtocolToLuaType_Error(lua,reply); break;
-    case '*': p = redisProtocolToLuaType_MultiBulk(lua,reply,*p); break;
+    case '*': p = redisProtocolToLuaType_Aggregate(lua,reply,*p); break;
-    case '%': p = redisProtocolToLuaType_MultiBulk(lua,reply,*p); break;
+    case '%': p = redisProtocolToLuaType_Aggregate(lua,reply,*p); break;
-    case '~': p = redisProtocolToLuaType_MultiBulk(lua,reply,*p); break;
+    case '~': p = redisProtocolToLuaType_Aggregate(lua,reply,*p); break;
+    case '_': p = redisProtocolToLuaType_Null(lua,reply); break;
+    case '#': p = redisProtocolToLuaType_Bool(lua,reply,p[1]); break;
+    case ',': p = redisProtocolToLuaType_Double(lua,reply); break;
     }
     return p;
 }
@@ -185,13 +191,13 @@ char *redisProtocolToLuaType_Error(lua_State *lua, char *reply) {
     return p+2;
 }
 
-char *redisProtocolToLuaType_MultiBulk(lua_State *lua, char *reply, int atype) {
+char *redisProtocolToLuaType_Aggregate(lua_State *lua, char *reply, int atype) {
     char *p = strchr(reply+1,'\r');
     long long mbulklen;
     int j = 0;
 
     string2ll(reply+1,p-reply-1,&mbulklen);
-    if (g_pserver->lua_caller->resp == 2 || atype == '*') {
+    if (serverTL->lua_client->resp == 2 || atype == '*') {
         p += 2;
         if (mbulklen == -1) {
             lua_pushboolean(lua,0);
@@ -203,11 +209,15 @@ char *redisProtocolToLuaType_MultiBulk(lua_State *lua, char *reply, int atype) {
             p = redisProtocolToLuaType(lua,p);
             lua_settable(lua,-3);
         }
-    } else if (g_pserver->lua_caller->resp == 3) {
+    } else if (serverTL->lua_client->resp == 3) {
         /* Here we handle only Set and Map replies in RESP3 mode, since arrays
-         * follow the above RESP2 code path. */
+         * follow the above RESP2 code path. Note that those are represented
+         * as a table with the "map" or "set" field populated with the actual
+         * table representing the set or the map type. */
         p += 2;
         lua_newtable(lua);
+        lua_pushstring(lua,atype == '%' ? "map" : "set");
+        lua_newtable(lua);
         for (j = 0; j < mbulklen; j++) {
             p = redisProtocolToLuaType(lua,p);
             if (atype == '%') {
@@ -217,10 +227,44 @@ char *redisProtocolToLuaType_MultiBulk(lua_State *lua, char *reply, int atype) {
             }
             lua_settable(lua,-3);
         }
+        lua_settable(lua,-3);
     }
     return p;
 }
 
+char *redisProtocolToLuaType_Null(lua_State *lua, char *reply) {
+    char *p = strchr(reply+1,'\r');
+    lua_pushnil(lua);
+    return p+2;
+}
+
+char *redisProtocolToLuaType_Bool(lua_State *lua, char *reply, int tf) {
+    char *p = strchr(reply+1,'\r');
+    lua_pushboolean(lua,tf == 't');
+    return p+2;
+}
+
+char *redisProtocolToLuaType_Double(lua_State *lua, char *reply) {
+    char *p = strchr(reply+1,'\r');
+    char buf[MAX_LONG_DOUBLE_CHARS+1];
+    size_t len = p-reply-1;
+    double d;
+
+    if (len <= MAX_LONG_DOUBLE_CHARS) {
+        memcpy(buf,reply+1,len);
+        buf[len] = '\0';
+        d = strtod(buf,NULL); /* We expect a valid representation. */
+    } else {
+        d = 0;
+    }
+
+    lua_newtable(lua);
+    lua_pushstring(lua,"double");
+    lua_pushnumber(lua,d);
+    lua_settable(lua,-3);
+    return p+2;
+}
+
 /* This function is used in order to push an error on the Lua stack in the
  * format used by redis.pcall to return errors, which is a lua table
  * with a single "err" field set to the error string. Note that this
@@ -295,6 +339,8 @@ void luaSortArray(lua_State *lua) {
  * Lua reply to Redis reply conversion functions.
  * ------------------------------------------------------------------------- */
 
+/* Reply to client 'c' converting the top element in the Lua stack to a
+ * Redis reply. As a side effect the element is consumed from the stack.  */
 void luaReplyToRedisReply(client *c, lua_State *lua) {
     int t = lua_type(lua,-1);
 
@@ -303,7 +349,11 @@ void luaReplyToRedisReply(client *c, lua_State *lua) {
         addReplyBulkCBuffer(c,(char*)lua_tostring(lua,-1),lua_strlen(lua,-1));
         break;
     case LUA_TBOOLEAN:
-        addReply(c,lua_toboolean(lua,-1) ? shared.cone : shared.null[c->resp]);
+        if (serverTL->lua_client->resp == 2)
+            addReply(c,lua_toboolean(lua,-1) ? shared.cone :
+                                               shared.null[c->resp]);
+        else
+            addReplyBool(c,lua_toboolean(lua,-1));
         break;
     case LUA_TNUMBER:
         addReplyLongLong(c,(long long)lua_tonumber(lua,-1));
@@ -313,6 +363,8 @@ void luaReplyToRedisReply(client *c, lua_State *lua) {
          * Error are returned as a single element table with 'err' field.
          * Status replies are returned as single element table with 'ok'
          * field. */
+
+        /* Handle error reply. */
         lua_pushstring(lua,"err");
         lua_gettable(lua,-2);
         t = lua_type(lua,-1);
@@ -324,8 +376,9 @@ void luaReplyToRedisReply(client *c, lua_State *lua) {
             lua_pop(lua,2);
             return;
         }
+        lua_pop(lua,1); /* Discard field name pushed before. */
 
-        lua_pop(lua,1);
+        /* Handle status reply. */
         lua_pushstring(lua,"ok");
         lua_gettable(lua,-2);
         t = lua_type(lua,-1);
@@ -334,12 +387,70 @@ void luaReplyToRedisReply(client *c, lua_State *lua) {
             sdsmapchars(ok,"\r\n","  ",2);
             addReplySds(c,sdscatprintf(sdsempty(),"+%s\r\n",ok));
             sdsfree(ok);
-            lua_pop(lua,1);
+            lua_pop(lua,2);
-        } else {
+            return;
+        }
+        lua_pop(lua,1); /* Discard field name pushed before. */
+
+        /* Handle double reply. */
+        lua_pushstring(lua,"double");
+        lua_gettable(lua,-2);
+        t = lua_type(lua,-1);
+        if (t == LUA_TNUMBER) {
+            addReplyDouble(c,lua_tonumber(lua,-1));
+            lua_pop(lua,2);
+            return;
+        }
+        lua_pop(lua,1); /* Discard field name pushed before. */
+
+        /* Handle map reply. */
+        lua_pushstring(lua,"map");
+        lua_gettable(lua,-2);
+        t = lua_type(lua,-1);
+        if (t == LUA_TTABLE) {
+            int maplen = 0;
+            void *replylen = addReplyDeferredLen(c);
+            lua_pushnil(lua); /* Use nil to start iteration. */
+            while (lua_next(lua,-2)) {
+                /* Stack now: table, key, value */
+                luaReplyToRedisReply(c, lua); /* Return value. */
+                lua_pushvalue(lua,-1);        /* Dup key before consuming. */
+                luaReplyToRedisReply(c, lua); /* Return key. */
+                /* Stack now: table, key. */
+                maplen++;
+            }
+            setDeferredMapLen(c,replylen,maplen);
+            lua_pop(lua,2);
+            return;
+        }
+        lua_pop(lua,1); /* Discard field name pushed before. */
+
+        /* Handle set reply. */
+        lua_pushstring(lua,"set");
+        lua_gettable(lua,-2);
+        t = lua_type(lua,-1);
+        if (t == LUA_TTABLE) {
+            int setlen = 0;
+            void *replylen = addReplyDeferredLen(c);
+            lua_pushnil(lua); /* Use nil to start iteration. */
+            while (lua_next(lua,-2)) {
+                /* Stack now: table, key, true */
+                lua_pop(lua,1);               /* Discard the boolean value. */
+                lua_pushvalue(lua,-1);        /* Dup key before consuming. */
+                luaReplyToRedisReply(c, lua); /* Return key. */
+                /* Stack now: table, key. */
+                setlen++;
+            }
+            setDeferredSetLen(c,replylen,setlen);
+            lua_pop(lua,2);
+            return;
+        }
+        lua_pop(lua,1); /* Discard field name pushed before. */
+
+        /* Handle the array reply. */
+        {
         void *replylen = addReplyDeferredLen(c);
         int j = 1, mbulklen = 0;
-
-            lua_pop(lua,1); /* Discard the 'ok' field value we popped */
         while(1) {
             lua_pushnumber(lua,j++);
             lua_gettable(lua,-2);
@@ -387,13 +498,6 @@ int luaRedisGenericCommand(lua_State *lua, int raise_error) {
     static size_t cached_objects_len[LUA_CMD_OBJCACHE_SIZE];
     static int inuse = 0;   /* Recursive calls detection. */
 
-    /* Reflect MULTI state */
-    if (g_pserver->lua_multi_emitted || (g_pserver->lua_caller->flags & CLIENT_MULTI)) {
-        c->flags |= CLIENT_MULTI;
-    } else {
-        c->flags &= ~CLIENT_MULTI;
-    }
-
     /* By using Lua debug hooks it is possible to trigger a recursive call
      * to luaRedisGenericCommand(), which normally should never happen.
      * To make this function reentrant is futile and makes it slower, but
@@ -582,15 +686,27 @@ int luaRedisGenericCommand(lua_State *lua, int raise_error) {
     if (g_pserver->cluster_enabled && !g_pserver->loading &&
         !(g_pserver->lua_caller->flags & CLIENT_MASTER))
     {
+        int error_code;
         /* Duplicate relevant flags in the lua client. */
         c->flags &= ~(CLIENT_READONLY|CLIENT_ASKING);
         c->flags |= g_pserver->lua_caller->flags & (CLIENT_READONLY|CLIENT_ASKING);
-        if (getNodeByQuery(c,c->cmd,c->argv,c->argc,NULL,NULL) !=
+        if (getNodeByQuery(c,c->cmd,c->argv,c->argc,NULL,&error_code) !=
                            g_pserver->cluster->myself)
         {
+            if (error_code == CLUSTER_REDIR_DOWN_RO_STATE) { 
+                luaPushError(lua,
+                    "Lua script attempted to execute a write command while the "
+                    "cluster is down and readonly");
+            } else if (error_code == CLUSTER_REDIR_DOWN_STATE) { 
+                luaPushError(lua,
+                    "Lua script attempted to execute a command while the "
+                    "cluster is down");
+            } else {
                 luaPushError(lua,
                     "Lua script attempted to access a non local key in a "
                     "cluster node");
+            }
+
             goto cleanup;
         }
     }
@@ -606,6 +722,9 @@ int luaRedisGenericCommand(lua_State *lua, int raise_error) {
     {
         execCommandPropagateMulti(g_pserver->lua_caller);
         g_pserver->lua_multi_emitted = 1;
+        /* Now we are in the MULTI context, the lua_client should be
+         * flag as CLIENT_MULTI. */
+        c->flags |= CLIENT_MULTI;
     }
 
     /* Run the command */
@@ -870,6 +989,25 @@ int luaLogCommand(lua_State *lua) {
     return 0;
 }
 
+/* redis.setresp() */
+int luaSetResp(lua_State *lua) {
+    int argc = lua_gettop(lua);
+
+    if (argc != 1) {
+        lua_pushstring(lua, "redis.setresp() requires one argument.");
+        return lua_error(lua);
+    }
+
+    int resp = lua_tonumber(lua,-argc);
+    if (resp != 2 && resp != 3) {
+        lua_pushstring(lua, "RESP version must be 2 or 3.");
+        return lua_error(lua);
+    }
+
+    serverTL->lua_client->resp = resp;
+    return 0;
+}
+
 /* ---------------------------------------------------------------------------
  * Lua engine initialization and reset.
  * ------------------------------------------------------------------------- */
@@ -968,10 +1106,12 @@ void scriptingInit(int setup) {
     if (setup) {
         for (int iel = 0; iel < cserver.cthreads; ++iel)
         {
-            g_pserver->rgthreadvar[iel].lua_client = createClient(-1, iel);
+            g_pserver->rgthreadvar[iel].lua_client = createClient(nullptr, iel);
             g_pserver->rgthreadvar[iel].lua_client->flags |= CLIENT_LUA;
         }
         g_pserver->lua_timedout = 0;
+        g_pserver->lua_caller = NULL;
+        g_pserver->lua_cur_script = NULL;
         ldbInit();
     }
 
@@ -1002,6 +1142,11 @@ void scriptingInit(int setup) {
     lua_pushcfunction(lua,luaLogCommand);
     lua_settable(lua,-3);
 
+    /* redis.setresp */
+    lua_pushstring(lua,"setresp");
+    lua_pushcfunction(lua,luaSetResp);
+    lua_settable(lua,-3);
+
     lua_pushstring(lua,"LOG_DEBUG");
     lua_pushnumber(lua,LL_DEBUG);
     lua_settable(lua,-3);
@@ -1281,7 +1426,11 @@ void luaMaskCountHook(lua_State *lua, lua_Debug *ar) {
     /* Set the timeout condition if not already set and the maximum
      * execution time was reached. */
     if (elapsed >= g_pserver->lua_time_limit && g_pserver->lua_timedout == 0) {
-        serverLog(LL_WARNING,"Lua slow script detected: still in execution after %lld milliseconds. You can try killing the script using the SCRIPT KILL command.",elapsed);
+        serverLog(LL_WARNING,
+            "Lua slow script detected: still in execution after %lld milliseconds. "
+            "You can try killing the script using the SCRIPT KILL command. "
+            "Script SHA1 is: %s",
+            elapsed, g_pserver->lua_cur_script);
         g_pserver->lua_timedout = 1;
         /* Once the script timeouts we reenter the event loop to permit others
          * to call SCRIPT KILL or SHUTDOWN NOSAVE if needed. For this reason
@@ -1298,6 +1447,22 @@ void luaMaskCountHook(lua_State *lua, lua_Debug *ar) {
     }
 }
 
+void prepareLuaClient(void) {
+    /* Select the right DB in the context of the Lua client */
+    selectDb(serverTL->lua_client,g_pserver->lua_caller->db->id);
+    serverTL->lua_client->resp = 2; /* Default is RESP2, scripts can change it. */
+
+    /* If we are in MULTI context, flag Lua client as CLIENT_MULTI. */
+    if (g_pserver->lua_caller->flags & CLIENT_MULTI) {
+        serverTL->lua_client->flags |= CLIENT_MULTI;
+    }
+}
+
+void resetLuaClient(void) {
+    /* After the script done, remove the MULTI state. */
+    serverTL->lua_client->flags &= ~CLIENT_MULTI;
+}
+
 void evalGenericCommand(client *c, int evalsha) {
     lua_State *lua = g_pserver->lua;
     char funcname[43];
@@ -1386,9 +1551,6 @@ void evalGenericCommand(client *c, int evalsha) {
     luaSetGlobalArray(lua,"KEYS",c->argv+3,numkeys);
     luaSetGlobalArray(lua,"ARGV",c->argv+3+numkeys,c->argc-3-numkeys);
 
-    /* Select the right DB in the context of the Lua client */
-    selectDb(serverTL->lua_client,c->db->id);
-
     /* Set a hook in order to be able to stop the script execution if it
      * is running for too much time.
      * We set the hook only if the time limit is enabled as the hook will
@@ -1397,6 +1559,7 @@ void evalGenericCommand(client *c, int evalsha) {
      * If we are debugging, we set instead a "line" hook so that the
      * debugger is call-back at every line executed by the script. */
     g_pserver->lua_caller = c;
+    g_pserver->lua_cur_script = funcname + 2;
     g_pserver->lua_time_start = mstime();
     g_pserver->lua_kill = 0;
     if (g_pserver->lua_time_limit > 0 && ldb.active == 0) {
@@ -1407,11 +1570,15 @@ void evalGenericCommand(client *c, int evalsha) {
         delhook = 1;
     }
 
+    prepareLuaClient();
+
     /* At this point whether this script was never seen before or if it was
      * already defined, we can call it. We have zero arguments and expect
      * a single return value. */
     err = lua_pcall(lua,0,1,-2);
 
+    resetLuaClient();
+
     /* Perform some cleanup that we need to do both on error and success. */
     if (delhook) lua_sethook(lua,NULL,0,0); /* Disable hook */
     if (g_pserver->lua_timedout) {
@@ -1430,6 +1597,7 @@ void evalGenericCommand(client *c, int evalsha) {
         }
     }
     g_pserver->lua_caller = NULL;
+    g_pserver->lua_cur_script = NULL;
 
     /* Call the Lua garbage collector from time to time to avoid a
      * full cycle performed by Lua, which adds too latency.
@@ -1464,11 +1632,7 @@ void evalGenericCommand(client *c, int evalsha) {
     if (g_pserver->lua_replicate_commands) {
         preventCommandPropagation(c);
         if (g_pserver->lua_multi_emitted) {
-            robj *propargv[1];
+            execCommandPropagateExec(c);
-            propargv[0] = createStringObject("EXEC",4);
-            alsoPropagate(cserver.execCommand,c->db->id,propargv,1,
-                PROPAGATE_AOF|PROPAGATE_REPL);
-            decrRefCount(propargv[0]);
         }
     }
 
@@ -1607,7 +1771,7 @@ NULL
 
 /* Initialize Lua debugger data structures. */
 void ldbInit(void) {
-    ldb.fd = -1;
+    ldb.conn = NULL;
     ldb.active = 0;
     ldb.logs = listCreate();
     listSetFreeMethod(ldb.logs,(void (*)(const void*))sdsfree);
@@ -1629,7 +1793,7 @@ void ldbFlushLog(list *log) {
 void ldbEnable(client *c) {
     c->flags |= CLIENT_LUA_DEBUG;
     ldbFlushLog(ldb.logs);
-    ldb.fd = c->fd;
+    ldb.conn = c->conn;
     ldb.step = 1;
     ldb.bpcount = 0;
     ldb.luabp = 0;
@@ -1684,7 +1848,7 @@ void ldbSendLogs(void) {
         proto = sdscatlen(proto,"\r\n",2);
         listDelNode(ldb.logs,ln);
     }
-    if (write(ldb.fd,proto,sdslen(proto)) == -1) {
+    if (connWrite(ldb.conn,proto,sdslen(proto)) == -1) {
         /* Avoid warning. We don't check the return value of write()
          * since the next read() will catch the I/O error and will
          * close the debugging session. */
@@ -1707,7 +1871,7 @@ void ldbSendLogs(void) {
 int ldbStartSession(client *c) {
     ldb.forked = (c->flags & CLIENT_LUA_DEBUG_SYNC) == 0;
     if (ldb.forked) {
-        pid_t cp = fork();
+        pid_t cp = redisFork();
         if (cp == -1) {
             addReplyError(c,"Fork() failed: can't run EVAL in debugging mode.");
             return 0;
@@ -1724,7 +1888,6 @@ int ldbStartSession(client *c) {
              * socket to make sure if the parent crashes a reset is sent
              * to the clients. */
             serverLog(LL_WARNING,"Redis forked for debugging eval");
-            closeListeningSockets(0);
         } else {
             /* Parent */
             listAddNodeTail(ldb.children,(void*)(unsigned long)cp);
@@ -1737,8 +1900,8 @@ int ldbStartSession(client *c) {
     }
 
     /* Setup our debugging session. */
-    anetBlock(NULL,ldb.fd);
+    connBlock(ldb.conn);
-    anetSendTimeout(NULL,ldb.fd,5000);
+    connSendTimeout(ldb.conn,5000);
     ldb.active = 1;
 
     /* First argument of EVAL is the script itself. We split it into different
@@ -1765,7 +1928,7 @@ void ldbEndSession(client *c) {
 
     /* If it's a fork()ed session, we just exit. */
     if (ldb.forked) {
-        writeToClient(c->fd, c, 0);
+        writeToClient(c,0);
         serverLog(LL_WARNING,"Lua debugging session child exiting");
         exitFromChild(0);
     } else {
@@ -1774,8 +1937,8 @@ void ldbEndSession(client *c) {
     }
 
     /* Otherwise let's restore client's state. */
-    anetNonBlock(NULL,ldb.fd);
+    connNonBlock(ldb.conn);
-    anetSendTimeout(NULL,ldb.fd,0);
+    connSendTimeout(ldb.conn,0);
 
     /* Close the client connectin after sending the final EVAL reply
      * in order to signal the end of the debugging session. */
@@ -2068,6 +2231,11 @@ char *ldbRedisProtocolToHuman_Int(sds *o, char *reply);
 char *ldbRedisProtocolToHuman_Bulk(sds *o, char *reply);
 char *ldbRedisProtocolToHuman_Status(sds *o, char *reply);
 char *ldbRedisProtocolToHuman_MultiBulk(sds *o, char *reply);
+char *ldbRedisProtocolToHuman_Set(sds *o, char *reply);
+char *ldbRedisProtocolToHuman_Map(sds *o, char *reply);
+char *ldbRedisProtocolToHuman_Null(sds *o, char *reply);
+char *ldbRedisProtocolToHuman_Bool(sds *o, char *reply);
+char *ldbRedisProtocolToHuman_Double(sds *o, char *reply);
 
 /* Get Redis protocol from 'reply' and appends it in human readable form to
  * the passed SDS string 'o'.
@@ -2082,6 +2250,11 @@ char *ldbRedisProtocolToHuman(sds *o, char *reply) {
     case '+': p = ldbRedisProtocolToHuman_Status(o,reply); break;
     case '-': p = ldbRedisProtocolToHuman_Status(o,reply); break;
     case '*': p = ldbRedisProtocolToHuman_MultiBulk(o,reply); break;
+    case '~': p = ldbRedisProtocolToHuman_Set(o,reply); break;
+    case '%': p = ldbRedisProtocolToHuman_Map(o,reply); break;
+    case '_': p = ldbRedisProtocolToHuman_Null(o,reply); break;
+    case '#': p = ldbRedisProtocolToHuman_Bool(o,reply); break;
+    case ',': p = ldbRedisProtocolToHuman_Double(o,reply); break;
     }
     return p;
 }
@@ -2136,6 +2309,62 @@ char *ldbRedisProtocolToHuman_MultiBulk(sds *o, char *reply) {
     return p;
 }
 
+char *ldbRedisProtocolToHuman_Set(sds *o, char *reply) {
+    char *p = strchr(reply+1,'\r');
+    long long mbulklen;
+    int j = 0;
+
+    string2ll(reply+1,p-reply-1,&mbulklen);
+    p += 2;
+    *o = sdscatlen(*o,"~(",2);
+    for (j = 0; j < mbulklen; j++) {
+        p = ldbRedisProtocolToHuman(o,p);
+        if (j != mbulklen-1) *o = sdscatlen(*o,",",1);
+    }
+    *o = sdscatlen(*o,")",1);
+    return p;
+}
+
+char *ldbRedisProtocolToHuman_Map(sds *o, char *reply) {
+    char *p = strchr(reply+1,'\r');
+    long long mbulklen;
+    int j = 0;
+
+    string2ll(reply+1,p-reply-1,&mbulklen);
+    p += 2;
+    *o = sdscatlen(*o,"{",1);
+    for (j = 0; j < mbulklen; j++) {
+        p = ldbRedisProtocolToHuman(o,p);
+        *o = sdscatlen(*o," => ",4);
+        p = ldbRedisProtocolToHuman(o,p);
+        if (j != mbulklen-1) *o = sdscatlen(*o,",",1);
+    }
+    *o = sdscatlen(*o,"}",1);
+    return p;
+}
+
+char *ldbRedisProtocolToHuman_Null(sds *o, char *reply) {
+    char *p = strchr(reply+1,'\r');
+    *o = sdscatlen(*o,"(null)",6);
+    return p+2;
+}
+
+char *ldbRedisProtocolToHuman_Bool(sds *o, char *reply) {
+    char *p = strchr(reply+1,'\r');
+    if (reply[1] == 't')
+        *o = sdscatlen(*o,"#true",5);
+    else
+        *o = sdscatlen(*o,"#false",6);
+    return p+2;
+}
+
+char *ldbRedisProtocolToHuman_Double(sds *o, char *reply) {
+    char *p = strchr(reply+1,'\r');
+    *o = sdscatlen(*o,"(double) ",9);
+    *o = sdscatlen(*o,reply+1,p-reply-1);
+    return p+2;
+}
+
 /* Log a Redis reply as debugger output, in an human readable format.
  * If the resulting string is longer than 'len' plus a few more chars
  * used as prefix, it gets truncated. */
@@ -2347,7 +2576,7 @@ int ldbRepl(lua_State *lua) {
     while(1) {
         while((argv = ldbReplParseCommand(&argc)) == NULL) {
             char buf[1024];
-            int nread = read(ldb.fd,buf,sizeof(buf));
+            int nread = connRead(ldb.conn,buf,sizeof(buf));
             if (nread <= 0) {
                 /* Make sure the script runs without user input since the
                  * client is no longer connected. */

src/sds.c

@@ -642,6 +642,10 @@ sds sdscatfmt(sds s, char const *fmt, ...) {
     long i;
     va_list ap;
 
+    /* To avoid continuous reallocations, let's start with a buffer that
+     * can hold at least two times the format string itself. It's not the
+     * best heuristic but seems to work in practice. */
+    s = sdsMakeRoomFor(s, initlen + strlen(fmt)*2);
     va_start(ap,fmt);
     f = fmt;    /* Next format specifier byte to process. */
     i = initlen; /* Position of the next byte to write to dest str. */

src/semiorderedset.h

@@ -198,6 +198,7 @@ public:
         m_data = decltype(m_data)();
         bits = bits_min;
         m_data.resize(1ULL << bits);
+        celem = 0;
         idxRehash = m_data.size();
     }
     

src/sentinel.cpp

@@ -30,6 +30,10 @@
 
 #include "server.h"
 #include "hiredis.h"
+#ifdef USE_OPENSSL
+#include "openssl/ssl.h"
+#include "hiredis_ssl.h"
+#endif
 #include "async.h"
 
 #include <ctype.h>
@@ -40,6 +44,10 @@
 
 extern char **environ;
 
+#ifdef USE_OPENSSL
+extern SSL_CTX *redis_tls_ctx;
+#endif
+
 #define REDIS_SENTINEL_PORT 26379
 
 /* ======================== Sentinel global state =========================== */
@@ -453,8 +461,8 @@ struct redisCommand sentinelcmds[] = {
     {"role",sentinelRoleCommand,1,"ok-loading",0,NULL,0,0,0,0,0},
     {"client",clientCommand,-2,"read-only no-script",0,NULL,0,0,0,0,0},
     {"shutdown",shutdownCommand,-1,"",0,NULL,0,0,0,0,0},
-    {"auth",authCommand,2,"no-script ok-loading ok-stale fast",0,NULL,0,0,0,0,0},
+    {"auth",authCommand,2,"no-auth no-script ok-loading ok-stale fast",0,NULL,0,0,0,0,0},
-    {"hello",helloCommand,-2,"no-script fast",0,NULL,0,0,0,0,0}
+    {"hello",helloCommand,-2,"no-auth no-script fast",0,NULL,0,0,0,0,0}
 };
 
 /* This function overwrites a few normal Redis config default with Sentinel
@@ -1995,6 +2003,19 @@ void sentinelSetClientName(sentinelRedisInstance *ri, redisAsyncContext *c, cons
     }
 }
 
+static int instanceLinkNegotiateTLS(redisAsyncContext *context) {
+#ifndef USE_OPENSSL
+    (void) context;
+#else
+    if (!redis_tls_ctx) return C_ERR;
+    SSL *ssl = SSL_new(redis_tls_ctx);
+    if (!ssl) return C_ERR;
+
+    if (redisInitiateSSL(&context->c, ssl) == REDIS_ERR) return C_ERR;
+#endif
+    return C_OK;
+}
+
 /* Create the async connections for the instance link if the link
  * is disconnected. Note that link->disconnected is true even if just
  * one of the two links (commands and pub/sub) is missing. */
@@ -2010,7 +2031,11 @@ void sentinelReconnectInstance(sentinelRedisInstance *ri) {
     /* Commands connection. */
     if (link->cc == NULL) {
         link->cc = redisAsyncConnectBind(ri->addr->ip,ri->addr->port,NET_FIRST_BIND_ADDR);
-        if (link->cc->err) {
+        if (!link->cc->err && g_pserver->tls_replication &&
+                (instanceLinkNegotiateTLS(link->cc) == C_ERR)) {
+            sentinelEvent(LL_DEBUG,"-cmd-link-reconnection",ri,"%@ #Failed to initialize TLS");
+            instanceLinkCloseConnection(link,link->cc);
+        } else if (link->cc->err) {
             sentinelEvent(LL_DEBUG,"-cmd-link-reconnection",ri,"%@ #%s",
                 link->cc->errstr);
             instanceLinkCloseConnection(link,link->cc);
@@ -2033,7 +2058,10 @@ void sentinelReconnectInstance(sentinelRedisInstance *ri) {
     /* Pub / Sub */
     if ((ri->flags & (SRI_MASTER|SRI_SLAVE)) && link->pc == NULL) {
         link->pc = redisAsyncConnectBind(ri->addr->ip,ri->addr->port,NET_FIRST_BIND_ADDR);
-        if (link->pc->err) {
+        if (!link->pc->err && g_pserver->tls_replication &&
+                (instanceLinkNegotiateTLS(link->pc) == C_ERR)) {
+            sentinelEvent(LL_DEBUG,"-pubsub-link-reconnection",ri,"%@ #Failed to initialize TLS");
+        } else if (link->pc->err) {
             sentinelEvent(LL_DEBUG,"-pubsub-link-reconnection",ri,"%@ #%s",
                 link->pc->errstr);
             instanceLinkCloseConnection(link,link->pc);
@@ -2584,8 +2612,9 @@ int sentinelSendHello(sentinelRedisInstance *ri) {
             return C_ERR;
         announce_ip = ip;
     }
-    announce_port = sentinel.announce_port ?
+    if (sentinel.announce_port) announce_port = sentinel.announce_port;
-                    sentinel.announce_port : g_pserver->port;
+    else if (g_pserver->tls_replication && g_pserver->tls_port) announce_port = g_pserver->tls_port;
+    else announce_port = g_pserver->port;
 
     /* Format and send the Hello message. */
     snprintf(payload,sizeof(payload),
@@ -3964,11 +3993,14 @@ int sentinelSendSlaveOf(sentinelRedisInstance *ri, const char *host, int port) {
      * an issue because CLIENT is variadic command, so Redis will not
      * recognized as a syntax error, and the transaction will not fail (but
      * only the unsupported command will fail). */
+    for (int type = 0; type < 2; type++) {
         retval = redisAsyncCommand(ri->link->cc,
-        sentinelDiscardReplyCallback, ri, "%s KILL TYPE normal",
+            sentinelDiscardReplyCallback, ri, "%s KILL TYPE %s",
-        sentinelInstanceMapCommand(ri,"CLIENT"));
+            sentinelInstanceMapCommand(ri,"CLIENT"),
+            type == 0 ? "normal" : "pubsub");
         if (retval == C_ERR) return retval;
         ri->link->pending_commands++;
+    }
 
     retval = redisAsyncCommand(ri->link->cc,
         sentinelDiscardReplyCallback, ri, "%s",
@@ -4278,7 +4310,7 @@ void sentinelFailoverDetectEnd(sentinelRedisInstance *master) {
             sentinelRedisInstance *slave = (sentinelRedisInstance*)dictGetVal(de);
             int retval;
 
-            if (slave->flags & (SRI_RECONF_DONE|SRI_RECONF_SENT)) continue;
+            if (slave->flags & (SRI_PROMOTED|SRI_RECONF_DONE|SRI_RECONF_SENT)) continue;
             if (slave->link->disconnected) continue;
 
             retval = sentinelSendSlaveOf(slave,