An exposed Redis instance on the internet can be cause of serious
issues. Since Redis, by default, binds to all the interfaces, it is easy
to forget an instance without any protection layer, for error.
Protected mode try to address this feature in a soft way, providing a
layer of protection, but giving clues to Redis users about why the
server is not accepting connections.
When protected mode is enabeld (the default), and if there are no
minumum hints about the fact the server is properly configured (no
"bind" directive is used in order to restrict the server to certain
interfaces, nor a password is set), clients connecting from external
intefaces are refused with an error explaining what to do in order to
fix the issue.
Clients connecting from the IPv4 and IPv6 lookback interfaces are still
accepted normally, similarly Unix domain socket connections are not
restricted in any way.
An exposed Redis instance on the internet can be cause of serious
issues. Since Redis, by default, binds to all the interfaces, it is easy
to forget an instance without any protection layer, for error.
Protected mode try to address this feature in a soft way, providing a
layer of protection, but giving clues to Redis users about why the
server is not accepting connections.
When protected mode is enabeld (the default), and if there are no
minumum hints about the fact the server is properly configured (no
"bind" directive is used in order to restrict the server to certain
interfaces, nor a password is set), clients connecting from external
intefaces are refused with an error explaining what to do in order to
fix the issue.
Clients connecting from the IPv4 and IPv6 lookback interfaces are still
accepted normally, similarly Unix domain socket connections are not
restricted in any way.
For non existing keys, we don't want to send -ASK redirections to
MIGRATE, since when moving slots from the migrating node to the
importing node, we want just to ignore keys that are no longer there.
They may be expired or deleted between the GETKEYSINSLOT call and the
MIGRATE call. Otherwise this causes an error during migrations with
redis-trib (or equivalent cluster management tools).
For non existing keys, we don't want to send -ASK redirections to
MIGRATE, since when moving slots from the migrating node to the
importing node, we want just to ignore keys that are no longer there.
They may be expired or deleted between the GETKEYSINSLOT call and the
MIGRATE call. Otherwise this causes an error during migrations with
redis-trib (or equivalent cluster management tools).
In issue #2948 a crash was reported in processCommand(). Later Oran Agra
(@oranagra) traced the bug (in private chat) in the following sequence
of events:
1. Some maxmemory is set.
2. The slave is the currently active client and is executing PING or
REPLCONF or whatever a slave can send to its master.
3. freeMemoryIfNeeded() is called since maxmemory is set.
4. flushSlavesOutputBuffers() is called by freeMemoryIfNeeded().
5. During slaves buffers flush, a write error could be encoutered in
writeToClient() or sendReplyToClient() depending on the version of
Redis. This will trigger freeClient() against the currently active
client, so a segmentation fault will likely happen in
processCommand() immediately after the call to freeMemoryIfNeeded().
There are different possible fixes:
1. Add flags to writeToClient() (recent versions code base) so that
we can ignore the write errors, and use this flag in
flushSlavesOutputBuffers(). However this is not simple to do in older
versions of Redis.
2. Use freeClientAsync() during write errors. This works but changes the
current behavior of releasing clients ASAP when possible. Normally
we write to clients during the normal event loop processing, in the
writable client, where there is no active client, so no care must be
taken.
3. The fix of this commit: to detect that the current client is no
longer valid. This fix is a bit "ad-hoc", but works across all the
versions and has the advantage of not changing the remaining
behavior. Only alters what happens during this race condition,
hopefully.
In issue #2948 a crash was reported in processCommand(). Later Oran Agra
(@oranagra) traced the bug (in private chat) in the following sequence
of events:
1. Some maxmemory is set.
2. The slave is the currently active client and is executing PING or
REPLCONF or whatever a slave can send to its master.
3. freeMemoryIfNeeded() is called since maxmemory is set.
4. flushSlavesOutputBuffers() is called by freeMemoryIfNeeded().
5. During slaves buffers flush, a write error could be encoutered in
writeToClient() or sendReplyToClient() depending on the version of
Redis. This will trigger freeClient() against the currently active
client, so a segmentation fault will likely happen in
processCommand() immediately after the call to freeMemoryIfNeeded().
There are different possible fixes:
1. Add flags to writeToClient() (recent versions code base) so that
we can ignore the write errors, and use this flag in
flushSlavesOutputBuffers(). However this is not simple to do in older
versions of Redis.
2. Use freeClientAsync() during write errors. This works but changes the
current behavior of releasing clients ASAP when possible. Normally
we write to clients during the normal event loop processing, in the
writable client, where there is no active client, so no care must be
taken.
3. The fix of this commit: to detect that the current client is no
longer valid. This fix is a bit "ad-hoc", but works across all the
versions and has the advantage of not changing the remaining
behavior. Only alters what happens during this race condition,
hopefully.
The old test, designed to do a transformation on the bits that was
invertible, in order to avoid touching the original memory content, was
not effective as it was redis-server --test-memory. The former often
reported OK while the latter was able to spot the error.
So the test was substituted with one that may perform better, however
the new one must backup the memory tested, so it tests memory in small
pieces. This limits the effectiveness because of the CPU caches. However
some attempt is made in order to trash the CPU cache between the fill
and the check stages, but not for the addressing test unfortunately.
We'll see if this test will be able to find errors where the old failed.
The old test, designed to do a transformation on the bits that was
invertible, in order to avoid touching the original memory content, was
not effective as it was redis-server --test-memory. The former often
reported OK while the latter was able to spot the error.
So the test was substituted with one that may perform better, however
the new one must backup the memory tested, so it tests memory in small
pieces. This limits the effectiveness because of the CPU caches. However
some attempt is made in order to trash the CPU cache between the fill
and the check stages, but not for the addressing test unfortunately.
We'll see if this test will be able to find errors where the old failed.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:
1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
which is planned.
Close#2710Close#2711
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:
1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
which is planned.
Close#2710Close#2711
We need to process replies after errors in order to delete keys
successfully transferred. Also argument rewriting was fixed since
it was broken in several ways. Now a fresh argument vector is created
and set if we are acknowledged of at least one key.
