There was a race condition in the AOF rewrite code that, with bad enough
timing, could cause a volatile key just about to expire to be turned
into a non-volatile key. The bug was never reported to cause actualy
issues, but was found analytically by an user in the Redis mailing list:
https://groups.google.com/forum/?fromgroups=#!topic/redis-db/Kvh2FAGK4Uk
This commit fixes issue #1079.
Tilt mode was too aggressive (not processing INFO output), this
resulted in a few problems:
1) Redirections were not followed when in tilt mode. This opened a
window to misinform clients about the current master when a Sentinel
was in tilt mode and a fail over happened during the time it was not
able to update the state.
2) It was possible for a Sentinel exiting tilt mode to detect a false
fail over start, if a slave rebooted with a wrong configuration
about at the same time. This used to happen since in tilt mode we
lose the information that the runid changed (reboot).
Now instead the Sentinel in tilt mode will still remove the instance
from the list of slaves if it changes state AND runid at the same
time.
Both are edge conditions but the changes should overall improve the
reliability of Sentinel.
Tilt mode was too aggressive (not processing INFO output), this
resulted in a few problems:
1) Redirections were not followed when in tilt mode. This opened a
window to misinform clients about the current master when a Sentinel
was in tilt mode and a fail over happened during the time it was not
able to update the state.
2) It was possible for a Sentinel exiting tilt mode to detect a false
fail over start, if a slave rebooted with a wrong configuration
about at the same time. This used to happen since in tilt mode we
lose the information that the runid changed (reboot).
Now instead the Sentinel in tilt mode will still remove the instance
from the list of slaves if it changes state AND runid at the same
time.
Both are edge conditions but the changes should overall improve the
reliability of Sentinel.
We used to always turn a master into a slave if the DEMOTE flag was set,
as this was a resurrecting master instance.
However the following race condition is possible for a Sentinel that
got partitioned or internal issues (tilt mode), and was not able to
refresh the state in the meantime:
1) Sentinel X is running, master is instance "A".
3) "A" fails, sentinels will promote slave "B" as master.
2) Sentinel X goes down because of a network partition.
4) "A" returns available, Sentinels will demote it as a slave.
5) "B" fails, other Sentinels will promote slave "A" as master.
6) At this point Sentinel X comes back.
When "X" comes back he thinks that:
"B" is the master.
"A" is the slave to demote.
We want to avoid that Sentinel "X" will demote "A" into a slave.
We also want that Sentinel "X" will detect that the conditions changed
and will reconfigure itself to monitor the right master.
There are two main ways for the Sentinel to reconfigure itself after
this event:
1) If "B" is reachable and already configured as a slave by other
sentinels, "X" will perform a redirection to "A".
2) If there are not the conditions to demote "A", the fact that "A"
reports to be a master will trigger a failover detection in "X", that
will end into a reconfiguraiton to monitor "A".
However if the Sentinel was not reachable, its state may not be updated,
so in case it titled, or was partiitoned from the master instance of the
slave to demote, the new implementation waits some time (enough to
guarantee we can detect the new INFO, and new DOWN conditions).
If after some time still there are not the right condiitons to demote
the instance, the DEMOTE flag is cleared.
We used to always turn a master into a slave if the DEMOTE flag was set,
as this was a resurrecting master instance.
However the following race condition is possible for a Sentinel that
got partitioned or internal issues (tilt mode), and was not able to
refresh the state in the meantime:
1) Sentinel X is running, master is instance "A".
3) "A" fails, sentinels will promote slave "B" as master.
2) Sentinel X goes down because of a network partition.
4) "A" returns available, Sentinels will demote it as a slave.
5) "B" fails, other Sentinels will promote slave "A" as master.
6) At this point Sentinel X comes back.
When "X" comes back he thinks that:
"B" is the master.
"A" is the slave to demote.
We want to avoid that Sentinel "X" will demote "A" into a slave.
We also want that Sentinel "X" will detect that the conditions changed
and will reconfigure itself to monitor the right master.
There are two main ways for the Sentinel to reconfigure itself after
this event:
1) If "B" is reachable and already configured as a slave by other
sentinels, "X" will perform a redirection to "A".
2) If there are not the conditions to demote "A", the fact that "A"
reports to be a master will trigger a failover detection in "X", that
will end into a reconfiguraiton to monitor "A".
However if the Sentinel was not reachable, its state may not be updated,
so in case it titled, or was partiitoned from the master instance of the
slave to demote, the new implementation waits some time (enough to
guarantee we can detect the new INFO, and new DOWN conditions).
If after some time still there are not the right condiitons to demote
the instance, the DEMOTE flag is cleared.
Sentinel redirected to the master if the instance changed runid or it
was the first time we got INFO, and a role change was detected from
master to slave.
While this is a good idea in case of slave->master, since otherwise we
could detect a failover without good reasons just after a reboot with a
slave with a wrong configuration, in the case of master->slave
transition is much better to always perform the redirection for the
following reasons:
1) A Sentinel may go down for some time. When it is back online there is
no other way to understand there was a failover.
2) Pointing clients to a slave seems to be always the wrong thing to do.
3) There is no good rationale about handling things differently once an
instance is rebooted (runid change) in that case.
Sentinel redirected to the master if the instance changed runid or it
was the first time we got INFO, and a role change was detected from
master to slave.
While this is a good idea in case of slave->master, since otherwise we
could detect a failover without good reasons just after a reboot with a
slave with a wrong configuration, in the case of master->slave
transition is much better to always perform the redirection for the
following reasons:
1) A Sentinel may go down for some time. When it is back online there is
no other way to understand there was a failover.
2) Pointing clients to a slave seems to be always the wrong thing to do.
3) There is no good rationale about handling things differently once an
instance is rebooted (runid change) in that case.
This prevents the kernel from putting too much stuff in the output
buffers, doing too heavy I/O all at once. So the goal of this commit is
to split the disk pressure due to the AOF rewrite process into smaller
spikes.
Please see issue #1019 for more information.
This prevents the kernel from putting too much stuff in the output
buffers, doing too heavy I/O all at once. So the goal of this commit is
to split the disk pressure due to the AOF rewrite process into smaller
spikes.
Please see issue #1019 for more information.
Previously redis-cli never tried to raise an error when an unrecognized
switch was encountered, as everything after the initial options is to be
transmitted to the server.
However this is too liberal, as there are no commands starting with "-".
So the new behavior is to produce an error if there is an unrecognized
switch starting with "-". This should not break past redis-cli usages
but should prevent broken options to be silently discarded.
As far the first token not starting with "-" is encountered, all the
rest is considered to be part of the command, so you cna still use
strings starting with "-" as values, like in:
redis-cli --port 6380 set foo --my-value
Previously redis-cli never tried to raise an error when an unrecognized
switch was encountered, as everything after the initial options is to be
transmitted to the server.
However this is too liberal, as there are no commands starting with "-".
So the new behavior is to produce an error if there is an unrecognized
switch starting with "-". This should not break past redis-cli usages
but should prevent broken options to be silently discarded.
As far the first token not starting with "-" is encountered, all the
rest is considered to be part of the command, so you cna still use
strings starting with "-" as values, like in:
redis-cli --port 6380 set foo --my-value
We used to copy this value into the server.cluster structure, however this
was not necessary.
The reason why we don't directly use server.cluster->node_timeout is
that things that can be configured via redis.conf need to be directly
available in the server structure as server.cluster is allocated later
only if needed in order to reduce the memory footprint of non-cluster
instances.
We used to copy this value into the server.cluster structure, however this
was not necessary.
The reason why we don't directly use server.cluster->node_timeout is
that things that can be configured via redis.conf need to be directly
available in the server structure as server.cluster is allocated later
only if needed in order to reduce the memory footprint of non-cluster
instances.
In commit de720e4 it was introduced the concept of sending a ping to
every node not receiving a ping since node_timeout/2 seconds.
However the code was located in a place that was not executed because of
a previous conditional causing the loop to re-iterate.
This caused false positives in nodes availability detection.
The current code is still not perfect as a node may be detected to be in
PFAIL state even if it does not reply for just node_timeout/2 seconds
that is not correct. There is a plan to improve this code ASAP.
In commit d728ec6 it was introduced the concept of sending a ping to
every node not receiving a ping since node_timeout/2 seconds.
However the code was located in a place that was not executed because of
a previous conditional causing the loop to re-iterate.
This caused false positives in nodes availability detection.
The current code is still not perfect as a node may be detected to be in
PFAIL state even if it does not reply for just node_timeout/2 seconds
that is not correct. There is a plan to improve this code ASAP.
When a BGSAVE fails, Redis used to flood itself trying to BGSAVE at
every next cron call, that is either 10 or 100 times per second
depending on configuration and server version.
This commit does not allow a new automatic BGSAVE attempt to be
performed before a few seconds delay (currently 5).
This avoids both the auto-flood problem and filling the disk with
logs at a serious rate.
The five seconds limit, considering a log entry of 200 bytes, will use
less than 4 MB of disk space per day that is reasonable, the sysadmin
should notice before of catastrofic events especially since by default
Redis will stop serving write queries after the first failed BGSAVE.
This fixes issue #849
When a BGSAVE fails, Redis used to flood itself trying to BGSAVE at
every next cron call, that is either 10 or 100 times per second
depending on configuration and server version.
This commit does not allow a new automatic BGSAVE attempt to be
performed before a few seconds delay (currently 5).
This avoids both the auto-flood problem and filling the disk with
logs at a serious rate.
The five seconds limit, considering a log entry of 200 bytes, will use
less than 4 MB of disk space per day that is reasonable, the sysadmin
should notice before of catastrofic events especially since by default
Redis will stop serving write queries after the first failed BGSAVE.
This fixes issue #849
