We have 24 total bits of space in each object in order to implement
an LFU (Least Frequently Used) eviction policy.
We split the 24 bits into two fields:
8 bits 16 bits
+--------+----------------+
| LOG_C | Last decr time |
+--------+----------------+
LOG_C is a logarithmic counter that provides an indication of the access
frequency. However this field must also be deceremented otherwise what used
to be a frequently accessed key in the past, will remain ranked like that
forever, while we want the algorithm to adapt to access pattern changes.
So the remaining 16 bits are used in order to store the "decrement time",
a reduced-precision unix time (we take 16 bits of the time converted
in minutes since we don't care about wrapping around) where the LOG_C
counter is halved if it has an high value, or just decremented if it
has a low value.
New keys don't start at zero, in order to have the ability to collect
some accesses before being trashed away, so they start at COUNTER_INIT_VAL.
The logaritmic increment performed on LOG_C takes care of COUNTER_INIT_VAL
when incrementing the key, so that keys starting at COUNTER_INIT_VAL
(or having a smaller value) have a very high chance of being incremented
on access.
The simulation starts with a power-law access pattern, and later converts
into a flat access pattern in order to see how the algorithm adapts.
Currenty the decrement operation period is 1 minute, however note that
it is not guaranteed that each key will be scanned 1 time every minute,
so the actual frequency can be lower. However under high load, we access
3/5 keys every newly inserted key (because of how Redis eviction works).
This is a work in progress at this point to evaluate if this works well.
The LRU eviction code used to make local choices: for each DB visited it
selected the best key to evict. This was repeated for each DB. However
this means that there could be DBs with very frequently accessed keys
that are targeted by the LRU algorithm while there were other DBs with
many better candidates to expire.
This commit attempts to fix this problem for the LRU policy. However the
TTL policy is still not fixed by this commit. The TTL policy will be
fixed in a successive commit.
This is an initial (partial because of TTL policy) fix for issue #2647.
To destroy and recreate the pool[].key element is slow, so we allocate
in pool[].cached SDS strings that can account up to 255 chars keys and
try to reuse them. This provides a solid 20% performance improvement
in real world workload alike benchmarks.
We start from the end of the pool to the initial item, zero-ing
every entry we use or every ghost entry, there is nothing to memmove
since to the right everything should be already set to NULL.
1. Scan keys with pause to account for actual LRU precision.
2. Test cross-DB with 100 keys allocated in DB1.
3. Output results that don't fluctuate depending on number of keys.
4. Output results in percentage to make more sense.
5. Save file instead of outputting to STDOUT.
6. Support running multiple times with average of outputs.
7. Label each square (DIV) with its ID as HTML title.
The rio structure is referenced in the global 'riostate' structure
in order for the logging functions to be always able to access the state
of the "pseudo-loading" of the RDB, needed for the check.
Courtesy of Valgrind.
They were under /deps since they originate from a different source tree,
however at this point they are very modified and we took ownership of
both the files making changes, fixing bugs, so there is no upgrade path
from the original code tree.
Given that, better to move the code under /src with proper dependencies
and with a more simpler editing experience.
strict_strtoll() has a bug that reports the empty string as ok and
parses it as zero.
Apparently nobody ever replaced this old call with the faster/saner
string2ll() which is used otherwise in the rest of the Redis core.
This commit close#3333.
In issues #3361 / #3365 a problem was reported / fixed with redis-cli
not updating correctly the current DB on error after SELECT.
In theory this bug was fixed in 0042fb0e, but actually the commit only
fixed the prompt updating, not the fact the state was set in a wrong
way.
This commit removes the check in the prompt update, now that hopefully
it is the state that is correct, there is no longer need for this check.
This commit both fixes the crash reported with issue #3364 and
also properly closes the old links after the Sentinel address for the
other masters gets updated.
The two problems where:
1. The Sentinel that switched address may not monitor all the masters,
it is possible that there is no match, and the 'match' variable is
NULL. Now we check for no match and 'continue' to the next master.
2. By ispecting the code because of issue "1" I noticed that there was a
problem in the code that disconnects the link of the Sentinel that
needs the address update. Basically link->disconnected is non-zero
even if just *a single link* (cc -- command link or pc -- pubsub
link) are disconnected, so to check with if (link->disconnected)
in order to close the links risks to leave one link connected.
I was able to manually reproduce the crash at "1" and verify that the
commit resolves the issue.
Close#3364.
So far we used an external program (later executed within Redis) and
parser in order to check RDB files for correctness. This forces, at each
RDB format update, to have two copies of the same format implementation
that are hard to keep in sync. Morover the former RDB checker only
checked the very high-level format of the file, without actually trying
to load things in memory. Certain corruptions can only be handled by
really loading key-value pairs.
This first commit attempts to unify the Redis RDB loadig code with the
task of checking the RDB file for correctness. More work is needed but
it looks like a sounding direction so far.
The quicklist takes a cached version of the ziplist representation size
in bytes. The implementation must update this length every time the
underlying ziplist changes. However quicklistReplaceAtIndex() failed to
fix the length.
During LSET calls, the size of the ziplist blob and the cached size
inside the quicklist diverged. Later, when this size is used in an
authoritative way, for example during nodes splitting in order to copy
the nodes, we end with a duplicated node that may contain random
garbage.
This commit should fix issue #3343, however several problems were found
reviewing the quicklist.c code in search of this bug that should be
addressed soon or later.
For example:
1. To take a cached ziplist length is fragile since failing to update it
leads to this kind of issues.
2. The node splitting code needs auditing. For example it works just for
a side effect of ziplistDeleteRange() to be able to cope with a wrong
count of elements to remove. The code inside quicklist.c assumes that
-1 means "delete till the end" while actually it's just a count of how
many elements to delete, and is an unsigned count. So -1 gets converted
into the maximum integer, and just by chance the ziplist code stops
deleting elements after there are no more to delete.
3. Node splitting is extremely inefficient, it copies the node and
removes elements from both nodes even when actually there is to move a
single entry from one node to the other, or when the new resulting node
is empty at all so there is nothing to copy but just to create a new
node.
However at least for Redis 3.2 to introduce fresh code inside
quicklist.c may be even more risky, so instead I'm writing a better
fuzzy tester to stress the internals a bit more in order to anticipate
other possible bugs.
This bug was found using a fuzzy tester written after having some clue
about where the bug could be. The tester eventually created a ~2000
commands sequence able to always crash Redis. I wrote a better version
of the tester that searched for the smallest sequence that could crash
Redis automatically. Later this smaller sequence was minimized by
removing random commands till it still crashed the server. This resulted
into a sequence of 7 commands. With this small sequence it was just a
matter of filling the code with enough printf() to understand enough
state to fix the bug.
