38169682f0
* add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
This directory contains all Redis dependencies, except for the libc that should be provided by the operating system.
- Jemalloc is our memory allocator, used as replacement for libc malloc on Linux by default. It has good performances and excellent fragmentation behavior. This component is upgraded from time to time.
- hiredis is the official C client library for Redis. It is used by redis-cli, redis-benchmark and Redis Sentinel. It is part of the Redis official ecosystem but is developed externally from the Redis repository, so we just upgrade it as needed.
- linenoise is a readline replacement. It is developed by the same authors of Redis but is managed as a separated project and updated as needed.
- lua is Lua 5.1 with minor changes for security and additional libraries.
How to upgrade the above dependencies
Jemalloc
Jemalloc is modified with changes that allow us to implement the Redis active defragmentation logic. However this feature of Redis is not mandatory and Redis is able to understand if the Jemalloc version it is compiled against supports such Redis-specific modifications. So in theory, if you are not interested in the active defragmentation, you can replace Jemalloc just following these steps:
- Remove the jemalloc directory.
- Substitute it with the new jemalloc source tree.
- Edit the Makefile located in the same directory as the README you are reading, and change the --with-version in the Jemalloc configure script options with the version you are using. This is required because otherwise Jemalloc configuration script is broken and will not work nested in another git repository.
However note that we change Jemalloc settings via the configure script of Jemalloc using the --with-lg-quantum option, setting it to the value of 3 instead of 4. This provides us with more size classes that better suit the Redis data structures, in order to gain memory efficiency.
If you want to upgrade Jemalloc while also providing support for active defragmentation, in addition to the above steps you need to perform the following additional steps:
- In Jemalloc tree, file
include/jemalloc/jemalloc_macros.h.in, make sure to add#define JEMALLOC_FRAG_HINT. - Implement the function
je_get_defrag_hint()insidesrc/jemalloc.c. You can see how it is implemented in the current Jemalloc source tree shipped with Redis, and rewrite it according to the new Jemalloc internals, if they changed, otherwise you could just copy the old implementation if you are upgrading just to a similar version of Jemalloc.
Hiredis
Hiredis uses the SDS string library, that must be the same version used inside Redis itself. Hiredis is also very critical for Sentinel. Historically Redis often used forked versions of hiredis in a way or the other. In order to upgrade it is advised to take a lot of care:
- Check with diff if hiredis API changed and what impact it could have in Redis.
- Make sure that the SDS library inside Hiredis and inside Redis are compatible.
- After the upgrade, run the Redis Sentinel test.
- Check manually that redis-cli and redis-benchmark behave as expected, since we have no tests for CLI utilities currently.
Linenoise
Linenoise is rarely upgraded as needed. The upgrade process is trivial since Redis uses a non modified version of linenoise, so to upgrade just do the following:
- Remove the linenoise directory.
- Substitute it with the new linenoise source tree.
Lua
We use Lua 5.1 and no upgrade is planned currently, since we don't want to break Lua scripts for new Lua features: in the context of Redis Lua scripts the capabilities of 5.1 are usually more than enough, the release is rock solid, and we definitely don't want to break old scripts.
So upgrading of Lua is up to the Redis project maintainers and should be a manual procedure performed by taking a diff between the different versions.
Currently we have at least the following differences between official Lua 5.1 and our version:
- Makefile is modified to allow a different compiler than GCC.
- We have the implementation source code, and directly link to the following external libraries:
lua_cjson.o,lua_struct.o,lua_cmsgpack.oandlua_bit.o. - There is a security fix in
ldo.c, line 498: The check forLUA_SIGNATURE[0]is removed in order to avoid direct bytecode execution.