Merge main with oss release sep29 2022 (#521)
* need to include stdint for uintptr_t
* need to include stdint for uintptr_t
* use atomic_load for g_pserver->mstime
* use atomic_load for g_pserver->mstime
* Integrate readwritelock with Pro Code
* Integrate readwritelock with Pro Code
* Defensive asserts for RWLock
* Defensive asserts for RWLock
* Save and restore master info in rdb to allow active replica partial sync (#371)
* save replid for all masters in rdb
* expanded rdbSaveInfo to hold multiple master structs
* parse repl-masters from rdb
* recover replid info from rdb in active replica mode, attempt partial sync
* save offset from rdb into correct variable
* don't change replid based on master in active rep
* save and load psync info from correct fields
* Save and restore master info in rdb to allow active replica partial sync (#371)
* save replid for all masters in rdb
* expanded rdbSaveInfo to hold multiple master structs
* parse repl-masters from rdb
* recover replid info from rdb in active replica mode, attempt partial sync
* save offset from rdb into correct variable
* don't change replid based on master in active rep
* save and load psync info from correct fields
* placement new instead of memcpy
* placement new instead of memcpy
* Remove asserts, RW lock can go below zero in cases of aeAcquireLock
* Remove asserts, RW lock can go below zero in cases of aeAcquireLock
* Inclusive language
* Inclusive language
* update packaging for OS merge
* update packaging for OS merge
* modify dockerfile to build within image
* modify dockerfile to build within image
* Make active client balancing a configurable option
* Make active client balancing a configurable option
* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior
* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior
* Only run the tls-name-validation test if --tls is passed into runtest
* Only run the tls-name-validation test if --tls is passed into runtest
* Fix KeyDB not building with TLS < 1.1.1
* Fix KeyDB not building with TLS < 1.1.1
* update changelog to use replica as terminology
* update changelog to use replica as terminology
* update copyright
* update copyright
* update deb copyright
* update deb copyright
* call aeThreadOnline() earlier
* call aeThreadOnline() earlier
* Removed mergeReplicationId
* Removed mergeReplicationId
* acceptTLS is threadsafe like the non TLS version
* acceptTLS is threadsafe like the non TLS version
* setup Machamp ci
* setup Machamp ci
* make build_test.sh executable
* make build_test.sh executable
* PSYNC production fixes
* PSYNC production fixes
* fix the Machamp build
* fix the Machamp build
* break into tests into steps
* break into tests into steps
* Added multimaster test
* Added multimaster test
* Update ci.yml
Change min tested version to 18.04
* Update ci.yml
Change min tested version to 18.04
* fork lock for all threads, use fastlock for readwritelock
* fork lock for all threads, use fastlock for readwritelock
* hide forklock object in ae
* hide forklock object in ae
* only need to include readwritelock in ae
* only need to include readwritelock in ae
* time thread lock uses fastlock instead of std::mutex
* time thread lock uses fastlock instead of std::mutex
* set thread as offline when waiting for time thread lock
* set thread as offline when waiting for time thread lock
* update README resource links
* update README resource links
* Fix MALLOC=memkind build issues
* Fix MALLOC=memkind build issues
* Fix module test break
* Fix module test break
* Eliminate firewall dialogs on mac for regular and cluster tests. There are still issues with the sentinel tests but attempting to bind only to localhost causes failures
* Eliminate firewall dialogs on mac for regular and cluster tests. There are still issues with the sentinel tests but attempting to bind only to localhost causes failures
* remove unused var in networking.cpp
* remove unused var in networking.cpp
* check ziplist len to avoid crash on empty ziplist convert
* check ziplist len to avoid crash on empty ziplist convert
* remove nullptr subtraction
* remove nullptr subtraction
* cannot mod a pointer
* cannot mod a pointer
* need to include stdint for uintptr_t
* need to include stdint for uintptr_t
* use atomic_load for g_pserver->mstime
* use atomic_load for g_pserver->mstime
* Integrate readwritelock with Pro Code
* Integrate readwritelock with Pro Code
* Defensive asserts for RWLock
* Defensive asserts for RWLock
* Save and restore master info in rdb to allow active replica partial sync (#371)
* save replid for all masters in rdb
* expanded rdbSaveInfo to hold multiple master structs
* parse repl-masters from rdb
* recover replid info from rdb in active replica mode, attempt partial sync
* save offset from rdb into correct variable
* don't change replid based on master in active rep
* save and load psync info from correct fields
* Save and restore master info in rdb to allow active replica partial sync (#371)
* save replid for all masters in rdb
* expanded rdbSaveInfo to hold multiple master structs
* parse repl-masters from rdb
* recover replid info from rdb in active replica mode, attempt partial sync
* save offset from rdb into correct variable
* don't change replid based on master in active rep
* save and load psync info from correct fields
* placement new instead of memcpy
* placement new instead of memcpy
* Remove asserts, RW lock can go below zero in cases of aeAcquireLock
* Remove asserts, RW lock can go below zero in cases of aeAcquireLock
* Inclusive language
* Inclusive language
* call aeThreadOnline() earlier
* call aeThreadOnline() earlier
* Removed mergeReplicationId
* Removed mergeReplicationId
* Make active client balancing a configurable option
* Make active client balancing a configurable option
* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior
* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior
* acceptTLS is threadsafe like the non TLS version
* acceptTLS is threadsafe like the non TLS version
* PSYNC production fixes
* PSYNC production fixes
* Ensure we are responsive during storagecache clears
* Ensure we are responsive during storagecache clears
* Ensure recreated tables use the same settings as ones made at boot
* Ensure recreated tables use the same settings as ones made at boot
* Converted some existing PSYNC tests for multimaster
* Converted some existing PSYNC tests for multimaster
* Inclusive language fix
* Inclusive language fix
* Cleanup test suite
* Cleanup test suite
* Updated test replica configs so tests make sense
* Updated test replica configs so tests make sense
* active-rep test reliability
* active-rep test reliability
* Quick fix to make psync tests work
* Quick fix to make psync tests work
* Fix PSYNC test crashes
* Fix PSYNC test crashes
* Ensure we force moves not copies when ingesting bulk insert files
* Ensure we force moves not copies when ingesting bulk insert files
* Disable async for hget commands as it is not ready
* Disable FLASH
* Fix crash in save of masterinfo
* Fix musl/Alpine build failures
* Remove unnecessary libs
* update readme
* update readme
* remove Enterprise references
* Limit max overage to 20% during RDB save
* Delete COPYING to replace with BSD license
* update deb master changelog
* Update license
* Fix Readme typo from github org transition
Replace mention of scratch-file-path with db-s3-object
* Fix reference counting failure in the dict. This is caused by std::swap also swapping refcounts
* Fix assertion in async rehash
* Prevent crash on shutdown by avoiding dtors (they are unnecessary anyways)
* Initialize noshrink, it was dangling
* Prevent us from starting a rehash when one wasn't already in progress. This can cause severe issues for snapshots
* Avoid unnecessary rehashing when a rehash is abandoned
* Dictionary use correct acquire/release semantics
* Add fence barriers for the repl backlog (important for AARCH64 and other weak memory models)
* Silence TSAN errors on ustime and mstime. Every CPU we support is atomic on aligned ints, but correctness matters
* Disable async commands by default
* Fix TSAN warnings on the repl backlog
* Merge OSS back into pro
* Fix unmerged files
* Fix O(n^2) algorithm in the GC cleanup logic
* Fix crash in expire when a snapshot is in flight. Caused by a perf optimization getting the expire map out of sync with the val
* On Alpine we must have a reasonable stack size
* Revert ci.yml to unstable branch version
* Implements the soft shutdown feature to allow clients to cooperatively disconnect preventing disruption during shutdown
* Ensure clean shutdown with multiple threads
* update dockerfiles
* update deb pkg references and changelog
* update gem reference
* lpGetInteger returns int64_t, avoid overflow (#10068)
Fix #9410
Crucial for the ms and sequence deltas, but I changed all
calls, just in case (e.g. "flags")
Before this commit:
`ms_delta` and `seq_delta` could have overflown, causing `currid` to be wrong,
which in turn would cause `streamTrim` to trim the entire rax node (see new test)
* Fix issue #454 (BSD build break)
* Do not allow commands to run in background when in eval, Issue #452
* Fix certificate leak during connection when tls-allowlists are used
* Fix issue #480
* Fix crash running INFO command while a disk based backlog is set
* check tracking per db
* fix warnings
* Fix a race when undoConnectWithMaster changes mi->repl_transfer_s but the connection is not yet closed and the event handler runs
* Fix a race in processChanges/trackChanges with rdbLoadRio by acquiring the lock when trackChanges is set
* Fix ASAN use after free
* Additional fixes
* Fix integer overflow of the track changes counter
* Fix P99 latency issue for TLS where we leave work for the next event loop
tlsProcessPendingData() needs to be called before we execute queued commands because it may enqueue more commands
* Fix race removing key cache
* Prevent crash on load in long running KeyDB instances
* Fixes a crash where the server assertion failed when the key exists in DB during RDB load
* Remove old assertion which is commented out.
* avoid from instatiating EpochHolder multiple times to improve performance and cpu utilization
* avoid from instatiating EpochHolder multiple times to improve performance and cpu utilization
* src\redis-cli.c: fix potential null pointer dereference found by cppcheck
src\redis-cli.c:5488:35: warning: Either the condition
'!table' is redundant or there is possible null pointer dereference:
table. [nullPointerRedundantCheck]
* Fix Issue #486
* Workaround bug in snapshot sync - abort don't crash
* Improve reliability of async parts of the soft shutdown tests
* Improve reliability of fragmentation tests
* Verify that partial syncs do indeed occur
* Fix O(n) algorithm in INFO command
* Remove incorrect assert that fires when the repl backlog is used fully
* Make building flash optional
* Remove unneeded gitlab CI file
* [BUG] Moves key to another DB, the source key was removed if the move failed due to the key exists in the destination db #497 (#498)
Co-authored-by: Paul Chen <mingchen@Mings-MacBook-Pro.local>
* trigger repl_curr_off!= master_repl_offset assert failure when having pending write case
* use debug for logging the message instead
* rocksdb log using up the diskspace on flash (#519)
* Fix OpenSSL 3.0.x related issues. (#10291)
* Drop obsolete initialization calls.
* Use decoder API for DH parameters.
* Enable auto DH parameters if not explicitly used, which should be the
preferred configuration going forward.
* remove unnecessary forward declaration
* remove internal ci stuff
* remove more internal ci/publishing
* submodule update step
* use with syntax instead
* bump ci ubuntu old ver as latest is now 22.04
* include submodules on all ci jobs
* install all deps for all ci jobs
Co-authored-by: Vivek Saini <vsaini@snapchat.com>
Co-authored-by: Christian Legge <christian@eqalpha.com>
Co-authored-by: benschermel <bschermel@snapchat.com>
Co-authored-by: John Sully <john@csquare.ca>
Co-authored-by: zliang <zliang@snapchat.com>
Co-authored-by: malavan <malavan@eqalpha.com>
Co-authored-by: John Sully <jsully@snapchat.com>
Co-authored-by: jfinity <38383673+jfinity@users.noreply.github.com>
Co-authored-by: benschermel <43507366+benschermel@users.noreply.github.com>
Co-authored-by: guybe7 <guy.benoish@redislabs.com>
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
Co-authored-by: root <paul.chen1@huawei.com>
Co-authored-by: Ilya Shipitsin <chipitsine@gmail.com>
Co-authored-by: Paul Chen <32553156+paulmchen@users.noreply.github.com>
Co-authored-by: Paul Chen <mingchen@Mings-MacBook-Pro.local>
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2022-12-14 12:17:36 -05:00
keydb (6:6.3.1-1distribution_placeholder) codename_placeholder; urgency=medium
* This point release contains fixes to bugs related to expires, active-rep, and rdb saving
* Issues fixed: #419, #422, #428
* PRs: #426, #429, #431, #433
-- Ben Schermel <ben@eqalpha.com> Mon, 23 May 2022 20:00:37 +0000
2022-05-18 01:29:15 +00:00
keydb (6:6.3.0-1distribution_placeholder) codename_placeholder; urgency=medium
* This release open sources KeyDB Enterprise features into the open source project along with PSYNC for active replication
* Partial synchronization for active replication is introduced
* MVCC introduced into codebase from KeyDB Enterprise
* Async commands added: GET, MGET. These will see perf improvements
* KEYS and SCAN commands will no longer be blocking calls
* Async Rehash implemented for additional stability to perf
* IStorage interface added
* In-process background saving (forkless) to comply with maxmemory setting
* See v6.3.0 tagged release notes on github for a detailed explanation of these changes
-- Ben Schermel <ben@eqalpha.com> Wed, 11 May 2022 20:00:37 +0000
2022-04-15 23:09:44 -04:00
keydb (6:6.2.2-1distribution_placeholder) codename_placeholder; urgency=medium
2020-07-10 04:13:42 +00:00
2022-04-15 23:09:44 -04:00
* Acquire lock in module.cpp to fix module test break
* Fix usage of nanosleep() on Apple hardware
* Fix networking deadlock (#183)
* Remove redundant zfree() in keydb-benchmark
* Issues resolved: #378, #384
2020-07-10 04:13:42 +00:00
2022-04-15 23:09:44 -04:00
-- Ben Schermel <ben@eqalpha.com> Sat, 15 Jan 2022 20:00:37 +0000
keydb (6:6.2.1-1distribution_placeholder) codename_placeholder; urgency=medium
* This release of KeyDB is at parity with Redis 6.2.6. In addition, the following changes were included:
* Systemd support for release packages
* Fixes to buffer overflows in the `BITOP SHIFT` and `CRON` commands
* Server times are now computed on a seperate thread to improve performance
* Now enforces syslog identity and facility as soon as possible
* Removed erroneous use of LOG_... flags when using syslog
* Fixed erroneous `#endif` leading to build errors on some platforms
* Fixed the incorrect counting of client connections
* Issues resolved: #355, #370
-- Ben Schermel <ben@eqalpha.com> Wed, 17 Nov 2021 20:00:37 +0000
keydb (6:6.2.0-1distribution_placeholder) codename_placeholder; urgency=medium
* Removed unused command line options(-a, -d, -P, -r, -q, --csv, -l, -I, -e, --precision, --cluster, --enable-tracking)
* --ms option changed to --time, added --clients, --host, --port, and --threads options
* Added keydb-diagnostic-tool
* Improved test reliability
* Fixed memory leak in keydb-benchmark
* Fixed a number of race conditions
* Fixed OSX build issues
* Feature parity with Redis 6.2.3
* Fixed Issues: #323 #325 #276
-- Ben Schermel <ben@eqalpha.com> Thu, 12 Aug 2021 20:00:37 +0000
keydb (6:6.0.18-1distribution_placeholder) codename_placeholder; urgency=medium
* 6.0.18 - This release contains a number of improvements to stability and performance.
* Additional tests were added with improvements to existing test reliability
* Merged Redis 6.0.10 into KeyDB
* Issues resolved in this release: #214, #222, #238, #257, #273, #276, #285
-- Ben Schermel <ben@eqalpha.com> Fri, 26 Mar 2021 16:00:00 +0000
keydb (6:6.0.16-1distribution_placeholder) codename_placeholder; urgency=medium
* 6.0.16 - This release contains significant performance improvements as well as fixes to the following issues: #234, #236, #233, #207, #229, #231
* Reduced memory consumption when Active Replication is not used
* Reduced CPU consumption
* Improved TLS performance
* replica-quorum config for multi-master.Intended to be used with "replica-serve-stale-data no", if at least N replicas are connected we will consider our master link up and serve data.
* Subkey expires performance improvement for big sets
-- Ben Schermel <ben@eqalpha.com> Mon, 28 Sep 2020 16:00:00 +0000
keydb (6:6.0.13-1distribution_placeholder) codename_placeholder; urgency=medium
* 6.0.13 New features added: KEYDB.MEXISTS, REPLICAOF REMOVE, multi-master-no-forward configuration option
* KEYDB.MEXISTS - return exactly which keys exist in the database not just the count
* REPLICAOF REMOVE - Remove specific masters in a multi master setup
* multi-master-no-forward configuration option - Reduce network traffic in mesh topology multimaster setups
* Redis 6.0.5 feature parity
* Fixed Issues: 194 Bad directive or wrong number of arguments error on 'lazyfree-lazy-user-del no'; 209 Timeout using modules; 210 Databases are not merged on multi-master sync; 211 rpm packages now signed; 212 rpm file permissions updated
-- Ben Schermel <ben@eqalpha.com> Mon, 13 Jul 2020 8:00:00 +0000
keydb (6:6.0.9-1distribution_placeholder) codename_placeholder; urgency=medium
* 6.0.9 Addressing issues 187 - cpu lockup with subkey expire, 190 - missing sentinel binary in keydb-tools.
-- Ben Schermel <ben@eqalpha.com> Sun, 07 Jun 2020 18:00:37 +0000
keydb (6:6.0.8-1distribution_placeholder) codename_placeholder; urgency=medium
* This is our first release to fully support all Redis 6.0.4 features, including but not limited to: TLS Support (fully supports multithreading!), Client side caching, RESP 3 Support
* KeyDB Has also added the following new features: Improved memory efficiency for short strings, Fastlock autotuning, KeyDB.HRENAME (rename a member of a hash)In addition we've spent a lot of time focussing on stability.
2022-04-20 14:28:18 -04:00
* The following bug fixes are resolved: 150 - Deadlock in ReplicationFeedMonitors, 170 - KeyDB dying via SIGABORT, 180 - crash after setting maxclients via cmd line, 169 - Write performance of a master dropped sharply when a replica is added
2022-04-15 23:09:44 -04:00
* In addition untracked issues were resolved: Potential deadlock when entering futex sleep, Pub/Sub Async messages may not be sent in a timely manner when load is low, Excessive logging during failed RREPLAY, KeyDB unresponsive handling clients on different threads during RDB load
* Updated deb package builds to build from source and phasing out init.d. dh_installsystemd from deb helper 11 used where applicable
* Naming conventions are now updated [ keydbpackage_version-build~distribution_architecture] and master changelog included and maintained
* keydb-pro-server binary is no longer inluded in the open source package
-- Ben Schermel <ben@eqalpha.com> Mon, 1 Jun 2020 8:00:37 +0000
keydb (5:5.3.3-1distribution_placeholder) codename_placeholder; urgency=medium
* 5.3.3 Updating deb source package, naming conventions, and build scripts.
-- Ben Schermel <ben@eqalpha.com> Wed, 06 May 2020 8:00:37 +0000
keydb (5:5.1.12-1chl1distribution_placeholder) codename_placeholder; urgency=medium
* 5.1.1 update. This update fixes several rare deadlock scenarios. Deadlock detection is also added.
-- Ben Schermel <ben@eqalpha.com> Fri, 25 Oct 2019 8:00:37 +0000
keydb (5:5.1.11-1chl1distribution_placeholder) codename_placeholder; urgency=medium
* 5.1 release. This release includes subkey expires (EXPIREMEMBER/EXPIREMEMBERAT), with updates to PTTL/TTL accordingly. New OBJECT LASTMODIFIED, BITIOP LSHIFT & BITOP RSHIFT commands. See https://docs.keydb.dev/blog/2019/10/20/blog-post/ for detailed review of release.
-- Ben Schermel <ben@eqalpha.com> Mon, 21 Oct 2019 8:00:37 +0000
keydb (5:5.0.1-1chl1distribution_placeholder) codename_placeholder; urgency=medium
* Arm build now included for bionic package
-- Ben Schermel <ben@eqalpha.com> Wed, 21 Aug 2019 22:58:37 +0000
keydb (5:5.0.0-1chl1distribution_placeholder) codename_placeholder; urgency=medium
* Initial release of KeyDB PPA. This PPA was originally derived from https://launchpad.net/~chris-lea/+archive/ubuntu/redis-server
-- Ben Schermel <ben@eqalpha.com> Wed, 21 Aug 2019 2:58:37 +0000
