58 Commits

Author SHA1 Message Date
Alex Cope
3e528f2619 clean 2023-06-30 11:20:29 -07:00
Alex Cope
4a9a9f9292 update 2023-06-29 13:23:16 -07:00
Alex Cope
e029c40fdb revert 2023-06-28 23:19:34 -07:00
Alex Cope
4ab128b86a fix conflict 2023-06-28 17:37:33 -07:00
John Sully
f53e0337ef Port load shedding and availability-zone (#202) 2023-06-27 15:37:28 -04:00
Malavan Sotheeswaran
b50f0cc21f
flash package update (#528)
* ship packages with flash option

* deb12 fix

Co-authored-by: benschermel <bschermel@snapchat.com>
2022-12-29 15:09:53 -05:00
Malavan Sotheeswaran
f5f1bd7605
Merge main with oss release sep29 2022 (#521)
* need to include stdint for uintptr_t

* need to include stdint for uintptr_t

* use atomic_load for g_pserver->mstime

* use atomic_load for g_pserver->mstime

* Integrate readwritelock with Pro Code

* Integrate readwritelock with Pro Code

* Defensive asserts for RWLock

* Defensive asserts for RWLock

* Save and restore master info in rdb to allow active replica partial sync (#371)

* save replid for all masters in rdb

* expanded rdbSaveInfo to hold multiple master structs

* parse repl-masters from rdb

* recover replid info from rdb in active replica mode, attempt partial sync

* save offset from rdb into correct variable

* don't change replid based on master in active rep

* save and load psync info from correct fields

* Save and restore master info in rdb to allow active replica partial sync (#371)

* save replid for all masters in rdb

* expanded rdbSaveInfo to hold multiple master structs

* parse repl-masters from rdb

* recover replid info from rdb in active replica mode, attempt partial sync

* save offset from rdb into correct variable

* don't change replid based on master in active rep

* save and load psync info from correct fields

* placement new instead of memcpy

* placement new instead of memcpy

* Remove asserts, RW lock can go below zero in cases of aeAcquireLock

* Remove asserts, RW lock can go below zero in cases of aeAcquireLock

* Inclusive language

* Inclusive language

* update packaging for OS merge

* update packaging for OS merge

* modify dockerfile to build within image

* modify dockerfile to build within image

* Make active client balancing a configurable option

* Make active client balancing a configurable option

* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior

* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior

* Only run the tls-name-validation test if --tls is passed into runtest

* Only run the tls-name-validation test if --tls is passed into runtest

* Fix KeyDB not building with TLS < 1.1.1

* Fix KeyDB not building with TLS < 1.1.1

* update changelog to use replica as terminology

* update changelog to use replica as terminology

* update copyright

* update copyright

* update deb copyright

* update deb copyright

* call aeThreadOnline() earlier

* call aeThreadOnline() earlier

* Removed mergeReplicationId

* Removed mergeReplicationId

* acceptTLS is threadsafe like the non TLS version

* acceptTLS is threadsafe like the non TLS version

* setup Machamp ci

* setup Machamp ci

* make build_test.sh executable

* make build_test.sh executable

* PSYNC production fixes

* PSYNC production fixes

* fix the Machamp build

* fix the Machamp build

* break into tests into steps

* break into tests into steps

* Added multimaster test

* Added multimaster test

* Update ci.yml

Change min tested version to 18.04

* Update ci.yml

Change min tested version to 18.04

* fork lock for all threads, use fastlock for readwritelock

* fork lock for all threads, use fastlock for readwritelock

* hide forklock object in ae

* hide forklock object in ae

* only need to include readwritelock in ae

* only need to include readwritelock in ae

* time thread lock uses fastlock instead of std::mutex

* time thread lock uses fastlock instead of std::mutex

* set thread as offline when waiting for time thread lock

* set thread as offline when waiting for time thread lock

* update README resource links

* update README resource links

* Fix MALLOC=memkind build issues

* Fix MALLOC=memkind build issues

* Fix module test break

* Fix module test break

* Eliminate firewall dialogs on mac for regular and cluster tests.  There are still issues with the sentinel tests but attempting to bind only to localhost causes failures

* Eliminate firewall dialogs on mac for regular and cluster tests.  There are still issues with the sentinel tests but attempting to bind only to localhost causes failures

* remove unused var in networking.cpp

* remove unused var in networking.cpp

* check ziplist len to avoid crash on empty ziplist convert

* check ziplist len to avoid crash on empty ziplist convert

* remove nullptr subtraction

* remove nullptr subtraction

* cannot mod a pointer

* cannot mod a pointer

* need to include stdint for uintptr_t

* need to include stdint for uintptr_t

* use atomic_load for g_pserver->mstime

* use atomic_load for g_pserver->mstime

* Integrate readwritelock with Pro Code

* Integrate readwritelock with Pro Code

* Defensive asserts for RWLock

* Defensive asserts for RWLock

* Save and restore master info in rdb to allow active replica partial sync (#371)

* save replid for all masters in rdb

* expanded rdbSaveInfo to hold multiple master structs

* parse repl-masters from rdb

* recover replid info from rdb in active replica mode, attempt partial sync

* save offset from rdb into correct variable

* don't change replid based on master in active rep

* save and load psync info from correct fields

* Save and restore master info in rdb to allow active replica partial sync (#371)

* save replid for all masters in rdb

* expanded rdbSaveInfo to hold multiple master structs

* parse repl-masters from rdb

* recover replid info from rdb in active replica mode, attempt partial sync

* save offset from rdb into correct variable

* don't change replid based on master in active rep

* save and load psync info from correct fields

* placement new instead of memcpy

* placement new instead of memcpy

* Remove asserts, RW lock can go below zero in cases of aeAcquireLock

* Remove asserts, RW lock can go below zero in cases of aeAcquireLock

* Inclusive language

* Inclusive language

* call aeThreadOnline() earlier

* call aeThreadOnline() earlier

* Removed mergeReplicationId

* Removed mergeReplicationId

* Make active client balancing a configurable option

* Make active client balancing a configurable option

* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior

* With TLS throttle accepts if server is under heavy load - do not change non TLS behavior

* acceptTLS is threadsafe like the non TLS version

* acceptTLS is threadsafe like the non TLS version

* PSYNC production fixes

* PSYNC production fixes

* Ensure we are responsive during storagecache clears

* Ensure we are responsive during storagecache clears

* Ensure recreated tables use the same settings as ones made at boot

* Ensure recreated tables use the same settings as ones made at boot

* Converted some existing PSYNC tests for multimaster

* Converted some existing PSYNC tests for multimaster

* Inclusive language fix

* Inclusive language fix

* Cleanup test suite

* Cleanup test suite

* Updated test replica configs so tests make sense

* Updated test replica configs so tests make sense

* active-rep test reliability

* active-rep test reliability

* Quick fix to make psync tests work

* Quick fix to make psync tests work

* Fix PSYNC test crashes

* Fix PSYNC test crashes

* Ensure we force moves not copies when ingesting bulk insert files

* Ensure we force moves not copies when ingesting bulk insert files

* Disable async for hget commands as it is not ready

* Disable FLASH

* Fix crash in save of masterinfo

* Fix musl/Alpine build failures

* Remove unnecessary libs

* update readme

* update readme

* remove Enterprise references

* Limit max overage to 20% during RDB save

* Delete COPYING to replace with BSD license

* update deb master changelog

* Update license

* Fix Readme typo from github org transition

Replace mention of scratch-file-path with db-s3-object

* Fix reference counting failure in the dict.  This is caused by std::swap also swapping refcounts

* Fix assertion in async rehash

* Prevent crash on shutdown by avoiding dtors (they are unnecessary anyways)

* Initialize noshrink, it was dangling

* Prevent us from starting a rehash when one wasn't already in progress.  This can cause severe issues for snapshots

* Avoid unnecessary rehashing when a rehash is abandoned

* Dictionary use correct acquire/release semantics

* Add fence barriers for the repl backlog (important for AARCH64 and other weak memory models)

* Silence TSAN errors on ustime and mstime.  Every CPU we support is atomic on aligned ints, but correctness matters

* Disable async commands by default

* Fix TSAN warnings on the repl backlog

* Merge OSS back into pro

* Fix unmerged files

* Fix O(n^2) algorithm in the GC cleanup logic

* Fix crash in expire when a snapshot is in flight.  Caused by a perf optimization getting the expire map out of sync with the val

* On Alpine we must have a reasonable stack size

* Revert ci.yml to unstable branch version

* Implements the soft shutdown feature to allow clients to cooperatively disconnect preventing disruption during shutdown

* Ensure clean shutdown with multiple threads

* update dockerfiles

* update deb pkg references and changelog

* update gem reference

* lpGetInteger returns int64_t, avoid overflow (#10068)

Fix #9410

Crucial for the ms and sequence deltas, but I changed all
calls, just in case (e.g. "flags")

Before this commit:
`ms_delta` and `seq_delta` could have overflown, causing `currid` to be wrong,
which in turn would cause `streamTrim` to trim the entire rax node (see new test)

* Fix issue #454 (BSD build break)

* Do not allow commands to run in background when in eval, Issue #452

* Fix certificate leak during connection when tls-allowlists are used

* Fix issue #480

* Fix crash running INFO command while a disk based backlog is set

* check tracking per db

* fix warnings

* Fix a race when undoConnectWithMaster changes mi->repl_transfer_s but the connection is not yet closed and the event handler runs

* Fix a race in processChanges/trackChanges with rdbLoadRio by acquiring the lock when trackChanges is set

* Fix ASAN use after free

* Additional fixes

* Fix integer overflow of the track changes counter

* Fix P99 latency issue for TLS where we leave work for the next event loop

tlsProcessPendingData() needs to be called before we execute queued commands because it may enqueue more commands

* Fix race removing key cache

* Prevent crash on load in long running KeyDB instances

* Fixes a crash where the server assertion failed when the key exists in DB during RDB load

* Remove old assertion which is commented out.

* avoid from instatiating EpochHolder multiple times to improve performance and cpu utilization

* avoid from instatiating EpochHolder multiple times to improve performance and cpu utilization

* src\redis-cli.c: fix potential null pointer dereference found by cppcheck

src\redis-cli.c:5488:35: warning: Either the condition
'!table' is redundant or there is possible null pointer dereference:
table. [nullPointerRedundantCheck]

* Fix Issue #486

* Workaround bug in snapshot sync - abort don't crash

* Improve reliability of async parts of the soft shutdown tests

* Improve reliability of fragmentation tests

* Verify that partial syncs do indeed occur

* Fix O(n) algorithm in INFO command

* Remove incorrect assert that fires when the repl backlog is used fully

* Make building flash optional

* Remove unneeded gitlab CI file

* [BUG] Moves key to another DB, the source key was removed if the move failed due to the key exists in the destination db #497 (#498)

Co-authored-by: Paul Chen <mingchen@Mings-MacBook-Pro.local>

* trigger repl_curr_off!= master_repl_offset assert failure when having pending write case

* use debug for logging the message instead

* rocksdb log using up the diskspace on flash (#519)

* Fix OpenSSL 3.0.x related issues. (#10291)

* Drop obsolete initialization calls.
* Use decoder API for DH parameters.
* Enable auto DH parameters if not explicitly used, which should be the
  preferred configuration going forward.

* remove unnecessary forward declaration

* remove internal ci stuff

* remove more internal ci/publishing

* submodule update step

* use with syntax instead

* bump ci ubuntu old ver as latest is now 22.04

* include submodules on all ci jobs

* install all deps for all ci jobs

Co-authored-by: Vivek Saini <vsaini@snapchat.com>
Co-authored-by: Christian Legge <christian@eqalpha.com>
Co-authored-by: benschermel <bschermel@snapchat.com>
Co-authored-by: John Sully <john@csquare.ca>
Co-authored-by: zliang <zliang@snapchat.com>
Co-authored-by: malavan <malavan@eqalpha.com>
Co-authored-by: John Sully <jsully@snapchat.com>
Co-authored-by: jfinity <38383673+jfinity@users.noreply.github.com>
Co-authored-by: benschermel <43507366+benschermel@users.noreply.github.com>
Co-authored-by: guybe7 <guy.benoish@redislabs.com>
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
Co-authored-by: root <paul.chen1@huawei.com>
Co-authored-by: Ilya Shipitsin <chipitsine@gmail.com>
Co-authored-by: Paul Chen <32553156+paulmchen@users.noreply.github.com>
Co-authored-by: Paul Chen <mingchen@Mings-MacBook-Pro.local>
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2022-12-14 12:17:36 -05:00
John Sully
807c443199 Merge branch 'keydbpro' of https://github.sc-corp.net/Snapchat/keydb-internal into keydbpro 2022-04-27 17:00:35 +00:00
John Sully
63e78ab7f3 Make active client balancing a configurable option 2022-04-18 23:07:15 +00:00
benschermel
9031ac02f9 update packaging for OS merge 2022-04-15 23:09:44 -04:00
John Sully
8a3a4b6e61 Support dynamic runtime modification of tls-allowlist, and add config rewrite support 2022-03-31 17:48:37 -04:00
Vivek Saini
ab9c21f315 Changed term whitelist to allowlist (#54) 2022-03-31 14:15:03 -04:00
Vivek Saini
8644595efc TLS Name Validation (#48)
Added TLS Name Validation
2022-03-03 13:59:07 -05:00
Vivek Saini
6c14fd3f3c Updated default config file with TLS rotation config option 2022-02-02 15:49:52 -05:00
Malavan Sotheeswaran
250e5b39a7 Merge branch 'keydbpro' into async_commands
Former-commit-id: 9eaddb8ca1424ff3225dac9c144d23848228c7d2
2021-11-30 11:47:51 -05:00
John Sully
bdff0121eb Make the replica weighting configurable
Former-commit-id: be6a8a7e68acb5cfbe950f13b903e6f7b98c5a39
2021-10-29 17:59:46 +00:00
malavan
67ffa578d7 Merge remote-tracking branch 'opensource/unstable' into keydbpro
Former-commit-id: cf40fa5da29ba9574d20d706d0a41ca7001df95a
2021-08-11 20:41:54 +00:00
malavan
6879564b48 update keydb.conf with new configs from redis unstable
Former-commit-id: 32ca2e8a7335ee7bc4889a9369c97177f34739ee
2021-08-10 21:34:52 +00:00
christianEQ
d10336b007 various branding and cleanup fixes
Former-commit-id: e3c619eca4755c96af83e1959a6ea5ba95734e93
2021-07-08 02:46:42 +00:00
benschermel
88e6c73906 update enterprise wording for keydb.config lk
Former-commit-id: feb2d31e6544763c94f4abaea69bc98bd1cb8b17
2021-03-21 05:19:28 +00:00
benschermel
05b2d1e798 update all packaging to support enterprise dist
Former-commit-id: 1fd4f27cd271090fc66dba3b09a3800bf1c33b0e
2021-03-17 02:30:34 +00:00
christianEQ
cca8d186aa Merge remote-tracking branch 'opensource/unstable' into keydbpro
Former-commit-id: 5bad058733de2c217340bb9ee48f02b07d754808
2021-02-03 18:10:27 +00:00
christianEQ
396f8316e8 fixed server.cpp mentions
Former-commit-id: 7ed2296fc5d10a1c41020558b4bc445953531bdd
2021-01-26 22:22:19 +00:00
christianEQ
e73af5617c added note about io-threads to keydb.conf
Former-commit-id: 0691a46669d6fbd7dd4a1962c7b1c1929c089b11
2021-01-26 22:10:53 +00:00
christianEQ
48fdfe7d8f Removed Redis-specific things from conf file
Former-commit-id: 4d71929c6000b6a5ceb4793f9b3ee8c7576f4b71
2021-01-26 21:50:45 +00:00
christianEQ
358debebfa Merge tag 'tags/6.0.10' into redismerge_2021-01-20
Former-commit-id: dadce055f897cee83946c2d3e5cbb76341b94230
2021-01-26 21:43:09 +00:00
christianEQ
b113977e94 removed 'comment arguments' from config options (doesn't ignore inline comments)
Former-commit-id: 7cd4b5e11c9c92e88cb551f7d83efc668aa259c7
2021-01-14 13:06:57 -05:00
christianEQ
e68cee23a1 added section to keydb.conf for new options, tweaked default
Former-commit-id: 9b81f3f263de8cffab20ef4d20c2ac4935f63a10
2021-01-14 13:06:57 -05:00
John Sully
706ba4e1dc Merge branch 'unstable' into keydbpro
Former-commit-id: b7a1e16c0f04e8aeb3764e3681fa14fc0f97f6a3
2020-12-10 02:37:28 +00:00
Kajaruban Surendran
fd03008bbf Configurable option for MOTD
Former-commit-id: 49a89d636ba698dbd0858d5059d3d6387c8c1fc7
2020-11-30 10:42:48 -05:00
John Sully
4f18a247e3 Merge tag '6.0.8' into unstable
Former-commit-id: 4c7e4b91a6bb2034636856b608b8c386d07f5541
2020-09-30 19:47:55 +00:00
John Sully
ad4c2390a8 Merge branch 'unstable' into keydbpro
Former-commit-id: d1986d9fcdfd56f2c30ee95edef6d5260e0aa777
2020-09-24 22:02:30 +00:00
John Sully
2c6fbac6ea Allow reads during an active replica load, optionally allow writes as an experimental feature
Former-commit-id: 655bb6e21c5e814980a7b8b3ac1481ef142c1845
2020-09-17 00:13:03 +00:00
John Sully
4eecb1825f Implement replica-quorum config
Former-commit-id: ab4cdd2ed4d3ee9752737d20662178d73d16b4c2
2020-09-16 03:26:31 +00:00
Yossi Gottlieb
bf6c002005 TLS: Session caching configuration support. (#7420)
* TLS: Session caching configuration support.
* TLS: Remove redundant config initialization.


Former-commit-id: d3834c50699bc4f31f381d6d03d4c1b022380895
2020-09-16 02:10:29 +00:00
John Sully
1ad2d96697 Merge branch 'unstable' into keydbpro
Former-commit-id: 0dafbc254a0efd5ee302d5c58fb2ca0a85110104
2020-07-13 03:31:47 +00:00
John Sully
c5f6cb1ba5 Add multi-master-no-forward command to reduce bus traffic with multi-master
Former-commit-id: d99d06b1250a51ea4bc54f678f451acbb7901e33
2020-07-12 19:25:19 +00:00
John Sully
654e72b88d Document min-clients-per-thread
Former-commit-id: 1b9ade7f657d9b142f78a46565e65b52904dad47
2020-07-12 18:49:02 +00:00
John Sully
44c1e6d5e7 Merge branch 'unstable' into keydbpro
Former-commit-id: 08a36155e3db9918048e87c3d691b7317787c9ab
2020-06-01 17:41:37 -04:00
John Sully
cfe9f8f3bc Merge tag '6.0.4' into unstable
Redis 6.0.4.


Former-commit-id: 9c31ac7925edba187e527f506e5e992946bd38a6
2020-05-29 00:57:07 -04:00
John Sully
d50c0c5ba3 Merge branch 'unstable' into keydbpro
Former-commit-id: b0636273806ce323627ce31c1ad7f86ceb39696d
2020-05-26 01:34:19 -04:00
Maxime de Roucy
b9b4b4cfde acl: @replication command group
containing mandatory commands for replication (used by masteruser user)


Former-commit-id: 123bc125cfe7b30fdeef833759a33ccc854b5c51
2020-05-25 02:35:22 -04:00
John Sully
cece963cf3 Merge branch 'unstable' into keydbpro
Former-commit-id: a830cf85df236885558c5571c0bf23cfb23e3655
2020-05-24 14:41:53 -04:00
John Sully
2d783a3cbf Merge tag '6.0.2' into unstable
Redis 6.0.2


Former-commit-id: a010e4a4b2cc2bcad1cb14604b7ebc596c35b05e
2020-05-22 16:45:18 -04:00
John Sully
67b7d512cb Merge commit 'cb683a84f7a9d530bec629f2b656e7d0842a0f75' into unstable
Former-commit-id: 817a11ff110772893eda0675912bbb0cfc1fca74
2020-05-22 15:56:35 -04:00
John Sully
27eb239f1a Fix bad merge in CI.yml
Former-commit-id: 6311d709c39b3bacaeab77b18033010f1b548f81
2020-05-21 22:09:06 -04:00
John Sully
3384af024d Merge commit 'cbcd07777dc569618a34f59e5fd0de53178f4f1d' into unstable
Former-commit-id: 49784c8122e827bd9de86bbae4d88c313400e86e
2020-05-21 18:48:49 -04:00
John Sully
05cc1fd3de Initial merge of unstable 6
Former-commit-id: aac140de199646914cc02997a45111c9c695e55d
2020-04-16 16:36:16 -04:00
John Sully
02147b4be0 Merge commit '0f7dfc378ce5ab0524ac28b6142545bc41435386' into redis_6_merge
Former-commit-id: cc3ebbe5194e9744fb84ce490e90ac5fbe7f8716
2020-04-14 22:19:29 -04:00
John Sully
aa48225a76 Merge commit '76d57161d960f27ac92765118e3b80fef8847b8f' into redis_6_merge
Former-commit-id: 320bc3c0329ff9e5a980b79426b719addae381cf
2020-04-14 21:04:42 -04:00