20964 Commits

Author SHA1 Message Date
Karthick Ariyaratnam
57f4a2e8b1
Fix a bug where flash CF options are being reset to default after flushall. (#718)
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
Co-authored-by: Malavan Sotheeswaran <105669860+msotheeswaran-sc@users.noreply.github.com>
2023-09-28 21:00:28 -04:00
Malavan Sotheeswaran
38169682f0
Merge latest internal to OSS (#720)
* add docker build

* fix the working dir in Dockerfile

* add release publish docker image

* address intentation and use default release

* migrate keydb_modstatsd to keydb-internal

* rm

* add submodule cpp-statsd-client

* include trigger keydb_modstatsd Makefile in modules Makefile

* update

* have clean also trigger keydb_modstatsd clean

* move cpp-statsd-client to deps

* checkout to a06a5b9359f31d946fe163b9038586982971ae49

* update relative path in compilation

* remove submodule instead use the source

* include building keydb statsd module

* fix check in Dockerfile docker-entrypoint.sh

* fix

* fix the comment caused stuck docker build

* use commit hash as tag template

* fix

* test tag

* Revert "test tag"

This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b.

* make docker build independent

* add new build to ci

* emit system free metrics with '/proc/meminfo'

* have emit system free memory within metrics_time_taken_us and also add metric time taken for it

* Remove Expireset (#217)

Major refactor to place expiry information directly in the object struct.

* update MemFree to MemAvailable in keydb statsd

* add metric emit for non-empty primary with less than 2 connected replicas

* address comments

* Multiply CPU percent metric by 100

* Fix memory leaks

* Fix slow to free when low lock contention

* fix nodename metricsname

* fix unnecessary replace

* Make propagating before freeing module context optional (#225)

* don't propogate on module context free for rdb load

* default in wrong place

* Flash expiration (#197)

Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing

* Emit more aggregate metrics in modstatsd (#223)

* Permit keys of differing slots as long as they are served by this cluster and we are not migrating

* Fix over pessimistic checks that prevent replicas from serving mget

* Fix logic bug

* async rehash is preventing rehashing during RDB load after a db flush.  Ensure it can't interefere after a flush

* make async rehash configurable

* only use internal locks when multithreaded (#205)

* Fix crossslot error migrating batches of keys

* Fix bug where we erroneously answer queries belonging to another shard

* fix mac compile

* enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229)

* enable level_compaction_dynamic_level_bytes after flush, and flush expires

* update debug reload for flash

* update debug reload for flash complete

* missing forward declare

* commit existing changes then track changes for debug reload

* missing args

* commitChanges is conditional

Co-authored-by: John Sully <jsully@snapchat.com>

---------

Co-authored-by: zliang <zliang@snapchat.com>
Co-authored-by: John Sully <jsully@snapchat.com>
Co-authored-by: Alex Cope <acope@snapchat.com>
Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
a00817524
5f5753ae53 build failure 2023-09-17 19:06:00 -04:00
a00817524
e42d83d208 build failure 2023-09-17 19:06:00 -04:00
a00817524
9c32d4c1c8 removed proc call 2023-09-17 19:06:00 -04:00
a00817524
337eb297a1 review rework 2023-09-17 19:06:00 -04:00
a00817524
cf13e7b594 Async FLASH Design: AE event loop changes to understand StorageToken 2023-09-17 19:06:00 -04:00
John Sully
aef01878a5 Build break fix 2023-07-25 21:32:52 -07:00
John Sully
bd6eb379d5 Prevent crash on free when using repl-disk-buffer-reserve (#207) 2023-07-25 21:32:52 -07:00
John Sully
06a49a6179 Fix OOM issues during full sync when fork bgsave is used (#209) 2023-07-25 21:32:52 -07:00
John Sully
9ab1278cf1 Prevent a deadlock when running CLIENT KILL with large numbers of clients (#206) 2023-07-25 21:32:52 -07:00
Alex Cope
cc8ed88577 nullptr 2023-07-25 21:32:52 -07:00
Alex Cope
0d2f98d6e1 fix compile error 2023-07-25 21:32:52 -07:00
Alex Cope
872c60566f use enum class 2023-07-25 21:32:52 -07:00
Alex Cope
b5a20a4f91 modify 2023-07-25 21:32:52 -07:00
Alex Cope
cf4735b61e ok 2023-07-25 21:32:52 -07:00
Alex Cope
b2ecf2c856 fix 2023-07-25 21:32:52 -07:00
Alex Cope
e80aba694c fix 2023-07-25 21:32:52 -07:00
Alex Cope
f774a17d8a fix 2023-07-25 21:32:52 -07:00
Alex Cope
329bf17e97 fix 2023-07-25 21:32:52 -07:00
Alex Cope
3792ffb580 fix 2023-07-25 21:32:52 -07:00
Alex Cope
271c2cb92c meminfo 2023-07-25 21:32:52 -07:00
Alex Cope
4c0cf9e5cd fix 2023-07-25 21:32:52 -07:00
Alex Cope
e3cdc765a1 fix 2023-07-25 21:32:52 -07:00
Alex Cope
60d74262dc fixes 2023-07-25 21:32:52 -07:00
Alex Cope
65b7fafd29 llu 2023-07-25 21:32:52 -07:00
Alex Cope
ac7e6c0691 fix compiler errors 2023-07-25 21:32:52 -07:00
Alex Cope
87012bf98e update sys free 2023-07-25 21:32:52 -07:00
Alex Cope
1dc5ea40f3 update 2023-07-25 21:32:51 -07:00
Alex Cope
c9d63c50c0 update 2023-07-25 21:32:51 -07:00
Alex Cope
39e3141a0a revert 2023-07-25 21:32:51 -07:00
Alex Cope
1f2c179124 with config 2023-07-25 21:32:24 -07:00
Alex Cope
4c3d9341fd port changes to 6.3 2023-07-25 21:31:42 -07:00
Karthick Ariyaratnam (A)
0d01b810c3 Fix a fug where repl-backlog-size config was modifed in keydb.conf with the runtime value during config rewrite. 2023-07-14 13:12:16 -04:00
Karthick Ariyaratnam
501ae473e1
Fix a bug where a temp rdb file with zero bytes is generated in flash mode. (#696)
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
2023-07-13 13:26:00 -04:00
Ikko Eltociear Ashimine
6f77a82480
Fix typo in module.cpp (#686)
comparision -> comparison
2023-07-05 18:38:39 -04:00
Malavan Sotheeswaran
c17b9f47ac Cherry picking keydb changes from keydbpro to main (#203)
* Audit Logging for KeyProxy and KeyDB (#144)

* Audit Log: log cert fingerprint (#151)

* Add more flash storage stats to info command.

* Remove unneeded libs when not building FLASH

* Fix mem leak

* Allow the reservation of localhost connections to ensure health checks always succeed even at maxclients (#181)

* Enable a force option for commands (#183)

* Fix missing newline and excessive logging in the CLI

* Support NO ONE for "CLUSTER REPLICATE" command.

Co-authored-by: Jacob Bohac <jbohac@snapchat.com>
Co-authored-by: Sergey Kolosov <skolosov@snapchat.com>
Co-authored-by: John Sully <jsully@snapchat.com>
Co-authored-by: John Sully <john@csquare.ca>
2023-06-27 16:23:20 -04:00
John Sully
f53e0337ef Port load shedding and availability-zone (#202) 2023-06-27 15:37:28 -04:00
Malavan Sotheeswaran
a77548e44a Make inserts in flight a shared_ptr to avoid double free (#198)
* remove keyproxy test from machamp

* Update build.yaml

* make insertsinflight shared
2023-06-21 16:28:57 -07:00
Malavan Sotheeswaran
dde2c27a7e have FreeMemoryLazyFree increment dict refcount so it doesn't try to use a dict that has already been released (#199) 2023-06-21 16:28:57 -07:00
Malavan Sotheeswaran
b258322f9e Fix propagate in transaction crash (#200)
* ignore propagate_in_transaction in processCommand if command is async

* make propogate_in_transaction thread local

* forgot module.cpp

* move client_pause_in_transaction to thread local

* don't do async commands while in eval or exec

* don't do keys or scan async if in transaction
2023-06-21 16:28:57 -07:00
Karthick Ariyaratnam
46660b9f18
Fix a bug where swapdb result was not recovered after keydb restarts in FLASH mode. (#668)
Co-authored-by: k00809413 <karthick.ariyaratnam1@huawei.com>
2023-06-21 16:26:14 -04:00
Malavan Sotheeswaran
62f3c37b87 Throw load error when replicaof config is before active-replica or multi-master configs (#666)
* load error when replicaof is set before multi-master or active-replica

* only an issue if enabled
2023-06-12 15:06:59 -07:00
Malavan Sotheeswaran
dc419a4e6e
keys after brpop causes crash due to timeout not being reset (#662) 2023-05-30 01:01:51 -04:00
Malavan Sotheeswaran
3336c4f43e
Flash slot to key (#653)
* add hash prefixing to rocksdb

* don't need to offset internal keys

* remove hashslot prefix when loading from rocksdb

* don't prefix internal keys

* implement slot to keys with rocksdb

* fix compile

* add hashslot enum to test storage provider

* add --flash option for tests

* forgot return statement in getKeysInSlot

* check for count mismatch

* forgot ;

* add assert to ensure correctness

* fix warnings

* add option to show logs of 1 server at a time

* add fixed length prefix extractor

* add machamp

* switch machamp to main branch

* add machamp script
2023-05-30 01:01:40 -04:00
Alexandre Chichmanian
16e8469316 Fix aws startup check preventing keydb from starting 2023-05-30 00:34:15 -04:00
Malavan Sotheeswaran
ef8b1bf373 add fPreSnapshot to evictionTimeProc 2023-05-25 00:03:39 -07:00
Attila Nagy
f629a76a2e
Make it compile on FreeBSD (#650) 2023-05-08 15:59:36 -04:00
Malavan Sotheeswaran
e4dacb467f add config to disable storage key cache 2023-04-25 11:15:57 -07:00
Malavan Sotheeswaran
a8c0f9b1c4 limit number of random elements returned based on config, exit loop early if client is being closed 2023-04-18 17:08:27 -07:00