27610 Commits

Author SHA1 Message Date
Malavan Sotheeswaran
de3c218a02 add release notes up to 6.3.4 2023-10-29 17:39:55 -07:00
Malavan Sotheeswaran
05945b60a8 missed a refernce to isRocksdbSnapshotRepl 2023-10-26 11:40:50 -07:00
Malavan Sotheeswaran
486a6bb7b1 Merge remote-tracking branch 'internal/main' 2023-10-26 11:21:46 -07:00
John Sully
70503f6b58 Disable fastsync by default 2023-10-23 15:23:11 -04:00
John Sully
46615ac099 Fix test 2023-10-23 15:23:11 -04:00
John Sully
a7ac6c1e43 Remove unused variable 2023-10-23 15:23:11 -04:00
John Sully
d7977c468f Fastsync in-mem initial implementation 2023-10-23 15:23:11 -04:00
John Sully
7ae27f61f0 rename constants 2023-10-23 15:23:11 -04:00
Malavan Sotheeswaran
47f635ee16 Fix tls auditlog tests and enable tls for CI (#234)
* fix auditlog test titles, and enable tls tests for CI

* fix tls auditlog key string
2023-10-23 14:34:23 -04:00
Malavan Sotheeswaran
79b0c84710
Fixing some compile issues for alpine and centos, also fix endian issue for FLASH hashslot prefix (#722)
*fix for rocksdb centos compile fail

* fix alpine compile

* fix hashslot enumerate

* update alpine docker to latest version

* fix mac build, use big endian for expire prefix

* use endian.h for expires too
2023-10-13 15:00:35 -04:00
John Sully
cd42776c9f Make the connect timeout configurable when in cluster mode to prevent huge delays 2023-10-12 12:11:18 -04:00
Einar Fløystad Dørum
7049b5f4e8
Add KEYDB_PASSWORD env variable to Dockerfiles (#220) (#704) 2023-09-28 21:03:45 -04:00
Karthick Ariyaratnam
5b12251627
Fix integer overflow issue in the temp rdb file naming. (#702)
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
2023-09-28 21:01:47 -04:00
michieldwitte
a0f280e1a2
rename zset to iterzset (#706)
Co-authored-by: Michiel De Witte <michiel.dewitte@basalte.be>
2023-09-28 21:01:05 -04:00
Karthick Ariyaratnam
57f4a2e8b1
Fix a bug where flash CF options are being reset to default after flushall. (#718)
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
Co-authored-by: Malavan Sotheeswaran <105669860+msotheeswaran-sc@users.noreply.github.com>
2023-09-28 21:00:28 -04:00
Malavan Sotheeswaran
38169682f0
Merge latest internal to OSS (#720)
* add docker build

* fix the working dir in Dockerfile

* add release publish docker image

* address intentation and use default release

* migrate keydb_modstatsd to keydb-internal

* rm

* add submodule cpp-statsd-client

* include trigger keydb_modstatsd Makefile in modules Makefile

* update

* have clean also trigger keydb_modstatsd clean

* move cpp-statsd-client to deps

* checkout to a06a5b9359f31d946fe163b9038586982971ae49

* update relative path in compilation

* remove submodule instead use the source

* include building keydb statsd module

* fix check in Dockerfile docker-entrypoint.sh

* fix

* fix the comment caused stuck docker build

* use commit hash as tag template

* fix

* test tag

* Revert "test tag"

This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b.

* make docker build independent

* add new build to ci

* emit system free metrics with '/proc/meminfo'

* have emit system free memory within metrics_time_taken_us and also add metric time taken for it

* Remove Expireset (#217)

Major refactor to place expiry information directly in the object struct.

* update MemFree to MemAvailable in keydb statsd

* add metric emit for non-empty primary with less than 2 connected replicas

* address comments

* Multiply CPU percent metric by 100

* Fix memory leaks

* Fix slow to free when low lock contention

* fix nodename metricsname

* fix unnecessary replace

* Make propagating before freeing module context optional (#225)

* don't propogate on module context free for rdb load

* default in wrong place

* Flash expiration (#197)

Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing

* Emit more aggregate metrics in modstatsd (#223)

* Permit keys of differing slots as long as they are served by this cluster and we are not migrating

* Fix over pessimistic checks that prevent replicas from serving mget

* Fix logic bug

* async rehash is preventing rehashing during RDB load after a db flush.  Ensure it can't interefere after a flush

* make async rehash configurable

* only use internal locks when multithreaded (#205)

* Fix crossslot error migrating batches of keys

* Fix bug where we erroneously answer queries belonging to another shard

* fix mac compile

* enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229)

* enable level_compaction_dynamic_level_bytes after flush, and flush expires

* update debug reload for flash

* update debug reload for flash complete

* missing forward declare

* commit existing changes then track changes for debug reload

* missing args

* commitChanges is conditional

Co-authored-by: John Sully <jsully@snapchat.com>

---------

Co-authored-by: zliang <zliang@snapchat.com>
Co-authored-by: John Sully <jsully@snapchat.com>
Co-authored-by: Alex Cope <acope@snapchat.com>
Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
Malavan Sotheeswaran
dfcc848e1c enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229)
* enable level_compaction_dynamic_level_bytes after flush, and flush expires

* update debug reload for flash

* update debug reload for flash complete

* missing forward declare

* commit existing changes then track changes for debug reload

* missing args

* commitChanges is conditional

Co-authored-by: John Sully <jsully@snapchat.com>
2023-09-28 16:59:23 -04:00
John Sully
e7f90f0096 Fix bug where we erroneously answer queries belonging to another shard 2023-09-27 07:38:09 +00:00
John Sully
025b7077f7 Fix crossslot error migrating batches of keys 2023-09-26 00:11:21 -04:00
Malavan Sotheeswaran
596c513d3e only use internal locks when multithreaded (#205) 2023-09-25 15:53:47 -04:00
John Sully
570bcb55c7 make async rehash configurable 2023-09-23 16:32:28 -04:00
John Sully
68d751046e async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush 2023-09-23 16:32:28 -04:00
John Sully
cfe257a9c7 Fix logic bug 2023-09-20 12:57:10 -07:00
John Sully
c6e20ea396 Fix over pessimistic checks that prevent replicas from serving mget 2023-09-20 12:57:10 -07:00
a00817524
5f5753ae53 build failure 2023-09-17 19:06:00 -04:00
a00817524
e42d83d208 build failure 2023-09-17 19:06:00 -04:00
a00817524
9c32d4c1c8 removed proc call 2023-09-17 19:06:00 -04:00
a00817524
337eb297a1 review rework 2023-09-17 19:06:00 -04:00
a00817524
cf13e7b594 Async FLASH Design: AE event loop changes to understand StorageToken 2023-09-17 19:06:00 -04:00
John Sully
9251f1c244 Permit keys of differing slots as long as they are served by this cluster and we are not migrating 2023-09-14 21:55:04 -04:00
Zhaozhen Liang
7b7c0ba192 Merge pull request #224 from Snapchat/remove-dot-in-nodename-metrics
fix nodename metricsname
2023-09-11 15:37:16 -06:00
zliang
98e306c59b Merge branch 'main' into remove-dot-in-nodename-metrics 2023-09-11 12:18:39 -06:00
Alex Cope
a55c687cd0 Emit more aggregate metrics in modstatsd (#223) 2023-09-11 10:43:25 -07:00
Malavan Sotheeswaran
ace783a00e Flash expiration (#197)
Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing
2023-09-08 16:25:53 -04:00
Malavan Sotheeswaran
7d4f461562 Make propagating before freeing module context optional (#225)
* don't propogate on module context free for rdb load

* default in wrong place
2023-09-08 15:51:48 -04:00
zliang
e31533bd7a fix unnecessary replace 2023-08-31 12:45:42 -06:00
zliang
c20af57cf1 fix nodename metricsname 2023-08-31 12:29:59 -06:00
John Sully
820482bffe Fix slow to free when low lock contention 2023-08-25 20:40:16 -04:00
John Sully
f435218ed9 Fix memory leaks 2023-08-25 16:44:10 +00:00
Alex Cope
8b755aee1b Multiply CPU percent metric by 100 2023-08-22 17:33:57 -04:00
Zhaozhen Liang
c97f696ea2 Merge pull request #219 from Snapchat/emit-non-empty-shard-less-replica-issue
add metric emit for non-empty primary with less than 2 connected replicas
2023-08-22 11:34:09 -06:00
zliang
cec570fdf1 address comments 2023-08-22 11:10:56 -06:00
zliang
6375e919b9 add metric emit for non-empty primary with less than 2 connected replicas 2023-08-21 18:29:05 -06:00
Zhaozhen Liang
bf29926686 Merge pull request #218 from Snapchat/change-memfree-to-memavailable
update MemFree to MemAvailable in keydb statsd
2023-08-21 16:46:38 -06:00
zliang
813bf15ae0 update MemFree to MemAvailable in keydb statsd 2023-08-21 15:28:26 -06:00
John Sully
ffac55273a Remove Expireset (#217)
Major refactor to place expiry information directly in the object struct.
2023-08-21 16:36:45 -04:00
Zhaozhen Liang
80dcbad30f Merge pull request #216 from Snapchat/add-free-memory-and-shard-num-metrics-emit-statsd
emit system free memory metrics with '/proc/meminfo'
2023-08-18 10:50:21 -06:00
zliang
eb14f4298c have emit system free memory within metrics_time_taken_us and also add metric time taken for it 2023-08-18 10:20:00 -06:00
zliang
6fa696bb98 emit system free metrics with '/proc/meminfo' 2023-08-17 19:31:52 -06:00
Zhaozhen Liang
abd6e77c95 Merge pull request #215 from Snapchat/make-docker-build-independent
make docker build independent
2023-08-16 12:24:25 -06:00