27547 Commits

Author SHA1 Message Date
michieldwitte
a0f280e1a2
rename zset to iterzset (#706)
Co-authored-by: Michiel De Witte <michiel.dewitte@basalte.be>
2023-09-28 21:01:05 -04:00
Karthick Ariyaratnam
57f4a2e8b1
Fix a bug where flash CF options are being reset to default after flushall. (#718)
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
Co-authored-by: Malavan Sotheeswaran <105669860+msotheeswaran-sc@users.noreply.github.com>
2023-09-28 21:00:28 -04:00
Malavan Sotheeswaran
38169682f0
Merge latest internal to OSS (#720)
* add docker build

* fix the working dir in Dockerfile

* add release publish docker image

* address intentation and use default release

* migrate keydb_modstatsd to keydb-internal

* rm

* add submodule cpp-statsd-client

* include trigger keydb_modstatsd Makefile in modules Makefile

* update

* have clean also trigger keydb_modstatsd clean

* move cpp-statsd-client to deps

* checkout to a06a5b9359f31d946fe163b9038586982971ae49

* update relative path in compilation

* remove submodule instead use the source

* include building keydb statsd module

* fix check in Dockerfile docker-entrypoint.sh

* fix

* fix the comment caused stuck docker build

* use commit hash as tag template

* fix

* test tag

* Revert "test tag"

This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b.

* make docker build independent

* add new build to ci

* emit system free metrics with '/proc/meminfo'

* have emit system free memory within metrics_time_taken_us and also add metric time taken for it

* Remove Expireset (#217)

Major refactor to place expiry information directly in the object struct.

* update MemFree to MemAvailable in keydb statsd

* add metric emit for non-empty primary with less than 2 connected replicas

* address comments

* Multiply CPU percent metric by 100

* Fix memory leaks

* Fix slow to free when low lock contention

* fix nodename metricsname

* fix unnecessary replace

* Make propagating before freeing module context optional (#225)

* don't propogate on module context free for rdb load

* default in wrong place

* Flash expiration (#197)

Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing

* Emit more aggregate metrics in modstatsd (#223)

* Permit keys of differing slots as long as they are served by this cluster and we are not migrating

* Fix over pessimistic checks that prevent replicas from serving mget

* Fix logic bug

* async rehash is preventing rehashing during RDB load after a db flush.  Ensure it can't interefere after a flush

* make async rehash configurable

* only use internal locks when multithreaded (#205)

* Fix crossslot error migrating batches of keys

* Fix bug where we erroneously answer queries belonging to another shard

* fix mac compile

* enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229)

* enable level_compaction_dynamic_level_bytes after flush, and flush expires

* update debug reload for flash

* update debug reload for flash complete

* missing forward declare

* commit existing changes then track changes for debug reload

* missing args

* commitChanges is conditional

Co-authored-by: John Sully <jsully@snapchat.com>

---------

Co-authored-by: zliang <zliang@snapchat.com>
Co-authored-by: John Sully <jsully@snapchat.com>
Co-authored-by: Alex Cope <acope@snapchat.com>
Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
Malavan Sotheeswaran
dfcc848e1c enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229)
* enable level_compaction_dynamic_level_bytes after flush, and flush expires

* update debug reload for flash

* update debug reload for flash complete

* missing forward declare

* commit existing changes then track changes for debug reload

* missing args

* commitChanges is conditional

Co-authored-by: John Sully <jsully@snapchat.com>
2023-09-28 16:59:23 -04:00
John Sully
e7f90f0096 Fix bug where we erroneously answer queries belonging to another shard 2023-09-27 07:38:09 +00:00
John Sully
025b7077f7 Fix crossslot error migrating batches of keys 2023-09-26 00:11:21 -04:00
Malavan Sotheeswaran
596c513d3e only use internal locks when multithreaded (#205) 2023-09-25 15:53:47 -04:00
John Sully
570bcb55c7 make async rehash configurable 2023-09-23 16:32:28 -04:00
John Sully
68d751046e async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush 2023-09-23 16:32:28 -04:00
John Sully
cfe257a9c7 Fix logic bug 2023-09-20 12:57:10 -07:00
John Sully
c6e20ea396 Fix over pessimistic checks that prevent replicas from serving mget 2023-09-20 12:57:10 -07:00
a00817524
5f5753ae53 build failure 2023-09-17 19:06:00 -04:00
a00817524
e42d83d208 build failure 2023-09-17 19:06:00 -04:00
a00817524
9c32d4c1c8 removed proc call 2023-09-17 19:06:00 -04:00
a00817524
337eb297a1 review rework 2023-09-17 19:06:00 -04:00
a00817524
cf13e7b594 Async FLASH Design: AE event loop changes to understand StorageToken 2023-09-17 19:06:00 -04:00
John Sully
9251f1c244 Permit keys of differing slots as long as they are served by this cluster and we are not migrating 2023-09-14 21:55:04 -04:00
Zhaozhen Liang
7b7c0ba192 Merge pull request #224 from Snapchat/remove-dot-in-nodename-metrics
fix nodename metricsname
2023-09-11 15:37:16 -06:00
zliang
98e306c59b Merge branch 'main' into remove-dot-in-nodename-metrics 2023-09-11 12:18:39 -06:00
Alex Cope
a55c687cd0 Emit more aggregate metrics in modstatsd (#223) 2023-09-11 10:43:25 -07:00
Malavan Sotheeswaran
ace783a00e Flash expiration (#197)
Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing
2023-09-08 16:25:53 -04:00
Malavan Sotheeswaran
7d4f461562 Make propagating before freeing module context optional (#225)
* don't propogate on module context free for rdb load

* default in wrong place
2023-09-08 15:51:48 -04:00
zliang
e31533bd7a fix unnecessary replace 2023-08-31 12:45:42 -06:00
zliang
c20af57cf1 fix nodename metricsname 2023-08-31 12:29:59 -06:00
John Sully
820482bffe Fix slow to free when low lock contention 2023-08-25 20:40:16 -04:00
John Sully
f435218ed9 Fix memory leaks 2023-08-25 16:44:10 +00:00
Alex Cope
8b755aee1b Multiply CPU percent metric by 100 2023-08-22 17:33:57 -04:00
Zhaozhen Liang
c97f696ea2 Merge pull request #219 from Snapchat/emit-non-empty-shard-less-replica-issue
add metric emit for non-empty primary with less than 2 connected replicas
2023-08-22 11:34:09 -06:00
zliang
cec570fdf1 address comments 2023-08-22 11:10:56 -06:00
zliang
6375e919b9 add metric emit for non-empty primary with less than 2 connected replicas 2023-08-21 18:29:05 -06:00
Zhaozhen Liang
bf29926686 Merge pull request #218 from Snapchat/change-memfree-to-memavailable
update MemFree to MemAvailable in keydb statsd
2023-08-21 16:46:38 -06:00
zliang
813bf15ae0 update MemFree to MemAvailable in keydb statsd 2023-08-21 15:28:26 -06:00
John Sully
ffac55273a Remove Expireset (#217)
Major refactor to place expiry information directly in the object struct.
2023-08-21 16:36:45 -04:00
Zhaozhen Liang
80dcbad30f Merge pull request #216 from Snapchat/add-free-memory-and-shard-num-metrics-emit-statsd
emit system free memory metrics with '/proc/meminfo'
2023-08-18 10:50:21 -06:00
zliang
eb14f4298c have emit system free memory within metrics_time_taken_us and also add metric time taken for it 2023-08-18 10:20:00 -06:00
zliang
6fa696bb98 emit system free metrics with '/proc/meminfo' 2023-08-17 19:31:52 -06:00
Zhaozhen Liang
abd6e77c95 Merge pull request #215 from Snapchat/make-docker-build-independent
make docker build independent
2023-08-16 12:24:25 -06:00
zliang
7a00a43170 add new build to ci 2023-08-15 15:52:59 -06:00
zliang
426eabe143 make docker build independent 2023-08-15 15:27:05 -06:00
Zhaozhen Liang
222776e98b Merge pull request #214 from Snapchat/update-docker-build-tag
use commit hash as tag template
2023-08-15 13:13:30 -06:00
zliang
76fec1c22e Revert "test tag"
This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b.
2023-08-15 12:40:10 -06:00
zliang
9cbc57137d test tag 2023-08-15 12:25:15 -06:00
zliang
94b40af1c3 fix 2023-08-15 11:56:03 -06:00
zliang
df9039d83f use commit hash as tag template 2023-08-15 11:37:29 -06:00
Zhaozhen Liang
a1b55eddf8 Merge pull request #211 from Snapchat/machamp-release-publish-image
add docker build
2023-08-15 09:40:43 -06:00
zliang
70f2a74ff1 fix the comment caused stuck docker build 2023-08-14 18:34:05 -06:00
zliang
e15a2b609c fix 2023-08-14 17:05:37 -06:00
zliang
7d96689ae6 fix check in Dockerfile docker-entrypoint.sh 2023-08-14 16:31:41 -06:00
Zhaozhen Liang
8cbb53e7cb Merge branch 'main' into machamp-release-publish-image 2023-08-14 12:55:30 -06:00
zliang
b6f91e4c26 include building keydb statsd module 2023-08-14 12:03:36 -06:00