27520 Commits

Author SHA1 Message Date
Guillem Jover
d67c65867a build: Add support for linking against a system librocksdb
Add a new USE_SYSTEM_ROCKSDB make variable to select whether to link
against the system librocksdb.
2024-03-25 21:01:05 -04:00
Guillem Jover
ae130d1a7f build: Add support for linking against a system libjemalloc
Add a new USE_SYSTEM_JEMALLOC make variable to select whether to link
against the system libjemalloc.
2024-03-25 21:01:05 -04:00
Guillem Jover
10c63d743c build: Add support for using the system concurrentqueue headers
Add a new USE_SYSTEM_CONCURRENTQUEUE make variable to select whether
to include the system concurrentqueue headers.
2024-03-25 21:01:05 -04:00
Ikko Eltociear Ashimine
5e643e9a9c Update README.md
intented -> intended
2024-03-25 20:58:26 -04:00
Teemu Kekkonen
2cb5d71073 Update RPM script for Rocky Linux 9
This uses the 'dist' RPM macro instead of relying on the version information of
CentOS packages.  This approach works on CentOS 7 and 8 as well.
2024-03-07 19:54:09 -05:00
Tibault Damman
011fdb4772 fix build on 32bit systems
Code had an assert that tests if sizeof(long) == sizeof(long long),
which obviously fails on 32-bit architectures.
I believe the assert is incorrect on any architecture, considering
the following code is actually putting a long long in an _int_.

I replaced it with code that checks if the value fits in an int.

Signed-off-by: Tibault Damman <tibault.damman@basalte.be>
2024-03-07 19:50:35 -05:00
Roman Donchenko
d94fddae0a Add more detail to the panic messages in blocked.cpp 2024-03-07 19:50:00 -05:00
k00809413
2ad14dec88 Fix swapdb bugs when multiple clients involved. 2024-01-10 15:32:51 -05:00
Malavan Sotheeswaran
2abe7086a8
Update README.md (#762) 2023-12-18 15:30:48 -05:00
Malavan Sotheeswaran
674d9fb7ee
stat64 is deprecated on mac (#737) 2023-11-16 03:05:43 -05:00
Malavan Sotheeswaran
de3c218a02 add release notes up to 6.3.4 2023-10-29 17:39:55 -07:00
Malavan Sotheeswaran
05945b60a8 missed a refernce to isRocksdbSnapshotRepl 2023-10-26 11:40:50 -07:00
Malavan Sotheeswaran
486a6bb7b1 Merge remote-tracking branch 'internal/main' 2023-10-26 11:21:46 -07:00
John Sully
70503f6b58 Disable fastsync by default 2023-10-23 15:23:11 -04:00
John Sully
46615ac099 Fix test 2023-10-23 15:23:11 -04:00
John Sully
a7ac6c1e43 Remove unused variable 2023-10-23 15:23:11 -04:00
John Sully
d7977c468f Fastsync in-mem initial implementation 2023-10-23 15:23:11 -04:00
John Sully
7ae27f61f0 rename constants 2023-10-23 15:23:11 -04:00
Malavan Sotheeswaran
47f635ee16 Fix tls auditlog tests and enable tls for CI (#234)
* fix auditlog test titles, and enable tls tests for CI

* fix tls auditlog key string
2023-10-23 14:34:23 -04:00
Malavan Sotheeswaran
79b0c84710
Fixing some compile issues for alpine and centos, also fix endian issue for FLASH hashslot prefix (#722)
*fix for rocksdb centos compile fail

* fix alpine compile

* fix hashslot enumerate

* update alpine docker to latest version

* fix mac build, use big endian for expire prefix

* use endian.h for expires too
2023-10-13 15:00:35 -04:00
John Sully
cd42776c9f Make the connect timeout configurable when in cluster mode to prevent huge delays 2023-10-12 12:11:18 -04:00
Einar Fløystad Dørum
7049b5f4e8
Add KEYDB_PASSWORD env variable to Dockerfiles (#220) (#704) 2023-09-28 21:03:45 -04:00
Karthick Ariyaratnam
5b12251627
Fix integer overflow issue in the temp rdb file naming. (#702)
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
2023-09-28 21:01:47 -04:00
michieldwitte
a0f280e1a2
rename zset to iterzset (#706)
Co-authored-by: Michiel De Witte <michiel.dewitte@basalte.be>
2023-09-28 21:01:05 -04:00
Karthick Ariyaratnam
57f4a2e8b1
Fix a bug where flash CF options are being reset to default after flushall. (#718)
Co-authored-by: Karthick Ariyaratnam (A) <k00809413@china.huawei.com>
Co-authored-by: Malavan Sotheeswaran <105669860+msotheeswaran-sc@users.noreply.github.com>
2023-09-28 21:00:28 -04:00
Malavan Sotheeswaran
38169682f0
Merge latest internal to OSS (#720)
* add docker build

* fix the working dir in Dockerfile

* add release publish docker image

* address intentation and use default release

* migrate keydb_modstatsd to keydb-internal

* rm

* add submodule cpp-statsd-client

* include trigger keydb_modstatsd Makefile in modules Makefile

* update

* have clean also trigger keydb_modstatsd clean

* move cpp-statsd-client to deps

* checkout to a06a5b9359f31d946fe163b9038586982971ae49

* update relative path in compilation

* remove submodule instead use the source

* include building keydb statsd module

* fix check in Dockerfile docker-entrypoint.sh

* fix

* fix the comment caused stuck docker build

* use commit hash as tag template

* fix

* test tag

* Revert "test tag"

This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b.

* make docker build independent

* add new build to ci

* emit system free metrics with '/proc/meminfo'

* have emit system free memory within metrics_time_taken_us and also add metric time taken for it

* Remove Expireset (#217)

Major refactor to place expiry information directly in the object struct.

* update MemFree to MemAvailable in keydb statsd

* add metric emit for non-empty primary with less than 2 connected replicas

* address comments

* Multiply CPU percent metric by 100

* Fix memory leaks

* Fix slow to free when low lock contention

* fix nodename metricsname

* fix unnecessary replace

* Make propagating before freeing module context optional (#225)

* don't propogate on module context free for rdb load

* default in wrong place

* Flash expiration (#197)

Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing

* Emit more aggregate metrics in modstatsd (#223)

* Permit keys of differing slots as long as they are served by this cluster and we are not migrating

* Fix over pessimistic checks that prevent replicas from serving mget

* Fix logic bug

* async rehash is preventing rehashing during RDB load after a db flush.  Ensure it can't interefere after a flush

* make async rehash configurable

* only use internal locks when multithreaded (#205)

* Fix crossslot error migrating batches of keys

* Fix bug where we erroneously answer queries belonging to another shard

* fix mac compile

* enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229)

* enable level_compaction_dynamic_level_bytes after flush, and flush expires

* update debug reload for flash

* update debug reload for flash complete

* missing forward declare

* commit existing changes then track changes for debug reload

* missing args

* commitChanges is conditional

Co-authored-by: John Sully <jsully@snapchat.com>

---------

Co-authored-by: zliang <zliang@snapchat.com>
Co-authored-by: John Sully <jsully@snapchat.com>
Co-authored-by: Alex Cope <acope@snapchat.com>
Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
Malavan Sotheeswaran
dfcc848e1c enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229)
* enable level_compaction_dynamic_level_bytes after flush, and flush expires

* update debug reload for flash

* update debug reload for flash complete

* missing forward declare

* commit existing changes then track changes for debug reload

* missing args

* commitChanges is conditional

Co-authored-by: John Sully <jsully@snapchat.com>
2023-09-28 16:59:23 -04:00
John Sully
e7f90f0096 Fix bug where we erroneously answer queries belonging to another shard 2023-09-27 07:38:09 +00:00
John Sully
025b7077f7 Fix crossslot error migrating batches of keys 2023-09-26 00:11:21 -04:00
Malavan Sotheeswaran
596c513d3e only use internal locks when multithreaded (#205) 2023-09-25 15:53:47 -04:00
John Sully
570bcb55c7 make async rehash configurable 2023-09-23 16:32:28 -04:00
John Sully
68d751046e async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush 2023-09-23 16:32:28 -04:00
John Sully
cfe257a9c7 Fix logic bug 2023-09-20 12:57:10 -07:00
John Sully
c6e20ea396 Fix over pessimistic checks that prevent replicas from serving mget 2023-09-20 12:57:10 -07:00
a00817524
5f5753ae53 build failure 2023-09-17 19:06:00 -04:00
a00817524
e42d83d208 build failure 2023-09-17 19:06:00 -04:00
a00817524
9c32d4c1c8 removed proc call 2023-09-17 19:06:00 -04:00
a00817524
337eb297a1 review rework 2023-09-17 19:06:00 -04:00
a00817524
cf13e7b594 Async FLASH Design: AE event loop changes to understand StorageToken 2023-09-17 19:06:00 -04:00
John Sully
9251f1c244 Permit keys of differing slots as long as they are served by this cluster and we are not migrating 2023-09-14 21:55:04 -04:00
Zhaozhen Liang
7b7c0ba192 Merge pull request #224 from Snapchat/remove-dot-in-nodename-metrics
fix nodename metricsname
2023-09-11 15:37:16 -06:00
zliang
98e306c59b Merge branch 'main' into remove-dot-in-nodename-metrics 2023-09-11 12:18:39 -06:00
Alex Cope
a55c687cd0 Emit more aggregate metrics in modstatsd (#223) 2023-09-11 10:43:25 -07:00
Malavan Sotheeswaran
ace783a00e Flash expiration (#197)
Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing
2023-09-08 16:25:53 -04:00
Malavan Sotheeswaran
7d4f461562 Make propagating before freeing module context optional (#225)
* don't propogate on module context free for rdb load

* default in wrong place
2023-09-08 15:51:48 -04:00
zliang
e31533bd7a fix unnecessary replace 2023-08-31 12:45:42 -06:00
zliang
c20af57cf1 fix nodename metricsname 2023-08-31 12:29:59 -06:00
John Sully
820482bffe Fix slow to free when low lock contention 2023-08-25 20:40:16 -04:00
John Sully
f435218ed9 Fix memory leaks 2023-08-25 16:44:10 +00:00
Alex Cope
8b755aee1b Multiply CPU percent metric by 100 2023-08-22 17:33:57 -04:00