91 Commits

Author SHA1 Message Date
Malavan Sotheeswaran
38169682f0
Merge latest internal to OSS (#720)
* add docker build

* fix the working dir in Dockerfile

* add release publish docker image

* address intentation and use default release

* migrate keydb_modstatsd to keydb-internal

* rm

* add submodule cpp-statsd-client

* include trigger keydb_modstatsd Makefile in modules Makefile

* update

* have clean also trigger keydb_modstatsd clean

* move cpp-statsd-client to deps

* checkout to a06a5b9359f31d946fe163b9038586982971ae49

* update relative path in compilation

* remove submodule instead use the source

* include building keydb statsd module

* fix check in Dockerfile docker-entrypoint.sh

* fix

* fix the comment caused stuck docker build

* use commit hash as tag template

* fix

* test tag

* Revert "test tag"

This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b.

* make docker build independent

* add new build to ci

* emit system free metrics with '/proc/meminfo'

* have emit system free memory within metrics_time_taken_us and also add metric time taken for it

* Remove Expireset (#217)

Major refactor to place expiry information directly in the object struct.

* update MemFree to MemAvailable in keydb statsd

* add metric emit for non-empty primary with less than 2 connected replicas

* address comments

* Multiply CPU percent metric by 100

* Fix memory leaks

* Fix slow to free when low lock contention

* fix nodename metricsname

* fix unnecessary replace

* Make propagating before freeing module context optional (#225)

* don't propogate on module context free for rdb load

* default in wrong place

* Flash expiration (#197)

Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing

* Emit more aggregate metrics in modstatsd (#223)

* Permit keys of differing slots as long as they are served by this cluster and we are not migrating

* Fix over pessimistic checks that prevent replicas from serving mget

* Fix logic bug

* async rehash is preventing rehashing during RDB load after a db flush.  Ensure it can't interefere after a flush

* make async rehash configurable

* only use internal locks when multithreaded (#205)

* Fix crossslot error migrating batches of keys

* Fix bug where we erroneously answer queries belonging to another shard

* fix mac compile

* enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229)

* enable level_compaction_dynamic_level_bytes after flush, and flush expires

* update debug reload for flash

* update debug reload for flash complete

* missing forward declare

* commit existing changes then track changes for debug reload

* missing args

* commitChanges is conditional

Co-authored-by: John Sully <jsully@snapchat.com>

---------

Co-authored-by: zliang <zliang@snapchat.com>
Co-authored-by: John Sully <jsully@snapchat.com>
Co-authored-by: Alex Cope <acope@snapchat.com>
Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
Ikko Eltociear Ashimine
870b0db23d Fix typo in module.cpp (#686)
comparision -> comparison
2023-07-25 21:33:30 -07:00
Malavan Sotheeswaran
b258322f9e Fix propagate in transaction crash (#200)
* ignore propagate_in_transaction in processCommand if command is async

* make propogate_in_transaction thread local

* forgot module.cpp

* move client_pause_in_transaction to thread local

* don't do async commands while in eval or exec

* don't do keys or scan async if in transaction
2023-06-21 16:28:57 -07:00
malavan
0ee3a5ace1 fork lock for all threads, use fastlock for readwritelock 2022-04-13 13:52:41 -04:00
John Sully
d77bbee238 Fix mac build breaks and remove license checks (won't work on mac) 2022-03-07 14:50:31 -05:00
christianEQ
b785b03068 Merge remote-tracking branch 'opensource/RELEASE_6' into keydbpro
Former-commit-id: 439c655a543f4d3224d90bcdeb21ba43c2ab8ab7
2022-01-14 22:19:42 +00:00
MalavanEQAlpha
0f4268579e Merge pull request #386 from EQ-Alpha/fix_rdb_hang
add readwrite lock for forking

Former-commit-id: dabb81960f6ccc7f62c53648127924fc2fef7cdc
2022-01-14 13:19:33 -08:00
John Sully
968d81dcf9 Fix module test break
Former-commit-id: 50dfee2039e199f60f927790e7ac81625402b191
2021-12-23 13:12:34 -05:00
John Sully
129f27f751 Fix module test failures with blocked clients
Former-commit-id: 18ec4b8600ea3462843b58a0f0e917824dfd7ff3
2021-12-23 02:10:20 -05:00
VivekSainiEQ
bfcea943ea Merge remote-tracking branch 'mainpro/PRO_RELEASE_6' into keydbpro
Former-commit-id: 5a32d66ee382b6d227a67073afc81ca058d605ed
2021-12-06 20:43:23 +00:00
VivekSainiEQ
f9a355900a Merge remote-tracking branch 'opensource/unstable' into PRO_RELEASE_6
Former-commit-id: 60da86471f68b13e2456e113ecd4aa117d95b134
2021-11-12 21:12:53 +00:00
VivekSainiEQ
93b0482376 Resolved merge conflicts in prior commit
Former-commit-id: b88f06b16f3d9e58ec884c61d2d074d7a489775e
2021-10-21 22:35:15 +00:00
VivekSainiEQ
1d882b5ddd Merge tag '6.2.6' into Redis_626_Merge
Former-commit-id: e6d7e01be6965110d487e12f40511fe0b3497695
2021-10-21 22:33:55 +00:00
John Sully
59855278e0 Initial implementation of async commands
Former-commit-id: 0d3a4ce7e60c42b2d81962f13100bd1789bf64a8
2021-09-21 20:43:16 +00:00
MalavanEQAlpha
e0af0bcb95 Merge pull request #313 from MalavanEQAlpha/fixissue295
Resolve Issue #295 by speeding up keyIsExpired and setting timeout on subkey expires.

Former-commit-id: 9e5e6c2f86548b18ae27b4e1ac20c72517392b07
2021-08-18 17:34:18 -04:00
malavan
67ffa578d7 Merge remote-tracking branch 'opensource/unstable' into keydbpro
Former-commit-id: cf40fa5da29ba9574d20d706d0a41ca7001df95a
2021-08-11 20:41:54 +00:00
VivekSainiEQ
23c0c8171e Merge remote-tracking branch 'upstream/unstable' into redisearch_fixes
Former-commit-id: 952ae9fbe38b99593ec175da6718a5d7fae49837
2021-08-11 19:28:04 +00:00
malavan
3a379f0934 use INVALID_EXPIRE instead of -1
Former-commit-id: 9e45984a97a293d87474f87612204a24c831a343
2021-07-14 22:13:29 +00:00
John Sully
6bb2075d3c Merge branch 'unstable' into keydbpro
Former-commit-id: 81ded8a35daa5100cac7299a7d0b5f43ee7ac74f
2021-07-09 04:41:47 +00:00
John Sully
da2aceabcf Merge tag '6.2.3' into unstable
Former-commit-id: 1895dbb7680fa9aadf6040912e89c733abc8c706
2021-07-09 04:40:31 +00:00
John Sully
6b4f686f5f Merge branch 'unstable' into keydbpro
Former-commit-id: 205d8f18d2bb8df5253bab40578b006b7aa73fd5
2021-05-28 23:32:46 +00:00
VivekSainiEQ
68cdb2f1ad Initialized serverTL in more places in module.cpp
Former-commit-id: 8d81592bec0a62b8f3eae6b8c924887839909e2c
2021-05-25 16:55:47 +00:00
John Sully
5267928381 Merge tag '6.2.2' into unstable
Former-commit-id: 93ebb31b17adec5d406d2e30a5b9ea71c07fce5c
2021-05-21 05:54:39 +00:00
John Sully
fe8efa916b Merge tag '6.2.1' into unstable
Former-commit-id: bfed57e3e0edaa724b9d060a6bb8edc5a6de65fa
2021-05-19 02:59:48 +00:00
VivekSainiEQ
c48144356e Created and initialized seperate thread variables for modules
Former-commit-id: 3bb6b16c4a8f692b46040b72a51bef57fa03f1e6
2021-03-30 13:46:03 -04:00
VivekSainiEQ
8714a26cd9 Removed hasModuleGIL boolean and added fix from PR #292
Former-commit-id: 68d213f4c9c1c3161929a5e20ca4f2b27665c8fd
2021-03-19 20:10:24 +00:00
VivekSainiEQ
189967e7af added lock releasing w/ hasModuleGIL, changed module serverTL, and moved module_blocking_pipe to global scope to fix issue #276
Former-commit-id: 7d9a2ce827a2f8d48e4682b3cc95460cc82f9778
2021-03-19 20:08:28 +00:00
John Sully
e5343f47c2 Merge branch 'unstable' into keydbpro
Former-commit-id: e2140793f2bf565972ada799af73bf4457e2718d
2021-02-08 18:17:09 +00:00
christianEQ
cca8d186aa Merge remote-tracking branch 'opensource/unstable' into keydbpro
Former-commit-id: 5bad058733de2c217340bb9ee48f02b07d754808
2021-02-03 18:10:27 +00:00
John Sully
23495c22a5 Simplify the handling of CreateTimer
Former-commit-id: 7b4e25008a352bd425582a3e60b26894826af626
2021-02-01 15:51:22 -05:00
VivekSainiEQ
55659b1569 Modified RM_CreateTimer to prevent more that one function being posted at a time
Former-commit-id: f66e5a9c3d6925c3c6d383fbc43fbfbd56721dad
2021-02-01 15:51:22 -05:00
VivekSainiEQ
ad11549020 Changed RM_CreateTimer to only call aePostFunction is existing aePostFunction isn't in flight
Former-commit-id: 9954f5b4a48286d07fb876fd9579801365b6c237
2021-02-01 15:51:22 -05:00
VivekSainiEQ
73e8825266 Now post entire timer installation process as one function to make atomic with respect to global locks
Former-commit-id: 53936661c88bd7eac88308afc75c510134a8e044
2021-02-01 15:51:22 -05:00
VivekSainiEQ
eb83f3d47b Removed more uses of fSynchronous and the use of condition variable and mutex on the control struct.
Former-commit-id: 6ab08cc3e1429178b26b55ed7aa8ba85240eb766
2021-02-01 15:51:22 -05:00
VivekSainiEQ
363293ef61 Added fix for scenario where module thread waiting for s_mutexModule in acquireGIL can deadlock with module thread waiting for s_mutex in releaseGIL
Former-commit-id: 3205373bb378f895824cc1936a6bae663b1abdcc
2021-02-01 15:51:22 -05:00
VivekSainiEQ
cba4dcffdf removed synchronous calls to aePostFunction and changed scope of g_fModuleThread in order to prevent module related deadlocks, issue #214
Former-commit-id: 3b8d1f7076e4ab2082cd0768abc7b0b6ed4f951a
2021-02-01 15:51:22 -05:00
christianEQ
358debebfa Merge tag 'tags/6.0.10' into redismerge_2021-01-20
Former-commit-id: dadce055f897cee83946c2d3e5cbb76341b94230
2021-01-26 21:43:09 +00:00
John Sully
706ba4e1dc Merge branch 'unstable' into keydbpro
Former-commit-id: b7a1e16c0f04e8aeb3764e3681fa14fc0f97f6a3
2020-12-10 02:37:28 +00:00
John Sully
b336d269ca Remove addReply*Async methods since we already know if its async or not. This is just a source of bugs
Former-commit-id: df22cdf6e91a1b9c390b69c4209c719ecf1e44f1
2020-10-24 02:18:03 +00:00
John Sully
2085e21dca Fix module test failures due to locking
Former-commit-id: 420ccdfbaebc452b9b374b54c6ebeec4a3ffea36
2020-10-01 00:08:54 +00:00
John Sully
4f18a247e3 Merge tag '6.0.8' into unstable
Former-commit-id: 4c7e4b91a6bb2034636856b608b8c386d07f5541
2020-09-30 19:47:55 +00:00
John Sully
6a57593467 Merge branch 'PRO_RELEASE_6' into keydbpro
Former-commit-id: bffe010ea5279bee869bc61cc6d933979e10bbea
2020-07-13 03:32:14 +00:00
John Sully
1ad2d96697 Merge branch 'unstable' into keydbpro
Former-commit-id: 0dafbc254a0efd5ee302d5c58fb2ca0a85110104
2020-07-13 03:31:47 +00:00
John Sully
efc8e719f9 Prevent deadlock in RM_ThreadSafeContextLock() when triggered as part of a module callback in a server thread
Former-commit-id: e01312642be3cc78e7b383dee958a9b5c0ffc103
2020-07-12 18:17:53 +00:00
John Sully
c83fe382ff COW pointer is not thread safe so we need a global expire lock
Former-commit-id: ac12eef75301564d920e622a08e4f586486fd395
2020-06-09 19:58:42 -04:00
John Sully
44c1e6d5e7 Merge branch 'unstable' into keydbpro
Former-commit-id: 08a36155e3db9918048e87c3d691b7317787c9ab
2020-06-01 17:41:37 -04:00
John Sully
4038288c4e Fix module multithreaded test failures
Former-commit-id: 1ef35cf466ea944c56974b3795d7d6b5e89f5a3d
2020-05-31 23:46:12 -04:00
John Sully
cece963cf3 Merge branch 'unstable' into keydbpro
Former-commit-id: a830cf85df236885558c5571c0bf23cfb23e3655
2020-05-24 14:41:53 -04:00
John Sully
27eb239f1a Fix bad merge in CI.yml
Former-commit-id: 6311d709c39b3bacaeab77b18033010f1b548f81
2020-05-21 22:09:06 -04:00
John Sully
c4db71f971 Merge commit '024c380b9da02bc4112822c0f5f9ac1388b4205b' into unstable
Former-commit-id: 7676f5b15f24a044257250b8891d23b14642da48
2020-05-21 17:36:53 -04:00