futriix-old/src/expire.h

310 lines
9.1 KiB
C
Raw Normal View History

#pragma once
/*
* INVALID_EXPIRE is the value we set the expireEntry's m_when value to when the main key is not expired and the value we return when we try to get the expire time of a key or subkey that is not expired
* Want this value to be LLONG_MAX however we use the most significant bit of m_when as a flag to see if the expireEntry is Fat or not so we want to ensure that it is unset hence the (& ~((1LL) << (sizeof(long long)*CHAR_BIT - 1)))
* Eventually this number might actually end up being a valid expire time, this could cause bugs so at that time it might be a good idea to use a larger data type.
*/
#define INVALID_EXPIRE (LLONG_MAX & ~((1LL) << (sizeof(long long)*CHAR_BIT - 1)))
class expireEntryFat
{
friend class expireEntry;
static const int INDEX_THRESHOLD = 16;
public:
struct subexpireEntry
{
long long when;
std::unique_ptr<const char, void(*)(const char*)> spsubkey;
subexpireEntry(long long when, const char *subkey)
: when(when), spsubkey(subkey, sdsfree)
{}
subexpireEntry(const subexpireEntry &other)
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
: spsubkey(nullptr, sdsfree)
{
when = other.when;
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
if (other.spsubkey != nullptr)
spsubkey = std::unique_ptr<const char, void(*)(const char*)>((const char*)sdsdupshared(other.spsubkey.get()), sdsfree);
}
subexpireEntry(subexpireEntry &&) = default;
subexpireEntry& operator=(subexpireEntry&&) = default;
subexpireEntry& operator=(const subexpireEntry &src) {
when = src.when;
spsubkey = std::unique_ptr<const char, void(*)(const char*)>((const char*)sdsdupshared(src.spsubkey.get()), sdsfree);
return *this;
}
bool operator<(long long when) const noexcept { return this->when < when; }
bool operator<(const subexpireEntry &se) { return this->when < se.when; }
};
private:
std::vector<subexpireEntry> m_vecexpireEntries; // Note a NULL for the sds portion means the expire is for the primary key
dict *m_dictIndex = nullptr;
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
long long m_whenPrimary = LLONG_MAX;
void createIndex();
public:
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
expireEntryFat() = default;
expireEntryFat(const expireEntryFat &);
~expireEntryFat();
long long when() const noexcept { return m_vecexpireEntries.front().when; }
bool operator<(long long when) const noexcept { return this->when() < when; }
void expireSubKey(const char *szSubkey, long long when);
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
bool FGetPrimaryExpire(long long *pwhen) const {
if (m_whenPrimary != LLONG_MAX) {
*pwhen = m_whenPrimary;
return true;
}
return false;
}
bool FEmpty() const noexcept { return m_vecexpireEntries.empty(); }
const subexpireEntry &nextExpireEntry() const noexcept { return m_vecexpireEntries.front(); }
void popfrontExpireEntry();
const subexpireEntry &operator[](size_t idx) const { return m_vecexpireEntries[idx]; }
size_t size() const noexcept { return m_vecexpireEntries.size(); }
};
class expireEntry {
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
struct {
uint64_t m_whenAndPtrUnion : 63,
fFat : 1;
} s;
static_assert(sizeof(expireEntryFat*) <= sizeof(int64_t), "The pointer must fit in the union");
public:
class iter
{
friend class expireEntry;
const expireEntry *m_pentry = nullptr;
size_t m_idx = 0;
public:
iter(const expireEntry *pentry, size_t idx)
: m_pentry(pentry), m_idx(idx)
{}
iter &operator++() { ++m_idx; return *this; }
const char *subkey() const
{
if (m_pentry->FFat())
return (*m_pentry->pfatentry())[m_idx].spsubkey.get();
return nullptr;
}
long long when() const
{
if (m_pentry->FFat())
return (*m_pentry->pfatentry())[m_idx].when;
return m_pentry->when();
}
bool operator!=(const iter &other)
{
return m_idx != other.m_idx;
}
const iter &operator*() const { return *this; }
};
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
expireEntry()
{
s.fFat = 0;
s.m_whenAndPtrUnion = 0;
}
expireEntry(const char *subkey, long long when)
{
if (subkey != nullptr)
{
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
auto pfatentry = new (MALLOC_LOCAL) expireEntryFat();
pfatentry->expireSubKey(subkey, when);
s.m_whenAndPtrUnion = reinterpret_cast<long long>(pfatentry);
s.fFat = true;
}
else
{
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
s.m_whenAndPtrUnion = when;
s.fFat = false;
}
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
expireEntry(expireEntryFat *pfatentry)
{
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
assert(pfatentry != nullptr);
s.m_whenAndPtrUnion = reinterpret_cast<long long>(pfatentry);
s.fFat = true;
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
expireEntry(const expireEntry &e) {
if (e.FFat()) {
s.m_whenAndPtrUnion = reinterpret_cast<long long>(new expireEntryFat(*e.pfatentry()));
s.fFat = true;
} else {
s = e.s;
}
}
expireEntry(expireEntry &&e)
{
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
s = e.s;
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
expireEntry &operator=(expireEntry &&e)
{
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
if (FFat())
delete pfatentry();
s = e.s;
e.s.m_whenAndPtrUnion = 0;
e.s.fFat = false;
return *this;
}
expireEntry &operator=(expireEntry &e) {
if (FFat())
delete pfatentry();
if (e.FFat()) {
s.m_whenAndPtrUnion = reinterpret_cast<long long>(new expireEntryFat(*e.pfatentry()));
s.fFat = true;
} else {
s = e.s;
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
return *this;
}
// Duplicate the expire, note this is intended to be passed directly to setExpire
expireEntry duplicate() const {
expireEntry dst;
if (FFat()) {
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
auto pfatentry = new expireEntryFat(*expireEntry::pfatentry());
dst.s.m_whenAndPtrUnion = reinterpret_cast<long long>(pfatentry);
dst.s.fFat = true;
} else {
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
dst.s.m_whenAndPtrUnion = s.m_whenAndPtrUnion;
dst.s.fFat = false;
}
return dst;
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
void reset() {
if (FFat())
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
delete pfatentry();
s.fFat = false;
s.m_whenAndPtrUnion = 0;
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
~expireEntry()
{
if (FFat())
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
delete pfatentry();
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
inline bool FFat() const noexcept { return s.fFat; }
expireEntryFat *pfatentry() {
assert(FFat());
return reinterpret_cast<expireEntryFat*>(s.m_whenAndPtrUnion);
}
const expireEntryFat *pfatentry() const {
return const_cast<expireEntry*>(this)->pfatentry();
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
bool operator<(const expireEntry &e) const noexcept
{
return when() < e.when();
}
bool operator<(long long when) const noexcept
{
return this->when() < when;
}
long long when() const noexcept
{
if (FFat())
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
return pfatentry()->when();
return s.m_whenAndPtrUnion;
}
void update(const char *subkey, long long when)
{
if (!FFat())
{
if (subkey == nullptr)
{
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
s.m_whenAndPtrUnion = when;
return;
}
else
{
// we have to upgrade to a fat entry
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
auto pfatentry = new (MALLOC_LOCAL) expireEntryFat();
pfatentry->expireSubKey(nullptr, s.m_whenAndPtrUnion);
s.m_whenAndPtrUnion = reinterpret_cast<long long>(pfatentry);
s.fFat = true;
// at this point we're fat so fall through
}
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
pfatentry()->expireSubKey(subkey, when);
}
iter begin() const { return iter(this, 0); }
iter end() const
{
if (FFat())
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
return iter(this, pfatentry()->size());
return iter(this, 1);
}
void erase(iter &itr)
{
if (!FFat())
throw -1; // assert
pfatentry()->m_vecexpireEntries.erase(
pfatentry()->m_vecexpireEntries.begin() + itr.m_idx);
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
size_t size() const {
if (FFat())
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
return pfatentry()->size();
return 1;
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
bool FGetPrimaryExpire(long long *pwhen) const noexcept
{
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
if (FFat()) {
return pfatentry()->FGetPrimaryExpire(pwhen);
} else {
*pwhen = s.m_whenAndPtrUnion;
return true;
}
}
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
void *release_as_void() {
uint64_t whenT = s.m_whenAndPtrUnion;
whenT |= static_cast<uint64_t>(s.fFat) << 63;
s.m_whenAndPtrUnion = 0;
s.fFat = 0;
return reinterpret_cast<void*>(whenT);
}
static expireEntry *from_void(void **src) {
uintptr_t llV = reinterpret_cast<uintptr_t>(src);
return reinterpret_cast<expireEntry*>(llV);
}
static const expireEntry *from_void(void *const*src) {
uintptr_t llV = reinterpret_cast<uintptr_t>(src);
return reinterpret_cast<expireEntry*>(llV);
}
explicit operator long long() const noexcept { return when(); }
};
Merge latest internal to OSS (#720) * add docker build * fix the working dir in Dockerfile * add release publish docker image * address intentation and use default release * migrate keydb_modstatsd to keydb-internal * rm * add submodule cpp-statsd-client * include trigger keydb_modstatsd Makefile in modules Makefile * update * have clean also trigger keydb_modstatsd clean * move cpp-statsd-client to deps * checkout to a06a5b9359f31d946fe163b9038586982971ae49 * update relative path in compilation * remove submodule instead use the source * include building keydb statsd module * fix check in Dockerfile docker-entrypoint.sh * fix * fix the comment caused stuck docker build * use commit hash as tag template * fix * test tag * Revert "test tag" This reverts commit 9cbc57137d57aab4fdd5a9283bae07391b3c7f8b. * make docker build independent * add new build to ci * emit system free metrics with '/proc/meminfo' * have emit system free memory within metrics_time_taken_us and also add metric time taken for it * Remove Expireset (#217) Major refactor to place expiry information directly in the object struct. * update MemFree to MemAvailable in keydb statsd * add metric emit for non-empty primary with less than 2 connected replicas * address comments * Multiply CPU percent metric by 100 * Fix memory leaks * Fix slow to free when low lock contention * fix nodename metricsname * fix unnecessary replace * Make propagating before freeing module context optional (#225) * don't propogate on module context free for rdb load * default in wrong place * Flash expiration (#197) Design Doc: https://docs.google.com/document/d/1NmnYGnHLdZp-KOUCUatX5iXpF-L3YK4VUc9Lm3Tqxpo/edit?usp=sharing * Emit more aggregate metrics in modstatsd (#223) * Permit keys of differing slots as long as they are served by this cluster and we are not migrating * Fix over pessimistic checks that prevent replicas from serving mget * Fix logic bug * async rehash is preventing rehashing during RDB load after a db flush. Ensure it can't interefere after a flush * make async rehash configurable * only use internal locks when multithreaded (#205) * Fix crossslot error migrating batches of keys * Fix bug where we erroneously answer queries belonging to another shard * fix mac compile * enable level_compaction_dynamic_level_bytes after flush, and flush expires for FLASH (#229) * enable level_compaction_dynamic_level_bytes after flush, and flush expires * update debug reload for flash * update debug reload for flash complete * missing forward declare * commit existing changes then track changes for debug reload * missing args * commitChanges is conditional Co-authored-by: John Sully <jsully@snapchat.com> --------- Co-authored-by: zliang <zliang@snapchat.com> Co-authored-by: John Sully <jsully@snapchat.com> Co-authored-by: Alex Cope <acope@snapchat.com> Co-authored-by: John Sully <john@csquare.ca>
2023-09-28 18:13:27 -04:00
static_assert(sizeof(expireEntry) == sizeof(long long), "This must fit in a long long so it can be put in a dictEntry");