futriix/src/unit/test_kvstore.c
Nadav Gigi f2510783f9
Accelerate hash table iterator with value prefetching (#1568)
This PR builds upon the [previous entry prefetching
optimization](https://github.com/valkey-io/valkey/pull/1501) to further
enhance performance by implementing value prefetching for hashtable
iterators.

## Implementation
Modified `hashtableInitIterator` to accept a new flags parameter,
allowing control over iterator behavior.
Implemented conditional value prefetching within `hashtableNext` based
on the new `HASHTABLE_ITER_PREFETCH_VALUES` flag.
When the flag is set, hashtableNext now calls `prefetchBucketValues` at
the start of each new bucket, preemptively loading the values of filled
entries into the CPU cache.
The actual prefetching of values is performed using type-specific
callback functions implemented in `server.c`:
- For `robj` the `hashtableObjectPrefetchValue` callback is used to
prefetch the value if not embeded.

This implementation is specifically focused on main database iterations
at this stage. Applying it to hashtables that hold other object types
should not be problematic, but its performance benefits for those cases
will need to be proven through testing and benchmarking.

## Performance

### Setup:
- 64cores Graviton 3 Amazon EC2 instance.
-  50 mil keys with different value sizes.
-  Running valkey server over RAM file system.
-  crc checksum and comperssion off.

### Action
- save command.

### Results
The results regarding the duration of “save” command was taken from
“info all” command.
```
+--------------------+------------------+------------------+ 
| Prefetching        | Value size (byte)| Time (seconds)   | 
+--------------------+------------------+------------------+ 
| No                 | 100              | 20.112279        | 
| Yes                | 100              | 12.758519        | 
| No                 | 40               | 16.945366        | 
| Yes                | 40               | 10.902022        |
| No                 | 20               | 9.817000         | 
| Yes                | 20               | 9.626821         |
| No                 | 10               | 9.71510          | 
| Yes                | 10               | 9.510565         |
+--------------------+------------------+------------------+
```
The results largely align with our expectations, showing significant
improvements for larger values (100 bytes and 40 bytes) that are stored
outside the robj. For smaller values (20 bytes and 10 bytes) that are
embedded within the robj, we see almost no improvement, which is as
expected.

However, the small improvement observed even for these embedded values
is somewhat surprising. Given that we are not actively prefetching these
embedded values, this minor performance gain was not anticipated.

perf record on save command **without** value prefetching:
```
                --99.98%--rdbSaveDb
                          |          
                          |--91.38%--rdbSaveKeyValuePair
                          |          |          
                          |          |--42.72%--rdbSaveRawString
                          |          |          |          
                          |          |          |--26.69%--rdbWriteRaw
                          |          |          |          |          
                          |          |          |           --25.75%--rioFileWrite.lto_priv.0
                          |          |          |          
                          |          |           --15.41%--rdbSaveLen
                          |          |                     |          
                          |          |                     |--7.58%--rdbWriteRaw
                          |          |                     |          |          
                          |          |                     |           --7.08%--rioFileWrite.lto_priv.0
                          |          |                     |                     |          
                          |          |                     |                      --6.54%--_IO_fwrite
                          |          |                     |                                         
                          |          |                     |          
                          |          |                      --7.42%--rdbWriteRaw.constprop.1
                          |          |                                |          
                          |          |                                 --7.18%--rioFileWrite.lto_priv.0
                          |          |                                           |          
                          |          |                                            --6.73%--_IO_fwrite
                          |          |                                                            
                          |          |          
                          |          |--40.44%--rdbSaveStringObject
                          |          |          
                          |           --7.62%--rdbSaveObjectType
                          |                     |          
                          |                      --7.39%--rdbWriteRaw.constprop.1
                          |                                |          
                          |                                 --7.04%--rioFileWrite.lto_priv.0
                          |                                           |          
                          |                                            --6.59%--_IO_fwrite
                          |                                                               
                          |          
                           --7.33%--hashtableNext.constprop.1
                                     |          
                                      --6.28%--prefetchNextBucketEntries.lto_priv.0
```
perf record on save command **with** value prefetching:
```
               rdbSaveRio
               |          
                --99.93%--rdbSaveDb
                          |          
                          |--79.81%--rdbSaveKeyValuePair
                          |          |          
                          |          |--66.79%--rdbSaveRawString
                          |          |          |          
                          |          |          |--42.31%--rdbWriteRaw
                          |          |          |          |          
                          |          |          |           --40.74%--rioFileWrite.lto_priv.0
                          |          |          |          
                          |          |           --23.37%--rdbSaveLen
                          |          |                     |          
                          |          |                     |--11.78%--rdbWriteRaw
                          |          |                     |          |          
                          |          |                     |           --11.03%--rioFileWrite.lto_priv.0
                          |          |                     |                     |          
                          |          |                     |                      --10.30%--_IO_fwrite
                          |          |                     |                                |          
                          |          |                     |          
                          |          |                      --10.98%--rdbWriteRaw.constprop.1
                          |          |                                |          
                          |          |                                 --10.44%--rioFileWrite.lto_priv.0
                          |          |                                           |          
                          |          |                                            --9.74%--_IO_fwrite
                          |          |                                                      |          
                          |          |          
                          |          |--11.33%--rdbSaveObjectType
                          |          |          |          
                          |          |           --10.96%--rdbWriteRaw.constprop.1
                          |          |                     |          
                          |          |                      --10.51%--rioFileWrite.lto_priv.0
                          |          |                                |          
                          |          |                                 --9.75%--_IO_fwrite
                          |          |                                           |          
                          |          |          
                          |           --0.77%--rdbSaveStringObject
                          |          
                           --18.39%--hashtableNext
                                     |          
                                     |--10.04%--hashtableObjectPrefetchValue
                                     |
                                      --6.06%--prefetchNextBucketEntries        

```
Conclusions:

The prefetching strategy appears to be working as intended, shifting the
performance bottleneck from data access to I/O operations.
The significant reduction in rdbSaveStringObject time suggests that
string objects(which are the values) are being accessed more
efficiently.

Signed-off-by: NadavGigi <nadavgigi102@gmail.com>
2025-01-23 12:17:20 +01:00

194 lines
5.6 KiB
C

#include "../kvstore.c"
#include "test_help.h"
uint64_t hashTestCallback(const void *key) {
return hashtableGenHashFunction((char *)key, strlen((char *)key));
}
int cmpTestCallback(const void *k1, const void *k2) {
return strcmp(k1, k2);
}
void freeTestCallback(void *val) {
zfree(val);
}
hashtableType KvstoreHashtableTestType = {
.hashFunction = hashTestCallback,
.keyCompare = cmpTestCallback,
.entryDestructor = freeTestCallback,
.rehashingStarted = kvstoreHashtableRehashingStarted,
.rehashingCompleted = kvstoreHashtableRehashingCompleted,
.trackMemUsage = kvstoreHashtableTrackMemUsage,
.getMetadataSize = kvstoreHashtableMetadataSize,
};
char *stringFromInt(int value) {
char buf[32];
int len;
char *s;
len = snprintf(buf, sizeof(buf), "%d", value);
s = zmalloc(len + 1);
memcpy(s, buf, len);
s[len] = '\0';
return s;
}
int test_kvstoreAdd16Keys(int argc, char **argv, int flags) {
UNUSED(argc);
UNUSED(argv);
UNUSED(flags);
int i;
int didx = 0;
kvstore *kvs1 = kvstoreCreate(&KvstoreHashtableTestType, 0, KVSTORE_ALLOCATE_HASHTABLES_ON_DEMAND);
kvstore *kvs2 = kvstoreCreate(&KvstoreHashtableTestType, 0, KVSTORE_ALLOCATE_HASHTABLES_ON_DEMAND | KVSTORE_FREE_EMPTY_HASHTABLES);
for (i = 0; i < 16; i++) {
TEST_ASSERT(kvstoreHashtableAdd(kvs1, didx, stringFromInt(i)));
TEST_ASSERT(kvstoreHashtableAdd(kvs2, didx, stringFromInt(i)));
}
TEST_ASSERT(kvstoreHashtableSize(kvs1, didx) == 16);
TEST_ASSERT(kvstoreSize(kvs1) == 16);
TEST_ASSERT(kvstoreHashtableSize(kvs2, didx) == 16);
TEST_ASSERT(kvstoreSize(kvs2) == 16);
kvstoreRelease(kvs1);
kvstoreRelease(kvs2);
return 0;
}
int test_kvstoreIteratorRemoveAllKeysNoDeleteEmptyHashtable(int argc, char **argv, int flags) {
UNUSED(argc);
UNUSED(argv);
UNUSED(flags);
int i;
void *key;
kvstoreIterator *kvs_it;
int didx = 0;
int curr_slot = 0;
kvstore *kvs1 = kvstoreCreate(&KvstoreHashtableTestType, 0, KVSTORE_ALLOCATE_HASHTABLES_ON_DEMAND);
for (i = 0; i < 16; i++) {
TEST_ASSERT(kvstoreHashtableAdd(kvs1, didx, stringFromInt(i)));
}
kvs_it = kvstoreIteratorInit(kvs1, HASHTABLE_ITER_SAFE);
while (kvstoreIteratorNext(kvs_it, &key)) {
curr_slot = kvstoreIteratorGetCurrentHashtableIndex(kvs_it);
TEST_ASSERT(kvstoreHashtableDelete(kvs1, curr_slot, key));
}
kvstoreIteratorRelease(kvs_it);
hashtable *ht = kvstoreGetHashtable(kvs1, didx);
TEST_ASSERT(ht != NULL);
TEST_ASSERT(kvstoreHashtableSize(kvs1, didx) == 0);
TEST_ASSERT(kvstoreSize(kvs1) == 0);
kvstoreRelease(kvs1);
return 0;
}
int test_kvstoreIteratorRemoveAllKeysDeleteEmptyHashtable(int argc, char **argv, int flags) {
UNUSED(argc);
UNUSED(argv);
UNUSED(flags);
int i;
void *key;
kvstoreIterator *kvs_it;
int didx = 0;
int curr_slot = 0;
kvstore *kvs2 = kvstoreCreate(&KvstoreHashtableTestType, 0, KVSTORE_ALLOCATE_HASHTABLES_ON_DEMAND | KVSTORE_FREE_EMPTY_HASHTABLES);
for (i = 0; i < 16; i++) {
TEST_ASSERT(kvstoreHashtableAdd(kvs2, didx, stringFromInt(i)));
}
kvs_it = kvstoreIteratorInit(kvs2, HASHTABLE_ITER_SAFE);
while (kvstoreIteratorNext(kvs_it, &key)) {
curr_slot = kvstoreIteratorGetCurrentHashtableIndex(kvs_it);
TEST_ASSERT(kvstoreHashtableDelete(kvs2, curr_slot, key));
}
kvstoreIteratorRelease(kvs_it);
/* Make sure the hashtable was removed from the rehashing list. */
while (kvstoreIncrementallyRehash(kvs2, 1000)) {
}
hashtable *ht = kvstoreGetHashtable(kvs2, didx);
TEST_ASSERT(ht == NULL);
TEST_ASSERT(kvstoreHashtableSize(kvs2, didx) == 0);
TEST_ASSERT(kvstoreSize(kvs2) == 0);
kvstoreRelease(kvs2);
return 0;
}
int test_kvstoreHashtableIteratorRemoveAllKeysNoDeleteEmptyHashtable(int argc, char **argv, int flags) {
UNUSED(argc);
UNUSED(argv);
UNUSED(flags);
int i;
void *key;
kvstoreHashtableIterator *kvs_di;
int didx = 0;
kvstore *kvs1 = kvstoreCreate(&KvstoreHashtableTestType, 0, KVSTORE_ALLOCATE_HASHTABLES_ON_DEMAND);
for (i = 0; i < 16; i++) {
TEST_ASSERT(kvstoreHashtableAdd(kvs1, didx, stringFromInt(i)));
}
kvs_di = kvstoreGetHashtableIterator(kvs1, didx, HASHTABLE_ITER_SAFE);
while (kvstoreHashtableIteratorNext(kvs_di, &key)) {
TEST_ASSERT(kvstoreHashtableDelete(kvs1, didx, key));
}
kvstoreReleaseHashtableIterator(kvs_di);
hashtable *ht = kvstoreGetHashtable(kvs1, didx);
TEST_ASSERT(ht != NULL);
TEST_ASSERT(kvstoreHashtableSize(kvs1, didx) == 0);
TEST_ASSERT(kvstoreSize(kvs1) == 0);
kvstoreRelease(kvs1);
return 0;
}
int test_kvstoreHashtableIteratorRemoveAllKeysDeleteEmptyHashtable(int argc, char **argv, int flags) {
UNUSED(argc);
UNUSED(argv);
UNUSED(flags);
int i;
void *key;
kvstoreHashtableIterator *kvs_di;
int didx = 0;
kvstore *kvs2 = kvstoreCreate(&KvstoreHashtableTestType, 0, KVSTORE_ALLOCATE_HASHTABLES_ON_DEMAND | KVSTORE_FREE_EMPTY_HASHTABLES);
for (i = 0; i < 16; i++) {
TEST_ASSERT(kvstoreHashtableAdd(kvs2, didx, stringFromInt(i)));
}
kvs_di = kvstoreGetHashtableIterator(kvs2, didx, HASHTABLE_ITER_SAFE);
while (kvstoreHashtableIteratorNext(kvs_di, &key)) {
TEST_ASSERT(kvstoreHashtableDelete(kvs2, didx, key));
}
kvstoreReleaseHashtableIterator(kvs_di);
hashtable *ht = kvstoreGetHashtable(kvs2, didx);
TEST_ASSERT(ht == NULL);
TEST_ASSERT(kvstoreHashtableSize(kvs2, didx) == 0);
TEST_ASSERT(kvstoreSize(kvs2) == 0);
kvstoreRelease(kvs2);
return 0;
}