This PR optimizes client query buffer handling in Valkey by introducing
a shared query buffer that is used by default for client reads. This
reduces memory usage by ~20KB per client by avoiding allocations for
most clients using short (<16KB) complete commands. For larger or
partial commands, the client still gets its own private buffer.
The primary changes are:
* Adding a shared query buffer `shared_qb` that clients use by default
* Modifying client querybuf initialization and reset logic
* Copying any partial query from shared to private buffer before command
execution
* Freeing idle client query buffers when empty to allow reuse of shared
buffer
* Master client query buffers are kept private as their contents need to
be preserved for replication stream
In addition to the memory savings, this change shows a 3% improvement in
latency and throughput when running with 1000 active clients.
The memory reduction may also help reduce the need to evict clients when
reaching max memory limit, as the query buffer is the main memory
consumer per client.
---------
Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
The races are between the '$rd' client and the 'r' client in the test
case.
Test case "client input output and command process statistics" in
unit/introspection.
---------
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
The replica sends its version when initiating replication, in
pipeline with other REPLCONF commands.
The primary stores it in the client struct. Other fields are made
smaller to avoid making the client struct consume more memory.
Fixes#414.
---------
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
This aligns the behaviour with established Valkey commands with a
TIMEOUT argument, such as BLPOP.
Fix#422
Signed-off-by: Ping Xie <pingxie@google.com>
- Replaces custom atomics logic with C11 default atomics logic.
- Drops "atomicvar_api" field from server info
Closes#485
---------
Signed-off-by: adetunjii <adetunjithomas1@outlook.com>
Signed-off-by: Samuel Adetunji <adetunjithomas1@outlook.com>
Co-authored-by: teej4y <samuel.adetunji@prunny.com>
Although I think this improves the readability of individual configs,
the fact there are now 1k more lines of configs makes this overall much
harder to parse. So reverting it back to the way it was before.
`,\n [ ]+` replace with `, `.
---------
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
After READONLY, make a cluster replica behave as its primary regarding
returning ASK redirects and TRYAGAIN.
Without this patch, a client reading from a replica cannot tell if a key
doesn't exist or if it has already been migrated to another shard as
part of an ongoing slot migration. Therefore, without an ASK redirect in
this situation, offloading reads to cluster replicas wasn't reliable.
Note: The target of a redirect is always a primary. If a client wants to
continue reading from a replica after following a redirect, it needs to
figure out the replicas of that new primary using CLUSTER SHARDS or
similar.
This is related to #21 and has been made possible by the introduction of
Replication of Slot Migration States in #445.
----
Release notes:
During cluster slot migration, replicas are able to return -ASK
redirects and -TRYAGAIN.
---------
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Undeprecate cluster slots command. This command is widely used by
clients to form the cluster topology and with the recent change to
improve performance of `CLUSTER SLOTS` command via #53 as well as us
looking to further improve the usability via #517, it makes sense to
undeprecate this command.
---------
Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
I have validated that these settings closely match the existing coding
style with one major exception on `BreakBeforeBraces`, which will be
`Attach` going forward. The mixed `BreakBeforeBraces` styles in the
current codebase are hard to imitate and also very odd IMHO - see below
```
if (a == 1) { /*Attach */
}
```
```
if (a == 1 ||
b == 2)
{ /* Why? */
}
```
Please do NOT merge just yet. Will add the github action next once the
style is reviewed/approved.
---------
Signed-off-by: Ping Xie <pingxie@google.com>
This commit adds a logic to cache `CLUSTER SLOTS` response for reduced
latency and also updates the cache when a change in the cluster is
detected.
Historically, `CLUSTER SLOTS` command was deprecated, however all the
server clients have been using `CLUSTER SLOTS` and have not migrated to
`CLUSTER SHARDS`. In future this logic can be added to any other
commands to improve the performance of the engine.
---------
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Issue #428.
Moved the SERVER_TEST block from ziplist.c into unit tests in
test_ziplist.c. I left the benchmark related tasks alone in their own
test, as I am not sure what to do with them.
Some of the assertions are a little vague/useless, but I will try to
refine them.
---------
Signed-off-by: Mason Hall <hallmason17@gmail.com>
Implemented in
3945a32177 (diff-a154d1fa454a9868e2c455acdae971e3605151516f9a8efac7f2c9b2845d914d),
this function is never called and never used. I was trying to understand
whether we could use this for another PR, but couldn't really find a
point for it because it didn't do exactly what I expected.
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
In 3f725b8, we introduced a change back in march to reduce the
entropy of ASLR, because ASAN didn't support it. Now the
vm.mmap_rnd_bits
was reverted in actions/runner-images#9491 so can remove this changes.
Closes#519.
Signed-off-by: Binbin <binloveplay1314@qq.com>
Commit 07ed0eafa98a66 introduced some SCAN improvements, but some
changes were postponed to a later version (8.0), which this PR finishes:
1. Prepare to move the TYPE filtering to the scan callback as well. this
was put on hold since it has side effects that can be considered a
breaking change, which is that we will not attempt to do lazy expire
(delete) a key that was filtered by not matching the TYPE (changing it
would mean TYPE filter starts behaving the same as MATCH filter already
does in that respect).
2. when the specified key TYPE filter is an unknown type, server will
reply a error immediately instead of doing a full scan that comes back
empty handed.
Fixes#235
Release notes:
> SCAN: Expired keys that don't match the TYPE argument for the SCAN are
no longer deleted by SCAN
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
This patch try to correct the actual allocated size from allocator
when call sdsRedize to align the logic with sdsnewlen function.
Maybe the https://github.com/valkey-io/valkey/pull/453 optimization
should depend on this.
Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
This PR added a new option for tcl test case which will fail fast once any test cases fail.
This can be useful while running redis CI pipeline, and you want to accelerate the CI pipeline.
usage for example
> ./runtest --single unit/type/hash --fast-fail
---------
Signed-off-by: arthur.lee <arthur-lee@qq.com>
The test logic is not smart enough to realize that a test is fully
#ifdef'd out, so it will try to attach it to the test suite anyways.
This is a minor work around for the reclaim file page test so that it
will still attach the test, it will just always succeed. Also remove an
unnecessary print statement that was missed in the same test.
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Cleaned up the minor cluster refactoring notes that were intended to be
follow ups that never happened. Basically:
1. Minor style nitpicks
2. Generalized clusterNodeIsMyself so that it wasn't implementation
dependent.
3. Removed getMyClusterId, and just make it an explicit call to myself's
name, which seems more straightforward and removes unnecessary
abstraction.
4. Remove clusterNodeGetSlaveof infavor of clusterNodeGetMaster. We
already do a check if it's a replica, and if it wasn't working it would
have been crashing.
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
This is the actual PR which is created to migrate all tests related to
zmalloc into new test framework as part of the parent issue
https://github.com/valkey-io/valkey/issues/428.
Signed-off-by: Karthick Ariyaratnam <karthyuom@gmail.com>
The script extracts the comments and prototypes from module.c and does
some pre-processing, e.g. converts URLs to markdown links. The URL
regexp didn't account for '#', '?' (and a few more chars) so an URL like
`https://example.com/#section` was converted to markdown as
[https://example.com/](https://example.com/)#section
With this change, it's instead correctly converted to
[https://example.com/#section](https://example.com/#section)
Additional change: Removes an unused metadata field.
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
In c7ad9feb52,
we missed removed endian coverage from the legacy unit tests, so it failed to find it when building.
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
This PR migrates all tests related to endianconv into new test framework
as part of the parent issue https://github.com/valkey-io/valkey/issues/428.
---------
Signed-off-by: Karthick Ariyaratnam <karthyuom@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Currently, we use select(2) on DragonFlyBSD while
`kqueue` is available on DragonFlyBSD since FreeBSD 4.1
and DragonFlyBSD was originally forked from FreeBSD 4.8
`select(2)` is a pretty old technique that has many defects
compared to `kqueue`, we should switch to `kqueue` on DragonFlyBSD.
Signed-off-by: Andy Pan <i@andypan.me>
If `valkey-server` was started with the `redis-server` symlink, the old
proc names are used, for backwards compatibility.
---------
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
This migrates unit tests related to sha1 to new framework, ref: #428.
---------
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
This patch migrates all tests in sds.c into new test framework as part
of the parent issue https://github.com/valkey-io/valkey/issues/428.
---------
Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Move dependency from `slowlog.c` into `slowlog.h`, make sure the
language server can work properly under `slowlog.h`
Signed-off-by: arthur.lee <arthur-lee@qq.com>
We currently has two disjoint TCL frameworks:
1. Normal testing framework, which trigger by runtest, which individually
launches nodes for testing.
2. Cluster framework, which trigger by runtest-cluster, which pre-allocates
N nodes and uses them for testing large configurations.
The normal TCL testing framework is much more readily tested and is also
automatically run as part of the CI for new PRs. The runtest-cluster since
it runs very slowly (cannot be parallelized), it currently only runs in daily
CI, this results in some changes to the cluster not being exposed in PR CI
in time.
This PR migrate the Cluster mode tests to normal framework. Some cluster
tests are kept in runtest-cluster because of timing issues or not yet
supported, we can process them later.
Signed-off-by: Binbin <binloveplay1314@qq.com>
All the intset unit tests are migrated to new test framework as part of
https://github.com/valkey-io/valkey/pull/344, but the old framework
declaration is missed to remove from intset.h. So removed the code.
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
Updated serverPanic output in db.c based on the
extended-redis-compatibility config. and also updated comments in other
files.
---------
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
This is a preparation for adding clang-format.
These comments prevent automatic formatting in some places. With these
exceptions, we will be able to run clang-format on the rest of the code.
This is a preparation for #323.
---------
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Fix the compile error with the following command:
`make all-with-unit-tests SERVER_CFLAGS='-Werror -DSERVER_TEST'
`
```
/usr/bin/ld: /home/ubuntu/valkey-shiv-repo/valkey/src/eval.c:1172: undefined reference to `lua_next'
/usr/bin/ld: /home/ubuntu/valkey-shiv-repo/valkey/src/eval.c:1154: undefined reference to `lua_toboolean'
/usr/bin/ld: /home/ubuntu/valkey-shiv-repo/valkey/src/eval.c:1175: undefined reference to `lua_type'
/usr/bin/ld: /home/ubuntu/valkey-shiv-repo/valkey/src/eval.c:1176: undefined reference to `lua_tonumber'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:469: valkey-unit-tests] Error 1
make[1]: Leaving directory '/home/ubuntu/valkey-shiv-repo/valkey/src'
make: *** [Makefile:6: all-with-unit-tests] Error 2
```
Issue is happened as all deps libraries not linked for
valkey-unit-tests, so linked all libraries to the binary.
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
This PR migrates all tests related to kvstore into new test framework as
part of the parent issue https://github.com/valkey-io/valkey/issues/428.
---------
Signed-off-by: Karthick Ariyaratnam <karthyuom@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
