Commit 07ed0eafa98a66 introduced some SCAN improvements, but some
changes were postponed to a later version (8.0), which this PR finishes:
1. Prepare to move the TYPE filtering to the scan callback as well. this
was put on hold since it has side effects that can be considered a
breaking change, which is that we will not attempt to do lazy expire
(delete) a key that was filtered by not matching the TYPE (changing it
would mean TYPE filter starts behaving the same as MATCH filter already
does in that respect).
2. when the specified key TYPE filter is an unknown type, server will
reply a error immediately instead of doing a full scan that comes back
empty handed.
Fixes#235
Release notes:
> SCAN: Expired keys that don't match the TYPE argument for the SCAN are
no longer deleted by SCAN
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
This patch try to correct the actual allocated size from allocator
when call sdsRedize to align the logic with sdsnewlen function.
Maybe the https://github.com/valkey-io/valkey/pull/453 optimization
should depend on this.
Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
This PR added a new option for tcl test case which will fail fast once any test cases fail.
This can be useful while running redis CI pipeline, and you want to accelerate the CI pipeline.
usage for example
> ./runtest --single unit/type/hash --fast-fail
---------
Signed-off-by: arthur.lee <arthur-lee@qq.com>
The test logic is not smart enough to realize that a test is fully
#ifdef'd out, so it will try to attach it to the test suite anyways.
This is a minor work around for the reclaim file page test so that it
will still attach the test, it will just always succeed. Also remove an
unnecessary print statement that was missed in the same test.
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Cleaned up the minor cluster refactoring notes that were intended to be
follow ups that never happened. Basically:
1. Minor style nitpicks
2. Generalized clusterNodeIsMyself so that it wasn't implementation
dependent.
3. Removed getMyClusterId, and just make it an explicit call to myself's
name, which seems more straightforward and removes unnecessary
abstraction.
4. Remove clusterNodeGetSlaveof infavor of clusterNodeGetMaster. We
already do a check if it's a replica, and if it wasn't working it would
have been crashing.
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
This is the actual PR which is created to migrate all tests related to
zmalloc into new test framework as part of the parent issue
https://github.com/valkey-io/valkey/issues/428.
Signed-off-by: Karthick Ariyaratnam <karthyuom@gmail.com>
The script extracts the comments and prototypes from module.c and does
some pre-processing, e.g. converts URLs to markdown links. The URL
regexp didn't account for '#', '?' (and a few more chars) so an URL like
`https://example.com/#section` was converted to markdown as
[https://example.com/](https://example.com/)#section
With this change, it's instead correctly converted to
[https://example.com/#section](https://example.com/#section)
Additional change: Removes an unused metadata field.
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
In c7ad9feb52,
we missed removed endian coverage from the legacy unit tests, so it failed to find it when building.
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
This PR migrates all tests related to endianconv into new test framework
as part of the parent issue https://github.com/valkey-io/valkey/issues/428.
---------
Signed-off-by: Karthick Ariyaratnam <karthyuom@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Currently, we use select(2) on DragonFlyBSD while
`kqueue` is available on DragonFlyBSD since FreeBSD 4.1
and DragonFlyBSD was originally forked from FreeBSD 4.8
`select(2)` is a pretty old technique that has many defects
compared to `kqueue`, we should switch to `kqueue` on DragonFlyBSD.
Signed-off-by: Andy Pan <i@andypan.me>
If `valkey-server` was started with the `redis-server` symlink, the old
proc names are used, for backwards compatibility.
---------
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
This migrates unit tests related to sha1 to new framework, ref: #428.
---------
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
This patch migrates all tests in sds.c into new test framework as part
of the parent issue https://github.com/valkey-io/valkey/issues/428.
---------
Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Move dependency from `slowlog.c` into `slowlog.h`, make sure the
language server can work properly under `slowlog.h`
Signed-off-by: arthur.lee <arthur-lee@qq.com>
We currently has two disjoint TCL frameworks:
1. Normal testing framework, which trigger by runtest, which individually
launches nodes for testing.
2. Cluster framework, which trigger by runtest-cluster, which pre-allocates
N nodes and uses them for testing large configurations.
The normal TCL testing framework is much more readily tested and is also
automatically run as part of the CI for new PRs. The runtest-cluster since
it runs very slowly (cannot be parallelized), it currently only runs in daily
CI, this results in some changes to the cluster not being exposed in PR CI
in time.
This PR migrate the Cluster mode tests to normal framework. Some cluster
tests are kept in runtest-cluster because of timing issues or not yet
supported, we can process them later.
Signed-off-by: Binbin <binloveplay1314@qq.com>
All the intset unit tests are migrated to new test framework as part of
https://github.com/valkey-io/valkey/pull/344, but the old framework
declaration is missed to remove from intset.h. So removed the code.
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
Updated serverPanic output in db.c based on the
extended-redis-compatibility config. and also updated comments in other
files.
---------
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
This is a preparation for adding clang-format.
These comments prevent automatic formatting in some places. With these
exceptions, we will be able to run clang-format on the rest of the code.
This is a preparation for #323.
---------
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Fix the compile error with the following command:
`make all-with-unit-tests SERVER_CFLAGS='-Werror -DSERVER_TEST'
`
```
/usr/bin/ld: /home/ubuntu/valkey-shiv-repo/valkey/src/eval.c:1172: undefined reference to `lua_next'
/usr/bin/ld: /home/ubuntu/valkey-shiv-repo/valkey/src/eval.c:1154: undefined reference to `lua_toboolean'
/usr/bin/ld: /home/ubuntu/valkey-shiv-repo/valkey/src/eval.c:1175: undefined reference to `lua_type'
/usr/bin/ld: /home/ubuntu/valkey-shiv-repo/valkey/src/eval.c:1176: undefined reference to `lua_tonumber'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:469: valkey-unit-tests] Error 1
make[1]: Leaving directory '/home/ubuntu/valkey-shiv-repo/valkey/src'
make: *** [Makefile:6: all-with-unit-tests] Error 2
```
Issue is happened as all deps libraries not linked for
valkey-unit-tests, so linked all libraries to the binary.
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
This PR migrates all tests related to kvstore into new test framework as
part of the parent issue https://github.com/valkey-io/valkey/issues/428.
---------
Signed-off-by: Karthick Ariyaratnam <karthyuom@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
This PR migrates all tests related to util into new test framework as
part of the parent issue https://github.com/valkey-io/valkey/issues/428.
---------
Signed-off-by: Karthick Ariyaratnam <karthyuom@gmail.com>
Despite the fact that SO_REUSEADDR can be set on a Unix domain socket
via setsockopt() without reporting an error, SO_REUSEADDR was actually
created for ipv4/ipv6 and it's not supported for sockets of AF_UNIX.
Therefore, setting this option on a Unix domain socket does nothing but
costs one extra system call.
Signed-off-by: Andy Pan <i@andypan.me>
Command syntax is now:
```
ZSCAN key cursor [MATCH pattern] [COUNT count] [NOSCORES]
```
Return format:
```
127.0.0.1:6379> zadd z 1 a 2 b 3 c
(integer) 3
127.0.0.1:6379> zscan z 0
1) "0"
2) 1) "a"
2) "1"
3) "b"
4) "2"
5) "c"
6) "3"
127.0.0.1:6379> zscan z 0 noscores
1) "0"
2) 1) "a"
2) "b"
3) "c"
```
when NOSCORES is on, the command will only return members in the zset,
without scores.
For client side parsing the command return, I believe it is fine as long
as the command is backwards compatible. The return structure are still
lists, what has changed is the content. And clients can tell the
difference by the subcommand they use.
Since `novalues` option of `HSCAN` is already accepted
(redis/redis#12765), I think similar thing can be done to `ZSCAN`.
---------
Signed-off-by: Chen Tianjie <TJ_Chen@outlook.com>
This PR fixes an error in the unit/READMED.md (see
https://github.com/valkey-io/valkey/issues/428) in order to correct the
steps for running single unit test file.
Signed-off-by: Karthick Ariyaratnam <karthyuom@gmail.com>
Updated strlcat function and macros name (redis_strlcat ->
valkey_strlcat).
I think the standard strcat function is not safe.
(https://codeql.github.com/codeql-query-help/cpp/cpp-unsafe-strcat/)
So, it would be better to keep it as a safe function.
Signed-off-by: NAM UK KIM <namuk2004@naver.com>
This commit does four things:
1. On various images, the linker was not able to correctly load the flto
optimizations from the archive generated for unit tests, and was
throwing errors. I was able to solve this by updating the plugin for the
fortify test, but was unable to reproduce it on the ASAN tests or find a
solution. So I decided to go with a single solution for now, which was
to just disable the linker optimizations for those tests. This shouldn't
weaken the protections provided by ASAN.
2. The change to remove flto for some reason caused some odd inlining
behavior in the intset test, that I wasn't really able to understand.
The error was basically that we were doing a 4 byte write, starting at
byte offset 8, for the first addition to listpack that was of size 10.
Practically this has no effect, since I'm not aware of any allocator
that would give us a 10 byte block as opposed to 12 (or more likely 16)
bytes. The isn't the correct behavior, since an uninitialized listpack
defaults to 16bit encoding, which should only be writing 2 bytes. I
rabbit holed like 2 hours into this, and gave up and just ignored the
warning on the file.
3. Now that address sanitizer was correctly running, it picked up two
issues. A memory leak and uninitialized value, so those were easy to
fix.
4. There is also a small change to the fortify to build the test up
front instead of later, this is just to be consistent with other tests
and has no functional change.
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
We already have global stats for input traffic, output traffic and how
many commands have been executed.
However, some users have the difficulty of locating the IP(s) which have
heavy network traffic. So here some stats for single client are
introduced.
```
tot-net-in // Total network input bytes read from the client
tot-net-out // Total network output bytes sent to the client
tot-cmds // Total commands the client has executed
```
These three stats are shown in `CLIENT LIST` and `CLIENT INFO`.
Though the metrics are handled in hot paths of the code, personally I
don't think it will slow down the server. Considering all other complex
operations handled nearby, this is only a small and simple operation.
However we do need to be cautious when adding more and more metrics, as
discussed in redis/redis#12640, we may need to find a way to tell
whether this has obvious performance degradation.
---------
Signed-off-by: Chen Tianjie <TJ_Chen@outlook.com>
Existing code does not build on macOS < 10.7. There are two issues:
1. The check for `MAC_OS_10_6_DETECTED` does the opposite of what it
should, since `AvailabilityMacros.h` does not define underscore-prefixed
versions of macros. `__MAC_OS_X_VERSION_MAX_ALLOWED` evaluates to 0, and
on 10.6 everything breaks down:
Credits to @mohd-akram who pointed out at possible origin of the this
problem.
2. Once that is fixed, on 10.6 when building for `ppc` there are new
errors, because the code uses inaccurate assumptions for archs. Fix that
too.
---------
Signed-off-by: Sergey Fedorov <vital.had@gmail.com>
I added details at the end with the Valkey entity information under an
"About Valkey" heading. Please feel free to adjust the heading if that's
not an appropriate one to use.
---------
Signed-off-by: Mike Dolan <mikedolan@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
`reclaimFilePageCache` did not set `errno` but `rdbSaveInternal` which
is logging the error assumed it did. This makes sure `errno` is set.
Signed-off-by: Ted Lyngmo <ted@lyncon.se>
The core idea was to take a lot of the stuff from the C unity framework
and adapt it a bit here. Each file in the `unit` directory that starts
with `test_` is automatically assumed to be a test suite. Within each
file, all functions that start with `test_` are assumed to be a test.
See unit/README.md for details about the implementation.
Instead of compiling basically a net new binary, the way the tests are
compiled is that the main valkey server is compiled as a static archive,
which we then compile the individual test files against to create a new
test executable. This is not all that important now, other than it makes
the compilation simpler, but what it will allow us to do is overwrite
functions in the archive to enable mocking for cross compilation unit
functions. There are also ways to enable mocking from within the same
compilation unit, but I don't know how important this is.
Tests are also written in one of two styles:
1. Including the header file and directly calling functions from the
archive.
2. Importing the original file, and then calling the functions. This
second approach is cool because we can call static functions. It won't
mess up the archive either.
---------
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
These JSON files were originally not intended to be used directly, since
they contain internals and some fiels like "acl_categories" that are not
the final ACL categories. (Valkey will apply some implicit rules to
compute the final ACL categories.) However, people see JSON files
and use them directly anyway.
So it's better to document them.
In a later PR, we can get rid of all implicit ACL categories and instead
populate them explicitly in the JSON files. Then, we'll add a validation
(e.g. in generate-command-code.py) that the implied categories are set.
---------
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Binbin <binloveplay1314@qq.com>
This is a minor change where only naming and links now points properly
to valkey.
Fixes#388
---------
Signed-off-by: Rolandas Šimkus <rolandas@simkus.io>
Signed-off-by: simkusr <rolandas.s@wilibox.com>
Signed-off-by: simkusr <rolandas@simkus.io>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: simkusr <rolandas.s@wilibox.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Renamed redis to valkey/server in aof.c serverlogs.
The AOF rewrite child process title is set to "redis-aof-rewrite" if
Valkey was started from a redis-server symlink, otherwise to
"valkey-aof-rewrite".
This is a breaking changes since logs are changed.
Part of #207.
---------
Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
Improve the performance of crc64 for large batches by processing large
number of bytes in parallel and combining the results.
## Performance
* 53-73% faster on Xeon 2670 v0 @ 2.6ghz
* 2-2.5x faster on Core i3 8130U @ 2.2 ghz
* 1.6-2.46 bytes/cycle on i3 8130U
* likely >2x faster than crcspeed on newer CPUs with more resources than
a 2012-era Xeon 2670
* crc64 combine function runs in <50 nanoseconds typical with vector +
cache optimizations (~8 *microseconds* without vector optimizations, ~80
*microseconds without cache, the combination is extra effective)
* still single-threaded
* valkey-server test crc64 --help (requires `make distclean && make
SERVER_TEST=yes`)
---------
Signed-off-by: Josiah Carlson <josiah.carlson@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
