125 Commits

Author SHA1 Message Date
Binbin
d6601de73f Fix that slot return in CLUSTER SHARDS should be integer (#12561)
An unintentional change was introduced in #10536, we used
to use addReplyLongLong and now it is addReplyBulkLonglong,
revert it back the previous behavior.

(cherry picked from commit 4031a187321be26fc96c044fec3ee759841e8379)
2023-10-18 10:44:10 +03:00
Binbin
413c8be5ad Support NO ONE block in REPLICAOF command json (#12633)
The current commands.json doesn't mention the special NO ONE arguments.
This change is also applied to SLAVEOF

(cherry picked from commit 8d92f7f2b7ff2d3e42425f65c5f65fe80888dfaf)
2023-10-18 10:44:10 +03:00
nihohit
3e7523aceb Update command tips on more admin / configuration commands (#12545)
Updated the command tips for ACL SAVE / SETUSER / DELUSER, CLIENT SETNAME / SETINFO, and LATENCY RESET.
The tips now match CONFIG SET, since there's a similar behavior for all of these commands - the
user expects to update the various configurations & states on all nodes, not only on a single, random node.
For LATENCY RESET the response tip is now agg_sum.

Co-authored-by: Shachar Langbeheim <shachlan@amazon.com>
(cherry picked from commit 90e9fc387c81d48d3d82ea7fcd2d3b6d896bf923)
2023-09-06 20:56:15 +03:00
Binbin
0b2cbf138e Update sort_ro reply_schema to mention the null reply (#12534)
Also added a test to cover this case, so this can
cover the reply schemas check.

(cherry picked from commit 9ce8c54d74f40bc5a70f210c2b14cf89c031ee10)
2023-09-06 20:56:15 +03:00
nihohit
68305c5b99 Align CONFIG RESETSTAT/REWRITE tips with SET. (#12530)
Since the three commands have similar behavior (change config, return
OK), the tips that govern how they should behave should be similar.

Co-authored-by: Shachar Langbeheim <shachlan@amazon.com>
(cherry picked from commit 4b281ce519974061d22eef7a9a8072d3611e64f9)
2023-09-06 20:56:15 +03:00
Binbin
f4549d1cf4
Fix CLUSTER REPLICAS time complexity, should be O(N) (#12477)
We iterate over all replicas to get the result, the time complexity
should be O(N), like CLUSTER NODES complexity is O(N).
2023-08-14 20:57:55 -07:00
Binbin
7af9f4b36e
Fix GEOHASH / GEODIST / GEOPOS time complexity, should be O(1) (#12445)
GEOHASH / GEODIST / GEOPOS use zsetScore to get the score, in skiplist encoding,
we use dictFind to get the score, which is O(1), same as ZSCORE command.
It is not clear why these commands had O(Log(N)), and O(N) until now.
2023-08-05 07:29:24 +03:00
nihohit
9f512017aa
Update request/response policies. (#12417)
changing the response and request policy of a few commands,
see https://redis.io/docs/reference/command-tips

1. RANDOMKEY used to have no response policy, which means
  that when sent to multiple shards, the responses should be aggregated.
  this normally applies to commands that return arrays, but since RANDOMKEY
  replies with a simple string, it actually requires a SPECIAL response policy
  (for the client to select just one)
2. SCAN used to have no response policy, but although the key names part of
  the response can be aggregated, the cursor part certainly can't.
3. MSETNX had a request policy of MULTI_SHARD and response policy of AGG_MIN,
  but in fact the contract with MSETNX is that when one key exists, it returns 0
  and doesn't set any key, routing it to multiple shards would mean that if one failed
  and another succeeded, it's atomicity is broken and it's impossible to return a valid
  response to the caller.

Co-authored-by: Shachar Langbeheim <shachlan@amazon.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2023-07-25 10:21:23 +03:00
Binbin
d306d86146
Fix ZRANK/ZREVRANK reply_schema description (#12331)
The parameter name is WITHSCORE instead of WITHSCORES.
2023-06-20 11:15:40 +03:00
Binbin
b510624978
Optimize PSUBSCRIBE and PUNSUBSCRIBE from O(N*M) to O(N) (#12298)
In the original implementation, the time complexity of the commands
is actually O(N*M), where N is the number of patterns the client is
already subscribed and M is the number of patterns to subscribe to.
The docs are all wrong about this.

Specifically, because the original client->pubsub_patterns is a list,
so we need to do listSearchKey which is O(N). In this PR, we change it
to a dict, so the search becomes O(1).

At the same time, both pubsub_channels and pubsubshard_channels are dicts.
Changing pubsub_patterns to a dictionary improves the readability and
maintainability of the code.
2023-06-19 16:31:18 +03:00
Harkrishn Patro
a9e32767f7
Allow cluster slots/shards api to respond during loading (#12269)
It would be helpful for clients to get cluster slots/shards information during a node failover and is loading data.
2023-06-13 18:16:32 +03:00
Binbin
e775b34e81
Add new loglevel nothing to sentinel-config.json (#12235)
It was missing in #12223, and the reply-schemas daily
was failing:
```
jsonschema.exceptions.ValidationError: 'nothing' is not valid under any of the given schemas

Failed validating 'oneOf' in schema[0]['properties']['loglevel']:
    {'oneOf': [{'const': 'debug'},
               {'const': 'verbose'},
               {'const': 'notice'},
               {'const': 'warning'},
               {'const': 'unknown'}]}

On instance['loglevel']:
    'nothing'
```
2023-05-26 09:12:23 +03:00
Wen Hui
df1890ef7f
Allow SENTINEL CONFIG SET and SENTINEL CONFIG GET to handle multiple parameters. (#10362)
Extend SENTINEL CONFIG SET and SENTINEL CONFIG GET to be
compatible with variadic CONFIG SET and CONFIG GET and allow multiple
parameters to be modified in a single call atomically.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-05-17 10:26:02 +03:00
Binbin
8597991e8f
Correct COMMAND DOCS summary, like COMMAND INFO (#12152)
This pattern is from COMMAND INFO:
Returns information about one, multiple or all commands.

Also re-generate commands.def, the GEO change was missing in #12151.
2023-05-10 08:26:46 +03:00
Leibale Eidelman
e04ebdb8d3
fix GEORADIUS[BYMEMBER] STORE & STOREDIST args spec (#12151)
in GEO commands, `STORE` and `STOREDIST` are mutually exclusive.
use `oneof` block to contain them
2023-05-09 14:24:37 +03:00
cui fliter
03d50e0c30
Remove several instances of duplicate "the" in comments (#12144)
Remove several instances of duplicate "the" in comments
2023-05-08 16:12:44 -07:00
Binbin
d659c73456
Add missing reply schema and coverage tests (#12079)
The change in #12018 break the CI (fixed by #12083).
There are quite a few sentinel commands that are missing both test coverage and also schema.

PR added reply-schema to the following commands:
- sentinel debug
- sentinel info-cache
- sentinel pendding-scripts
- sentinel reset
- sentinel simulate-failure

Added some very basic tests for other sentinel commands, just so that they have some coverage.
- sentinel help
- sentinel masters
- sentinel myid
- sentinel sentinels
- sentinel slaves

These tests should be improved / replaced in a followup PR.
2023-04-27 09:32:14 +03:00
Juho Kim
2f46e96d44
Fix typos (#12076)
"lst" to "last" in commands help
2023-04-20 11:50:28 +03:00
Wen Hui
b220e6b450
Updating reply_schema for sentinal commands (#12018)
Some sentinel subcommands are missing the reply_schema in the json file,
so add the proper reply_schema part in json file as sentinel replicas commands.

The schema validator was skipping coverage test for sentinel commands, this was initially
done just in order to focus on redis commands and leave sentinel coverage for later,
so this check is now removed.

sentinel commands that were missing reply schema:
* sentinel masters
* sentinel myid
* sentinel sentinels <master-name>
* sentinel slaves (deprecated)  <master-name>
2023-04-19 09:08:11 +03:00
Wen Hui
1250c3cf80
Update reply_schema details for info and hset commands json files accordingly. (#12017)
These commands had an empty description.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-04-13 12:17:28 +03:00
Itamar Haber
0c3b8b7e90
Overhauls command summaries and man pages. (#11942)
This is an attempt to normalize/formalize command summaries.

Main actions performed:

* Starts with the continuation of the phrase "The XXXX command, when called, ..." for user commands.
* Starts with "An internal command...", "A container command...", etc... when applicable.
* Always uses periods.
* Refrains from referring to other commands. If this is needed, backquotes should be used for command names.
* Tries to be very clear about the data type when applicable.
* Tries to mention additional effects, e.g. "The key is created if it doesn't exist" and "The set is deleted if the last member is removed."
* Prefers being terse over verbose.
* Tries to be consistent.
2023-03-29 20:48:59 +03:00
Oran Agra
ef50118c92
add 7.2 history details to xinfo json files (#11949) 2023-03-22 09:47:39 +02:00
Igor Malinovskiy
c3b9f2fbd9
Allow clients to report name and version (#11758)
This PR allows clients to send information about the client library to redis
to be displayed in CLIENT LIST and CLIENT INFO.

Currently supports:
`CLIENT [lib-name | lib-ver] <value>`
Client libraries are expected to pipeline these right after AUTH, and ignore
the failure in case they're talking to an older version of redis.

These will be shown in CLIENT LIST and CLIENT INFO as:
* `lib-name` - meant to hold the client library name.
* `lib-ver` - meant to hold the client library version.

The values cannot contain spaces, newlines and any wild ASCII characters,
but all other normal chars are accepted, e.g `.`, `=` etc (same as CLIENT NAME).

The RESET command does NOT clear these, but they can be cleared to the
default by sending a command with a blank string.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-03-22 08:17:20 +02:00
Meir Shpilraien (Spielrein)
d0da0a6a3f
Support for RM_Call on blocking commands (#11568)
Allow running blocking commands from within a module using `RM_Call`.

Today, when `RM_Call` is used, the fake client that is used to run command
is marked with `CLIENT_DENY_BLOCKING` flag. This flag tells the command
that it is not allowed to block the client and in case it needs to block, it must
fallback to some alternative (either return error or perform some default behavior).
For example, `BLPOP` fallback to simple `LPOP` if it is not allowed to block.

All the commands must respect the `CLIENT_DENY_BLOCKING` flag (including
module commands). When the command invocation finished, Redis asserts that
the client was not blocked.

This PR introduces the ability to call blocking command using `RM_Call` by
passing a callback that will be called when the client will get unblocked.
In order to do that, the user must explicitly say that he allow to perform blocking
command by passing a new format specifier argument, `K`, to the `RM_Call`
function. This new flag will tell Redis that it is allow to run blocking command
and block the client. In case the command got blocked, Redis will return a new
type of call reply (`REDISMODULE_REPLY_PROMISE`). This call reply indicates
that the command got blocked and the user can set the on_unblocked handler using
`RM_CallReplyPromiseSetUnblockHandler`.

When clients gets unblocked, it eventually reaches `processUnblockedClients` function.
This is where we check if the client is a fake module client and if it is, we call the unblock
callback instead of performing the usual unblock operations.

**Notice**: `RM_CallReplyPromiseSetUnblockHandler` must be called atomically
along side the command invocation (without releasing the Redis lock in between).
In addition, unlike other CallReply types, the promise call reply must be released
by the module when the Redis GIL is acquired.

The module can abort the execution on the blocking command (if it was not yet
executed) using `RM_CallReplyPromiseAbort`. the API will return `REDISMODULE_OK`
on success and `REDISMODULE_ERR` if the operation is already executed.
**Notice** that in case of misbehave module, Abort might finished successfully but the
operation will not really be aborted. This can only happened if the module do not respect
the disconnect callback of the blocked client. 
For pure Redis commands this can not happened.

### Atomicity Guarantees

The API promise that the unblock handler will run atomically as an execution unit.
This means that all the operation performed on the unblock handler will be wrapped
with a multi exec transaction when replicated to the replica and AOF.
The API **do not** grantee any other atomicity properties such as when the unblock
handler will be called. This gives us the flexibility to strengthen the grantees (or not)
in the future if we will decide that we need a better guarantees.

That said, the implementation **does** provide a better guarantees when performing
pure Redis blocking command like `BLPOP`. In this case the unblock handler will run
atomically with the operation that got unblocked (for example, in case of `BLPOP`, the
unblock handler will run atomically with the `LPOP` operation that run when the command
got unblocked). This is an implementation detail that might be change in the future and the
module writer should not count on that.

### Calling blocking commands while running on script mode (`S`)

`RM_Call` script mode (`S`) was introduced on #0372. It is used for usecases where the
command that was invoked on `RM_Call` comes from a user input and we want to make
sure the user will not run dangerous commands like `shutdown`. Some command, such
as `BLPOP`, are marked with `NO_SCRIPT` flag, which means they will not be allowed on
script mode. Those commands are marked with  `NO_SCRIPT` just because they are
blocking commands and not because they are dangerous. Now that we can run blocking
commands on RM_Call, there is no real reason not to allow such commands on script mode.

The underline problem is that the `NO_SCRIPT` flag is abused to also mark some of the
blocking commands (notice that those commands know not to block the client if it is not
allowed to do so, and have a fallback logic to such cases. So even if those commands
were not marked with `NO_SCRIPT` flag, it would not harm Redis, and today we can
already run those commands within multi exec).

In addition, not all blocking commands are marked with `NO_SCRIPT` flag, for example
`blmpop` are not marked and can run from within a script.

Those facts shows that there are some ambiguity about the meaning of the `NO_SCRIPT`
flag, and its not fully clear where it should be use.

The PR suggest that blocking commands should not be marked with `NO_SCRIPT` flag,
those commands should handle `CLIENT_DENY_BLOCKING` flag and only block when
it's safe (like they already does today). To achieve that, the PR removes the `NO_SCRIPT`
flag from the following commands:
* `blmove`
* `blpop`
* `brpop`
* `brpoplpush`
* `bzpopmax`
* `bzpopmin`
* `wait`

This might be considered a breaking change as now, on scripts, instead of getting
`command is not allowed from script` error, the user will get some fallback behavior
base on the command implementation. That said, the change matches the behavior
of scripts and multi exec with respect to those commands and allow running them on
`RM_Call` even when script mode is used.

### Additional RedisModule API and changes

* `RM_BlockClientSetPrivateData` - Set private data on the blocked client without the
  need to unblock the client. This allows up to set the promise CallReply as the private
  data of the blocked client and abort it if the client gets disconnected.
* `RM_BlockClientGetPrivateData` - Return the current private data set on a blocked client.
  We need it so we will have access to this private data on the disconnect callback.
* On RM_Call, the returned reply will be added to the auto memory context only if auto
  memory is enabled, this allows us to keep the call reply for longer time then the context
  lifetime and does not force an unneeded borrow relationship between the CallReply and
  the RedisModuleContext.
2023-03-16 14:04:31 +02:00
Slava Koyfman
9344f654c6
Implementing the WAITAOF command (issue #10505) (#11713)
Implementing the WAITAOF functionality which would allow the user to
block until a specified number of Redises have fsynced all previous write
commands to the AOF.

Syntax: `WAITAOF <num_local> <num_replicas> <timeout>`
Response: Array containing two elements: num_local, num_replicas
num_local is always either 0 or 1 representing the local AOF on the master.
num_replicas is the number of replicas that acknowledged the a replication
offset of the last write being fsynced to the AOF.

Returns an error when called on replicas, or when called with non-zero
num_local on a master with AOF disabled, in all other cases the response
just contains number of fsync copies.

Main changes:
* Added code to keep track of replication offsets that are confirmed to have
  been fsynced to disk.
* Keep advancing master_repl_offset even when replication is disabled (and
  there's no replication backlog, only if there's an AOF enabled).
  This way we can use this command and it's mechanisms even when replication
  is disabled.
* Extend REPLCONF ACK to `REPLCONF ACK <ofs> FACK <ofs>`, the FACK
  will be appended only if there's an AOF on the replica, and already ignored on
  old masters (thus backwards compatible)
* WAIT now no longer wait for the replication offset after your last command, but
  rather the replication offset after your last write (or read command that caused
  propagation, e.g. lazy expiry).

Unrelated changes:
* WAIT command respects CLIENT_DENY_BLOCKING (not just CLIENT_MULTI)

Implementation details:
* Add an atomic var named `fsynced_reploff_pending` that's updated
  (usually by the bio thread) and later copied to the main `fsynced_reploff`
  variable (only if the AOF base file exists).
  I.e. during the initial AOF rewrite it will not be used as the fsynced offset
  since the AOF base is still missing.
* Replace close+fsync bio job with new BIO_CLOSE_AOF (AOF specific)
  job that will also update fsync offset the field.
* Handle all AOF jobs (BIO_CLOSE_AOF, BIO_AOF_FSYNC) in the same bio
  worker thread, to impose ordering on their execution. This solves a
  race condition where a job could set `fsynced_reploff_pending` to a higher
  value than another pending fsync job, resulting in indicating an offset
  for which parts of the data have not yet actually been fsynced.
  Imposing an ordering on the jobs guarantees that fsync jobs are executed
  in increasing order of replication offset.
* Drain bio jobs when switching `appendfsync` to "always"
  This should prevent a write race between updates to `fsynced_reploff_pending`
  in the main thread (`flushAppendOnlyFile` when set to ALWAYS fsync), and
  those done in the bio thread.
* Drain the pending fsync when starting over a new AOF to avoid race conditions
  with the previous AOF offsets overriding the new one (e.g. after switching to
  replicate from a new master).
* Make sure to update the fsynced offset at the end of the initial AOF rewrite.
  a must in case there are no additional writes that trigger a periodic fsync,
  specifically for a replica that does a full sync.

Limitations:
It is possible to write a module and a Lua script that propagate to the AOF and doesn't
propagate to the replication stream. see REDISMODULE_ARGV_NO_REPLICAS and luaRedisSetReplCommand.
These features are incompatible with the WAITAOF command, and can result
in two bad cases. The scenario is that the user executes command that only
propagates to AOF, and then immediately
issues a WAITAOF, and there's no further writes on the replication stream after that.
1. if the the last thing that happened on the replication stream is a PING
  (which increased the replication offset but won't trigger an fsync on the replica),
  then the client would hang forever (will wait for an fack that the replica will never
  send sine it doesn't trigger any fsyncs).
2. if the last thing that happened is a write command that got propagated properly,
  then WAITAOF will be released immediately, without waiting for an fsync (since
  the offset didn't change)

Refactoring:
* Plumbing to allow bio worker to handle multiple job types
  This introduces infrastructure necessary to allow BIO workers to
  not have a 1-1 mapping of worker to job-type. This allows in the
  future to assign multiple job types to a single worker, either as
  a performance/resource optimization, or as a way of enforcing
  ordering between specific classes of jobs.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-03-14 20:26:21 +02:00
guybe7
4ba47d2d21
Add reply_schema to command json files (internal for now) (#10273)
Work in progress towards implementing a reply schema as part of COMMAND DOCS, see #9845
Since ironing the details of the reply schema of each and every command can take a long time, we
would like to merge this PR when the infrastructure is ready, and let this mature in the unstable branch.
Meanwhile the changes of this PR are internal, they are part of the repo, but do not affect the produced build.

### Background
In #9656 we add a lot of information about Redis commands, but we are missing information about the replies

### Motivation
1. Documentation. This is the primary goal.
2. It should be possible, based on the output of COMMAND, to be able to generate client code in typed
  languages. In order to do that, we need Redis to tell us, in detail, what each reply looks like.
3. We would like to build a fuzzer that verifies the reply structure (for now we use the existing
  testsuite, see the "Testing" section)

### Schema
The idea is to supply some sort of schema for the various replies of each command.
The schema will describe the conceptual structure of the reply (for generated clients), as defined in RESP3.
Note that the reply structure itself may change, depending on the arguments (e.g. `XINFO STREAM`, with
and without the `FULL` modifier)
We decided to use the standard json-schema (see https://json-schema.org/) as the reply-schema.

Example for `BZPOPMIN`:
```
"reply_schema": {
    "oneOf": [
        {
            "description": "Timeout reached and no elements were popped.",
            "type": "null"
        },
        {
            "description": "The keyname, popped member, and its score.",
            "type": "array",
            "minItems": 3,
            "maxItems": 3,
            "items": [
                {
                    "description": "Keyname",
                    "type": "string"
                },
                {
                    "description": "Member",
                    "type": "string"
                },
                {
                    "description": "Score",
                    "type": "number"
                }
            ]
        }
    ]
}
```

#### Notes
1.  It is ok that some commands' reply structure depends on the arguments and it's the caller's responsibility
  to know which is the relevant one. this comes after looking at other request-reply systems like OpenAPI,
  where the reply schema can also be oneOf and the caller is responsible to know which schema is the relevant one.
2. The reply schemas will describe RESP3 replies only. even though RESP3 is structured, we want to use reply
  schema for documentation (and possibly to create a fuzzer that validates the replies)
3. For documentation, the description field will include an explanation of the scenario in which the reply is sent,
  including any relation to arguments. for example, for `ZRANGE`'s two schemas we will need to state that one
  is with `WITHSCORES` and the other is without.
4. For documentation, there will be another optional field "notes" in which we will add a short description of
  the representation in RESP2, in case it's not trivial (RESP3's `ZRANGE`'s nested array vs. RESP2's flat
  array, for example)

Given the above:
1. We can generate the "return" section of all commands in [redis-doc](https://redis.io/commands/)
  (given that "description" and "notes" are comprehensive enough)
2. We can generate a client in a strongly typed language (but the return type could be a conceptual
  `union` and the caller needs to know which schema is relevant). see the section below for RESP2 support.
3. We can create a fuzzer for RESP3.

### Limitations (because we are using the standard json-schema)
The problem is that Redis' replies are more diverse than what the json format allows. This means that,
when we convert the reply to a json (in order to validate the schema against it), we lose information (see
the "Testing" section below).
The other option would have been to extend the standard json-schema (and json format) to include stuff
like sets, bulk-strings, error-string, etc. but that would mean also extending the schema-validator - and that
seemed like too much work, so we decided to compromise.

Examples:
1. We cannot tell the difference between an "array" and a "set"
2. We cannot tell the difference between simple-string and bulk-string
3. we cannot verify true uniqueness of items in commands like ZRANGE: json-schema doesn't cover the
  case of two identical members with different scores (e.g. `[["m1",6],["m1",7]]`) because `uniqueItems`
  compares (member,score) tuples and not just the member name. 

### Testing
This commit includes some changes inside Redis in order to verify the schemas (existing and future ones)
are indeed correct (i.e. describe the actual response of Redis).
To do that, we added a debugging feature to Redis that causes it to produce a log of all the commands
it executed and their replies.
For that, Redis needs to be compiled with `-DLOG_REQ_RES` and run with
`--reg-res-logfile <file> --client-default-resp 3` (the testsuite already does that if you run it with
`--log-req-res --force-resp3`)
You should run the testsuite with the above args (and `--dont-clean`) in order to make Redis generate
`.reqres` files (same dir as the `stdout` files) which contain request-response pairs.
These files are later on processed by `./utils/req-res-log-validator.py` which does:
1. Goes over req-res files, generated by redis-servers, spawned by the testsuite (see logreqres.c)
2. For each request-response pair, it validates the response against the request's reply_schema
  (obtained from the extended COMMAND DOCS)
5. In order to get good coverage of the Redis commands, and all their different replies, we chose to use
  the existing redis test suite, rather than attempt to write a fuzzer.

#### Notes about RESP2
1. We will not be able to use the testing tool to verify RESP2 replies (we are ok with that, it's time to
  accept RESP3 as the future RESP)
2. Since the majority of the test suite is using RESP2, and we want the server to reply with RESP3
  so that we can validate it, we will need to know how to convert the actual reply to the one expected.
   - number and boolean are always strings in RESP2 so the conversion is easy
   - objects (maps) are always a flat array in RESP2
   - others (nested array in RESP3's `ZRANGE` and others) will need some special per-command
     handling (so the client will not be totally auto-generated)

Example for ZRANGE:
```
"reply_schema": {
    "anyOf": [
        {
            "description": "A list of member elements",
            "type": "array",
            "uniqueItems": true,
            "items": {
                "type": "string"
            }
        },
        {
            "description": "Members and their scores. Returned in case `WITHSCORES` was used.",
            "notes": "In RESP2 this is returned as a flat array",
            "type": "array",
            "uniqueItems": true,
            "items": {
                "type": "array",
                "minItems": 2,
                "maxItems": 2,
                "items": [
                    {
                        "description": "Member",
                        "type": "string"
                    },
                    {
                        "description": "Score",
                        "type": "number"
                    }
                ]
            }
        }
    ]
}
```

### Other changes
1. Some tests that behave differently depending on the RESP are now being tested for both RESP,
  regardless of the special log-req-res mode ("Pub/Sub PING" for example)
2. Update the history field of CLIENT LIST
3. Added basic tests for commands that were not covered at all by the testsuite

### TODO

- [x] (maybe a different PR) add a "condition" field to anyOf/oneOf schemas that refers to args. e.g.
  when `SET` return NULL, the condition is `arguments.get||arguments.condition`, for `OK` the condition
  is `!arguments.get`, and for `string` the condition is `arguments.get` - https://github.com/redis/redis/issues/11896
- [x] (maybe a different PR) also run `runtest-cluster` in the req-res logging mode
- [x] add the new tests to GH actions (i.e. compile with `-DLOG_REQ_RES`, run the tests, and run the validator)
- [x] (maybe a different PR) figure out a way to warn about (sub)schemas that are uncovered by the output
  of the tests - https://github.com/redis/redis/issues/11897
- [x] (probably a separate PR) add all missing schemas
- [x] check why "SDOWN is triggered by misconfigured instance replying with errors" fails with --log-req-res
- [x] move the response transformers to their own file (run both regular, cluster, and sentinel tests - need to
  fight with the tcl including mechanism a bit)
- [x] issue: module API - https://github.com/redis/redis/issues/11898
- [x] (probably a separate PR): improve schemas: add `required` to `object`s - https://github.com/redis/redis/issues/11899

Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
Co-authored-by: Hanna Fadida <hanna.fadida@redislabs.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Shaya Potter <shaya@redislabs.com>
2023-03-11 10:14:16 +02:00
Binbin
61acf515bc
Add missing since filed for new CLIENT NO-TOUCH command (#11829)
CLIENT NO-TOUCH added in #11483, but forgot to add the since
field in the JSON file. This PR adds the since field to it
with a value of 7.2.0
2023-02-23 10:56:52 +02:00
Chen Tianjie
897c3d522c
Add CLIENT NO-TOUCH for clients to run commands without affecting LRU/LFU of keys (#11483)
When no-touch mode is enabled, the client will not touch LRU/LFU of the
keys it accesses, except when executing command `TOUCH`.
This allows inspecting or modifying the key-space without affecting their eviction.

Changes:
- A command `CLIENT NO-TOUCH ON|OFF` to switch on and off this mode.
- A client flag `#define CLIENT_NOTOUCH (1ULL<<45)`, which can be shown
  with `CLIENT INFO`, by the letter "T" in the "flags" field.
- Clear `NO-TOUCH` flag in `clearClientConnectionState`, which is used by `RESET`
  command and resetting temp clients used by modules.
- Also clear `NO-EVICT` flag in `clearClientConnectionState`, this might have been an
  oversight, spotted by @madolson.
- A test using `DEBUG OBJECT` command to verify that LRU stat is not touched when
  no-touch mode is on.
 

Co-authored-by: chentianjie <chentianjie@alibaba-inc.com>
Co-authored-by: Madelyn Olson <34459052+madolson@users.noreply.github.com>
Co-authored-by: sundb <sundbcn@gmail.com>
2023-02-23 09:07:49 +02:00
guybe7
fd82bccd0e
SCAN/RANDOMKEY and lazy-expire (#11788)
Starting from Redis 7.0 (#9890) we started wrapping everything a command
 propagates with MULTI/EXEC. The problem is that both SCAN and RANDOMKEY can
lazy-expire arbitrary keys (similar behavior to active-expire), and put DELs in a transaction.

Fix: When these commands are called without a parent exec-unit (e.g. not in EVAL or
MULTI) we avoid wrapping their DELs in a transaction (for the same reasons active-expire
and eviction avoids a transaction)

This PR adds a per-command flag that indicates that the command may touch arbitrary
keys (not the ones in the arguments), and uses that flag to avoid the MULTI-EXEC.
For now, this flag is internal, since we're considering other solutions for the future.

Note for cluster mode: if SCAN/RANDOMKEY is inside EVAL/MULTI it can still cause the
same situation (as it always did), but it won't cause a CROSSSLOT because replicas and AOF
do not perform slot checks.
The problem with the above is mainly for 3rd party ecosystem tools that propagate commands
from master to master, or feed an AOF file with redis-cli into a master.
This PR aims to fix the regression in redis 7.0, and we opened #11792 to try to handle the
bigger problem with lazy expire better for another release.
2023-02-14 09:33:21 +02:00
Roshan Khatri
ac31295438
Added fields to ACL LOG error entries for precise time logging (#11477)
Added 3 fields to the ACL LOG - adds entry_id, timestamp_created and timestamp_last_updated, which updates similar existing log error entries. The pair - entry_id, timestamp_created is a unique identifier of this entry, in case the node dies and is restarted, it can detect that if it's a new series.

The primary use case of Unique id is to uniquely identify the error messages and not to detect if the server has restarted.

entry-id is the sequence number of the entry (starting at 0) since the server process started. Can also be used to check if items were "lost" if they fell between periods.
timestamp-created is the unix-time in ms at the time the entry was first created.
timestamp-last-updated is the unix-time in ms at the time the entry was last updated
Time_created gives the absolute time which better accounts for network time as compared to time since. It can also be older than 60 secs and presently there is no field that can display the original time of creation once the error entry is updated.
The reason of timestamp_last_updated field is that it provides a more precise value for the “last time” an error was seen where as, presently it is only in the 60 second period.

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2023-02-02 12:12:16 -08:00
Binbin
e7f35edb13
Document some fields history of CLIENT LIST command (#11729)
Change history:
- `user` added in 6.0.0, 0f42447a0ec841f0b3e83328ac16a573012e2880
- `argv-mem` and `tot-mem` added in 6.2.0, bea40e6a41e31a52e9e6efee77ea5a4bd873b759
- `redir` added in 6.2.0, dd1f20edc5ecda7848c31601782c5e9d7bce4788
- `resp` added in 7.0.0, 7c376398b1cd827282e17804e230c41cbb48a89c
- `multi-mem` added in 7.0.0, 2753429c99425e3d0216cba79e0e61192975f252
- `rbs` and `rbp` added in 7.0.0, 47c51d0c7858dc8ce7747b78b73cf8cec2e59ff3
- `ssub` added in 7.0.3, 35c2ee8716dc9b1d4edbbb409815a585af491335
2023-02-01 11:48:48 +02:00
Wen Hui
cc97f4cf35
update sentinel config condition (#11751)
The command:
sentinel config set option value
and
sentinel config get option

They should include at least 4 arguments instead of 3,
This PR fixes this issue.
the only impact on the client is a different error message
2023-01-26 10:10:17 +02:00
Wen Hui
81bf14c848
fix format for evalsha_ro.json file (#11756)
We should always use space instead of Tab, this PR fix the wrong code format
2023-01-25 12:42:39 -08:00
Wen Hui
5a355883c1
Fix EVAL_RO json command format (#11755)
We should always use space instead of Tab, this PR fix the wrong code format
2023-01-25 10:11:38 -08:00
knggk
44c6770372
Add minimum version information to new xsetid arguments (#11694)
the metadata for the new arguments of XSETID,
entries-added and max-deleted-id, which have been added
in Redis 7.0 was missing.
2023-01-10 09:09:51 +02:00
C Charles
eeca7f2911
Add withscore option to ZRANK and ZREVRANK. (#11235)
Add an option "withscores" to ZRANK and ZREVRANK.

Add `[withscore]` option to both `zrank` and `zrevrank`, like this:
```
z[rev]rank key member [withscore]
```
2022-11-28 11:57:11 +02:00
Wen Hui
75c66fb02c
Update Sentinel Debug command json file and add test case for it (#11513)
Command SENTINEL DEBUG could be no arguments, which display all
configurable arguments and their values.
Update the command arguments in the docs (json file) to indicate that
arguments are optional
2022-11-24 13:10:41 +02:00
Itamar Haber
f36eb5a1ba
Deprecates SETEX, PSETEX and SETNX (#11512)
Technically, these commands were deprecated as of 2.6.12, with the
introduction of the respective arguments to SET.
In reality, the deprecation note will only be added in 7.2.0.
2022-11-22 18:10:21 +02:00
Ping Xie
203b12e41f
Introduce Shard IDs to logically group nodes in cluster mode (#10536)
Introduce Shard IDs to logically group nodes in cluster mode.
1. Added a new "shard_id" field to "cluster nodes" output and nodes.conf after "hostname"
2. Added a new PING extension to propagate "shard_id"
3. Handled upgrade from pre-7.2 releases automatically
4. Refactored PING extension assembling/parsing logic

Behavior of Shard IDs:

Replicas will always follow the shards of their reported primaries. If a primary updates its shard ID, the replica will follow. (This need not follow for cluster v2) This is not an expected use case.
2022-11-16 19:24:18 -08:00
Viktor Söderqvist
07d187066a
Deprecate QUIT (#11439)
Clients should not use this command.
Instead, clients should simply close the connection when they're not used anymore.
Terminating a connection on the client side is preferable, as it eliminates `TIME_WAIT`
lingering sockets on the server side.
2022-11-09 14:41:00 +02:00
Wen Hui
fea9bbbe0f
Fix command BITFIELD_RO and BITFIELD argument json file, add some test cases for them (#11445)
According to the source code, the commands can be executed with only key name,
and no GET/SET/INCR operation arguments.
change the docs to reflect that by marking these arguments as optional.
also add tests.
2022-11-02 15:15:12 +02:00
Wen Hui
7c3916ae6e
Fix command GEOHASH and GEOPOS argument doc, mark member as optional (#11417)
These commands take a list of members, which can be empty (i.e. running
the command with just a key name).
this always results in an empty array reply, so it doesn't make much sense,
but changing it is a breaking change.

This PR fixes the documentation, making the member field as optional, just makes
sure the command format documentation is consistent with the command behavior.

The command format will be:

127.0.0.1:6381> GEOPOS key [member [member ...]]
127.0.0.1:6381> GEOHASH key [member [member ...]]
2022-10-25 14:26:08 +03:00
Binbin
9e1b879f5b
Make PFMERGE source key optional in docs, add tests with one input key, add tests on missing source keys (#11205)
The following two cases will create an empty destkey HLL:
1. called with no source keys, like `pfmerge destkey`
2. called with non-existing source keys, like `pfmerge destkey non-existing-source-key`

In the first case, in `PFMERGE`, the dest key is actually one of the source keys too.
So `PFMERGE k1 k2` is equivalent to `SUNIONSTORE k1 k1 k2`,
and `PFMERGE k1` is equivalent to `SUNIONSTORE k1 k1`.
So the first case is reasonable, the source key is actually optional.

And the second case, `PFMERGE` on missing keys should succeed and create an empty dest.
This is consistent with `PFCOUNT`, and also with `SUNIONSTORE`, no need to change.
2022-10-22 20:41:17 +03:00
Nikita Tolkachev
4c53bdc241
Fix typo in scan commands complexity statement (#11370)
remove double period at the end of sentence.
2022-10-11 09:17:55 +03:00
Binbin
3c02d1acc4
code, typo and comment cleanups (#11280)
- fix `the the` typo
- `LPOPRPUSH` does not exist, should be `RPOPLPUSH`
- `CLUSTER GETKEYINSLOT` 's time complexity should be O(N)
- `there bytes` should be `three bytes`, this closes #11266
- `slave` word to `replica` in log, modified the front and missed the back
- remove useless aofReadDiffFromParent in server.h
- `trackingHandlePendingKeyInvalidations` method adds a void parameter
2022-10-02 13:56:45 +03:00
Huang Zhw
2804eefc58
fix some commands json file (#11201)
- BITOP: turn argument `operation` from string to oneof
- CLIENT KILL: turn argument `skipme` from string to oneof
- COMMAND GETKEYS / GETKEYSANDFLAGS: change arguments to optional, and change arity to -3 (fixes regression in redis 7.0)
- CLIENT PAUSE: this command was added in v3.0.0
2022-10-02 10:52:13 +03:00
Wen Hui
e21c059967
Update sentinel-config.json to consistent with Config Get and Set operation (#11334)
The sentinel CONFIG GET command doesn't support multiple arguments, but the json file did.
remove that.
2022-09-29 09:33:37 +03:00
Wen Hui
5389fa62a6
Update group and consumer description in json file for Unifying Stream command format (#11190)
For the stream data type, some commands, such as **XGROUP CREATE, XGROUP DESTROY, XGROUP CREATECONSUMER, 
XGROUP DELCONSUMER and XINFO CONSUMERS** use groupname and consumername in the command description;

However, for the commands **XREADGROUP GROUP, XPENDING, XACK , XCLAIM  and XAUTOCLAIM**  use term "group and consumer", clients could be confused.

This PR goal is to unify all the commands to groupname and consumername.
2022-09-08 09:37:49 +03:00
Huang Zhw
a7da7473cb
Remove the NONDETERMINISTIC_OUTPUT flag from most CLUSTER sub-commands. (#11157)
TLDR: the CLUSTER command originally had the `random` flag,
so all the sub-commands initially got that new flag, but in fact many
of them don't need it.
The only effect of this change is on the output of COMMAND INFO.
2022-08-28 11:24:47 +03:00
Itamar Haber
407b5c912f
Replaces a made-up term with a real one (#11169) 2022-08-22 15:05:01 +03:00