75 Commits

Author SHA1 Message Date
christianEQ
c068f2cd3d Merge tag 'tags/6.0.10' into redismerge_2021-01-20
Former-commit-id: dadce055f897cee83946c2d3e5cbb76341b94230
2021-01-26 21:43:09 +00:00
Meir Shpilraien (Spielrein)
58c00320e3 Moved RMAPI_FUNC_SUPPORTED location such that it will be visible to modules (#8037)
The RMAPI_FUNC_SUPPORTED was defined in the wrong place on redismodule.h
and was not visible to modules.

(cherry picked from commit 560d2dc0081bc35b81b6a64d25c4077fa2e69ad9)
2021-01-12 16:25:37 +02:00
Oran Agra
fb0783c710 Handle output buffer limits for Module blocked clients (#8141)
Module blocked clients cache the response in a temporary client,
the reply list in this client would be affected by the recent fix
in #7202, but when the reply is later copied into the real client,
it would have bypassed all the checks for output buffer limit, which
would have resulted in both: responding with a partial response to
the client, and also not disconnecting it at all.

(cherry picked from commit 11e0c4739b40c23a70aff5287e67e8ef6418bb2a)
2021-01-12 16:25:37 +02:00
Yossi Gottlieb
c6c0155725 Modules: fix RM_GetCommandKeys API. (#7901)
This cleans up and simplifies the API by passing the command name as the
first argument. Previously the command name was specified explicitly,
but was still included in the argv.

(cherry picked from commit a94ddb27fe919d60c598c2403b230b27c6e3a11c)
2020-10-27 09:12:01 +02:00
Meir Shpilraien (Spielrein)
dbf3902481 Add Module API for version and compatibility checks (#7865)
* Introduce a new API's: RM_GetContextFlagsAll, and
RM_GetKeyspaceNotificationFlagsAll that will return the
full flags mask of each feature. The module writer can
check base on this value if the Flags he needs are
supported or not.

* For each flag, introduce a new value on redismodule.h,
this value represents the LAST value and should be there
as a reminder to update it when a new value is added,
also it will be used in the code to calculate the full
flags mask (assuming flags are incrementally increasing).
In addition, stated that the module writer should not use
the LAST flag directly and he should use the GetFlagAll API's.

* Introduce a new API: RM_IsSubEventSupported, that returns for a given
event and subevent, whether or not the subevent supported.

* Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether
or not a function API is supported by comparing it to NULL.

* Introduce a new API: int RM_GetServerVersion();, that will return the
current Redis version in the format 0x00MMmmpp; e.g. 0x00060008;

* Changed unstable version from 999.999.999 to 255.255.255

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
(cherry picked from commit e9c837ad3118f5ea2cb4defb0ab82cbd113733e7)
2020-10-27 09:12:01 +02:00
Yossi Gottlieb
02edabc9aa Module API: Add RM_GetClientCertificate(). (#7866)
This API function makes it possible to retrieve the X.509 certificate
used by clients to authenticate TLS connections.

(cherry picked from commit 929f1e2ec73c147251d693631a746baaaf5d4127)
2020-10-27 09:12:01 +02:00
Yossi Gottlieb
298808cf32 Modules: Add RM_GetDetachedThreadSafeContext(). (#7886)
The main motivation here is to provide a way for modules to create a
single, global context that can be used for logging.

Currently, it is possible to obtain a thread-safe context that is not
attached to any blocked client by using `RM_GetThreadSafeContext`.
However, the attached context is not linked to the module identity so
log messages produced are not tagged with the module name.

Ideally we'd fix this in `RM_GetThreadSafeContext` itself but as it
doesn't accept the current context as an argument there's no way to do
that in a backwards compatible manner.

(cherry picked from commit e270302cdf06b6a7b38c7dcb59680419b1b589c8)
2020-10-27 09:12:01 +02:00
Yossi Gottlieb
4af8059e61 Modules: add RM_GetCommandKeys().
This is essentially the same as calling COMMAND GETKEYS but provides a
more efficient interface that can be used in every context (i.e. not a
Redis command).

(cherry picked from commit 9e58b52a19fed12c4e9fbe120ec0d4baf9c2bc28)
2020-10-27 09:12:01 +02:00
Wen Hui
d3f36e93a9 Add Swapdb Module Event (#7804)
(cherry picked from commit 0db3223bc6090556c920912d9c92dd42878e316c)
2020-10-27 09:12:01 +02:00
John Sully
14daf6f909 Merge tag '6.0.8' into unstable
Former-commit-id: 4c7e4b91a6bb2034636856b608b8c386d07f5541
2020-09-30 19:47:55 +00:00
Oran Agra
4de93718a7 Fix leak in new blockedclient module API test
(cherry picked from commit 0e20ad14a6a857cb168b808f94721df19b23dc0c)
2020-09-10 14:09:00 +03:00
Roi Lipman
ee3e45ac6e RM_ThreadSafeContextTryLock a non-blocking method for acquiring GIL (#7738)
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit b1de173ec0f6a03d6083b87f1505fbf843708685)
2020-09-10 14:09:00 +03:00
Meir Shpilraien (Spielrein)
63e3f1e449 see #7544, added RedisModule_HoldString api. (#7577)
Added RedisModule_HoldString that either returns a
shallow copy of the given String (by increasing
the String ref count) or a new deep copy of String
in case its not possible to get a shallow copy.

Co-authored-by: Itamar Haber <itamar@redislabs.com>
(cherry picked from commit 4f99b22118ca91e3a7fe9c1c68c19dd717dfdbb5)
2020-09-01 09:27:58 +03:00
Meir Shpilraien (Spielrein)
f63e428e5b This PR introduces a new loaded keyspace event (#7536)
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
(cherry picked from commit 73198c50194cbf0254afd4cc5245f9274a538d13)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb
d13c44583c Module API: fix missing RM_CLIENTINFO_FLAG_SSL. (#7666)
The `REDISMODULE_CLIENTINFO_FLAG_SSL` flag was already a part of the `RedisModuleClientInfo` structure but was not implemented.

(cherry picked from commit 2ec11f941ae41188e517670fc3224b12c7666541)
2020-09-01 09:27:58 +03:00
meir@redislabs.com
cb06a48ef3 Fix RM_ScanKey module api not to return int encoded strings
The scan key module API provides the scan callback with the current
field name and value (if it exists). Those arguments are RedisModuleString*
which means it supposes to point to robj which is encoded as a string.
Using createStringObjectFromLongLong function might return robj that
points to an integer and so break a module that tries for example to
use RedisModule_StringPtrLen on the given field/value.

The PR introduces a fix that uses the createObject function and sdsfromlonglong function.
Using those function promise that the field and value pass to the to the
scan callback will be Strings.

The PR also changes the Scan test module to use RedisModule_StringPtrLen
to catch the issue. without this, the issue is hidden because
RedisModule_ReplyWithString knows to handle integer encoding of the
given robj (RedisModuleString).

The PR also introduces a new test to verify the issue is solved.

(cherry picked from commit e37c16e42551a3a5c61e1f8a90cfc672d3e010e4)
2020-07-20 21:08:26 +03:00
John Sully
6158cdfd29 Prevent deadlock in RM_ThreadSafeContextLock() when triggered as part of a module callback in a server thread
Former-commit-id: e01312642be3cc78e7b383dee958a9b5c0ffc103
2020-07-12 18:17:53 +00:00
Oran Agra
31bd963557 32bit CI needs to build modules correctly 2020-05-28 10:09:51 +02:00
Guy Benoish
f1ed8d93a0 Fix memory corruption in moduleHandleBlockedClients
By using a "circular BRPOPLPUSH"-like scenario it was
possible the get the same client on db->blocking_keys
twice (See comment in moduleTryServeClientBlockedOnKey)

The fix was actually already implememnted in
moduleTryServeClientBlockedOnKey but it had a bug:
the funxction should return 0 or 1 (not OK or ERR)

Other changes:
1. Added two commands to blockonkeys.c test module (To
   reproduce the case described above)
2. Simplify blockonkeys.c in order to make testing easier
3. cast raxSize() to avoid warning with format spec
2020-04-07 16:52:03 +02:00
antirez
6354b71663 Modify the propagate unit test to show more cases. 2020-03-31 16:57:20 +02:00
antirez
514ca204bb Fix module commands propagation double MULTI bug.
b512cb40 introduced automatic wrapping of MULTI/EXEC for the
alsoPropagate API. However this collides with the built-in mechanism
already present in module.c. To avoid complex changes near Redis 6 GA
this commit introduces the ability to exclude call() MUTLI/EXEC wrapping
for also propagate in order to continue to use the old code paths in
module.c.
2020-03-31 16:57:20 +02:00
Oran Agra
5daed7baee fix race in module api test for fork
in some cases we were trying to kill the fork before it got created
2020-02-27 18:02:30 +01:00
Guy Benoish
5a15c0edf9 Fix memory leak in test_ld_conv 2020-02-27 18:00:47 +01:00
Guy Benoish
2ad427f862 ld2string should fail if string contains \0 in the middle
This bug affected RM_StringToLongDouble and HINCRBYFLOAT.
I added tests for both cases.

Main changes:
1. Fixed string2ld to fail if string contains \0 in the middle
2. Use string2ld in getLongDoubleFromObject - No point of
   having duplicated code here

The two changes above broke RM_SaveLongDouble/RM_LoadLongDouble
because the long double string was saved with length+1 (An innocent
mistake, but it's actually a bug - The length passed to
RM_SaveLongDouble should not include the last \0).
2020-02-04 10:23:48 +01:00
Guy Benoish
d9f508d527 Modules: Fix blocked-client-related memory leak
If a blocked module client times-out (or disconnects, unblocked
by CLIENT command, etc.) we need to call moduleUnblockClient
in order to free memory allocated by the module sub-system
and blocked-client private data

Other changes:
Made blockedonkeys.tcl tests a bit more aggressive in order
to smoke-out potential memory leaks
2020-02-04 10:23:48 +01:00
Madelyn Olson
a48e2b721a Fixed some documentation 2019-12-17 07:49:21 +00:00
Madelyn Olson
f71437d391 Added some documentation and fixed a test 2019-12-17 07:15:04 +00:00
Madelyn Olson
05b7628687 Add module APIs for custom authentication 2019-12-17 06:59:59 +00:00
Yossi Gottlieb
937a32d84c Improve RM_ModuleTypeReplaceValue() API.
With the previous API, a NULL return value was ambiguous and could
represent either an old value of NULL or an error condition. The new API
returns a status code and allows the old value to be returned
by-reference.

This commit also includes test coverage based on
tests/modules/datatype.c which did not exist at the time of the original
commit.
2019-12-12 18:50:11 +02:00
Oran Agra
7450df6150 fix crash in module short read test 2019-12-02 19:17:35 +02:00
Salvatore Sanfilippo
22c68cc3bf Merge branch 'unstable' into rm_get_server_info 2019-11-21 10:06:15 +01:00
Salvatore Sanfilippo
bf67482a88 Merge branch 'unstable' into module-long-double 2019-11-19 12:15:45 +01:00
Salvatore Sanfilippo
cdae22b9b2 Merge pull request #6557 from oranagra/rm_lru_lfu_revized
rename RN_SetLRUOrLFU -> RM_SetLRU and RN_SetLFU
2019-11-19 11:58:07 +01:00
Salvatore Sanfilippo
f339a1a64d Merge pull request #6558 from oranagra/module_testrdb_leak
fix leak in module api rdb test
2019-11-19 11:49:43 +01:00
Salvatore Sanfilippo
ed51330a12 Merge branch 'unstable' into scan_module_impl 2019-11-19 11:08:02 +01:00
Oran Agra
4c7b540ccd Add RM_ScanKey to scan hash, set, zset, changes to RM_Scan API
- Adding RM_ScanKey
- Adding tests for RM_ScanKey
- Refactoring RM_Scan API

Changes in RM_Scan
- cleanup in docs and coding convention
- Moving out of experimantal Api
- Adding ctx to scan callback
- Dont use cursor of -1 as an indication of done (can be a valid cursor)
- Set errno when returning 0 for various reasons
- Rename Cursor to ScanCursor
- Test filters key that are not strings, and opens a key if NULL
2019-11-11 16:05:55 +02:00
meir@redislabs.com
5642a1da20 Added scan implementation to module api.
The implementation expose the following new functions:
1. RedisModule_CursorCreate - allow to create a new cursor object for
keys scanning
2. RedisModule_CursorRestart - restart an existing cursor to restart the
scan
3. RedisModule_CursorDestroy - destroy an existing cursor
4. RedisModule_Scan - scan keys

The RedisModule_Scan function gets a cursor object, a callback and void*
(used as user private data).
The callback will be called for each key in the database proving the key
name and the value as RedisModuleKey.
2019-11-11 16:01:41 +02:00
Oran Agra
86e1f73bd6 rename RN_SetLRUOrLFU -> RM_SetLRU and RN_SetLFU
- the API name was odd, separated to two apis one for LRU and one for LFU
- the LRU idle time was in 1 second resolution, which might be ok for RDB
  and RESTORE, but i think modules may need higher resolution
- adding tests for LFU and for handling maxmemory policy mismatch
2019-11-10 09:27:01 +02:00
Oran Agra
bd206b583a fix leak in module api rdb test
recently added more reads into that function, if a later read fails, i must
either free what's already allocated, or return the pointer so that the free
callback will release it.
2019-11-10 09:21:19 +02:00
Yossi Gottlieb
a73881f818 Add ModuleDataType to/from string serialization.
Add two new functions that leverage the RedisModuleDataType mechanism
for RDB serialization/deserialization and make it possible to use it
to/from arbitrary strings:

* RM_SaveDataTypeToString()
* RM_LoadDataTypeFromString()
2019-11-04 21:53:59 +02:00
Salvatore Sanfilippo
adcaa4e464 Merge pull request #6540 from oranagra/simple_module_api_tests
Test coverage for new module APIs: dbsize, flushall, randomkey, lru get/set
2019-11-04 18:52:31 +01:00
artix
29d1e22cd1 RM_CreateStringFromLongDouble: use new ld2string 'mode' type 2019-11-04 18:49:56 +01:00
artix
52a00f773b Fix RedisModule_ReplyWithLongDouble ptr definition, add tests 2019-11-04 18:49:50 +01:00
Oran Agra
9822d541d4 Test coverage for new module APIs: dbsize, flushall, randomkey, lru get/set 2019-11-04 19:30:31 +02:00
Salvatore Sanfilippo
ab520ef80f Merge pull request #6531 from oranagra/rm_save_long_double
Module API for loading and saving long double
2019-11-04 17:33:35 +01:00
Salvatore Sanfilippo
bec140f28a Merge pull request #6535 from guybe7/module_block_on_keys_tests
Modules: Test RedisModule_BlockClientOnKeys
2019-11-04 10:53:59 +01:00
Guy Benoish
4f630e931b Modules: Test RedisModule_BlockClientOnKeys 2019-11-04 14:20:24 +05:30
Oran Agra
ba1478c9f9 Add RM_ServerInfoGetFieldUnsigned
rename RM_ServerInfoGetFieldNumerical RM_ServerInfoGetFieldSigned
move string2ull to util.c
fix leak in RM_GetServerInfo when duplicate info fields exist
2019-11-04 08:50:29 +02:00
Oran Agra
8541c6a058 Adding RM_ServerInfoGetFieldC 2019-11-04 07:57:52 +02:00
Oran Agra
04df0cc328 Module API for loading and saving long double
looks like each platform implements long double differently (different bit count)
so we can't save them as binary, and we also want to avoid creating a new RDB
format version, so we save these are hex strings using "%La".

This commit includes a change in the arguments of ld2string to support this.
as well as tests for coverage and short reads.

coded by @guybe7
2019-11-03 16:42:31 +02:00