Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:
- Avoid `LOADING` response during failed and successful synchronization for cases where the
replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
to spend the extra memory usage.
General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline.
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
here anyways to avoid complications.
Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
be cases where async_loading should be checked as well and cases where it shouldn't (would require
very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
handle commands coming from the AOF client differently than ones coming from real clients, changed
to check CLIENT_ID_AOF instead.
**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
to allow modules to declare they support the diskless replication with async loading (when absent, we fall
back to disk-based loading).
Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-04 09:46:50 +01:00
|
|
|
# Check that replica keys and keys to slots map are right after failing to diskless load using SWAPDB.
|
2020-12-02 19:56:11 +08:00
|
|
|
|
2024-05-09 10:14:47 +08:00
|
|
|
start_cluster 1 1 {tags {external:skip cluster}} {
|
2020-12-02 19:56:11 +08:00
|
|
|
|
|
|
|
test "Cluster should start ok" {
|
2024-05-09 10:14:47 +08:00
|
|
|
wait_for_cluster_state ok
|
2020-12-02 19:56:11 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
test "Cluster is writable" {
|
2024-05-09 10:14:47 +08:00
|
|
|
cluster_write_test [srv 0 port]
|
2020-12-02 19:56:11 +08:00
|
|
|
}
|
|
|
|
|
Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:
- Avoid `LOADING` response during failed and successful synchronization for cases where the
replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
to spend the extra memory usage.
General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline.
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
here anyways to avoid complications.
Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
be cases where async_loading should be checked as well and cases where it shouldn't (would require
very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
handle commands coming from the AOF client differently than ones coming from real clients, changed
to check CLIENT_ID_AOF instead.
**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
to allow modules to declare they support the diskless replication with async loading (when absent, we fall
back to disk-based loading).
Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-04 09:46:50 +01:00
|
|
|
test "Main db not affected when fail to diskless load" {
|
2024-05-09 10:14:47 +08:00
|
|
|
set master [srv 0 "client"]
|
|
|
|
set replica [srv -1 "client"]
|
2020-12-02 19:56:11 +08:00
|
|
|
set master_id 0
|
2024-05-09 10:14:47 +08:00
|
|
|
set replica_id -1
|
2020-12-02 19:56:11 +08:00
|
|
|
|
|
|
|
$replica READONLY
|
|
|
|
$replica config set repl-diskless-load swapdb
|
2021-01-12 09:41:57 +02:00
|
|
|
$replica config set appendonly no
|
|
|
|
$replica config set save ""
|
2020-12-02 19:56:11 +08:00
|
|
|
$replica config rewrite
|
|
|
|
$master config set repl-backlog-size 1024
|
|
|
|
$master config set repl-diskless-sync yes
|
|
|
|
$master config set repl-diskless-sync-delay 0
|
|
|
|
$master config set rdb-key-save-delay 10000
|
|
|
|
$master config set rdbcompression no
|
|
|
|
$master config set appendonly no
|
|
|
|
$master config set save ""
|
|
|
|
|
|
|
|
# Write a key that belongs to slot 0
|
|
|
|
set slot0_key "06S"
|
|
|
|
$master set $slot0_key 1
|
2021-03-22 10:51:13 +02:00
|
|
|
wait_for_ofs_sync $master $replica
|
2020-12-02 19:56:11 +08:00
|
|
|
assert_equal {1} [$replica get $slot0_key]
|
|
|
|
assert_equal $slot0_key [$replica CLUSTER GETKEYSINSLOT 0 1]
|
|
|
|
|
2021-01-12 09:41:57 +02:00
|
|
|
# Save an RDB and kill the replica
|
|
|
|
$replica save
|
2024-05-09 10:14:47 +08:00
|
|
|
pause_process [srv $replica_id pid]
|
2020-12-02 19:56:11 +08:00
|
|
|
|
|
|
|
# Delete the key from master
|
|
|
|
$master del $slot0_key
|
|
|
|
|
|
|
|
# Replica must full sync with master when start because replication
|
|
|
|
# backlog size is very small, and dumping rdb will cost several seconds.
|
|
|
|
set num 10000
|
|
|
|
set value [string repeat A 1024]
|
2024-04-18 09:57:17 -04:00
|
|
|
set rd [valkey_deferring_client valkey $master_id]
|
2020-12-02 19:56:11 +08:00
|
|
|
for {set j 0} {$j < $num} {incr j} {
|
|
|
|
$rd set $j $value
|
|
|
|
}
|
|
|
|
for {set j 0} {$j < $num} {incr j} {
|
|
|
|
$rd read
|
|
|
|
}
|
|
|
|
|
|
|
|
# Start the replica again
|
2024-05-09 10:14:47 +08:00
|
|
|
resume_process [srv $replica_id pid]
|
|
|
|
restart_server $replica_id true false
|
|
|
|
set replica [srv -1 "client"]
|
2020-12-02 19:56:11 +08:00
|
|
|
$replica READONLY
|
|
|
|
|
Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:
- Avoid `LOADING` response during failed and successful synchronization for cases where the
replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
to spend the extra memory usage.
General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline.
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
here anyways to avoid complications.
Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
be cases where async_loading should be checked as well and cases where it shouldn't (would require
very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
handle commands coming from the AOF client differently than ones coming from real clients, changed
to check CLIENT_ID_AOF instead.
**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
to allow modules to declare they support the diskless replication with async loading (when absent, we fall
back to disk-based loading).
Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-04 09:46:50 +01:00
|
|
|
# Start full sync, wait till after db started loading in background
|
2020-12-02 19:56:11 +08:00
|
|
|
wait_for_condition 500 10 {
|
Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:
- Avoid `LOADING` response during failed and successful synchronization for cases where the
replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
to spend the extra memory usage.
General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline.
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
here anyways to avoid complications.
Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
be cases where async_loading should be checked as well and cases where it shouldn't (would require
very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
handle commands coming from the AOF client differently than ones coming from real clients, changed
to check CLIENT_ID_AOF instead.
**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
to allow modules to declare they support the diskless replication with async loading (when absent, we fall
back to disk-based loading).
Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-04 09:46:50 +01:00
|
|
|
[s $replica_id async_loading] eq 1
|
2020-12-02 19:56:11 +08:00
|
|
|
} else {
|
|
|
|
fail "Fail to full sync"
|
|
|
|
}
|
|
|
|
|
|
|
|
# Kill master, abort full sync
|
2024-05-09 10:14:47 +08:00
|
|
|
pause_process [srv $master_id pid]
|
2020-12-02 19:56:11 +08:00
|
|
|
|
2021-03-22 10:51:13 +02:00
|
|
|
# Start full sync, wait till the replica detects the disconnection
|
|
|
|
wait_for_condition 500 10 {
|
Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:
- Avoid `LOADING` response during failed and successful synchronization for cases where the
replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
to spend the extra memory usage.
General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline.
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
here anyways to avoid complications.
Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
be cases where async_loading should be checked as well and cases where it shouldn't (would require
very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
handle commands coming from the AOF client differently than ones coming from real clients, changed
to check CLIENT_ID_AOF instead.
**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
to allow modules to declare they support the diskless replication with async loading (when absent, we fall
back to disk-based loading).
Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-04 09:46:50 +01:00
|
|
|
[s $replica_id async_loading] eq 0
|
2021-03-22 10:51:13 +02:00
|
|
|
} else {
|
2024-05-09 10:14:47 +08:00
|
|
|
fail "Fail to stop the full sync"
|
2021-03-22 10:51:13 +02:00
|
|
|
}
|
|
|
|
|
2024-10-02 04:14:30 +08:00
|
|
|
# Since we paused the primary node earlier, the replica may enter
|
|
|
|
# cluster down due to primary node pfail. Here set allow read to
|
|
|
|
# prevent subsequent read errors.
|
|
|
|
$replica config set cluster-allow-reads-when-down yes
|
|
|
|
|
2020-12-02 19:56:11 +08:00
|
|
|
# Replica keys and keys to slots map still both are right
|
|
|
|
assert_equal {1} [$replica get $slot0_key]
|
|
|
|
assert_equal $slot0_key [$replica CLUSTER GETKEYSINSLOT 0 1]
|
2024-05-09 10:14:47 +08:00
|
|
|
|
|
|
|
resume_process [srv $master_id pid]
|
2021-01-12 09:41:57 +02:00
|
|
|
}
|
2024-05-09 10:14:47 +08:00
|
|
|
|
|
|
|
} ;# start_cluster
|