gvsafronov/futriix
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

Try to fix SENTINEL SIMULATE-FAILURE test by re-source init-tests before each test (#12194)

Browse Source 
This test was introduced in #12079, it works well most of the time, but
occasionally fails:
```
00:34:45> SENTINEL SIMULATE-FAILURE crash-after-election works: OK
00:34:45> SENTINEL SIMULATE-FAILURE crash-after-promotion works: FAILED: Sentinel set crash-after-promotion but did not exit
```

Don't know the reason, it may be affected by the exit of the previous
crash-after-election test. Because it doesn't really make much sense to
go deeper into it now, we re-source init-tests to get a clean environment
before each test, to try to fix this.

After applying this change, we found a new error:
```
16:39:33> SENTINEL SIMULATE-FAILURE crash-after-election works: FAILED: caught an error in the test couldn't open socket: connection refused
couldn't open socket: connection refused
```

I am guessing the sentinel triggers failover and exits before SENTINEL FAILOVER,
added a new || condition in wait_for_condition to fix it.
This commit is contained in:
Binbin 2023-05-29 18:43:26 +08:00 committed by GitHub
parent 32f45215c3
commit da8f7428fa
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 7 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
tests/sentinel/tests/05-manual.tcl
View File

@ -63,15 +63,21 @@ test "The old master eventually gets reconfigured as a slave" {
} }
foreach flag {crash-after-election crash-after-promotion} { foreach flag {crash-after-election crash-after-promotion} {
# Before each SIMULATE-FAILURE test, re-source init-tests to get a clean environment
source "../tests/includes/init-tests.tcl"
test "SENTINEL SIMULATE-FAILURE $flag works" { test "SENTINEL SIMULATE-FAILURE $flag works" {
assert_equal {OK} [S 0 SENTINEL SIMULATE-FAILURE $flag] assert_equal {OK} [S 0 SENTINEL SIMULATE-FAILURE $flag]
# Trigger a failover, failover will trigger leader election, replica promotion # Trigger a failover, failover will trigger leader election, replica promotion
# Sentinel may enter failover and exit before the command, catch it and allow it
wait_for_condition 300 50 { wait_for_condition 300 50 {
[catch {S 0 SENTINEL FAILOVER mymaster}] == 0 [catch {S 0 SENTINEL FAILOVER mymaster}] == 0
||
([catch {S 0 SENTINEL FAILOVER mymaster} reply] == 1 &&
[string match {*couldn't open socket: connection refused*} $reply])
} else { } else {
catch {S 0 SENTINEL FAILOVER mymaster} reply catch {S 0 SENTINEL FAILOVER mymaster} reply
puts [S 0 SENTINEL REPLICAS mymaster]
fail "Sentinel manual failover did not work, got: $reply" fail "Sentinel manual failover did not work, got: $reply"
} }