mirror of https://github.com/OpenIdentityPlatform/OpenDJ.git

Valery Kharseko
4 hours ago f251af3f68a38e2f83154ed80be92667326d286c
refs
author Valery Kharseko <vharseko@3a-systems.ru>
Saturday, November 7, 2026 12:48 +0100
committer GitHub <noreply@github.com>
Saturday, November 7, 2026 12:48 +0100
commitf251af3f68a38e2f83154ed80be92667326d286c
tree 7515ea64c462d0cba697850cce920df860324347 tree | zip | gz
parent d9d626c0d2d932b664fd7893673af83e96cd68e9 view | diff
[#1208] Test directory and replication server crashes and restarts (#1244)

Part of #1208: add independent replication-server kill/restart runs and
directory-server crash runs to the replication convergence matrix, and
scope expected DNS failures to the disturbed replication server. Other
topology disturbances remain separate work, so this PR does not close
#1208.

Each matrix row starts a fresh topology and performs at most one
disturbance:

| Topology | Disturbances |
|---|---|
| `rs534` | `none`, `restart-ds`, `kill-ds`, `kill-rs`, `restart-rs` |
| `combined` | `none`, `restart-ds`, `kill-ds` |

`kill-rs` kills `cv-r0`, waits for its container to exit, and starts it
again. `restart-rs` gracefully restarts that server with `docker restart
-t 120`. In both cases, `cv-r1` stays available and every DS continues
writing. Recovery requires the startup message and connections to every
RS peer. These independent scenarios replace the earlier sequential
disturbance and its catch-up barrier; they do not establish that
sequential failover during recovery is safe.

`kill-ds` crashes the same second DS as `restart-ds`: `cv-d1` in
`rs534`, or `cv-c1` in `combined`. Its clients finish before the
disturbance and resume after startup on the existing volume, while the
other DS continue writing. In `combined`, the container also hosts an
RS. The load-budget checks require at least 30 seconds of useful load to
remain before the RS operation and after recovery; DS clients also
require at least 30 seconds to resume writing. Bootstrap completion is
required before the final convergence wait.

All eight rows use 900 seconds of load, with disturbances scheduled at
300 seconds, the existing load rates and 10,000 seeded users. Reducing
the former 1,800-second runs lowers CI time and sustained-overload
coverage; this change adds no scheduled long run. `QUIESCE_TIMEOUT`
remains 4,500 seconds, and successful runs stop waiting once stable
convergence is confirmed. Every row has its own artifact name.

Error windows close 10 seconds after recovery. In addition to the
existing handshake allowances, the scenarios permit:

| Scenario | Additional expected records inside its window |
|---|---|
| `rs534 / restart-ds` | None |
| `combined / restart-ds` | ERROR 51 for the exact disturbed hostname |
| `rs534 / kill-ds` | ERROR 211 |
| `combined / kill-ds` | ERROR 51 for the exact disturbed hostname; 180,
181, 211, 284 |
| `kill-rs` | ERROR 51 for the exact disturbed hostname; 180, 181, 284 |
| `restart-rs` | ERROR 51 for the exact disturbed hostname; 180 |

The DNS exception requires SYNC ERROR 51, its complete
resolution-failure message and the exact `dns:<hostname>` token supplied
by that window. `restart-ds` supplies the token only when the disturbed
node is also in `RS_NODES`; a DS-only restart does not receive it. Other
hostnames, altered messages and records outside the window remain
failures. Expected records are retained in `logs/<node>.expected`. The
numeric msgID allowances match records within the window without
additional peer or domain filtering. ERROR 180 is accepted during a
graceful RS restart based on observed CI receive failures, without
establishing their underlying cause. Errors 66 and 108 remain rejected.

Final indexed searches, offline LDIF comparisons, index verification,
changelog-gap checks, RS-mesh checks and replication-error checks remain
strict. Summary self-tests preserve an inherited GitHub step summary.

The latest revision also covers the previously deferred `combined /
restart-ds` DNS flake. In [run
42269542941](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/42269542941/job/128178403818),
`cv-c2` logged ten ERROR 51 records for `cv-c1` during its restart. The
replicas converged 531 seconds after the load; offline LDIF comparisons
and index checks passed, and every RS changelog contained all workload
origins with no sequence gaps. The generic online-check failure message
did not indicate data divergence. This evidence supports the scoped
exception; the run did not capture a DNS trace establishing the
underlying resolution failure.

Revision `076b9f1531` is based on freshly fetched `origin/master` at
`d9d626c0d2`. The local rebase had no conflicts, and `range-diff`
confirmed that all 13 preceding PR commits retained their patches.

Validation:

- All 207 shell checks pass on macOS Bash 3.2 and Linux Bash 5.2.15.
Every replication shell script passes `bash -n`, `git diff --check`
passes, and the self-tests preserve an inherited GitHub step summary on
both shells.
- The combined restart lifecycle and filter cases failed before the
production change and pass afterward. Coverage includes DS-only
restarts, exact RS membership, other hostnames, window boundaries and
rejection of disconnect/recovery/monitor errors.
- All five targeted mutations were detected: missing/wrong DNS token,
accidental DS-only allowance, substring membership and an added ERROR
180 allowance.
- Replaying the ten actual CI error records with the combined restart
token produces zero unexpected errors and preserves all ten in
`.expected`. Without the token, or outside the window, all ten remain
errors. Independent local review found no issues.

Full Docker convergence and the Java reactor were not run locally for
this shell-only revision. A fresh CI run must validate the rebased
revision end to end.
8 files modified
7 files added
830 ■■■■■ changed files
.github/scripts/replication/assert-converged.sh 83 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/collect.sh 15 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/disturb-kill-ds.sh 50 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/disturb-kill-rs.sh 28 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/disturb-restart-ds.sh 11 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/disturb-restart-rs.sh 27 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/lib.sh 3 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/load.sh 30 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/rs-disturbance.sh 71 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/selftest-disturb-kill-ds.sh 112 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/selftest-disturb-restart-ds.sh 51 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/selftest-disturb-rs.sh 147 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/selftest-replication-errors.sh 119 ●●●●● diff | view | raw | blame | history
.github/scripts/replication/selftest-summary.sh 59 ●●●●● diff | view | raw | blame | history
.github/workflows/build.yml 24 ●●●●● diff | view | raw | blame | history