[#1090] Check replication convergence under load next to the Docker tests, and run the Docker tests of both images after their builds (#1192)

Fixes #1090 with the base variant (points 1, 2, 4, 5 and 6 of the
proposal). The disturbances of point 3 follow one at a time in later
PRs, under #1208.

## What it does

`.github/scripts/replication/run.sh <image>` writes to every directory
server of a replication topology at once, then checks that the replicas
hold the same data. It runs in CI and on a laptop with Docker alone;
every setting comes from the environment (`lib.sh`).

1. **Topology** (`topology-up.sh`): `TOPOLOGY=rs534` - four directory
servers without a replication server and two replication servers without
data, the topology of #534 - or `TOPOLOGY=combined` - four combined
servers. Every server joins through the background join of the image
(`OPENDJ_REPLICATION_TYPE=simple`) in its `REPLICATION_ROLE` (#1178),
and the run waits for every join to end, not only for every container to
be healthy: a join goes on repairing the replication server lists after
that. The run checks who runs a replication server and that every
directory server holds the seeded users.
2. **Load** (`load.sh`), on every directory server at once for
`DURATION` seconds, from containers of the same image, over plain LDAP
with eight connections per client kept open and bound once
(`--noRebind`), up to `LOAD_MOD_RATE` modifications and `LOAD_ADD_RATE`
adds or deletes a second per directory server (423 and 85), with no
container memory limit and the heap bounded (`SERVER_JAVA_ARGS` `-server
-Xmx1g`, `CLIENT_JAVA_ARGS` `-Xmx256m`): `modrate` on the shared users
from every server (concurrent modifications of one entry, #139),
`addrate` with fifo deletes under a uid prefix of each server (no naming
conflicts). The values `modrate` writes start with the name of their
server: the rate tools seed `rand` and `randstr` alike in every client,
so all of them would write the same values in the same order, a change
lost on its way could hide behind the same value written later by
another server, and a value in a diff would not tell which server made
it. Writes only: a read sends nothing to the replication servers. A rate
tool exits with 0 whatever it got back, and one whose connection breaks
stops early, still with 0, leaving out the interval that would count the
error: so each client must have done work, reported no errors in its CSV
output, exited with 0, printed nothing besides its CSV output and
covered the whole load.
3. **Convergence** (`monitor.sh`): every replication server knows every
directory server of the domain and none misses a change, on three
samples in a row, the last one started more than a monitoring period and
5 s after the first one ended (60 s: a replication server computes
`missing-changes` once a period, waiting up to 5 s for its peers,
answers every monitor search from that computation, and a server its
last computation does not know reads as 0) - a missing monitor entry is
not a zero. A sampler records `missing-changes` and `approximate-delay`
for the peaks, asking every replication server at once (each ask is an
`ldapsearch`, a JVM that takes tens of seconds to start on a busy
runner); stopping the sampler removes what the sample it cuts short
leaves behind.
4. **Checks on the data** (`assert-converged.sh`), since
`missing-changes` reaching 0 is not enough:
- online: the same equality searches on `sn`, each under the index entry
limit, return the same DNs on every replica, and the access log of each
replica shows that they went through the index (#1021-class index drift,
which an export does not show; `(objectClass=inetOrgPerson)` over the
seeded users was a scan); no `ds-sync-conflict` entry; every replication
server connected to every other one; no `SYNC` error, TLS handshake
failure or replay give-up in any errors log after the topology came up.
A search or a read of a log that fails is an error of its own: searches
that failed alike on every replica would give the same empty answer;
- offline, servers stopped: `export-ldif --offline` of every replica
without `ds-sync-hist`/`ds-sync-state`, a guard against empty exports,
`ldifdiff` against the first replica (the differences go to `diff/`),
`verify-index`;
- the changelogs, servers stopped: `changelog-gaps.py` reads the
changelog of every replication server, streamed out by `docker cp`, and
reports the sequence numbers missing among the changes each replica made
during the load. A directory server numbers its changes one after the
other (`CSNGenerator.newCSN()`) and no operation of the load fails, so a
gap is a change that reached no replication server - which the data
shows only until a later change overwrites it, and `missing-changes`
never (#1195). A change lost after the changelog, on its way to a
directory server, leaves no gap there; the data shows it (#1206). Every
changelog must hold changes of every directory server, so that records
the script no longer reads do not pass for a changelog without gaps.
What it cannot see: a change lost at either end of a replica's changes
has no gap after or before it, and a directory server restarted during
the load would read as a lost change - its `ReplicaOfflineMsg` takes a
sequence number that only `offline.state` holds, and only the newest
one; the run restarts none.

The offline checks and the changelogs are checked after failed online
checks as well, so that a run says which changes are missing where.
5. **What is left behind** (`collect.sh`): `summary.md` and the job
summary - convergence time, how often the monitor was sampled, peak
`missing-changes` and `approximate-delay` (seconds) and the
reconnections to a replication server per directory server (msgID 62,
whatever made them: a switch the broker decided on, 188 or 213, a
connection the replication server closed or that broke; changes made
around one were lost in #1195), counted also when a run fails before its
checks (`-` for a server whose errors log cannot be read), the changes
missing from the changelogs, throughput per client, time per phase; on
failure `state.tgz` with every server's container log, `logs/`,
`config/` and `changelogDb`. `OUT` must be new or empty: the files of an
earlier run would mix into the summary.

## CI

The registry of a build job is a service of that job and is gone with
it, so the image goes over as an artifact:

```
build-maven ─┬─ build-docker         (every platform, as before; + linux/amd64 image → artifact)
             └─ build-docker-alpine  (same)
                    ├─ docker-test        matrix: default, alpine   - the test steps of both jobs
                    └─ docker-convergence matrix: rs534, combined   - this run, DURATION=1800
```

`docker-test` holds the test steps of `build-docker` and
`build-docker-alpine` once. Expanded per flavor, its steps were byte for
byte the old ones (checked with a script); since then two settings
changed on purpose: the containers of both flavors run with `512m`
(alpine had `1g`), and both benchmark artifacts are kept 5 days (3 and
90 before). The PDB vs JE benchmark stays with the default image. The
tests, Trivy and the benchmarks now run next to the convergence check
instead of after the build in the same job.

Three couplings of the jobs, each stated in a comment where it happens:

- A red `docker-convergence` row turns the Build run red, and on a push
to master `deploy.yml` then skips the snapshot deploy and the publishing
of the docs. `rs534` fails until #1206 is fixed and can fail until #1215
is (a switch that connects to the same replication server again while it
still holds the session before logs a duplicate server id), and
`combined` can fail until #1195 is.
- `docker-test` runs when either image was built (as the tests of each
image ran in its own build job before): a matrix row cannot need its own
build only, so the row of an image that was not built fails at its
download.
- The linux/amd64 images, about 250 MB each, are kept 3 days: "Re-run
failed jobs" does not build them again, so it works within those days,
and "Re-run all jobs" after them.

`docker-convergence` loads for 1800 s and gives the replicas 4500 s to
converge after the load (`QUIESCE_TIMEOUT`, 900 s by default): in run
39043600905 they fell 1307 s behind in 1800 s at 423 and 85 a second and
converged 2489 s after the load (367 s behind in 900 s at 325 and 65 a
second and 396 s in run 38944365728), and a run that does not converge
in time ends before the checks of the data. Its artifact is kept 5 days.

## Testing

This round: `QUIESCE_TIMEOUT` is 4500 s (2700 s before), and the comment
of the job names #1215 among the failures of `rs534`; actionlint 1.7.7,
no findings. The branch is on master `8fc60b90d5` again.

CI, run
[39043600905](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/39043600905)
(the round of the second review, at d3fff4f, the first run at 1800 s):
every job green but `docker-convergence (rs534)`.

- `combined` converged 34 s after the load (121 s at 900 s in run
38944365728). Its clients reached 95-100% of their ceilings.
- `rs534` converged 2489 s after the load, 211 s short of the 2700 s it
had: when the load ended, every directory server was 1307 s and about
1.87 million changes behind (367 s at most at 900 s). Its clients
reached 94-97% of their ceilings.
- `rs534` failed rightly on #1206, a fifth time: `cv-d0` never got 19
minutes of the changes of `cv-d2` (sequence numbers 295855 to 870340,
about 574,000 changes), both connected to `cv-r1` and neither
reconnecting in any domain after the topology came up. Its replay of
`cv-d2` jumped at 18:51:29, four minutes after the load, to the last 10
s of the changes of `cv-d2`. Both changelogs are complete. The data
showed it on 102 entries of `cv-d2` that only `cv-d0` kept: the
`modrate` client of `cv-d2` was the fastest of the four, and the others
modified each entry it modified after it, so no skipped modification was
the newest change of its entry.
- `cv-r0` logged one error: a switch of `cv-d1` in `cn=admin data`
(#1215; 34 switches there and 38 in `cn=schema`) connected to it again
while it still held the session before (msgID 54, a duplicate server
id). That alone fails the online checks.
- The checks of the round of the second review held on the run: both
`sn` searches went through the index on every replica, and the
convergence was taken 20 s after the last change replayed. `state.tgz`
came out whole at 627 MB.

The round of the second review of maximthomas: each check it changes was
run on stubs, against the scripts before the round:

| case | before | now |
|---|---|---|
| the access logs of run 38944365728, the searches of the online checks
| no such check | green: both `sn` searches went through the index |
| the same logs, `(objectClass=inetOrgPerson)` among the filters | no
such check, the scan passed | red on every replica: ran without an index
|
| the request of a search in a rotated access log, its unindexed result
in the current one | no such check | red |
| access logs that hold none of the searches | no such check | red:
holds no search of `(sn=AddRate)`, `(sn=Diverged)` |
| a container whose logs cannot be read | no such check | red: the
access log could not be read |
| converged on every sample, samples of 2 s | done at 63 s, the last
sample started 54 s after the first ended | done at 77 s, 68 s after |
| the same, samples of 20 s | done at 100 s, 55 s after | done at 125 s,
80 s after |
| the errors log of `cv-d2` unreadable: `reconnections.txt`, and the
summary of the ERR trap on the artifact of run 38944365728 | `cv-d2 0`,
summary 0 | `cv-d2` left out, summary - |

The other online cases - a conflict entry, answers that differ, searches
that fail, a conflict search that fails, an errors log that cannot be
read - came out as before, less the lines of
`(objectClass=inetOrgPerson)`.

The round before that: the load runs 1800 s at 423 and 85 a second (900
s at 325 and 65 before), the replicas get 2700 s to converge after it,
and the artifact of a convergence run is kept 5 days (14 before); the
branch is on master again, with the versions of the actions #1212 moved.
Run
[39032032780](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/39032032780)
at 0b7a028 was cancelled by the push of the round of the second review
in the build of the default image, after 15 green jobs
(`build-docker-alpine` among them) and before its `docker-convergence`
rows started, so no row has run at 1800 s yet.

The round of the review of maximthomas: each check it changes was run on
stubs and on the artifact of run 38893291531, against the scripts before
the round:

| case | before | now |
|---|---|---|
| a client whose output stops at 390 of 900 s (39 lines) | passes | red:
stopped after 39 of 89 intervals |
| a client that printed a connection error | passes | red |
| a client that exited with 1 | passes | red |
| the indexed searches failing alike on every replica | pass | red |
| the `ds-sync-conflict` search failing on one replica | passes | red |
| the errors log of one server that cannot be read | passes | red |
| `UPDATE_TYPES = {}` in `changelog-gaps.py`, on the changelog of
`cv-r0` | passes | red: no change of the load by `cv-d0` ... `cv-d3` |
| converged on every sample | after 3 samples, 21 s | after 9 samples,
63 s; `CONVERGED_IN` is the first of them |
| converged on 4 samples, behind on the fifth at 30 s | done after the
third | the fifth starts the series again |
| `rm` fails in the stop of the sampler | the shell exits | the run goes
on |
| a directory server without a server id | passes | red |
| a run that fails before the online checks | reconnections `-` |
counted |

What the round does not change came out the same before and after: the
client output of a whole run, the same answers everywhere, a conflict
entry, answers that differ, the whole changelog of `cv-r0` (green,
green, red, red, green). The reconnections in the errors logs of that
run, which go on to the shutdown: `cv-d2` 8 - each of its 8 switches
(#1215) connected it again to the replication server it left -, `cv-d1`
1, its reconnection while the servers stopped, the others 0. `run.sh`
with an `OUT` that holds a file exits with 2 before it touches Docker.
The wait for the end of the joins is the one of
`docker-test-replication.sh`, which runs on both images; it has not run
here (no Docker on the host). `build.yml`: actionlint 1.7.7, no
findings.

CI, run
[38944365728](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/38944365728)
(the round of the review, at e4a8431): every job green but
`docker-convergence (rs534)`; `combined` converged 121 s after the load.
`rs534` failed rightly on #1206, twice, with both changelogs complete:
`cv-d1` never got about 70 thousand changes of `cv-d3` (sequence numbers
250852 to 321547) and kept 102 entries of `cv-d3` the others had deleted
and 2733 stale values; `cv-d3`, connected again at 23:15:57, never got
about 61 thousand changes of `cv-d1` (149763 to 210787), went on from
those made when it connected, and kept 289 entries the others had
deleted. It converged 396 s after the load, 367 s behind at its end. The
clients reached 94-96% of their ceilings on `rs534`, 73-100% on
`combined`.

CI, run
[38893291531](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/38893291531)
(the sixth round: values of its own for every `modrate` client, the stop
of the sampler): `docker-test (default)` and `docker-test (alpine)`
green, `combined` green, converged 212 s after the load. `rs534` failed
rightly, again on a loss on the way to a directory server (#1206):
`cv-d2` kept 12 entries the others had deleted - 7 of `cv-d0`, 5 of
`cv-d3` - while both changelogs held every change; it converged 40 s
after the load. `cv-d2` switched replication servers 8 times (msgID
188), and each switch connected it again to the replication server it
left - the stale flag of #1215, not #1198 as this said before. Before
CI, with `start_client` stubbed, each `modrate` client got a template of
its own (`description:cv-d0-%2$s`, `description:cv-d1-%2$s`, ...), which
`String.format` - as ModRate calls it - turns into
`description:cv-d0-AbCdEfGhIjKlMnOp`; stopping the sampler in the middle
of a sample (a stubbed `admin_search` that takes 3 s) left its
`sample.*` directory behind before that round and left nothing after it.

CI, run
[38865878001](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/38865878001)
(the fifth round: every replication server asked at once, the switch
count, the changelog check): every job that finished was green but
`docker-convergence (rs534)` (the `docker-test` jobs were still running
when this round was pushed). `rs534` failed rightly, on a loss of
another kind than #1195: a replication server skipped minutes of the
changes of one directory server for another that was catching up - about
70 thousand changes of `cv-d2` never reached `cv-d3`, about 24 thousand
of `cv-d3` never reached `cv-d2` - with nothing logged and
`missing-changes` at 0 (converged 348 s after the load), now #1206. The
changelog check found no gap in the changelog of either replication
server - every change was there, the loss was on the way out - and none
in `combined`, so it raised no false alarm on a complete changelog. The
sampler took 22 samples in 900 s on `rs534` (13 in the fourth round) and
counted 10 switches of `cv-d1` (the pattern of #1215), none elsewhere.
Two things this round changes came out of it: which server made a stale
value took the changelog to tell, since every `modrate` client wrote the
same values, and a `sample.*` directory the stop of the sampler had cut
short was uploaded with the run. So `docker-convergence (rs534)` can
fail until #1206 is fixed, as `combined` until #1195.

The fifth round, before CI: `changelog-gaps.py`, the switch count and
the summary were run on the state of the failed `combined` run of #1195
(run 38605759421 attempt 1, artifact `convergence-combined`) with
`docker` stubbed: the replica of `cv-c3` misses 81 changes in all four
changelogs - the number the analysis of #1195 found independently - the
other three replicas none, also over the whole history including the
setup of the topology; `cv-c3` switched replication servers once. Asking
the replication servers at once took one ask's time for four of them. A
whole local run was not possible (no Docker on the host at the time).

CI, run
[38755406531](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/38755406531)
(the fourth round, 325 modifications and 65 adds or deletes a second for
900 s, heaps of 1 GB and 256 MB): every job green. The clients reached
92-98% of their ceilings, about 1500 writes a second over the topology.
`rs534` converged 269 s after the load: `cv-d1` and `cv-d2` fell behind
steadily, about 300 thousand changes and 320 s at the end of the load,
and caught up. `combined` converged 83 s after the load. The sampler
took 13 samples in 900 s there, one ask after the other, which this
round changes.

CI, run
[38694855431](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/38694855431)
(the third round, 325 modifications and 65 adds or deletes a second for
900 s, no bound on any heap): both `docker-convergence` runners lost
communication with GitHub in the middle of the run, with no log and no
artifact left. Fourteen JVMs that may each take a quarter of a 16 GB
runner are the likely cause, so the heaps are bounded again.
`docker-test (default)` failed in `Docker test replication` (`dj-r1 did
not join through dj-r0`, the joins on empty volumes of #1178), which
this PR does not touch.

CI, run
[38605759421](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/38605759421)
(the second round, at 250 modifications and 50 adds or deletes a second
for 300 s): the clients reached their ceilings, about 1180 writes a
second over the topology. `rs534` passed. `combined` failed in attempt
1, and rightly: two deletes `cv-c3` made while its directory server
switched replication servers reached no replication server, the other
replicas kept both entries, and `missing-changes` reached 0 all the same
- #1195. Attempt 2 of the same job passed, with peaks of
`missing-changes` up to 58078 and of `approximate-delay` up to 105 s:
the loss depends on a delete landing in the window of a switch. So
`docker-convergence (combined)` can fail until #1195 is fixed.

CI, run 38528537312 (the first round of this PR, before the load moved
to plain LDAP on open connections): every job green.
`docker-convergence` took 14 minutes for each topology, next to
`docker-test` (103 and 117 minutes). `rs534` converged 22 s after the
load, `combined` 75 s; the clients reached 10-30 operations a second
each, since each operation opened an LDAPS connection and bound first,
which the second round removed.

Locally, Docker 29.8.2, 4 CPU / 4 GB, image built from this branch:

- `combined` and `rs534` with the load for 120 s: green; `combined` for
300 s: green, converged 83 s after the load, `ldifdiff` found no
attribute that differs beyond the two excluded.
- `combined` with the load of this round (plain LDAP, open connections,
writes only) for 60 s: green, converged 158 s after the load.
- `INJECT_DIVERGENCE=1` (one change and one entry on one replica only,
through the replication repair control): red on the indexed searches and
on `ldifdiff`, whose output is exactly the injected change.
- Each guard red on its own case: an export guard demanding more entries
than there are, a planted `SYNC` error in an errors log, a stopped
replication server (the others see 2 of 3 peers), a directory server
whose monitor entry is gone (the wait times out and prints the monitor),
a load client that reaches no server or binds with a wrong password.
- The load checks once summed the client rates in the host's locale,
where awk reads `0.5` as 0 with a decimal comma; the run sets `LC_ALL=C`
(red on a saved client output before, green after).
- `build.yml`: actionlint 1.7.7, no findings.

Not done: `rs534` with the load for 300 s locally. The first run failed
on a load client that printed nothing (the run now records each client's
exit code and OOM kill); the second was stopped for lack of memory on
the host. On 4 CPU / 4 GB with the host itself busy, the servers get a
few operations a second from each client, so the local runs check the
logic, not the throughput. Once, locally, a combined server did not turn
healthy within an hour while the topology came up; its logs were lost to
the cleanup, which `KEEP=1` and `state.tgz` now prevent.