[#1227] Keep the record a changelog cursor read in advance when the head log file it is positioned in is rotated (#1228)
Fixes #1227
## Problem
A replication server can skip one change of a replica for a directory
server which is catching up, and the state of that directory server then
moves past the change for good. This is the one-change loss of #1225,
which `docker-convergence (combined)` showed again in run
[39407529872](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/39407529872)
on a build with #1219. #1227 has the analysis of that run.
A cursor on a changelog log reads the record it is positioned on in
advance:
1. `LogFileCursor.positionTo()` keeps that record in `initialRecord` for
the next `next()`, and leaves the file pointer after it.
2. When the head log file is rotated, `Log.rotateHeadLogFile()` moves
every cursor open on it to the read-only file it was renamed to. It does
so from the state `InternalLogCursor.getState()` saved: the file
position and the current record, but not `initialRecord`.
3. A cursor positioned on the head log file and moved before it read
anything therefore starts after the record it is positioned on.
`FileReplicaDBCursor` takes the next one for the first change after its
start CSN.
`positionTo()` and the first `next()` take the shared lock of the log
one after the other. So a rotation which asks for the exclusive lock in
between runs before the read. Two roads go through that window:
- `MessageHandler.fillLateQueue()` makes such cursors anew for every
batch it reads.
-
`FileReplicaDBCursor.nextWhenCursorIsExhaustedOrNotCorrectlyPositionned()`
positions again each time it looks for the changes written since.
## Fix
- `Log.CursorState` keeps the record read in advance, and the state
moved to the rotated file carries it.
- `LogFile.LogFileCursor.getInitialRecord()` gives that record to the
state.
- `LogFile.getCursorInitialisedTo()` gives it back to the cursor made on
the rotated file.
## Tests
In `LogTest`, on its log which rotates after two records:
-
`aCursorReadsTheRecordItIsPositionedOnWhenTheHeadLogFileIsRotatedBeforeItsFirstRead`:
a cursor positioned on the head log file reads that record first after a
rotation. It covers every key matching and positioning strategy which
reads a record in advance, 6 cases.
-
`aCursorPositionedAgainReadsTheRecordItIsPositionedOnWhenTheHeadLogFileIsRotatedBeforeItsNextRead`:
the same for a cursor which read the newest record and is positioned
again once more were written. This is the road `FileReplicaDBCursor`
takes.
## Verification
- **Without the fix:** 7 of the 95 tests of `LogTest` fail, each one on
the skipped record (expected `key009`, `key010` or `key012`, read the
record after it).
- **With the fix:** 95 of 95 pass.
- **Mutants:** each link of the fix was broken in turn, 5 mutants:
- the state taken before the rotation drops the record;
- the state moved to the rotated file drops it;
- the cursor made from the state is not given it;
- the cursor does not keep it;
- the getter returns the current record instead.
Each one fails all 7 new cases.
- **Reactor** (`-Pprecommit verify -pl opendj-server-legacy -am`, JDK
27, with javadoc), `org/opends/server/replication/**` and
`ChangelogBackendTestCase`:
- 3967 tests, 2 failures. Both failures are the embedded server failing
to start in `setUp` of `FileReplicaDBTest` and `ComputeBestServerTest`,
on `0.0.0.0:65534: Address already in use`: another test server held the
admin port.
- Run again on their own, both classes pass (17 and 46 tests).
- Every class of `replication/server/changelog/file` passes, and
`ChangelogBackendTestCase` passes (31).