[#1206] Take a late handler of a replication server back to its queue only once its consumer has every change the queue lacks (#1219)

Fixes #1206
Fixes #1224
Fixes #1225

## Problem

A replication server stopped sending a directory server the changes of
another replica, and went on minutes further. The changes in between
were in the changelog of every replication server, never reached that
directory server, and `missing-changes` still reached 0.

A handler which is behind reads the changes for its consumer from the
changelog. `MessageHandler.getNextMessage()` takes it back to its
in-memory queue once the queue holds the first change of what it read,
which is the oldest change the consumer lacks. Holding that change does
not mean holding the ones after it:

- The queue is ordered by CSN, and drops its oldest changes when it is
full.
- A change which reaches the queue late with an older CSN goes in front:
one relayed by another replication server, or published again by a
recovery.
- Behind it, the queue can lack newer changes of another replica: the
ones it dropped before that change came, or the ones which reached the
domain before the handler was registered.

That change is then the oldest one the consumer lacks, and it is in the
queue. The handler went back to the queue, the consumer was given the
changes the queue still held of the other replica, and its state moved
past the ones in between. The changelog is read again only for a change
the state of the consumer does not cover (#963), so they were never
sent.

A full queue keeps such a change because it is bounded in bytes as well
and drops whole changes: the room a large change leaves takes a small
one in.

The changelog is read without the lock of the queue, so a change
arriving in between is lost the same way:

- with an older CSN than the first change read, it was dropped along
with the changes before that one, which the consumer has;
- into a full queue as its oldest change, it was dropped at once, and a
handler whose read gave nothing went back to the queue all the same.

This fits what #1206 found:

- The replica skipped was connected to the same replication server as
the directory server. The replicas whose changes came through the other
replication server were not skipped.
- Both jumps of the description landed on the last 10 to 13 s of the
skipped replica, which is what the queue held of it.
- The new handlers of cases 2 and 3 of the comment skipped from the
moment they connected: their queues had nothing from before.

#1225 is the same with a hole of one change, in the `combined` topology
of the convergence check. The replication server of `cv-c2` never sent
its directory server one delete of `cv-c1`, though it sent the changes
of `cv-c1` on both sides of it and every add and delete of the other
replicas. That is test 1 below: the queue dropped the change, and a
change of another replica with an older CSN came in after it.

## Change

The handler records, for each replica, the newest change of it the queue
may lack:

- what the domain held when it handed the queue its first change: the
domain hands a registered handler every change it receives which is
meant for its consumer, and writes a change to the changelog before it
hands it over;
- every change the queue drops when it is full.

It goes back to the queue only once the state of the consumer covers
them, and once the queue holds nothing before the first change read that
the consumer lacks (`MsgQueue.holdsAChangeBefore()`). It reads the
changelog until then. All three ways back to the queue check what the
queue lacks: the one which finds the first change read in the queue, the
one taken when the read gave nothing, and the one taken when both queues
are empty and nothing is found after the state of the consumer.

- Only the changes a read of the changelog can still give count, for
what the queue lacks and for what it holds before the first change read.
A change the changelog no longer holds, as one of a changelog cleared
for a new generation id, is left out once the consumer has every change
of its replica the changelog holds. Otherwise a read which gives nothing
but replica offline messages would be made again and again, with no
wait.
- A replica offline message in the queue is left out as well: no state
of a consumer ever holds its CSN.
- When the queue holds every change after the state of the consumer,
nothing changes: what it lacks is older than the oldest change the
consumer lacks.
- For the handler of a peer replication server, what the domain held
includes changes of replicas connected to other replication servers,
which the domain never hands it. The handler reads those the state of
the peer lacks from the changelog once, up to what the domain held then,
before it goes back to its queue. The peer gets them from the
replication servers they come from as well. Its changelog keeps one copy
of each, and with #1224 fixed below, its cookie based persistent
searches are not told of them twice either.
- Changes the domain does not hand a handler on purpose, to a directory
server with another generation id or being initialized, are not
recorded. Such a directory server ends that status only with a new
session (`StatusMachine` lets neither status go to normal): after its
import (`ReplicationDomain`, `broker.reStart(false)`), or once the
changelog is cleared for its generation id, when the replication server
closes its session (`DataServerHandler.changeStatusForResetGenId()`).

### Cookie based persistent searches (#1224)

`FileChangelogDB.publishUpdateMsg()` notified the cookie based
persistent searches of `cn=changelog` of every change it was given, also
of one its replica DB did not take: `Log.append()` drops a record whose
key is not after the newest one, which is what a change the changelog
already holds is. A change a replication server receives twice reached
those searches twice. `LogFile.append()`, `Log.append()` and
`FileReplicaDB.add()` now tell whether the record was appended, and
`publishUpdateMsg()` notifies the persistent searches only of a change
the replica DB took.

Not addressed here, from the comment on #1206:

- case 4: the span all three streams skipped after the changes sent
again at the reconnection of 23:08:57;
- `RemotePendingChanges.putRemoteUpdate()` taking a change sent again
which the state of the directory server covers.

## Tests

`MessageHandlerCatchUpTest` (new) drives a handler no domain feeds from
one thread, as the writer of a session does, over the changelog of a
real replication server. A change arrives the way
`ReplicationServerDomain.put()` hands it over: written to the changelog,
then given to the queue. For the reads, the domain of the handler can
hand it a change right after it has read the changelog, or once a read
gave nothing.

| # | Test | What it pins |
|---|---|---|
| 1 | `aChangeTheFullQueueDroppedIsSentWhenAnOlderOneCameInAfterIt` |
The queue dropped a change of A, then took in a change of B with an
older CSN, in the room the drop left: the consumer is sent the dropped
change. Red on master. |
| 2 |
`aChangeWhichReachedTheDomainBeforeTheHandlerIsSentWhenAnOlderOneIsQueued`
| The queue never had the changes of A which reached the domain before
it, then got a change of B with an older CSN and a later change of A:
the consumer is sent them. Red on master. |
| 3 |
`aChangeOfAReplicaTheQueueIsNeverHandedIsSentWhenAnOlderOneIsQueued` |
The same with no later change of A. Red on master. |
| 4 | `aChangeWhichArrivedWhileTheChangelogWasReadIsSent` | A change of
B with an older CSN than the first change read arrives once the
changelog is read: the consumer is sent it. Red on master. |
| 5 | `aChangeTheFullQueueDroppedWhileTheChangelogWasReadIsSent` | A
change the full queue drops at once arrives once a read gave nothing:
the consumer is sent it. Red on master. |
| 6 | `aHandlerWhoseQueueHoldsWhatTheConsumerLacksFollowsTheQueueAgain`
| The queue of test 1, for a consumer which has the dropped change: the
handler goes back to the queue. |
| 7 | `aDroppedChangeTheChangelogNoLongerHoldsDoesNotHoldTheHandlerBack`
| The queue dropped a change the changelog no longer holds, of a replica
which writes nothing more: the handler goes back to the queue. |
| 8 | `aReplicaOfflineMessageDoesNotHoldTheHandlerBack` | A replica
offline message in the queue before the first change read does not keep
the handler from it, also once its replica has written a change the
consumer lacks. |
| 9 | `aQueuedChangeTheChangelogNoLongerHoldsDoesNotHoldTheHandlerBack`
| The queue holds a change the changelog no longer holds, and the
changelog gives nothing but a replica offline message: the handler goes
back to the queue instead of reading the changelog again and again. Red
before round 2. |
| 10 | `aDroppedReplicaOfflineMessageDoesNotHoldTheHandlerBack` | The
full queue dropped a replica offline message, for a consumer which has
every change of its replica: the handler goes back to the queue. |
| 11 | `aHandlerWhichCaughtUpFromTheChangelogFollowsTheQueueAgain` | A
handler which has been handed a change catches up from the changelog
until a read gives nothing: the change handed after that read is served
from the queue. |
| 12 |
`aHandlerWhichReadTheChangesOfAReplicaItIsNeverHandedFollowsTheQueueAgain`
| The domain held a change of a replica the queue is never handed when
it handed the queue its first change: the handler reads it from the
changelog, then goes back to the queue once a read gives nothing. Red on
master. |

For #1224:

| Test | What it pins |
|---|---|
|
`ChangelogBackendTestCase.aChangeTheChangelogAlreadyHoldsIsNotNotifiedAgainToACookieBasedPersistentSearch`
| A directory server publishes a change, the same change again, then a
newer one: a changes-only cookie based persistent search of
`cn=changelog` is notified of each once. Red on master: `[csn1, csn1,
csn2]`. |
| `LogTest.appendTellsWhetherTheRecordWasAppended` | A record whose key
is not after the newest one is not appended, whether the head log file
is rotated or not, and `append()` says so. |

Mutants of `MessageHandler` and `MsgQueue`, each run against
`MessageHandlerCatchUpTest`:

| Mutant | Caught by |
|---|---|
| the changes the queue drops are not recorded | 1, 5 |
| what the domain held when it handed the queue its first change is not
recorded | 2, 3, 12 |
| what the domain holds is recorded at every change handed to the queue
| 6, 7, 8, 10, 11, 12 |
| a change the queue dropped counts when the changelog no longer holds
it | 7, 10 |
| a change of the queue before the first change read counts when the
changelog no longer holds it | 9 |
| a change counts when the consumer has every change of its replica the
changelog holds | 10 |
| the queue before the first change read is not checked | 4 |
| a replica offline message counts as a change the consumer lacks | 8 |
| the handler never goes back to the queue on finding there the first
change read | 6, 7, 8, 9, 10 |
| the handler goes back to the queue when the read gave nothing,
whatever the queue lacks | 5 |
| a handler which has been handed a change never goes back to the queue
when a read gives nothing | 11, 12 |
| the third way back to the queue is not checked | none: no state the
handler reaches lets the check decide on that way |

Mutants of the fix of #1224, each run against its two tests:

| Mutant | Caught by |
|---|---|
| the persistent searches are notified whatever the replica DB took |
`ChangelogBackendTestCase` |
| the replica DB tells it added a change the log did not append |
`ChangelogBackendTestCase` |
| the log file tells it appended a record breaking the key ordering |
both |
| the log tells it appended a record when its head log file is not to be
rotated | both |
| the log tells it appended a record when it gave up a rotation |
`LogTest` |

## Verification

Run directly with TestNG, JDK 21, macOS:

- `MessageHandlerCatchUpTest`: 12 tests, 0 failures. On master 6 of them
fail (1-5, 12); at the head of round 1, `ad3e21ad6d`, 1 fails (9).
- The two tests of #1224: 0 failures. The one of
`ChangelogBackendTestCase` fails on master.

In the reactor (`-Pprecommit verify -pl opendj-server-legacy -am`,
javadoc included), JDK 27, macOS, `org/opends/server/replication/**` and
`ChangelogBackendTestCase`: 3879 tests, 0 failures, 0 errors, 0 skipped.
Among them is `MissedUpdateRecoveryTest` (#963), which drives the same
handler through real sessions: 8 tests, 0 failures.