[#1198] Take the own state of a replication server again after each answer of its peers when it gathers the monitoring data (#1205)
Fixes #1198
## Summary
Under a steady write load a directory server left its replication server
for a peer which only had its changes from it (#1198) - the switch which
closed the session of #1195 with changes still on their way. The states
it compares were taken at different moments by the replication server
that sends the monitoring message: its own state first, before it asked
its peers, the state of each peer later, in the peer's answer. The
replication server now takes its own state again after each answer of a
peer is stored, so it is never shown behind a peer on the changes of its
directly connected directory servers.
## Root cause
- A directory server checks on every second monitoring message whether
its replication server still holds its newest change, and leaves it when
another replication server holds the change and it does not
(`ReplicationBroker.receive()`, `filterServersWithAllLocalDSChanges()`).
- The states of the replication servers in that message come from
`ReplicationDomainMonitor.recomputeMonitorData()`. It took the state of
the replication server itself - together with the states of its directly
connected directory servers - in `initializePendingMonitorData()`,
before it sent the monitor requests. The state of each peer is the one
the peer put in its answer, after it got the request.
- The changes of a directory server reach the peers only through the
replication server it is connected to, and that one stores a change
before it forwards it (`ReplicationServerDomain.put()`:
`publishUpdateMsg()`, then `addUpdate()` for each peer). So at any
moment a peer holds no change of that directory server which its
replication server does not hold. A peer can only look ahead when its
state was taken later - which under load it always was.
- In the run of #1195: RS(3637) on `cv-c0` at or past seqnum 84376 of
DS(20750), next to RS(13750) at 84374, while RS(13750) is the only way
the changes of DS(20750) reach RS(3637).
## Change
`ReplicationDomainMonitor`:
- `initializePendingMonitorData()` runs once when the computing starts,
as before, and again each time the answer of a peer is stored
(`receiveMonitorDataResponse()`, under the lock the answers are stored
under). The state of the replication server itself is then never older
than an answer, and never later than the last one.
- It is not taken once the wait for the answers is over. A peer which
does not answer would put the whole wait - up to 5 seconds - between the
own state and the answers of the others. Those peers would then be shown
missing the changes which came in meanwhile, and the local directory
servers missing the changes of each other.
- The block which publishes the computed data also takes it out of the
reach of the answers (`pendingMonitorData = null`). Before, an answer
coming late, between that block and the `finally` which cleared the
pending data, went into the published data with a state taken after the
own one.
A peer which really holds changes this replication server lacks -
published to it by the directory server on an earlier session - is still
reported ahead.
## Why the directory server side is left as it is
The directions in #1198 were on the directory server side: judging the
replication server by the changes published when the states were taken,
or only after several checks. Once the states are taken consistently,
comparing them with the live state of the directory server no longer
makes it leave its replication server. When the changes still on their
way are missing on its replication server, they are missing on every
peer too. The criterion then finds no replication server with the newest
change and keeps them all (`RSEvaluations.keepBest()` keeps the list
when nothing was accepted). The states the directory server keeps per
replication server are merged with a max and survive topology messages
(`ReplicationServerInfo.update()`, `Topology.computeRSInfos()`), so that
holds across messages as well.
## Not covered
- When a directory server connects, the phase-one handshake asks the
replication servers for their states one after the other
(`collectReplicationServersInfo()`). On the tail of a closed session, a
later answer could in theory be ahead in the same way. This was not seen
in the runs and this change does not touch it.
- An older replication server in a mixed topology still sends the states
as before.
## Tests
`ReplicationDomainMonitorTest` (new) gathers the monitor data of a
mocked domain whose peers answer from reader threads of their own:
| Case | What it pins |
|---|---|
| `aPeerDoesNotLookAheadOnTheChangesItGetsFromThisReplicationServer` | A
change comes in once the requests are sent and the requester waits, and
the peer answers with it: the own state is not older than the peer's.
Red on master, green with the change. |
| `aPeerWhichHoldsChangesThisReplicationServerLacksIsReportedAhead` | A
peer really ahead is still reported ahead. |
| `aPeerWhichAnswersAfterTheWaitLeavesTheOthersUpToDate` | One peer
answers at once and another only after the wait, with changes coming in
meanwhile: the first one shows no missing changes, and the late answer
stays out of the published data. Takes the 5 seconds of the wait. |
Mutants of `ReplicationDomainMonitor`, each run against the class:
| Mutant | Caught by |
|---|---|
| master | `aPeerDoesNotLookAhead…` |
| no new snapshot after an answer | `aPeerDoesNotLookAhead…` |
| one new snapshot right after the requests are sent |
`aPeerDoesNotLookAhead…` |
| one new snapshot once the wait is over |
`aPeerWhichAnswersAfterTheWait…` |
| own state reported as the max of its and of the peers' |
`aPeerWhichHoldsChanges…` |
| published data not taken out of the reach of the answers | not caught:
the late answer cannot be put between the two blocks without a hook, so
the change closes the gap by construction |
## Verification
- `org/opends/server/replication/**` in the reactor (`-pl
opendj-server-legacy -am`): 3835 tests, 0 failures, 0 errors, 0 skipped.
