[#1259] Avoid heartbeat send errors during local connection shutdown (#1260)
Fixes #1259
Forwarding a change-time heartbeat can race with an orderly connection
close: the reader closes its session before stopping the handler, and a
sender retaining that handler then writes to the closed socket. The
heartbeat forwarding path reported every such failure as ERROR 66,
causing false replication alerts and failed convergence log checks.
`ReplicationServerDomain` now reports the send failure at ERROR only
when the handler is not shutting down and this side has not initiated
the session close. Both flags are checked after the send fails and
before cleanup. Exception tracing and `stopServer(handler, false)`
remain unconditional; failures on an otherwise open handler retain ERROR
66.
The reader remains responsible for reporting why the connection ended.
With #1254 in the base, an unexpected EOF from a V4+ peer is reported as
a bad disconnection (ERROR 181 for an RS, 211 for a DS), unless the
handler is shutting down or this side has initiated the close. EOF from
a V3 peer remains at NOTICE 53, because those peers do not send
`StopMsg`; the reader's general `catch (Exception)` also retains NOTICE
53. Suppressing a subsequent heartbeat ERROR 66 is intentional in all of
these cases. Raising the severity of other reader failures would be a
separate change, independent of whether a heartbeat happens to race with
reader cleanup. This PR does not change reader reporting, session
behavior or convergence filters.
Eight regression cases use real TLS sessions and the real handler send
path, with domain construction and teardown mocked. They cover all four
combinations of local close and handler shutdown, a controlled close
after the sender enters `send()`, shutdown engaged after handler
selection while the local-close flag remains false, successful heartbeat
delivery, and exception tracing when ERROR 66 is suppressed. The trace
case installs its own method-scoped debug publisher and removes it in
`finally`; it checks the actual IOException thrown by the real send.
Failed sends must request cleanup; successful delivery must not. The
cleanup mock sets the shutdown flag as the real `stopServer()` does, so
the live-failure case detects a guard incorrectly moved after cleanup.
ERROR 66 records are counted by category, severity and message ID, then
checked for the expected local RS port, base DN and server ID.
Validation on JDK 21, base `a77994116f`:
- Maven reactor `-Pprecommit verify`, including Javadoc, passes all 36
selected tests: 8 heartbeat, 8 reader, 4 writer, 6 session and 10
publisher-drain cases, with no failures, errors or skips.
- Caching `shuttingDown()` before `send()` fails only
`heartbeatAfterConcurrentHandlerShutdownIsNotReported`, on one
unexpected ERROR 66; the other seven cases pass.
- Moving `traceException(e)` inside the ERROR guard, or deleting it,
fails only `suppressedHeartbeatFailureIsStillTraced`, on missing debug
output; the other seven cases pass in each run.
- The previous six-case test passes against all three mutations. The
updated test passes 8/8 against the unmodified code. Mutation sources
and classes were compiled outside reactor build outputs.
- `git diff --check` passes; independent local review found no remaining
issues.
Docker convergence was not rerun locally.
```sh
mvn -o -Pprecommit verify -pl opendj-server-legacy -am \
-Dit.test=ReplicationServerDomainHeartbeatTest,ServerReaderDisconnectTest,ServerWriterDisconnectTest,SessionTest,SessionPublisherDrainTest \
-Dfailsafe.failIfNoSpecifiedTests=false \
-Dtest=NoTestsSelected -Dsurefire.failIfNoSpecifiedTests=false
```