mirror of https://github.com/OpenIdentityPlatform/OpenDJ.git

Valery Kharseko
81 mins ago 5154f34503981b2f96755b1b6da9b425bb20ca7f
refs
author Valery Kharseko <vharseko@3a-systems.ru>
Thursday, November 5, 2026 15:32 +0100
committer GitHub <noreply@github.com>
Thursday, November 5, 2026 15:32 +0100
commit5154f34503981b2f96755b1b6da9b425bb20ca7f
tree a1131c01e32f108d2837037108451f7895bf5f90 tree | zip | gz
parent 7367aa439db9731c364a256444f0afa1c61973a4 view | diff
[#1269] Preserve replay order across duplicate replication deliveries (#1270)

After a directory server reconnects, duplicate delivery could replay an
already applied ADD after its DELETE, or replace a queued ADD and move
it behind its DELETE. Either path could leave a deleted entry present on
the replica.

Preserve the accepted queued delivery and reject changes already covered
by replay progress. Remote-origin duplicates use the applied
ServerState. Own-origin catch-up uses a separate watermark initialized
from backend state and advanced by committed replay, so a newer local
publication cannot hide an older own-origin change already delivered
after restore. Reloading backend state resets that watermark.

Keep uncommitted changes in the dependency index from admission through
commit, including failed queue hand-offs. This prevents a DELETE from
overtaking the retry of a cancelled ADD. Each delivery has its own
cancellation token. Enqueue, dequeue and claim failures request
redelivery without reacquiring the pending lock. The consumer retains
the queue head before removing it, covering a failure inside `poll()`
after unlinking as well as a failure before claim. Failed dependency
registration cancels and requests recovery before attempting rollback,
so another allocation failure during cleanup cannot skip that request.

A normal listener shutdown cancels an unqueued delivery without
requesting an extra restart while that thread remains the registered
listener: a completed join already precedes the next session's
redelivery. Cancellation occurs before comparing listener identity. If
an interrupted join has detached the old listener, its late cancellation
requests a fresh restart, including when a newer session has already
refused the still-live delivery. Exceptional hand-off failures always
request recovery. Cancellation affects only its original delivery.

Queued duplicates increment the processed-delivery counter without
returning early window credit or a SAFE_READ ack. The accepted copy
retains its queue position, takes the latest redelivery's assured
settings, and completes after replay with its actual result. Applied,
committed and owned duplicates retain their existing completion
behavior. Assured-field updates affect in-memory ACK metadata; the code
documents that any cached wire encoding remains stale.

Add deterministic backend regressions for both resurrection paths and
cancellation/retry ordering, plus coverage for completion counts,
assured success/error, changing assured flags and modes, own-origin
catch-up, backend state reload, interrupted claims, completed and
interrupted listener joins, and late cancellation. Constructor-supplied
thread-index and dependency-set state inject claim, registration and
rollback failures without replacing private final fields. Retry tests
hold recovery to prevent the checkpointer from restarting the session
during their controlled retry. A real replay consumer test verifies
notification from an empty queue, prompt draining of its backlog, and
committed replay before shutdown. Two naming-conflict fixtures use
different origins for changes deliberately replayed in reverse CSN
order, preserving their conflict-resolution coverage.

Fixes #1269.

## Validation

- Backend regressions reproduced both original resurrection paths.
Further regressions reproduced early duplicate completion, suppression
of own-origin catch-up, lost queue hand-offs, DELETE overtaking a
cancelled ADD's retry, an unnecessary restart after a completed listener
join, and missed cancellation after a repeated OOME during rollback.
- Java 21.0.12.1: the complete replication/changelog selection passed
**4,155 tests**, with zero failures, errors or skips. Reactor build and
Javadoc/doclint passed.
- Focused compilation and execution passed **54/54 tests on Java 11 and
Java 21**, with zero failures or skips.
- The initial **seven targeted mutations** were detected: queued
dependency registration, claim cancellation, cancellation restart, early
completion, assured metadata, own-origin filtering and backend reload.
- A further **eight targeted mutations** were detected: assured-mode
copy, producer notification, unconditional consumer wait, cancellation
after rollback, missing registration rollback, unnecessary
normal-shutdown restart, missing detached-listener restart, and missing
exceptional-handoff restart.
- Independent read-only review of the latest change found a
consumer-test shutdown race; it was corrected before the final full run.
- After the full run, the branch was rebased without conflicts onto the
documentation-only #1268 update to `origin/master`. All six verified
source hashes remained unchanged; the full reactor result precedes this
base-only update. `git diff --check` passed after rebase.

```sh
mvn -o -Pprecommit verify -pl opendj-server-legacy -am \
'-Dit.test=**/replication/**/*Test*,ChangelogBackendTestCase' \
-Dfailsafe.failIfNoSpecifiedTests=false -DfailIfNoTests=false
```

## Limits

- Fault injection covers the failure boundaries without exhausting the
JVM heap. The listener-join test explicitly interrupts a controlled stop
thread using the real `disableService()`/`join()` path and a supplied
listener-slot update; it does not establish how an external production
interrupt reaches that path or reproduce a full RS reconnect with a
saturated shared queue.
- The pre-existing early completion of owned duplicates and RS-side
suppression of own changes not yet sent before a newer local publication
are unchanged.
- The Docker convergence scenario from #1244 and a throughput benchmark
were not run locally.
8 files modified
1 files added
1459 ■■■■■ changed files
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java 87 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/PendingChange.java 52 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/RemotePendingChanges.java 186 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/ReplayThread.java 62 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/UpdateToReplay.java 37 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/service/ReplicationDomain.java 6 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/DuplicateReplayTest.java 746 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/NamingConflictTest.java 11 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/RemotePendingChangesTest.java 272 ●●●●● diff | view | raw | blame | history