mirror of https://github.com/OpenIdentityPlatform/OpenDJ.git

Valery Kharseko
10 hours ago c6650aedac5c1ade8aee9e7a76eb0dd100a3e179
refs
author Valery Kharseko <vharseko@3a-systems.ru>
Monday, November 2, 2026 16:39 +0100
committer GitHub <noreply@github.com>
Monday, November 2, 2026 16:39 +0100
commitc6650aedac5c1ade8aee9e7a76eb0dd100a3e179
tree ed8329794e70ff9e9b393ceae5371e924d274c40 tree | zip | gz
parent fdebe46a43ccc098df06a06210dbd18ad647b755 view | diff
[#1246] Keep the messages of the last local changes, and have a recovery publish the changes done before it began from them, as they were done (#1251)

Fixes #1246

## Problem and behavior

When a directory server connects to a replication server that is behind
it, rebuilding missing changes from entry history cannot reproduce every
original operation. A later modify can replace the historical record of
an earlier value, and deleting an entry removes the history of its add.
Once the replication server accepts a newer CSN, its ordered changelog
cannot accept those missing older changes.

The directory server now keeps a bounded buffer of recent local change
messages. Recovery publishes covered changes as they were originally
made, including changes made before recovery began. Older ranges still
fall back to historical reconstruction.

## Implementation

- `PendingChanges` retains local messages whether the broker accepted
them, refused them, or left them to recovery. Repeated attempts to push
the same change do not count it twice.
- Each written domain keeps at most 100,000 messages and the smaller of
64 MiB or one sixteenth of the maximum JVM heap. The memory estimate is
`2 * message.size() + 560`; it is an estimate, not an exact heap
measurement. A recovery snapshot can temporarily retain approximately
another buffer's worth of messages. Multiple written domains each have
their own limit.
- A recovery uses recent messages where coverage is known and
reconstructs older ranges from entry history and retained deletes.
WARNING 335 is emitted once per recovery when a reconstructed round
finds changes or retained deletes. It is not emitted solely because the
recovery starts before the buffer's coverage boundary.
- The boundary before a message with sequence number zero is the
preceding millisecond with `Integer.MAX_VALUE`, preserving the
historical index's ordering.
- Assured messages are republished as non-assured copies. The original
messages remain unchanged for operations still awaiting acknowledgment.

## Data replacement and failed import cleanup

Reloading data state forgets recent messages **before** reading the new
state. It also marks every pending local operation, including operations
not yet marked committed. When such an operation reaches the head of the
queue, its message is not added to the recent buffer; the gap
invalidates coverage and sends that range back to historical
reconstruction. Aborted operations remove their markers.

A total update forgets recent messages before cleanup and again when
loading state after enabling the backend. The second forget can also
mark operations on the new data, conservatively reducing coverage. This
behavior is documented at the call site.

A failed cleanup must not leave a live replication domain using the
replaced data's old state:

- Before replacement, the import waits for replay and an in-flight
server-state save to finish, with a bounded wait. A timeout abandons the
import with `UNAVAILABLE` / error 337 and logs that diagnostic at ERROR
level. An interrupted preparation logs a distinct error 339 and
preserves the interrupt flag. Neither is wrapped in a generic stack
trace or reported as a shutdown-only replay warning. Saves resume on an
abandoned preparation; after replacement, they resume only after
successful cleanup and state loading.
- Cleanup tracks whether this import actually acquired the backend lock.
A failed acquisition does not release another owner's lock.
- State and generation-ID loading after total update treat read failures
as errors rather than silently using a stale fallback. Failure to save a
recomputed generation ID is also reported, except when the suffix has no
base entry.
- A cleanup failure before data replacement reports the failure without
quarantining intact data. Availability is checked by the import's exact
backend ID, so a nested suffix cannot mistake its parent backend for its
own. If the original backend remains available, the domain stays usable.
Otherwise, the domain stops reversibly by aborting the session without
an offline-CSN announcement or a wait for send credit, including with
`connection-timeout: 0`.
- The domain retains the ID of the backend it is waiting for. After
import completion, the state-checkpointer thread rechecks availability
and recovers an enable event ignored during cleanup. Backend callbacks
and this recheck serialize through the service lock; callbacks filter
foreign backends before taking that lock and cannot treat a parent
backend as the missing child. An explicit domain disable clears the
pending recovery. The old listener is joined before a replacement
session starts.
- Pre-finalization callbacks use the same backend identity check as
initialization callbacks. An unavailable top-level backend does not
cause an unrelated backend's removal to throw, and disabling then
enabling a parent backend preserves the awaited child ID. Unexpected
automatic-restoration failures are reported as ERROR 340 with the
domain, backend ID and cause.
- Error 338 names the stopped domain and backend, explains that the
original data was not replaced, and tells the operator to correct the
backend failure and disable then enable the backend, or restart the
server process. The disable/enable sequence also covers an apply-phase
failure that left the configuration marked enabled while the backend was
not opened.
- A cleanup failure after replacement stops the broker permanently,
without an offline-CSN announcement or automatic import retry. It
records an operator-facing error and blocks writes overlapping the
affected suffix until the server process is restarted. An in-core
restart, including a restart task or `stop-ds --restart` with connection
options, does not clear this barrier. Neither does re-enabling the
backend, recreating the replication domain, or removing the
synchronization provider. Rename checks both source and destination.
- Import completion consumes replay restart requests already satisfied
by its reconnect, including when preparation was abandoned. Pending
remote changes are preserved when the original backend remains usable;
they are cleared after replacement or when an unavailable backend
requires stopping the domain.

After a cleanup failure following replacement, the operator must correct
the backend failure before restarting the server process. A partial
import may require another full initialization. Successful imports
resume normal state saving and replication.

The exporter also bounds its end-of-initialization wait for disconnected
targets. Each target gets its own ten-second grace, measured with a
monotonic clock and reset when it reconnects. A target that stays absent
is recorded as failed, allowing the exporter to release its
import/export context. Other targets still in full update do not prevent
tracking absent peers. The ten seconds remain an explicit absence bound,
not a guarantee covering every successful reconnect: the importer's
connection timeout and replication-server list are not available in the
topology and can differ from the exporter's. A sufficiently slow
successful reconnect can therefore still be reported as failed.

## Limits

The recent-message buffer does not provide durable history across
directory-server restarts, and cannot guarantee original operations
outside its coverage. WARNING 335 can fire for a retained delete even
when no earlier operation was lost; an overwritten value that leaves no
trace may still produce no warning. The pending-at-reload markers
protect the new recent-message buffer; they do not redesign the existing
live-publication or recovery queues.

The reversible stop does not force a final ServerState save. Re-enabling
can redeliver changes since the last checkpoint; the unchanged data and
its historical metadata absorb those repeats.

The reversible stop after aborted import preparation deliberately sends
no `ReplicaOfflineMsg`: the listener must not wait for send credit or a
blocked socket write. If the replica participates in the external
changelog, its last known CSN can hold back new change-number assignment
in `cn=changelog` until it reconnects and advances that CSN. The
consistency point is shared across the replication server's ECL-enabled
domains, so other suffixes can also be affected. This has the same
effect on change numbering as an abruptly disconnected directory server;
it does not stop ordinary replication among the remaining servers. A
free send permit alone would not make offline publication non-blocking.

This PR does not change replication-server selection or load balancing.

## Validation

JDK 21 offline Maven reactor: `mvn -o -Pprecommit verify -pl
opendj-server-legacy -am` with the nine integration-test classes below.
The final run passed **194/194 selected cases**, with zero failures,
errors or skips; the reactor, including its upstream-module tests and
Javadoc generation, completed successfully. `git diff --check` passed.

| Test class | Passed / run |
| --- | --- |
| ConfigChangeDuringImportTest | 4 / 4 |
| DisabledDomainServerStateTest | 2 / 2 |
| HistoricalCsnOrderingTest | 70 / 70 |
| PendingChangesTest | 19 / 19 |
| PersistentServerStateTest | 12 / 12 |
| ReplayDuringImportTest | 28 / 28 |
| ReplicationBrokerStopTest | 15 / 15 |
| ServerStateFlushTest | 4 / 4 |
| ReplicationDomainTest | 40 / 40 |

- Before the callback fix, the two extended failed-preparation cases
failed as expected: a foreign backend's pre-finalization threw an NPE
when the top-level backend was absent, and the parent's pre/post
callbacks revived a domain awaiting its nested child. Both now pass and
verify a subsequent successful import.
- Interruption tests cover both replay drain and an in-flight
ServerState save. Each checks fresh ERROR 339 records and completes
another import afterward. A compiled mutant deleting the save-wait
catch's re-interrupt failed the save row; the replay row still passed.
The mutation was compiled outside the working tree and was not included
in the final reactor run.
- All fifteen broker-stop cases passed in the final run, including the
heartbeat case that failed in the previous round. This run does not
establish the cause of that earlier timing failure.
- The selected suites also cover pending-at-reload markers, recovery
coverage and warning frequency, memory limits, sequence-number zero,
save suspension, backend lock ownership, failed import cleanup,
permanent write quarantine, ignored backend enables, empty send credit,
independent export absence timers and reset on reconnect.
- An independent read-only review of this round found no additional
issues.

The convergence scenario `rs534, restart-rs` from #1208 was not rerun in
this round. The completion-loop tests still use real elapsed time;
injecting both a test clock and controlled sleeps is deferred. The
optional assertion of WARNING 335's exact message arguments has not been
added; its occurrence and once-per-recovery behavior are tested.
15 files modified
2595 ■■■■■ changed files
opendj-server-legacy/src/main/java/org/opends/server/api/LocalBackend.java 35 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java 470 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/MultimasterReplication.java 10 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/PendingChanges.java 225 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/PersistentServerState.java 88 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/service/ReplicationBroker.java 81 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/replication/service/ReplicationDomain.java 73 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/workflowelement/localbackend/LocalBackendModifyDNOperation.java 2 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/main/java/org/opends/server/workflowelement/localbackend/LocalBackendWorkflowElement.java 6 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/messages/org/opends/messages/replication.properties 22 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/HistoricalCsnOrderingTest.java 466 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/PendingChangesTest.java 250 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/PersistentServerStateTest.java 55 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/ReplayDuringImportTest.java 621 ●●●●● diff | view | raw | blame | history
opendj-server-legacy/src/test/java/org/opends/server/replication/service/ReplicationDomainTest.java 191 ●●●●● diff | view | raw | blame | history