OpenDJ uses advanced data replication with automated conflict resolution to help ensure your directory services remain available in the event a server crashes or a network goes down, and also as you backup or upgrade your directory service. You can configure data replication as part of OpenDJ installation, and in many cases let replication do its work in the background.

The easiest way to set up replication for the first time involves using the setup wizard. In the Topology Options screen for the first server you set up, select This server will be part of a replication topology. If you also choose Configure as Secure, then replication traffic is protected by SSL. In the Topology Options screen for subsequent servers, also select There is already a server in the topology, providing the Host Name, Administration Connector Port number, Admin User, and Admin Password for the first replica you set up. You also set up a global administrator account, stored under cn=admin data across replicas, used to manage replication in the topology. You further set up what to replicate. Once replication is set up, it works for all the replicas. You can monitor the replication connection and status through the OpenDJ Control Panel.

Before you take replication further than setting up replication in the setup wizard, read this section to learn more about how OpenDJ replication works. Replication is the process of copying updates between OpenDJ directory servers such that all servers converge on identical copies of directory data. Replication is designed to let convergence happen over time by default. Assured replication can require, however, that the convergence happen before the client application is notified that the operation was successful. Letting convergence happen over time means that different replicas can be momentarily out of sync, but it also means that if you lose an individual server or even an entire data center, your directory service can keep on running, and then get back in sync when the servers are restarted or the network is repaired. Replication is specific to the OpenDJ directory service. Replication uses a specific protocol that replays update operations quickly, storing enough historical information about the updates to resolve most conflicts automatically. For example, if two client applications separately update a user entry to change the phone number, replication can work out which was the latest change, and apply that change across servers. The historical information needed to resolve these issues is periodically purged to avoid growing larger and larger forever. As a directory administrator, you make sure that you do not purge the historical information more often than you backup your directory data. The primary unit of replication is the suffix, specified by a base DN such as dc=example,dc=com. When you configure partial and fractional replication, however, you can replicate only part of a suffix, or only certain attributes on entries. Also, if you split your suffix across multiple backends, then you need to set up replication separately for each part of suffix in a different backend. Replication also depends on the directory schema, defined on cn=schema, and the cn=admin data suffix with administrative identities and certificates for protecting communications. Thus that content gets replicated as well. The set of replicas sharing data in a given suffix is called a replication topology. You can have more than one replication topology. For example, one topology could be devoted to dc=example,dc=com, and another to dc=example,dc=org. Directory servers are capable of serving more than one suffix. They are also capable of participating in more than one replication topology.

For some deployments you choose not to configure replication using the setup wizard. This section shows how to configure replication with command-line tools.

You can start the replication process by using the dsreplication enable command. $ dsreplication enable -I admin -w password -X -n -b dc=example,dc=com \ > --host1 `hostname` --port1 4444 --bindDN1 "cn=Directory Manager" \ > --bindPassword1 password --replicationPort1 8989 \ > --host2 `hostname` --port2 5444 --bindDN2 "cn=Directory Manager" \ > --bindPassword2 password --replicationPort2 9989 Establishing connections ..... Done. Checking registration information ..... Done. Updating remote references on server localhost:4444 ..... Done. Configuring Replication port on server localhost:5444 ..... Done. Updating replication configuration for baseDN dc=example,dc=com on server localhost:4444 ..... Done. Updating replication configuration for baseDN dc=example,dc=com on server localhost:5444 ..... Done. Updating registration configuration on server localhost:4444 ..... Done. Updating registration configuration on server localhost:5444 ..... Done. Updating replication configuration for baseDN cn=schema on server localhost:4444 ..... Done. Updating replication configuration for baseDN cn=schema on server localhost:5444 ..... Done. Initializing registration information on server localhost:5444 with the contents of server localhost:4444 ..... Done. Initializing schema on server localhost:5444 with the contents of server localhost:4444 ..... Done. Replication has been successfully enabled. Note that for replication to work you must initialize the contents of the base DN's that are being replicated (use dsreplication initialize to do so). See /var/.../opends-replication-7958637258600693490.log for a detailed log of this operation. As you see in the command output, replication is set up to function once enabled. You must however initialize replication in order to start the process, however. If you need to add another OpenDJ directory server to participate in replication, use the dsreplication enable with the new server as the second server.

Although you can enable replication before you have user data, you must initialize replication after you enable it for the first time. You can perform initialization either over the replication protocol, by importing the same LDIF data on all server before performing initialization when starting out, by importing data from LDIF that you exported from another replica when adding a server to the topology, or by restoring a backup from an existing replica onto a new server. Make sure you have enabled servers you want to participate in replication. Start replication with the dsreplication initialize-all command. $ dsreplication initialize-all -I admin -w password -X -n -b dc=example,dc=com \ > -h `hostname` -p 4444 Initializing base DN dc=example,dc=com with the contents from localhost:4444: 160 entries processed (100 % complete). Base DN initialized successfully. See /var/.../opends-replication-5020375834904394170.log for a detailed log of this operation. Follow these steps to prepare a replication topology starting from directory data in LDIF. Import the same LDIF on all servers you want to participate in replication. Make sure you have enabled servers you want to participate in replication. Start replication with the dsreplication initialize-all command. $ dsreplication initialize-all -I admin -w password -X -n -b dc=example,dc=com \ > ;-h `hostname` -p 4444 Initializing base DN dc=example,dc=com with the contents from localhost:4444: 161 entries processed (100 % complete). Base DN initialized successfully. See /var/.../opends-replication-5745536041520679254.log for a detailed log of this operation. Follow these steps to add another server to the topology by copying the database Backup the database to replica on an existing server. Install a new server. Restore the new server database from the backup archive. Enable replication on the new server. $ dsreplication enable -I admin -w password -X -n -b dc=example,dc=com \ > --host1 `hostname` --port1 4444 --bindDN1 "cn=Directory Manager" \ > --bindPassword1 password --replicationPort1 8989 \ > --host2 `hostname` --port2 6444 --bindDN2 "cn=Directory Manager" \ > --bindPassword2 password --replicationPort2 10989 Establishing connections ..... Done. Checking registration information ..... Done. Updating remote references on server localhost:4444 ..... Done. Configuring Replication port on server localhost:6444 ..... Done. Updating replication configuration for baseDN dc=example,dc=com on server localhost:4444 ..... Done. Updating replication configuration for baseDN dc=example,dc=com on server localhost:6444 ..... Done. Updating replication configuration for baseDN dc=example,dc=com on server localhost:5444 ..... Done. Updating remote references on server localhost:5444 ..... Done. Updating registration configuration on server localhost:4444 ..... Done. Updating registration configuration on server localhost:6444 ..... Done. Updating registration configuration on server localhost:5444 ..... Done. Updating replication configuration for baseDN cn=schema on server localhost:4444 ..... Done. Updating replication configuration for baseDN cn=schema on server localhost:6444 ..... Done. Updating replication configuration for baseDN cn=schema on server localhost:5444 ..... Done. Initializing registration information on server localhost:6444 with the contents of server localhost:4444 ..... Done. Replication has been successfully enabled. Note that for replication to work you must initialize the contents of the base DN's that are being replicated (use dsreplication initialize to do so). See /var/.../opends-replication-1672058070147419978.log for a detailed log of this operation. Initialize replication on the new server with the dsreplication initialize command. $ dsreplication initialize -I admin -w password -X -n -b dc=example,dc=com \ > -h `hostname` -p 6444 Initializing base DN dc=example,dc=com with the contents from localhost:6444: 161 entries processed (100 % complete). Base DN initialized successfully. See /var/.../opends-replication-4529243617755617341.log for a detailed log of this operation.

How you stop replication depends on whether the change is meant to be temporary, or meant to be permanent. If you need to stop a server from replicating temporarily, you can do so using dsconfig command. Do not update directory data on the server while replication is interrupted. Get the replication server property that identifies one of the replication service host:port combinations that you need to restart replication. $ dsconfig -p 5444 -h `hostname` -D "cn=Directory Manager" -w password \ > get-replication-server-prop --provider-name "Multimaster Synchronization" \ > --property replication-server -X Property : Value(s) -------------------:----------------------------------------------------------- replication-server : localhost:8989, : localhost:9989 Reset the replication server property to the default (no replication server) to pause replication. $ dsconfig -p 5444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-server-prop --provider-name "Multimaster Synchronization" \ > --reset replication-server -X -n Do not modify the replica for which replication is paused. When you are ready to resume replication, set the replication server property to the host:port combination of an active replication server. $ dsconfig -p 5444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-server-prop --provider-name "Multimaster Synchronization" \ > --set replication-server:localhost:8989 -X -n If you need to stop a server from replicating permanently, for example in preparation to remove a server, you can do so with the dsreplication disable command. Stop replication using the dsreplication disable command. $ dsreplication disable -a -p 5444 -h `hostname` -D "cn=Directory Manager" \ > -w password -X -n Establishing connections ..... Done. Disabling replication on base DN cn=admin data of server localhost:5444 ..... Done. Disabling replication on base DN dc=example,dc=com of server localhost:5444 ..... Done. Disabling replication on base DN cn=schema of server localhost:5444 ..... Done. Disabling replication port 9989 of server localhost:5444 ..... Done. Removing registration information ..... Done. Removing truststore information ..... Done. See /var/.../opends-replication-125248191132797765.log for a detailed log of this operation. The dsreplication disable as shown removes the replication configuration information. If you want to restart replication for the server, you use the dsreplication enable and dsreplication initialize commands again.

Replication in OpenDJ is designed to be both easy to implement in environments with a few servers, and also scalable in environments with many servers. You can enable the replication service on each OpenDJ directory server in your deployment, for example, to limit the number of servers you deploy. Yet in a large deployment, you can use stand-alone replication servers — OpenDJ servers that do nothing but relay replication messages — to configure (and troubleshoot) the replication service separately from the directory service. You only need a few stand-alone replication servers publishing changes to serve many directory servers subscribed to the changes. Furthermore, replication is designed such that you need only connect a directory server to the nearest replication server for the directory server to replicate with all others in your topology. Yet only the stand-alone replication servers participate in fully-meshed replication. This example sets up a stand-alone replication server to handle the replication traffic between two directory servers that do not handle replication themselves. Here the replication server has admin port 6444. The directory servers have admin ports 4444 and 5444. In a real deployment, you would have more replication servers to avoid a single point of failure. Setup the replication server as a directory server that has no database. Setup the directory servers as stand-alone directory servers. Enable replication with the appropriate --noReplicationServer and --onlyReplicationServer options. $ dsreplication enable -I admin -w password -X -n -b dc=example,dc=com \ > --host1 `hostname` --port1 4444 --bindDN1 "cn=Directory Manager" \ > --bindPassword1 password --noReplicationServer1 \ > --host2 `hostname` --port2 6444 --bindDN2 "cn=Directory Manager" \ > --bindPassword2 password --replicationPort2 8989 --onlyReplicationServer2 Establishing connections ..... Done. Only one replication server will be defined for the following base DN's: dc=example,dc=com It is recommended to have at least two replication servers (two changelogs) to avoid a single point of failure in the replication topology. Checking registration information ..... Done. Configuring Replication port on server localhost:6444 ..... Done. Updating replication configuration for baseDN dc=example,dc=com on server localhost:4444 ..... Done. Updating registration configuration on server localhost:4444 ..... Done. Updating registration configuration on server localhost:6444 ..... Done. Updating replication configuration for baseDN cn=schema on server localhost:4444 ..... Done. Initializing registration information on server localhost:6444 with the contents of server localhost:4444 ..... Done. Replication has been successfully enabled. Note that for replication to work you must initialize the contents of the base DN's that are being replicated (use dsreplication initialize to do so). See /var/.../opends-replication-1720959352638609971.log for a detailed log of this operation. $ dsreplication enable -I admin -w password -X -n -b dc=example,dc=com \ > --host1 `hostname` --port1 5444 --bindDN1 "cn=Directory Manager" \ > --bindPassword1 password --noReplicationServer1 \ > --host2 `hostname` --port2 6444 --bindDN2 "cn=Directory Manager" \ > --bindPassword2 password --replicationPort2 8989 --onlyReplicationServer2 Establishing connections ..... Done. Only one replication server will be defined for the following base DN's: dc=example,dc=com It is recommended to have at least two replication servers (two changelogs) to avoid a single point of failure in the replication topology. Checking registration information ..... Done. Updating remote references on server localhost:6444 ..... Done. Updating replication configuration for baseDN dc=example,dc=com on server localhost:5444 ..... Done. Updating replication configuration for baseDN dc=example,dc=com on server localhost:4444 ..... Done. Updating registration configuration on server localhost:5444 ..... Done. Updating registration configuration on server localhost:6444 ..... Done. Updating registration configuration on server localhost:4444 ..... Done. Updating replication configuration for baseDN cn=schema on server localhost:5444 ..... Done. Updating replication configuration for baseDN cn=schema on server localhost:4444 ..... Done. Initializing registration information on server localhost:5444 with the contents of server localhost:6444 ..... Done. Replication has been successfully enabled. Note that for replication to work you must initialize the contents of the base DN's that are being replicated (use dsreplication initialize to do so). See /var/folders/.../opends-replication-5893037538856033562.log for a detailed log of this operation. Initialize replication from one of the directory servers. $ dsreplication initialize-all -I admin -w password -X -n -b dc=example,dc=com \ > -h `hostname` -p 4444 Initializing base DN dc=example,dc=com with the contents from localhost:4444: 160 entries processed (100 % complete). Base DN initialized successfully. See /var/.../opends-replication-7677303986403997574.log for a detailed log of this operation.

Replication lets you define groups so that replicas communicate first with replication servers in the group before going to replication servers outside the group. Groups are identified with unique numeric group IDs. For each group, set the appropriate group ID for the topology on both the replication servers and the directory servers. The example commands in this procedure set up two replication groups, each with a replication server and a directory server. The directory servers have admin ports 4444 and 5444. The replication servers have admin ports 6444 and 7444. In a full-scale deployment, you would have multiple servers of each type in each group, such as all the replicas and replication servers in each data center being in the same group. Pick a group ID for each group. The default group ID is 1. Set the group ID for each group by replication domain on the directory servers. $ dsconfig -p 4444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-domain-prop --provider-name "MultimasterSynchronization" \ > --domain-name "dc=example,dc=com" --set group-id:1 -X -n $ dsconfig -p 5444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-domain-prop --provider-name "Multimaster Synchronization" \ > --domain-name "dc=example,dc=com" --set group-id:2 -X -n Set the group ID for each group on the replication servers. $ dsconfig -p 6444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-server-prop --provider-name "Multimaster Synchronization" \ > --set group-id:1 -X -n $ dsconfig -p 7444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-server-prop --provider-name "Multimaster Synchronization" \ > --set group-id:2 -X -n

By default all directory servers in a replication topology are read-write. You can however choose to make replicas take updates only from the replication protocol, and refuse updates from client applications. $ dsconfig -p 5444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-global-configuration-prop --set writability-mode:internal-only -X -n

In standard replication, when a client requests an update operation the directory server performs the update and, if the update is successful, sends information about the update to the replication service, and sends a result code to the client application right away. As a result, the client application can conclude that the update was successful, but only on the replica that handled the update. Assured replication lets you force the replica performing the initial update to wait for confirmation that the update has been received elsewhere in the topology before sending a result code to the client application. You can configure assured replication either to wait for one or more replication servers to acknowledge having received the update, or to wait for all directory servers to have replayed the update. As you might imagine, assured replication is theoretically safer than standard replication, yet it is also slower, potentially waiting for a timeout before failing when the network or other servers are down. Safe data mode requires the update be sent to assured-sd-level replication servers before acknowledgement is returned to the client application. For each directory server, set safe data mode for the replication domain, and also set the safe data level. $ dsconfig -p 4444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-domain-prop --provider-name "Multimaster Synchronization" \ > --domain-name "dc=example,dc=com" \ > --set assured-type:safe-data --set assured-sd-level:1 -X -n $ dsconfig -p 5444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-domain-prop --provider-name "Multimaster Synchronization" \ > --domain-name "dc=example,dc=com" \ > --set assured-type:safe-data --set assured-sd-level:1 -X -n Safe read mode requires the update be replayed on all directory servers before acknowledgement is returned to the client application. For each directory server, set safe read mode for the replication domain. $ dsconfig -p 4444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-domain-prop --provider-name "Multimaster Synchronization" \ > --domain-name "dc=example,dc=com" --set assured-type:safe-read -X -n $ dsconfig -p 5444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-domain-prop --provider-name "Multimaster Synchronization" \ > --domain-name "dc=example,dc=com" --set assured-type:safe-read -X -n When working with assured replication, the replication server property degraded-status-threshold (default: 5000), sets the number of operations allowed to build up in the replication queue before the server is assigned degraded status. When a replication server has degraded status, assured replication ceases to have an effect.

OpenDJ lets you do subtree replication, for example replicating ou=People,dc=example,dc=com, but not the rest of dc=example,dc=com, by putting the subtree in a separate backend from the rest of the suffix. For example, in this case you might have a userRoot backend containing everything in dc=example,dc=com except ou=People,dc=example,dc=com, and a separate peopleRoot backend for ou=People,dc=example,dc=com. Then you replicate ou=People,dc=example,dc=com in its own topology.

OpenDJ lets you do fractional replication, whereby you specify the attributes to include in the replication process, or alternatively specify the attributes to exclude. You set fractional replication configuration as fractional-include or fractional-exclude properties for a replication domain. When you include attributes, the attributes that must be kept on the relevant object classes are also included, whether you specify them or not. When you exclude attributes, the excluded attributes must be optional attributes for the relevant object classes. Fractional replica still respect schema definitions. For example, you might configure an externally facing fractional replica to include only some inetOrgPerson attributes. $ dsconfig -p 4444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-domain-prop --provider-name "Multimaster Synchronization" \ --domain-name "dc=example,dc=com" -X -n --set \ fractional-include:inetorgperson:cn,givenname,mail,mobile,sn,telephonenumber As another example, you might exclude a custom attribute called sessionToken from being replicated. dsconfig -p 4444 -h `hostname` -D "cn=Directory Manager" -w password \ > set-replication-domain-prop --provider-name "Multimaster Synchronization" \ --domain-name "dc=example,dc=com" --set fractional-exclude:*:sessionToken -X -n This last example only works if you first define a sessionToken attribute in the directory server schema.

Some of your applications might require notification when directory data updates occur. For example, the application might need to sync directory data with another database, or the application might need to kick off other processing when certain updates occur. In addition to supporting peristent search operations, OpenDJ provides a change log mechanism to allow applications to be notified of changes to directory data. TODO