* {@code ReplicationServerHandler.toServerAddressURL()}, which this change removes, took * the host of a handler from {@code session.getRemoteAddress()} and its port from the start * message that handler received, so the address a peer was registered under was an artefact * of which interface its connection used, not an identity. Two things followed, and the * tests below drive both: * the already connected test of {@code runConnect()} compared a configured address against * one which never matches it, so the peer was dialled again on every pass, about once a * second, for as long as it stayed where it was; and the handshake offered to that same * peer ran into a handler holding its server id under another address URL, which is * {@code ERR_DUPLICATE_REPLICATION_SERVER_ID} -- an error logged with no throttle for a * topology which is merely multi homed. *
* The fixture is the mechanism itself rather than a stand in for it: the peer dials the * server under test over the loopback interface and names another address in its start * message, which is what a peer behind a NAT looks like to the server it dials, and the * session the server dials that other address on reports the address it dialled, which is * what reaching the same peer at its configured address gives. The addresses named are * documentation addresses (RFC 5737): nothing answers at them, which is what the connect * thread of the server under test finds when it dials one -- a host stack sends that SYN to * its default route, and the connect fails, once per six passes -- a failed dial is * blacklisted for the five passes after it -- until the peer has registered. */ @SuppressWarnings("javadoc") public class MultiHomedPeerTest extends ReplicationTestCase { private static final int SOCKET_TIMEOUT_MS = 30000; /** How long the registration of the fake peer is waited for, in milliseconds. */ private static final long REGISTRATION_TIMEOUT_MS = 30000; private static final int RS_ID = 8251; private static final int PEER_RS_ID = 8252; /** The server id of the second fake peer, which only the removal case registers. */ private static final int SECOND_PEER_RS_ID = 8253; /** TEST-NET-1: the address the peer is configured under and answers on. */ private static final byte[] PEER_ADDRESS = { (byte) 192, 0, 2, 1 }; /** TEST-NET-2: the address of a second, genuinely different server. */ private static final byte[] OTHER_ADDRESS = { (byte) 198, 51, 100, 1 }; /** * A name under the .invalid top level domain (RFC 6761), which resolvers answer does not * exist: the host name of a peer which this server has no way to resolve, and what * {@code ReplicationServer.setServerURL()} falls back to naming. */ private static final String UNRESOLVABLE_HOST = "nonexistent.invalid"; /** * Tests that a peer which dialled this server from an address it is not configured under * is still found by the address it is configured under. *
* This is what stops the connect thread from dialling it on every pass: the address the * handler is registered under is the loopback address the peer's own connection came * from, and the only address which can match the configured one is the address the peer * named in its start message. *
* Both halves are asserted, because the predicate and the call site which reads it fail * apart: what the connect thread does with a peer it finds connected is reported by the * outage it closes, and an outage is recorded here for it to close. */ @Test public void aPeerRegisteredUnderTheAddressItDialledFromIsFoundByItsConfiguredAddress() throws Exception { TestCaseUtils.startServer(); final DN baseDN = DN.valueOf(TEST_ROOT_DN_STRING); final int[] ports = TestCaseUtils.findFreePorts(2); final HostPort peerAddress = documentationAddress(PEER_ADDRESS, ports[1]); ReplicationServer rs = null; Session inbound = null; try { rs = startServerWithPeerConfiguredAt(ports[0], "multiHomedPeerSkipDb", peerAddress); final ReplicationServerDomain domain = rs.getReplicationServerDomain(baseDN, true); /* * An outage for the configured address, which nothing but the already connected * branch of runConnect() closes: a peer that branch skips is dialled by no one, so a * connection is reported for it nowhere else. Recorded from this thread rather than * waited for, because whether the connect thread has dialled that address before the * peer registers is a race and what is asserted below is not. */ assertThat(rs.connect(peerAddress, baseDN)) .as("nothing answers at " + peerAddress).isFalse(); inbound = registerPeerFrom(ports[0], peerAddress, baseDN); final ReplicationServerHandler registered = waitForRegistration(domain); // The precondition of the case rather than an assumption: a peer registered under the // address it is configured under is a peer this test says nothing about. assertThat(registered.getServerAddressURL()) .as("the peer should be registered under the loopback address it dialled from") .isEqualTo(loopbackAt(ports[1]).toString()); assertThat(registered.getServerURL()) .as("the peer should name the address it is configured under in its start message") .isEqualTo(peerAddress.toString()); assertThat(domain.isConnectedToServerAt(peerAddress)) .as("the peer is connected, so its configured address must not be dialled again") .isTrue(); assertThat(waitForConnectRestored(rs, peerAddress, baseDN)) .as("the connect thread must find the peer at its configured address and close the" + " outage reported for it, rather than dial it again") .isTrue(); } finally { close(inbound); removeQuietly(rs); } } /** * Tests that a second session with a peer already connected under another address is * dropped as the duplicate connection it is, and not reported as two servers sharing a * server id. *
* The server under test dials the peer at its configured address while holding the
* session that same peer dialled it on. Both sessions are with one server, which names
* one address in both of its start messages; only the addresses the two sockets carry
* differ, which is what being reachable at more than one address means.
*/
@Test
public void aPeerReachedAtItsConfiguredAddressIsNotReportedAsADuplicateServerId()
throws Exception
{
TestCaseUtils.startServer();
final DN baseDN = DN.valueOf(TEST_ROOT_DN_STRING);
final int[] ports = TestCaseUtils.findFreePorts(2);
final HostPort peerAddress = documentationAddress(PEER_ADDRESS, ports[1]);
ReplicationServer rs = null;
Session inbound = null;
try
{
rs = startServerWithPeerConfiguredAt(ports[0], "multiHomedPeerDuplicateDb", peerAddress);
final ReplicationServerDomain domain = rs.getReplicationServerDomain(baseDN, true);
inbound = registerPeerFrom(ports[0], peerAddress, baseDN);
final ReplicationServerHandler registered = waitForRegistration(domain);
final List
* The peer of the first case names the address it is configured under, and is found by
* it. This one is the other way round: it is configured at the address it dials this
* server from, and names one this topology does not configure it at -- the host name of
* its machine under another of its addresses, which {@code setServerURL()} falls back to.
* The address it names matches nothing here, so the address its session came from is what
* has to match.
*/
@Test
public void aPeerWhichNamesAnAddressThisConfigurationDoesNotUseIsFoundByTheOneItDialledFrom()
throws Exception
{
TestCaseUtils.startServer();
final DN baseDN = DN.valueOf(TEST_ROOT_DN_STRING);
final int[] ports = TestCaseUtils.findFreePorts(2);
final HostPort peerAddress = loopbackAt(ports[1]);
final HostPort namedAddress = documentationAddress(OTHER_ADDRESS, ports[1]);
ReplicationServer rs = null;
Session inbound = null;
try
{
rs = startServerWithPeerConfiguredAt(ports[0], "multiHomedPeerNamedDb", peerAddress);
final ReplicationServerDomain domain = rs.getReplicationServerDomain(baseDN, true);
inbound = registerPeerFrom(ports[0], namedAddress, baseDN);
final ReplicationServerHandler registered = waitForRegistration(domain);
// The precondition of the case rather than an assumption, as in the first case: a
// peer which names the address it is configured under is the peer of that one.
assertThat(registered.getServerURL())
.as("the peer should name the address this configuration does not use")
.isEqualTo(namedAddress.toString());
assertThat(registered.getServerAddressURL())
.as("the peer should be registered under the loopback address it dialled from")
.isEqualTo(peerAddress.toString());
assertThat(domain.isConnectedToServerAt(peerAddress))
.as("the address the connection of such a peer came from is the only one which"
+ " can match the address it is configured at")
.isTrue();
}
finally
{
close(inbound);
removeQuietly(rs);
}
}
/**
* Tests that a peer which names a host this server cannot resolve is recognised by that
* name all the same, rather than reported as two servers sharing a server id.
*
* {@code setServerURL()} falls back to the host name of the machine when none of the
* addresses a peer is configured with is local to it, and the rest of the topology has no
* reason to resolve that name. Both handlers of such a peer name it, so it is what tells
* this server they are one -- but a comparison which resolves both sides answers that a
* name it cannot resolve is equivalent to nothing at all, not even to itself, and the
* handshake this server offers such a peer would abort on a duplicate server id on every
* pass which reaches it.
*/
@Test
public void aPeerWhichNamesAnUnresolvableHostIsNotReportedAsADuplicateServerId()
throws Exception
{
TestCaseUtils.startServer();
final DN baseDN = DN.valueOf(TEST_ROOT_DN_STRING);
final int[] ports = TestCaseUtils.findFreePorts(2);
final HostPort peerAddress = documentationAddress(PEER_ADDRESS, ports[1]);
final HostPort namedAddress = new HostPort(UNRESOLVABLE_HOST, ports[1]);
ReplicationServer rs = null;
Session inbound = null;
try
{
rs = startServerWithPeerConfiguredAt(ports[0], "multiHomedPeerUnresolvedDb", peerAddress);
final ReplicationServerDomain domain = rs.getReplicationServerDomain(baseDN, true);
inbound = registerPeerFrom(ports[0], namedAddress, baseDN);
final ReplicationServerHandler registered = waitForRegistration(domain);
/*
* The premise of this case rather than an assumption about the network it runs on:
* HostPort reports a name it cannot resolve each time it builds an address from one,
* so those records are what says this machine answers that the name below does not
* exist. A resolver which synthesises an address for a name which does not -- consumer
* ISPs, captive portals, some corporate DNS -- would have isEquivalentTo() answer this
* case on its own, which leaves the arm the case is here for unread, the count below
* vacuous and nothing red to say so.
*/
assertThat(recordsContaining(TestCaseUtils.ERROR_TEXT_WRITER.getMessages(),
ERR_COULD_NOT_SOLVE_HOSTNAME.get(UNRESOLVABLE_HOST).toString()))
.as("the resolver of this machine must answer that " + UNRESOLVABLE_HOST
+ " does not exist, which is the premise of this case")
.isNotEmpty();
/*
* What the addresses are built from is read once, when the start message names them:
* a name which cannot be resolved is reported by HostPort each time one is built from
* it, and the connect thread compares them on every pass. The comparison below is the
* one that thread makes.
*/
TestCaseUtils.ERROR_TEXT_WRITER.clear();
for (int pass = 0; pass < 3; pass++)
{
domain.isConnectedToServerAt(peerAddress);
}
assertThat(recordsContaining(TestCaseUtils.ERROR_TEXT_WRITER.getMessages(),
ERR_COULD_NOT_SOLVE_HOSTNAME.get(UNRESOLVABLE_HOST).toString()))
.as("comparing the addresses of a handler must not build them again")
.isEmpty();
final List
* Nothing is shared here: the connected peer names one address and dialled from the
* loopback interface, and the server answering the handshake names, and is reached at,
* another address on the same port. The port is the same one on purpose: it is the host
* which tells the two servers apart, and a second port would answer the comparison before
* any host of it is read.
*/
@Test
public void twoServersSharingAServerIdAreStillReported() throws Exception
{
TestCaseUtils.startServer();
final DN baseDN = DN.valueOf(TEST_ROOT_DN_STRING);
final int[] ports = TestCaseUtils.findFreePorts(2);
final HostPort peerAddress = documentationAddress(PEER_ADDRESS, ports[1]);
final HostPort otherAddress = documentationAddress(OTHER_ADDRESS, ports[1]);
ReplicationServer rs = null;
Session inbound = null;
try
{
rs = startServerWithPeerConfiguredAt(ports[0], "multiHomedPeerConflictDb", peerAddress);
final ReplicationServerDomain domain = rs.getReplicationServerDomain(baseDN, true);
inbound = registerPeerFrom(ports[0], peerAddress, baseDN);
waitForRegistration(domain);
final List
* This is the other arm of the comparison the handshake makes, and what it gives up:
* a session from the address a peer is connected on is that peer, so two replication
* servers which share a server id and reach this one from behind a single gateway are read
* as one and the second of them is dropped in silence. The base did the same with them --
* it compared the addresses the two sockets carried, which for such a pair is one string
* twice -- and this is the arm which recognises the peer of the case above, whose named
* address matches nothing in this configuration.
*/
@Test
public void aSecondSessionFromTheAddressAPeerIsConnectedOnIsReadAsThatPeer() throws Exception
{
TestCaseUtils.startServer();
final DN baseDN = DN.valueOf(TEST_ROOT_DN_STRING);
final int[] ports = TestCaseUtils.findFreePorts(2);
final HostPort peerAddress = documentationAddress(PEER_ADDRESS, ports[1]);
final HostPort otherAddress = documentationAddress(OTHER_ADDRESS, ports[1]);
ReplicationServer rs = null;
Session inbound = null;
try
{
rs = startServerWithPeerConfiguredAt(ports[0], "multiHomedPeerGatewayDb", peerAddress);
final ReplicationServerDomain domain = rs.getReplicationServerDomain(baseDN, true);
inbound = registerPeerFrom(ports[0], peerAddress, baseDN);
final ReplicationServerHandler registered = waitForRegistration(domain);
/*
* The handshake is answered from the loopback address the session of the registered
* peer came from, and what answers it names an address neither handler carries
* otherwise: the named addresses of the two differ, so the address the two sessions
* have in common is the only thing which can match them.
*/
final List
* {@code disconnectRemovedReplicationServers()} hands the addresses which were removed
* from {@code ds-cfg-replication-server} to this domain, and a handler which does not
* answer to any of them stays connected: the peer an administrator took out of the
* topology keeps replicating with this server until one of the two is restarted.
*/
@Test
public void aPeerRemovedFromTheConfigurationIsDisconnectedByItsConfiguredAddress()
throws Exception
{
TestCaseUtils.startServer();
final DN baseDN = DN.valueOf(TEST_ROOT_DN_STRING);
final int[] ports = TestCaseUtils.findFreePorts(2);
final HostPort peerAddress = documentationAddress(PEER_ADDRESS, ports[1]);
ReplicationServer rs = null;
Session inbound = null;
try
{
rs = startServerWithPeerConfiguredAt(ports[0], "multiHomedPeerRemovedDb", peerAddress);
final ReplicationServerDomain domain = rs.getReplicationServerDomain(baseDN, true);
inbound = registerPeerFrom(ports[0], peerAddress, baseDN);
waitForRegistration(domain);
domain.stopReplicationServers(Collections.singletonList(peerAddress));
assertThat(domain.getConnectedRSs().keySet())
.as("the peer removed from the configuration must be disconnected")
.doesNotContain(PEER_RS_ID);
}
finally
{
close(inbound);
removeQuietly(rs);
}
}
/**
* Tests that taking the entry of this server out of {@code ds-cfg-replication-server}
* stops no peer, while taking the entry of a peer out still stops that peer.
*
* {@code HostPort} normalises every address local to this machine to {@code localhost}, so
* the entry of this server and the loopback address a peer names for itself are one
* address on this port: a peer whose own configuration lists {@code localhost:P} for
* itself names exactly that in its start messages -- {@code setServerURL()} takes the
* first configured entry which is local to it -- and on this server that name normalises
* to what its own entry does. The connect thread skips the entry of this server before it
* dials; the road which disconnects removed replication servers has to skip it as well, or
* removing it -- a configuration which does not list this server is supported, see
* {@code runConnect()} -- costs such a peer its session, which it then has to dial again.
*
* Both entries go in one change, so what tells the two peers apart is that guard alone:
* the peer which names the entry of this server stays, the peer whose own entry was
* removed goes. What the in JVM fixture cannot give is the fold itself, which needs one
* port on two machines; what it names is the address of this server, which normalises the
* same way.
*/
@Test
public void removingTheEntryOfThisServerStopsNoPeer() throws Exception
{
TestCaseUtils.startServer();
final DN baseDN = DN.valueOf(TEST_ROOT_DN_STRING);
final int[] ports = TestCaseUtils.findFreePorts(2);
final HostPort ownAddress = loopbackAt(ports[0]);
final HostPort peerAddress = loopbackAt(ports[1]);
final String dbDirName = "multiHomedPeerOwnEntryDb";
ReplicationServer rs = null;
Session namesThisServer = null;
Session namesItsOwnEntry = null;
try
{
rs = startServerWithPeersConfiguredAt(ports[0], dbDirName, ownAddress, peerAddress);
final ReplicationServerDomain domain = rs.getReplicationServerDomain(baseDN, true);
namesThisServer = registerPeerFrom(ports[0], PEER_RS_ID, ownAddress, baseDN);
final ReplicationServerHandler namedThisServer = waitForRegistration(domain, PEER_RS_ID);
namesItsOwnEntry = registerPeerFrom(ports[0], SECOND_PEER_RS_ID, peerAddress, baseDN);
waitForRegistration(domain, SECOND_PEER_RS_ID);
// The precondition of the case rather than an assumption: a peer which does not answer
// to the entry of this server is a peer this case says nothing about.
assertThat(namedThisServer.isServerAt(HostPort.localAddress(ports[0])))
.as("the peer which names a loopback address on the port of this server should"
+ " answer to the entry of this server")
.isTrue();
// Both entries removed at once, which is what an administrator emptying the list does.
rs.applyConfigurationChange(configurationWithPeersAt(ports[0], dbDirName));
assertThat(domain.getConnectedRSs().keySet())
.as("a peer which names the address this server listens on must outlive the removal"
+ " of the entry of this server")
.contains(PEER_RS_ID);
assertThat(domain.getConnectedRSs().keySet())
.as("the peer whose own entry was removed must still be disconnected")
.doesNotContain(SECOND_PEER_RS_ID);
}
finally
{
close(namesThisServer);
close(namesItsOwnEntry);
removeQuietly(rs);
}
}
/**
* Starts a replication server whose only configured peer is at the provided address.
*
* Nothing ever answers there -- the address is a documentation address -- which is what
* the connect thread of the server does with it: it dials it, fails, and comes back to it
* six passes later, the failed dial being blacklisted for the five in between. Whether it
* dials it at all is what the first case is about.
*/
private ReplicationServer startServerWithPeerConfiguredAt(int port, String dbDirName,
HostPort peerAddress) throws Exception
{
return startServerWithPeersConfiguredAt(port, dbDirName, peerAddress);
}
/** Starts a replication server whose configured peers are the provided addresses. */
private ReplicationServer startServerWithPeersConfiguredAt(int port, String dbDirName,
HostPort... peerAddresses) throws Exception
{
return new ReplicationServer(configurationWithPeersAt(port, dbDirName, peerAddresses));
}
/**
* Returns the configuration of the server under test, listing the provided addresses as
* the replication servers of its topology.
*/
private ReplServerFakeConfiguration configurationWithPeersAt(int port, String dbDirName,
HostPort... peerAddresses)
{
final SortedSet
* The session is dialled over the loopback interface and reports {@code dialledAt} as the
* address it reaches the peer at, which is what dialling a peer at its configured address
* gives whatever interface the peer's own connection to this server used. The answer is
* a well formed start message naming {@code answersAs}: what ends the handshake is the
* handler this server already holds for that server id, not what the peer answers here.
*
* @param listenPort
* the port the fake peer answers the handshake on
* @param dialledAt
* the address the session reports the peer at
* @param answersAs
* the address the peer names in the start message it answers with
*/
private List
* That report is what the already connected branch of {@code runConnect()} does with a
* peer it finds registered, and the only thing this server does with one: the branch
* which dials a peer reports a connection only for a handshake it completed, which the
* documentation address of these tests never gives.
*/
private boolean waitForConnectRestored(ReplicationServer rs, HostPort peer, DN baseDN)
throws Exception
{
final String message =
NOTE_REPLICATION_SERVER_CONNECT_RESTORED.get(RS_ID, peer, baseDN).toString();
final long deadline = System.currentTimeMillis() + REGISTRATION_TIMEOUT_MS;
while (true)
{
if (!recordsContaining(TestCaseUtils.ERROR_TEXT_WRITER.getMessages(), message).isEmpty())
{
return true;
}
if (System.currentTimeMillis() > deadline)
{
return false;
}
rs.waitConnections();
}
}
/**
* Returns the records of the provided log which report the two provided address URLs as
* two replication servers sharing a server id.
*
* The addresses are the ones the message names, so a record of the handshake next door
* cannot be read as one of the handshake under test.
*/
private List