external-software/github_OpenIdentityPlatform_OpenDJ.git

parent: fc4e7423 | patch | commit | show whitespace

[#1213] Try another free port when the offline connect of the proxy tests c...

Valery Kharseko

73 mins ago 8fc60b90d5ba0641b3a693286ecf65881c165d90

[#1213] Try another free port when the offline connect of the proxy tests connects the client to itself (#1214)

Fixes #1213

### Problem

`GrizzlyLDAPListenerTestCase#testLDAPListenerProxyDuringHandleBind`
failed on `windows-latest` / JDK 27 with `Connection to offline server
succeeded unexpectedly` ([run
38947864951](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/38947864951/job/117131465938),
attempt 1). This is the failure that #1181 made visible for #1180.

Both proxy tests take a free loopback port P from
`findFreeSocketAddress()` and expect a connect to it to be refused. P
comes from the same dynamic range that the client socket takes its own
source port from. When the client gets P as its source port, its SYN
reaches the socket itself. The TCP simultaneous open then connects the
socket to itself, and the "offline" connect succeeds.

### Evidence

- **The failed job.** Just before the failure, Grizzly logged
`GRIZZLY0013: Exception during FilterChain execution` with a
`NullPointerException` in `LDAPClientFilter.handleReadException`. None
of the other 8 `build-maven` jobs of the run logged it. That method runs
when the client side fails to decode what it read. It hits `null` when
the connection was already closed. So a closed client connection read a
message that a server never sends to a client. On the offline connection
the client sent only its Unbind, from `close()`.
- **Linux reproduces the mechanism** (container, kernel 7.0, JDK 21). An
`LDAPConnectionFactory` that connects again and again to one free even
port connected to itself on 11 ports out of 11, after 676 to 41470
attempts. `/proc/net/tcp` showed `127.0.0.1:P <-> 127.0.0.1:P`
`ESTABLISHED`. A bind on such a connection failed with `Decoding Error:
Unexpected LDAP request: id=1, message=GenericBindRequest(...)`: the
client read its own request. The probe-then-connect order of the tests
does not reach this on Linux, because `bind(0)` and `connect()` take
ports of opposite parity: 0 of 100000 connects.
- **macOS fails such a connect** with `SocketException: Invalid
argument`: 6 of 100000 connects to one free port (about 1 in 16384),
none succeeded.

Windows was not checked on a Windows host. The conclusion for Windows
rests on the log of the failed job.

### Fix (test only)

`failOverToOnlineServer` now calls a new `connectToOfflineServer()`.
When a connect to the free port succeeds, that method closes the
connection and tries a fresh free port, up to 3 times. If every attempt
succeeds, the error lists the ports that accepted: `Connection to
offline server succeeded unexpectedly on ports [...]`. The online leg
and the error messages of the other failures stay the same.

Keeping P bound during the connect would stop the client from getting
it, but #1181 showed that on macOS a bound but not listening socket
swallows the connect (it times out instead of being refused).

### Verification

Mutants of the offline address, each run against both proxy tests:

| Variant | Result |
|---|---|
| BASE, no mutant | 2/2 pass |
| BASE, the offline port accepts | 2/2 fail with `Connection to offline
server succeeded unexpectedly` (the CI signature) |
| PR, no mutant | 2/2 pass |
| PR, the 1st attempt accepts | 2/2 pass |
| PR, the 1st and 2nd attempts accept | 2/2 pass |
| PR, every attempt accepts | 2/2 fail in 2 s with `... succeeded
unexpectedly on ports [50887, 50887, 50887]` |

In the mutants, the accepting address is a `ServerSocket` that listens
and never calls `accept()`.

Without a mutant, all of `opendj-grizzly` passes: 1040 tests, 0
failures.

### Not changed

- The load-balancer tests in this class and in
`ConnectionFactoryTestCase` (`:201`, `:206`) rely on the same assumption
about their offline ports. This can happen only on Windows, and it has
not been seen there. In those tests the load balancer makes the
connects, so the test has no place to retry.
- `LDAPClientFilter.handleReadException` does not check for a missing
connection, unlike the `*Result` handlers. Its `NullPointerException`
replaced the `DecodeException` in the log of the failed job. That is a
change to the main code, outside this test fix.

1 files modified

opendj-grizzly/src/test/java/org/forgerock/opendj/grizzly/GrizzlyLDAPListenerTestCase.java

41 ●●●●● patch | view | raw | blame | history

 opendj-grizzly/src/test/java/org/forgerock/opendj/grizzly/GrizzlyLDAPListenerTestCase.java

@@ -32,8 +32,10 @@
import java.lang.reflect.Field;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
@@ -204,37 +206,56 @@
        return socket;
    }

    /** How many free ports {@link #connectToOfflineServer()} tries before it gives up. */
    private static final int OFFLINE_CONNECT_ATTEMPTS = 3;

    /**
     * Fails over the way a proxy does: connects to a free port, which must refuse the connection,
     * then to the online server.
     */
    private static void failOverToOnlineServer(final InetSocketAddress onlineAddress) throws LdapException {
        // The numeric host avoids a forward lookup of "localhost".
        final InetSocketAddress offlineAddress = findFreeSocketAddress();
        final LDAPConnectionFactory offlineFactory =
                new LDAPConnectionFactory(offlineAddress.getAddress().getHostAddress(), offlineAddress.getPort());
        try {
            offlineFactory.getConnection().close();
        } catch (final ConnectionException expected) {
            // This is expected - so go to online server.
        connectToOfflineServer();
            final LDAPConnectionFactory onlineFactory =
                    new LDAPConnectionFactory(onlineAddress.getHostString(), onlineAddress.getPort());
            try {
                onlineFactory.getConnection().close();
                return;
            } catch (final Exception e) {
                throw newLdapException(ResultCode.OTHER,
                        "Unexpected exception when connecting to online server", e);
            } finally {
                onlineFactory.close();
            }
    }

    /**
     * Connects to a free port, which must refuse the connection.
     * <p>
     * A free port can still accept it: when the client socket gets that same port as its own
     * ephemeral port, it connects to itself (TCP simultaneous open). Linux and Windows allow such
     * a connection, macOS fails it. The connection is then closed and another free port is tried,
     * a bounded number of times.
     */
    private static void connectToOfflineServer() throws LdapException {
        final List<Integer> acceptingPorts = new ArrayList<>();
        do {
            // The numeric host avoids a forward lookup of "localhost".
            final InetSocketAddress offlineAddress = findFreeSocketAddress();
            final LDAPConnectionFactory offlineFactory =
                    new LDAPConnectionFactory(offlineAddress.getAddress().getHostAddress(), offlineAddress.getPort());
            try {
                offlineFactory.getConnection().close();
                acceptingPorts.add(offlineAddress.getPort());
            } catch (final ConnectionException expected) {
                return;
        } catch (final Exception e) {
            throw newLdapException(ResultCode.OTHER,
                    "Unexpected exception when connecting to offline server", e);
        } finally {
            offlineFactory.close();
        }
        throw newLdapException(ResultCode.OTHER, "Connection to offline server succeeded unexpectedly");
        } while (acceptingPorts.size() < OFFLINE_CONNECT_ATTEMPTS);
        throw newLdapException(ResultCode.OTHER,
                "Connection to offline server succeeded unexpectedly on ports " + acceptingPorts);
    }

    /** Disables logging before the tests. */

			@@ -32,8 +32,10 @@
			import java.lang.reflect.Field;
			import java.net.InetSocketAddress;
			import java.net.ServerSocket;
			import java.util.ArrayList;
			import java.util.Arrays;
			import java.util.Collections;
			import java.util.List;
			import java.util.concurrent.CountDownLatch;
			import java.util.concurrent.TimeUnit;
			import java.util.logging.Level;
			@@ -204,37 +206,56 @@
			return socket;
			}

			/** How many free ports {@link #connectToOfflineServer()} tries before it gives up. */
			private static final int OFFLINE_CONNECT_ATTEMPTS = 3;

			/**
			* Fails over the way a proxy does: connects to a free port, which must refuse the connection,
			* then to the online server.
			*/
			private static void failOverToOnlineServer(final InetSocketAddress onlineAddress) throws LdapException {
			// The numeric host avoids a forward lookup of "localhost".
			final InetSocketAddress offlineAddress = findFreeSocketAddress();
			final LDAPConnectionFactory offlineFactory =
			new LDAPConnectionFactory(offlineAddress.getAddress().getHostAddress(), offlineAddress.getPort());
			try {
			offlineFactory.getConnection().close();
			} catch (final ConnectionException expected) {
			// This is expected - so go to online server.
			connectToOfflineServer();
			final LDAPConnectionFactory onlineFactory =
			new LDAPConnectionFactory(onlineAddress.getHostString(), onlineAddress.getPort());
			try {
			onlineFactory.getConnection().close();
			return;
			} catch (final Exception e) {
			throw newLdapException(ResultCode.OTHER,
			"Unexpected exception when connecting to online server", e);
			} finally {
			onlineFactory.close();
			}
			}

			/**
			* Connects to a free port, which must refuse the connection.
			* <p>
			* A free port can still accept it: when the client socket gets that same port as its own
			* ephemeral port, it connects to itself (TCP simultaneous open). Linux and Windows allow such
			* a connection, macOS fails it. The connection is then closed and another free port is tried,
			* a bounded number of times.
			*/
			private static void connectToOfflineServer() throws LdapException {
			final List<Integer> acceptingPorts = new ArrayList<>();
			do {
			// The numeric host avoids a forward lookup of "localhost".
			final InetSocketAddress offlineAddress = findFreeSocketAddress();
			final LDAPConnectionFactory offlineFactory =
			new LDAPConnectionFactory(offlineAddress.getAddress().getHostAddress(), offlineAddress.getPort());
			try {
			offlineFactory.getConnection().close();
			acceptingPorts.add(offlineAddress.getPort());
			} catch (final ConnectionException expected) {
			return;
			} catch (final Exception e) {
			throw newLdapException(ResultCode.OTHER,
			"Unexpected exception when connecting to offline server", e);
			} finally {
			offlineFactory.close();
			}
			throw newLdapException(ResultCode.OTHER, "Connection to offline server succeeded unexpectedly");
			} while (acceptingPorts.size() < OFFLINE_CONNECT_ATTEMPTS);
			throw newLdapException(ResultCode.OTHER,
			"Connection to offline server succeeded unexpectedly on ports " + acceptingPorts);
			}

			/** Disables logging before the tests. */