[#1161] Keep bc-fips from seeding its DRBG from RDSEED in the test JVMs (#1...

Valery Kharseko

yesterday 10fc27e672753c6718197cf36da388a5d66bd285

[#1161] Keep bc-fips from seeding its DRBG from RDSEED in the test JVMs (#1168)

Fixes #1161

### Problem

On the Linux `build-maven` legs the embedded test server sometimes fails
to start: bc-fips 2.1.3 seeds its DRBG from the CPU's RDSEED instruction
through its native libraries, and on a busy CI host RDSEED runs dry
while the server generates its self-signed certificates (`RDSEED
persistently failed to produce entropy`). The class that starts the
server fails in `@BeforeClass` and the rest of it is skipped.

### Change

Both test `argLine` values in the root `pom.xml` (the default one and
the `jdk17.options` one) now carry
`-Dorg.bouncycastle.native.cpu_variant=java`. With it bc-fips does not
load its native libraries, and
`FipsDRBG.getDefaultEntropySourceProvider()` falls back to
`SecureRandom.getInstanceStrong()` from the JDK. The
`opendj-server-legacy` failsafe configuration picks it up through
`@{argLine}`.

The argLine does not cross a process boundary.
`AdsTrustStoreInstallTestCase` and `QuickSetupTestCase` start the
packaged `setup`, and `ServerControllerTest` starts `start-ds` through
`ServerController`, which keeps the environment of the fork and drops
only `OPENDJ_JAVA_ARGS` and `CLASSPATH`. Those servers generate their
certificates through bc-fips too, so the `opendj-server-legacy` failsafe
execution also sets `JAVA_TOOL_OPTIONS` to the same flag in
`<environmentVariables>`, and every JVM started from the fork reads it.

Two pins:
- `BcFipsNativeLibrariesOffTestCase` (`opendj-core`) fails when either
root argLine loses the flag. `LDAPServer` generates its key pairs with
bc-fips in that module, and the module takes the root argLine as it is:
the default one below JDK 17, the `jdk17.options` one from JDK 17 on.
- `BcFipsNativeLibrariesOffTest` (`opendj-server-legacy`) fails when the
failsafe fork loses `JAVA_TOOL_OPTIONS`.

Only the test JVMs and the JVMs they start change. The product (start
scripts, `setup`) keeps loading the native libraries, and the "Test on
Unix FIPS" step in `build.yml`, which runs the packaged server outside
Maven, still exercises that path.

### Verification

- `mvn help:evaluate -Dexpression=argLine` shows the property with
`jdk17.options` active, with it disabled, and in `opendj-server-legacy`.
- A small program that makes the same `KeyPairGenerator` call as
`Platform.newKeyPair()`, run against bc-fips 2.1.3 in an
`eclipse-temurin:17` linux/amd64 container on a CPU with `rdseed`:

  | | without the property | `cpu_variant=java` |
  |---|---|---|
  | native status / variant | `READY` / `avx` | `UNSUPPORTED` / none |
  | native DRBG / NRBG | `true` / `true` | `false` / `false` |
| entropy source | `FipsDRBG$1` (`NativeEntropySource`, RDSEED) |
`BasicEntropySourceProvider` (JDK) |
  | RSA 2048 key pair | OK | OK |

- Reactor run (JDK 26): `BcFipsNativeLibrariesOffTestCase` 1/1,
`AdsTrustStoreInstallTestCase` 6/6, `ServerControllerTest` 2/2,
`BcFipsNativeLibrariesOffTest` 2/2. The JVM's `Picked up
JAVA_TOOL_OPTIONS` line breaks nothing.
- The built package's `setup` and `start-ds`, run with the fork's
environment: `setup`, `start-ds` and the server's `logs/server.out` all
print `Picked up JAVA_TOOL_OPTIONS:
-Dorg.bouncycastle.native.cpu_variant=java`, and the server generates
`ads-certificate`.
- Mutants: flag removed from the default argLine → `opendj-core` pin red
on JDK 11; removed from the `jdk17.options` argLine → red on JDK 26
(both green at the head); `JAVA_TOOL_OPTIONS` missing →
`BcFipsNativeLibrariesOffTest` red.

The RDSEED failure itself cannot be reproduced on demand, so the CI legs
are the end-to-end check.

### Follow-up, not in this PR

bc-fips 2.1.4 (a release candidate, see bcgit/bc-java#2434, not yet on
Maven Central) adds `org.bouncycastle.native.rand=NONE`, which turns off
only the hardware RNG and keeps the native AES/SHA acceleration, and
raises the RDSEED retry limit to 1500. That is the better fit for the
product side (start scripts, `setup`) once it is released.

 opendj-core/src/test/java/org/forgerock/opendj/ldap/BcFipsNativeLibrariesOffTestCase.java

New file
@@ -0,0 +1,35 @@
/*
 * The contents of this file are subject to the terms of the Common Development and
 * Distribution License (the License). You may not use this file except in compliance with the
 * License.
 *
 * You can obtain a copy of the License at legal/CDDLv1.0.txt. See the License for the
 * specific language governing permission and limitations under the License.
 *
 * When distributing Covered Software, include this CDDL Header Notice in each file and include
 * the License file at legal/CDDLv1.0.txt. If applicable, add the following below the CDDL
 * Header, with the fields enclosed by brackets [] replaced by your own identifying
 * information: "Portions copyright [year] [name of copyright owner]".
 *
 * Copyright 2026 3A Systems, LLC.
 */
package org.forgerock.opendj.ldap;

import static org.testng.Assert.assertEquals;

import org.testng.annotations.Test;

/**
 * Pins the test argLine flag that keeps bc-fips from seeding its DRBG from the CPU's RDSEED
 * instruction, see issue #1161. {@link LDAPServer} generates its key pairs with bc-fips, and this
 * module takes the argLine of the root pom as it is: the default one below JDK 17, the
 * jdk17.options one from JDK 17 on.
 */
@SuppressWarnings("javadoc")
public class BcFipsNativeLibrariesOffTestCase extends SdkTestCase {
    @Test
    public void testJvmRunsWithoutTheNativeLibraries() {
        assertEquals(System.getProperty("org.bouncycastle.native.cpu_variant"), "java",
                "the test argLine lost the bc-fips cpu_variant flag, see #1161");
    }
}

 opendj-server-legacy/pom.xml

@@ -1282,6 +1282,15 @@
                    <!-- Matched against the name of the test class, see org.opends.server.TestListener.onStart(). -->
                    <org.opends.test.trace.pattern>(org\.opends\.server\.replication\.service\..*)|(org\.opends\.server\.replication\.GenerationIdTest)|(org\.opends\.server\.types\.HostPortTest)|(org\.openidentityplatform\.opendj\.AliasTestCase)</org.opends.test.trace.pattern>
                  </systemPropertyVariables>
                  <!--
                    The argLine flag that keeps bc-fips off RDSEED (issue #1161) does not reach the JVMs
                    the tests start from the built package: setup, and start-ds through ServerController,
                    which keeps the environment of this fork and drops only OPENDJ_JAVA_ARGS and CLASSPATH.
                    Every JVM reads JAVA_TOOL_OPTIONS, so they pick the flag up from here.
                  -->
                  <environmentVariables>
                    <JAVA_TOOL_OPTIONS>-Dorg.bouncycastle.native.cpu_variant=java</JAVA_TOOL_OPTIONS>
                  </environmentVariables>
                  <argLine>@{argLine}</argLine>
                  <reuseForks>false</reuseForks>
                  <forkCount>1</forkCount>

 opendj-server-legacy/src/test/java/org/opends/server/util/BcFipsNativeLibrariesOffTest.java

New file
@@ -0,0 +1,52 @@
/*
 * The contents of this file are subject to the terms of the Common Development and
 * Distribution License (the License). You may not use this file except in compliance with the
 * License.
 *
 * You can obtain a copy of the License at legal/CDDLv1.0.txt. See the License for the
 * specific language governing permission and limitations under the License.
 *
 * When distributing Covered Software, include this CDDL Header Notice in each file and include
 * the License file at legal/CDDLv1.0.txt. If applicable, add the following below the CDDL
 * Header, with the fields enclosed by brackets [] replaced by your own identifying
 * information: "Portions copyright [year] [name of copyright owner]".
 *
 * Copyright 2026 3A Systems, LLC.
 */
package org.opends.server.util;

import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertTrue;

import java.util.Arrays;

import org.opends.server.DirectoryServerTestCase;
import org.testng.annotations.Test;

/**
 * Pins the flag that keeps bc-fips from seeding its DRBG from the CPU's RDSEED instruction in the
 * test JVMs and in the JVMs they start from the built package, see issue #1161.
 */
@SuppressWarnings("javadoc")
public class BcFipsNativeLibrariesOffTest extends DirectoryServerTestCase
{
  private static final String FLAG = "-Dorg.bouncycastle.native.cpu_variant=java";

  @Test
  public void theTestJvmRunsWithoutTheNativeLibraries()
  {
    assertEquals(System.getProperty("org.bouncycastle.native.cpu_variant"), "java",
        "the test JVM lost the bc-fips cpu_variant flag, see #1161");
  }

  /** setup, and start-ds through ServerController, inherit the environment of the failsafe fork. */
  @Test
  public void theJvmsStartedFromThePackageInheritTheFlag()
  {
    final String toolOptions = System.getenv("JAVA_TOOL_OPTIONS");
    assertNotNull(toolOptions, "the failsafe fork lost JAVA_TOOL_OPTIONS, see #1161");
    assertTrue(Arrays.asList(toolOptions.trim().split("\\s+")).contains(FLAG),
        "JAVA_TOOL_OPTIONS of the failsafe fork lost the bc-fips cpu_variant flag, see #1161: " + toolOptions);
  }
}

 pom.xml

@@ -59,7 +59,14 @@
        <checkstylePluginVersion>2.9.1</checkstylePluginVersion>
        <checkstyleVersion>5.5</checkstyleVersion>
        <ant.contrib.version>1.0b3</ant.contrib.version>
        <argLine>-Xmx512m</argLine>
        <!--
          org.bouncycastle.native.cpu_variant=java keeps bc-fips from loading its native libraries in the
          test JVMs, so its DRBG is seeded from the JDK instead of the CPU's RDSEED instruction, which
          runs dry on busy CI hosts ("RDSEED persistently failed to produce entropy"), see issue #1161.
          Keep it in the jdk17.options argLine below as well; BcFipsNativeLibrariesOffTestCase in
          opendj-core fails when either argLine loses it.
        -->
        <argLine>-Xmx512m -Dorg.bouncycastle.native.cpu_variant=java</argLine>
        <maven.cargo.containerId>tomcat10x</maven.cargo.containerId>

        <docHomepageUrl>https://doc.openidentityplatform.org/opendj/</docHomepageUrl>
@@ -735,7 +742,7 @@
            <jdk>[17,)</jdk>
          </activation>
          <properties>
              <argLine>-Xmx512m --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED  --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util.regex=ALL-UNNAMED --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED</argLine>
              <argLine>-Xmx512m -Dorg.bouncycastle.native.cpu_variant=java --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED  --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util.regex=ALL-UNNAMED --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED</argLine>
              <maven.cargo.containerId>tomcat11x</maven.cargo.containerId>
          </properties>
        </profile>