[#1287] Complete large TLS handshakes without blocking writes (#1290)
A TLS handshake can stall when the server certificate flight exceeds the
transport's write buffer. With the default 4096-byte buffer, Grizzly's
blocking handshake can prevent its own pending writes from being
flushed, causing LDAPS and the administration connector to time out.
Start the externally supplied SSL engine explicitly and process the
initial handshake asynchronously. Preserve the configured read timeout
during the handshake, reject duplicate TLS upgrades while StartTLS is
pending, and refuse upgrades on closed connections. A regression test
covers an expiry task that has started before the handshake deadline is
renewed. Document the `enableTLS` refusal cases, report the possible
reasons when StartTLS cannot be enabled, and include the new refusal
message in the log-message reference.
Fixes #1287.
Fixes #1285.
The #1285 client log also reports a certificate trust failure; the
client must trust the server certificate or its issuing chain. Full TLS
1.2 renegotiation still uses Grizzly's existing blocking path and is
outside this fix; the configured timeout is restored for that path.
Validation:
- Initial fix: Grizzly test suite on Java 11 and Java 21, 1,058 tests
passed on each; LDAP connection limits and transport integration tests
on Java 21, 41 tests passed.
- Initial fix: OpenDJ 5.2.0 with the patched filter and default
4096-byte buffers succeeded on LDAPS and the administration connector
with a large certificate and `--trustAll`.
- Review follow-up: `GrizzlyLDAPListenerTestCase` on Java 21, 28 tests
passed; the seven targeted timeout and closed-connection cases also
passed on Java 11 and Java 21.
- Review follow-up: `LDAPConnectionLimitsTestCase` on Java 21, 24 tests
passed, including closed-connection and pending-upgrade refusal
diagnostics.
- Mutation checks confirmed that the tests detect removal of the
closed-connection guard, the disabled-timeout guard, deadline renewal,
and restoration of the renegotiation timeout.
- Second review follow-up: rendered the log-message appendix with
Asciidoctor and verified the new entry's anchor, ID, severity, and text
against the message definition.
- Deadline race follow-up: `GrizzlyLDAPListenerTestCase` passed 29 tests
on Java 21; the new race test passed separately on Java 11 and Java 21
and failed when the deadline generation check was removed.
- Exploratory Java 21 test using `ApplicationTrustManager`, a
three-certificate 7.7 KB chain, and up to 1,000 additional trusted
certificates: the original filter stalled for about 10 seconds; the
patched filter completed the handshake and LDAP bind when the chain was
trusted. An untrusted chain still failed certificate validation.