Add SPI for binding multiple datagram sockets per HTTP/3 bind target

[simonjbeaumont]

Motivation

Each HTTP/3 bind target currently binds a single UDP socket and installs one QUIC channel handler, tied to the next available event loop. As a result each bound address is served by one event loop so HTTP/3 throughput is single-threaded.

It's possible to bind multiple sockets to the same port with SO_REUSEPORT and have the system load-balance incoming traffic between the sockets in the group.

By default, the kernel will route datagrams based on a stable hash of the 4-tuple, so UDP traffic with a stable 4-tuple will arrive at the same socket.

Unlike TCP, where connections are identified by their 4-tuple, QUIC connection IDs are part of the UDP payload, which allows for QUIC connections to migrate between paths, so serving multithreaded QUIC with a simple SO_REUSEPORT group, will not support connection migration.

While userspace packet steering is possible, high-performance multithreaded QUIC servers take advantage of platform-specific kernel mechanisms to route datagrams by QUIC Connection ID instead of 4-tuple (e.g. sk_reuseport on Linux).

This PR adds support for binding multiple sockets in an SO_REUSEPORT group with an SPI for such a mechanism to participate in the server bootstrap.

Modifications

• Added protocol QUICDatagramSocketGroup (behind @_spi(QUICDatagramSockets)): conforming types provide a hook called on socket bind, and a connection ID generator for each socket.
• Added QUICDatagramSocketGroupFactory (behind @_spi(QUICDatagramSockets)): called during bootstrap for adopters to provide a socket group implementation, or nil.
• Added QUICConfiguration.datagramSocketGroupFactory (behind @_spi(QUICDatagramSockets)): Note, this is code-only config because it can't be expressed in Swift Configuration.
• Reworked serve to call addHTTP3Listeners (plural) (was addHTTP3Listener), which binds one socket per event loop in the event loop group using SO_REUSEPORT. The datagramSocketGroupFactory is consulted to determine the socket group implementation, if any, which is called with each bound socket and to provide a connection ID generator.
• Added tests that drive the SPI with a stub conformer.
• Note: If datagramSocketGroupFactory is not configured, the behavior remains as-was: only one socket is bound and tied to the next available event loop in the event loop group.
• Aside: Bumps Swift NIO QUIC dependency to 0.3.0, for the connectionIDGenerator parameter.
• Aside: Bumps Swift NIO dependency to 2.103.0, for the .so_reuseport socket option.

Result

A bind target can serve HTTP/3 across several event loops, with an optional socket group mechanism provided via SPI.

When no factory is configured, nothing is changed: one socket is used per bind target (without SO_REUSEPORT), tied to the next available event loop, with the default (random) connection ID generator.

[FranzBusch]

This looks sensible. letting @aryan-25 do the very detailed review and approval

[simonjbeaumont]

Thanks @aryan-25 for the review comments. I think I've addressed them all. PTAL.

[gjcairo]

LGTM once the open threads are resolved.

[aryan-25]

Thank you!

[simonjbeaumont]

@FranzBusch apparently this repo needs admin merges at the moment while there is no functional CI?

% swift test
...
􁁛  Test run with 191 tests in 41 suites passed after 5.180 seconds.

[FranzBusch]

The failure is due to a build error in swift-network-evolution

error: SwiftCompile normal x86_64 /swift-http-server/.build/checkouts/swift-network-evolution/Sources/SwiftNetwork/Protocols/FrameArray.swift failed with a nonzero exit code. Command line:     cd /
    builtin-SwiftPerFileCompile FrameArray.swift
/swift-http-server/.build/checkouts/swift-network-evolution/Sources/SwiftNetwork/Protocols/FrameArray.swift:71:23: error: lifetime-dependent value escapes its scope
 68 |     #if !NETWORK_EMBEDDED
 69 |     @_lifetime(borrow self)
 70 |     func bytes(at index: Int) -> RawSpan? {
    |          `- note: it depends on the lifetime of this parent value
 71 |         frames[index].bytes
    |                       `- error: lifetime-dependent value escapes its scope
 72 |     }
    |     `- note: this use causes the lifetime-dependent value to escape
 73 |    

That seems fixable.

[simonjbeaumont]

The failure is due to a build error in swift-network-evolution
...
That seems fixable.

Great! Are we just waiting on pulling a new version of swift-network-evolution through the dependency graph? Is someone working on that?

[josephnoir]

Yes, looking into it.