Skip to content

refactor!: plugins on std:: containers (4/6) - #104

Open
Segfaultd wants to merge 32 commits into
refactor/modern-memory-3-core-transportfrom
refactor/modern-memory-4-plugins
Open

Segfaultd wants to merge 32 commits into
refactor/modern-memory-3-core-transportfrom
refactor/modern-memory-4-plugins

Conversation

@Segfaultd

Copy link
Copy Markdown
Member

Summary

Every plugin moves to std:: containers:

  • ReplicaManager3, FullyConnectedMesh2, ConnectionGraph2, TeamManager/TeamBalancer, ReadyEvent, Router2
  • the sectorizers, RPC4, MessageFilter, StatisticsHistory, TwoWayAuthentication
  • CloudServer/Client, PacketizedTCP/TCPInterface, UDPProxy, ThreadPool

List-returning APIs now fill a std::vector<T>&.

Bugs fixed on the way, each with a test:

  • TeamBalancer accepted a team index ≥ the team count
  • std::sort on NaN statistics values was UB; they now sort last
  • MessageFilter used a hasher inconsistent with AddressOrGUID::operator==
  • PacketizedTCP had no message size cap; it is now 64 MiB by default
  • the TCP send buffer erased from the front (O(n²)); it now consumes through an offset
  • CloudServer: uploader count, row bounds, and the isUploaded flag
  • UDPProxyServer authentication
  • ThreadPool lifecycle fixes
  • format strings, the Huffman rename (huffman_encoding_tree*), DecodeArray sizes

Breaking

Signatures that took or returned DataStructures::List/OrderedList/Map now use std::vector/std::map. See the migration guide in 6/6.

Test plan

  • Debug -Werror build; unit tests plus the Reliability/Simulated/RakPeer/Encrypt/Mtu/Split integration tests

Commits

  • refactor!: ReplicaManager3 on std:: containers
  • refactor!: FCM2, ConnectionGraph2, Team*, ReadyEvent, Router2, sectorizers and RakVoice on std:: containers
  • fix: TeamBalancer rejects a team id equal to the team count
  • fix: delete ConnectionGraph2 copy operations
  • refactor!: rename ds_huffman_encoding_tree* to huffman_encoding_tree*
  • refactor!: Huffman tree and StringCompressor on std:: containers
  • refactor!: RPC4 on std:: containers
  • refactor!: MessageFilter, RelayPlugin and TwoWayAuthentication on std:: containers
  • refactor!: StatisticsHistory on std:: containers
  • refactor!: StringTable, VariadicSQLParser and CommandParserInterface on std:: containers
  • test: pass the '%' identifier to TwoWayAuthentication verbatim
  • fix: drop the public std::hash
  • fix: StatisticsHistory sort comparators order NaN last
  • refactor!: FileList, FileListTransfer and DirectoryDeltaTransfer on std:: containers
  • refactor!: TCPInterface, PacketizedTCP, HTTP and transports on std:: containers
  • refactor!: ThreadPool, CCRakNetUDT and RakNetRandomSync on std:: containers
  • refactor!: CloudClient and CloudServer on std:: containers
  • refactor!: NAT punchthrough, NAT type detection and UDP proxy on std:: containers
  • test: cover HTTPConnection queues when the extras library is built
  • test: make the simulated network's clock safe to read from a plugin thread
  • fix: include where it used to arrive through the DataStructures headers
  • test: brace an if around an EXPECT (GCC -Wdangling-else)
  • fix: ThreadPool::AddInput signals its event only while the threads run
  • fix: TCPInterface send buffer consumes from an offset, not a front erase
  • fix: Cloud upload-limit undo and response row count
  • test: a logged-in UDPProxyServer still sets up forwarding
  • test: pin RemovePendingRequest, PacketizedTCP::Stop and the duplicate NAT user fixes
  • test: no namespace-scope RakString in CloudApiTests; brace an ASSERT under if
  • style: clang-format UdpProxyLiveTests
  • fix: CloudServer marks an upload into a specific-subscription placeholder as uploaded

Stack, merge in order: 1/6 safety net, 2/6 allocator, 3/6 core transport, 4/6 plugins, 5/6 Table + extensions, 6/6 delete DataStructures. Each PR targets the branch of the one before it. Breaking change for the next major version; no version bump yet.

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 2

  • main
  • master

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: eb33eed8-86c7-4356-8f27-92e2a52bbf87

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-4-plugins branch from 50e293d to f887476 Compare November 16, 2026 11:43
@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-3-core-transport branch from e4ad4d4 to 2721afb Compare November 16, 2026 11:43
Every DataStructures::List in ReplicaManager3's API becomes std::vector:
DereferenceList, GetReplicasCreatedByGuid/ByMe, GetReferencedReplicaList,
GetConnectionsInVirtualWorld, GetGuidsInVirtualWorld,
GetConnectionsThatHaveReplicaConstructed, BroadcastDestructionList,
AutoCreateConnectionList, and the Connection_RM3 virtuals
GetConstructedReplicas, QueryReplicaList, QuerySerializationList and
SendConstruction. Overrides must change their parameter types.

constructedReplicaList stays sorted by Replica3LSRComp (referenceIndex) as a
sorted std::vector searched with lower_bound, so serialization order is
unchanged; downloadGroup is a std::deque.
…izers and RakVoice on std:: containers

Public out-parameters move from DataStructures::List to std::vector:
FullyConnectedMesh2 GetHostOrder, GetParticipantList and the virtuals
GetVerifiedJoinAcceptedAdditionalData / GetVerifiedJoinRequiredProcessingList;
TeamManager GetRequestedSpecificTeams, GetCurrentTeams, GetLastTeams,
GetTeamMembers, GetParticipantList, GetSortedJoinRequests; TeamBalancer
GetMemberIndexToSwitchTeams, GetOverpopulatedTeams; GridSectorizer and
PointGridSectorizer GetEntries; RakVoice GetActiveSpeakers.
ConnectionGraph2::GetParticipantList fills a std::set<RakNetGUID> (no caller
indexes it).

Sorted OrderedLists become sorted vectors (or std::map) over the same
comparators, so FCM2 host order, ReadyEvent and RakVoice index order and
ConnectionGraph2 connection order are unchanged.

Also fixes: FCM2 leaked each departing participant; ConnectionGraph2 never
freed its remote systems and inserted unread entries from a truncated
ID_REMOTE_NEW_INCOMING_CONNECTION; TM_World::Clear() left stale pointers in
its networkID lookups; TeamBalancer::AddTeamMember could index one past its
count table. Router2 picks the forwarding system with a strict ordering
(lowest weighted ping, then guid) instead of an inconsistent comparator.
The range checks on network-derived team ids used '>' where '>=' was
needed, so a remote member could name the team one past the last
(OnStatusUpdateToNewHost, OnRequestSpecificTeam), which AddTeamMember then
counted and later code read out of bounds. Such messages are now dropped
silently instead of asserting, AddTeamMember refuses unknown or unassigned
teams, and the table-growing branch that hid the bug is gone.
ConnectionGraph2 is exported (RAK_DLL_EXPORT) and now owns its remote
systems through std::unique_ptr. MSVC instantiates the special members of an
exported class, so the implicitly declared copy would fail to compile in the
DLL build. Delete it explicitly.
The Huffman tree headers lose the ds_ prefix: include
mafianet/huffman_encoding_tree.h and mafianet/huffman_encoding_tree_node.h.
No shim headers are left behind.
HuffmanEncodingTree keeps its nodes in one reserved std::vector and sorts
with a std::vector + lower_bound that reproduces the old linked-list
insertion order, so encodings stay byte-identical (StringCompressorGolden).
StringCompressor holds its trees in std::map<int, std::unique_ptr<>>.

Decoding treats its input as untrusted:
- node weights are 64-bit, so network-supplied frequencies near 2^32
  cannot wrap the sum and degenerate the tree;
- DecodeArray returns a size_t count (never negative in the caller) and
  DECODE_ERROR instead of following a missing branch or an empty tree;
  the BitStream overload returns false in that case;
- DecodeString terminates inside the buffer using unsigned arithmetic.

GenerateTreeFromStrings with empty input keeps the language's tree
instead of leaving a freed pointer in the map. The dead and
uncompilable _CSTRING_COMPRESSOR overloads are removed.
HuffmanEncodingTree and StringCompressor are explicitly non-copyable.
RPC4 keeps functions and slots in std::unordered_map keyed by RakString,
slots as std::unique_ptr<LocalSlot> (sorted std::vector of slot objects),
and local callbacks in a std::map of sorted name vectors.

API changes:
- protected GetLocalSlotIndex(const char *) -> public
  LocalSlot *GetLocalSlot(const char *), nullptr if absent; the pointer is
  valid until UnregisterSlot() or the plugin is destroyed.
- protected InvokeSignal(HashIndex, ...) -> public
  InvokeSignal(LocalSlot *, ...); a nullptr slot is a no-op.
- RPC4::LocalSlot is only forward-declared; the public localSlots member
  is now protected. LocalCallbackComp is gone.
- RPC4 is explicitly non-copyable.
- UnregisterLocalCallback returns true whenever it removed the pair, as
  documented, not only when the message id's last function went.

Identifiers are stored and looked up verbatim. Received function names
were passed through RakString's printf-style constructor, so a '%' in a
name from a remote peer made vsnprintf read stray varargs.

std::hash is specialized for RakString, RakNetGUID and AddressOrGUID with
their existing ToInteger/ToUint32 hashes.
…:: containers

MessageFilter: filter sets in an id-sorted std::vector<std::unique_ptr>
(GetFilterSetIDByIndex keeps id order), systems in an
std::unordered_map<AddressOrGUID, FilteredSystem> (Update walks a
snapshot, since its callbacks may change the map), allowed RPC4 names
in a std::set<RakString>. autoAddNewConnectionsToFilter now starts at
its documented -1 instead of uninitialized.

RelayPlugin: participants owned by std::unordered_map<RakString,
std::unique_ptr<>> and indexed by GUID in a second unordered_map; rooms
in std::vector<std::unique_ptr<RP_Group>> (SwapRemove keeps the old
removal order, which the group list reports).

TwoWayAuthentication: public outgoingChallenges is a std::deque,
NonceGenerator::generatedNonces a std::vector<std::unique_ptr<>>,
passwords an std::unordered_map. Received password identifiers are
looked up verbatim instead of through RakString's printf-style
constructor.

All three classes, and NonceGenerator, are explicitly non-copyable.
SetAllowRPC4 stores names verbatim, matching RPC4.
API changes:
- GetHistorySorted(..., List<TimeAndValueQueue *> &) ->
  std::vector<TimeAndValueQueue *> & (still appends; ties by key).
- GetUniqueKeyList(List<RakString> &) -> std::vector<RakString> &
  (replaced; order unspecified, as with the hash it came from).
- TimeAndValueQueue::values and ResizeSampleSet's histogram:
  Queue<TimeAndValue> -> std::deque<TimeAndValue>.
- StatisticsHistory and StatisticsHistoryPlugin are explicitly
  non-copyable.

Tracked objects are an id-sorted std::vector<std::unique_ptr<>>, so the
index accessors keep id order; each object's queues are owned by an
std::unordered_map<RakString, std::unique_ptr<>>. TrackedObject's
destructor walked an empty list and leaked every queue; they are freed
now.

The discrete branch of TimeAndValueQueue::MergeSets indexed lhs with the
rhs index, emitting the wrong samples or reading past the end of lhs.
…on std:: containers

API changes:
- VariadicSQLParser::GetTypeMappingIndices / ExtractArguments /
  FreeArguments take std::vector<IndexAndType> instead of
  DataStructures::List (PostgreSQLInterface updated).
- StringTable and CommandParserInterface keep their sorted
  OrderedLists as sorted std::vectors searched with lower_bound over
  the unchanged StrAndBoolComp / RegisteredCommandComp, so StringTable's
  wire indices and the command list order are unchanged. StringTable is
  explicitly non-copyable.

StringTable::DecodeString no longer RakAsserts on an out-of-range index
read from the network; it rejects it. A duplicate AddString frees the
copy it made instead of leaking it.

log_command_parser.{h,cpp} include ds_list.h and assert.h themselves
instead of through command_parser_interface.h.
The literal went through RakString's printf-style constructor at the call site, so the test read stray varargs (caught by ASan).
AddressOrGUID::operator== matches on the GUID or the address, while the hash used the GUID when set and the address otherwise, so equal keys could hash differently: not a valid std::hash. MessageFilter, its only user, now hashes its system table with a private MessageFilterSystemHash that documents the inherited mismatch. Equality semantics are unchanged.
GetHistorySorted now uses std::sort, which needs a strict weak ordering; a NaN sort value (a NaN sample, or inf - inf) broke it, which is undefined behaviour. Both comparators put NaN after every number and tie NaNs on the key, so the order is total and deterministic.
…containers

ByteQueue is gone: TCPInterface buffers outgoing bytes in a std::vector and
PacketizedTCP reassembles each connection's stream in one. PacketizedTCP now
bounds the peer-supplied length prefix (SetMaxMessageLength, default 64 MiB):
a longer prefix drops the connection instead of being buffered.
Ordered lists become sorted vectors or std::set/std::map with the same
ordering, so query results keep their order. GetRequests are keyed by
request id in a std::map: the old comparator never returned 1.
…: containers

NatTypeDetection's receive buffers become LockedQueue<unique_ptr>. UDPProxyServer
now honours a forwarding request only from a coordinator it logged in to (the
old index check never failed).
…hread

UDPProxyServer's UDPForwarder thread reads the time while Step() advances it;
TSan flagged the plain global.
…ures headers

libstdc++ does not pull it in transitively; dyn_dns.cpp failed to build on Linux.
Before StartThreads the event's pthread condition is uninitialized; signalling
it hung ThreadPool.InputQueueKeepsEachCallbackWithItsInput on glibc.
Erasing the sent prefix after every send shifted the whole backlog under
outgoingDataMutex: O(n^2) for a large message. The prefix is now dropped only
once it is at least half the buffer, so draining n bytes moves fewer than n.
A rejected over-limit Post left uploaderCount raised, so the next real upload
was never announced to other servers. A response's row count is peer input:
reading stops at the first row the stream does not hold, rows are
value-initialized, and CloudKey/CloudQueryRow::Serialize report read failure.
…under if

A global CloudKey was built during static initialization, before RakString's
allocator in another translation unit (LeakSanitizer reported its pool).
GCC -Wdangling-else flagged the unbraced ASSERT.
…lder as uploaded

isUploaded stayed false, so an over-limit re-post erased the key from the
client's uploadedKeys (its Release was then ignored and its data outlived the
connection), and an unsubscribe deleted the live upload. Also tests the
over-limit overwrite branch's uploaderCount undo.
A worker counted itself out of numThreadsRunning and only then read the pool to find and run the per-thread data destructor. StopThreads() returns when that count reaches 0, so the owner could destroy the pool while workers still read it: TSan caught WorkersProcessEveryInputExactlyOnce's workers reading the stack slot of a pool that had gone out of scope. The destructor now runs first and the count is the worker's last touch of the pool.
@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-4-plugins branch from f887476 to 81ee9cd Compare November 16, 2026 13:58
@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-3-core-transport branch from 2721afb to 1965815 Compare November 16, 2026 13:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant