Skip to content

refactor!: delete DataStructures, CI guard, migration guide (6/6) - #106

Open
Segfaultd wants to merge 22 commits into
refactor/modern-memory-5-table-extensionsfrom
refactor/modern-memory-6-delete-ds
Open

Segfaultd wants to merge 22 commits into
refactor/modern-memory-5-table-extensionsfrom
refactor/modern-memory-6-delete-ds

Conversation

@Segfaultd

Copy link
Copy Markdown
Member

Summary

  • Removed: every ds_* header, the DataStructures namespace and _FILE_AND_LINE_. What remains as helpers in MafiaNet::: vector_util.h, instance_ptr.h, locked_queue.h, object_pool.h, range_list.h, table.h.
  • CI guard: the linux-native job fails if DataStructures::, ds_*.h, rakMalloc/rakFree, OP_NEW/OP_DELETE, memory_override.h, alloca(, RakNetSmartPtr, _FILE_AND_LINE_ or MAX_UNSIGNED_LONG reappear.
  • Docs: a migration guide (docs/advanced/migrating-to-std-containers.rst), utilities API docs, and unreleased-breaking entries in the changelog and README.
  • Final review fixes, each with a test that fails without it:
    • MSVC /W4 /WX: removes 11 size_t narrowing warnings that would break that build.
    • NAT punchthrough: a request that targets the sender itself is rejected; it used to leave a dangling pointer.
    • TableSerializer: the table built from peer input is capped at 1,000,000 cells.
    • RPC4: a malformed ID_RPC_PLUGIN packet is dropped instead of over-read and asserted on.
    • Table cells: text no longer goes through printf-style formatting, and lengths are no longer cast unchecked.
    • Router2: the load-weighted ping is computed in 64 bits.
    • Asserts on peer input: a too-short ID_CONNECTION_REQUEST_ACCEPTED and full TeamBalancer teams are dropped instead of asserting.
    • Autopatcher: repository error strings no longer carry source paths, and the load notifier is called outside ThreadPool's input lock. ThreadPool's mutex is no longer recursive on Windows, so calling it inside the lock would deadlock.
  • GCC + ASan on AArch64: ASan's use-after-return check is turned off there, because the toolchain never releases those frames. Turning it off exposed two old uninitialised-member bugs, fixed here: splitMessageProgressInterval and the FileListNode fields.

Breaking

See docs/advanced/migrating-to-std-containers.rst.

Test plan

  • Debug with samples: 1019/1019
  • asan preset 1015/1015, tsan preset 1015/1015, FuzzCorpus 6/6 (the four extra Debug tests are only built with samples)
  • Docker Ubuntu 24.04, Debug and Release: all pass; Docker LeakSanitizer (detect_leaks=1): no leaks
  • Format check, CI guard grep, Clang narrowing/shadow sweep vs master: 0 new warnings
  • MSVC: not verifiable locally; Windows CI is the check

Follow-ups (pre-existing, not fixed here)

  • Router2::OnReceive reads data[1] without a length check
  • AutopatcherPostgreRepository.cpp:1013 puts a server path into a string that can reach clients
  • TeamManager host team switch handled twice; Cloud unread flag bit written on reads; UDPProxy coordinator ping lists; FileList asserts on peer data; Table filter-query wire bugs; Lobby2 leaks and missing re-sort; an audit of format-string calls
  • Enable detect_leaks=1 in the CI ASan job

Commits

  • refactor!: remove the DataStructures namespace
  • refactor!: drop FILE_AND_LINE allocation tracking parameters
  • fix: include where the DS headers used to provide it
  • docs: restore the RakString, RakWString and PointGridSectorizer API reference
  • ci: forbid DataStructures, rakMalloc and alloca
  • docs: migration guide
  • docs: list every DependentExtensions signature change in the migration guide
  • build: disable ASan fake stack for GCC on AArch64
  • fix: initialise splitMessageProgressInterval
  • fix: initialise FileListNode members
  • fix: remove size_t narrowing that breaks the MSVC /W4 /WX build
  • fix: reject a NAT punchthrough request that targets the sender itself
  • fix: bound the table TableSerializer builds from peer input
  • fix: drop malformed ID_RPC_PLUGIN packets in RPC4 instead of asserting
  • fix: stop Table cells formatting their text and casting unchecked lengths
  • fix: compute Router2's load-weighted ping in 64 bits
  • fix: drop a too-short ID_CONNECTION_REQUEST_ACCEPTED instead of asserting
  • fix: drop TeamBalancer requests when every team is full instead of asserting
  • test: make PeerSystemList cleanup ASSERT-safe and ObjectPool counts relative
  • fix: print the NATCompleteServer system count with %zu
  • fix: keep source paths out of Autopatcher repository error strings
  • style: drop stale declarations and comments, include what RakAssert needs
  • fix: value-initialise RangeList::Deserialize locals and pin the Insert quirk
  • fix: call the Autopatcher load notifier outside ThreadPool's input lock
  • docs: document the final fix wave in the migration guide and changelog

Stack, merge in order: 1/6 safety net, 2/6 allocator, 3/6 core transport, 4/6 plugins, 5/6 Table + extensions, 6/6 delete DataStructures. Each PR targets the branch of the one before it. Breaking change for the next major version; no version bump yet.

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 2

  • main
  • master

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 06999917-515e-4e38-bc85-1bbca576c3e6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-5-table-extensions branch from 64f2cb0 to b4a5c4e Compare November 16, 2026 11:43
@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-6-delete-ds branch from e743d22 to 4624462 Compare November 16, 2026 11:43
Delete the last ds_*.h headers (ds_list, ds_queue, ds_ordered_list,
ds_map, ds_hash, ds_heap, ds_linked_list, ds_memory_pool, ds_multilist)
with no forwarding headers, and drop them from the CMake header list.

Convert the stragglers that still used them to std:: containers:
ConsoleServer, LogCommandParser, Rackspace, UDPForwarder,
VariableDeltaSerializer (receipt history on std::map, ChangedVariablesList
on plain new/delete instead of a MemoryPool), VariableListDeltaTracker and
the Linux _findfirst emulation. Files that got <assert.h> through the DS
headers include <cassert> themselves. Remove a commented-out RakString
benchmark main that was written against DataStructures::List.

Delete the two legacy Xcode projects (RakNet_IOS_XCode, Samples/iOS
ChatClient): every source path they list, the ds_* headers included, no
longer exists. Delete docs/api/data-structures.rst and describe std::
containers in the system overview, the virtual-worlds guide and CLAUDE.md.

Add VariableDeltaSerializer unit tests for receipt tracking.
ReliabilityLayer::AllocInternalPacketData(packet, numBytes, allowStack)
and FreeInternalPacketData(packet) lose the file and line parameters
that only fed the removed allocator tracking.

Where file and line are printed (Router2 diagnostics, the CCRakNetUDT
CapMinSnd debug print, the Autopatcher PostgreSQL error strings and the
SQLiteClientLogger rakSqlLog/rakFnLog macros) callers now pass
__FILE__, __LINE__ explicitly. The remaining uses were in comments.

Remove _FILE_AND_LINE_ from defines.h together with the unused memory
pool page sizes BUFFERED_PACKETS_PAGE_SIZE and INTERNAL_PACKET_PAGE_SIZE,
and the commented-out Native Client ProcessChromePacket.
On Linux the deleted ds_* headers were the only source of <cstring> for the _findfirst emulation and UDPForwarder. List _FILE_AND_LINE_ and the memory pool page sizes in the removed-API section of the memory management guide.
…eference

The deleted data-structures page was their only API reference. Move them to a new api/utilities page and list every remaining container helper in the system overview.
Fail the build when a removed container, allocator macro, alloca, RakNetSmartPtr, _FILE_AND_LINE_ or MAX_UNSIGNED_LONG reappears in Source, Tests, Samples, DependentExtensions or Benchmarks. The InternalPacket::NORMAL comment named rakMalloc; it now says std::malloc, which is what the code uses.
Add docs/advanced/migrating-to-std-containers.rst: the container and allocator mappings, every removed header, macro and type, the changed public signatures by subsystem, the non-copyable types, the behaviour changes and the Windows CRT/toolset rule. Add an Unreleased (breaking) section to the changelog and README.
…n guide

Cover Lobby2 (Lobby2Client ignore list, Lobby2Server SendMsg/HasAdminAddress/HasRankingAddress/SendRemoteLoginNotification, PGSQL helpers, Steam, RoomsContainer/RoomTypes/RoomsPlugin), RPC3, SQLite3Plugin, Autopatcher and PostgreSQLInterface from the full header diff. Note that public headers no longer include <cstring>, <new> and friends transitively. Complete the changelog and README fix lists.
GCC 13/14 on arm64 never releases stack-use-after-return fake frames, so
after ~16k calls per thread every instrumented call scans a full size
class. The network thread then falls behind and
Rm3ReceiveThread.ConnectionsAreServedInParallelWhileReplicasComeAndGo
times out under ASan. x86-64 GCC and Clang are unaffected.
FileList::Deserialize left isAReference and the length fields
uninitialised; UBSan flagged the copy of an invalid bool.
The self check compared the sender with ca->recipient before it was assigned, so it was always false: the same ConnectionAttempt went into the sender's list twice and OnClosedConnection freed it once, leaving a dangling entry. Compare the looked-up recipient instead and answer ID_NAT_TARGET_NOT_CONNECTED.
A zero-entry row is 8 bytes on the wire but one Cell per column in memory, so within the existing 10000-column and 100000-row caps under 1 MB of input could demand about 1e9 allocations. DeserializeTable now rejects a row count the unread bits cannot hold and a table above 1e6 cells, before building any row.
A one-byte ID_RPC_PLUGIN packet read data[1] past the end, and an unknown sub-identifier hit RakAssert(ID_RPC4_RETURN) and was then stored as a pending blocking call's return value. Both are now dropped.
…gths

Cell::ToString built a STRING cell's RakString through the printf-style constructor, so peer-supplied text such as %n was interpreted as a format; it is now assigned verbatim. OwnedLength clamps the double before converting it to size_t (undefined for infinity or values past the range), and SetByType treats a binary length that is not a valid int as a zero-length blob instead of casting it.
ping * (usedForwardingEntries + 1) multiplied two relay-supplied 16-bit figures in int: signed overflow, and a wrapped weight could make the busiest relay look like the best one.
…ting

A connected remote could send a truncated connection accept and hit RakAssert(0), aborting a Debug build. The message is now dropped.
…serting

OnStatusUpdateToNewHost, OnRequestAnyTeam and OnRequestSpecificTeam hit RakAssert("Too many members asking for teams!") when no team had room, which a remote member can cause; a Debug build aborted. The request is now dropped.
…elative

GetSocketsReturnsBoundSockets owns its peer through InstancePtr so an early ASSERT return still destroys it; AcquireAfterExhaustionAllocates compares the live count with its starting value; TcpOutgoingBufferTests includes <algorithm> for std::min and std::equal.
GetLastError() text can be sent to clients. _RETAIL builds used to blank the file and line through _FILE_AND_LINE_; since that macro is gone they carried the server's build paths. The file and line are dropped from these messages.
…eeds

Remove RakPeer's unused incomingQueueMutex (a leftover of the pre-LockedQueue incoming queue) and point the bufferedCommands comment at RunUpdateCycle; delete socket_layer.h's commented-out SendTo declarations that still took file/line; group udp_forwarder.h's standard includes. http_connection2, telnet_transport and udp_forwarder included <cassert> only for RakAssert, which defines.h expands to assert(); they now include mafianet/assert.h, the header that provides it.
…t quirk

min and max are now value-initialised before DeserializeSingleRange fills them. A characterization test pins the merge quirk kept from DataStructures::RangeList (one past a middle range starts a new adjacent range), which must not change because it shapes the ACK/NAK bytes.
RemoveFromThreadPool called CallPatchCompleteCallback while holding LockInput. ThreadPool's queue mutexes are std::mutex now, not a recursive CRITICAL_SECTION as on Windows before, so a notifier that queued more work deadlocked. The aborted requests are counted under the lock and reported after it is released.
ThreadPool's mutexes are not recursive; Lobby2 binary buffers are std::malloc/std::free owned; the network-input list names RPC4, RakPeer and TeamBalancer as they now behave; TableSerializer limits, verbatim Table string cells, the NAT self-punch answer and Router2's 64-bit weight; Autopatcher error strings no longer carry file and line.
@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-6-delete-ds branch from 4624462 to eab97b6 Compare November 16, 2026 13:58
@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-5-table-extensions branch from b4a5c4e to d44d4d3 Compare November 16, 2026 13:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant