Skip to content

refactor!: drop the custom allocator (rakMalloc / OP_NEW / memory_override) (2/6) - #102

Open
Segfaultd wants to merge 9 commits into
refactor/modern-memory-1-safety-netfrom
refactor/modern-memory-2-allocator
Open

Segfaultd wants to merge 9 commits into
refactor/modern-memory-1-safety-netfrom
refactor/modern-memory-2-allocator

Conversation

@Segfaultd

Copy link
Copy Markdown
Member

Summary

Removes the custom allocator layer entirely: rakMalloc/rakFree/rakRealloc, OP_NEW/OP_DELETE, memory_override.h, alloca, and the _FILE_AND_LINE_ allocation-tracking arguments on the allocation paths. Allocation is now plain new/std::make_unique/std::vector, with ownership moved into smart pointers where it was manual.

  • DataCompressor::DecompressAndAllocate now rejects an unbounded size taken from the network instead of allocating it
  • StringCompressor clears its decode buffer before an unknown-language early return
  • RakPeer::GetLiveRemoteSystemIndexEntriesForTesting() (test-only) replaces the rakMalloc hook counting in master's RemoteSystemLookupTests; the converted test still fails when the leak is put back

Breaking

Applications that installed custom allocators through rakMalloc_Ex and friends, or called OP_NEW/OP_DELETE, must use standard allocation. See the migration guide in 6/6.

Notes

  • Master's soak test asserted that rakMalloc blocks don't grow between waves. That hook no longer exists, and the remaining operator new check is coarser. The unit test above is now the precise guard for that leak; LeakSanitizer covers the rest.

Test plan

  • Debug -Werror build; unit tests plus the Reliability/Simulated/RakPeer/Encrypt/Mtu/Split integration tests

Commits

  • refactor!: allocate with new/delete and std::malloc instead of OP_NEW/rakMalloc
  • refactor!: remove memory_override, dlmalloc hooks and alloca
  • fix: allocation-free unicast in SendImmediate; review follow-ups
  • refactor!: return owned buffers as unique_ptr, replace RakNetSmartPtr with shared_ptr
  • fix: bound DataCompressor output size; restore CRLF line endings
  • style: clang-format DataCompressor tests
  • perf: stack buffer for StringCompressor decode instead of zero-filled vector
  • fix: clear StringCompressor decode buffer before an unknown-language early return

Stack, merge in order: 1/6 safety net, 2/6 allocator, 3/6 core transport, 4/6 plugins, 5/6 Table + extensions, 6/6 delete DataStructures. Each PR targets the branch of the one before it. Breaking change for the next major version; no version bump yet.

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 2

  • main
  • master

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3a56de16-20b3-4319-89cd-01d30169b383

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-2-allocator branch from 78f100e to 39465ca Compare November 16, 2026 11:43
Delete memory_override.h/.cpp and alloca.h together with the SetMalloc/
SetFree/SetRealloc hook API, UseRaknetFixedHeap and the _USE_RAK_MEMORY_OVERRIDE,
_LINK_DL_MALLOC, USE_ALLOCA and MAX_ALLOCA_STACK_ALLOCATION defines. Headers
now include only what they use. alloca sites (reliability layer split,
string compressor, file list, peer send list, profanity filter) use
std::vector / std::string. Rewrite the custom memory management page: replace
global operator new/delete or link a malloc replacement.
… with shared_ptr

Owned-buffer APIs now fill std::unique_ptr<unsigned char[]> (allocated with
new[], matching the deleter):
- BitStream::CopyData(std::unique_ptr<unsigned char[]> &) replaces the
  unsigned char ** overload; an empty stream yields null and returns 0
  instead of asserting
- BitStream::ReadAlignedBytesSafeAlloc (both int/unsigned overloads)
- DataCompressor::DecompressAndAllocate
- HuffmanEncodingTree::CharacterEncoding::encoding (internal owner of the
  CopyData result)

Callers updated: HuffmanEncodingTree, TableSerializer (the Table cell keeps
its own malloc'd copy via Cell::Set), bit_stream fuzzer, BitStreamBounds test.

RakNetSmartPtr (smart_ptr.h) is deleted; Lobby2 BinaryDataBlock handles use
std::shared_ptr. Not converted: ReliabilityLayer::Receive (internal, feeds
Packet::data), CloudAllocator callbacks, Get*OfflinePingResponse/GetSessionConfig
(non-owning views).
DecompressAndAllocate checks every read, rejects payloads longer than the stream and declared sizes above the payload bit count (each decoded byte costs at least one bit), and never asserts on network data. Restores the CRLF line endings the previous commit converted to LF in seven files.
A ReliabilityLayer that was never given SetSplitMessageProgressInterval read the interval uninitialised. Where the memory held a small nonzero value, the receiver emitted ID_DOWNLOAD_PROGRESS while reassembling a split message and handed it out ahead of the message itself: ReassemblesMessageOfTensOfThousandsOfFragments then took a 1353-byte progress message for its 32 MiB message on the macOS runner. The new test builds the layer over memory holding 1 in every word.
@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-2-allocator branch from 39465ca to d622b2d Compare November 16, 2026 13:58
@Segfaultd
Segfaultd force-pushed the refactor/modern-memory-1-safety-net branch from efb84bf to 7464dfd Compare November 16, 2026 13:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant