Spatial memory: object centric particle index over exact Gaussian world state

[ruvnet]

Problem

RuVector already has a GaussianSplat bridge for point cloud clusters with center, color, opacity, scale, label, point count, and an optional dense trajectory. This is useful for rendering, but it is not an object centric memory abstraction and it grows poorly when downstream agents need persistent object identity, compact task context, and repeated spatial retrieval.

Fresh research: ParticleSplat, arXiv:2609.19463, submitted 2026-09-16, learns a compact set of 3D latent particles and decodes each particle into a local Gaussian field. The paper reports 73.0 percent overall success on single view RLBench versus 46.6 percent for its DLP ablation with the same policy, 83.0 percent on multi view RLBench versus 59.4 percent for DLP, and 56.1 percent on MimicGen versus 48.1 percent for 3D DLP. The representation uses 20 latent particles while its image plane Gaussian decoder uses 128 squared Gaussians, a nominal 819.2 to 1 latent count reduction before decoder expansion.

Sources:
https://arxiv.org/abs/2609.19463
https://lyuxinghe.github.io/ParticleSplat-website/
https://github.com/lyuxinghe/ParticleSplat

Important limitation: the official GitHub repository currently says the code release is coming soon. Do not add it as a dependency or claim reproduction yet.

Why it matters

RuVector should distinguish authoritative spatial state from compact task memory.

Exact geometry, timestamps, coordinate frames, covariance, track identity, and provenance should remain deterministic and witnessed. A small object centric particle index can then provide fast semantic retrieval, manipulation context, and browser or agent working memory without replacing the authoritative world representation.

Current architecture

crates/ruvector-robotics/src/bridge/gaussian.rs currently maps point cloud clusters to GaussianSplat values. Each Gaussian stores center, visual attributes, point count, semantic label, and a raw trajectory vector.

Recent RuVector HNSW fixes also restored exact stored point coverage and improved recall on measured benchmarks, so new spatial memory experiments should use the corrected index as their retrieval baseline rather than building a duplicate vector store.

Observed limitation

1. No explicit object level memory identity above individual Gaussians.
2. No compact object state that can reference a set of exact Gaussians and trajectories.
3. Dense trajectory vectors grow with observation duration.
4. Current viewer oriented representation does not carry uncertainty or source evidence at object granularity.
5. Agents must either retrieve too much Gaussian state or rely on lossy semantic summaries.

Proposed architecture

Add an experimental SpatialParticle layer in ruvector-robotics or the existing spatial memory crate selected by architecture review.

A particle is an index over authoritative state, not a replacement for it.

Suggested fields:

1. stable object or region id
2. world frame id
3. exact anchor center and covariance
4. bounded extent or oriented box
5. semantic embedding and optional bounded label set
6. motion state summary with links to exact trajectory anchors
7. list or range of authoritative Gaussian ids
8. source RVF roots or retrieval receipt ids
9. first seen and last seen timestamps
10. confidence ceiling derived from source evidence
11. representation version

The object centric layer may be compact and learned. Geometry and provenance remain exact.

Experiment arms

1. Current dense GaussianSplatCloud retrieval.
2. Current Gaussian cloud plus deterministic object clustering.
3. Exact object anchors plus compressed semantic particle embeddings.
4. Optional learned particle encoder only after arms 1 through 3 establish a deterministic baseline.

Do not copy ParticleSplat architecture directly until its source and license are actually released and reviewed.

Benchmark plan

Use RuView or robotics sequences with object tracks and exact geometry. Include static objects, moving people, repeated occlusion, object reappearance, scene rearrangement, and long horizon operation.

Measure:

1. bytes retained per object hour
2. compression ratio against dense Gaussian and trajectory state
3. object retrieval recall at k
4. spatial center and extent error
5. temporal last seen error
6. task query success
7. p50 and p95 retrieval latency
8. index build and update latency
9. peak memory
10. browser WASM memory where exposed
11. source receipt coverage
12. deterministic output for the deterministic baseline

Success metrics

Promotion to production candidate requires all of:

1. At least 10 times smaller task working memory than dense Gaussian plus trajectory retrieval on a 24 hour equivalent sequence.
2. No more than 2 centimetres additional median geometry error because exact anchors remain authoritative.
3. No more than 3 percentage points loss in object retrieval recall at k versus dense state.
4. p95 retrieval below 50 milliseconds on x86 and below 100 milliseconds on the reference ARM64 edge device.
5. 100 percent source evidence coverage for every particle returned as authoritative context.
6. Zero confidence amplification above the strongest source evidence.
7. WASM path, if implemented, remains within a documented memory ceiling and never gains file or network capability beyond the host contract.

Security and privacy

1. Object particles can concentrate location and identity information, so privacy classification must be at least the maximum of their source state.
2. Semantic labels and embeddings are untrusted residual data and cannot execute instructions or override exact state.
3. Bound object counts, embedding dimensions, Gaussian references, and trajectory anchor counts to prevent memory exhaustion.
4. Validate every RVF root or source receipt before returning an authoritative particle.
5. Protect against object identity linkage across privacy domains.
6. A malicious or poisoned learned encoder cannot modify authoritative geometry or provenance.
7. Browser WASM serialization must reject oversized and non finite inputs.

Compatibility and migration

Additive experimental layer only. Existing Gaussian, HNSW, RVF, and viewer formats stay unchanged. No RVF wire change until a dedicated profile with golden fixtures and migration rules is justified by benchmark results.

MetaHarness plan

Use stable ruvnet/metaharness v0.4.4 with repository specific objectives for memory reduction, retrieval quality, geometry preservation, latency, determinism, and provenance coverage. Darwin may tune bounded clustering or admission thresholds only after a frozen real holdout is created. Any candidate that changes authoritative geometry, loses receipt coverage, or regresses corrected HNSW retrieval fails immediately.

Use RuVector native retrieval and WASM rather than introducing another vector database.

Rollback

Disable the particle feature flag and fall back to existing dense Gaussian and trajectory retrieval. No source data migration is required.

Definition of done

1. ADR defining authoritative state versus particle residual state.
2. Deterministic non learned baseline implementation.
3. Unit, property, adversarial, serialization, and long horizon tests.
4. x86, ARM64, and WASM benchmarks where supported.
5. Before and after storage, recall, latency, and memory report across at least three repeated runs.
6. Optional learned path only after open source license review and independent reproduction.
7. MetaHarness evidence receipt bound to the exact tested commit.

Production classification

Experimental. The representation concept is promising, but the fresh upstream repository does not yet contain released code.