ingot

⚠️ Work in progress. ingot is under active development and its design is changing rapidly: interfaces, on-disk formats, and the Forge upload path are all still in flux. Expect breaking changes.

ingot is an S3 gateway over the Forge network, built around a Merkle Search Tree (MST) ported from bluesky-social/indigo.

It speaks the S3 REST protocol on one side and the Forge UCAN control plane on the other, and runs two ways: as an embeddable Go library a host process imports in-process, or as a standalone daemon (ingot serve). Either way it presents each S3 bucket as a per-bucket MST, uploads object bodies to Forge storage as content-addressed blobs before a write is acked, journals the catalog to a local per-bucket log, and ships sealed catalog segments to Forge as a guppy-style edge client. Reads fall through local tiers and finally to the network.

For the architecture as it operates today, see DESIGN_NOTES.md; for the target design, docs/architecture.md; for the as-built diagram set, docs/diagrams.md; for the log internals, logstore/README.md.

Why a forked MST, not a direct dependency

The MST in mst/ is a fork of indigo's, not an import of it. atproto's MST is exactly the data structure ingot wants — an ordered, content-addressed key/value map that commits to an entire keyspace under a single root CID — but ingot ports it rather than depending on it, for three reasons:

• A different key space. atproto validates keys as repo record paths (collection/rkey, with charset and length limits). ingot relaxes this to accept arbitrary S3 object keys (any non-empty UTF-8 string up to 1024 bytes, NUL excluded) — a behavioral change to the structure, not just a repackage.
• A small dependency surface. Importing indigo's mst subpackage would pull the broader atproto module graph in for one self-contained data structure. The fork's only external dependency is go-cid.
• Freedom to diverge. The on-disk format starts identical to atproto's, but cross-implementation compatibility is intentionally not a goal.

How buckets map onto the MST and Postgres

Each bucket is one MST: an ordered map from object key → value block. A key's value is its object manifest (size, BLAKE3 digest and tree, md5, the S3 system headers and user metadata, and the ordered list of content-addressed body blobs that cover it); a key with retained versions groups them under a per-key leaf (docs/s3-versioning.md). Because every node is addressed by its own hash, a bucket rolls up to a single root CID — a cryptographic commitment to the exact set of objects it holds — and the tree's ordered keys make S3 prefix/delimiter listings fall out of ordinary traversal.

Writes are functional: a PUT or DELETE rewrites only the nodes on the path from the changed key up to the root (every other node is immutable and shared), producing a new root CID. ingot journals the changed blocks to the local log, then compare-and-swaps the bucket's root in Postgres from the old CID to the new one. That split is the heart of the design:

• The MST is the data — immutable, content-addressed, self-verifying, and shippable to Forge exactly as it sits on disk.
• Postgres is the mutable index — the authoritative current root per bucket (the CAS keeps a bucket single-writer-correct), the segment lifecycle, and the blob claim ledger, all under the ingot schema.

Bodies and catalog

A write splits into two paths:

• Bodies stream into a local spool (the BLAKE3 tree and md5 in one pass), split into blobs of at most max_blob_size, and upload to Forge storage synchronously: /blob/add via sprue, an HTTP PUT to the allocated piri, and the accepted location commitment, all before the catalog commit. Identical bytes dedup by digest; a reference index (blob_refs) counts which versions still claim each blob and releases it (/blob/remove) when the count reaches zero.
• The catalog (the dag-cbor MST nodes and manifests) journals to a local per-bucket log, seals into CAR segments by size or age, and ships to the bucket's Forge space in the background; shipping advances the bucket's forge_root_cid under a guard. See logstore/README.md.

Reads fall through tiers: local disk (cache, then spool), the local log (catalog blocks), and finally the network, resolved by a local locator (blob_locations + shard_inclusions) and fetched with a ranged content/retrieve against the storing piri. The two routes are drawn side by side in docs/diagrams.md.

Running it

As a library (fx). A host adds the module and provides a logger, a Postgres pool, and the agent identity (via Config):

app := fx.New(
    // host provides: *zap.Logger, *pgxpool.Pool, libforge identity.Identity
    ingot.Module(cfg), // cfg sets UploadServiceURL/DID, AuthServiceURL/DID, ...
)

There is also a non-fx escape hatch — New(ctx, ServerConfig, ServerDeps) + Server.Start/Server.Stop — for hosts and tests that construct the collaborators themselves.

As a daemon (cmd/, cobra/viper/fx):

ingot serve --config /etc/ingot/config.yaml   # the gateway
ingot whoami                                  # agent DID (and key DID) + sprue/hilt endpoints
ingot version
ingot blake3 hash < file                      # the object CID (x-cid) of the data
ingot blake3 hash --bao < file                # plus the ChunkLog and Bao Outboard the Blake3 attribute carries
ingot blake3 hash --bao --chunk-log 4 < file  # an outboard at iroh's 16 KiB block instead of ingot's block size
ingot blake3 verify <cid> < file              # whole-object check against an x-cid
ingot blake3 range --outboard "$OUTBOARD" --chunk-log 12 5000-200000   # aligned Range to fetch
ingot blake3 verify --outboard "$OUTBOARD" --chunk-log 12 --offset 0 <cid> < part  # ranged check

blake3 range and the ranged verify take the Blake3 attribute of GetObjectAttributes: its ChunkLog (the Bao block size as a base-2 exponent of 1 KiB chunks), and its Outboard either inline as the base64 string from the XML (--outboard) or from a file holding that string or the raw bytes (--outboard-file). A mismatch exits 1; any other failure exits 2.

serve requires Postgres (postgres_dsn), the sprue edge client (upload_service_url/_did), and the hilt auth/tenant service (auth_service_url/_did plus the auth_service_proofs delegation chains hilt issues to this agent). Tenants, access keys, and buckets are owned by hilt: it authorizes every S3 request signed with a hilt-issued access key, mints each bucket's Forge space, and issues the S3 credentials; ingot never self-provisions. The gateway has no root account, and a key that is not a did:key is rejected before hilt is consulted. The deployment context is the Forge deployment RFC: the S3 facade runs at the edge, co-located with a provider's piri or as a standalone client — not inside the central upload-service.

serve exports OpenTelemetry traces over OTLP/HTTP when OTEL_EXPORTER_OTLP_ENDPOINT names a collector; with no endpoint, tracing is off. Each S3 request is a trace named for its S3 action, with the hilt, sprue, piri and Postgres calls it makes as child spans. The other standard OTEL_* environment variables apply: OTEL_EXPORTER_OTLP_HEADERS authenticates to the collector, and OTEL_TRACES_SAMPLER_ARG sets the fraction of requests traced (0.01 traces 1%; the default traces every request). As a library, ingot records spans on the global tracer provider, so an embedding host decides where they go. Metrics go to the same collector under the same variables (OTEL_EXPORTER_OTLP_METRICS_ENDPOINT overrides the endpoint for them alone, and OTEL_METRIC_EXPORT_INTERVAL sets how often they are pushed); as a library, ingot records them on the global meter provider.

Local disk

serve keeps object bodies in two directories under data_dir, which must be on one filesystem (serve checks this at startup):

• <data_dir>/spool holds each body as it is written and uploaded, until the provider holds it;
• <data_dir>/cache holds a body once the provider holds it (it moves there by rename), so a read soon after a write is served from local disk.

A cached copy is not needed, so it goes:

• when a deleted or overwritten object's release runs (release_grace, default 60s, after its last reference drops). The release also removes the provider's copy, so a GET still streaming the old object once the grace has passed fails when it reaches a blob it has not yet opened: release_grace bounds how long such a read may take;
• with local_blob_max_bytes set, when a sweeper, checking every 30 seconds, finds the two directories over the budget and evicts cached bodies, oldest first, down to 90% of the budget. Later reads of an evicted body go to the provider and take provider-read latency.

Two windows keep cached bodies local while the budget allows: cache_min_residency (default 10m) passes over bodies whose upload changed state that recently (for a committed object, its commit), so a client reading back what it just wrote reads from disk, and cache_read_retention (default 1h) passes over bodies read from the cache that recently (remembered in memory for up to 65,536 bodies, and forgotten on restart). If usage is still over budget after that, the sweeper evicts inside the windows, since a full disk fails every write, but only down to the budget: first bodies inside cache_min_residency that no one has read recently, then, only if that is not enough, recently read ones too. It warns at most once an hour when it does; the budget_forced removals count each time. 0s turns either window off.

A multipart part's copy goes from the spool as soon as the part parks on its provider, so Complete relies on the provider keeping the parked allocation until then; keep multipart_session_ttl (default 168h) well under how long the provider keeps parked allocations. The TTL counts from the session's last state change, and a failed Complete returns the session to open, which restarts it, so a part can outlive the TTL. The budget is off by default (local_blob_max_bytes: 0): without it the cache grows with every live object's bodies. A node that rarely reads back what it just wrote can keep no written bodies at all: with cache_writes: false, each body's copy goes as soon as the provider accepts it instead of moving to the cache, and every read goes to the provider. The spool then holds only bodies still being written or uploaded and the bodies of failed uploads, apart from a copy whose removal failed, which waits for its object's release or the budget; bodies cached before the setting was turned off stay until they are released or evicted. With or without a budget, the sweeper also deletes, at startup and then hourly, unfinished .tmp-* writes in either directory and spool files with no upload intent, once they are older than local_blob_orphan_age (default 24h, at least 1h). Once at each startup, serve also deletes the copies of objects deleted by a version of ingot that kept them.

Sizing. The filesystem needs room for:

• local_blob_max_bytes, plus 10% headroom that must exceed the ingest rate × 30 seconds (60 GB at 2 GB/s). To evict outside cache_min_residency, the budget must also exceed the ingest rate × the window (1.2 TB at 2 GB/s for the default 10m); below that, sweeps evict inside the window, and warn at most once an hour. Likewise, to keep recently read bodies, it must exceed the bodies read within cache_read_retention (up to 65,536 of them, each up to max_blob_size); a read working set bigger than the budget is evicted inside the window every sweep;
• the spool's bodies in flight, when they outgrow the budget: each concurrent PUT or UploadPart writes its body as it streams, and a part can be up to 5 GiB. The budget counts those bytes as they land, so the sweeper evicts cached bodies to make room for them. But a body cannot be evicted until the provider holds it, so if the spool alone exceeds the budget, usage runs over it by the difference;
• the bodies of uploads that failed: their upload intents stay spooled or uploading, and nothing reclaims a failed PUT's spool files yet, so they stay and count against the budget. (A multipart part's go with its session, at Complete, Abort or multipart_session_ttl.) They are not safe to delete by hand (see Manual cleanup below), so size the budget with headroom for them. ingot.local_blobs.stalled_bytes reports how much they hold;
• the catalog log, if it shares the filesystem: <data_dir>/segments, per bucket about (retain + the open and unshipped segments) × seal_bytes.

Metrics. With metrics on, local blob storage reports:

Metric	Meaning
ingot.local_blobs.usage	Bytes held, by dir: spool (writes in progress and bodies awaiting upload, which eviction cannot touch) or cache
ingot.local_blobs.budget	local_blob_max_bytes (0: no budget)
ingot.local_blobs.stalled_bytes	Bytes of bodies whose upload has stalled, by state: intents still spooled or uploading an hour after their last state change, which nothing reclaims yet for a failed PUT (a multipart part's go with its session). spooled bodies never left this node; uploading ones may have reached their provider. Growth means uploads are failing
ingot.local_blobs.removals, ingot.local_blobs.removed_bytes	Files and bytes removed, by reason: released, parked, accepted (with cache_writes: false), budget, budget_forced (inside a retention window), orphan
ingot.local_blobs.reads	Body-blob reads, by tier: local or network (the local hit ratio)

Multipart uploads' parts are accepted in the background while the client uploads them (see DESIGN_NOTES), and report:

Metric	Meaning
ingot.multipart.parked_age	Seconds the oldest parked part of an open or completing upload has waited for acceptance, measured every 30s; 0 when none waits. Parts are accepted within seconds, so growth means the acceptance queue is stuck
ingot.multipart.accept_passes	Acceptor passes that concluded parts, by outcome: accepted or failed (retried after a backoff; a timed-out pass included)
ingot.multipart.complete_deferred	CompleteMultipartUpload requests answered OperationAborted because a part they name was still queued. Botocore clients (the AWS CLI) treat that as a failed upload

Manual cleanup. Every file in <data_dir>/cache is safe to delete: the provider holds its body. A file in <data_dir>/spool may be the only copy. The one kind safe to delete is a body the provider already holds, left in the spool by an interrupted move; this query lists those (the file names are the hex digests):

SELECT encode(i.digest, 'hex') FROM ingot.upload_intents i
WHERE i.state IN ('accepted', 'published')
  AND EXISTS (SELECT 1 FROM ingot.blob_locations l WHERE l.digest = i.digest);

Never delete the spool file of a spooled or uploading intent: it may be the only copy. Restart serve after deleting files by hand: the byte counts see such a change at startup, or only at an hourly scan that no write overlapped.

Build & test

make build   # GOWORK=off go build ./...
make test    # unit tests (seconds, no Docker)
make itest   # integration tests (boots the Forge stack in Docker, ~6 min)
make gen     # regenerate CBOR marshalers after changing bucket types

The itest/ package boots the real Forge stack via smelt's Go SDK and runs this working tree's binary against it — including the curated S3 conformance partition (per-group expected-pass and known-fail tables of versitygw cases). CI runs it after unit tests pass. See itest/README.md.

Deploying to the dev node

Every merge to main publishes ghcr.io/fil-forge/ingot:main and dispatches the digest of the prod image to infra-nodes. That workflow rewrites the pin the FilOne Appliance dev node runs and opens a pull request with auto-merge armed. Merging queues the image; the node picks it up on its next reconcile pass, waits for a safe proving window, and then restarts.

The dev matrix leg dispatches nothing. It carries delve and a debug build, and the node pins the prod image.

The dispatch needs two repository credentials: the Actions variable FORGE_BOT_APP_ID and the Actions secret FORGE_BOT_PRIVATE_KEY, which mint a token scoped to infra-nodes alone. Without them the publish step fails and the image still lands in GHCR.

Dependencies

ingot depends only on the Forge stack — ucantone (UCAN 1.0 primitives), libforge (Forge capability definitions), the hilt client (request authorization + tenancy), the indexing-service query client, versitygw (the S3 front end) — plus standard plumbing (pgx, goose, fx, zap, go-cid). It must never import fil-forge/sprue or fil-forge/guppy (guppy embeds ingot → cycle); the Forge-client subset it needs is carried in forgeclient/ + tokenstore/.

Status

The S3 core is exercised two ways: make test runs the unit tier in seconds, and make itest boots the real Forge stack (smelt, in Docker) and runs this working tree's binary against it, including a curated partition of the upstream versitygw S3 conformance suite. Known gaps are tracked in DESIGN_NOTES.md.