StableRail is a Go reference implementation of a provider-neutral payment orchestration platform. It exposes one HTTP payment model for inbound pay-ins and outbound payouts, keeps payment, double-entry ledger, outbox, and saga state in PostgreSQL, and uses Kafka to coordinate policy checks, accounting, provider execution, compensation, and manual review. Settlement-provider adapters implement the external pay-in and payout operations without changing the public payment API.
- Separate payment, settlement, ledger, and reconciliation statuses so business outcomes do not obscure operational or accounting state
- Immutable, balanced double-entry ledger postings
- Transactional outbox and inbox with retries, dead-lettering, and redrive
- Persisted sagas with timeouts, provider returns, compliance holds, and audited manual review
- Versioned event payloads with consumer upcasting
- Tenant API keys stored as hashes and HTTP idempotency backed by PostgreSQL
- Signed tenant webhooks with retry and delivery history
- Health, readiness, Prometheus metrics, structured logs, and graceful shutdown
StableRail currently runs as one process, while preserving boundaries that allow its components to be deployed separately later.
flowchart LR
Client[Client] --> API[Payment API]
Operator[Operator] --> API
API --> DB[(PostgreSQL)]
DB --> Relay[Outbox relay]
Relay --> Kafka[[Kafka]]
Kafka --> Consumers[Inbox consumers]
Consumers --> Saga[Saga and workers]
Saga --> DB
Saga <-->|Generic pay-in/payout contract| Provider[Settlement provider]
Provider --> External[External provider adapter]
External -->|Signed webhook| API
DB --> Webhooks[Tenant webhooks]
Every business update and outgoing event commits in one PostgreSQL transaction. Consumers record an event in their inbox and apply its effects in another single transaction. Delivery is at least once; deterministic event IDs, uniqueness constraints, and inbox records make retries safe.
| Package | Responsibility |
|---|---|
paymentapi |
HTTP transport, authentication, and tenant/operator endpoints |
paymentcore |
Shared payment, funds, quote, execution, refund, history, and direction-neutral payment-query models |
paymentcore/payin |
Inbound quote and operation model, saga coordination, provider execution, and result persistence |
paymentcore/payout |
Outbound creation, refunds, quotes, saga coordination, provider execution, and result persistence |
ledger |
Transactional double-entry payout reservation, settlement, and release; inbound pay-in receipts; and provider-return journals |
policy |
Payment policy evaluation contracts |
reconciliation |
Comparison of payment, ledger, and provider records plus discrepancy resolution |
eventbus |
Shared event envelopes, topic and version contracts, and Kafka producer/consumer adapters |
outbox |
Transactional event publication, retry and dead-letter handling, and operator redrive |
settlement |
Provider-neutral payout/pay-in contracts and the deterministic mock provider |
settlement/blindpay |
BlindPay client, resource mapping, quote/execution adapters, and webhooks |
workers |
Policy, ledger, and provider command execution plus runtime timeout polling for both settlement directions |
paymentcore owns the shared merchant-facing payment lifecycle, with payin and payout handling their direction-specific workflows. eventbus defines shared events and routing, workers executes asynchronous work, and settlement providers implement separate quote and execution capabilities for each supported direction.
Pay-in and payout services manage quotes and provider execution. Their saga coordinators retry uncertain operations with stable idempotency keys. Provider adapters translate external API responses and webhooks into normalized results; payment, ledger, and merchant-notification updates remain owned by StableRail.
All application topic names are defined in eventbus/topics.go. Pay-in and payout topics carry internal workflow events; payment-events carries the stable merchant-facing lifecycle shared by both directions.
| Topic | Constant | Producers | Consumers | Purpose |
|---|---|---|---|---|
payout-events |
eventbus.PayoutEventsTopic |
Payout services, workers, and provider webhooks | Payout saga coordinator | Internal payout workflow facts |
payin-events |
eventbus.PayinEventsTopic |
Pay-in services, workers, and provider webhooks | Pay-in saga coordinator | Internal pay-in workflow facts |
payment-events |
eventbus.PaymentEventsTopic |
Pay-in and payout workflows | Tenant-webhook dispatcher | Merchant-facing payment lifecycle facts |
settlement-commands |
eventbus.SettlementCommandsTopic |
Pay-in and payout saga coordinators | Policy, ledger, and provider command workers | Durable workflow commands for both settlement directions |
stablerail-dead-letter |
eventbus.DeadLetterTopic |
Outbox relay | Operator inspection and redrive tooling | Events that exhausted outbox publication retries or exceeded the retry age |
Workflow events use direction-specific names such as payout.created, payout.provider_completed, payin.created, and payin.received. Merchant integrations receive only the shared payment.created, payment.processing, payment.succeeded, and payment.failed events.
The API exposes payment_status as the public business outcome:
stateDiagram-v2
state "Payment status" as payment {
[*] --> created
created --> processing: submitted
created --> failed: rejected before submission
processing --> succeeded: recipient paid
processing --> failed: settlement unsuccessful
}
Operational and accounting state stays with the subsystem that owns it:
| State | Owner | Meaning |
|---|---|---|
payment_status |
payments |
Public business outcome: created, processing, succeeded, or failed |
settlement_status |
payins / payouts |
Normalized external movement state, including uncertainty, holds, receipt/completion, failure, and return |
ledger_status |
ledger_journals |
Accounting recognition: pending, posted, or failed |
reconciliation_status |
payins / payouts |
Agreement with external records: unmatched, matched, or exception |
Balances, reservations, and availability are calculated from posted ledger_entries; they are not mutable payment attributes. An ambiguous provider submission remains payment_status=processing with payouts.settlement_status=unknown, while the posted reservation journal continues to determine availability until a retry, webhook, or reconciliation establishes the outcome.
A bank or provider can return funds after a payout was already confirmed. StableRail records that as a separate financial operation:
payment_returns contains only completed provider returns; it is an immutable record rather than a return lifecycle. Its required journal debits cash:operating for the asset received back and credits settlement:payable to restore the obligation. The original payment is not rewritten. StableRail emits payment.return.succeeded; return details are read from the separate return operation rather than encoded as payment state.
Merchant-issued refunds are separate linked payments. POST /v1/payments/{id}/refunds accepts an idempotency key, a positive amount, a reason, and an optional fresh payout_quote_id for BlindPay routing. Partial refunds are supported up to the original payment amount. StableRail creates a new payment and uses its ID as payment_refunds.refund_payment_id; original_payment_id links back to the payment being refunded. It binds the payout quote when supplied and emits payout.created for workflow coordination and payment.created for merchant notification. From there, policy, ledger reservation, settlement, and failure handling use the ordinary payout workflow. Refund eligibility is established by the original payment outcome and its posted settlement journal. Provider-originated returns remain separate and continue to use payment_returns and reversal accounting.
The persisted payout saga tracks internal workflow progress in more detail than the public payment status:
stateDiagram-v2
state "funds_returned (saga)" as returned
[*] --> awaiting_policy: payout.created
awaiting_policy --> awaiting_ledger: payout.policy.approved / ledger.reserve
awaiting_policy --> failed: payout.policy.rejected / payment.fail
awaiting_ledger --> awaiting_settlement: payout.funds_reserved / settlement.execute
awaiting_ledger --> failed: payout.ledger_failed / payment.fail
awaiting_settlement --> settling_payment: payout.provider_completed / payment.settle
settling_payment --> completed: payout.completed
awaiting_settlement --> awaiting_settlement: settlement timeout / retry settlement.execute
awaiting_settlement --> failed: payout.provider_failed / payment.fail_reserved
awaiting_settlement --> failed: submission_failed / payment.fail
awaiting_settlement --> returning: payout.provider_returned / ledger.release
failed --> returning: late payout.provider_returned / ledger.release
returning --> returned: payout.funds_released / payment.return
awaiting_settlement --> on_hold: payout.on_hold
on_hold --> settling_payment: payout.provider_completed / payment.settle
on_hold --> failed: payout.provider_failed / payment.fail_reserved
on_hold --> returning: payout.provider_returned / ledger.release
on_hold --> manual_review: compliance timeout
manual_review --> on_hold: operator retry
manual_review --> settling_payment: operator complete / payment.settle
manual_review --> failed: operator fail / payment.fail_reserved
manual_review --> returning: operator return / ledger.release
The saga's funds_returned label is workflow state, not a payment status. The resulting payment remains payment_status=failed; the posted release journal records the accounting effect. The transition from failed handles a provider return that arrives after a reservation-related failure was already recorded.
Timeout handling is conservative: settlement timeouts retry the idempotent provider command while preserving the reservation, compliance timeouts require manual review, and ambiguous submissions remain in processing until a retry, webhook, or reconciliation establishes an outcome. A reservation becomes available only after a confirmed pre-capture failure, and becomes returned only after the provider confirms the funds came back.
The pay-in coordinator uses the same settlement_sagas table with direction=payin, but has direction-specific states and commands:
stateDiagram-v2
[*] --> awaiting_policy: payin.created
awaiting_policy --> awaiting_execution: payin.policy.approved / payin.execute
awaiting_policy --> failed: rejection or timeout / payin.fail
awaiting_execution --> processing: payin.processing
awaiting_execution --> on_hold: payin.on_hold
awaiting_execution --> awaiting_ledger: payin.received / payin.ledger.record
awaiting_execution --> awaiting_execution: execution timeout / retry payin.execute
processing --> processing: provider polling or retry
processing --> on_hold: payin.on_hold
processing --> awaiting_ledger: payin.received / payin.ledger.record
on_hold --> awaiting_ledger: payin.received / payin.ledger.record
on_hold --> failed: compliance timeout / payin.fail
awaiting_ledger --> completed: payin.succeeded
awaiting_ledger --> awaiting_ledger: ledger timeout / retry payin.ledger.record
awaiting_ledger --> failed: payin.failed
awaiting_execution --> failed: payin.failed
processing --> failed: payin.failed
on_hold --> failed: payin.failed
awaiting_execution --> refunded: payin.refunded
processing --> refunded: payin.refunded
on_hold --> refunded: payin.refunded
awaiting_ledger --> refunded: payin.refunded
completed --> refunded: payin.refunded / reverse ledger
Pay-in policy and compliance waits fail on timeout. Provider execution and ledger commands are retried with their original idempotent operation identity. Each active pay-in state stores a deadline, and the timeout worker claims only direction=payin rows; the payout timeout worker similarly claims only direction=payout rows.
Pay-ins and payouts are directions of the same public payment resource. Create a direction-aware quote with POST /v1/payment-quotes when the client needs to preview and lock pricing, FX, fees, or routing, then create the operation with POST /v1/payments using its quote_id. Both directions also support direct creation. A direct pay-in supplies amount, currency, funding method, and destination account; a direct payout additionally supplies source account and destination instrument. StableRail creates and persists an implicit provider quote before accepting either direct payment, so provider execution always uses a durable provider_quote_id. Creating either direction persists a created payment and transactional outbox events. A dedicated pay-in coordinator sends payin.execute through Kafka only after payin.policy.evaluate is approved, and the command worker performs the provider call. Provider confirmation moves the pay-in to received; the saga then sends payin.ledger.record, and only a successful balanced journal advances the pay-in to succeeded. Provider instructions such as an ACH memo, bank details, Pix code, or CLABE become available asynchronously; retrieve current state with GET /v1/payments/{id}.
Both direction-specific coordinators store orchestration state in settlement_sagas, keyed by payment ID and direction. The provider-facing payins and payouts tables remain separate because their execution details and provider statuses differ; they are not separate public API resources.
Direction-specific quote boundaries use opaque StableRail resource IDs resolved through provider_resources; provider wallet and bank-account identifiers do not appear in the shared payment API. Both execution boundaries receive only a stable idempotency key and provider quote ID. Verified payin.* webhooks can advance an executed pay-in to processing, on_hold, received, failed, or refunded. The saga turns received into succeeded only after the ledger command debits cash:operating and credits settlement:payable. A refund after that successful journal posts the inverse journal; a refund before ledger completion has no completed pay-in journal to reverse. Early webhooks are retained and reconciled after the local pay-in becomes visible.
Pay-in and payout records own their source, destination, method, and monetary snapshot. A quote locks fees, FX, provider routing, and source/destination amounts. Clients may supply an explicit quote or use direct creation, but StableRail persists a quote in both cases and provider execution always references it. Accepting the quote copies its terms into the direction-specific operation snapshot.
Shared APIs and workflow tables use provider-resource IDs instead of naming provider-specific bank-account or wallet fields:
Payout: source account -> destination payment instrument
Pay-in: optional source payment instrument -> destination account
An account represents a balance-holding resource, such as a managed wallet. A payment instrument represents an external routing endpoint, such as a bank account or blockchain address. provider_resources maps those stable IDs to a provider and its reference. For example:
acct_123 -> blindpay / managed wallet / bl_...
instrument_456 -> blindpay / bank account / ba_...
Callers must treat resource IDs as opaque; the current BlindPay reference sync may reuse a provider reference as the local resource ID for compatibility, but the adapter still resolves it through provider_resources. Adding another provider requires new resource mappings and an adapter, not new columns in payins, payouts, or their quote tables. Raw provider responses are isolated in provider_payload, while raw webhook events remain adapter-owned data.
The schema separates payment workflow state, provider execution, double-entry accounting, event delivery, and reconciliation while linking each operation back to its payment.
Migrations are ordered by dependency. Provider-neutral platform and workflow schema is created before the BlindPay-specific adapter schema.
| Migration | Main purpose |
|---|---|
| 001_payment_core.sql | Payments, shared quotes and provider resources, audit/timeline history, and refunds |
| 002_eventing.sql | Transactional outbox and consumer inbox |
| 003_ledger.sql | Accounts and balanced payment journals |
| 004_payouts.sql | Payout provider operations and post-success returns backed by ledger journals |
| 005_payins.sql | Pay-in provider operations linked to payments |
| 006_payment_workflow.sql | Direction-aware settlement sagas, manual review actions, and settlement submission records |
| 007_webhooks.sql | Merchant webhook delivery and provider webhook ingestion/application tracking |
| 008_reconciliation.sql | Reconciliation runs and discrepancies |
| 009_tenant_access.sql | Tenant API-key authentication |
| 010_blindpay.sql | BlindPay-owned customers, bank accounts, wallets, and raw webhook events |
Requirements: Go, Docker, and Docker Compose.
Start PostgreSQL and Kafka:
docker compose up -d
docker compose psApply migrations:
for migration in migrations/*.sql; do
docker compose exec -T postgres psql -U stablerail -d stablerail -f - < "$migration"
doneCreate Kafka topics:
for topic in payout-events payin-events payment-events settlement-commands stablerail-dead-letter; do
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server localhost:9092 \
--create --if-not-exists \
--topic "$topic" \
--partitions 1 \
--replication-factor 1
doneRun StableRail:
export STABLERAIL_DATABASE_URL=postgresql://stablerail:stablerail@localhost:5432/stablerail
export STABLERAIL_KAFKA_BROKERS=localhost:9092
export STABLERAIL_OPERATOR_TOKEN='replace-with-a-secret-token'
go run ./cmd/stablerailThe API listens on :8080. Create a tenant API key:
curl -X POST http://localhost:8080/v1/operator/tenants/tenant-1/api-keys \
-H "Authorization: Bearer $STABLERAIL_OPERATOR_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"name":"local development"}'Save the returned srk_... value as STABLERAIL_API_KEY. For the mock provider, seed provider-neutral development resources:
docker compose exec -T postgres psql -U stablerail -d stablerail -c "
INSERT INTO provider_resources
(id, tenant_id, provider, resource_type, provider_reference, metadata, created_at, updated_at)
VALUES
('acct_local', 'tenant-1', 'mock', 'account', 'mock_wallet', '{\"kind\":\"wallet\"}', now(), now()),
('instrument_local', 'tenant-1', 'mock', 'payment_instrument', 'mock_bank', '{\"kind\":\"bank_account\"}', now(), now())
ON CONFLICT (id) DO NOTHING;"Then create and inspect a payment:
curl -X POST http://localhost:8080/v1/payments \
-H "Authorization: Bearer $STABLERAIL_API_KEY" \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: request-123' \
-d '{"direction":"payout","external_reference":"order-123","currency":"USD","amount_minor":2500,"funding_method":"bank","source_account_id":"acct_local","destination_instrument_id":"instrument_local"}'
curl -H "Authorization: Bearer $STABLERAIL_API_KEY" \
http://localhost:8080/v1/payments/PAYMENT_ID/timelinePayment processing is asynchronous. Poll the payment or timeline endpoint to observe status changes. Reusing an idempotency key with the same request returns the original payment; reusing it with different fields returns 409 Conflict.
The following reset permanently deletes the local PostgreSQL volume and all of its data. It also stops and removes the other containers in this Compose project. Do not run it against an environment containing data you need to preserve.
Remove the containers and named volumes:
docker compose down -vRecreate PostgreSQL and wait until it accepts connections:
docker compose up -d postgres
docker compose exec -T postgres pg_isready -U stablerail -d stablerailApply all migrations to the empty database:
STABLERAIL_DATABASE_URL='postgresql://stablerail:stablerail@localhost:5432/stablerail?sslmode=disable' \
go run ./cmd/migrateVerify that the migration history is present and the payment table is empty:
docker compose exec -T postgres psql -U stablerail -d stablerail \
-c "SELECT count(*) AS applied_migrations FROM schema_migrations; SELECT count(*) AS payments FROM payments;"Start the remaining services when needed:
docker compose up -dTo stop the environment without deleting its data, run docker compose down
without -v.
| Endpoint | Purpose |
|---|---|
POST /v1/payments |
Create a pay-in or payout payment (direction: "payin" or "payout") |
GET /v1/payments/{id} |
Read a payment |
GET /v1/payments/{id}/timeline |
Read payment history |
POST /v1/payments/{id}/refunds |
Create a merchant-issued refund as a linked payment |
POST /v1/payment-quotes |
Create a provider-neutral pay-in or payout quote |
POST /v1/providers/blindpay/webhooks |
Receive signed BlindPay events |
POST /v1/webhook-endpoints |
Register a tenant webhook |
GET /v1/webhook-endpoints |
List tenant webhooks |
DELETE /v1/webhook-endpoints/{id} |
Disable a tenant webhook |
POST /v1/operator/tenants/{id}/api-keys |
Issue a tenant API key |
DELETE /v1/operator/api-keys/{id} |
Revoke a tenant API key |
POST /v1/operator/payments/{id}/manual-review |
Resolve a held payment |
POST /v1/operator/mock-settlements/{id} |
Resolve a local mock settlement when enabled |
GET /healthz |
Liveness check |
GET /readyz |
PostgreSQL readiness check |
GET /metrics |
Prometheus metrics |
Tenant endpoints require Authorization: Bearer <api-key>. Operator endpoints are available only when STABLERAIL_OPERATOR_TOKEN is set and require that token. Payment reads are tenant-scoped. direction is required and must be payin or payout.
POST /v1/payments returns 202 Accepted for both directions. Provider instructions and later lifecycle statuses are asynchronous. Direct creation may create a provider quote before the payment is persisted; provider execution itself remains asynchronous.
The application selects one settlement.SettlementProvider, which composes direction-specific QuoteProvider and ExecutionProvider capabilities. Quote requests carry provider-neutral route IDs. Both execution capabilities use the shared paymentcore.ExecuteRequest, containing only an idempotency key and provider quote ID. The runtime includes a deterministic mock provider and a BlindPay adapter with durable payout submission, pay-ins, signed webhooks, compliance holds, reconciliation, and coordinator-driven ambiguous-outcome retries.
BlindPay refund semantics depend on direction and timing. A payout or pay-in refunded before success becomes payment_status=failed, while its direction record and journals retain settlement and accounting detail. After payout success, returned funds are recorded as a separate payment_returns operation and reversal journal while the original payment remains succeeded. None of these provider-originated events is a merchant-issued refund.
Configure the BlindPay provider with:
export STABLERAIL_BLINDPAY_API_KEY='...'
export STABLERAIL_BLINDPAY_INSTANCE_ID='in_...'
export STABLERAIL_BLINDPAY_WEBHOOK_SECRET='whsec_...'
export STABLERAIL_BLINDPAY_NETWORK='base'
export STABLERAIL_BLINDPAY_TOKEN='USDC'
export STABLERAIL_BLINDPAY_MANAGED_WALLET_ID='bl_...'
export STABLERAIL_BLINDPAY_MANAGED_WALLET_ADDRESS='0x...'Register the public HTTPS URL https://your-host/v1/providers/blindpay/webhooks in the BlindPay dashboard. This is an inbound provider endpoint, not a client API. StableRail verifies its Svix signature headers before storing or processing a delivery.
StableRail persists a payout submission attempt before calling BlindPay. If the response is lost, the saga coordinator re-enqueues execution with the payment's original provider idempotency key; reconciliation can independently confirm the result. A verified webhook or reconciliation result, rather than the initial HTTP response alone, determines the final payment and settlement statuses.
See BlindPay lifecycle testing for provider scenarios and expected accounting behavior.
The initial production architecture runs the current single-process application as a replicated monolith on AWS ECS Fargate, backed by RDS PostgreSQL and Amazon MSK. ADR 0001 records why this is the current decision, the measured triggers for moving to role-based deployments, and the stricter requirements for selectively extracting microservices.
The AWS Terraform deployment provisions the network, load balancer and DNS, ECS service, ECR repository, Multi-AZ database, TLS Kafka cluster, secrets access, logging, autoscaling, and one-off migration task.
Run the unit and package tests:
go test ./...Run race detection and static analysis:
go test -race ./...
go vet ./...Run the end-to-end lifecycle suites:
# Provider-free lifecycle tests using the mock settlement provider
./scripts/test-e2e-local.sh
# BlindPay adapter lifecycle tests using the local BlindPay mock server
./scripts/test-e2e-blindpay.shPass normal go test arguments to select a scenario:
./scripts/test-e2e-local.sh -run '^TestLOCAL001SuccessfulPaymentLifecycle$'Each script starts an isolated PostgreSQL and Kafka stack, applies migrations, builds StableRail, runs its suite, and removes the stack afterward. Set STABLERAIL_E2E_KEEP_STACK=1 to retain the containers for inspection. See the local lifecycle test guide and BlindPay lifecycle test guide for the executable scenario specifications.
The core payment, saga, managed-wallet payout, pay-in, merchant-issued refund, provider-return, webhook, reconciliation, and manual-review paths are implemented. Merchant refunds are linked payments and reuse the ordinary payout saga; provider-originated returns remain separate operations with reversal accounting. Remaining production work includes distributed tracing, alert integrations, credential rotation, provider rate limiting, rollout controls, and external-wallet payout submission.
The mock provider and local Compose environment are intended for development and verification. A production deployment still requires environment-specific security controls, topic provisioning, monitoring, and a limited provider pilot.