Skip to content

fix(lore-0046): give Horizon 2 s so a hung one can't fail over the RPC - #162

Open
karolko9 wants to merge 1 commit into
developfrom
lore-0046-horizon-timeout
Open

karolko9 wants to merge 1 commit into
developfrom
lore-0046-horizon-timeout

Conversation

@karolko9

@karolko9 karolko9 commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Upstream's verify() reads the last 20 ledgers from Horizon on every payment, to turn maxTimeoutSeconds into ledgers. The read has no timeout of its own and runs inside our 10 s deadline per RPC URL. A hung Horizon took every RPC URL past its deadline: the facilitator moved off healthy RPCs and answered the payment as an RPC outage.

 main()
-  setGlobalDispatcher(new Agent({ headersTimeout: 10 s, bodyTimeout: 10 s }))
+  setGlobalDispatcher(boundedAgent(10 s, {
+    "https://horizon-testnet.stellar.org": 2 s,
+    "https://horizon.stellar.org": 2 s,
+  }))
Situation Before After
Horizon answers (0.46 to 0.73 s, 30 testnet calls) verified the same
Horizon hangs for 15 s every RPC URL times out; calls move off a healthy RPC for about 60 s; the payment is refused the read fails after about 2 s; upstream assumes 5 s a ledger; the payment is verified on the same URL
Horizon answers an error upstream assumes 5 s a ledger the same
An RPC URL hangs 10 s, then the next URL the same

Unchanged: every other origin keeps 10 s, RPC included. boundedAgent (src/adapters/http-timeouts.ts) is a function, not a type: one undici Agent whose factory gives a listed origin's pool its own header and body timeouts.

Evidence

  • New test, before the module existed: http-timeouts.test.ts failed to import. After: a hung listed origin fails in under 2 s at a 100 ms bound, and a 300 ms answer from another origin passes under the 5 s default.
  • The SDK's Horizon client obeys it: new Horizon.Server(<hung local origin>).ledgers().call() with this dispatcher installed globally failed after 2512 ms (fetch failed).
  • The whole facilitator suite: 1008 passed, 3 skipped. Typecheck, pnpm lint and format:check pass.

Not run: a real Horizon outage.

Considered and rejected

  • A PR to upstream that drops the Horizon read for a configured or constant ledger time — the right fix, but it waits on maintainers; noted for later.
  • A local patch of @x402/stellar — the spine keeps upstream unchanged (AD-3), and the patch would need redoing on every bump.
  • Renaming /verify's 503 from rpc_unavailable — with Horizon bounded, the timeout that gives it means an RPC again.

Merge Danger

Door: two-way

Revert the commit.

Blast Radius: payments

A Horizon slower than 2 s but still answering makes upstream use 5 s a ledger instead of the measured mean. Ledgers closed in exactly 5 s in 8,000 sampled intervals. Once #152 is in, its runbook step "check Horizon when every URL timed out" describes what this PR removes; it gets a follow-up edit.

🤖 Generated with Claude Code

Upstream verify reads the ledger time from Horizon on every payment, with
no timeout of its own, inside each RPC URL's 10 s deadline. A hung Horizon
took every URL past it, moved calls off healthy RPCs and answered the
payment as an RPC outage. The process's HTTP dispatcher now ends requests
to the two Horizon origins after 2 s; upstream then assumes 5 s a ledger.

Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant