Repository navigation
Conversation
Upstream verify reads the ledger time from Horizon on every payment, with no timeout of its own, inside each RPC URL's 10 s deadline. A hung Horizon took every URL past it, moved calls off healthy RPCs and answered the payment as an RPC outage. The process's HTTP dispatcher now ends requests to the two Horizon origins after 2 s; upstream then assumes 5 s a ledger. Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upstream's
verify()reads the last 20 ledgers from Horizon on every payment, to turnmaxTimeoutSecondsinto ledgers. The read has no timeout of its own and runs inside our 10 s deadline per RPC URL. A hung Horizon took every RPC URL past its deadline: the facilitator moved off healthy RPCs and answered the payment as an RPC outage.Unchanged: every other origin keeps 10 s, RPC included.
boundedAgent(src/adapters/http-timeouts.ts) is a function, not a type: one undiciAgentwhosefactorygives a listed origin's pool its own header and body timeouts.Evidence
http-timeouts.test.tsfailed to import. After: a hung listed origin fails in under 2 s at a 100 ms bound, and a 300 ms answer from another origin passes under the 5 s default.new Horizon.Server(<hung local origin>).ledgers().call()with this dispatcher installed globally failed after 2512 ms (fetch failed).pnpm lintandformat:checkpass.Not run: a real Horizon outage.
Considered and rejected
@x402/stellar— the spine keeps upstream unchanged (AD-3), and the patch would need redoing on every bump./verify's 503 fromrpc_unavailable— with Horizon bounded, the timeout that gives it means an RPC again.Merge Danger
Door: two-way
Revert the commit.
Blast Radius: payments
A Horizon slower than 2 s but still answering makes upstream use 5 s a ledger instead of the measured mean. Ledgers closed in exactly 5 s in 8,000 sampled intervals. Once #152 is in, its runbook step "check Horizon when every URL timed out" describes what this PR removes; it gets a follow-up edit.
🤖 Generated with Claude Code