Skip to content

chore(lore-0026): promote develop to production for the fourth deploy - #167

Open
karczuRF wants to merge 156 commits into
deploy/productionfrom
develop
Open

karczuRF wants to merge 156 commits into
deploy/productionfrom
develop

Conversation

@karczuRF

@karczuRF karczuRF commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

The fourth deploy (task 0026, for 0027's x402@ fix). It promotes develop at 67f03a6e onto deploy/production (68e778e3, the third deploy): 155 commits, 18 pull requests. The head is develop itself, so a later merge into develop joins this pull request too: check the list again if it moves before the merge.

The reason for it now is #163: every planned stop of x402@production ended failed (Compose's 130 after systemd's stop signal) and counted facilitator_failures_total. Since facilitator-failed went live on 2026-11-09, every deploy or restart would fire it falsely.

Merge it with Create a merge commit, like #119, #132 and #137, so deploy/production keeps develop's commits as they are.

Before the merge

  • One new migration, 0006_catalog_seller_check (feat(lore-0063): list a resource from the seller's own 402 #89). ALTER TABLE catalog ADD COLUMN checked_at bigint (nullable) and a new catalog_attempts table with an index. It expands only, so the image in digest.good keeps working on it after a rollback. Its number is older than the migrations the server already ran (0007–0012). migrate applies every name not yet in schema_migrations, whatever its place, so it runs once, after 0012.
  • The new image works with the server's current files. No change to the production compose file, the Dockerfile or the workflows. The only change under deploy/ is fix(lore-0027): count a planned facilitator stop as a success #163's unit line and its test.
  • CI on the head is green. develop's push run at 67f03a6e is running; e7c921bf before it is green.

After the merge

  1. Deploy production builds the image, then waits for the production environment's approval from someone other than whoever merged. It migrates, drains the old facilitator, starts the new image and waits for ready. On a failure it rolls back to digest.good.
  2. Expect one facilitator-failed in #stellar-x402-bot: this deploy's restart still runs under the old unit. Post in the channel first.
  3. Check /health and /supported. /health's commit is the merge commit on deploy/production.
  4. Once the workflow run has ended, run the playbook from this merge (runbook, The playbook), --check --diff first. Its diff must show x402@.service gaining SuccessExitStatus=130 and nothing else, then a daemon-reload.
  5. Check it: systemctl show x402@production -p SuccessExitStatus must print SuccessExitStatus=130 …. From the next stop on, a planned stop ends inactive and writes no failure.

Testing

CI on develop: green at e7c921bf, and the run at 67f03a6e is running. Merge only once the run on the head is green. #163's own test (deploy/test/units.test.ts) fails without the line. Testnet: N/A. This pull request is the deploy, and the steps above are its checks.

karolko9 and others added 30 commits October 28, 2026 08:35
Rows 1-4, 6-12, 19, 22 and 23 of the task, red before the code: a
listing comes from the seller's unpaid 402, the check is budgeted per
key, host and in all, offers are filtered, stripped and capped, the
later-started check wins, and every attempt is a row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A settled payment's copy of a listing no longer writes it. When the
copy differs from the stored listing, the facilitator asks the seller
with one unpaid request through the seller-fetch guard, and stores what
the seller's 402 says: its text, its bazaar block and only its offers
to the key's network and payTo, replacing the stored ones.

- budgets: once a minute per key, 30 a minute per host, 300 in all
- offers: the spec's seven fields, extra up to 4 KiB, up to 8 offers;
  over a limit the whole listing is refused as too_large
- of two checks of one key, the later-started one is kept
- every attempt is a row in catalog_attempts, pruned after 30 days
- mcp listings wait for the MCP check and are not listed meanwhile
- ADR 0008 records who may change a listing and the entry limits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-listing

# Conflicts:
#	apps/facilitator/src/adapters/postgres.ts
A seller that settles through a facilitator built by main(), a buyer
with upstream's client and an outsider calling /settle itself: the
listing follows the seller's 402, an unchanged copy asks nobody, and an
outsider's 1-stroop copy changes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rows 5 and 18 of task 0063: an https MCP listing is checked with an MCP
client (initialize, tools/list, an unpaid tools/call with the example
arguments); a tool the server does not list, a server that is down and a
free tool give no listing; an mcp://tool URL is not askable. The guard
ends a request its caller aborts, so a closed client ends its stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An https MCP listing is now checked like an HTTP one, with an MCP
client: initialize, tools/list, then the listing's tool called without
payment with its example arguments, read with upstream's
getToolPaymentRequirements. A tool the server does not list, a server
that does not answer and a free tool change nothing; an mcp://tool URL
names no server, so it stays unlisted. The seller-fetch guard now ends
a request its caller aborts, so a closed MCP client ends its stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…describe

The SDK's listTools() keeps each tool's output schema and callTool then
checks a 402 error result against it, which the terms never match: one
live seller with 29 of the 126 MCP listings in the largest public
catalog failed so. The check lists tools with a plain tools/list
request instead. A seller that sends the terms only as text (9 of 20
probed) is covered by a test too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tool list carries every tool's schemas: the largest of 34 live MCP
servers sent 122,729 B and a second 67,408 B, both over the 64 KiB an
HTTP seller's answer may take, so 40 of 126 live MCP listings could not
be checked. The MCP check reads up to twice the largest measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fields

Upstream's PaymentRequired schema refuses a whole 402 whose tags or
service name are over the Bazaar limits, which the spec says to drop
instead: 47 of 400 live sellers sent such fields. The seller's terms are
now checked without serviceName, tags and iconUrl, and the listing's
validator trims those as it does a payment's copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng-check

# Conflicts:
#	apps/facilitator/src/catalog/seller-check.ts
Upstream's MCP client parses the tool's 402 terms with the whole
PaymentRequired schema and reads terms with over 5 tags as no 402 at
all, which one of 34 live MCP sellers sends. The check now reads the
terms from the error result's structuredContent or first text, as
upstream's payment wrapper sends them, and runs the same shape check
as for an HTTP seller. @x402/mcp is no longer needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…age-listing

# Conflicts:
#	apps/facilitator/test/port-contracts.ts
#	apps/facilitator/test/settle-pool.test.ts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 10 s deadline covered only an open HTTP request, so a server that
accepted tools/call (202) and never sent its result held the check for
the SDK's own 60 s. The deadline now ends the SDK's wait too, and is
reported as upstream_error like an HTTP seller's timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A database error or a bug thrown mid-check left a log line and a counter
but no catalog_attempts row, though row 23 makes every attempt a row. The
outer catch now records it as internal_error, its reason the step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One payer could fill the 300-a-minute budget over all hosts and hold back
every other seller's listing. Checks are now limited per listing and per
host only; /settle's per-IP limit and the settled payment each check
needs bound the rest.

ADR 0008 also says what the unpaid request can reach (any method to a
public https URL, without credentials) and that plain http is not listed,
as the seller check calls https only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto lore-0063-mcp-listing-check

# Conflicts:
#	apps/facilitator/src/catalog/cataloger.ts
The check stored the listing under the key rebuilt from the seller's
answer, so on a host or MCP server several sellers share, one route's
402 naming another route, method or tool replaced that one's listing.
An answer whose key differs from the one asked about is now refused as
other_key, and the stored listing stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once a payment's copy differs from the stored listing, its attempt row
keeps the copy and the seller's 402 as answered, so what a payer sent can
be compared with what the seller says. A copy equal to the listing keeps
neither. Both are pruned with the row after 30 days.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
close() only aborts the client's requests, so against a server that
hands out sessions every check left one open until the server's own
cleanup. The check now sends the session DELETE first; a failed one is
logged and the answer stands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
karolko9 and others added 29 commits November 9, 2026 12:44
…ields

fix(lore-0063): list the attempt counter's labels in the catalog
test(lore-0046): give the MCP e2e its own facilitator and time limit [low risk]
Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
check-series passed for all three on 2026-11-06, after the third
deploy's restart window moved the units' output to the shipped files
(decision 26). host goes on with logVolume, its cost ceiling.
log-volume's text now names the logs' event field: msg doesn't exist.
Each group's section lands with the PR that turns it on. The page now
points at the log files and the app group instead of the journal, which
keeps only a unit's start and stop since the restart window.
Review of #150: each alarm's Slack message names its docs/monitoring.md
section, and nothing checked the section exists. A renamed alarm or a
group turned on without its section now fails the test.
Review of #150: the runbook and four alarm texts sent on-call to pool
events in the app log, but the facilitator only counts them. The texts
now say so and point at pool_events_total; 0125 logs the events.

uptime's texts point at the log files, not the journal; log-volume's
query fits each log group; probe-silent and the server section name the
alarms the probe covers. The runbook greps the rotated files and leaves
the metric lines out of a tail, names the units still on the journal,
and the count is 35 alarms under 32 names. AC 15 is ticked with the
2026-11-06 check-series run, and decision 65 names both teams.
A channel is quarantined only when the RPC answers that its account
doesn't exist, so the runbook and alarm text point at a gone account,
name the channel through settlement_submit_failed and re-create it.
fee-raised and send-retry have no type dimension: their counts are
found by Logs Insights on the EMF lines. probe-silent's text names
what goes blind. infra/README.md reads a group's Level signals before
its deploy and lists ALARM states after it. 0125 narrowed to three
events.
The Alarms stack went out at 12:19 UTC on 2026-11-09 from develop
1b4229b: 39 new alarms, 47 in all, none in ALARM after it. The
delivery test at 12:22:01 UTC reached Slack, and the gauges' Values
arrays aggregate as samples, so that check leaves the list of what
isn't checked before a deploy.
Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
…nswer

/settle runs the same checks as /verify before it claims. An RPC that gave
them no answer was a thrown error, so /settle answered 500 while /verify
now answers 503 for the same outage. It now answers its own
settle_exact_stellar_rpc_unavailable in an x402 body, with nothing claimed,
so the same payment may be sent again.

Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
The #149 review: an hourly read of three files fired on spec doc edits
and missed the discovery spec the RFP names. The check now runs once a
day and adds specs/extensions/bazaar.md; a read that fails for two days
pages, and one deadline keeps the section inside the Lambda's timeout.

Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
The reconcile comment said no call adds a hash past the budget plus the
margin. A write sent just before the budget can reach the store after it;
what keeps it out is addHash's state check. The comment now says so, a store
contract test holds it, and the runbook names what a deploy that lowers
SETTLE_BUDGET_MS does to the claims in flight.

Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
test(lore-0046): pin the store guard a claim sweep relies on [low risk]
…ilable

fix(lore-0046): answer rpc_unavailable when /settle's checks get no answer
Decision 71: the theft and secret-read PR moves to the new task 0127
with criteria 6, 7 and 9, and 0026 turns on the backups and snapshots
groups once their jobs run, with criterion 11's WAL, backup and
snapshot ages. It also records the drills plan for the 47 live alarms.
After ExecStop, systemd signals the docker compose up that is still
exiting, and Compose answers with 130, so every deploy, drain and
systemctl stop left x402@production failed and counted
facilitator_failures_total. 0027's drill showed it: facilitator-failed
fired for the drill's own stop. The unit now lists 130 in
SuccessExitStatus. The facilitator exits 0 or 1 itself, and the
watchdog's kill of a hung process ends 137, so crashes still count.
The 2026-11-09 drill fired 37 alarms on purpose: a 20-minute stop
window for the silence alarms, fake points for the others. Every
alarm's row now says how and when it fired, or why it wasn't, and
Drills records the steps. Criteria 2, 3, 4, 5, 10, 11 and 13 are
ticked, with what the drill didn't prove. Decision 72 turns memory on
at 90% after three days of data.
On ch-prod-01 OpenBao's audit volume is on / with everything else, so
disk and audit-disk read the same series and at one threshold would
send two messages for one fact. Agreed with Stan and Adam (decision
73): disk at 80% warns, audit-disk at 90% is the last call before
signing stops. ADR 0013 and ADR 0011 amended; 80% leaves about 334 GB
of the 1.67 TB, 90% about 167 GB.
A testnet run showed that our facilitator refuses payments from an
OpenZeppelin smart account with a spending-limit policy, because
upstream verify rejects the policy's event. Fixes wait in upstream
PRs; recorded here so the on-chain daily limit idea keeps the fact.

Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
On pubnet a payment that creates payTo's balance stays refused under the
250,000 ceiling, and the refusal will say why. Whether the facilitator
pays for it is reconsidered in 0107, once each key has a fee budget.

Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>
docs/monitoring.md now lists the catalog's 61 metrics and the 5 AWS
ones with the alarms that read them, every alarm that is off with the
task that turns it on, and the uptime figure: 99.56% since uptime went
on, 100% without the drill's planned stop. A test fails when the page
lacks a metric or an alarm. disk-root fired on fake points at 85%
while audit-disk stayed silent; memory can't be faked while the agent
writes real samples. Criteria 12, 16 and 17 ticked; 0027 stays active
for criterion 1, which needs a real testnet settlement.
0096 set out to raise the pubnet fee ceiling for a new balance's rent;
the measurements showed the ceiling should stay. Its figures move to a
wiki note, and what is left of it (whether to pay, the refusal's reason,
the pubnet figures in the docs) becomes part of 0107's decision.

Co-Authored-By: Claude <209825114+claude[bot]@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants