Repository navigation
fix(lore-0027): count a planned facilitator stop as a success - #163
Merged
Merged
Conversation
After ExecStop, systemd signals the docker compose up that is still exiting, and Compose answers with 130, so every deploy, drain and systemctl stop left x402@production failed and counted facilitator_failures_total. 0027's drill showed it: facilitator-failed fired for the drill's own stop. The unit now lists 130 in SuccessExitStatus. The facilitator exits 0 or 1 itself, and the watchdog's kill of a hung process ends 137, so crashes still count.
stkrolikiewicz
approved these changes
Nov 9, 2026
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every planned stop of
x402@production(a deploy, a drain,systemctl stop) endedfailedand countedfacilitator_failures_total. Since the alarms went live on 2026-11-09, every deploy would fire a falsefacilitator-failed. Task 0027.What happens
ExecStop(docker compose stop facilitator) stops the container cleanly, with exit 0.docker compose upthat is still exiting, and Compose answers a signal with 130.failed(Result: exit-code).ExecStopPost=x402-host-probe stoppedseesSERVICE_RESULT=exit-codeand writesfacilitator_failures_total 1.The evidence
systemctl stop x402@productionat 13:26:45 endedfailed:ExecStart … (code=exited, status=130),ExecStop … status=0/SUCCESS. When the agent started again,facilitator-failedfired at 13:46:53.facilitator_failures_totalalso has a point at 2026-11-06 13:37, the third deploy's restart.The fix
SuccessExitStatus=130indeploy/systemd/x402@.service, with a comment. A planned stop now endsinactive,SERVICE_RESULTissuccess, and the hook writes nothing.server.ts). The watchdog stops a hung container withdocker stop --time 10, which ends 137. A drain pastTimeoutStopSecends intimeout, and a failed start inexit-codewith another status.deploy/test/units.test.tsrequiresSuccessExitStatus=130onx402@. Removing the line fails it (1 failed, 31 passed).docs/monitoring.md'sfacilitator-failedsay that a planned stop is a success. 0027 gets a history entry.The other
compose upunits (x402-db-testnet@,x402-openbao,x402-tunnel@) probably also endfailedon a stop. They don't feed this alarm, so they're left as they are.To go live
The playbook installs the unit from
deploy/production, so this needs the next promotion, then a playbook run (daemon-reload). It takes effect at the next stop of the unit. Until then, a deploy firesfacilitator-failedonce. Expect it and say so in the channel. The server side is in Stan's lane, so a review from him is welcome.