Repository navigation
feat(alastor): ping krater's /healthz/edge every minute - #107
Open
Adambomb210 wants to merge 1 commit into
Open
Adambomb210 wants to merge 1 commit into
Adambomb210 wants to merge 1 commit into
Conversation
Krater stops a box, or fails a job and destroys its machine, when the runner on it goes silent. Runners report over Cloudflare, Traefik on alastor and the tailnet to weft, so when that path is down every runner looks dead and none is. A krater-edge-ping timer on alastor requests /healthz/edge on staging, and on production while its routes are on, every minute over that same path. Krater records it; when it stops arriving, Krater holds actions on silent runners and alerts (patchworklabsorg/krater, claude/network-hold). An HTTPS request rather than an ICMP ping, so it tests what runners use. alastor builds from jaspermayone/infra: this goes live when that input is bumped there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
alastor now requests Krater's
/healthz/edgeevery minute, so Krater can tell its runners going silent apart from its own network being down (patchworklabsorg/krater#83).Runners report to Krater over Cloudflare → alastor → tailnet → weft. When that path breaks, every runner looks dead, and Krater used to stop boxes or fail jobs for it. With this ping, Krater holds those actions while the ping is stale, and alerts after 10 min ("alastor's ping stopped reaching Krater").
What changes in
hosts/alastor/edge.nix:patchwork.edge.kraterEdgePing(defaulttrue).krater-edge-ping.service: a oneshot withDynamicUser. It runscurl -fsS -o /dev/null --max-time 20against each URL, and the unit fails if any of them fails.kraterProductionis on, like its routes.krater-edge-ping.timer:OnCalendar = "minutely".Deploy order:
/healthz/edgebefore this reaches alastor. Until it does, the production request gets a 404 and the unit fails every minute. That's noise only: Krater never holds for a ping it has never seen.After deploy:
systemctl status krater-edge-ping.timer krater-edge-pingon alastor.curl -fsS https://krater-staging.patchworklabs.org/healthz/edgeshould return{"status":"ok"}.Tested:
nix build .#checks.x86_64-linux.krater-weftand theweave-alastorcheck pass.krater-wefthas two new checks: the edge pings Krater every minute, and it pings production while production's routes are on. Formatted with the flake's nixfmt-tree. It hasn't run on alastor yet.🤖 Generated with Claude Code