Skip to content

fix(bootstrap): first-boot FK race + single-flight - #1019

Open
mmcintosh wants to merge 4 commits into
mainfrom
mark/fix-beta25-bootstrap-fk-race
Open

mmcintosh wants to merge 4 commits into
mainfrom
mark/fix-beta25-bootstrap-fk-race

Conversation

@mmcintosh

@mmcintosh mmcintosh commented Jul 20, 2026 •

Copy link
Copy Markdown
Collaborator

Description

On a fresh database's first boot, bootstrapMiddleware seeds system data in one Promise.all. In that batch, two steps create document_types rows while two others insert documents that reference them by foreign key.

When an insert wins the race, D1 throws FOREIGN KEY constraint failed and the per-step .catch() swallows it. RBAC roles and core plugins are silently left unseeded, and the admin gets "You do not have permission to access this area".

Bootstrap then writes its KV skip-marker anyway (24h TTL), which locks the broken state in. It happens on every first boot of a fresh D1: 11 of 11 local boots on main. A user also confirmed it on a production deploy of beta.28 (see #1017).

Fixes #1017. Refs #1054 (the "no permission" part).

Part of a batch of five login fixes: #1103 (Better Auth 1.7.7), #1104 (register-form credential row), #1105 (password changes reaching Better Auth) and #1063. The #1063 seed-admin gate should merge after this one.

Changes

All in packages/core/src/middleware/bootstrap.ts, plus a small change to services/plugin-bootstrap.ts:

  • Two-phase seed. Document-type producers run and finish before the consumers that insert documents.
  • Single-flight. Concurrent requests on the same cold isolate wait on one bootstrap instead of racing parallel ones.
  • Marker only on success. The completion latch and KV marker are written only when every step succeeded.
  • Hardening, added after an independent code review:
    • A cancelled first boot no longer stalls the isolate. Bootstrap runs under executionCtx.waitUntil. Waiting requests give up after 20s and carry on without bootstrapping, so a missing role means 403, not open access. A latch older than 60s is treated as dead.
    • Backoff after a failed run. The next full attempt waits at least 30s per isolate instead of re-running on every request. The telemetry snapshot is sent only after a successful run.
    • The KV fast path checks the seed, not just the marker. It trusts the marker only if this database has the role-admin document, plus the core-auth plugin document unless plugins are disabled. This matters in CI, where every PR preview shares one KV namespace and the marker key depends only on the version.
    • A failed core plugin install counts as a failed step. bootstrapCorePlugins() returns the failed ids. The limit is documented in the code: isBootstrapNeeded() looks only at core-* plugins, so for a bootstrapped plugin whose name doesn't start with core-, a failure blocks the marker for one run only. Widening it would reinstall plugins a user uninstalled.
    • Known gap: autoRegisterCollectionDocumentTypes still catches each collection's failure, so a failed collection type doesn't block the marker.

Conflict: #1097 also edits bootstrap.ts and conflicts with this PR in two places, including resetBootstrap(). Whichever merges second needs a rebase with a hand merge there; I'll do it. Once #1097 is in, the security-check calls this PR adds to the waiting and backoff paths can go, because #1097 runs the check on every request.

Testing

  • bootstrap-fk-ordering.test.ts (real SQLite, foreign_keys = ON): on the pre-fix code, the end-to-end middleware case's RBAC seed hits FOREIGN KEY constraint failed, which is swallowed, so no roles are written and the test fails. It passes with the fix.

  • bootstrap-resilience.test.ts (new, 8 tests) covers:

    • a hung run plus the timeout;
    • the stale latch;
    • the backoff count;
    • telemetry only on success;
    • the fast path rejecting a marker over an unseeded database;
    • no marker after a failed plugin install.

    Removing each fix fails exactly its own test.

  • Fresh greenfield boots (clean D1 + empty KV), measured on the original commit 7b2151c25: a single first request gave zero FK errors and a fully seeded database, and four concurrent cold requests ran the bootstrap once. No unit test covers single-flight itself.

Unit Tests

  • Added/updated unit tests
  • All unit tests passing (this branch: 1742 / 0 on its base; all five batch PRs merged on current main: 2000 / 0)

E2E Tests

  • Added/updated E2E tests. None added: CI creates a fresh D1 per run, but its readiness probe is the first request, before any spec runs, so a spec can't observe first boot. The unit tests above cover it with a real SQLite database.
  • All E2E tests passing (not run locally, per the repo's E2E policy)

Screenshots/Videos

N/A. No UI changes.

Checklist

  • Code follows project conventions
  • Tests added/updated and passing
  • Type checking passes
  • No console errors or warnings
  • Documentation updated (if needed): N/A

@mmcintosh
mmcintosh marked this pull request as ready for review July 20, 2026 16:43
@mmcintosh
mmcintosh requested a review from lane711 as a code owner July 20, 2026 16:43
@mmcintosh

Copy link
Copy Markdown
Collaborator Author

Note on the red CI. The failing check here is the unit-test step (about 25 tests), and those failures are pre-existing on main — this branch is cut straight from current main, and none of them touch bootstrap.ts. They are the known beta.25 test-harness issue (a per-database cache leaks generated columns across in-memory test DBs, producing "no such column: q_*" errors) and are fixed by the test-infra PR (#1018, Fixes #1016). Type-check passes here, and this PR's purpose is to fix the E2E against preview failure: the loginAsAdmin timeout caused by the first-boot FK race that leaves the RBAC roles and core plugins unseeded (#1017). The two are a pair: #1018 fixes this PR's unit suite, and this PR fixes #1018's E2E preview. Recommend landing #1018 first, then this one — after both, CI is green across the stack.

@mmcintosh mmcintosh self-assigned this Jul 20, 2026
@mmcintosh
mmcintosh force-pushed the mark/fix-beta25-bootstrap-fk-race branch from 1f18e3c to ef70ba7 Compare August 19, 2026 02:26
…rs + single-flight guard

Fixes the greenfield first-boot race (two coupled facets), observed live on a fresh
D1 + empty KV on beta.25:

FACET 1 — intra-bootstrap FK ordering. The system-data seed ran all five steps in one
Promise.all, but two of them (RBAC seed, core-plugin bootstrap) INSERT INTO documents
whose type_id FK-references document_types rows that the other two steps
(bootstrapDocumentTypes, autoRegisterCollectionDocumentTypes) are still creating in
parallel. A consumer insert winning the race throws FOREIGN KEY constraint failed; the
per-step .catch() swallows it, so RBAC roles/verbs and core plugins are silently left
unseeded (admin routes then 403, plugins inactive).

FACET 2 — no single-flight guard. bootstrapComplete only flips at the END of a run, so
two concurrent requests on the same cold isolate both pass the completion/KV checks and
start SECOND parallel seed — re-opening the FK window across the two runs.

Compounding both: the KV skip-marker was written UNCONDITIONALLY after the batch (24h
TTL), latching the partial state so every later request/isolate skips D1 and the broken
bootstrap persists across restarts until the marker expires or is cleared.

Fix (bootstrap.ts):
- Two phases. Phase A awaits the document-type PRODUCERS first; Phase B runs the
  CONSUMERS + independent credential repair in parallel (latency preserved).
- Single-flight in-flight promise: a concurrent request on a cold isolate awaits the
  running bootstrap instead of starting a second one. Check-and-set is synchronous
  (atomic), released in finally.
- runStep() records failures; the completion latch + KV marker are written ONLY when
  every step succeeded, so a transient failure self-heals on the next request.

Regression test (bootstrap-fk-ordering.test.ts): real SQLite, foreign_keys=ON (the
shared harness disables them; here the FK is the subject). Proves insert-before-type
throws FK + writes nothing, and that registering types first lets consumer inserts land.

Reality-tested on fresh greenfield boots (clean D1 + empty KV) against beta.25:
- single first request → clean boot, 0 FK errors, 21 types / 4 roles / 6 verbs / 4 plugins.
- 4 CONCURRENT cold-isolate requests → bootstrap body runs exactly ONCE, completes clean,
  0 FK errors (proves the single-flight guard).
Before the fix the same first boot always FK-failed and required a KV-clear + reboot.

Pre-existing: full suite shows 25 failures on this branch, identical to pristine
origin/main; none touch bootstrap. They are the separately-owned beta.25 unit-suite
repair (per-DB cache + executionCtx). Green CI requires that repair to land first.

Coordination: touches ONLY bootstrap.ts + its test — no overlap with the beta.25 repair
(plugin-middleware.ts, document-scalar-schema.ts, api.ts, catalog.ts).
…rify the seed behind the KV marker

Follow-ups to the first-boot race fix (#1017):

- A run that never settles no longer blocks the isolate. The bootstrap runs under
  waitUntil so it outlives a cancelled first request; requests wait on it for at most
  20s and then proceed; a latch older than 60s is treated as dead and replaced.
- After a run that finished with errors, the next full attempt waits 30s instead of
  re-running ~10s of D1 work on every request. The project snapshot telemetry is sent
  only for a successful run.
- The KV fast path now checks that THIS database has the system seed (role-admin, and
  the core-auth plugin unless plugins are disabled) instead of just a non-empty
  document_types. The marker is keyed by version, not database, and CI previews share
  one KV namespace.
- A core plugin that fails to install now counts as a failed step, so the marker is not
  written. bootstrapCorePlugins() returns the ids that failed (it still installs the
  rest).
- resetBootstrap() releases waiters; a dangling comment reference is fixed.

Tests: bootstrap-resilience.test.ts (real SQLite), one case per fix; each fails with its
fix reverted.
A failed non-core plugin keeps the marker unwritten for that run only: the next run
skips plugin bootstrap when the core-* plugins are present (isBootstrapNeeded).
Widening the check would reinstall plugins a user uninstalled, so the limit is
documented rather than changed.

This branch was successfully deployed

1 active deployment
internal — 75b5a4c2 Deployed Oct 13, 2026 by mmcintosh via authorize #1611
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

First-boot FK race + no single-flight leaves RBAC/plugins unseeded and latches the broken state (beta.25)

1 participant