Skip to content

Make CI tests ~3x faster - #58

Merged
danigb merged 2 commits into
mainfrom
chore/faster-ci
Sep 12, 2026
Merged

danigb merged 2 commits into
mainfrom
chore/faster-ci

Conversation

@danigb

@danigb danigb commented Sep 12, 2026 •

Copy link
Copy Markdown
Owner

CI runs were taking about 7 minutes, and Run tests was 354 s of that 415 s. Everything else — install, formatting, build, both generated-code checks — came to 61 s together.

Measured on this PR's own CI run: 415 s → 170 s.

Where the time went

Not where it looked. Measured by re-running the suite three ways:

Variant CPU time
jest --coverage (what CI ran) 377 s
no coverage 342 s → coverage was 8%
no type-checking 334 s → ts-jest was 2%

The remaining ~334 s was the test bodies, and most of it was not DSP. The suite made 3.5M expect() calls, 3.2M of them in the innermost loop of an exhaustive sweep. A matcher call costs ~19 µs whether it passes or fails, so the arp sweep alone spent 17 s inside jest proving the same invariant 900,000 times.

Benchmarked directly: 900k expect() in a loop is 16,760 ms; the same checks as && with one assertion after the loop is 26 ms.

What changed

Sweeps report once. Same corners, same order, same seeds, same conditions — only the reporting moved out of the hot loop. Failure messages get better, since naming the offending corner (mode, set size, octave, step) beats an expect(index).toBeGreaterThanOrEqual(0) that has lost its loop variables by the time it prints.

Negated forms are written !(a >= b) rather than a < b so that a NaN still fails, matching the matcher it replaced.

Verified by mutation: deleting the note fold in arp's dsp.ts still fails "keeps the pitch class when it folds", exactly as before. A mutation that this style misses was checked against the original tests too — they miss it as well, so nothing was narrowed.

Suite Before After
arp/dsp 50.7 s 3.6 s
clock/dsp 33.0 s 4.5 s
wavetable-oscillator/dsp 27.0 s 7.5 s
virtual-analog-filter/filters 46.8 s 18.6 s

Build split from test, tests sharded four ways. The tests import package sources (jest maps @synthlet/* at packages/*/src), so they never read dist/ and had no reason to wait for a build. Formatting, the build and its two checks now run beside the shards. Six shards were tried and the maximum did not move, so four it is.

--coverage dropped from CI. No coverageThreshold, no upload, no artifact: the report was written to a gitignored directory and discarded. npm run test:coverage still produces it on demand, and npm test is now the fast path locally too.

Result

This PR's CI run, all 2823 tests passing:

Job Time
build 66 s
shard 1 67 s
shard 2 73 s
shard 3 160 s
shard 4 105 s
test (gate) 3 s
wall clock 170 s (was 415 s)

No shipping code changed — only test files, the workflow, and package.json.

What still sets the floor

packages/karplus-strong/src/dsp.test.ts is 147 s of shard 3's 160 s on its own. Jest shards by file, so that one file is now the critical path and more shards cannot help. Its cost is genuine DSP rather than matcher overhead — it renders 30 s and 60 s of audio per test to measure decay, which needs the horizon.

Getting below ~170 s means splitting that file or shortening its renders. Left alone here: it is a change to what the tests measure, not to how they report, and it deserves its own PR.

Note on branch protection

test is the required status check on main, so a matrix job of that name would have produced test (1)…test (4) and blocked every PR. test is now an aggregate job that passes only when the build and all four shards did; the name survives and protection needs no re-pointing. Confirmed green on this run.

The test suite spent most of its time inside jest's matchers rather than
inside the DSP. 3.5M `expect()` calls, and 3.2M of them sat in the
innermost loop of an exhaustive sweep, where the assertion is re-proved
for every step of every corner.

A matcher call costs ~19 us whether it passes or fails. Measured on the
arp sweep: 900k `expect()` in a loop is 16.8 s, and the same checks
written as `&&` with one assertion after the loop is 26 ms.

So the sweeps now scan and report once. Nothing about the coverage
changes: the same corners are visited in the same order with the same
seeds, and the conditions are identical. The failure messages get
better, because naming the offending corner - mode, set size, octave,
step - says more than a bare `expect(index).toBeGreaterThanOrEqual(0)`
that has lost the loop variables by the time it prints.

Verified by mutation: deleting the note fold in `arp`'s `dsp.ts` still
fails "keeps the pitch class when it folds", as it did before.

  arp/dsp                    50.7 s -> 3.6 s
  clock/dsp                  33.0 s -> 4.5 s
  wavetable-oscillator/dsp   27.0 s -> 7.5 s
  virtual-analog-filter      46.8 s -> 18.6 s

Whole suite: 398 s of CPU -> 306 s, 2823 tests still passing.
The suite is CPU-bound and a hosted runner has 4 cores, so the wall clock
was set by how much work landed on one machine: 354 s of a 415 s run, with
install, formatting, the build and both generated-code checks together
accounting for the other 61 s.

Three changes.

Split build from test. The tests import package sources - jest's
moduleNameMapper points `@synthlet/*` at `packages/*/src` - so they never
read `dist/` and had no reason to wait for a build. Formatting, the build
and the two checks that depend on it now run beside the shards instead of
in front of them.

Shard the tests four ways. Measured at a runner's parallelism, the
shards come out 22 s / 16 s / 45 s / 42 s: uneven, because jest shards by
file and one package's test file is the long pole, but the maximum is
what counts. Six shards were tried and the maximum did not move, so the
extra runners would have been spent on setup.

Drop `--coverage`. There is no `coverageThreshold`, no upload and no
artifact - the report was written to a gitignored directory and thrown
away. `npm run test:coverage` still produces it on demand, and `npm test`
is now the fast path locally too.

`test` stays the name of the required status check on main. It is now an
aggregate job that passes only when the build and all four shards did,
so branch protection keeps working without being re-pointed.
@danigb
danigb merged commit ca38b94 into main Sep 12, 2026
6 checks passed
@danigb
danigb deleted the chore/faster-ci branch September 12, 2026 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant