Repository navigation
Make CI tests ~3x faster - #58
Merged
Merged
Conversation
The test suite spent most of its time inside jest's matchers rather than inside the DSP. 3.5M `expect()` calls, and 3.2M of them sat in the innermost loop of an exhaustive sweep, where the assertion is re-proved for every step of every corner. A matcher call costs ~19 us whether it passes or fails. Measured on the arp sweep: 900k `expect()` in a loop is 16.8 s, and the same checks written as `&&` with one assertion after the loop is 26 ms. So the sweeps now scan and report once. Nothing about the coverage changes: the same corners are visited in the same order with the same seeds, and the conditions are identical. The failure messages get better, because naming the offending corner - mode, set size, octave, step - says more than a bare `expect(index).toBeGreaterThanOrEqual(0)` that has lost the loop variables by the time it prints. Verified by mutation: deleting the note fold in `arp`'s `dsp.ts` still fails "keeps the pitch class when it folds", as it did before. arp/dsp 50.7 s -> 3.6 s clock/dsp 33.0 s -> 4.5 s wavetable-oscillator/dsp 27.0 s -> 7.5 s virtual-analog-filter 46.8 s -> 18.6 s Whole suite: 398 s of CPU -> 306 s, 2823 tests still passing.
The suite is CPU-bound and a hosted runner has 4 cores, so the wall clock was set by how much work landed on one machine: 354 s of a 415 s run, with install, formatting, the build and both generated-code checks together accounting for the other 61 s. Three changes. Split build from test. The tests import package sources - jest's moduleNameMapper points `@synthlet/*` at `packages/*/src` - so they never read `dist/` and had no reason to wait for a build. Formatting, the build and the two checks that depend on it now run beside the shards instead of in front of them. Shard the tests four ways. Measured at a runner's parallelism, the shards come out 22 s / 16 s / 45 s / 42 s: uneven, because jest shards by file and one package's test file is the long pole, but the maximum is what counts. Six shards were tried and the maximum did not move, so the extra runners would have been spent on setup. Drop `--coverage`. There is no `coverageThreshold`, no upload and no artifact - the report was written to a gitignored directory and thrown away. `npm run test:coverage` still produces it on demand, and `npm test` is now the fast path locally too. `test` stays the name of the required status check on main. It is now an aggregate job that passes only when the build and all four shards did, so branch protection keeps working without being re-pointed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI runs were taking about 7 minutes, and
Run testswas 354 s of that 415 s. Everything else — install, formatting, build, both generated-code checks — came to 61 s together.Measured on this PR's own CI run: 415 s → 170 s.
Where the time went
Not where it looked. Measured by re-running the suite three ways:
jest --coverage(what CI ran)The remaining ~334 s was the test bodies, and most of it was not DSP. The suite made 3.5M
expect()calls, 3.2M of them in the innermost loop of an exhaustive sweep. A matcher call costs ~19 µs whether it passes or fails, so the arp sweep alone spent 17 s inside jest proving the same invariant 900,000 times.Benchmarked directly: 900k
expect()in a loop is 16,760 ms; the same checks as&&with one assertion after the loop is 26 ms.What changed
Sweeps report once. Same corners, same order, same seeds, same conditions — only the reporting moved out of the hot loop. Failure messages get better, since naming the offending corner (mode, set size, octave, step) beats an
expect(index).toBeGreaterThanOrEqual(0)that has lost its loop variables by the time it prints.Negated forms are written
!(a >= b)rather thana < bso that a NaN still fails, matching the matcher it replaced.Verified by mutation: deleting the note fold in arp's
dsp.tsstill fails "keeps the pitch class when it folds", exactly as before. A mutation that this style misses was checked against the original tests too — they miss it as well, so nothing was narrowed.arp/dspclock/dspwavetable-oscillator/dspvirtual-analog-filter/filtersBuild split from test, tests sharded four ways. The tests import package sources (jest maps
@synthlet/*atpackages/*/src), so they never readdist/and had no reason to wait for a build. Formatting, the build and its two checks now run beside the shards. Six shards were tried and the maximum did not move, so four it is.--coveragedropped from CI. NocoverageThreshold, no upload, no artifact: the report was written to a gitignored directory and discarded.npm run test:coveragestill produces it on demand, andnpm testis now the fast path locally too.Result
This PR's CI run, all 2823 tests passing:
No shipping code changed — only test files, the workflow, and
package.json.What still sets the floor
packages/karplus-strong/src/dsp.test.tsis 147 s of shard 3's 160 s on its own. Jest shards by file, so that one file is now the critical path and more shards cannot help. Its cost is genuine DSP rather than matcher overhead — it renders 30 s and 60 s of audio per test to measure decay, which needs the horizon.Getting below ~170 s means splitting that file or shortening its renders. Left alone here: it is a change to what the tests measure, not to how they report, and it deserves its own PR.
Note on branch protection
testis the required status check onmain, so a matrix job of that name would have producedtest (1)…test (4)and blocked every PR.testis now an aggregate job that passes only when the build and all four shards did; the name survives and protection needs no re-pointing. Confirmed green on this run.