Skip to content

GPUI frame benchmarks: interleaved A/B per platform, scene-equivalence gate, Cap editor A/B #3886

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
richiemcilroy merged 13 commits into from
Nov 6, 2026
Merged

GPUI frame benchmarks: interleaved A/B per platform, scene-equivalence gate, Cap editor A/B #3886

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
b128989
Add GPUI frame benchmarks with interleaved A/B runs on macOS, Windows…
richiemcilroy Nov 6, 2026
944b5eb
Bound Criterion samples per benchmark round
richiemcilroy Nov 6, 2026
7c8eb20
Add an interleaved A/B run of the Cap editor's draw benchmark
richiemcilroy Nov 6, 2026
de368f5
Add a callgrind profile of the GPUI frame benchmarks
richiemcilroy Nov 6, 2026
138ebad
Build both benchmark sides from one checkout path
richiemcilroy Nov 6, 2026
02c5a76
Compare the editor benchmark by median draw over five rounds
richiemcilroy Nov 6, 2026
e0b4c01
Resolve benchmark executables before changing directory; allow choosi…
richiemcilroy Nov 6, 2026
e519b4d
Report each benchmark size on its own, and compare per-frame draw time
richiemcilroy Nov 6, 2026
5e28ae0
Profile 15 s of frames per benchmark
richiemcilroy Nov 6, 2026
1825c6c
Let local runs pick the benchmark toolchain (GPUI_BENCH_TOOLCHAIN, de…
richiemcilroy Nov 6, 2026
05126c6
Gate GPUI patch changes on identical scenes, frame by frame
richiemcilroy Nov 6, 2026
e3a8c7f
Scene gate: cover atlas pixels and missing frames; keep fork PRs off …
richiemcilroy Nov 6, 2026
e9fad53
Fix relative bench output paths; trim comments to non-obvious context
richiemcilroy Nov 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 142 additions & 0 deletions .github/workflows/gpui-app-bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
name: GPUI app benchmark

on:
pull_request:
paths:
- "apps/desktop-gpui/patches/**"
- "scripts/gpui-app-bench.py"
- ".github/workflows/gpui-app-bench.yml"

@superagent-security superagent-security Bot Nov 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Untrusted fork PRs can repeatedly queue multi-hour GPUI benchmark jobs by editing benchmark-only paths

New PR workflows trigger 120-150 min macOS/Windows/Linux benchmark jobs when only the workflow, its scripts, or gpui-benches crate change.

Gate self-file/benchmark-triggered jobs to internal PRs or a maintainer-approved label/environment; keep fork runs bounded.

AI prompt 
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/gpui-app-bench.yml">
<violation number="1" location=".github/workflows/gpui-app-bench.yml:15">
<priority>P2</priority>
<title>Untrusted fork PRs can repeatedly queue multi-hour GPUI benchmark jobs by editing benchmark-only paths</title>
<evidence>The new `pull_request` trigger lists the workflow itself and `scripts/gpui-app-bench.py` as paths (lines 11-15), so a fork PR that changes nothing but this workflow or its runner script starts the `editor` job on `blacksmith-6vcpu-macos-15` with `timeout-minutes: 150` (lines 36-39). That job runs `node scripts/setup.js`, prepares head and base GPUI checkouts, builds `cap-gpui` twice (lines 107-112), and then runs the real editor timeline benchmark for `--rounds 5` interleaved A/B rounds over the reference recording (lines 126-139). The new `.github/workflows/gpui-bench.yml` has the same shape: paths at lines 12-17 include `apps/desktop-gpui/gpui-benches/**`, `scripts/gpui-bench-*`, and the workflow file itself, and each matching PR push queues three `bench` jobs (150-minute timeout) plus three `test` jobs (120-minute timeout) across macOS, Windows, and Linux. Neither workflow carries a `github.event.pull_request.head.repo.full_name == github.repository` condition, a required label, or an `environment:` approval gate; `concurrency` with `cancel-in-progress: true` only cancels overlapping runs and does not stop sequential pushes from each consuming a full run. The 150-minute and 120-minute timeouts are the worst-case per-run cost, and each run performs two full Cargo builds plus the interleaved measurement workload, so a fork contributor can repeatedly trigger expensive shared-runner work with a trivial edit to benchmark-only files.</evidence>
<recommendation>Restrict the self-file/benchmark-only triggers to internal pull requests, e.g. add `if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository` to the `bench`/`test`/`editor` jobs (or drop the workflow/script/crate paths from `pull_request.paths` and run them only from `workflow_dispatch` / `push` on trusted branches). For fork PRs, keep the benchmark suite reachable only through a maintainer-applied label or an `environment:` approval, and keep a short bounded validation run for workflow-only changes.</recommendation>
</violation>
</file>

@richiemcilroy richiemcilroy Nov 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e3a8c7f: the bench and test jobs in gpui-bench.yml and the editor job in gpui-app-bench.yml now run only when github.event.pull_request.head.repo.full_name == github.repository (or on push/workflow_dispatch). Fork PRs no longer start them; maintainers can still measure a fork's patch with workflow_dispatch.

@superagent-security superagent-security Bot Nov 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctly fixed: gating the long-running bench, test, and editor jobs to internal PRs via github.event.pull_request.head.repo.full_name == github.repository (and push/workflow_dispatch) prevents untrusted forks from automatically queuing expensive benchmark runs, while preserving maintainer flexibility. Dismissing.

push:
branches:
- "gpui-perf/**"
workflow_dispatch:
inputs:
base:
description: "Ref whose patch stack is the baseline (default: the merge base with main)"
default: ""
rounds:
description: "Interleaved rounds per side"
default: "5"

concurrency:
group: gpui-app-bench-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

permissions:
contents: read

jobs:
editor:
name: Editor draw benchmark (macOS)
# Benchmarks build GPUI twice and run for up to two hours, so pull requests
# from forks do not start them; maintainers can run them with workflow_dispatch.
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: blacksmith-6vcpu-macos-15
timeout-minutes: 150
env:
CARGO_PROFILE_DEV_DEBUG: "0"
TARGET: aarch64-apple-darwin
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
persist-credentials: false

- name: Checkout the reference recording
uses: actions/checkout@v4
with:
repository: CapSoftware/cap-performance-fixtures
path: performance-fixtures
persist-credentials: false

- name: Rust setup
uses: dtolnay/rust-toolchain@1.95.0
with:
targets: ${{ env.TARGET }}

- uses: ./.github/actions/setup-rust-cache
with:
target: ${{ env.TARGET }}
prefix-key: v1-rust-gpui
save-cache: "false"
workspaces: apps/desktop-gpui -> target

- name: Run setup
shell: bash
run: node scripts/setup.js
env:
RUST_TARGET_TRIPLE: ${{ env.TARGET }}

- name: Prepare head and base GPUI checkouts
shell: bash
env:
BASE_INPUT: ${{ inputs.base }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
run: |
set -euo pipefail
node scripts/sync-desktop-versions.mjs
./scripts/prepare-gpui-dependency.sh
base="${BASE_INPUT:-}"
if [[ -z "$base" ]]; then
base="$(git merge-base HEAD "${PR_BASE:-origin/main}")"
fi
echo "Head $(git rev-parse --short HEAD), base patch stack from $(git rev-parse --short "$base")" | tee -a "$GITHUB_STEP_SUMMARY"
git worktree add --detach "$RUNNER_TEMP/base/cap" "$base"
(cd "$RUNNER_TEMP/base/cap" && ./scripts/prepare-gpui-dependency.sh)

- name: Build head and base
shell: bash
run: |
set -euo pipefail
zed="$(cd .. && pwd)/zed-cap"
bin_dir="$GITHUB_WORKSPACE/apps/desktop-gpui/target/$TARGET/debug"
build() {
# The app patches gpui to ../zed-cap, so each build swaps the
# checkout in. Cargo decides freshness of path dependencies by
# mtime, so touch the swapped-in sources to force a rebuild.
find "$zed/crates" -type f -name '*.rs' -exec touch {} +
cargo +1.95.0 build --manifest-path apps/desktop-gpui/Cargo.toml --locked \
--target "$TARGET" --bin cap-gpui
cp "$bin_dir/cap-gpui" "$bin_dir/cap-gpui-$1"
}
build head
mv "$zed" "$zed-head"
mv "$RUNNER_TEMP/base/zed-cap" "$zed"
build base
mv "$zed" "$RUNNER_TEMP/base/zed-cap"
mv "$zed-head" "$zed"
for frameworks in "$bin_dir/Frameworks" "$GITHUB_WORKSPACE/apps/desktop-gpui/target/$TARGET/Frameworks"; do
mkdir -p "$frameworks"
ln -sfn "$GITHUB_WORKSPACE/target/Frameworks/Spacedrive.framework" "$frameworks/Spacedrive.framework"
done
if cmp -s "$bin_dir/cap-gpui-head" "$bin_dir/cap-gpui-base"; then
echo "Head and base builds are identical (no GPUI change)." | tee -a "$GITHUB_STEP_SUMMARY"
fi

- name: Keep the display awake
run: |
sudo pmset -a displaysleep 0 sleep 0 || true
caffeinate -u -t 2

- name: Measure
shell: bash
env:
ROUNDS: ${{ inputs.rounds || '5' }}
run: |
bin_dir="$GITHUB_WORKSPACE/apps/desktop-gpui/target/$TARGET/debug"
python3 scripts/gpui-app-bench.py \
--base "$bin_dir/cap-gpui-base" \
--head "$bin_dir/cap-gpui-head" \
--fixture "$GITHUB_WORKSPACE/performance-fixtures/reference-recording.cap" \
--out "$RUNNER_TEMP/app-bench" \
--rounds "$ROUNDS" \
--label macOS \
--summary "$GITHUB_STEP_SUMMARY"

- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: always()
with:
name: gpui-app-bench-macos
path: ${{ runner.temp }}/app-bench
if-no-files-found: ignore
254 changes: 254 additions & 0 deletions .github/workflows/gpui-bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
name: GPUI frame benchmarks

on:
pull_request:
paths:
- "apps/desktop-gpui/patches/**"
- "apps/desktop-gpui/gpui-benches/**"
- "scripts/prepare-gpui-dependency.sh"
- "scripts/gpui-bench-*"
- "scripts/gpui-scene-digest.py"
- ".github/workflows/gpui-bench.yml"
push:
branches:
- "gpui-perf/**"
workflow_dispatch:
inputs:
base:
description: "Ref whose patch stack is the baseline (default: the merge base with main)"
default: ""
rounds:
description: "Interleaved rounds per side"
default: "5"

concurrency:
group: gpui-bench-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

permissions:
contents: read

jobs:
bench:
name: GPUI benchmarks (${{ matrix.settings.name }})
# Benchmarks build GPUI twice and run for up to two hours, so pull requests
# from forks do not start them; maintainers can run them with workflow_dispatch.
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
timeout-minutes: 150
strategy:
fail-fast: false
matrix:
settings:
- name: macOS
slug: macos
runner: blacksmith-6vcpu-macos-15
- name: Windows
slug: windows
runner: blacksmith-8vcpu-windows-2025
- name: Linux
slug: linux
runner: blacksmith-8vcpu-ubuntu-2404
runs-on: ${{ matrix.settings.runner }}
env:
CARGO_TARGET_DIR: ${{ github.workspace }}/../gpui-bench-target
CARGO_PROFILE_BENCH_DEBUG: "false"
CARGO_INCREMENTAL: "0"
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
persist-credentials: false

- name: Install Linux dependencies
if: ${{ runner.os == 'Linux' }}
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends build-essential pkg-config clang libclang-dev \
libx11-dev libx11-xcb-dev libxcb1-dev libxkbcommon-dev libxkbcommon-x11-dev libwayland-dev \
libfontconfig1-dev libfreetype-dev libvulkan1 mesa-vulkan-drivers libssl-dev libasound2-dev \
libzstd-dev fonts-dejavu-core

- name: Allow long paths in Zed's git dependencies
if: ${{ runner.os == 'Windows' }}
run: git config --global core.longpaths true

- name: Rust setup
uses: dtolnay/rust-toolchain@1.95.0

- uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Prepare head and base GPUI checkouts
shell: bash
env:
BASE_INPUT: ${{ inputs.base }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
run: |
set -euo pipefail
./scripts/prepare-gpui-dependency.sh
base="${BASE_INPUT:-}"
if [[ -z "$base" ]]; then
base="$(git merge-base HEAD "${PR_BASE:-origin/main}")"
fi
echo "Head $(git rev-parse --short HEAD), base patch stack from $(git rev-parse --short "$base")" | tee -a "$GITHUB_STEP_SUMMARY"
git worktree add --detach "$RUNNER_TEMP/base/cap" "$base"
(cd "$RUNNER_TEMP/base/cap" && ./scripts/prepare-gpui-dependency.sh)
echo "ZED_HEAD=$(cd ../zed-cap && pwd)" >> "$GITHUB_ENV"
echo "ZED_LOCK_HASH=$(python -c 'import hashlib,sys; print(hashlib.sha256(open(sys.argv[1],"rb").read()).hexdigest()[:16])' ../zed-cap/Cargo.lock)" >> "$GITHUB_ENV"
echo "ZED_BASE=$RUNNER_TEMP/base/zed-cap" >> "$GITHUB_ENV"

- name: Cargo cache
uses: actions/cache@v4
with:
path: |
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
${{ github.workspace }}/../gpui-bench-target
key: gpui-bench-v1-${{ runner.os }}-${{ env.ZED_LOCK_HASH }}-${{ github.run_id }}
restore-keys: |
gpui-bench-v1-${{ runner.os }}-${{ env.ZED_LOCK_HASH }}-
gpui-bench-v1-${{ runner.os }}-

# Both sides build from the same checkout path, swapped in turn. Paths
# end up in symbol hashes and panic locations, so building them from two
# paths gives two code layouts that differ by a few percent with
# identical sources. Cargo judges path dependencies by mtime, so the
# swapped-in sources are touched to force the rebuild.
- name: Build benchmarks
shell: bash
run: |
set -euo pipefail
build() {
find "$ZED_HEAD/crates" -type f -name '*.rs' -exec touch {} +
./scripts/gpui-bench-build.sh "$ZED_HEAD" "$RUNNER_TEMP/bench/$1"
}
mv "$ZED_HEAD" "$ZED_HEAD.head"
mv "$ZED_BASE" "$ZED_HEAD"
build base
mv "$ZED_HEAD" "$ZED_BASE"
mv "$ZED_HEAD.head" "$ZED_HEAD"
build head
if cmp -s "$RUNNER_TEMP/bench/base/frame"* "$RUNNER_TEMP/bench/head/frame"*; then
echo "Base and head benchmark builds are identical." | tee -a "$GITHUB_STEP_SUMMARY"
fi

- name: Measure
shell: bash
env:
ROUNDS: ${{ inputs.rounds || '5' }}
run: |
python scripts/gpui-bench-compare.py \
--base "$RUNNER_TEMP/bench/base" \
--head "$RUNNER_TEMP/bench/head" \
--out "$RUNNER_TEMP/bench/runs" \
--rounds "$ROUNDS" \
--label "${{ matrix.settings.name }}" \
--summary "$GITHUB_STEP_SUMMARY"

# Only macOS has a headless renderer that sees submitted frames; the
# step reports a skip elsewhere.
- name: Scene equivalence
shell: bash
run: |
python scripts/gpui-scene-digest.py \
--base "$RUNNER_TEMP/bench/base" \
--head "$RUNNER_TEMP/bench/head" \
--out "$RUNNER_TEMP/bench/digests" \
--summary "$GITHUB_STEP_SUMMARY"

- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: always()
with:
name: gpui-bench-${{ matrix.settings.slug }}
path: |
${{ runner.temp }}/bench/runs/*.md
${{ runner.temp }}/bench/runs/*.json
${{ runner.temp }}/bench/runs/*.log
if-no-files-found: ignore

test:
name: GPUI tests (${{ matrix.settings.name }})
# Same as the benchmarks: GPUI's test builds are long, so forks do not start them.
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
settings:
- name: macOS
runner: blacksmith-6vcpu-macos-15
- name: Windows
runner: blacksmith-8vcpu-windows-2025
- name: Linux
runner: blacksmith-8vcpu-ubuntu-2404
runs-on: ${{ matrix.settings.runner }}
env:
CARGO_PROFILE_DEV_DEBUG: "0"
CARGO_INCREMENTAL: "0"
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
persist-credentials: false

- name: Install Linux dependencies
if: ${{ runner.os == 'Linux' }}
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends build-essential pkg-config clang libclang-dev \
libx11-dev libx11-xcb-dev libxcb1-dev libxkbcommon-dev libxkbcommon-x11-dev libwayland-dev \
libfontconfig1-dev libfreetype-dev libvulkan1 mesa-vulkan-drivers libegl1 libegl-mesa0 \
libgl1-mesa-dri libssl-dev libasound2-dev libzstd-dev fonts-dejavu-core

- name: Allow long paths in Zed's git dependencies
if: ${{ runner.os == 'Windows' }}
run: git config --global core.longpaths true

- name: Rust setup
uses: dtolnay/rust-toolchain@1.95.0

- uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Prepare GPUI dependency
shell: bash
run: |
./scripts/prepare-gpui-dependency.sh
echo "ZED_LOCK_HASH=$(python -c 'import hashlib,sys; print(hashlib.sha256(open(sys.argv[1],"rb").read()).hexdigest()[:16])' ../zed-cap/Cargo.lock)" >> "$GITHUB_ENV"

- name: Cargo cache
uses: actions/cache@v4
with:
path: |
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
${{ github.workspace }}/../zed-cap/target
key: gpui-test-v1-${{ runner.os }}-${{ env.ZED_LOCK_HASH }}-${{ github.run_id }}
restore-keys: |
gpui-test-v1-${{ runner.os }}-${{ env.ZED_LOCK_HASH }}-
gpui-test-v1-${{ runner.os }}-

- name: gpui tests
shell: bash
working-directory: ../zed-cap
run: cargo +1.95.0 test -p gpui --features test-support --lib --tests --no-fail-fast

- name: Platform crate tests
shell: bash
working-directory: ../zed-cap
env:
GPUI_REQUIRE_TEST_DEVICE: "1"
run: |
set -euo pipefail
if [[ "$RUNNER_OS" == "Linux" ]]; then
WGPU_BACKEND=vulkan cargo +1.95.0 test -p gpui_wgpu --lib --no-fail-fast
elif [[ "$RUNNER_OS" == "macOS" ]]; then
cargo +1.95.0 test -p gpui_macos --lib --no-fail-fast
else
cargo +1.95.0 test -p gpui_windows --lib --no-fail-fast
fi
Loading
Loading