[richiemcilroy]

Fixes #4016. Fixes #3928.

Intel's Media Foundation H.264 encoder (mfx_mft_h264ve_64.dll) faults with an access violation instead of returning an error at some frame sizes. FFmpeg's h264_mf loads it into Cap's own process during the pre-flight self-test, so the fault ends the app before any fallback can run:

• Crush on Export on Windows #4016: export crashes every time at mfx_mft_h264ve_64.dll+0x00143100 (same offset every time) at the recorded area's size, on 0.6.2 and 0.6.3. A clean 16:9 export works. The 0.6.3 log shows h264_qsv failing its self-test, which leaves h264_mf next in the Intel order.
• force close when start recording (Windows 26H2 fresh install) #3928: Studio recording start force-closes on Intel UHD 630, WER APPCRASH in mfx_mft_h264ve_64.dll, c0000005. Moving Cap to the NVIDIA GPU avoids it.
• fix(desktop-gpui): stop a crashing hardware encoder from crash-looping recordings #3904 (0.6.3) only stops the crash loop: it skips the encoder at the next launch, and resets on every app update. So every user still loses one export or recording per version.

Root cause

• FFmpeg's h264_mf copies each frame into a tightly packed system-memory buffer at the exact output size (stride = width) and sets MF_MT_FRAME_SIZE to that size. Intel documents its encoder transform as taking widths in multiples of 16. At other widths it reads past the buffer.
• Export widths are rounded to a multiple of 4 and recording widths to a multiple of 2 (ProjectUniforms::get_output_size, ensure_even). So any recorded area or window that isn't a standard 16:9 size reaches the transform at an unsupported width. The 16:9 presets (1280, 1920, 3840 wide) are all multiples of 16, which is why those work.
• FFmpeg's h264_qsv pads its own surfaces to 16 alignment, so it is not affected.

What changed

flowchart LR
  A[next H264 candidate] --> B{h264_mf and Intel MFT first<br/>and width % 16 != 0 or over level 5.2?}
  B -- yes --> N[skip, next candidate]
  B -- no --> C{h264_mf on Windows<br/>and host registered?}
  C -- yes --> D[self-test in a child copy of Cap]
  D -- ok --> E[open encoder in process]
  D -- fail / crash / 20 s timeout --> N
  C -- no --> F[self-test in process, as before] --> E

Loading

• Never feed Intel's transform a size it cannot take (cap_media_info::intel_h264_mft_accepts_frame_size): width a multiple of 16, both sides at most 4096, at most 36864 macroblocks (H.264 level 5.2). Heights such as 1080 stay allowed, because the transform pads height itself.
  • FFmpeg path (crates/enc-ffmpeg/src/video/h264.rs unsupported_frame_size): applies only to h264_mf, and only when the first hardware H.264 transform Media Foundation lists (the one FFmpeg opens, cap_enc_mediafoundation::mft::first_hardware_h264_encoder_name) is Intel's. Selection then moves on to the next candidate (NVENC, AMF, then libx264).
  • Direct Media Foundation path (crates/enc-mediafoundation/src/video/h264.rs): the same check returns NewVideoEncoderError::UnsupportedFrameSize when the chosen transform is Intel's. All four callers (win.rs screen and camera muxers, win_segmented.rs, win_segmented_camera.rs) already fall back to the FFmpeg order on a constructor error.
• A crash in the self-test no longer takes Cap down (crates/enc-ffmpeg/src/video/isolated_self_test.rs). On Windows, h264_mf's pre-flight self-test runs in a child copy of the host executable: the same encoder, options, size, pixel format and gray-frame round trip. Exit code 3 with a verdict line means the test failed; any other ending (0xc0000005, 0xc0000409, no verdict) or 20 s without an answer counts as a crash. Either way that candidate is skipped and the verdict is cached per process, like before.
  • The cap-gpui and cap main call isolated_self_test::run_child_or_register_host() first, so a child runs only the test and exits before logging, crash capture, single-instance or any window. Executables that don't register keep testing in process, exactly as before.
  • NVENC, AMF, QSV and VideoToolbox keep testing in process: no report implicates them, and a child process would add start-up time to every recording start on those GPUs.
• Debug builds only: CAP_H264_SELF_TEST_FAULT=access_violation|abort|hang|fail[:codec] makes the self-test fault the way Intel's transform does, and CAP_DEBUG_H264_ENCODER_ORDER forces an encoder order on runners without a GPU. Neither exists in release builds.

What could regress and why it doesn't

• NVIDIA/AMD encoder choice and output. Their primary encoders (h264_nvenc, h264_amf) are untouched and still test in process. Measured below: bit-identical bitstreams at 13 sizes, recording and export settings, main vs this branch.
• Recording start time. The child process only runs when h264_mf is reached, which on NVIDIA and AMD means their own encoder already failed. Median cost when it does run: about 100 ms (659 vs 556 ms, AMD, 13 sizes).
• Intel machines where h264_mf works today at a 16-aligned width. Those still get h264_mf, after the child self-test passes.
• Intel machines at an odd width with a failing QSV now encode with libx264 instead of crashing. Slower, but the export finishes.
• Callers of H264EncoderBuilder::build/build_standalone: recording (win_fragmented_m4s.rs, oop_fragmented_m4s_win.rs, win_segmented*.rs, win.rs fallbacks, instant mode), export (crates/export/src/mp4.rs, turbo) and the CLI all go through the same two selection loops, so they all get both changes. HEVC and ProRes are untouched.

Tests added

• cap-media-info: intel_h264_mft_frame_sizes, intel_h264_mft_names.
• cap-enc-ffmpeg (all OSes): isolated_self_test::tests::isolated_self_test_outcomes (pass, access violation, abort, fail, hang killed at the timeout, unknown encoder, codec-targeted faults), request_survives_the_trip_to_the_child, a_child_that_prints_no_verdict_counts_as_a_crash, only_media_foundation_self_tests_in_a_child_process, only_media_foundation_is_skipped_for_its_frame_size.
• cap-enc-ffmpeg (Windows): media_foundation_self_test_fault_falls_back_to_libx264_at_odd_sizes at 1438x812, 1366x768 and 1448x814. In CI it runs with CAP_H264_SELF_TEST_FAULT=access_violation and =abort, and asserts the child crashed, the test process survived and libx264 encoded the frames.
• cap-export (Windows): tests/windows_odd_size_export.rs exports 1446x812, 1366x770 and 1914x1078 projects through the Windows order with h264_mf first and its self-test faulting. It checks the output size (1448x812, 1368x770, 1916x1078, none 16-aligned) and frame count.
• All of the above run in .github/workflows/windows-recording-encoder.yml on windows-2022.

Measured

Release builds of this branch and of main (8393446fd) from the same Blacksmith Windows job. Boxes: AWS g4dn.2xlarge (Tesla T4, GRID 596.86, WDDM) and Azure NV12ads V710 (Radeon PRO V710), Windows Server 2025 and 2022, run in the logged-on console session.

Encoder matrix (60 frames of a moving pattern per case, recording and export settings, 1438x812, 1366x768, 1448x814, 1914x1078, 1436x1080, 1082x1922, 2542x1078, 998x562, 1920x1080, 1280x720, 2560x1440, 3840x2160, 5120x2880):

GPU	encoder chosen (main / branch)	bitstream identical	open time, sum of 26 cases (main / branch)
T4	h264_nvenc up to 4K, libx264 at 5K / same	26 of 26	16.4 s / 11.5 s
V710	h264_amf up to 4K, libx264 at 5K / same	26 of 26	14.1 s / 6.3 s

h264_mf self-test faulting like Intel's transform (debug build, order h264_mf, then the vendor encoder, then libx264, all 26 cases):

GPU	fault	in process (main's behaviour)	child process (this branch)
T4	access violation	process dies 0xc0000005	exits 0, 24 h264_nvenc + 2 libx264 (5K), output identical to the normal run
T4	fast fail	process dies 0xc0000409	exits 0, same
V710	access violation	process dies 0xc0000005	exits 0, 24 h264_amf + 2 libx264, output identical to the normal run
V710	fast fail	process dies 0xc0000409	exits 0, same

Real app (release cap-gpui, CAP_GPUI_AUTO_RECORD=studio:6 area recording, then an editor export at the default 720p):

GPU	build	recording	export
T4	main	h264_nvenc 736x510	h264_nvenc 1040x720, 166 frames
T4	branch	h264_nvenc 736x510	h264_nvenc 1040x720, 166 frames
V710	main	h264_amf 1436x810	h264_amf 1276x720, 170 frames
V710	branch	h264_amf 1436x810	h264_amf 1276x720, 172 frames (6 s timed recording, 2 more frames captured)

• Release cap-gpui.exe as the self-test child: h264_mf (AMD transform) answers ok in 459 ms, h264_amf in 212 ms and h264_nvenc in 420 ms. With no NVIDIA transform installed, h264_mf answers fail ... Function not implemented (exit 3) in 190 ms. It creates no profile, no logs and no window.
• windows_odd_size_export on both GPUs: passes with the faulting h264_mf setup and with the default order.

Not proven on Intel hardware: no cloud instance offers an Intel iGPU with its Media Foundation encoder, so the Intel transform itself has not been run. The width rule comes from Intel's documented constraint and from #4016 (16:9 works, the recorded area's size crashes), and the child-process self-test covers any size the rule misses.

Confidence Score: 5/5

The reviewed update appears safe to merge.

Summary

The PR skips frame sizes that Intel’s H.264 transform cannot accept and runs Windows h264_mf self-tests in a child process.

• Intel Media Foundation H.264 encoders skip frame sizes they cannot take.
• Windows h264_mf self-tests run in a child process.
• The encoder dependency list now includes Media Foundation.

_{Reviews (3) · Last reviewed commit: "test(enc-ffmpeg): build the Windows test..." · Reviewed by Greptile}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
cap-status	Ignored	Preview	Nov 14, 2026 7:31am UTC

[sandchest-com]

Sandchest reviewed 713fa6e: nothing exploitable found in this change.

_{Sandchest reviews every change for vulnerabilities an attacker could use · 1 credit · Review details}

[capy-ai]

Capy found 1 potential issue (1 medium).

View 1 other finding in Capy.

[richiemcilroy]

hey @greptileai, please re-review the PR

[capy-ai]

Capy found no issues.

[richiemcilroy]

hey @greptileai, please re-review the PR

[capy-ai]

Capy found no issues.

View 1 other finding in Capy.