Repository navigation
PR #5200 breaks clients that negotiate maxNumReferenceFrames=1 (macOS) #5469
Description
Is there an existing issue for this?
- I have searched the existing issues
Is your issue described in the documentation?
- I have read the documentation
Is your issue present in the latest beta/pre-release?
This issue is present in the latest pre-release
Describe the Bug
After updating to the pre-release, the client does receive twice the frames per second, but the image redraws and then reverts completely to the previous background. I had to go back to the stable version, which only delivers half of the captured frames. The pre-release says this is fixed, but since the screen does not refresh correctly, it is unusable on my system.
Expected Behavior
I expected the pre-release to deliver both things at once: the full 60 fps that #5200 restores, and a picture that stays on screen. Instead the fix trades one problem for a worse one on my client. Going back to the pre-#5200 build gives me a correct picture again, but only half the frames.
Additional Context
Client log, one line per frame (3585/minute at 60 fps):
[h264 @ 0x...] number of reference frames (1+1) exceeds max
(1; probably corrupt input), discarding one
Host telemetry on the pre-#5200 build (VCPRateControlSession):
input_fps=60.00 enc_fps=60.00 tx_fps=30.00 idr_fps=30.00
bit_rate 18190023/10988000 enc_time=2.96 ms
So on the old build the encoder emits half the captured frames, all of them
IDR, and overshoots its bitrate target by 1.66x (18.19 Mbps used against an
11.0 Mbps target). Because every frame is intra-coded, an IDLE desktop costs
the same as full motion: 23.6 Mbps measured at the client, ~10 GB/hour.
It also costs client-side diagnostics: 99.3% of the Moonlight log is
"Network dropped 1 frame" (164691 of 165775 lines), which fills the 10 MB
log cap in about 100 minutes of streaming.
Reproduced twice on the same host, checking which binary was actually running
each time with the "txt" line of lsof -p .
There is no configuration workaround: the binary exposes only vt_coder,
vt_realtime and vt_software, and the reference-frame restriction is internal
(REF_FRAMES_RESTRICT), driven by what the client negotiates.
Host Operating System
macOS
Operating System Version
macOS 26.5.2 (build 25F84). Affected: sunshine-beta 2026.726.710 (post-#5200). Working: sunshine 2026.516.143833 (pre-#5200)
Architecture
arm64/aarch64
Package
macOS - Homebrew
GPU Type
Apple Silicon
GPU Model
Apple M4
GPU Driver/Mesa Version
n/a (VideoToolbox, macOS 26.5.2 build 25F84)
Capture Method
AVCaptureScreen (macOS)
Apps
Log output
Online logs
No response
Activity
Optimiza commentedon Aug 3, 2026
Some details I left out of the original report.
The client, which is the side that triggers this:
maxNumReferenceFrames: 1Which build each block in the report came from, since they are different:
number of reference frames (1+1) exceeds maxflood is from thepost-fix(macos): drop max_ref_frames=1 for h264_videotoolbox and enable PARALLEL_ENCODING #5200 build (sunshine-beta 2026.726.710), one line per frame.
VCPRateControlSessiontelemetry withtx_fps=30 idr_fps=30is fromthe pre-fix(macos): drop max_ref_frames=1 for h264_videotoolbox and enable PARALLEL_ENCODING #5200 build (sunshine 2026.516.143833).
And the contrast that matters: on the post-#5200 build the host side is
correct. The same telemetry reads
idr_fps=0with60/60/60fps andenc_timearound 5.6 ms. So #5200 does fix the encoder. The problem is onlythat the resulting stream references more frames than this client negotiated,
and its decoder cannot follow it.
ReenigneArcher commentedon Aug 7, 2026
@Nottlespike FYI
Optimiza commentedon Aug 22, 2026
Looking at the diff for #5200, it drops {"max_ref_frames"s, 1} from the h264_videotoolbox common options unconditionally. Making that conditional on what the client negotiated would cover both cases: when config.numRefFrames > 1, omit max_ref_frames, which is the current #5200 behaviour and gives the full frame rate with no forced IDR; when config.numRefFrames == 1, keep max_ref_frames=1, which is the pre-#5200 behaviour and the only stream this client's decoder can follow. The value is already available at that point, since the generic path uses config.numRefFrames for ctx->refs.
One data point that may be relevant to the unresolved review comment on that diff, which argues the all-IDR behaviour should persist through ctx->refs = config.numRefFrames even after the AVOption is gone: on my host it does not. My client negotiates maxNumReferenceFrames=1, and the post-#5200 build reports idr_fps=0 with 60/60/60 fps and enc_time around 5.6 ms. So the encoder side is genuinely fixed here, and what remains is only that the stream references more frames than the client asked for.
Happy to test a build. Intel Broadwell iMac on H.264, easy to reproduce.
martona commentedon Sep 23, 2026
Possible root cause found, and I think this issue, the freeze in #5511's test report, and probably #5697 are all the same thing.
VT's low-latency H.264 sessions use long-term reference pictures by default. Parsing the elementary stream from a current master host (AVFoundation capture, ffmpeg h264_videotoolbox): the IDR is marked as a long-term reference and the P-frames that follow carry reference-marking operations; the SPS advertises 5-15 reference frames. An x264 stream from the same host has none of that and a single short-term reference. The "(1+1) exceeds max (1)" log line is a decoder seeing that second reference.
#5200 exposed it because with max_ref_frames=1, VT produced all-IDR output, so there were never any P-frames to carry long-term references. Dropping it fixed the frame rate and let the references through. Making max_ref_frames conditional on the negotiated count would bring back the all-IDR stream for your client and wouldn't help clients that negotiate more than one reference but whose hardware decoders mishandle long-term references anyway; my NVIDIA-based Moonlight clients on Windows show the same freeze/garble with hardware decode and are fine with software decode. The stream itself is valid: the dumped file plays in mpv and VLC and decodes cleanly in libavcodec, both software and with the VideoToolbox hwaccel.
I tested VT session settings in a standalone harness that mirrors Sunshine's session config:
EnableLTR=false is a public key (macOS 12+), low-latency rate control stays on, and the stream comes out with plain short-term references. I verified it end to end on a native VideoToolbox encoder I have on a separate branch: same host, same NVIDIA clients, H.264 1080p120 with hardware decode now renders correctly.
For ffmpeg, there is no option to set it. Since build-deps already carries ffmpeg patches, the smallest fix for master is a patch to videotoolboxenc.c that sets EnableLTR=false on H.264 sessions. I can put that patch together if there's interest.
@Optimiza, when you have the Intel iMac on Thursday: I expect this to satisfy a 1-reference client too, since the extra reference is exactly what disappears. https://github.com/martona/Sunshine/tree/macos-latency is my branch that combines SCKit capture and the native VT encoder, with this fix applied.
Optimiza commentedon Sep 24, 2026
Results from a Windows NVIDIA client that reproduces the freeze, run as an A/B with controls. They support the long-term reference explanation.
Setup. Host: Mac mini M4, macOS 26.7, two Sunshine instances on the same machine: the Homebrew release (2026.914, commit
63d35f7, ffmpegh264_videotoolbox) and yourmacos-latencybranch built at04483743. Client: Windows desktop with an RTX 4090, Moonlight, codec forced to H.264. Every encoder below is quoted from the host's own log for that session, not inferred from the configuration.h264_videotoolboxmacos-latency,encoder = videotoolbox_nativeh264_vt [videotoolbox_native]macos-latency,encoder = videotoolboxh264_videotoolboxh264_videotoolboxh264_videotoolboxmacos-latency, nativeh264_vt [videotoolbox_native]What each comparison isolates:
EnableLTR = falseis.Limits, so the table is read for what it is: freezes showed within the first minute of each failing run, and the clean runs were watched with video and window movement for one and a half to three minutes each, not for long sessions; the verdicts are visual; and one Windows client with one GPU was tested. The release does not log the capture resolution; its runs are tied to the client settings by the requested bitrate, identical in runs 1 and 4 and higher for 1b, and by the frame rate the host derives from the request (below).
Host log lines for each run
Run 1, release, hardware decode:
Run 2,
macos-latency, native encoder:Run 3,
macos-latency, ffmpeg encoder (copied during the run; the bench's log is rewritten at each start and this session's file is gone):Run 4, release, software decode (the IDR line then repeats continuously until the client leaves):
Run 1b, release, hardware decode, 120 fps requested:
Run 2b,
macos-latency, native encoder, 120 fps:Versions from the same logs:
Sunshine version: 2026.914.233613 commit: 63d35f702ee9e362e43263742981836ec0710384(release) andSunshine version: 0.0.0-04483743(bench).One host-side detail that looked like evidence and is not:
Encoder did not produce IDR frame when requested!does not track the freeze here. It was absent in the frozen run 1 and logged continuously in the clean run 4, so it follows the client asking for recovery frames, which it does constantly on software decoding.Our understanding, for you to correct: VT's low-latency H.264 sessions emit long-term reference pictures that some hardware decoders mishandle; #5200 exposed it by dropping
max_ref_frames=1, which had kept VT on all-IDR streams; the fix for master iskVTCompressionPropertyKey_EnableLTR = false, which ffmpeg does not expose, so it needs a patch tovideotoolboxenc.cin build-deps; and it is out of scope for #5511.Still to come from our side: the Intel iMac (HD 6000, one negotiated reference frame) that first showed the freeze in the #5511 report, the client closest to #5469. The same pairs will be run there, the release on software and hardware decoding and
macos-latencyon hardware decoding, and reported here.Optimiza commentedon Sep 25, 2026
Results from the Intel iMac, the client this issue was filed for. Your expectation holds: the 1-reference client is clean on the native encoder with
EnableLTR = false, and freezes on everything that still emits long-term references.Setup. Host: the same Mac mini M4 and the same two builds as the Alienware results above, the Homebrew release (2026.914, commit
63d35f7) andmacos-latencyat04483743, still the branch head. Client: iMac Late 2015, Intel HD 6000, macOS 14.8.1, Moonlight over WAN, codec forced to H.264, 1080p60, default bitrate. Every encoder below is theCreating encoderline afterCLIENT CONNECTEDin the host's log for that session.h264_videotoolboxh264_videotoolboxmacos-latency,encoder = videotoolboxh264_videotoolboxmacos-latency,encoder = videotoolboxh264_videotoolboxmacos-latency,encoder = videotoolbox_nativeh264_vt [videotoolbox_native]Limits: the freezes were immediate, so the contrast is sharp, but the clean runs were short (about 1.5 minutes for run 2, 35 seconds for 1-sw, almost 4 minutes for 0b); the verdicts are visual; and I did not collect Moonlight's own log this time, so I cannot show the
(1+1) exceeds maxline going away, only the freeze. Runs 1 and 2 used Moonlight's Automatic decoder, the same in both; run 1 freezing is what says it picked hardware.Host log lines for each run
Run 0, release, hardware decode:
Run 0b, release, software decode:
Run 1-sw,
macos-latency, ffmpeg encoder, software decode:Run 1,
macos-latency, ffmpeg encoder, hardware decode:Run 2,
macos-latency, native encoder, hardware decode:So both clients that reproduce, an Intel iGPU negotiating one reference frame and an RTX 4090, are fixed by the same change. From my side that supports the
videotoolboxenc.cpatch for master you offered.A note on how this is written: I run these tests automated with an AI assistant, and I use it to write in English and to carry the log lines over without copying them by hand. The runs, the verdicts and the decision to post are mine.