Replace pydub with a PyAV sound mixer and place audio in movie time by behackl · Pull Request #5059 · ManimCommunity/manim

behackl · 2026-10-10T20:15:00Z

Overview: What does this pull request change?

Stacked on #5006.

Mixes sounds with PyAV instead of pydub, and places sounds and subcaptions in movie time.

Sounds are placed at scene time, independently of the plays. All sounds are mixed into one track, and the movie keeps the parts of the track that fall into the spans of scene time it shows. Plays left out of the output cut sound like the video. Parts before the scene starts or after it ends are cut with a warning. This replaces the rule from Refactor: run play calls through one loop with listeners #5004 that dropped the sounds of excluded plays.
Subcaptions and section videos are placed in movie time the same way; section videos carry their part of the audio.
The audio track is exactly as long as the video. No audio path needs pydub, audioop-lts, or an ffmpeg binary.
Timelines record each sound's duration, read from the file headers.

Breaking changes

SceneFileWriter: init_audio, create_audio_segment, add_audio_segment, audio_segment, includes_sound, and the module function convert_audio are removed. add_sound(sound_file, time, gain=None, gain_to_background=None) requires time. combine_files(..., audio=...) replaces includes_sound. New: add_movie_interval(start, end) and Section.movie_start.
Scene.add_sound() and Manager.add_sound() check the sound file when called, in every run: a missing file raises OSError, and a file that cannot be read as audio raises ValueError. Before, Manager.evaluate() skipped the check unless it captured a timeline, so a scene could pass evaluation and fail when rendered (6a6fcb6).
Scene.add_sound() and Manager.add_sound() take gain_to_background as a parameter instead of **kwargs. Timeline sound entries have gain_to_background instead of options.

Links to added or changed documentation pages

Tests

Mixer unit tests; audio starts on time and matches the video length (mp4, mov, webm); partial renders, skipped sections and section videos; warnings for sounds and captions after the scene end; reused segments give identical audio; sound durations in timelines.

GniLudio

The pydub segment in the mypy.ini should be removed as well.

behackl · 2026-11-02T02:41:55Z

removed the pydub segment from mypy.ini ✅

behackl

... and the final few comments now. 👍

manim/scene/audio_mixer.py probes and decodes sound files with PyAV and mixes them into one track per scene, placed at scene time. The mix is cut to the parts of the scene that the movie shows, so movie time runs through those parts in turn. Nothing uses the mixer yet.

Sounds no longer belong to plays. Every add_sound() call records a sound at its scene time, and the movie's audio is the mix of all sounds, cut to the spans of scene time that the movie shows. The encoder reports those spans with the new SceneFileWriter.add_movie_interval(). Plays left out of the output, by -n or a skipped section, therefore cut sound like the video instead of dropping it; parts before the scene starts or after the movie ends are cut with a warning. Subcaptions and section videos are placed in movie time the same way, and the audio track is exactly as long as the video. Audio is decoded, mixed and encoded with PyAV in the same pass that copies the video packets. pydub and audioop-lts are no longer dependencies, and no audio path needs an ffmpeg binary. add_sound() takes gain_to_background as a parameter instead of **kwargs, and timelines record it in place of "options".

When a timeline is captured, each sound's duration is read from its file's headers, without decoding it. A missing or unreadable sound file therefore fails the capture, as it fails a render. The reference scene's test writes a short click sound for it.

The compatibility path behind CairoRenderer.add_frame() and freeze_current_frame() marked the frames' scene time as shown in the movie. Between plays no segment is open and the frames are not written, so the audio ran longer than the video; during a play the span overlapped the play's own, which shifted later sounds and subcaptions. Only plays now mark movie time, which every segment of the movie comes from.

Probing a sound file reads only its headers, so a file with damaged audio data passed add_sound() and failed with a raw av error when the movie was assembled. Decoding now raises the same ValueError as the probe, naming the file.

The writer's sound mix was made once, on first use, and kept even if a sound or a span of movie time was added afterwards. That cannot happen in one render, but a writer that combines and then goes on playing would have used the old mix.

With a timeline captured, every add_sound() opened the file twice: the timeline records its duration, and the file writer checks it. Scenes also often play the same file many times. Probe results are now kept per file, path, size and modification time.

The checks for sounds and subcaptions that the movie cuts were split over three places that compared against different ends: the Manager warned at the call about starts before the scene, the encoder kept its own list of calls to warn about those after the scene, and the file writer warned about audio past the last span shown in the movie. That last warning did not name the sound, needed the decoded mix, and advised adding a wait() also when a skipped final section had cut the sound. The file writer now checks every sound and subcaption once, before the movie is assembled, against the scene's end, which the encoder reports. Each warning names the cut item. A cut by a skipped final section does not warn, like any other cut by plays left out of the output. Evaluation writes no movie and no longer warns about cuts.

The file writer (for sections and subcaptions) and the sound mixer (for its intervals' sample boundaries) each computed movie time from the shown scene intervals, one with a running sum and one per call. Both now use audio_mixer._movie_time(). The sample boundaries are unchanged (checked on 20000 random interval layouts): an interval still ends at the sample where the next one starts. Suggested in review.

The mixer decoded every sound file to float32 stereo and kept it in memory while the movie and each section video were assembled: a 10-minute music bed took about 230 MB, twice what pydub's 16-bit samples took. Sounds are now decoded to int16 and converted to float block by block while mixing. For 16-bit sources the mix is bit-identical; sources with more precision are rounded to 16 bits, as pydub did. Raised in review.

A sound call made two nearly identical records: the run loop's _Sound (the file as the scene named it) and the mixer's _PlacedSound (the file's resolved path), and the path was resolved twice, by the timeline record and by the file writer. There is now one frozen _Sound (audio_mixer.py) with the file as named, its resolved path, start and gains, made once by _Sound.at() in Manager.add_sound and passed through the loop, the timeline record, the file writer and the mixer. _Sound.at() checks the file, so a missing or unreadable sound file now fails at the add_sound() call in every run. An evaluation used to skip the check unless it captured a timeline, so the same scene could pass evaluation and fail when rendered. Raised in review.

The sound test helper wrote its files at 44_100 Hz and scaled levels by 32767, and two sound tests wrote their own WAV files with 44100 again. tests/helpers/audio.py now names the rate, SOURCE_RATE (the CD rate, unlike the mixer's 48 kHz, so that reading the files resamples, as for most real sound files), and derives the full scale from int16; write_wav can write stereo files, and both tests use it. Raised in review.

GniLudio · 2026-11-11T14:03:11Z

Did a run of the lissajous benchmark (with verbosity set to warning) to make sure the performance isn't hugely effected by this stack.

main
===========================================================================
  RadiusOne                          27194ms  (runs: 27194ms)
  RadiusHalf                        423765ms  (runs: 423765ms)
  RadiusThreeFourths                 92016ms  (runs: 92016ms)
---------------------------------------------------------------------------
  TOTAL                             542976ms

this
===========================================================================
  RadiusOne                          26381ms  (runs: 26381ms)
  RadiusHalf                        421851ms  (runs: 421851ms)
  RadiusThreeFourths                 92045ms  (runs: 92045ms)
---------------------------------------------------------------------------
  TOTAL                             540277ms

GniLudio

LGTM

behackl added this pull request to stack #5003 October 10, 2026 20:41

behackl force-pushed the refactor/audio-mixer branch from 189fb4c to 3d6405c Compare October 26, 2026 17:46

behackl mentioned this pull request Oct 26, 2026

Refactor: run play calls through one loop with listeners #5004

Open

behackl removed this pull request from stack #5003 October 26, 2026 18:00

behackl changed the base branch from refactor/manager-targeted-frame to refactor/timeline-export October 26, 2026 18:01

behackl added this pull request to stack #5104 October 26, 2026 18:01

behackl force-pushed the refactor/audio-mixer branch 3 times, most recently from ff731a1 to fd963fd Compare October 26, 2026 21:42

behackl force-pushed the refactor/audio-mixer branch from fd963fd to 09b51a0 Compare October 29, 2026 13:06

hoangchunghien mentioned this pull request Oct 31, 2026

Audio broken after silence in rerendered video #4751

Open

GniLudio requested changes Nov 1, 2026

View reviewed changes

behackl force-pushed the refactor/audio-mixer branch from 09b51a0 to deabc7b Compare November 2, 2026 02:06

behackl force-pushed the refactor/audio-mixer branch from deabc7b to c095df7 Compare November 2, 2026 02:47

GniLudio requested changes Nov 2, 2026

View reviewed changes

behackl force-pushed the refactor/audio-mixer branch 2 times, most recently from cc2a8f1 to 7ad802c Compare November 4, 2026 15:43

GniLudio requested changes Nov 6, 2026

View reviewed changes

Comment thread manim/scene/audio_mixer.py Resolved

Comment thread manim/scene/audio_mixer.py Resolved

Comment thread manim/scene/audio_mixer.py Resolved

Comment thread manim/scene/scene_file_writer.py Resolved

GniLudio requested changes Nov 8, 2026

View reviewed changes

Comment thread manim/scene/audio_mixer.py Resolved

Comment thread tests/helpers/audio.py Resolved

Comment thread tests/helpers/audio.py Resolved

Comment thread tests/helpers/audio.py Resolved

behackl force-pushed the refactor/audio-mixer branch from 74636a1 to 7fdf4ce Compare November 9, 2026 14:37

behackl commented Nov 10, 2026

View reviewed changes

behackl and others added 13 commits November 11, 2026 12:36

Drop the mypy override for pydub, which Manim no longer imports

687cc39

Raise a ValueError for audio data that cannot be decoded

6329820

Probing a sound file reads only its headers, so a file with damaged audio data passed add_sound() and failed with a raw av error when the movie was assembled. Decoding now raises the same ValueError as the probe, naming the file.

behackl force-pushed the refactor/audio-mixer branch from 7fdf4ce to 8c9ee43 Compare November 11, 2026 12:08

GniLudio approved these changes Nov 11, 2026

View reviewed changes

Conversation

behackl commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview: What does this pull request change?

Breaking changes

Links to added or changed documentation pages

Tests

Uh oh!

GniLudio left a comment

Choose a reason for hiding this comment

Uh oh!

behackl commented Nov 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

behackl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GniLudio commented Nov 11, 2026

Uh oh!

GniLudio left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

behackl commented Oct 10, 2026 •

edited

Loading