Repository navigation
[CustomDevice][Performance] Skip per-Run default-stream Wait when a Run stays on the default stream - #79960
Open
Kafka-Hatsune wants to merge 7 commits into
Open
[CustomDevice][Performance] Skip per-Run default-stream Wait when a Run stays on the default stream#79960Kafka-Hatsune wants to merge 7 commits into
Kafka-Hatsune wants to merge 7 commits into
Conversation
…reter Run The PIR interpreter issues kernels on the default compute stream, which is the same stream eager uses after run_program returns. Stream FIFO therefore already orders the handoff, so the unconditional per-Run DeviceContext::Wait() on custom devices is unnecessary and serializes host against device, scaling the sync count with the number of eager<->static boundaries. Replace it with: skip when the Run only used the default stream; otherwise make the default stream wait on side-stream (D2H/H2D/exec) events asynchronously. FLAGS_custom_device_force_run_sync restores the old behavior. Verified on a custom device: loss bit-identical to the blocking path; sync count no longer grows with compile scope.
…lt-stream Runs Remove FLAGS_custom_device_force_run_sync and the async-event cross-stream path. stream_wait_event is an optional custom-device interface and silently no-ops when absent, so the async path is unsafe without a fallback. Final behavior is a minimal, universally-correct subset of the original: skip the per-Run blocking default-stream Wait only when the Run used solely the default (eager-shared) stream, where stream FIFO already orders the eager handoff; keep the original blocking Wait when any side stream (D2H/H2D/exec) was used. Verified on a custom device: compiled (interpreter) path matches eager; full- graph / per-layer CINN training keep their speedup (default-stream-only Runs).
…known issues Two correctness tests on a custom device: - compiled (to_static) training matches eager (default-stream handoff is correctly ordered by stream FIFO after skipping the per-Run sync); - .cpu() device->host readback read immediately with no explicit sync matches a synchronized reference (generic .cpu() uses a blocking copy). Known limitations (to be addressed on other backends later): - Cross-Run named/side-stream consumer: a later Run's first op reading a prior default-stream Run's output on a user execution stream has no cross-Run event ordering on custom devices (jit-input event is is_gpu_place only) -> fast path may race. Not covered by these tests. - Backends that override memcpy with a non-blocking D2H can let the host read a stale buffer; the generic .cpu() path is blocking and is covered. - The skip is currently enabled for all custom devices; devices with no-op stream events (e.g. Intel GPU / MPS) or async .cpu() (e.g. MLU memcpy_d2h >64KB) need a per-device opt-in (capability negotiation) in a follow-up.
Kafka-Hatsune
marked this pull request as ready for review
November 18, 2026 11:04
Contributor
Paddle-Bot Review Board (review完成)
可维护性建议(总分 88.3 / 100,只给建议,不影响评审结论)
各维度得分(仅评本 PR 改动,点击展开)
Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully. |
Paddle-Review-Bot
suggested changes
Nov 18, 2026
Address review: the previous end-of-Run skip only inspected the current Run, so a later Run reading a prior default-stream output on a side stream (e.g. 1F1B BACKWARD -> SEND_BACKWARD on send_recv_stream) had no cross-Run ordering on custom devices. Instead of blocking the host at the end of a Run, order side streams at the start: a default-stream-only Run does nothing; a Run that uses side streams records an event on the default stream and makes each side stream wait on it asynchronously, ordering side-stream ops after all prior default-stream work (including a previous Run's outputs) without blocking the host. Also from review: - header declaration/members guarded by PADDLE_WITH_CUSTOM_DEVICE; - rename to OrderCustomDeviceSideStreamsBeforeRun with the contract in the header comment; - scan the instruction list once (cached side-stream contexts); - document the intentional difference vs ProgramInterpreter (legacy IR keeps the blocking Wait, not yet migrated). Verified on a custom device: correctness tests pass; CINN training keeps its speedup (full ~17ms, per-layer ~31ms); loss unchanged.
- test: build the custom_cpu plugin in setUp (PLUGIN_URL/PLUGIN_TAG) like the sibling custom_runtime tests so it actually exercises the custom-device Run boundary on CI instead of skipping; refresh the stale header comment to the event-based ordering. - drop the redundant const_cast in the side-stream event wait. - invalidate the cached side-stream contexts in BuildInstruction() when the instruction list is rebuilt; document the contract in the header.
Address review: - check the custom_cpu build return code (subprocess.run(..., check=True)) so a clone/compile failure is surfaced instead of leaving CUSTOM_DEVICE_ROOT at a missing dir and silently skipping (false CI pass); - build the plugin once in setUpClass instead of per-test in setUp.
The existing cases only use the default stream, so the Record/Wait branch of OrderCustomDeviceSideStreamsBeforeRun (custom_side_ctxs_ non-empty) was never exercised. Add a static-graph case that puts a matmul on a named execution stream (set_execution_stream), which routes that op to a side-stream context, and assert the result is identical to the default-stream baseline.
Paddle-Review-Bot
approved these changes
Nov 19, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Category
Custom Device
PR Types
Performance
Description
自定义设备上
PirInterpreter::{TraceRunImpl,MultiThreadRunImpl}在每次 Run 结束后都会无条件执行一次阻塞的默认计算流同步(DeviceContextPool::Get(place_)->Wait(),即synchronize_stream,非整设备同步);GPU 路径没有这一步。由于解释器与 eager 共用默认计算流,流内 FIFO 已能保证交回 eager 时的顺序,这次阻塞对主机是多余的,且每个InterpreterCore::Run()都执行一次,使主机侧同步次数随 eager↔静态图 的边界数增长。本 PR 改动(updated in commit 9a982b9):在 Run 开始前对侧流做异步事件排序,取代每次 Run 末尾阻塞主机的默认流同步。
已知限制(后续在其他后端完善):
是否引起精度变化
否