Skip to content

[CustomDevice][Performance] Skip per-Run default-stream Wait when a Run stays on the default stream - #79960

Open
Kafka-Hatsune wants to merge 7 commits into
PaddlePaddle:developfrom
Kafka-Hatsune:opt/customdevice-wait
Open

Kafka-Hatsune wants to merge 7 commits into
PaddlePaddle:developfrom
Kafka-Hatsune:opt/customdevice-wait

Conversation

@Kafka-Hatsune

@Kafka-Hatsune Kafka-Hatsune commented Nov 18, 2026 •

Copy link
Copy Markdown

PR Category

Custom Device

PR Types

Performance

Description

自定义设备上 PirInterpreter::{TraceRunImpl,MultiThreadRunImpl} 在每次 Run 结束后都会无条件执行一次阻塞的默认计算流同步(DeviceContextPool::Get(place_)->Wait(),即 synchronize_stream,非整设备同步);GPU 路径没有这一步。由于解释器与 eager 共用默认计算流,流内 FIFO 已能保证交回 eager 时的顺序,这次阻塞对主机是多余的,且每个 InterpreterCore::Run() 都执行一次,使主机侧同步次数随 eager↔静态图 的边界数增长。

本 PR 改动(updated in commit 9a982b9):在 Run 开始前对侧流做异步事件排序,取代每次 Run 末尾阻塞主机的默认流同步。

  • 默认流-only 的 Run:不做任何同步(流内 FIFO 保证交回 eager 的顺序)。
  • 含侧流(D2H/H2D/exec)的 Run:在默认流记录事件、各侧流异步等待它,使侧流 op 排在所有先前默认流工作(含上一个 Run 的输出)之后,不阻塞主机,并补上跨 Run 的 default→side 依赖(如 1F1B BACKWARD→SEND_BACKWARD)。

已知限制(后续在其他后端完善):

  • 依赖可选接口 stream_wait_event;事件为 no-op 的后端(Intel GPU/MPS)侧流不会真正等待。
  • .cpu() 的主机读属 host-read 非跨流,不在本机制覆盖内;通用 .cpu() 为阻塞拷贝,已安全。
  • side→default 的跨 Run 反向依赖沿用既有行为(原阻塞 Wait 同样未覆盖)。

是否引起精度变化

否

…reter Run

The PIR interpreter issues kernels on the default compute stream, which is
the same stream eager uses after run_program returns. Stream FIFO therefore
already orders the handoff, so the unconditional per-Run DeviceContext::Wait()
on custom devices is unnecessary and serializes host against device, scaling
the sync count with the number of eager<->static boundaries.

Replace it with: skip when the Run only used the default stream; otherwise
make the default stream wait on side-stream (D2H/H2D/exec) events
asynchronously. FLAGS_custom_device_force_run_sync restores the old behavior.

Verified on a custom device: loss bit-identical to the blocking path; sync
count no longer grows with compile scope.
…lt-stream Runs

Remove FLAGS_custom_device_force_run_sync and the async-event cross-stream path.
stream_wait_event is an optional custom-device interface and silently no-ops
when absent, so the async path is unsafe without a fallback.

Final behavior is a minimal, universally-correct subset of the original: skip
the per-Run blocking default-stream Wait only when the Run used solely the
default (eager-shared) stream, where stream FIFO already orders the eager
handoff; keep the original blocking Wait when any side stream (D2H/H2D/exec)
was used.

Verified on a custom device: compiled (interpreter) path matches eager; full-
graph / per-layer CINN training keep their speedup (default-stream-only Runs).
…known issues

Two correctness tests on a custom device:
- compiled (to_static) training matches eager (default-stream handoff is
  correctly ordered by stream FIFO after skipping the per-Run sync);
- .cpu() device->host readback read immediately with no explicit sync matches a
  synchronized reference (generic .cpu() uses a blocking copy).

Known limitations (to be addressed on other backends later):
- Cross-Run named/side-stream consumer: a later Run's first op reading a prior
  default-stream Run's output on a user execution stream has no cross-Run event
  ordering on custom devices (jit-input event is is_gpu_place only) -> fast path
  may race. Not covered by these tests.
- Backends that override memcpy with a non-blocking D2H can let the host read a
  stale buffer; the generic .cpu() path is blocking and is covered.
- The skip is currently enabled for all custom devices; devices with no-op
  stream events (e.g. Intel GPU / MPS) or async .cpu() (e.g. MLU memcpy_d2h
  >64KB) need a per-device opt-in (capability negotiation) in a follow-up.
@Kafka-Hatsune Kafka-Hatsune changed the title Opt/customdevice wait [CustomDevice][Performance] Skip per-Run default-stream Wait when a Run stays on the default stream Nov 18, 2026
@Kafka-Hatsune
Kafka-Hatsune marked this pull request as ready for review November 18, 2026 11:04
@Paddle-Review-Bot

Paddle-Review-Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Paddle-Bot Review Board (review完成)

序号 位置 优先级 规则来源 状态
1 Run 边界跳过同步的判定 P1 默认规则 ✅
2 Run 边界测试覆盖 P2 默认规则 ✅

可维护性建议(总分 88.3 / 100,只给建议,不影响评审结论)

序号 位置 维度 状态
1 头文件声明缺少条件编译 约定偏离 ✅
2 同步函数命名与注释 命名质量 ✅
3 每次 Run 重复扫描指令 中层优雅度 ✅
4 两个解释器 Run 边界语义不一致 高层优雅度 ✅
5 侧流等待中多余的常量转换 类型安全 ✅
6 侧流上下文缓存的失效条件 契约一致性 ✅
7 插件构建失败时测试静默跳过 错误处理一致性 ✅
各维度得分(仅评本 PR 改动,点击展开)
85 中层优雅度
3
85 AI 生成债务
85 未完成迁移
85 约定偏离
1
86 测试策略
86 高层优雅度
4
87 底层优雅度
88 抽象适配度
88 API 表面一致性
88 跨模块架构
88 设计一致性
88 错误处理一致性
7
89 初始化耦合
89 命名质量
2
90 契约一致性
6
90 逻辑清晰度
90 包组织结构
95 依赖健康度
95 类型安全
5
— 鉴权一致性

Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully.

@paddle-bot paddle-bot Bot added the contributor External developers label Nov 18, 2026
Address review: the previous end-of-Run skip only inspected the current Run,
so a later Run reading a prior default-stream output on a side stream (e.g.
1F1B BACKWARD -> SEND_BACKWARD on send_recv_stream) had no cross-Run ordering
on custom devices.

Instead of blocking the host at the end of a Run, order side streams at the
start: a default-stream-only Run does nothing; a Run that uses side streams
records an event on the default stream and makes each side stream wait on it
asynchronously, ordering side-stream ops after all prior default-stream work
(including a previous Run's outputs) without blocking the host.

Also from review:
- header declaration/members guarded by PADDLE_WITH_CUSTOM_DEVICE;
- rename to OrderCustomDeviceSideStreamsBeforeRun with the contract in the
  header comment;
- scan the instruction list once (cached side-stream contexts);
- document the intentional difference vs ProgramInterpreter (legacy IR keeps
  the blocking Wait, not yet migrated).

Verified on a custom device: correctness tests pass; CINN training keeps its
speedup (full ~17ms, per-layer ~31ms); loss unchanged.
- test: build the custom_cpu plugin in setUp (PLUGIN_URL/PLUGIN_TAG) like the
  sibling custom_runtime tests so it actually exercises the custom-device Run
  boundary on CI instead of skipping; refresh the stale header comment to the
  event-based ordering.
- drop the redundant const_cast in the side-stream event wait.
- invalidate the cached side-stream contexts in BuildInstruction() when the
  instruction list is rebuilt; document the contract in the header.
Address review:
- check the custom_cpu build return code (subprocess.run(..., check=True)) so a
  clone/compile failure is surfaced instead of leaving CUSTOM_DEVICE_ROOT at a
  missing dir and silently skipping (false CI pass);
- build the plugin once in setUpClass instead of per-test in setUp.
The existing cases only use the default stream, so the Record/Wait branch of
OrderCustomDeviceSideStreamsBeforeRun (custom_side_ctxs_ non-empty) was never
exercised. Add a static-graph case that puts a matmul on a named execution
stream (set_execution_stream), which routes that op to a side-stream context,
and assert the result is identical to the default-stream baseline.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor External developers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants