Skip to content

kqueue: don't take pending events in submit - #262

Open
cw228 wants to merge 1 commit into
mitchellh:mainfrom
cw228:kqueue-submit-no-events
Open

cw228 wants to merge 1 commit into
mitchellh:mainfrom
cw228:kqueue-submit-no-events

Conversation

@cw228

@cw228 cw228 commented Nov 17, 2026

Copy link
Copy Markdown

Problem

Loop.submit passes the full event list to kevent, so besides per-change errors, kevent also returns events that are pending for completions that are already registered, and submit performs them and pushes them onto the completion queue.

For a read that rearms, that path ends with the completion dead while it is still in the kqueue (tick's completion-queue branch sets dead and, since c_active was true, doesn't resubmit). If the fd happens to be drained at that point, the next submit takes its event again, sees dead, and its .rearm pushes it onto the submission queue. The normal event path then sets it active, and the following tick logs:

error(libxev_kqueue): invalid state in submission queue state=.active

From there the completion can also be stopped (its callback gets a cancellation) while the fd is still live.

I hit this with a PTY read on a loop that also queues a socket write for every read (a terminal server whose client acks instantly); it shows up within a few MiB of output. The old NOTE in submit says completions aren't processed there (zero-length event list), but the code no longer does that.

Fix

Every change in submit carries EV_RECEIPT, and the event list has exactly one slot per change, so kevent can only return receipts (EV_ERROR with data = 0 on success). A failed change still reaches its callback with the errno; readiness is left to tick. Unlike a zero-length event list, this keeps submission errors (e.g. EBADF) from being lost.

Test

kqueue: submit must not run a registered read's event drives the loop tick by tick with .no_wait: a rearming read on a pipe, one new write (to another pipe) per tick so submit always calls kevent, and 1 then 2 bytes fed before the first two ticks. On main it fails with the error log above; with the fix it passes. All 60 tests pass on macOS (aarch64, Zig 0.17.0). I haven't run it on FreeBSD.

`submit` passed the full event list to kevent, so besides per-change
errors, kevent also returned events that were pending for completions
already registered, and `submit` ran them through the completion queue.

For a read that rearms, that path ends with the completion `dead` while
it is still in the kqueue. If its fd is drained right then, the next
`submit` takes its event again, sees `dead` and pushes it onto the
submission queue. The normal event path then sets it `active`, and the
tick after that logs "invalid state in submission queue state=.active";
from there the completion can be stopped (cancelled) while its fd is
still live.

Found with a PTY read on a loop that also queues a socket write for each
read (a terminal server with a client that acks instantly): the error
shows up within a few MiB of output.

The fix: every change in `submit` carries EV_RECEIPT and the event list
has exactly one slot per change, so kevent can only return receipts.
A failed change still reaches its callback with the errno; readiness is
left to `tick`. This is also what the old NOTE in `submit` described
(no completions processed there), which the code no longer did.

Adds a tick-by-tick test that reproduces the invalid state on main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant