Repository navigation
Conversation
`submit` passed the full event list to kevent, so besides per-change errors, kevent also returned events that were pending for completions already registered, and `submit` ran them through the completion queue. For a read that rearms, that path ends with the completion `dead` while it is still in the kqueue. If its fd is drained right then, the next `submit` takes its event again, sees `dead` and pushes it onto the submission queue. The normal event path then sets it `active`, and the tick after that logs "invalid state in submission queue state=.active"; from there the completion can be stopped (cancelled) while its fd is still live. Found with a PTY read on a loop that also queues a socket write for each read (a terminal server with a client that acks instantly): the error shows up within a few MiB of output. The fix: every change in `submit` carries EV_RECEIPT and the event list has exactly one slot per change, so kevent can only return receipts. A failed change still reaches its callback with the errno; readiness is left to `tick`. This is also what the old NOTE in `submit` described (no completions processed there), which the code no longer did. Adds a tick-by-tick test that reproduces the invalid state on main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Loop.submitpasses the full event list tokevent, so besides per-change errors, kevent also returns events that are pending for completions that are already registered, andsubmitperforms them and pushes them onto the completion queue.For a read that rearms, that path ends with the completion
deadwhile it is still in the kqueue (tick's completion-queue branch setsdeadand, sincec_activewas true, doesn't resubmit). If the fd happens to be drained at that point, the nextsubmittakes its event again, seesdead, and its.rearmpushes it onto the submission queue. The normal event path then sets itactive, and the following tick logs:From there the completion can also be stopped (its callback gets a cancellation) while the fd is still live.
I hit this with a PTY read on a loop that also queues a socket write for every read (a terminal server whose client acks instantly); it shows up within a few MiB of output. The old
NOTEinsubmitsays completions aren't processed there (zero-length event list), but the code no longer does that.Fix
Every change in
submitcarriesEV_RECEIPT, and the event list has exactly one slot per change, so kevent can only return receipts (EV_ERRORwithdata = 0on success). A failed change still reaches its callback with the errno; readiness is left totick. Unlike a zero-length event list, this keeps submission errors (e.g.EBADF) from being lost.Test
kqueue: submit must not run a registered read's eventdrives the loop tick by tick with.no_wait: a rearming read on a pipe, one new write (to another pipe) per tick sosubmitalways calls kevent, and 1 then 2 bytes fed before the first two ticks. Onmainit fails with the error log above; with the fix it passes. All 60 tests pass on macOS (aarch64, Zig 0.17.0). I haven't run it on FreeBSD.