Repository navigation
Test that a receiver recovered mid-stream gets every message once, in order - #636
Merged
Merged
Conversation
… order A workflow receives 300 messages in order while a client sends them, without idempotency keys, and it is recovered onto a new executor every 250 ms through recoverPendingWorkflows, the call Conductor's recovery makes. In one variant each executor is closed before the next recovers it, as a redeploy does, so some messages arrive while nothing runs the receiver; in the other it is left running through the hand-off and is refused at its next write. Either way the receiver gets every message exactly once. A second test shows the one way a message does arrive twice: a send retried after the first attempt went through, without an idempotency key. With the same key on both attempts, the message is delivered once.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The recovery test can pass without verifying repeated mid-stream receiver resumptions.
1 open issue
What changed in this PR
Adds notification tests for receiver recovery and send idempotency without changing the SDK.
Changes:
- Exercises ordered delivery during repeated executor replacement using close and abandon scenarios.
- Compares retried sends with and without idempotency keys.
| File | Description |
|---|---|
| transact/src/test/java/dev/dbos/transact/notifications/RecvAcrossRecoveryTest.java | Adds recovery and duplicate-send tests. |
Review effort: Balanced
A recovered workflow waits for a queue poll, about a second, before anything runs it again, so with a hand-off every 250 ms most new executors were replaced before they claimed the receiver, which could sit ENQUEUED until the sending ended and still pass. Each hand-off now waits for a new execution to claim the receiver (its start time changes) and receive another message while messages are still arriving, and the test asserts at least five such resumptions. 1000 messages keep the sender going long enough.
On CockroachDB the receiver was still working through its backlog when the fixed 60 s wait after the sending ran out, and each variant took about 90 s of the 2-minute test timeout. The final wait now lasts while the receiver keeps receiving and gives up after 30 s without progress, and 500 messages sent 20 ms apart still keep the sender going through several resumptions: about 17 s per variant on Postgres and 50 s on CockroachDB.
Collaborator
Author
|
merging w/o review since it's only test changes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds tests only; it doesn't change the SDK.
RecvAcrossRecoveryTestchecks that a long-running receiver gets every message exactly once, in order, while it is recovered onto a new executor again and again. On DBOS Cloud, Conductor recovers a dead executor's workflows by callingrecoverPendingWorkflows, and that's the call these tests make.aReceiverRecoveredRepeatedlyGetsEveryMessageInOrderDBOSClientsends them about 20 ms apart, with no idempotency keys.close: each executor is closed before the next one recovers it, as in a redeploy. Some messages arrive while nothing is running the receiver.abandon: each executor keeps running through the hand-off, as on a host wrongly presumed dead. Its execution of the receiver is refused at its next write.aRetriedSendIsDeliveredTwiceWithoutAnIdempotencyKeyUnexpected message 0, expected 1.Both tests pass on Postgres and on CockroachDB.
These came out of an intermittent DBOS Cloud chaos-test failure (dbos-inc/dbos-cloud#1392), where a Java receiver ends in ERROR. The tests show the SDK delivers correctly across recovery. That points to the chaos test retrying its non-idempotent send.
🤖 Generated with Claude Code