Repository navigation
Rerun Seqera tasks the scheduler marks retryable as new tasks - #7933
Merged
pditommaso merged 8 commits intoOct 27, 2026
Merged
Conversation
nf-seqera: when the scheduler reports a FAILED task with `retryable=true` (e.g. a failed Fusion snapshot restore, exit 175/177), fail it with a ProcessRetryableException so it is rerun as a new task in a new work dir, regardless of errorStrategy. Capped at 3 reruns, then the exit code goes through errorStrategy. A missing field keeps the old path. core: count these reruns in a new `TaskRun.rerunCount` instead of `failCount`, so they don't spend maxRetries nor bump task.attempt (also applies to K8s node termination and spot retries). Signed-off-by: Jordi Deu-Pons <jordi@jordeu.net>
✅ Deploy Preview for nextflow-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
jordeu
marked this pull request as ready for review
October 27, 2026 06:42
pditommaso
self-requested a review
October 27, 2026 11:57
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Replace the hardcoded MAX_RETRYABLE_RERUNS with the task maxRetries directive, and cover the fallback to errorStrategy when the cap is reached and the scheduler reports no exit code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
…ci fast] Keep sched-client 0.111.0-SNAPSHOT: the 0.111.0 release does not include DescribeTaskResponse.retryable and retryReason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
The retryable branch of TaskProcessor.resumeOrDie already increments failCount and so reruns the task in a new work dir. Drop the new TaskRun.rerunCount and the related core changes, cap the reruns on failCount against maxRetries, and inline the retry check in SeqeraTaskHandler.checkIfCompleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
…fast] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
pditommaso
deleted the
task/nf-seqera-rerun-a-sched-task-marked-retr-998
branch
October 27, 2026 13:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a Fusion snapshot restore fails, rerunning the task in the same work dir can hit stale state left by the failed attempt. The Seqera scheduler now fails such a task and marks it
retryable, and Nextflow must rerun it in a new work dir. This PR makes the Seqera executor do that whatever the pipeline'serrorStrategysays, up to themaxRetriesdirective. Core code is unchanged: the existingProcessRetryableExceptionpath does the rerun.Changes
nf-seqera
SeqeraTaskHandlerkeeps the wholeDescribeTaskResponse, not onlytaskState, so it can read the new optionalretryableandretryReasonfields.FAILED,retryableis true andtask.failCount < maxRetries,task.erroris set to a newSeqeraRetryableException, aProcessRetryableException.TaskProcessor.resumeOrDiethen incrementsfailCountand reruns the task in the next work dir, outsideerrorStrategy, as it already does for K8s node terminations. The reason, exit code and error message are logged.maxRetriesreruns, or when the scheduler doesn't send the field, the exit code goes througherrorStrategyas before.task.abortedis deliberately not set, unlike inK8sTaskHandler.makeCopy()would carry the flag over, and a rerun that succeeded would be reported asABORTEDand not treated as cached on-resume.io.seqera:sched-clientbumped to0.111.0-SNAPSHOT, which addsDescribeTaskResponse.retryableandretryReason(the0.111.0release does not include them).docs
docs/reference/executor/seqera.mdx): reruns in a new work dir, how they count towardsmaxRetriesandtask.attempt, and how to allow more reruns.Behaviour
An infrastructure rerun counts as one of the user's
maxRetries(default 1) and raisestask.attemptfor a later real failure, the same as K8s node terminations today.Before merging
sched-clientSNAPSHOT with a release that includes the new fields.Tests
SeqeraTaskHandlerTest: covers retryable (first and last rerun), at the cap with and without an exit code,retryablefalse or absent, andSUCCEEDED.TaskProcessorTest: aProcessRetryableExceptioncause reruns the task regardless oferrorStrategy, incrementingfailCount(the core path had no test).