Skip to content

aws_s3: close the object scanner when reading an object fails - #4914

Open
a-palamarchuk wants to merge 1 commit into
redpanda-data:mainfrom
a-palamarchuk:aws-s3-close-scanner-on-error
Open

a-palamarchuk wants to merge 1 commit into
redpanda-data:mainfrom
a-palamarchuk:aws-s3-close-scanner-on-error

Conversation

@a-palamarchuk

Copy link
Copy Markdown

Fixes #2617.

What

When the aws_s3 input's scanner returned an error other than io.EOF while
reading an object, ReadBatch cleared a.object and returned without closing the
scanner. Because a.object was already cleared, Close couldn't reach it later
either. The scanner, and the S3 GetObject response body (plus any decompression
reader) it wraps, leaked for every object that failed mid-read, holding a
connection from the HTTP client's pool.

This closes the scanner on every read error, not only at end of object. The close
result goes into its own variable, so a close failure is logged and can't replace
the original read error returned to the caller.

Is it safe to close after an error?

Yes. Closing doesn't turn the failure into a successful acknowledgement of the S3
object, which in SQS mode would delete the notification:

  • Scanners built on service.AutoAggregateBatchScannerAcks (every built-in Benthos
    and Connect scanner) acknowledge the source with the read error inside
    NextBatch, through a once-only guard. Their Close then only releases the
    reader; the later acknowledgements are no-ops.
  • The deprecated codec: readers in Benthos follow the same pattern.
  • The wrapper scanners (decompress, skip_bom, switch) pass the source
    acknowledgement straight to the scanner they wrap.

So the object is still reported as failed exactly as before; the only change is
that its resources are released.

Tests

  • New TestReadBatchClosesScannerOnError plants a scanner that fails on read and
    records whether it was closed. It fails on main (scanner never closed) and
    passes with this change, and also checks the original error is still returned.
  • go test -race -shuffle=on ./internal/impl/aws/s3/... passes.
  • TestIntegrationS3 (LocalStack) passes, including the SQS line-reading subtests
    for both the current scanner and the deprecated codec: path, which exercise the
    normal end-of-object close that this change moves.
  • golangci-lint run and golangci-lint fmt --diff are clean.

When the scanner returned an error other than io.EOF, ReadBatch cleared
a.object and returned without closing the scanner, and with a.object cleared
Close couldn't reach it either. Every object that failed mid-read leaked its
scanner and the S3 GetObject response body behind it.

Close the scanner on every read error, keeping the original error as the
result. Scanners report the failure to the source before returning it, so
closing afterwards only releases resources and doesn't acknowledge the
object as processed.

Fixes redpanda-data#2617

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aws_s3: Scanner and backing reader not closed on non io.EOF error

2 participants