Skip to content

out_s3: address review comments on optional Arrow schema (#12529) - #12832

Open
Shrey93 wants to merge 3 commits into
fluent:cosmo0920-make-an-optional-arrow-schemafrom
Shrey93:address-review-12529
Open

Shrey93 wants to merge 3 commits into
fluent:cosmo0920-make-an-optional-arrow-schemafrom
Shrey93:address-review-12529

Conversation

@Shrey93

@Shrey93 Shrey93 commented Nov 22, 2026

Copy link
Copy Markdown

Follow-up to #12529 that addresses its open review comments, so the PR can come off hold. It targets the #12529 branch, not master. No rebase onto master is included; I left that to @cosmo0920 to avoid adding a merge commit.

Changes

  • out_s3: CodeRabbit flagged (Major) that with parquet.schema_file set, the $INDEX sequence file lived under <buffer>/msgpack. Restarting with the schema, or turning it off, restarted the index at 0. $INDEX keys have no random suffix, so the next upload could overwrite an existing S3 object. Schema mode now writes the index to <buffer>/sequence/index_metadata, the same path schema-less mode uses, outside the MessagePack store. The other-buffer-mode warning skips that directory so it is not reported as pending JSON-lines data.
  • aws: renamed json to data in compress_columnar(), since it now carries MessagePack too.
  • tests: integration: test_out_s3_schema_restart_preserves_seq_index restarts with an $INDEX-only key (schema on → on, schema on → off) and asserts that no key is reused. It fails on 614a5f8 (both runs upload /out_s3/0) and passes with this change.

The Codex comment about Arrow metadata keys was already handled in 614a5f8, and test_out_s3_schema_accepts_arrow_metadata covers it.

Out of scope: schema-less $INDEX also restarts at 0 after a clean shutdown when nothing is buffered. flb_fstore_destroy() deletes chunk-less stream directories, including sequence/. That happens on master today and is independent of this feature, so I am fixing it in a separate PR against master. Until that lands, switching the schema from off to on still starts at 0, the same as any schema-less restart today.

Testing (macOS arm64, arrow-glib/parquet-glib 25.0.1, -DFLB_ARROW=On -DFLB_DEV=On)

  • flb-it-aws_compress: all tests pass
  • ctest --test-dir build -R flb-rt-out_s3 --output-on-failure: 2/2 passed (flb-rt-out_s3, flb-rt-out_s3_otlp_json)
  • commit_prefix_check.py in pull-request mode against this branch: passed
  • ./run_tests.py scenarios/out_s3/tests/test_out_s3_001.py -k "schema or format": 25 passed (the 23 existing tests plus 2 new)
  • macOS leaks: ./run_tests.py --leaks-strict scenarios/out_s3/tests/test_out_s3_001.py -k "schema or format" -n 0: 25 passed, every Fluent Bit run exited leaks -atExit clean (no leaks reported)
  • Valgrind: not run here (macOS).

Enter [N/A] in the box, if an item is not applicable to your change.

Testing

  • [N/A] Example configuration file for the change (no new options; the regression test uses s3_key_format /$TAG/$INDEX with and without parquet.schema_file)
  • [N/A] Debug log output from testing the change (covered by the integration test above)
  • [N/A] Valgrind output: run on macOS, so the in-tree --leaks-strict mode was used instead (results above)

Documentation

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

With parquet.schema_file set, the file store is rooted at
<buffer>/msgpack and the $INDEX sequence stream was created inside it.
Its directory holds no chunks, so store shutdown removed it, and it also
did not match the location used without a schema. Either way the index
restarted at zero, and because $INDEX keys have no random suffix, new
uploads could overwrite existing S3 objects.

In schema mode, write the index to <buffer>/sequence/index_metadata,
the same path schema-less mode uses, outside the MessagePack store.
Skip the shared sequence directory when warning about buffers from the
other mode, so it is not reported as pending JSON-lines data.

Assisted-by: Kiro CLI 3.5.0 (claude-opus-5.5)
Signed-off-by: Shrey Nahata <nshreyan@amazon.com>
compress_columnar() now receives either JSON lines or Fluent Bit
MessagePack events depending on whether a schema is set, so the
parameter name json was misleading.

Assisted-by: Kiro CLI 3.5.0 (claude-opus-5.5)
Signed-off-by: Shrey Nahata <nshreyan@amazon.com>
Restart Fluent Bit with an $INDEX-only key while keeping or removing
parquet.schema_file, and assert that no S3 key is reused.

Assisted-by: Kiro CLI 3.5.0 (claude-opus-5.5)
Signed-off-by: Shrey Nahata <nshreyan@amazon.com>
@coderabbitai

coderabbitai Bot commented Nov 22, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a7b5600b-b574-4453-a5b9-297bee2687aa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-22T20:03:25.884339Z 5a99c56 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant