Repository navigation
Conversation
With parquet.schema_file set, the file store is rooted at <buffer>/msgpack and the $INDEX sequence stream was created inside it. Its directory holds no chunks, so store shutdown removed it, and it also did not match the location used without a schema. Either way the index restarted at zero, and because $INDEX keys have no random suffix, new uploads could overwrite existing S3 objects. In schema mode, write the index to <buffer>/sequence/index_metadata, the same path schema-less mode uses, outside the MessagePack store. Skip the shared sequence directory when warning about buffers from the other mode, so it is not reported as pending JSON-lines data. Assisted-by: Kiro CLI 3.5.0 (claude-opus-5.5) Signed-off-by: Shrey Nahata <nshreyan@amazon.com>
compress_columnar() now receives either JSON lines or Fluent Bit MessagePack events depending on whether a schema is set, so the parameter name json was misleading. Assisted-by: Kiro CLI 3.5.0 (claude-opus-5.5) Signed-off-by: Shrey Nahata <nshreyan@amazon.com>
Restart Fluent Bit with an $INDEX-only key while keeping or removing parquet.schema_file, and assert that no S3 key is reused. Assisted-by: Kiro CLI 3.5.0 (claude-opus-5.5) Signed-off-by: Shrey Nahata <nshreyan@amazon.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Follow-up to #12529 that addresses its open review comments, so the PR can come off hold. It targets the #12529 branch, not master. No rebase onto master is included; I left that to @cosmo0920 to avoid adding a merge commit.
Changes
out_s3: CodeRabbit flagged (Major) that withparquet.schema_fileset, the$INDEXsequence file lived under<buffer>/msgpack. Restarting with the schema, or turning it off, restarted the index at 0.$INDEXkeys have no random suffix, so the next upload could overwrite an existing S3 object. Schema mode now writes the index to<buffer>/sequence/index_metadata, the same path schema-less mode uses, outside the MessagePack store. The other-buffer-mode warning skips that directory so it is not reported as pending JSON-lines data.aws: renamedjsontodataincompress_columnar(), since it now carries MessagePack too.tests: integration:test_out_s3_schema_restart_preserves_seq_indexrestarts with an$INDEX-only key (schema on → on, schema on → off) and asserts that no key is reused. It fails on614a5f8(both runs upload/out_s3/0) and passes with this change.The Codex comment about Arrow
metadatakeys was already handled in614a5f8, andtest_out_s3_schema_accepts_arrow_metadatacovers it.Out of scope: schema-less
$INDEXalso restarts at 0 after a clean shutdown when nothing is buffered.flb_fstore_destroy()deletes chunk-less stream directories, includingsequence/. That happens on master today and is independent of this feature, so I am fixing it in a separate PR against master. Until that lands, switching the schema from off to on still starts at 0, the same as any schema-less restart today.Testing (macOS arm64, arrow-glib/parquet-glib 25.0.1,
-DFLB_ARROW=On -DFLB_DEV=On)flb-it-aws_compress: all tests passctest --test-dir build -R flb-rt-out_s3 --output-on-failure: 2/2 passed (flb-rt-out_s3,flb-rt-out_s3_otlp_json)commit_prefix_check.pyin pull-request mode against this branch: passed./run_tests.py scenarios/out_s3/tests/test_out_s3_001.py -k "schema or format": 25 passed (the 23 existing tests plus 2 new)leaks:./run_tests.py --leaks-strict scenarios/out_s3/tests/test_out_s3_001.py -k "schema or format" -n 0: 25 passed, every Fluent Bit run exitedleaks -atExitclean (no leaks reported)Enter
[N/A]in the box, if an item is not applicable to your change.Testing
s3_key_format /$TAG/$INDEXwith and withoutparquet.schema_file)--leaks-strictmode was used instead (results above)Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.