You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Parquet scan read errors name no file, or every file in the task, instead of the file that failed #7213
When a native Parquet scan fails to read a file, the FAILED_READ_FILE.NO_HINT error does not name the file the way Spark's does.
Corrupt file (random bytes in place of one part file): Spark's message names the bad file. Comet's message reads Encountered error while reading file ., with an empty path. The native error does carry the file name in its inner message (Failed to fetch metadata for file .../part-00001-....parquet: ... Corrupt footer), but params.filePath is "". In the branch-1.2 audit, a test with several files per task got the opposite problem: the path listed every file of the task's partition. That comes from the per-partition path fallback (perPartitionPaths).
Duplicate field in case-insensitive mode (seen in the audit, not re-run here): Spark wraps Found duplicate field(s) "a": [a, A] in case-insensitive mode in FAILED_READ_FILE.NO_HINT with the path. Comet raises the bare SparkRuntimeException. In field-id mode Comet wraps it, but with an empty path.
The empty path for PARQUET_COLUMN_DATA_TYPE_MISMATCH is already tracked in #4316.
The 1.1.0 release behaves the same way by code. The error conversion and the per-partition path fallback are unchanged, and #7094's metadata prefetch does not change the outcome.
Activity