feat(storage): Add HDFS support via opendal services-hdfs-native by mixermt · Pull Request #3111 · apache/iceberg-rust

mixermt · 2026-08-30T06:58:22Z

Which issue does this PR close?

Closes Support tables which live on HDFS #2440.
Part of HDFS support in iceberg-rust #1130.

This revives #1131's successor #2441 by @jordepic (closed by the stale bot after a first review round), rebased onto the current Storage-trait layout and updated for opendal 0.58, where services-hdfs-native changed behavior in ways that required design changes (details below).

What changes are included in this PR?

Adds an opendal-hdfs-native cargo feature to iceberg-storage-opendal, with OpenDalStorageFactory::HdfsNative / OpenDalStorage::HdfsNative variants and hdfs:// routing in OpenDalResolvingStorage. The backend uses OpenDAL's services-hdfs-native (pure-Rust HDFS RPC via hdfs-native — no JNI/libhdfs). The feature is experimental and not part of opendal-all.

NameNode resolution (differs from #2441, forced by opendal's hdfs-native service, where name_node is mandatory and, since 0.56 (apache/opendal#7248), its comma-split list is the HA mechanism):

An authority with a port (hdfs://host:port/path) is used as is; a configured hdfs.name-node never overrides it. Userinfo and port 0 in an authority are rejected with DataInvalid (every error the HDFS backend raises for a path or NameNode value masks everything between the scheme and the last @, e.g. hdfs://***@nn:8020/x).
A portless authority (hdfs://ns1/path) must be a declared nameservice: hdfs.name-node.<nameservice> takes a single endpoint or a comma-separated list for HA failover and is sugar for Hadoop's dfs.ha.namenodes.<nameservice> / dfs.namenode.rpc-address.<nameservice>.<id>, which are honored as well when passed through hadoop.*. Unlike Hadoop, an undeclared one is an error naming the key to set, rather than being resolved from $HADOOP_CONF_DIR or dialed on port 8020: opendal always builds the client as hdfs://nameservice from an explicit list, so hdfs-native never sees the path's authority (noted in the README; lifting it needs opendal to accept a logical nameservice as name_node). Every NameNode spelling, configured or from a path, is normalized to hdfs://host:port (host lowercased, as Hadoop's FileSystem cache key does) so all sources share cache keys; other schemes, userinfo, paths, port 0 and unbracketed IPv6 are rejected, and hdfs-native has no default port, so a portless NameNode can only be a logical name.
An authority-less path (hdfs:///path) resolves through plain hdfs.name-node (new HDFS_NAME_NODE / HDFS_HADOOP_CONF_PREFIX constants in iceberg::io), else fs.defaultFS: set from PyIceberg's hdfs.host/hdfs.port (new HDFS_HOST / HDFS_PORT constants, port defaults to 8020, validated at config time; as in PyIceberg they apply only without a path authority) or from hadoop.fs.defaultFS. Malformed declarations, sugar or passed through, and a non-HDFS fs.defaultFS that serves authority-less paths fail when the config is parsed; an undeclared nameservice or a missing default fails when a path needs it, with an error naming the property to set.

The properties are read through #[derive(Properties)]. hadoop.-prefixed properties pass through to the HDFS client config with the prefix stripped, overriding $HADOOP_CONF_DIR values (mirroring the hadoop. catalog-property convention of the Java integrations); hdfs-native still loads core-site.xml/hdfs-site.xml from $HADOOP_CONF_DIR/$HADOOP_HOME for everything else, and Kerberos works via libgssapi_krb5 (runtime dlopen). PyIceberg's hdfs.user/hdfs.kerberos_ticket have no equivalent in opendal's config and log a warning.

Operators are cached per effective NameNode inside an opaque HdfsNativeStorage (built only by the factories). hdfs-native binds a client's NameNode connections to the runtime current when the client is built and panics once that runtime is gone, so clients are built inside a crate-owned runtime (started on first use) and a cached operator works from any caller's runtime; create_operator stays synchronous for every backend. The build reads the Hadoop XML synchronously (~0.1 ms), once per NameNode per storage. A poisoned cache lock is recovered rather than failing every later call.

Relative paths are returned opendal-style without a leading / — opendal::Deleter::delete (used by delete_stream) rejects leading slashes, which an integration test caught.

Test infrastructure: a single-node HDFS docker fixture (apache/hadoop:3.5.0) on the shared bridge network with published ports, started by make docker-up like the other fixtures. The DataNode advertises 127.0.0.1 and the tests set dfs.client.use.datanode.hostname, since hdfs-native otherwise dials the DataNode's container IP, which the host cannot reach on macOS or Windows. The DataNode healthcheck gates on NameNode registration so --wait means writable. The tests run in the default CI job; ICEBERG_TEST_HDFS_ENDPOINT overrides the endpoint.

Are these changes tested?

34 unit tests across config parsing and normalization (per-entry trimming, port validation, hdfs.host/hdfs.port into fs.defaultFS, rejected and ignored keys, declarations validated at parse), path parsing (authority/port/IPv6/authority-less/non-hierarchical), NameNode precedence and nameservice declarations, spelling normalization (including host case), operator caching (first-insert-wins for racing builds, poisoned-lock recovery), userinfo masking in errors, serde round-trip, batch keys, relativize, and scheme resolution.
14 integration tests against the docker fixture covering exists/read/write/overwrite/metadata/range reads/streaming writes/delete/delete_prefix/delete_stream (+empty, +two NameNode spellings), authority-less paths through hdfs.name-node and hdfs.host/hdfs.port, resolving storage over hdfs://, and the HA flow (logical authority in the path, declared with a dead first NameNode so every call fails over), plus a fixture-free test that reuses a FileIO from a second runtime after the first is dropped (it panics if clients are built on the caller's runtime). The docker-backed suite runs in the default CI job.
make check (incl. cargo-deny licenses), full-workspace unit/doc tests, MSRV, the public API check, and the s3/gcs/hms/resolving integration suites pass locally; public-api.txt regenerated for both crates.

AI Disclosure

Developed with AI assistance (Claude Code): drafting code/tests/fixtures starting from #2441, and cross-checking the design against the locked opendal 0.58.1 / hdfs-native 0.14.6 sources. I reviewed the implementation and ran all verification locally. Areas worth reviewer attention: the NameNode-resolution semantics above (opendal's synthetic-nameservice behavior constrains what hdfs://<nameservice> paths can do without the property), and the Windows --all-features build of hdfs-native, which I could only verify via CI.

🤖 Generated with Claude Code

@jordepic

Adds an opendal-hdfs-native cargo feature plus OpenDalStorageFactory::Hdfs and OpenDalStorage::Hdfs variants in iceberg-storage-opendal, using OpenDAL's services-hdfs-native (pure-Rust HDFS RPC, no JNI/libhdfs). The NameNode for a path resolves as: the hdfs.name-node property when set (comma-separated endpoints enable HA failover), otherwise the path authority. hadoop.-prefixed properties are forwarded to the HDFS client configuration, overriding values loaded from $HADOOP_CONF_DIR. Operators are cached per effective NameNode since each holds live RPC connections. Revives and updates PR apache#2441 (by @jordepic) against the current storage layer and opendal 0.58, where name_node became mandatory and the comma list is the HA mechanism. Closes apache#2440 Co-authored-by: Jordan Epstein <jordepic@users.noreply.github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

blackmwk

I found two correctness issues that need to be addressed before merge: malformed hdfs: URLs can panic, and bulk deletion can conflate NameNodes running on different ports. Details and suggested regression coverage are inline. Please also resolve the two existing review threads.

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Rust maintainer. After you've addressed the points above and pushed an update, an Apache Iceberg Rust maintainer — a real person — will take the next look at the PR. The findings cite the project's review criteria; if you think one of them is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Apache Iceberg Rust handles maintainer review: CONTRIBUTING.md.

…S CI special-casing - HdfsConfig is now pub(crate) and parsed via #[derive(Properties)] (key/prefix attributes) instead of hand-written TryFrom + TypedBuilder. - HDFS integration tests are no longer #[ignore]d and the dedicated CI step is gone; they run under the default nextest invocation since make docker-up already starts the fixture and the Tests job is Linux-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…eat-storage-hdfs-native

…effective NameNode - Url::parse accepts non-hierarchical forms like `hdfs:x`; the byte-7 slice then panicked. Require the literal `hdfs://` prefix instead. - batch_key_for_path grouped by URL host only, so NameNodes differing by port shared one deleter; key by the effective NameNode (configured hdfs.name-node, else authority incl. port), matching the operator cache. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

blackmwk

Thanks @mixermt for this pr. I think the variant should be named to Hdfs-native rather than hdfs. Hdfs should be reserved for non native hdfs support.

# Conflicts: # Cargo.lock

…sConfig Per review: the opendal module, its functions and the storage/factory variants are now hdfs_native-prefixed (leaving room for a libhdfs-backed variant, see apache#1130), and the never-consumed HdfsConfig struct is removed from the core crate — config/hdfs.rs keeps only the property constants that iceberg-storage-opendal uses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mixermt · 2026-09-05T20:20:06Z

@blackmwk
I've addressed all the recent feedback.
Could you please give this one last review? 🙏

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

comphead

Reviewed the backend against the locked opendal-service-hdfs-native 0.58.1 and hdfs-native 0.14.5 sources, and against the existing hf/oss/azdls backends in this crate.

The overall shape fits the crate well: *_config_parse / *_batch_key / (Operator, &str) mirror hf, and the dedicated batch_key arm is genuinely needed, since host-only keying would merge NameNodes that differ only by port. The operator cache is justified too, given each client re-reads the Hadoop XML and opens RPC connections.

Leaving 5 inline comments: 2 blockers (test gating versus what the description claims, and blocking I/O under the cache write lock) and 3 majors (the cache leaking into the public API, duplicated NameNode-precedence logic, and empty-name-node handling).

I also have a handful of minor simplification notes and two questions (operator caching pins the Client to the tokio runtime that first built it, since hdfs-native captures Handle::try_current() eagerly; and the hadoop. prefix scope). Happy to add those if useful, but they seemed like noise next to the above.

…gle NameNode rule, env-gated HDFS tests - HdfsNativeOperatorCache newtype keeps the cache representation out of the public API. - Operators are built outside the cache lock (the build reads Hadoop XML synchronously); a racing first caller's duplicate is dropped unopened. - hdfs_native_effective_name_node is the single source of the configured-else-authority rule for both create_operator and the delete_stream batch key; unresolvable paths key on themselves. - hdfs.name-node is trimmed and an empty value is treated as unset, so it no longer shadows the path-authority fallback. - HDFS integration tests self-skip unless ICEBERG_TEST_HDFS_ENDPOINT is set (as the HF tests do) and the compose services sit behind the hdfs profile; the Linux CI job opts in via env. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Inherited from main (tracked in apache#3222); same bump as apache#3165. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mixermt · 2026-09-17T09:55:46Z

Thanks @comphead for the thorough review — all five items are addressed in d17ae38 and answered inline. The security_audit failure was RUSTSEC-2026-0285 in rustls 0.23.41, which is main's current pin (AWS SDK chain, unrelated to hdfs-native; tracked in #3222). Bumped to 0.23.45 in b8cc860, the same fix #3165 applied. The minor simplification notes you held back are welcome — please send them.

"Bumped to 0.23.45 in b8cc860, the same fix #3165 applied" if we keep it without it CI will stay red, tell me if we want to revert this under this PR

# Conflicts: # Cargo.lock

mixermt · 2026-09-22T12:05:23Z

@comphead @blackmwk any other comments?
I really want to push this forward

I'm testing this with datafusion-comet, so far no issues in this code.

A panic elsewhere while the cache lock was held turned every later HDFS call into a permanent `Unexpected` error. Writes only ever replace whole entries, so the map stays consistent: both lock sites now recover the guard with `PoisonError::into_inner`, and `get`/`insert` are infallible. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The runtime test sampled its dial counter before the listener thread had necessarily counted every dial from the first runtime. It now drops that runtime and waits for the listener to close a marker connection; accepts are served in order, so every earlier dial is counted first. A failed accept no longer stops the listener. The runtime-shutdown unit test also asserts that the rebuilt operator is a new instance, not the stale one with a fresh sentinel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Hadoop resolves a portless authority from hdfs-site.xml, else dials port 8020. Here it must be declared with `hdfs.name-node.<nameservice>`, because opendal builds the client as `hdfs://nameservice` and never hands hdfs-native the authority. The `HDFS_NAME_NODE` and resolver docs no longer claim Hadoop parity, the error says to add a port or declare the nameservice, and the README notes the limitation in its own note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

blackmwk

Just finised another round of review, need more iterations.

comphead · 2026-10-09T15:13:34Z

Thanks folks for helping out, I'll do another round of review today as well

comphead

Thanks for the work here. I reviewed this from a Rust idioms and design angle and did not re-audit the NameNode semantics that earlier rounds settled. Overall it follows the existing per-backend pattern, and the concurrency and public-API concerns from earlier rounds look addressed.

The main point is the first inline comment on typed resolution and config-time validation. The rest are questions and nits.

One test gap: the fixture is single-node, so comma-list HA failover is only asserted as strings. A bump of opendal = "0.58" that changes the synthetic-nameservice behavior would not be caught.

I did not build or run anything, and I did not check the Windows --all-features build. Please check and address all review comments in this PR.

comphead

Follow-up on idioms and design

Matches its siblings, which are named after their storage feature (azdls, gcs, hf, oss, s3). The module is private and re-exported, so the public iceberg::io::HDFS_* paths are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The serde round-trip and relativize_path tests now live next to the code they cover, with the module's test_hdfs_native_ prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hdfs.name-node, the hdfs.name-node.<nameservice> and hadoop. prefixes, hdfs.host and hdfs.port are declared in one struct parsed by the Properties macro instead of being looked up by hand. Host and port stay trimmed strings so a blank value still counts as unset, as in PyIceberg. Validation, warnings and error messages are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The HDFS fixture joins the shared bridge network with published ports instead of using host networking behind a compose profile, so make docker-up starts it on every platform: the DataNode advertises 127.0.0.1 and the tests set dfs.client.use.datanode.hostname. The tests no longer skip without ICEBERG_TEST_HDFS_ENDPOINT, and the dedicated Tests (hdfs) job is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An hdfs-native client spawns its NameNode connections on the runtime current when it is built and panics once that runtime is dropped. The cache used to detect that with a sentinel task per entry, which needed Handle::try_current, and the build ran under spawn_blocking, which made create_operator async for every backend. Clients are now built inside a crate-owned runtime started on first use, so a cached operator works from any caller's runtime: the sentinel, Handle::try_current and spawn_blocking are gone, and create_operator is sync again as on main. The Hadoop XML read (~0.1 ms) runs inline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ns at parse hdfs_native_parse_path returns an HdfsNativeAuthority (NameNode, Nameservice, Default), so the resolver matches on it instead of re-parsing the authority to learn whether it has a port. Every HA nameservice declaration, from hdfs.name-node.<nameservice> or passed through as hadoop.dfs.ha.namenodes.*, and the fs.defaultFS that serves authority-less paths are now validated when the config is parsed, so a broken one fails the first call rather than only the paths that use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…FS backend The nameservice lookup is a for loop with let-else, and the userinfo masking and the relative-path extraction use split_once/rsplit_once instead of slicing at computed indices. A test pins that doubled slashes after the authority are still trimmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The parse error was interpolated into the message; it is now attached as the error's source, which iceberg prints after the message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

host_str keeps the host's case for hdfs:// URLs, so hdfs://NN1:8020 and hdfs://nn1:8020 cached two clients for one NameNode. Endpoint hosts are now lowercased by one helper used by the config normalizer and the path parser, as Hadoop's FileSystem cache key does; nameservice names stay case-sensitive like the Hadoop keys that declare them. The cache's doc states that it is unbounded by design. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

opendal's hdfs-native service does not expose hdfs-native's ClientBuilder::with_io_runtime (Kimahriman/hdfs-native#255); a TODO at the crate-owned runtime records that a runtime could be passed in once it does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The declared-nameservice test listed the single-node fixture twice, which drove opendal's list path but never needed a failover. Its first NameNode is now a local port nothing listens on, so every call has to fail over through opendal's synthetic nameservice; an opendal change that kept only the first entry of the list would fail it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Cargo.lock is main's, plus the hdfs-native dependency set this branch adds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mixermt · 2026-10-09T20:58:28Z

@comphead on the review notes: the declared-nameservice test now lists a dead NameNode first, so every call has to fail over through opendal's synthetic nameservice, and keeping only the first entry of the list makes it fail. The Windows --all-features build is covered by CI's build (windows-latest) job (cargo build --all-targets --all-features --workspace), which passes.

Docker Hub rate-limited the previous run's image pulls, so the default test job never started its fixtures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The previous run hit Docker Hub's unauthenticated pull rate limit again before any test ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mixermt · 2026-10-10T09:59:15Z

@blackmwk @comphead @laskoviymishka
Thank you for taking the time to review and provide feedback.
I believe I have addressed all the issues.

comphead

Thanks @mixermt it looks good to me, would you also mind to document the HDFS support and how to enable it for the users

The iceberg-storage-opendal README gains an HDFS section: enabling the opendal-hdfs-native feature, the factory, the hdfs.* and hadoop.* properties, the Hadoop config, user and Kerberos it reads, and the portless-authority limitation. The experimental-backends note now points to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mixermt · 2026-10-10T18:51:51Z

@comphead Done: the iceberg-storage-opendal README now has an HDFS section covering how to enable opendal-hdfs-native, the factory, the hdfs.*/hadoop.* properties, the Hadoop config, user and Kerberos it reads, and the portless-authority limitation.

…-rust#3111 ef451b66 merges apache/iceberg-rust main into the PR: typed NULL arrays for schema-added nested columns on read (apache#2635), an azdls relative-path fix (apache#3311) and dependency bumps. The HDFS resolver is unchanged, so the hdfs.name-node.<nameservice> declaration Comet emits still applies, and the head still contains main's pin (af1da4c5). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Brings in origin/main up to 9e13649 and moves the iceberg-rust pin to the current head of apache/iceberg-rust#3111 (0145ca6b), which still contains main's pin (af1da4c5). Main now loads the Iceberg scheme allow-lists from the native builtin_storage_schemes over JNI, so hdfs is admitted there (read and write) instead of in the two Scala sets. The pin brings apache/iceberg-rust apache#3145, which makes FileScanTaskDeleteFile's fields private behind a validating builder and its deletion-vector offsets unsigned: the planner builds delete files through the builder (rejecting a negative offset or size as iceberg-rust does for a manifest entry), and the delete-file sizing rebuilds each delete through it. The reworked HDFS backend validates every declared nameservice and a forwarded fs.defaultFS when it builds the storage, failing all I/O otherwise. Comet's gate now simulates its option expansion (hadoop.* keys win over the hdfs.name-node.<ns> sugar key by key) and declines those cases at planning. It also builds HDFS clients in a runtime of its own, so the AbortOnDrop comment about runtime-bound operators is replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mixermt and others added 2 commits August 29, 2026 18:57

Merge branch 'main' into feat-storage-hdfs-native

4e0f745

blackmwk reviewed Aug 31, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Outdated

Comment thread crates/iceberg/src/io/storage/config/hdfs.rs Outdated

blackmwk requested changes Aug 31, 2026

View reviewed changes

Comment thread crates/storage/opendal/src/hdfs.rs Outdated

Comment thread crates/storage/opendal/src/lib.rs Outdated

mixermt and others added 4 commits August 31, 2026 12:17

Merge remote-tracking branch 'origin/feat-storage-hdfs-native' into f…

7939b68

…eat-storage-hdfs-native

Merge branch 'main' into feat-storage-hdfs-native

4141f4e

mixermt requested a review from blackmwk September 1, 2026 12:05

Merge branch 'main' into feat-storage-hdfs-native

2578c03

blackmwk reviewed Sep 4, 2026

View reviewed changes

Comment thread crates/iceberg/src/io/storage/config/hdfs.rs Outdated

Comment thread crates/storage/opendal/src/hdfs_native.rs

Comment thread crates/storage/opendal/src/hdfs.rs Outdated

mixermt and others added 2 commits September 5, 2026 12:11

Merge remote-tracking branch 'apache/main' into feat-storage-hdfs-native

13911c5

# Conflicts: # Cargo.lock

mixermt requested a review from blackmwk September 5, 2026 21:28

mixermt and others added 3 commits September 8, 2026 18:44

Merge branch 'main' into feat-storage-hdfs-native

a9a6786

Merge branch 'main' into feat-storage-hdfs-native

806bd89

ci: retrigger after setup-python manifest timeout on macos runner

9d7d2d8

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This was referenced Sep 13, 2026

Support Iceberg tables on HDFS in the native scan apache/datafusion-comet#5894

Open

feat(iceberg): support HDFS storage via iceberg-rust hdfs-native backend [NOT FOR MERGE: pending apache/iceberg-rust#3111] apache/datafusion-comet#5898

Draft

comphead reviewed Sep 15, 2026

View reviewed changes

Comment thread dev/docker-compose.yaml

mixermt and others added 3 commits September 16, 2026 15:58

Merge branch 'main' into feat-storage-hdfs-native

c561cfc

chore(deps): bump rustls to 0.23.45 (RUSTSEC-2026-0285)

b8cc860

Inherited from main (tracked in apache#3222); same bump as apache#3165. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge remote-tracking branch 'apache/main' into feat-storage-hdfs-native

587688c

# Conflicts: # Cargo.lock

andygrove mentioned this pull request Sep 21, 2026

deps: bump the iceberg-rust pin to bb1e4a4 and document why it is pinned apache/datafusion-comet#6094

Merged

Merge branch 'main' into feat-storage-hdfs-native

72305b6

mixermt and others added 3 commits October 8, 2026 11:45

blackmwk requested changes Oct 9, 2026

View reviewed changes

comphead reviewed Oct 9, 2026

View reviewed changes

Comment thread crates/storage/opendal/src/hdfs_native.rs Outdated

Comment thread crates/storage/opendal/src/hdfs_native.rs

Comment thread crates/storage/opendal/src/hdfs_native.rs Outdated

Comment thread crates/storage/opendal/src/lib.rs

mixermt and others added 12 commits October 9, 2026 21:45

test(storage): move the HDFS tests out of lib.rs into hdfs_native

a347b84

The serde round-trip and relativize_path tests now live next to the code they cover, with the module's test_hdfs_native_ prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(storage): attach the hdfs.port parse error as the source

80f2149

The parse error was interpolated into the message; it is now attached as the error's source, which iceberg prints after the message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge apache/main into feat-storage-hdfs-native

fdd9562

Cargo.lock is main's, plus the hdfs-native dependency set this branch adds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mixermt mentioned this pull request Oct 9, 2026

Split the iceberg-storage-opendal lib.rs tests into per-backend modules #3385

Open

mixermt requested a review from blackmwk October 9, 2026 21:01

mixermt and others added 2 commits October 10, 2026 00:07

chore: retrigger CI

8aba12b

Docker Hub rate-limited the previous run's image pulls, so the default test job never started its fixtures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chore: retrigger CI

650e793

The previous run hit Docker Hub's unauthenticated pull rate limit again before any test ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

comphead approved these changes Oct 10, 2026

View reviewed changes

Conversation

mixermt commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

AI Disclosure

Uh oh!

Uh oh!

Uh oh!

blackmwk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

blackmwk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mixermt commented Sep 5, 2026

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mixermt commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mixermt commented Sep 22, 2026

Uh oh!

blackmwk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

comphead commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

comphead left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mixermt commented Oct 9, 2026

Uh oh!

mixermt commented Oct 10, 2026

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

mixermt commented Oct 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

mixermt commented Aug 30, 2026 •

edited

Loading

mixermt commented Sep 17, 2026 •

edited

Loading

comphead commented Oct 9, 2026 •

edited

Loading

comphead left a comment •

edited

Loading