Repository navigation
[DRAFT] All changes for pyiceberg-core wheel license attribution - #3380
Draft
dannycjones wants to merge 3 commits into
Draft
dannycjones wants to merge 3 commits into
dannycjones wants to merge 3 commits into
Conversation
This change migrates from the old shell script for cargo-deny to a new Python script. This new script will contain logic for dealing with dependencies and their licenses. This commit simply translates the existing logic. In later changes, this script will be extended with: - Clarifications for existing dependencies where licenses are misreported, with additional verifications added in the script itself - Logic for generating the attribution documentation for the pyiceberg-core package.
…hey ship Two dependencies declare licenses that do not describe the code actually compiled into the project: zstd-sys declares only "MIT/Apache-2.0", which covers its Rust wrapper and the pre-generated bindings. It also vendors Zstandard's C sources, which are dual BSD-3-Clause / GPL-2.0-only. There is no GPL entry in the allow list; the BSD-3-Clause half of that choice is what makes the crate acceptable, and the clarification is what records that this was decided rather than missed. brotli-decompressor declares "BSD-3-Clause/MIT", the legacy slash form, which reads as a choice. It is not one: src/context.rs is Google's Brotli code under MIT and is compiled in, so both licenses apply at once and Dropbox cannot offer that file under BSD-3-Clause. Upstream reached the same conclusion for the sibling crate after dropbox/rust-brotli#218. This is not cosmetic -- under the declared OR, BSD-3-Clause alone satisfies the allow list, so removing MIT from it would greenlight this crate while its MIT code still shipped. A clarification is attested by the hash of every license file it relies on, and that is where the problem lies: cargo-deny discards the entire block if any one hash stops matching, reports it at warning level only, and then judges the crate on its own manifest terms. The run still exits 0. Corrupting one of the five hashes in the zstd-sys block changes nothing a reader would notice. So a crate upgrade can retire a clarification silently, which is the opposite of what attesting a hash is for. `dependencies check` therefore cross-checks that every block applied, comparing sets of crate versions rather than counting clarifications. cargo-deny reports one per crate *instance*, not one per block, so a count fails open: a block matching two versions of a crate inflates the total and hides a second block that was discarded. Comparing versions also catches a block that reached some versions of a crate and not others, which is a real case -- getrandom appears three times in this graph with three different license texts. Two further checks come from the same reading. A clarification naming a crate no longer in the graph is reported separately, because the remedy is to delete it rather than re-hash it. And a version-pinned spec is rejected outright: cargo-deny accepts `foo@<2.1` as valid config and then applies the block to nothing, with no diagnostic, so the pin a reader would reach for to say "before the version that relicensed" is the one that silently does nothing. `dependencies clarification-hash` prints the hash deny.toml should record for a license file, so the remediation the failure suggests is a command rather than an algorithm. It needs xxhash, declared in the script's PEP 723 header, and imports it lazily -- every other command stays standard library only, so CI keeps invoking this with plain `python`. Recorded in deny.toml for the next audit: libsqlite3-sys has the same shape as zstd-sys, declaring MIT while vendoring SQLite's C sources, but SQLite is public domain and adds nothing to satisfy. aws-lc-sys already declares every license of everything it vendors.
…wheels The wheels ship a compiled extension module that statically links every Rust dependency, so they redistribute that code and take on its attribution conditions: the full license text of each crate, and for Apache-2.0 dependencies a relay of their NOTICE files under section 4(d). Today they carry only the project's own LICENSE and NOTICE. Source releases and crates.io publications bundle none of that code and are unaffected. Generated per wheel, not once. Each platform links a different set of crates, so a single bundle would either overclaim or omit; cargo-about runs against the Rust target triples behind each maturin target, and universal2-apple-darwin covers two triples because it is a fat binary. None of it is checked in: the release workflows generate the files once, upload them as an artifact, and stage the right set per wheel before maturin runs. Nothing upstream makes a mistake here loud, so the generator refuses to produce a bundle it cannot vouch for: maturin exits 0 and warns nothing when a `license-files` glob matches nothing, contrary to what PEP 639 requires of it, so `wheel verify` opens the built wheels and asserts the files are present, non-trivial, and generated for that wheel. cargo-about reproduces only the license files it matched by name, silently, so a crate keeping its license under an unmatched name loses its attribution. Four crates needed clarifying: ring, whose vendored BoringSSL and fiat-crypto code is compiled in; zstd-sys, which vendors Zstandard's C sources; libm; and alloc-stdlib, whose text exists only upstream. cargo-about also discards a clarification whose checksum no longer matches, at warning level, and then matches by name again -- the same failure mode as cargo-deny's, which is why both halves of this script cross-check that the clarifications they were given actually applied. Corrupting one of ring's three checksums drops the fiat-crypto notice and 16KB from the bundle without failing. Three crates ship no matchable license file at all, so cargo-about fell back to the SPDX template and emitted "Copyright (c) <year> <copyright holders>" -- a notice attributing nobody. require_no_placeholder_notices rejects those. brotli-decompressor is disclosed rather than fixed: src/context.rs is Google's MIT-licensed Brotli code and is compiled in, but the crate ships no MIT text, so it is listed under BSD-3-Clause only. The MIT text and its notice do reach the wheel via the brotli crate in the same graph, so the obligation is met; what is missing is the cross-reference. dropbox/rust-brotli-decompressor#32 fixes it upstream. Where a crate offers a choice of licenses this selects one, preferring Apache-2.0, and reproduces only that, per https://www.apache.org/legal/resolved.html#mutually-exclusive. LICENSE records the choice and points at THIRD-PARTY-LICENSES.
dannycjones
force-pushed
the
wheel-license-attribution-v2
branch
from
October 9, 2026 18:42
b9ccc01 to
69b7daf
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a draft of all the changes required to have wheels with correct license attribution, the blocker for releasing 0.11.
Please refer to individual PRs for review. This is available to preview what the full changes will look like.