Skip to content

Major dependency update and bugfix - #878

Open
lyda wants to merge 1 commit into
peak:masterfrom
lyda:master
Open

lyda wants to merge 1 commit into
peak:masterfrom
lyda:master

Conversation

@lyda

@lyda lyda commented Sep 8, 2026

Copy link
Copy Markdown

Note: this might be too big for a PR, but normally when I fix a bug I first update dependencies and there was a lot to to here. The bulk of it was the removal of the vendor directory.

tl;dr: Upgraded dpendencies, removed vendor/, modernised code, fixed some small bugs.

Follow-on work from go get -u ./... and go mod vendor, which left the tree building against newer major-version behaviour it was not written for. Each item below was verified against the test suite rather than by inspection.

go fix / go vet

  • command/sync.go: drop unneccesary var blocks. extsort v1.4.2 returns a receive-only channel from New, so the pre-declared bidirectional channel vars no longer matched.
  • e2e: 13 call sites annoyed go vet. Fixed

urfave/cli v2 -> v3
--------------------------------------------------- After an aborted upgrade to v2.27, I noticed there was a v3. Some of these chnages were from the older v2; most changes needed for v3. Overall v3 is a better module.

  • cli.App and cli.Context are gone: handlers take (context.Context, *cli.Command); Before returns a context.
  • The Flag interface changed shape, so the custom MapFlag was ported to the PreParse/PostParse/Set lifecycle. It is kept rather than replaced by v3's StringMapFlag because it rejects duplicate keys.
  • v3 runs the Before hooks of a command's whole lineage. Run-mode executes each line as a child of the root, so log.Init and parallel.Init were re-entered per line; they are now guarded by sync.Once.
  • GenericFlag values read back as "" through both cmd.String (asserts to string) and cmd.Generic (asserts to cli.Value, which the by-value stored EnumValue does not satisfy). This had silently disabled --log, --set, --metadata-directive and --structure; genericString() fixes it.
  • v3 runs Before before the shell-completion callback, and every command validates its argument count there, so completion produced nothing. Sub-command Before hooks are dropped for completion requests.
  • The run-mode help workaround is removed: v3 builds its help command per-setup instead of mutating package state.
  • Error messages return to the v2.3.0 form -- "cp s3://a b" with no command prefix for root-level failures.

sync: keys containing tab or newline (NoSuchKey on copy) -------------------------------------------------------- This was the bug that kicked off all these changes.

sync serialises each copy into a command line that run-mode parses back with a shell parser. The problem is it uses %q, which renders tabs as an escape, not a tab. Now quoted with shellquote.Join, the exact inverse of the shellquote.Split that reads it. Use the same thing on both sides kids.

Keys containing a newline failed differently: run-mode reads commands line by line, so a quoted newline split the command and aborted the whole run with exit 1 and no message. The reader now joins lines while a quote is open, and reports unterminated input with its line number.

Regression tests cover tab, newline, CR, a literal backslash-t and space, both as a round-trip unit test and end to end.

aws-sdk-go v1 -> v2

v1 reached end of support on 2025-07-31 and is archived. Production code no longer imports it; it remains only as an indirect requirement of gofakes3. Four behaviour differences needed handling:

  • v2 computes a CRC32 checksum for every upload by default. v1 sent none, and many S3-compatible services reject them, so checksums are set to WhenRequired.
  • manager.GetBucketRegion clears credentials, so private buckets need them restored -- but not the anonymous provider, which v2's signer rejects outright. It also returns "" with no error when the region header is absent, which was blanking the region.
  • v2 wraps API errors with the operation, request id and a 100-character host id. cleanupError now renders the concise "Code: message status code: N" form s5cmd has always printed.
  • HeadObject returns a nil metadata map where v1 returned an empty one, which changed head --json output to "metadata":null.

The transfer manager (feature/s3/manager) is deprecated in favour of feature/s3/transfermanager, which is still pre-1.0; s5cmd stays on the stable package, with the deprecation confined to two type aliases.

Dependencies and tooling

  • go.mod tool directives replace internal/tools/tools.go and its //go:build tools tag, which is why staticcheck, unparam and mockgen were invisible to go get -u ./.... Invoked via go tool.
  • staticcheck v0.4.7 -> v0.8.1, unparam to current, strcase to v0.3.0 (test-only deps need go get -u -t), plus toml, kr/pretty, x/mod, x/exp/typeparams and check.v1.
  • extsort's deprecated SortType interface migrated to the Generic API, removing the per-item type assertions.
  • The e2e harness moved off SDK v1, which surfaced three more differences: uploads now carry a Content-Type, metadata keys come back lowercased, and Expires is parsed (now read via ExpiresString).
  • staticcheck: 34 findings -> 0. unparam: 0.

vendor/ removed

Versions and integrity are pinned by go.mod and go.sum; vendoring added no locking, only 27MB of a 29MB checkout -- 5.8MB of it lint tooling that never ships in the binary. Removing it drops -mod=vendor from six Makefile targets and from the e2e harness's own go build.

CI

The matrices tested Go 1.20/1.21/1.22 against a go.mod that already required 1.25, so they could not have built. Now 1.26, which the tool directives and the upgraded linters require. goreleaser and the Dockerfile follow.

The e2e suite now pins its umask: s5cmd creates directories with os.ModePerm, so ten tests asserting on directory permissions passed under umask 022 and failed under 002.

User-visible changes

  • Shell completion is triggered by --generate-shell-completion in v3; anyone with an installed completion script must re-run --install-completion.
  • Building requires Go 1.26 (raised by the linters' own requirements).
  • "Incorrect Usage" now goes to stderr rather than stdout.
  • Help output differs slightly in flag rendering, per v3's templates.

tl;dr: Upgraded dpendencies, removed vendor/, modernised code,
fixed some small bugs.

Follow-on work from `go get -u ./...` and `go mod vendor`, which left the
tree building against newer major-version behaviour it was not written
for. Each item below was verified against the test suite rather than by
inspection.

go fix / go vet
---------------
* command/sync.go: drop unneccesary var blocks. extsort v1.4.2 returns a
  receive-only channel from New, so the pre-declared bidirectional
  channel vars no longer matched.
* e2e: 13 call sites annoyed go vet.  Fixed

urfave/cli v2 -> v3
---------------------------------------------------
After an aborted upgrade to v2.27, I noticed there was a v3.  Some of
these chnages were from the older v2; most changes needed for v3.
Overall v3 is a better module.

* cli.App and cli.Context are gone: handlers take
  (context.Context, *cli.Command); Before returns a context.
* The Flag interface changed shape, so the custom MapFlag was ported to
  the PreParse/PostParse/Set lifecycle. It is kept rather than replaced
  by v3's StringMapFlag because it rejects duplicate keys.
* v3 runs the Before hooks of a command's whole lineage. Run-mode
  executes each line as a child of the root, so log.Init and
  parallel.Init were re-entered per line; they are now guarded by
  sync.Once.
* GenericFlag values read back as "" through both cmd.String (asserts to
  string) and cmd.Generic (asserts to cli.Value, which the by-value
  stored EnumValue does not satisfy). This had silently disabled --log,
  --set, --metadata-directive and --structure; genericString() fixes it.
* v3 runs Before before the shell-completion callback, and every command
  validates its argument count there, so completion produced nothing.
  Sub-command Before hooks are dropped for completion requests.
* The run-mode help workaround is removed: v3 builds its help command
  per-setup instead of mutating package state.
* Error messages return to the v2.3.0 form -- "cp s3://a b" with no
  command prefix for root-level failures.

sync: keys containing tab or newline (NoSuchKey on copy)
--------------------------------------------------------
This was the bug that kicked off all these changes.

sync serialises each copy into a command line that run-mode parses back
with a shell parser.  The problem is it uses %q, which renders tabs
as an escape, not a tab.  Now quoted with shellquote.Join, the exact
inverse of the shellquote.Split that reads it.  Use the same thing on
both sides kids.

Keys containing a newline failed differently: run-mode reads commands
line by line, so a quoted newline split the command and aborted the whole
run with exit 1 and no message. The reader now joins lines while a quote
is open, and reports unterminated input with its line number.

Regression tests cover tab, newline, CR, a literal backslash-t and
space, both as a round-trip unit test and end to end.

aws-sdk-go v1 -> v2
-------------------
v1 reached end of support on 2025-07-31 and is archived. Production code
no longer imports it; it remains only as an indirect requirement of
gofakes3. Four behaviour differences needed handling:

* v2 computes a CRC32 checksum for every upload by default. v1 sent
  none, and many S3-compatible services reject them, so checksums are
  set to WhenRequired.
* manager.GetBucketRegion clears credentials, so private buckets need
  them restored -- but not the anonymous provider, which v2's signer
  rejects outright. It also returns "" with no error when the region
  header is absent, which was blanking the region.
* v2 wraps API errors with the operation, request id and a 100-character
  host id. cleanupError now renders the concise "Code: message status
  code: N" form s5cmd has always printed.
* HeadObject returns a nil metadata map where v1 returned an empty one,
  which changed `head --json` output to "metadata":null.

The transfer manager (feature/s3/manager) is deprecated in favour of
feature/s3/transfermanager, which is still pre-1.0; s5cmd stays on the
stable package, with the deprecation confined to two type aliases.

Dependencies and tooling
------------------------
* go.mod tool directives replace internal/tools/tools.go and its
  //go:build tools tag, which is why staticcheck, unparam and mockgen
  were invisible to `go get -u ./...`. Invoked via `go tool`.
* staticcheck v0.4.7 -> v0.8.1, unparam to current, strcase to v0.3.0
  (test-only deps need `go get -u -t`), plus toml, kr/pretty, x/mod,
  x/exp/typeparams and check.v1.
* extsort's deprecated SortType interface migrated to the Generic API,
  removing the per-item type assertions.
* The e2e harness moved off SDK v1, which surfaced three more
  differences: uploads now carry a Content-Type, metadata keys come back
  lowercased, and Expires is parsed (now read via ExpiresString).
* staticcheck: 34 findings -> 0. unparam: 0.

vendor/ removed
---------------
Versions and integrity are pinned by go.mod and go.sum; vendoring added
no locking, only 27MB of a 29MB checkout -- 5.8MB of it lint tooling
that never ships in the binary. Removing it drops -mod=vendor from six
Makefile targets and from the e2e harness's own `go build`.

CI
--
The matrices tested Go 1.20/1.21/1.22 against a go.mod that already
required 1.25, so they could not have built. Now 1.26, which the tool
directives and the upgraded linters require. goreleaser and the
Dockerfile follow.

The e2e suite now pins its umask: s5cmd creates directories with
os.ModePerm, so ten tests asserting on directory permissions passed
under umask 022 and failed under 002.

User-visible changes
--------------------
* Shell completion is triggered by --generate-shell-completion in v3;
  anyone with an installed completion script must re-run
  --install-completion.
* Building requires Go 1.26 (raised by the linters' own requirements).
* "Incorrect Usage" now goes to stderr rather than stdout.
* Help output differs slightly in flag rendering, per v3's templates.
@lyda
lyda requested review from a team, igungor and sonmezonur and removed request for a team September 8, 2026 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant