Skip to content

Bug: sam local invoke unconditionally rebuilds docker image on every invocation when a layer is declared #9255

Description

@marcoreni

Description:

sam local invoke unconditionally rebuilds the Docker image on every single invocation when the function has a declared layer in the template (AWS::Serverless::LayerVersion with a local ContentUri).

Root cause (traced in source, v1.165.0):

  • samcli/local/docker/lambda_image.py:276-281 forces _build_image() whenever any(layer.is_defined_within_template for layer in downloaded_layers) is true — no hash/content comparison, this is a blanket condition.
  • samcli/lib/providers/provider.py:301 sets is_defined_within_template = bool(codeuri) — true for any layer built locally, unconditionally, regardless of whether its content actually changed since the last build/invoke.
  • The rebuild itself (_build_image → create_tarball, samcli/lib/utils/tar.py:51-53) re-tars the entire layer directory via Python's tarfile, one file at a time, before Docker's build API/cache is ever reached. On our layer (~30MB, ~5300 files under node_modules) this alone costs 60-90s, every call.

SAM CLI already has the infrastructure to avoid this: sam build itself decides whether to reinstall a layer's dependencies by hashing the manifest (package.json/lockfile) via DependencyHashGenerator in samcli/lib/build/build_strategy.py, not by hashing/re-reading the installed node_modules tree. lambda_image.py's invoke-time rebuild check does not reuse this at all.

Steps to reproduce:

  1. Template with a function that has Layers: [!Ref DepsLayer], where DepsLayer is AWS::Serverless::LayerVersion with a local ContentUri and Metadata: BuildMethod: makefile (i.e. built locally, not referencing a published/remote layer ARN).
  2. sam build
  3. sam local invoke MyFunction -e event.json --debug
  4. Immediately run the exact same command again, with no changes to any file.
  5. Both invocations pay the same "Building image" cost.

Observed result:

Two consecutive invokes, same template, same build, zero changes in between:

=== invoke 1 ===
2026-09-04 19:31:57,863 | Local image is up-to-date
2026-09-04 19:33:24,098 | Mounting /path/.aws-sam/build/MyLambda as /var/task:ro,delegated, inside runtime container
sam local invoke MyLambda -e event.json --debug 1:31.82 total

=== invoke 2 ===
2026-09-04 19:33:29,058 | Local image is up-to-date
2026-09-04 19:34:29,438 | Mounting /path/.aws-sam/build/MyLambda as /var/task:ro,delegated, inside runtime container
sam local invoke MyLambda -e event.json --debug 1:05.05 total

86s and 60s respectively spent between "Local image is up-to-date" and "Mounting" — the silent image-rebuild window — on both calls, despite nothing having changed between them.

Expected result:

sam local invoke should reuse the existing rapid image as much as possible — the same way SAM already skips reinstalling a layer's dependencies during sam build when its manifest hash is unchanged.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: macOS 26.6.2 (arm64)
  2. sam --version: SAM CLI, version 1.165.0

{
"version": "1.165.0",
"system": {
"python": "3.14.7",
"os": "macOS-26.6.2-arm64-arm-64bit-Mach-O"
},
"additional_dependencies": {
"container_engine": "Docker(v29.7.2)",
"aws_cdk": "Not available",
"terraform": "1.7.4"
}
}

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions