Skip to content

Drop Python 3.11 support and promote clodius to a core dependency - #119

Merged
conradbzura merged 1 commit into
masterfrom
drop-python-3.11-promote-clodius
Sep 29, 2026
Merged

conradbzura merged 1 commit into
masterfrom
drop-python-3.11-promote-clodius

Conversation

@conradbzura

Copy link
Copy Markdown
Collaborator

Summary

Raise requires-python to >=3.12 and promote clodius (the matrix tile backend, issue #82) from an optional tiles extra to an ordinary core dependency.

The extra existed only to keep Python 3.11 installable: clodius requires >= 3.12, so it carried a python_version >= '3.12' marker and stayed optional rather than force the floor up. The consequence was that every image built from the standard Dockerfile.api/Dockerfile.wool — including what actually deploys to dev and prod — carried no tile backend at all, so /tileset_info and /tiles answered 501 regardless of database content. This surfaced concretely while testing PR #118's tileset endpoints against the dev deployment: the routes were live, but tile serving itself was unavailable because dev's image never carried clodius in the first place.

Every image already builds on python:3.12-slim-bookworm, so this changes nothing about the runtime base — it only drops the promise that a 3.11 interpreter can install cfdb at all, and both the API and worker images now carry the tile backend by default.

Verified: full test suite passes (1479 passed, 3 skipped on unrelated missing-tool conditions, 1 xfailed), and both Dockerfile.api and Dockerfile.wool build successfully from a clean git worktree with clodius installed unconditionally. Re-ran the exact trivy invocation CI uses (severity: HIGH,CRITICAL, ignore-unfixed: true) against the rebuilt worker image, which now carries clodius's full dependency tree (numpy, h5py, cooler, pandas, scipy, pyarrow, etc.) — 0 findings, exit code 0.

Proposed changes

Dependency and version floor

  • pyproject.toml: requires-python = ">=3.12", clodius moved into dependencies unconditionally, the python_version >= '3.12' marker dropped, the 3.11 classifier removed.
  • .github/workflows/test.yml: drop the 3.11 leg from the CI matrix (now ["3.12", "3.13"]).

Retire the now-redundant local-dev scaffolding

Dockerfile.api-tiles and the api-tiles/worker-tiles Makefile targets existed solely to layer clodius onto a local image and share a cache volume with a worker, because make api's image carried no clodius. Both are deleted — make api and make wool now carry the tile backend directly, exactly as anticipated when that scaffolding was written ("Retire both once tiles becomes an ordinary dependency ... make api then covers this on its own"). The README's "Serving tiles locally" section is updated to point at the ordinary uv run local worker pool flow instead.

Remove version-gated test guards

The pytest.importorskip("clodius") / pytest.importorskip("numpy") guards added to keep the (now nonexistent) 3.11 CI leg passing are removed across tests/fixtures/coolers.py, tests/test_tiles.py, tests/test_tilesets/test_backend.py, tests/test_tilesets/test_service.py, tests/test_tilesets_router.py, and tests/test_api/test_lifespan_tilesets.py — they would never skip anything again. The sys.modules-patching tests that simulate a genuinely backendless build (test_should_raise_when_clodius_is_absent, test_lifespan_should_leave_the_service_unset_without_clodius, etc.) are untouched: that degradation path remains real defensive architecture — a broken or partial install — not a version-support artifact, so its coverage stays.

Documentation and error-message accuracy

Updates every place that described clodius as optional or gated on Python 3.11 — the Quickstart install note, the "Matrix tile serving" Limitations bullet (removed outright, since it's no longer a limitation), the /tiles status-code table, and the docstrings/error messages in cfdb.tilesets.backend, cfdb.tilesets.errors, and cfdb.api.main that referenced "the tiles extra" or a "sibling checkout" as the reason a build might lack clodius. They now correctly describe a missing backend as a broken or incomplete install rather than an unconfigured optional feature.

Test cases

# Test Suite Given When Then Coverage Target
1 TestLoadBackend sys.modules patched so both clodius modules the gate imports from raise ImportError, simulating a broken or incomplete install of the now-core clodius dependency load_backend is called It raises TileBackendUnavailable, chained from the underlying ImportError Backendless-build degradation stays real defensive coverage, independent of Python version
2 TestIsTileError A loaded backend and a real clodius error instance (no environment gating — clodius is always present) is_tile_error is called It classifies correctly per the existing TilesetError hierarchy Real-backend classification no longer needs a version-conditional skip
3 Full suite (uv run pytest) The standard dev extra synced with no tiles-specific step Every test module is collected and run 1479 passed, 3 skipped on unrelated missing-tool conditions (gffread, bedToBigBed), 1 xfailed, 0 failed No regression from removing the 3.11-era import guards
4 Docker build (Dockerfile.api, Dockerfile.wool) A clean git worktree at this branch's tip docker build --platform linux/amd64 runs for both files Both complete successfully, with clodius/numpy/cooler/h5py installed via the plain pip install . step The standard image-build path now carries the tile backend with no extra flags
5 trivy scan (cfdb-wool) The rebuilt worker image, now carrying clodius's full dependency tree trivy image --severity HIGH,CRITICAL --ignore-unfixed, matching CI's exact invocation 0 findings, exit code 0 Promoting clodius introduces no new HIGH/CRITICAL vulnerability surface

The tiles extra existed only to keep 3.11 installable: clodius
(matrix tile serving, issue #82) requires Python >= 3.12, so it
carried a python_version marker and stayed optional rather than
force the floor up. That meant every image built from the standard
Dockerfiles -- including what actually deploys to dev/prod -- carried
no tile backend at all, so the tile routes always answered 501
regardless of database content.

Raise requires-python to >=3.12 and move clodius into dependencies
unconditionally. Every image already builds on python:3.12-slim-bookworm,
so this changes nothing about the runtime base -- it only drops the
promise that a 3.11 interpreter can install cfdb, and both the API and
worker images now carry the tile backend by default.

Retires Dockerfile.api-tiles and the api-tiles/worker-tiles Makefile
scaffolding that existed solely to work around clodius being optional
locally, exactly as anticipated when that scaffolding was written --
`make api` and `make wool` cover it now. Drops the 3.11 leg from the
CI test matrix, and removes the pytest.importorskip("clodius")/("numpy")
guards added to keep the (now nonexistent) 3.11 CI leg passing; the
sys.modules-patching tests that simulate a genuinely backendless build
are untouched, since that degradation path remains real architecture,
not a version-support artifact.
@conradbzura conradbzura self-assigned this Sep 29, 2026
@conradbzura
conradbzura marked this pull request as ready for review September 29, 2026 16:35
@conradbzura
conradbzura merged commit 8df827e into master Sep 29, 2026
3 checks passed
conradbzura added a commit that referenced this pull request Sep 29, 2026
pyproject.toml's [tool.uv.sources] pins clodius to a commit on the
abdenlab/clodius fork -- the public PyPI package is a different
codebase entirely, with no tiles_v2 subpackage. That pin is a uv-only
concept: bare `pip install .`, which both Dockerfiles used, has no
idea it exists and silently resolves `clodius` from PyPI instead.

This went undetected through both the build and the trivy scan --
`pip install .` succeeds either way, since the wrong package installs
cleanly and only fails at import (`ModuleNotFoundError: No module
named 'clodius.core'`) -- and shipped in the PR #119 merge that
promoted clodius to a core dependency. The deployed dev image reported
`clodius tile backend` unavailable on every tile route despite the
build logs showing clodius installed successfully, because it was
installing and then failing to import the wrong package the whole
time.

Switch both Dockerfiles to `uv pip install --system`, which resolves
[tool.uv.sources] the same way `uv sync` does, and add `git` to both
images' apt install (uv needs it to fetch the pinned commit).

Verified by actually running the built images and importing clodius
as the app runtime would -- the gap the previous verification missed,
since `uv sync`/`uv run` locally and a bare `docker build` both hid
this: the former because it correctly reads [tool.uv.sources], the
latter because it only checks that pip install succeeds, not which
package it installed. `cfdb.tilesets.backend.is_available()` now
returns True inside both freshly built images. Re-ran the exact CI
trivy invocation against the worker image with git added: 0
HIGH/CRITICAL, exit code 0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant