clearscale

clearscale is a small dependency-free Python package for clear multiscale image metadata manipulation.

Features

• Zero dependencies, pure Python
• Compatible with Python 3.10+
• Read OME-Zarr metadata and even Neuroglancer Precomputed (very basic support)
• Read all OME-Zarr versions (0.1 to 0.6) using one code path - no version branching, future versions come for free
• Write OME-Zarr versions 0.4, 0.5 and 0.6
• Virtually no knowledge about OME-Zarr required
• Create, manipulate, save and load multiscale metadata
• Make sure metadata are correct for any scaling method
• Find out how exactly your scaling method resamples with characterization
• Read and write multiscale transformations into alternate coordinate systems
• Read and write transformations between multiscales using scenes

Minimal OME-Zarr example

The absolute minimum requirements for writing OME-Zarr:

• axis keys ("zyx")
• writing the data to a zarr array inside a zarr group (and recording the array sub-path)
• zarr_format=3 for OME-Zarr version="0.5"

# Assuming you have numpy and zarr-python 3.* installed
import numpy as np
import zarr
import clearscale as cs

# You have some array
image = np.random.random((128, 1024, 1024)).astype(np.float32)

# 1. Data: Make a group and write the array into it
# (no clearscale involved)
group = zarr.open_group(
    "demo-output/example1.ome.zarr", 
    mode="w", 
    zarr_format=3
)
array_path = "s0"
group.create_array(array_path, data=image)

# 2. Metadata: Scale -> Multiscale -> Group -> attrs
group.attrs.update(
    cs.OmeZarrGroup
    .from_single(
        cs.Multiscale.from_single(
            scale=cs.Scale.from_lists("zyx"), 
            scale_key=array_path
        )
    )
    .to_attrs(version="0.5")
)

Zero dependencies

Being dependency-free means:

• clearscale fits in any Python environment. Installing or uninstalling it will not modify any other installed packages.
• clearscale does not interact with data (numpy/zarr arrays). You implement data processing according to your needs, and use clearscale to generate metadata that describes your processing.

This provides full flexibility for data processing, but it comes at the risk of desynchronization between data and metadata.

You can mitigate this by coupling data processing to the metadata, for example by using the generated metadata as the parameter for image scaling and data writing:

import clearscale as cs

# 1. Define operation plan as metadata
shape = cs.Shape(zip("tzyx", my_data.shape))
scaling_blueprint = cs.BlueprintShapes.downscale_powers_of_2_xyz(
    base_shape=shape,
    rounding="ceil",
    limit_all=cs.Shape(z=8, y=128, x=128),
)

# 2. Scale data according to the blueprint.
# Metadata as parameter for data processing.
for scale_key, target_shape in scaling_blueprint.items():
    # Target shape from blueprint
    scaled_data = do_my_scaling(my_data, target_shape.values_tuple())
    # Target array path from blueprint
    zarr_group.create_array(scale_key, data=scaled_data)

# 3. Write metadata: Scale -> Multiscale -> Group -> attrs
pixel_size = cs.PixelSize(t=5.0, z=260.0, y=0.53, x=0.53)
unit       = cs.Unit(t="s", z="micrometer", y="micrometer", x="micrometer")
base       = cs.Scale(shape, pixel_size, unit)

ms         = cs.Multiscale.from_single(base, blueprint=scaling_blueprint)
group_meta = cs.OmeZarrGroup.from_single(ms).to_attrs(version="0.6")
zarr_group.attrs.update(group_meta)

For example, if you use numpy, scikit-image and zarr-python, the placeholders above could look like:

import os, numpy, skimage, zarr

my_data = numpy.random.rand(3, 12, 512, 512)
do_my_scaling = skimage.transform.resize
zarr_group = zarr.open_group(os.path.expanduser("demo-output/sweet.ome.zarr"), mode="w")

# Now you can actually run the example snippet above and produce a valid `sweet.ome.zarr`

Install

clearscale is considered in "beta" status for a short time, with 1.0 release intended soon(TM), when we have some feedback from first users.

Please try it and let us know if you encounter anything unclear!

Install the 0.9.0 release candidate from PyPI:

python -m pip install clearscale==0.9.0rc1

Or add to your conda env.yaml:

dependencies:
  - pip
  - pip:
      - clearscale==0.9.0rc1

Note that until we publish 1.0, the API is considered subject to change so we can ensure it "solidifies" with as much convenience and robustness as possible. Though we hope there won't be too many changes necessary at this point :)

Documentation

Rendered guides (OME-Zarr how-to, core concepts, metadata scaling, ...) and automatically generated API Reference are in the clearscale documentation.

The handwritten guides are also directly under docs/ in this repository.

End-to-end examples

Downsample a numpy array and save it as OME-Zarr

The previous example used a "metadata-first" approach (first compute metadata blueprint, then scale data according to it).

This example works the other way round (first scale data, then record what was done in blueprint).

# This example assumes numpy, scikit-image and zarr-python 3.* are installed
import numpy as np
import zarr
from skimage.transform import pyramid_gaussian

import clearscale as cs

# 1. Generate a pyramid
image = np.random.random((128, 1024, 1024)).astype(np.float32)
pyramid = [level.astype(np.float32) for level in pyramid_gaussian(image, downscale=2)]

# 2. Write arrays to zarr on disk and record scaled shapes
group = zarr.open_group("demo-output/example2.ome.zarr", mode="w")
scaled_shapes = []
for i, level in enumerate(pyramid):
    scale_key = f"s{i}"
    group.create_array(scale_key, data=level)
    scaled_shapes.append(
        (
            scale_key,
            cs.Shape(zip("zyx", level.shape))
        )
    )

# 3. Describe the full-resolution image
base = cs.Scale.from_lists(
    keys="zyx",
    shape=image.shape,
    pixel_size=[25, 240, 240],
    unit=["micrometer", "nanometer", "nanometer"],
)

# 4. Use the recorded scale shapes as a blueprint to expand a Multiscale
blueprint = cs.BlueprintShapes(scaled_shapes)
multiscale = cs.Multiscale.from_single(base, blueprint=blueprint)

# 5. Save OME-Zarr metadata
group_meta = cs.OmeZarrGroup.from_single(multiscale).to_attrs(version="0.5")
group.attrs.update(group_meta)

Download a single scale of a public OME-Zarr as a valid local OME-Zarr

from pathlib import Path
import zarr  # This example assumes zarr-python 3.* is installed (plus fsspec, requests and aiohttp for https reading)
import clearscale as cs

URL = "https://s3.embl.de/i2k-2020/platy-raw.ome.zarr"
SCALE_KEY = "s6"
LOCAL_PATH = Path(f"demo-output/platy-raw-{SCALE_KEY}.ome.zarr")

# 1. Discover what's on the server
remote_group = zarr.open_group(URL)

remote_meta = cs.OmeZarrGroup.from_group(remote_group)
assert remote_meta.kind is cs.GroupKind.MULTISCALE, "This tells us there is exactly one Multiscale in this group"
assert remote_meta.version == "0.4", "OME-Zarr version is a property of the group, not of individual Multiscales"

remote_multiscale = remote_meta.multiscales[0]
assert SCALE_KEY in remote_multiscale, "Multiscale is dict-like, with the sub-paths to arrays as keys"
assert remote_multiscale[SCALE_KEY].shape == {"z": 357, "y": 405, "x": 430}

# 2. Create the local target and download the data
source_array = remote_group[SCALE_KEY]
local_group = zarr.open_group(str(LOCAL_PATH), mode="w", zarr_format=2)  # OME-Zarr v0.4 must use zarr-format v2
print(f"Downloading {SCALE_KEY} data...")
local_array = local_group.create_array(SCALE_KEY, data=source_array)

# 3. Derive a new Multiscale from the chosen Scale entry in the existing Multiscale
local_multiscale = remote_multiscale.derive(SCALE_KEY)

# 4. Write the new metadata to the downloaded store
local_group_meta = cs.OmeZarrGroup.from_single(local_multiscale)
local_group.attrs.update(local_group_meta.to_attrs(version="0.4"))

Why clearscale?

The motivation behind clearscale is to make upgrading from

My tool/script does zarr

to

My tool/script does OME-Zarr

as easy as possible.

Why not use one of the existing libraries?

You should probably use one of the existing libraries, if they work for you :)

Check out the NGFF Tools page for some examples. Python libraries include e.g. bioio, ngff-zarr, ngio, ome-zarr-py, and yaozarrs. Several of these handle not only the metadata, but also data manipulation all in one.

Dependencies are hard

But maybe you can't, or don't want to, use any of the above.

• Technical reasons (dependency conflicts)
• Legal reasons (licensing conflicts, commercial use)
• You already have data processing in place that you don't want to reimplement or risk adapting inefficiently.
• You just want to minimise bloat.

clearscale is tiny, dependency-free, permissively licensed, and works around your existing data processing.

Zarr is complex enough

There are several backend packages your application or script might use for reading and writing zarr data. Implementing data processing in an efficient, chunk-wise (shard-wise) manner is complex on its own. You shouldn't need to learn the OME metadata specification on top to make your datasets accessible to others, or interoperable with other tools.

With clearscale, no matter how you handle zarr data, handling multiscale metadata like OME-Zarr looks the same.

Custom applications

There are thousands of ways to scale or otherwise transform an image. Maybe none of the existing libraries that can write OME-Zarr implements quite what you need, so you have to roll your own.

clearscale isn't tied to data processing. Whatever you're doing, clearscale can simplify metadata manipulation.

Should I use clearscale in my project right now?

If you're willing to adapt to future changes in the API: Yes, please do try it!

If you are looking for a stable API, please wait until the first formal release to PyPI. After that point, API changes will follow semver and deprecations will be avoided or give appropriate notice through DeprecationWarnings.

Contributing / Feature scope

Any form of feedback, issues, bug reports, feature requests, pull requests and other contributions are welcome! See CONTRIBUTING.md.

License

Licensed under either the MIT license or the Apache License, Version 2.0, at your option.