Add PRR collection generation; point OSC item assets to PRR (v0.2.0)

.github/workflows/unittest-workflow.yaml

@@ -25,7 +25,8 @@ jobs:
 
       - name: Lint with ruff
         run: |
-            pip install ruff
+            # TODO upgrade to latest later
+            pip install ruff==0.9.10
             ruff check
 
       - name: Run unit tests

CHANGES.md

@@ -85,8 +85,27 @@
 - Removed redundant `via` access link from the OSC STAC collection; access is already expressed via typed assets (`zarr-data`, `zarr-consolidated-metadata`) on the STAC item.
 - `osc_project` is now omitted from `OscDatasetStacGenerator` when not provided, preserving the callee's default instead of passing `None`.
 
-## Changes in 0.1.10 (in Development)
+## Changes in 0.2.0 (in Development)
 
 - Generated project collection now includes required STAC extensions (`osc`, `themes`, `contacts`) and OSC-mandatory fields (`osc:type`, `osc:status`, `themes`, `contacts`) to pass OSC catalog validation.
 - Added optional `osc_project_url` field to the dataset config; used as the `via` link in the project collection. Falls back to `documentation_link` if omitted; defaults to the existing DeepESDL project collection when neither is provided.
 - `dataset_status` now defaults to `"ongoing"` when not specified in the dataset config.
+- Added a new CLI command `generate-prr-collection` that writes a Project Results Repository (PRR) STAC collection as local files, ready for submission to the ESA EarthCODE PRR endpoint. It reuses the dataset config and needs no GitHub credentials or S3 write access. See [PRR collection specification](https://eoresults.esa.int/prr_collection_specifications.html).
+- The PRR command produces a self-contained `Collection → Item → Assets` tree: the Item carries the `datacube` extension (`cube:dimensions` / `cube:variables` extracted from the Zarr) with `zarr-data` and `zarr-consolidated-metadata` assets, and the Collection declares the OSC, Scientific, Processing, Themes and CF extensions.
+- Added PRR-specific dataset config fields: `osc_initiative` (default `earthcode`), `osc_missions`, `osc_contract_number`, `osc_project_website`, `osc_project_description`, `thumbnail`, `thumbnail_media_type`, `sci_doi`, `sci_citation`, and `prr_output_dir`. Missing PRR-required fields produce a warning rather than a failure.
+- `generate-config` templates now document the PRR fields under a dedicated section.
+- `stac_catalog_s3_root` is optional again. By default the OSC collection now links to the dataset's collection in PRR (`child` → `https://eoresults.esa.int/stac/collections/{collection_id}`, `via` "Access" → the PRR STAC browser), matching how other PRR-hosted OSC products are linked; nothing is written to S3 and no S3 credentials are needed. PRR ingestion must happen before OSC publishing: publishing fails with a descriptive error if the PRR collection does not exist. Set `stac_catalog_s3_root` to publish and link a deep-code STAC catalog on S3 instead, for datasets not in PRR.
+- The S3-hosted STAC item's `zarr-data` and `zarr-consolidated-metadata` assets now point to an absolute Zarr URL: the new optional `access_link` config field, or else the `zarr-data` asset looked up from the dataset's PRR item. Previously the href was relative (`./{dataset_id}`) and resolved to a non-existent path next to the item on S3.
+- Fixed the S3-hosted STAC item's `self` link to match the path it is written to (`{stac_catalog_s3_root}/{collection_id}/items/{item_id}.json` instead of `.../{collection_id}/item.json`).
+- Renamed the dataset config field `dataset_status` to `osc_status`, now defaulting to `"completed"` (used by both `publish` and `generate-prr-collection`). The old `dataset_status` key is still read as a fallback but logs a deprecation warning; rename it in existing dataset configs.
+- `publish` and `generate-prr-collection` now report a clear error when an `items_config` entry is missing `dataset_id` or `item_id` (e.g. `dataset_id is required in items_config entry 0.`), instead of a bare `KeyError`.
+- `dataset_id` in `items_config` can now be the `https://` URL of a consolidated Zarr store (e.g. the copy served by PRR), so `publish` works for datasets that are not in a DeepESDL S3 bucket. `publish` no longer computes the Zarr store's file sizes, which it did not use and which required listing every chunk of the store.
+- Fixed `publish` crashing with `AttributeError: 'OscDatasetStacGenerator' object has no attribute 'variables_metadata'` when a dataset introduces a variable that has no OSC variable catalog yet. Variable metadata is now extracted once, so missing GCMD keyword URLs are prompted for only once.
+- Workflow and experiment records now link to the project given by `osc_project`, instead of always linking to the DeepESDL project.
+- Fixed `publish` overwriting upstream OSC catalogs (e.g. `products/catalog.json`) with the stale copies from an out-of-date fork. The fork is now synced with upstream before existing catalogs are read.
+- `publish` now updates the existing `self` link of `workflows/catalog.json` and `experiments/catalog.json` in place, instead of moving it to the end of the links list.
+- Fixed OSC validation failures when `publish` creates a new project collection:
+  - The project's title (`osc_project_title`) is now used verbatim for the project collection and for every link to it (projects catalog, product collection, workflow and experiment records). If the project already exists in OSC, its existing title is used. Before, the title was reformatted differently in each place (e.g. `ARNETLAB` became `Arnetlab`).
+  - New optional dataset config field `osc_project_contacts` sets the project collection's contacts. OSC requires a contact with role `technical_officer` and an email; `publish` warns if it is missing.
+  - The project collection now has `related` links to its themes and declares the contacts extension v0.1.1, as OSC requires.
+- Extra `links` from the workflow config that OSC does not allow on experiment records (`root`, `parent`, `child` or `related` links whose type is not `application/json`) are now added to the workflow record only.

deep_code/cli/generate_config.py

@@ -18,5 +18,5 @@
     help="Output directory for templates",
 )
 def generate_config(output_dir):
-    TemplateGenerator.generate_workflow_template(f"{output_dir}/workflow_config.yaml")
-    TemplateGenerator.generate_dataset_template(f"{output_dir}/dataset_config.yaml")
+    TemplateGenerator.generate_workflow_template(f"{output_dir}/workflow.yaml")
+    TemplateGenerator.generate_dataset_template(f"{output_dir}/dataset.yaml")

deep_code/cli/main.py

@@ -8,18 +8,19 @@
 
 from deep_code.cli.generate_config import generate_config
 from deep_code.cli.lint import lint_dataset
+from deep_code.cli.prr import generate_prr_collection_cmd
 from deep_code.cli.publish import publish
 
 
 @click.group()
 def main():
     """Deep Code CLI."""
-    pass
 
 
 main.add_command(publish)
 main.add_command(generate_config)
 main.add_command(lint_dataset)
+main.add_command(generate_prr_collection_cmd)
 
 if __name__ == "__main__":
     main()

deep_code/cli/prr.py

@@ -0,0 +1,34 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+import click
+
+from deep_code.tools.prr import generate_prr_collection
+
+
+@click.command(name="generate-prr-collection")
+@click.argument("dataset_config", type=click.Path(exists=True))
+@click.option(
+    "--output-dir",
+    "-o",
+    type=click.Path(file_okay=False),
+    default=None,
+    help=(
+        "Directory to write the PRR collection tree into "
+        "(default: 'prr_output_dir' from the config, else 'prr/<collection_id>')."
+    ),
+)
+def generate_prr_collection_cmd(dataset_config, output_dir):
+    """Generate a PRR-style STAC Collection as local files.
+
+    Reads a dataset config (the same YAML used by 'publish') and writes a
+    self-contained Collection -> Item -> Assets tree for submission to the
+    ESA EarthCODE Project Results Repository.
+
+    Example:
+      deep-code generate-prr-collection dataset-config.yaml -o ./prr
+    """
+    out_dir = generate_prr_collection(dataset_config, output_dir=output_dir)
+    click.echo(f"PRR collection written to: {out_dir}")

deep_code/constants.py

@@ -7,7 +7,7 @@
 OSC_SCHEMA_URI = "https://stac-extensions.github.io/osc/v1.0.0/schema.json"
 CF_SCHEMA_URI = "https://stac-extensions.github.io/cf/v0.2.0/schema.json"
 THEMES_SCHEMA_URI = "https://stac-extensions.github.io/themes/v1.0.0/schema.json"
-CONTACTS_SCHEMA_URI = "https://stac-extensions.github.io/contacts/v0.1.0/schema.json"
+CONTACTS_SCHEMA_URI = "https://stac-extensions.github.io/contacts/v0.1.1/schema.json"
 OSC_THEME_SCHEME = "https://github.com/stac-extensions/osc#theme"
 OSC_REPO_OWNER = "ESA-EarthCODE"
 OSC_REPO_NAME = "open-science-catalog-metadata"
@@ -41,3 +41,17 @@
     "https://stac-extensions.github.io/application/v0.1.0/schema.json"
 )
 ZARR_MEDIA_TYPE = "application/vnd+zarr"
+DATACUBE_SCHEMA_URI = "https://stac-extensions.github.io/datacube/v2.2.0/schema.json"
+FILE_SCHEMA_URI = "https://stac-extensions.github.io/file/v2.1.0/schema.json"
+PROCESSING_SCHEMA_URI = (
+    "https://stac-extensions.github.io/processing/v1.2.0/schema.json"
+)
+SCIENTIFIC_SCHEMA_URI = (
+    "https://stac-extensions.github.io/scientific/v1.0.0/schema.json"
+)
+# STAC API of the ESA Project Results Repository (PRR). The OSC item points its
+# assets at the Zarr store PRR serves, looked up from this API.
+PRR_STAC_API_ROOT = "https://eoresults.esa.int/stac"
+PRR_STAC_BROWSER_ROOT = (
+    "https://eoresults.esa.int/browser/#/external/eoresults.esa.int/stac"
+)

deep_code/tests/tools/test_new.py

@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+import os
+import tempfile
+import unittest
+
+
+from deep_code.tools.new import TemplateGenerator
+
+
+class TestTemplateGenerator(unittest.TestCase):
+    def test_generate_dataset_template_writes_file(self):
+        with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f:
+            path = f.name
+        try:
+            TemplateGenerator.generate_dataset_template(output_path=path)
+            with open(path) as f:
+                content = f.read()
+        finally:
+            os.unlink(path)
+
+        # Required fields present
+        self.assertIn("dataset_id", content)
+        self.assertIn("collection_id", content)
+        self.assertIn("license_type", content)
+        self.assertIn("stac_catalog_s3_root", content)
+        # Optional fields present
+        self.assertIn("osc_themes", content)
+        self.assertIn("visualisation_link", content)
+        self.assertIn("osc_project_title", content)
+        # Section headers
+        self.assertIn("REQUIRED", content)
+        self.assertIn("OPTIONAL", content)
+
+    def test_generate_workflow_template_writes_file(self):
+        with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f:
+            path = f.name
+        try:
+            TemplateGenerator.generate_workflow_template(output_path=path)
+            with open(path) as f:
+                content = f.read()
+        finally:
+            os.unlink(path)
+
+        # Required fields present
+        self.assertIn("workflow_id", content)
+        self.assertIn("title", content)
+        self.assertIn("license", content)
+        # Optional fields present
+        self.assertIn("jupyter_notebook_url", content)
+        self.assertIn("contact", content)
+        # Section headers
+        self.assertIn("REQUIRED", content)
+        self.assertIn("OPTIONAL", content)
+
+    def test_generate_dataset_template_no_path_returns_none(self):
+        result = TemplateGenerator.generate_dataset_template()
+        self.assertIsNone(result)
+
+    def test_generate_workflow_template_no_path_returns_none(self):
+        result = TemplateGenerator.generate_workflow_template()
+        self.assertIsNone(result)

deep_code/tests/tools/test_prr.py

@@ -0,0 +1,200 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+import json
+import os
+import tempfile
+import unittest
+from datetime import datetime
+from unittest.mock import patch
+
+import numpy as np
+import yaml
+from xarray import Dataset
+
+from deep_code.tools.prr import generate_prr_collection
+
+
+def _make_dataset():
+    return Dataset(
+        coords={
+            "lon": ("lon", np.linspace(-20, 20, 4)),
+            "lat": ("lat", np.linspace(-10, 10, 3)),
+            "time": ("time", [np.datetime64(datetime(2021, 1, 1), "ns")]),
+        },
+        data_vars={
+            "sst": (
+                ("time", "lat", "lon"),
+                np.random.rand(1, 3, 4),
+                {"units": "K", "long_name": "Sea surface temperature"},
+            )
+        },
+        attrs={
+            "description": "PRR tool test cube",
+            # set by open_dataset from the store's file sizes
+            "size": 123456,
+            "metadata_size": 123,
+        },
+    )
+
+
+class TestGeneratePrrCollection(unittest.TestCase):
+    def _write_config(self, tmp, **overrides):
+        config = {
+            "collection_id": "tool-prr",
+            "items_config": [{"dataset_id": "test.zarr", "item_id": "tool-item"}],
+            "license_type": "CC-BY-4.0",
+            "osc_project": "tool-project",
+            "osc_project_url": "https://project.example.org",
+            "osc_status": "ongoing",
+            "osc_region": "Global",
+            "osc_themes": ["oceans"],
+            "documentation_link": "https://example.org",
+        }
+        config.update(overrides)
+        path = os.path.join(tmp, "dataset-config.yaml")
+        with open(path, "w") as f:
+            yaml.safe_dump(config, f)
+        return path
+
+    @patch(
+        "deep_code.utils.dataset_stac_generator.open_dataset",
+        side_effect=lambda *args, **kwargs: _make_dataset(),
+    )
+    def test_writes_collection_tree(self, _mock_open_ds):
+        with tempfile.TemporaryDirectory() as tmp:
+            config_path = self._write_config(tmp)
+            out = os.path.join(tmp, "out")
+            result = generate_prr_collection(config_path, output_dir=out)
+
+            self.assertEqual(result, out)
+            coll_path = os.path.join(out, "tool-prr", "collection.json")
+            item_path = os.path.join(out, "tool-prr", "items", "tool-item.json")
+            self.assertTrue(os.path.isfile(coll_path))
+            self.assertTrue(os.path.isfile(item_path))
+            with open(coll_path) as f:
+                coll = json.load(f)
+            self.assertEqual(coll["id"], "tool-prr")
+            self.assertEqual(coll["type"], "Collection")
+            self.assertEqual(coll["osc:type"], "product")
+            self.assertEqual(coll["osc:status"], "ongoing")
+            with open(item_path) as f:
+                item = json.load(f)
+            # PRR ingests the Zarr next to the item, so asset hrefs are relative.
+            self.assertEqual(item["assets"]["zarr-data"]["href"], "./test.zarr")
+
+    @patch(
+        "deep_code.utils.dataset_stac_generator.open_dataset",
+        side_effect=lambda *args, **kwargs: _make_dataset(),
+    )
+    def test_default_output_dir_from_collection_id(self, _mock_open_ds):
+        with tempfile.TemporaryDirectory() as tmp:
+            config_path = self._write_config(tmp)
+            cwd = os.getcwd()
+            os.chdir(tmp)
+            try:
+                out = generate_prr_collection(config_path)
+                self.assertEqual(out, "prr")
+                self.assertTrue(
+                    os.path.isfile(os.path.join(out, "tool-prr", "collection.json"))
+                )
+            finally:
+                os.chdir(cwd)
+
+    @patch(
+        "deep_code.utils.dataset_stac_generator.open_dataset",
+        side_effect=lambda *args, **kwargs: _make_dataset(),
+    )
+    def test_output_dir_from_config(self, _mock_open_ds):
+        with tempfile.TemporaryDirectory() as tmp:
+            out = os.path.join(tmp, "from-config")
+            config_path = self._write_config(tmp, prr_output_dir=out)
+            result = generate_prr_collection(config_path)
+            self.assertEqual(result, out)
+            self.assertTrue(
+                os.path.isfile(os.path.join(out, "tool-prr", "collection.json"))
+            )
+
+    @patch(
+        "deep_code.utils.dataset_stac_generator.open_dataset",
+        side_effect=lambda *args, **kwargs: _make_dataset(),
+    )
+    def test_prr_spec_fields_plumbed_from_config(self, _mock_open_ds):
+        with tempfile.TemporaryDirectory() as tmp:
+            out = os.path.join(tmp, "out")
+            config_path = self._write_config(
+                tmp,
+                osc_missions=["sentinel-3"],
+                osc_project_description="Project description.",
+                osc_project_website="https://project.example.org",
+                osc_contract_number="4000114410/15/NL/BW",
+                thumbnail="https://example.org/logo.jpeg",
+                sci_doi="10.1000/xyz123",
+            )
+            generate_prr_collection(config_path, output_dir=out)
+            with open(os.path.join(out, "tool-prr", "collection.json")) as f:
+                coll = json.load(f)
+            self.assertEqual(coll["osc:initiative"], "earthcode")
+            self.assertEqual(coll["osc:project_website"], "https://project.example.org")
+            self.assertEqual(coll["osc:project_description"], "Project description.")
+            self.assertEqual(coll["osc:contract-number"], "4000114410/15/NL/BW")
+            self.assertEqual(coll["sci:doi"], "10.1000/xyz123")
+            self.assertIn("thumbnail", coll["assets"])
+            self.assertEqual(coll["assets"]["thumbnail"]["roles"], ["thumbnail"])
+
+    @patch(
+        "deep_code.utils.dataset_stac_generator.open_dataset",
+        side_effect=lambda *args, **kwargs: _make_dataset(),
+    )
+    def test_deprecated_dataset_status(self, _mock_open_ds):
+        with tempfile.TemporaryDirectory() as tmp:
+            out = os.path.join(tmp, "out")
+            config_path = self._write_config(tmp, osc_status=None)
+            with open(config_path) as f:
+                config = yaml.safe_load(f)
+            del config["osc_status"]
+            config["dataset_status"] = "planned"
+            with open(config_path, "w") as f:
+                yaml.safe_dump(config, f)
+            generate_prr_collection(config_path, output_dir=out)
+            with open(os.path.join(out, "tool-prr", "collection.json")) as f:
+                coll = json.load(f)
+            self.assertEqual(coll["osc:status"], "planned")
+
+    def _assert_missing_raises(self, field: str):
+        with tempfile.TemporaryDirectory() as tmp:
+            config_path = self._write_config(tmp)
+            with open(config_path) as f:
+                config = yaml.safe_load(f)
+            del config[field]
+            with open(config_path, "w") as f:
+                yaml.safe_dump(config, f)
+            with self.assertRaisesRegex(ValueError, f"{field} is required"):
+                generate_prr_collection(config_path)
+
+    def test_items_config_entry_missing_dataset_id_raises(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            config_path = self._write_config(
+                tmp, items_config=[{"item_id": "tool-item"}]
+            )
+            with self.assertRaisesRegex(
+                ValueError, "dataset_id is required in items_config entry 0"
+            ):
+                generate_prr_collection(config_path)
+
+    def test_missing_required_fields_raise(self):
+        for field in (
+            "collection_id",
+            "osc_project",
+            "osc_project_url",
+            "license_type",
+            "items_config",
+        ):
+            with self.subTest(field=field):
+                self._assert_missing_raises(field)
+
+
+if __name__ == "__main__":
+    unittest.main()