Skip to content

SQL exporter kopf handling - #934

Draft
goat-ssh wants to merge 5 commits into
masterfrom
s/sql-exporter-kopf-handling
Draft

goat-ssh wants to merge 5 commits into
masterfrom
s/sql-exporter-kopf-handling

Conversation

@goat-ssh

@goat-ssh goat-ssh commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary of changes

On top of #931

SQL exporter image and configuration were only written when a cluster was created, so version upgrades were manual action.

With new env var SQL_EXPORTER_AUTO_UPDATE enabled, a resume handler updates the
ConfigMap, restarts only the sql-exporter container one pod at a time, and then patches the StatefulSets.
An image that does not start is reverted on the first pod and not retried until the operator restarts.

🤖

Checklist

  • Link to issue this PR refers to:
  • Relevant changes are reflected in CHANGES.rst
  • Added or changed code is covered by tests
  • Documentation has been updated if necessary
  • Changed code does not contain any breaking changes (or this is a major version change)

goat-ssh and others added 4 commits November 12, 2026 18:24
Their only caller, the ConfigMap resume handler, was removed in 0c7478e.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The exporter image and configuration were only written when a cluster was
created, so sql_exporter bumps never reached existing clusters without
manual kubectl work.

With SQL_EXPORTER_AUTO_UPDATE enabled, a resume handler updates the
ConfigMap, restarts only the sql-exporter container one pod at a time, and
then patches the StatefulSets. An image that does not start is reverted on
the first pod and not retried until the operator restarts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@goat-ssh
goat-ssh marked this pull request as draft November 13, 2026 11:28
@goat-ssh
goat-ssh requested a balanced review from Copilot November 16, 2026 08:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Interrupted rollouts can be mistaken for completed updates, bypassing readiness and rollback checks and allowing a broken exporter image to spread to remaining pods and StatefulSets.

4 open issues
What changed in this PR

Adds opt-in SQL exporter updates for existing CrateDB clusters when the operator starts, extending the exporter upgrade guidance from #931.

Changes:

  • Adds a resume handler to update configuration, restart exporter containers sequentially, and patch StatefulSet templates.
  • Tracks configuration hashes and adds rollout, rollback, and integration tests.
  • Documents automatic and manual exporter upgrades.
File Description
tests/​test_sql_exporter.py Adds unit and Kubernetes integration tests for exporter updates.
tests/​test_create.py Accounts for the configuration hash in the ConfigMap.
docs/​source/​upgrading-exporters.rst Documents automatic and manual exporter upgrades.
docs/​source/​index.rst Links the exporter upgrade guide.
docs/​source/​configuration.rst Documents exporter settings and automatic updates.
crate/​operator/​sql_exporter.py Implements configuration updates, sequential restarts, and image rollback.
crate/​operator/​operations.py Removes unused SQL exporter configuration helpers.
crate/​operator/​main.py Registers the opt-in resume handler.
crate/​operator/​create.py Generates configuration hashes and annotates pod templates.
crate/​operator/​constants.py Defines shared exporter names and hash keys.
crate/​operator/​config.py Adds the disabled-by-default automatic update setting.
CHANGES.rst Records automatic updates and upgrade documentation.

Review effort: Balanced

Comment thread crate/operator/sql_exporter.py Outdated
Comment on lines +231 to +232
if not stale_image and not stale_config:
return

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d26f749. While an image change is unverified, the pod carries a sql-exporter-rollout-from annotation with the previous image, written in the same patch as the image. A later run that finds it waits for the exporter to be ready. It then clears the annotation, or reverts to the recorded image and raises PermanentError. The config-hash annotation is now written only after that check.

Comment thread crate/operator/sql_exporter.py Outdated
Comment on lines +219 to +223
or status is None
or pod.metadata.deletion_timestamp is not None
or pod.status.phase != "Running"
):
return

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d26f749. update_pod and update_pods now report a stale pod that is not running yet. After the templates are updated, update_sql_exporter raises TemporaryError while one remains. A pod that already has the new image and hash does not trigger it. One consequence is that a pod stuck Pending keeps requeueing the handler.

Comment on lines +353 to +355
status = _container_status(pod)
if status and status.ready and status.restart_count > restart_count:
return True

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d26f749. _wait_for_restart takes the pod UID. A different UID, or a 404, counts as the pod having been replaced: skip it, no revert, and the second pass handles the replacement.

Comment thread crate/operator/sql_exporter.py Outdated
Comment on lines +354 to +355
if status and status.ready and status.restart_count > restart_count:
return True

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly fixed in d26f749. After the config-projection wait, the pod is read again and the restart baseline and UID come from that read, so an unrelated restart during the wait can no longer satisfy the check. I did not add a check of the running container image against status.image, because kubelet may report it in a normalised form and a strict comparison could cause false failures. A restart between the fresh read and the patch is still possible, though it is a much smaller window.

Record the previous image on the pod while an image change is unverified,
so a resumed run confirms or reverts it instead of treating matching image
and hash as done. Requeue while stale pods are not running yet, refresh the
restart baseline after the config projection wait, and treat a pod with a
new UID as replaced.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@goat-ssh
goat-ssh requested a review from tomach November 16, 2026 10:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants