Repository navigation
Conversation
Their only caller, the ConfigMap resume handler, was removed in 0c7478e. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The exporter image and configuration were only written when a cluster was created, so sql_exporter bumps never reached existing clusters without manual kubectl work. With SQL_EXPORTER_AUTO_UPDATE enabled, a resume handler updates the ConfigMap, restarts only the sql-exporter container one pod at a time, and then patches the StatefulSets. An image that does not start is reverted on the first pod and not retried until the operator restarts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Interrupted rollouts can be mistaken for completed updates, bypassing readiness and rollback checks and allowing a broken exporter image to spread to remaining pods and StatefulSets.
4 open issues
Matching image and hash can skip rollout readiness and rollback checks · New Pending stale pods are skipped, leaving the cluster unreconciled · New Replacement pod UID can cause a false restart timeout and rollback · New Unrelated restart can satisfy the baseline without verifying the new image · New
What changed in this PR
Adds opt-in SQL exporter updates for existing CrateDB clusters when the operator starts, extending the exporter upgrade guidance from #931.
Changes:
- Adds a resume handler to update configuration, restart exporter containers sequentially, and patch StatefulSet templates.
- Tracks configuration hashes and adds rollout, rollback, and integration tests.
- Documents automatic and manual exporter upgrades.
| File | Description |
|---|---|
tests/test_sql_exporter.py |
Adds unit and Kubernetes integration tests for exporter updates. |
tests/test_create.py |
Accounts for the configuration hash in the ConfigMap. |
docs/source/upgrading-exporters.rst |
Documents automatic and manual exporter upgrades. |
docs/source/index.rst |
Links the exporter upgrade guide. |
docs/source/configuration.rst |
Documents exporter settings and automatic updates. |
crate/operator/sql_exporter.py |
Implements configuration updates, sequential restarts, and image rollback. |
crate/operator/operations.py |
Removes unused SQL exporter configuration helpers. |
crate/operator/main.py |
Registers the opt-in resume handler. |
crate/operator/create.py |
Generates configuration hashes and annotates pod templates. |
crate/operator/constants.py |
Defines shared exporter names and hash keys. |
crate/operator/config.py |
Adds the disabled-by-default automatic update setting. |
CHANGES.rst |
Records automatic updates and upgrade documentation. |
Review effort: Balanced
| if not stale_image and not stale_config: | ||
| return |
There was a problem hiding this comment.
Fixed in d26f749. While an image change is unverified, the pod carries a sql-exporter-rollout-from annotation with the previous image, written in the same patch as the image. A later run that finds it waits for the exporter to be ready. It then clears the annotation, or reverts to the recorded image and raises PermanentError. The config-hash annotation is now written only after that check.
| or status is None | ||
| or pod.metadata.deletion_timestamp is not None | ||
| or pod.status.phase != "Running" | ||
| ): | ||
| return |
There was a problem hiding this comment.
Fixed in d26f749. update_pod and update_pods now report a stale pod that is not running yet. After the templates are updated, update_sql_exporter raises TemporaryError while one remains. A pod that already has the new image and hash does not trigger it. One consequence is that a pod stuck Pending keeps requeueing the handler.
| status = _container_status(pod) | ||
| if status and status.ready and status.restart_count > restart_count: | ||
| return True |
There was a problem hiding this comment.
Fixed in d26f749. _wait_for_restart takes the pod UID. A different UID, or a 404, counts as the pod having been replaced: skip it, no revert, and the second pass handles the replacement.
| if status and status.ready and status.restart_count > restart_count: | ||
| return True |
There was a problem hiding this comment.
Partly fixed in d26f749. After the config-projection wait, the pod is read again and the restart baseline and UID come from that read, so an unrelated restart during the wait can no longer satisfy the check. I did not add a check of the running container image against status.image, because kubelet may report it in a normalised form and a strict comparison could cause false failures. A restart between the fresh read and the patch is still possible, though it is a much smaller window.
Record the previous image on the pod while an image change is unverified, so a resumed run confirms or reverts it instead of treating matching image and hash as done. Requeue while stale pods are not running yet, refresh the restart baseline after the config projection wait, and treat a pod with a new UID as replaced. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>


Summary of changes
On top of #931
SQL exporter image and configuration were only written when a cluster was created, so version upgrades were manual action.
With new env var
SQL_EXPORTER_AUTO_UPDATEenabled, a resume handler updates theConfigMap, restarts only the sql-exporter container one pod at a time, and then patches the StatefulSets.
An image that does not start is reverted on the first pod and not retried until the operator restarts.
🤖
Checklist
CHANGES.rst