Skip to content

[agentbricks] Delete runtime store before role in tool-matrix cleanup - #688

Open
elainewang-db wants to merge 1 commit into
databricks:mainfrom
elainewang-db:mason-tool-matrix-cleanup
Open

elainewang-db wants to merge 1 commit into
databricks:mainfrom
elainewang-db:mason-tool-matrix-cleanup

Conversation

@elainewang-db

@elainewang-db elainewang-db commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

What

Two changes to the Agent Bricks e2e tool-matrix cleanup (integrations/agentbricks/tests/e2e/tool_matrix.py):

  1. Delete the runtime store before the role. cleanup() tore down each app with raw databricks apps delete, which leaves the deploy-created managed runtime store runtime-stores/{app} and its dedicated Lakebase DB. The subsequent databricks postgres delete-role then fails because the App SP role still owns that DB. Now, for each app with stores, the harness issues databricks api delete /api/2.0/agents/runtime-stores/{app} (404-tolerant) before the role delete, matching the product teardown order in deployments delete (verified in cli/deploy.py + agentkit _api_client.delete_runtime_store). The role delete only runs once the store delete succeeds.
  2. Surface failed cleanup entries. verify_evidence's gate printed only a generic one-liner; it now prints each failed cleanup entry (resource + detail) and flags any app deleted without an absence confirmation, so cleanup failures name themselves in CI (where evidence.json isn't uploaded).

Why

On the latest nightly dispatch the matrix was fully green (24 passed, 0 failed) but the job exited 1 solely on the cleanup gate. The failing cleanup resource was not visible in CI output. The leading suspect — grounded in the code path and a previously-seen gotcha — is the Lakebase role delete failing on the leftover runtime-store DB; this PR fixes that teardown ordering and adds the diagnostics so the next run confirms the exact failing entry (rather than guessing again).

Test

uv run --group tests pytest tests/unit_tests -k "cleanup or tool_matrix or evidence or author" → 82 passed. Added tool_matrix_cleanup_test.py cases: runtime-store DELETE in the expected command sequence; store-delete-fails → role skipped + gate trips; 404 → treated as deleted + role proceeds.

Live-validated: dispatched the Agent Bricks integration suite against this PR's head → the agentbricks-tests job passed — the first fully-green end-to-end run (all 24 matrix rows pass and the cleanup gate now clears). This confirms the leftover runtime-store DB blocking the role delete was indeed the cause. Run: https://github.com/databricks-eng/ai-oss-integration-tests-runner/actions/runs/37081527823

This pull request and its description were written by Isaac.

Also surface failed cleanup entries in the evidence gate output.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant