Skip to content

docs: correct workflow publishing security-review claim, add catalog vetting notes - #4736

Open
chelsealong wants to merge 3 commits into
github:mainfrom
chelsealong:docs/4733-catalog-trust-vetting-consistency
Open

chelsealong wants to merge 3 commits into
github:mainfrom
chelsealong:docs/4733-catalog-trust-vetting-consistency

Conversation

@chelsealong

Copy link
Copy Markdown
Contributor

Fixes #4733.

Problem

workflows/PUBLISHING.md described the workflow submission "Verification Process" as including a Security check ("no malicious shell commands, safe operations") and stated that "catalog-listed workflows are reviewed at submission time," then said the workflow "appears in specify workflow search" once "verified." In reality maintainers only check submission form and completeness (schema validity, step correctness, input design, documentation) — they do not review, audit, or endorse the shell step content a workflow actually runs. This directly contradicts the file's own later "Security: shell steps execute arbitrary code" section, which correctly tells users to audit run fields themselves.

Separately, docs/reference/presets.md and docs/reference/bundles.md document how to mark a catalog install_allowed but, unlike docs/reference/extensions.md (added in #4177), never warn that a project-supplied catalog or an install_allowed flag is not evidence the catalog's contents were vetted.

docs/community/extensions.md, docs/community/presets.md, and docs/community/bundles.md already use the correct "entry form and completeness only, not component code" language, so no changes were needed there.

Changes

  • workflows/PUBLISHING.md: rewrote the "Verification Process" section to drop the Security review claim and state plainly that the check covers submission form/completeness, not the shell code the workflow runs; updated the follow-on sentence that referenced "workflows are reviewed at submission time" to match. A follow-up commit then reconciled this new wording with the pre-existing "Authors" bullet under "Security: shell steps execute arbitrary code," which said maintainers "will reject submissions whose shell steps can't be justified at review time" — as first worded, the two statements flatly contradicted each other (one said maintainers never look at shell content, the other said they reject based on it). All three references now consistently say maintainers may reject a submission whose shell steps look obviously dangerous during the form/completeness pass, but this is never a security audit and shouldn't be relied on as one.
  • docs/reference/presets.md: added a note in "Catalog Management" that a project's .specify/preset-catalogs.yml can point at an unvetted catalog, and that install_allowed should only be set on a catalog you authored or vetted yourself (mirrors the existing guidance in docs/reference/extensions.md).
  • docs/reference/bundles.md: added an equivalent note in "Manage Catalog Sources" covering both the bundle catalog source itself and the component catalogs (extension/preset/workflow) a bundle can pull in.
  • Added tests/test_catalog_trust_docs.py with regression assertions for the three changes above.

Test plan

$ python3 -m pytest tests/test_catalog_trust_docs.py -v
tests/test_catalog_trust_docs.py::test_workflow_publishing_does_not_claim_security_review PASSED
tests/test_catalog_trust_docs.py::test_preset_catalog_docs_warn_about_vetting_install_allowed PASSED
tests/test_catalog_trust_docs.py::test_bundle_catalog_docs_cover_bundle_source_and_component_catalogs PASSED
3 passed in 0.04s

Confirmed the new test fails without the fix by checking out the pre-change versions of the three docs files (git checkout HEAD~2 -- workflows/PUBLISHING.md docs/reference/presets.md docs/reference/bundles.md) and re-running — all three assertions failed as expected, then passed again once the docs were restored.

npx markdownlint-cli2 on the edited doc: 0 issues in 0 files.

AI disclosure

Implemented autonomously by Claude Code (model: Claude Sonnet 5), running non-interactively end-to-end: issue re-verification, doc edits, regression test, and lint validation were all AI-generated and AI-executed with no human line-by-line review prior to this push. A follow-up commit (also autonomous, same model) reconciled a contradiction an independent review caught in the first commit's PUBLISHING.md wording.

🤖 Generated with Claude Code

…vetting notes

workflows/PUBLISHING.md described the workflow submission check as a
security review of shell step content and called listed workflows
"reviewed" for that purpose. Maintainers only check submission form and
completeness, matching the language already used for extensions,
presets, and bundles.

Preset and bundle catalog docs demonstrated install_allowed without the
vetting guidance extensions.md already gives, so a project-supplied
catalog could look implicitly trusted. Add the same "vet before marking
install_allowed" note to docs/reference/presets.md and
docs/reference/bundles.md (covering both bundle sources and the
component catalogs a bundle can pull in).
The prior commit's new Verification Process wording ("maintainers do
not review... at all") contradicted the untouched Authors bullet under
Security, which said maintainers reject submissions whose shell steps
can't be justified at review time. Reword both, plus the parallel
sentence in the Security section, so all three consistently say
maintainers may reject an obviously dangerous submission during the
form/completeness pass, but this is never a security audit.

Assisted-by: Claude Sonnet 5 (model: claude-sonnet-5, autonomous)
@chelsealong
chelsealong requested a review from mnriem as a code owner September 24, 2026 14:53
@mnriem mnriem added the triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate label Sep 24, 2026
@mnriem
mnriem requested a balanced review from Copilot September 24, 2026 17:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Workflow user guidance remains incomplete, and bundle guidance omits step catalogs.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 3 Low severity

Open (3)
What changed in this PR

Clarifies catalog trust boundaries and workflow review scope.

Changes:

  • Corrects workflow publishing security-review claims.
  • Adds preset and bundle catalog vetting guidance.
  • Adds documentation regression tests.
File Description
workflows/​PUBLISHING.md Clarifies submission checks and security responsibility.
docs/​reference/​presets.md Adds install-enabled catalog vetting guidance.
docs/​reference/​bundles.md Adds bundle and component-source vetting guidance.
tests/​test_catalog_trust_docs.py Tests the new documentation requirements.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/reference/bundles.md Outdated
Comment thread tests/test_catalog_trust_docs.py
Comment thread workflows/PUBLISHING.md
4. **Documentation** — clear README explaining what the workflow does and when to use it

Once verified, the workflow appears in `specify workflow search`.
This is a check of the submission's **form and completeness**, not a security review — maintainers may reject a submission if its `shell` step content looks obviously dangerous during this pass, but they do not systematically audit, endorse, or support the workflow's code. Treat every workflow, including catalog-listed ones, as untrusted until you've read its `run` fields yourself (see [Security: shell steps execute arbitrary code](#security-shell-steps-execute-arbitrary-code)).
…eference docs

Address review feedback on github#4736: the bundle catalog trust warning
omitted step catalogs and their inspection command, and the new
catalog-vetting guidance only lived in the author-facing publishing
guide, not in the user-facing workflow reference docs (workflow run/add,
.specify/workflow-catalogs.yml). Adds an equivalent warning to
docs/reference/workflows.md and extends the regression test to assert
the step-catalog inspection command explicitly.
@chelsealong

Copy link
Copy Markdown
Contributor Author

Addressed all three review comments in 9369001:

  • docs/reference/bundles.md: the catalog-trust warning now includes step catalogs (fourth component source) and the specify workflow step catalog list inspection command.
  • docs/reference/workflows.md: added an equivalent catalog-vetting warning in "Catalog Management" (next to workflow run/add/.specify/workflow-catalogs.yml), since the earlier fix only touched the author-facing publishing guide.
  • tests/test_catalog_trust_docs.py: added assertions that explicitly check for the specify workflow step catalog list command in both files, so an incomplete list regresses loudly.

Confirmed both new assertions fail against the pre-fix docs (git checkout HEAD~1 -- docs/reference/bundles.md docs/reference/workflows.md) and pass again after restoring. Full suite: 2563 passed, 5 skipped.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The documentation is consistent with the stated trust model, references valid commands, and includes focused regression coverage.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
Resolved since last review (2)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clarify catalog trust and community review scope across documentation

3 participants