Skip to content

Update export tool discovery description for Azure Terraform tools - #3737

Draft
Yun Liu (liuwuliuyun) wants to merge 1 commit into
microsoft:mainfrom
liuwuliuyun:liuwuliuyun-terraform-namespace-fix
Draft

Update export tool discovery description for Azure Terraform tools#3737
Yun Liu (liuwuliuyun) wants to merge 1 commit into
microsoft:mainfrom
liuwuliuyun:liuwuliuyun-terraform-namespace-fix

Conversation

@liuwuliuyun

@liuwuliuyun Yun Liu (liuwuliuyun) commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

In namespace mode, the Azure Terraform export prompts in e2eTestPrompts.md did not select the aztfexport tools. The tools worked; the model did not pick them. The namespace description advertised "aztfexport command generation" without saying what aztfexport does, and several tools in the namespace competed for the same prompts.

This PR keys each description on the signal that distinguishes its tool from its nearest competitor:

Tool Distinguishing signal
aztfexport resource Leads with the full Azure resource ID shape; states it exports only that resource, not its resource group
aztfexport resourcegroup Keys on a named resource group rather than a resource ID
azurerm get Keys on the azurerm_ prefix; states it reads reference docs rather than querying live resources
azapi get Keys on ARM namespace format (Microsoft.Storage/storageAccounts); same live-resource disclaimer
conftest workspace / conftest plan Distinguishes a folder of .tf source files from a generated tfplan.json

Behavior is unchanged. These tools still only generate commands for local execution and never execute an export or modify Azure resources.

Validation

Real model. Ran the checked-in Vally spec against gpt-5.6-sol, the model named in the issue, through the namespace tool: 10/10 stimuli passed, including all three prompts reported as broken. Every run routed through azmcp-azureterraform, the namespace path the issue says failed.

ToolDescriptionEvaluator. Top-choice selection across the 30 azureterraform prompts went from 70.0% to 93.3%, with all prompts at 0.40 or above and in the top 3. The resource-ID prompt moved from rank 2 to rank 1.

Two prompts still miss on top-choice, both pre-existing and both for azapi_get: Microsoft.Storage/storageAccounts loses to storage_account_get, and one AzAPI prompt loses to azurerm_get. Both remain within the top 3 and above 0.40. Fixing them means editing tools outside this area, so they are left for a follow-up.

Tests

  • AztfexportToolLoadingTests — namespace and all-tools discovery, routing, provider forwarding, installation help, HTTP-mode filtering
  • AzureTerraformToolDescriptionTests — pins the disambiguating phrases so rewording cannot silently regress selection
  • tools/Azure.Mcp.Tools.AzureTerraform/tests/eval.yaml — checked-in Vally spec. The conftest prompts are marked context-required in e2eTestPrompts.md and are skipped by generated specs, so this file is their only Vally coverage.

189/189 unit tests pass.

Notes for reviewers

  • Some descriptions read as repetitive, for example the ARM type examples in azapi get. This is deliberate and measured: removing the duplication scored 90.0% instead of 93.3% and broke a passing prompt. Embeddings match on token overlap, so the repeated literals are doing retrieval work.
  • Tuning was measured against one embedding model plus one Vally run per stimulus. Vally is non-deterministic, so a repeat run may differ.
  • This PR modifies descriptions for six tools in one namespace rather than one tool. They compete with each other for the same prompts, so splitting them would mean landing measurements that a later PR invalidates. Happy to split if maintainers prefer.

GitHub issue number?

Fixes #3553

Pre-merge Checklist

  • Required for All PRs
    • Read contribution guidelines
    • PR title clearly describes the change
    • Commit history is clean with descriptive messages (cleanup guide)
    • Added comprehensive tests for new/modified functionality
    • Created a changelog entry if the change falls among the following: new feature, bug fix, UI/UX update, breaking change, or updated dependencies. Follow the changelog entry guide
  • For MCP tool changes:
    • One tool per PR: This PR adds or modifies only one MCP tool for faster review cycles
    • Updated servers/Azure.Mcp.Server/README.md and/or servers/Fabric.Mcp.Server/README.md documentation — not applicable, the README does not list these tools individually
    • Validate README.md changes running the script ./eng/scripts/Process-PackageReadMe.ps1 — not applicable, no README change
    • For new or modified tool descriptions, ran ToolDescriptionEvaluator and obtained a score of 0.4 or more and a top 3 ranking for all related test prompts
    • For tools with new names, including new tools or renamed tools, update consolidated-tools.json — not applicable, no tools added or renamed
    • For renamed tools, follow the Tool Rename Checklist and tag the PR with the breaking-change label — not applicable, no renames
    • For new tools associated with Azure services or publicly available tools/APIs/products, add URL to documentation in the PR description — not applicable, no new tools. Reference for the underlying tool: Azure Export for Terraform
  • Extra steps for Azure MCP Server tool changes:

@liuwuliuyun
fix(azureterraform): clarify export tool discovery
f05740d 
Make export intent visible in namespace and individual tool descriptions while preserving local command generation. Cover discovery, routing, provider forwarding, installation help, and HTTP restrictions with regression tests.

Disambiguate tools that competed for the same prompts by keying each description on its distinguishing selection signal. The aztfexport resource tool leads with the full Azure resource ID shape and states it exports only that resource, while the resourcegroup tool keys on a named resource group. The azurerm and azapi documentation tools state that they read reference documentation rather than querying live Azure resources, and identify their inputs by the azurerm_ prefix and the ARM namespace format respectively. The conftest workspace and plan tools distinguish a folder of Terraform .tf source files from an already generated tfplan.json.

Validated with the tool description evaluator against the azureterraform prompts, which improved top-choice selection from 70.0% to 93.3%, with all prompts remaining above the 0.40 confidence bar.

Add a checked-in Vally specification covering the prompts reported in the issue plus the tool pairs that compete for similar prompts. The conftest prompts in e2eTestPrompts.md are marked context-required and are therefore skipped by generated specifications, so the checked-in specification is the only Vally coverage for them.

Pin the disambiguating phrases with unit tests so that rewording a description cannot silently regress tool selection.

Addresses microsoft#3553

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

azure-pipelines Bot commented Sep 21, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

@liuwuliuyun Yun Liu (liuwuliuyun) changed the title fix(azureterraform): clarify export tool discovery Update export tool discovery description for Azure Terraform tools Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

server-Azure.Mcp Azure.Mcp.Server tools-Terraform

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Azure Terraform][Namespace] Failed to trigger mcp tools

1 participant

@liuwuliuyun