Skip to content

Update Resilience and FileShare tool descriptions - #3716

Merged
Connie Yau (conniey) merged 5 commits into
mainfrom
conniey/prompt-updates
Sep 21, 2026
Merged

Connie Yau (conniey) merged 5 commits into
mainfrom
conniey/prompt-updates

Conversation

@conniey

Copy link
Copy Markdown
Member

What does this PR do?

Based on eval failures using e2eTestPrompts.md updates tool descriptions or test prompts to be clearer with their intention. The updates are for instances where model chose the incorrect tool namespace or child command.

  • Strengthened descriptions for usage plans, enrollments, drill supporting-resource relocation, operation-level validation, and reprotect validation.
  • Replaced three recovery-plan update prompts that required a successful prerequisite GET with complete, directly executable create prompts.
  • Clarified vague usage-plan, reprotect, Failover, and  TestFailoverCleanup  prompts.

Pre-merge Checklist

  • Required for All PRs
    • Read contribution guidelines
    • PR title clearly describes the change
    • Commit history is clean with descriptive messages (cleanup guide)
    • Added comprehensive tests for new/modified functionality
    • Created a changelog entry if the change falls among the following: new feature, bug fix, UI/UX update, breaking change, or updated dependencies. Follow the changelog entry guide
  • For MCP tool changes:
    • One tool per PR: This PR adds or modifies only one MCP tool for faster review cycles
    • Updated servers/Azure.Mcp.Server/README.md and/or servers/Fabric.Mcp.Server/README.md documentation
    • Validate README.md changes running the script ./eng/scripts/Process-PackageReadMe.ps1. See Package README
    • For new or modified tool descriptions, ran ToolDescriptionEvaluator and obtained a score of 0.4 or more and a top 3 ranking for all related test prompts
    • For tools with new names, including new tools or renamed tools, update consolidated-tools.json
    • For renamed tools, follow the Tool Rename Checklist and tag the PR with the breaking-change label
    • For new tools associated with Azure services or publicly available tools/APIs/products, add URL to documentation in the PR description
  • Extra steps for Azure MCP Server tool changes:
    • Updated command list in servers/Azure.Mcp.Server/docs/azmcp-commands.md
    • Ran ./eng/scripts/Update-AzCommandsMetadata.ps1 to update tool metadata in azmcp-commands.md (required for CI)
    • Updated test prompts in servers/Azure.Mcp.Server/docs/e2eTestPrompts.md
    • 👉 For Community (non-Microsoft team member) PRs:
      • Security review: Reviewed code for security vulnerabilities, malicious code, or suspicious activities before running tests (crypto mining, spam, data exfiltration, etc.)
      • Manual tests run: added comment /azp run mcp - pullrequest - live to run Live Test Pipeline

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Address the prompt parameter issue and synchronize required documentation, consolidated metadata, evaluation evidence, and changelog/policy requirements.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Improves Azure MCP tool descriptions and prompts for Resilience Management and Azure File Shares to improve tool routing and invocation accuracy.

Changes:

  • Clarifies resilience command scope and operation distinctions.
  • Updates File Shares descriptions and tests.
  • Revises recovery prompts and reference documentation.
File summaries
File Summary
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/UsagePlans/UsagePlanGetCommand.cs Clarifies usage-plan scope.
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/UsagePlans/Enrollments/UsagePlanEnrollmentGetCommand.cs Clarifies enrollment semantics.
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/Recovery/Plans/RecoveryPlanValidateForReprotectCommand.cs Clarifies reprotect validation.
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/Recovery/Plans/RecoveryPlanValidateForOperationCommand.cs Distinguishes operation-level validation.
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/Drills/DrillUpdateCommand.cs Clarifies supporting-resource relocation.
tools/Azure.Mcp.Tools.FileShares/tests/Azure.Mcp.Tools.FileShares.Tests/Informational/FileShareGetLimitsCommandTests.cs Verifies the limits description.
tools/Azure.Mcp.Tools.FileShares/tests/Azure.Mcp.Tools.FileShares.Tests/FileShare/FileShareGetCommandTests.cs Verifies the file-share description.
tools/Azure.Mcp.Tools.FileShares/src/Commands/Informational/FileShareGetLimitsCommand.cs Clarifies service limits versus quotas.
tools/Azure.Mcp.Tools.FileShares/src/Commands/FileShare/FileShareGetCommand.cs Clarifies file-share retrieval behavior.
servers/Azure.Mcp.Server/docs/e2eTestPrompts.md Updates File Shares and resilience prompts.
servers/Azure.Mcp.Server/docs/azmcp-commands.md Updates File Shares command documentation.
.github/skills/resilience-management-operations/references/tools.md Updates resilience tool references.
Review details

Suppressed comments (8)

.github/skills/resilience-management-operations/references/tools.md:11

  • This reference is updated, but the shipped CLI reference is not: servers/Azure.Mcp.Server/docs/azmcp-commands.md still uses the old generic wording for usage-plan get/enrollment, drill update, and both recovery-plan validation commands (for example, lines 4211, 4230, 4323, 4330, and 4414). Update those entries so the public command documentation carries the same Azure Resilience and command-routing distinctions.
| `mcp_azure_mcp_ser_resilience_usageplan_get` | List or get Azure Resilience Management usage plans; not quota, API Management, or load-testing plans | `subscription?`, `resource-group?`, `name?`, `tenant?`. A specific `name` requires `resource-group`. Omit both to list across the subscription. |

servers/Azure.Mcp.Server/docs/e2eTestPrompts.md:1153

  • Please add the final ToolDescriptionEvaluator evidence to the PR description. It currently only says the changes are based on evaluation failures; the repository gate requires at least 0.4 and top-three ranking, while the Resilience Management override requires every changed resilience prompt to rank #1 at 0.6 or higher and include representative prompts.
| resilience_recoveryplan_create | Change recoveryplan <recoveryplan_name> in service group <service_group> to a system-assigned managed identity and description <plan_description>. Keep its Zonal plan type and existing recovery groups | none |
| resilience_recoveryplan_create | Create Zonal recoveryplan <recoveryplan_name> in service group <service_group> with a system-assigned managed identity, plan description <plan_description>, default recovery group description <default_group_description>, and one additional recovery group described as <additional_group_description> | none |
| resilience_recoveryplan_create | Create Zonal recoveryplan <recoveryplan_name> in service group <service_group> with a system-assigned managed identity and plan description <plan_description>. Add manual pre-action <manual_action_name> with timeout <timeout_minutes> to the default group, and add CustomRunbook post-action <runbook_action_name> with timeout <runbook_timeout_minutes> using Automation runbook <runbook_resource_id> to an additional group described as <additional_group_description> | none |

servers/Azure.Mcp.Server/docs/e2eTestPrompts.md:1152

  • These prompts are marked none and are intended to be directly executable, but --additional-groups requires each group object to include an integer orderId (the parser rejects objects with only description; see RecoveryPlanCreateCommand.cs:165-172). Add order ID 1 for the single additional group in both prompts, otherwise the generated invocation must invent a required value or fails validation.
| resilience_recoveryplan_create | Change recoveryplan <recoveryplan_name> in service group <service_group> to a system-assigned managed identity and description <plan_description>. Keep its Zonal plan type and existing recovery groups | none |
| resilience_recoveryplan_create | Create Zonal recoveryplan <recoveryplan_name> in service group <service_group> with a system-assigned managed identity, plan description <plan_description>, default recovery group description <default_group_description>, and one additional recovery group described as <additional_group_description> | none |

tools/Azure.Mcp.Tools.FileShares/src/Commands/Informational/FileShareGetLimitsCommand.cs:19

  • This PR changes both the Resilience Management and FileShares toolsets, but the repository requires one toolset per MCP tool pull request (AGENTS.md:194). Please split the FileShares changes from the Resilience changes, or obtain an explicit exception before merging.
        Get Azure File Shares service limits and provisioning constants for a subscription and location,

tools/Azure.Mcp.Tools.FileShares/src/Commands/Informational/FileShareGetLimitsCommand.cs:22

  • This is a user-facing tool-selection/UX change, but no Azure MCP Server changelog entry is included. Similar description and prompt fixes are recorded in servers/Azure.Mcp.Server/CHANGELOG.md (for example entries at lines 118 and 744), and the pull request checklist requires a changelog for UI/UX updates. Add a valid YAML entry under servers/Azure.Mcp.Server/changelog-entries/.
        Get Azure File Shares service limits and provisioning constants for a subscription and location,
        including supported file share counts, storage, IOPS, and throughput ranges.
        Use this command for Azure File Shares service limits, not Azure resource quota usage.
        """,

tools/Azure.Mcp.Tools.FileShares/src/Commands/Informational/FileShareGetLimitsCommand.cs:21

  • consolidated-tools.json:3489 still describes the mapped planning tool as including quotas. In consolidated mode that parent description is the routing surface, so it contradicts the new exclusion of Azure resource quota usage and can continue sending quota requests to File Shares. Update the consolidated description to say service limits rather than resource quotas.
        Use this command for Azure File Shares service limits, not Azure resource quota usage.

tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/Recovery/Plans/RecoveryPlanValidateForOperationCommand.cs:25

  • The PR description does not include the final ToolDescriptionEvaluator results. Because this changes Resilience Management tool descriptions, the Resilience-specific gate requires at least two prompts per changed tool, the expected tool ranked #1 with a score of at least 0.6 for every prompt, and representative scores recorded in the PR description; please add that evidence before merge.
        Runs operation-level pre-validation for an Azure Resilience Management recoveryplan before execution.
        It checks whether the plan's current state supports Failover, FailoverCommit, Reprotect, TestFailover,
        or TestFailoverCleanup, including readiness and permissions. When the operation is named, call this
        command directly with that operation; ask the user only when it is omitted. For operation-level Failover
        validation, use this command rather than validateforfailover, which evaluates per-resource qualification
        using source locations or resource IDs. This command does not execute the recovery operation.

tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/UsagePlans/UsagePlanGetCommand.cs:21

  • The PR description does not record the required ToolDescriptionEvaluator results for these changed descriptions. The Resilience-specific gate requires at least two prompts per changed tool, rank #1 with a score of at least 0.6 for every prompt, and representative scores/prompts in the PR description; the generic gate also applies to the FileShares descriptions. Please add the final evaluation evidence before merging.
        Gets Azure Resilience Management usage plan resources. Use this command when the user asks to get or list
  • Files reviewed: 10/10 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@adishiritwick

Copy link
Copy Markdown
Contributor

.github/skills/resilience-management-operations - please see if files in this folder needs update. Skill added for resilience

@conniey

Copy link
Copy Markdown
Member Author

.github/skills/resilience-management-operations - please see if files in this folder needs update. Skill added for resilience

I updated those files too

@conniey
Connie Yau (conniey) merged commit a330b69 into main Sep 21, 2026
19 checks passed
@conniey
Connie Yau (conniey) deleted the conniey/prompt-updates branch September 21, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants