Update Resilience and FileShare tool descriptions - #3716
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
45c9d64 to
819c094
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Address the prompt parameter issue and synchronize required documentation, consolidated metadata, evaluation evidence, and changelog/policy requirements.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Improves Azure MCP tool descriptions and prompts for Resilience Management and Azure File Shares to improve tool routing and invocation accuracy.
Changes:
- Clarifies resilience command scope and operation distinctions.
- Updates File Shares descriptions and tests.
- Revises recovery prompts and reference documentation.
File summaries
| File | Summary |
|---|---|
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/UsagePlans/UsagePlanGetCommand.cs |
Clarifies usage-plan scope. |
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/UsagePlans/Enrollments/UsagePlanEnrollmentGetCommand.cs |
Clarifies enrollment semantics. |
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/Recovery/Plans/RecoveryPlanValidateForReprotectCommand.cs |
Clarifies reprotect validation. |
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/Recovery/Plans/RecoveryPlanValidateForOperationCommand.cs |
Distinguishes operation-level validation. |
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/Drills/DrillUpdateCommand.cs |
Clarifies supporting-resource relocation. |
tools/Azure.Mcp.Tools.FileShares/tests/Azure.Mcp.Tools.FileShares.Tests/Informational/FileShareGetLimitsCommandTests.cs |
Verifies the limits description. |
tools/Azure.Mcp.Tools.FileShares/tests/Azure.Mcp.Tools.FileShares.Tests/FileShare/FileShareGetCommandTests.cs |
Verifies the file-share description. |
tools/Azure.Mcp.Tools.FileShares/src/Commands/Informational/FileShareGetLimitsCommand.cs |
Clarifies service limits versus quotas. |
tools/Azure.Mcp.Tools.FileShares/src/Commands/FileShare/FileShareGetCommand.cs |
Clarifies file-share retrieval behavior. |
servers/Azure.Mcp.Server/docs/e2eTestPrompts.md |
Updates File Shares and resilience prompts. |
servers/Azure.Mcp.Server/docs/azmcp-commands.md |
Updates File Shares command documentation. |
.github/skills/resilience-management-operations/references/tools.md |
Updates resilience tool references. |
Review details
Suppressed comments (8)
.github/skills/resilience-management-operations/references/tools.md:11
- This reference is updated, but the shipped CLI reference is not:
servers/Azure.Mcp.Server/docs/azmcp-commands.mdstill uses the old generic wording for usage-plan get/enrollment, drill update, and both recovery-plan validation commands (for example, lines 4211, 4230, 4323, 4330, and 4414). Update those entries so the public command documentation carries the same Azure Resilience and command-routing distinctions.
| `mcp_azure_mcp_ser_resilience_usageplan_get` | List or get Azure Resilience Management usage plans; not quota, API Management, or load-testing plans | `subscription?`, `resource-group?`, `name?`, `tenant?`. A specific `name` requires `resource-group`. Omit both to list across the subscription. |
servers/Azure.Mcp.Server/docs/e2eTestPrompts.md:1153
- Please add the final ToolDescriptionEvaluator evidence to the PR description. It currently only says the changes are based on evaluation failures; the repository gate requires at least 0.4 and top-three ranking, while the Resilience Management override requires every changed resilience prompt to rank #1 at 0.6 or higher and include representative prompts.
| resilience_recoveryplan_create | Change recoveryplan <recoveryplan_name> in service group <service_group> to a system-assigned managed identity and description <plan_description>. Keep its Zonal plan type and existing recovery groups | none |
| resilience_recoveryplan_create | Create Zonal recoveryplan <recoveryplan_name> in service group <service_group> with a system-assigned managed identity, plan description <plan_description>, default recovery group description <default_group_description>, and one additional recovery group described as <additional_group_description> | none |
| resilience_recoveryplan_create | Create Zonal recoveryplan <recoveryplan_name> in service group <service_group> with a system-assigned managed identity and plan description <plan_description>. Add manual pre-action <manual_action_name> with timeout <timeout_minutes> to the default group, and add CustomRunbook post-action <runbook_action_name> with timeout <runbook_timeout_minutes> using Automation runbook <runbook_resource_id> to an additional group described as <additional_group_description> | none |
servers/Azure.Mcp.Server/docs/e2eTestPrompts.md:1152
- These prompts are marked
noneand are intended to be directly executable, but--additional-groupsrequires each group object to include an integerorderId(the parser rejects objects with onlydescription; seeRecoveryPlanCreateCommand.cs:165-172). Add order ID1for the single additional group in both prompts, otherwise the generated invocation must invent a required value or fails validation.
| resilience_recoveryplan_create | Change recoveryplan <recoveryplan_name> in service group <service_group> to a system-assigned managed identity and description <plan_description>. Keep its Zonal plan type and existing recovery groups | none |
| resilience_recoveryplan_create | Create Zonal recoveryplan <recoveryplan_name> in service group <service_group> with a system-assigned managed identity, plan description <plan_description>, default recovery group description <default_group_description>, and one additional recovery group described as <additional_group_description> | none |
tools/Azure.Mcp.Tools.FileShares/src/Commands/Informational/FileShareGetLimitsCommand.cs:19
- This PR changes both the Resilience Management and FileShares toolsets, but the repository requires one toolset per MCP tool pull request (
AGENTS.md:194). Please split the FileShares changes from the Resilience changes, or obtain an explicit exception before merging.
Get Azure File Shares service limits and provisioning constants for a subscription and location,
tools/Azure.Mcp.Tools.FileShares/src/Commands/Informational/FileShareGetLimitsCommand.cs:22
- This is a user-facing tool-selection/UX change, but no Azure MCP Server changelog entry is included. Similar description and prompt fixes are recorded in
servers/Azure.Mcp.Server/CHANGELOG.md(for example entries at lines 118 and 744), and the pull request checklist requires a changelog for UI/UX updates. Add a valid YAML entry underservers/Azure.Mcp.Server/changelog-entries/.
Get Azure File Shares service limits and provisioning constants for a subscription and location,
including supported file share counts, storage, IOPS, and throughput ranges.
Use this command for Azure File Shares service limits, not Azure resource quota usage.
""",
tools/Azure.Mcp.Tools.FileShares/src/Commands/Informational/FileShareGetLimitsCommand.cs:21
consolidated-tools.json:3489still describes the mapped planning tool as includingquotas. In consolidated mode that parent description is the routing surface, so it contradicts the new exclusion of Azure resource quota usage and can continue sending quota requests to File Shares. Update the consolidated description to say service limits rather than resource quotas.
Use this command for Azure File Shares service limits, not Azure resource quota usage.
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/Recovery/Plans/RecoveryPlanValidateForOperationCommand.cs:25
- The PR description does not include the final ToolDescriptionEvaluator results. Because this changes Resilience Management tool descriptions, the Resilience-specific gate requires at least two prompts per changed tool, the expected tool ranked #1 with a score of at least 0.6 for every prompt, and representative scores recorded in the PR description; please add that evidence before merge.
Runs operation-level pre-validation for an Azure Resilience Management recoveryplan before execution.
It checks whether the plan's current state supports Failover, FailoverCommit, Reprotect, TestFailover,
or TestFailoverCleanup, including readiness and permissions. When the operation is named, call this
command directly with that operation; ask the user only when it is omitted. For operation-level Failover
validation, use this command rather than validateforfailover, which evaluates per-resource qualification
using source locations or resource IDs. This command does not execute the recovery operation.
tools/Azure.Mcp.Tools.ResilienceManagement/src/Commands/UsagePlans/UsagePlanGetCommand.cs:21
- The PR description does not record the required ToolDescriptionEvaluator results for these changed descriptions. The Resilience-specific gate requires at least two prompts per changed tool, rank #1 with a score of at least 0.6 for every prompt, and representative scores/prompts in the PR description; the generic gate also applies to the FileShares descriptions. Please add the final evaluation evidence before merging.
Gets Azure Resilience Management usage plan resources. Use this command when the user asks to get or list
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
.github/skills/resilience-management-operations - please see if files in this folder needs update. Skill added for resilience |
|
.github/skills/resilience-management-operations - please see if files in this folder needs update. Skill added for resilience I updated those files too |
What does this PR do?
Based on eval failures using e2eTestPrompts.md updates tool descriptions or test prompts to be clearer with their intention. The updates are for instances where model chose the incorrect tool namespace or child command.
Pre-merge Checklist
servers/Azure.Mcp.Server/README.mdand/orservers/Fabric.Mcp.Server/README.mddocumentationREADME.mdchanges running the script./eng/scripts/Process-PackageReadMe.ps1. See Package READMEToolDescriptionEvaluatorand obtained a score of0.4or more and a top 3 ranking for all related test promptsconsolidated-tools.jsonbreaking-changelabelservers/Azure.Mcp.Server/docs/azmcp-commands.md./eng/scripts/Update-AzCommandsMetadata.ps1to update tool metadata inazmcp-commands.md(required for CI)servers/Azure.Mcp.Server/docs/e2eTestPrompts.mdcrypto mining, spam, data exfiltration, etc.)/azp run mcp - pullrequest - liveto run Live Test Pipeline