[quality-improver] Repository Quality: AzureFoundry Extension Missing from Self-Registration Acceptance Test

[github-actions]

🎯 Repository Quality Improvement Report — Extension Self-Registration Acceptance Coverage

Analysis Date: 2026-09-21
Focus Area: extension-self-registration-acceptance-coverage
Strategy Type: Custom

Executive Summary

Every Microsoft.Testing.Platform extension that self-registers via a TestingPlatformBuilderHook (AzureDevOpsReport, CrashDump, CtrfReport, GitHubActionsReport, HangDump, HotReload, HtmlReport, JUnitReport, Retry, Telemetry, TrxReport, VideoRecorder) is exercised by the MSBuildTests_KnownExtensionRegistration.Microsoft_Testing_Platform_Extensions_ShouldBe_Correctly_Registered acceptance test in test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/MSBuild.KnownExtensionRegistration.cs. This test packs each extension, builds a consuming asset, and asserts both that the generated _GenerateSelfRegisteredExtensions MSBuild target embeds <Extension>.TestingPlatformBuilderHook.AddExtensions in the source, and that the resulting host's --help output surfaces the extension's CLI options.

Microsoft.Testing.Extensions.AzureFoundry ships the identical wiring — a TestingPlatformBuilderHook class, PackagedApp-style build/buildTransitive/buildMultiTargeting .props importing the same TestingPlatformBuilderHook item shape (<TestingPlatformBuilderHook Include="3D8F9C1A-7B2E-4F6D-A9C3-5E8B4D1F7A6C"> with DisplayName/TypeFullName pointing at Microsoft.Testing.Extensions.AzureFoundry.TestingPlatformBuilderHook) — yet it is absent from this test's SourceCode package list and from all twelve Assert.Contains(...TestingPlatformBuilderHook.AddExtensions...) assertions. A regression that breaks AzureFoundry's self-registration (for example a typo in the hook GUID, TypeFullName, or a build-props import mistake) would not be caught by this acceptance test, even though the sibling HelpInfoAllExtensionsTests asset does at least reference the AzureFoundry package (without asserting on its self-registration source generation specifically).

This is a genuine coverage gap distinct from previously filed findings in this repository — it concerns the MSBuildTests_KnownExtensionRegistration test's known-extension list, not any of the runtime logic areas (Retry, HangDump, CrashDump, HotReload, ChatClientManager, OpenTelemetry, AzureDevOps run coordination, etc.) covered by prior quality-improver issues.

Full Analysis Report

Focus Area: Extension Self-Registration Acceptance Coverage

Current State Assessment

Metrics Collected:

Metric	Value	Status
Extensions with TestingPlatformBuilderHook.cs	15 (AzureDevOpsReport, AzureFoundry, CrashDump, CtrfReport, GitHubActionsReport, HangDump, HotReload, HtmlReport, JUnitReport, MSBuild, PackagedApp, Retry, Telemetry, TrxReport, VideoRecorder)	i️
Extensions asserted in MSBuildTests_KnownExtensionRegistration	12 of 15	⚠️
Extensions missing from that test (excluding intentionally-excluded MSBuild/PackagedApp hosting hooks)	AzureFoundry	❌
AzureFoundry package referenced in sibling HelpInfoAllExtensionsTests asset	Yes, but only for --help/--info CLI text, not self-registration source generation	⚠️

Findings

Strengths

• The acceptance test correctly enumerates and asserts self-registration for 12 of the platform's shipping extensions, giving strong regression protection for the common case.
• AzureFoundry's build-props wiring (build/buildTransitive/buildMultiTargeting .props + TestingPlatformBuilderHook item) exactly follows the same convention as its siblings, so adding coverage is mechanical.

Areas for Improvement

• ❌ High severity (regression-detection gap): Microsoft.Testing.Extensions.AzureFoundry is not included in MSBuildTests_KnownExtensionRegistration's package references or Assert.Contains list, so a broken TestingPlatformBuilderHook GUID/TypeFullName/props import for this extension would silently pass CI.
• ⚠️ Medium severity: MSBuild and PackagedApp are also excluded, but for different, defensible reasons (MSBuild is IsPackable=false/non-shipping infra, and PackagedApp is Windows/packaged-app specific and excluded from the cross-TFM [AllTargetFrameworks] combinatorial matrix already used elsewhere, e.g. HelpInfoAllExtensionsTests's net462 condition). These exclusions should be explicitly documented with a comment in MSBuild.KnownExtensionRegistration.cs so future readers don't mistake missing coverage there for an oversight, distinguishing them from the AzureFoundry gap which has no such rationale.

---

🤖 Suggested Improvement Tasks

Task 1: Add Microsoft.Testing.Extensions.AzureFoundry to MSBuildTests_KnownExtensionRegistration

Priority: High
Estimated Effort: Small

In test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/MSBuild.KnownExtensionRegistration.cs:

• Add a <PackageReference Include="Microsoft.Testing.Extensions.AzureFoundry" Version="$MicrosoftTestingExtensionsAzureFoundryVersion$" /> to the SourceCode project template (mirroring the pattern already used for CtrfReport/GitHubActionsReport/JUnitReport/VideoRecorder, which have their own version tokens because they are non-shipping or differently-versioned packages).
• Add the $MicrosoftTestingExtensionsAzureFoundryVersion$ token to the .PatchCodeWithReplace(...) chain, using the existing MicrosoftTestingExtensionsAzureFoundryVersion static property already defined in test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs (used by HelpInfoAllExtensionsTests).
• Add Assert.Contains("Microsoft.Testing.Extensions.AzureFoundry.TestingPlatformBuilderHook.AddExtensions", generatedSource.Text, generatedSource.Text); alongside the other 12 assertions.
• Note: AzureFoundry does not currently register any --help-visible CLI options (it only registers a chat client provider), so no corresponding testHostResult.AssertOutputContains(...) line is needed unless one is added — verify by running --help against a test asset referencing the package.

---

Task 2: Document the exclusion of MSBuild and PackagedApp from this test

Priority: Medium
Estimated Effort: Small

Add a short comment above the SourceCode package list or near the Assert.Contains block in MSBuild.KnownExtensionRegistration.cs explaining that Microsoft.Testing.Extensions.MSBuild (IsPackable=false, build-infra-only) and Microsoft.Testing.Extensions.PackagedApp (Windows-only, conditionally excluded per the net462 handling already present in HelpInfoAllExtensionsTests) are intentionally excluded, so a future contributor adding a new extension doesn't misread the omission as a template to copy for a genuinely shippable extension like AzureFoundry.

---

Task 3: Cross-check other combinatorial/self-registration acceptance tests for the same gap

Priority: Medium
Estimated Effort: Small

Search the acceptance test suite (test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/) for any other place that enumerates the full extension set (besides HelpInfoAllExtensionsTests, which already includes AzureFoundry) and confirm none of them silently drop newly added shipping extensions the way MSBuild.KnownExtensionRegistration.cs did. PackageMetadataCompletenessTests.cs already references AzureFoundry, so it is not affected; verifying the remainder should be quick.

---

Task 4: Add a lightweight "all known extensions" list check to prevent future drift

Priority: Low
Estimated Effort: Medium

Consider extracting the list of shipping extension package IDs with TestingPlatformBuilderHook self-registration into a single shared constant/array in test infrastructure (e.g. Microsoft.Testing.TestInfrastructure), consumed by both MSBuildTests_KnownExtensionRegistration and HelpInfoAllExtensionsTests. This would make it structurally harder for a new extension (or a future one) to be added to one test's asset but omitted from the other's assertions, as happened here with AzureFoundry.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type
2026-09-18	msbuild-namespace-sanitization-coverage-gap	Custom
2026-09-17	opentelemetry-platformservice-metrics-and-tracestate-coverage-gap	Custom
2026-09-16	hotreload-extension-unit-test-coverage-gap	Custom
2026-09-15	security-dependency-and-attack-surface-audit	Standard
2026-09-14	azuredevops-runidcoordinator-inherited-and-mismatch-paths-untested	Custom

---

🎯 Recommendations

Immediate Actions (This Week)

1. Add AzureFoundry to MSBuildTests_KnownExtensionRegistration (Task 1) — Priority: High

Short-term Actions (This Month)

1. Document intentional exclusions and audit other enumeration tests (Tasks 2–3) — Priority: Medium
2. Evaluate a shared extension-list constant to prevent recurrence (Task 4) — Priority: Low

Next analysis: 2026-09-22 (or next scheduled run) — Focus area selected based on diversity algorithm

🤖 Automated content by GitHub Copilot. Generated by the Repository Quality Improver workflow. · copilot · auto · 148.4 AIC · ⌖ 8.6 AIC · ⊞ 16.5K · [◷]( · ◷)

_{Add this agentic workflow to your repo}

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/repository-quality-improver.md@main

• expires on Sep 23, 2026, 10:31 PM UTC