Finalizing the initial vally work - #10124
Richard Park (richardpark-msft) wants to merge 3 commits into
Conversation
… evals work: - Adding a comment in so we don't accidentally judge luna against luna, for instance. - Fixing one of the tests so it doesn't accidentally early exit and not show all failures. And some other work just to clean things up: - action.yml had some cruft from my previous refactorings - consolidated some of the stages, and also handle some of the failures (like a missing vally results folder, etc..) in a way that we'd actually see so we know we have to fix it. Fixes #9340
|
Azure Pipelines: Successfully started running 1 pipeline(s). 24 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The action still does not fail when individual trials fail while Vally exits successfully.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
Refines Vally eval CI reliability and evaluation maintenance.
Changes:
- Simplifies the composite action and improves missing-result handling.
- Uses subtests to report all assertion failures.
- Documents judge-model separation and cleans YAML formatting.
| File | Description |
|---|---|
.github/actions/vally-eval/action.yml |
Refactors setup, failure handling, reporting, and artifact upload. |
cli/azd/test/evals/vally_report_test.go |
Isolates stimulus checks into subtests. |
cli/azd/test/evals/eval-azd.experiments.yaml |
Documents subject/judge model separation. |
cli/azd/test/evals/eval-azd-qna.yaml |
Clarifies judge-model constraints. |
cli/azd/test/evals/eval-azd-deploy.yaml |
Clarifies judge-model constraints and removes whitespace. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 24 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|

Fixing the last miscellaneous bits to finish the first round of vally evals work:
And some other work just to clean things up:
Fixes #9340