azd ai eval Build-32 testing surfaced a batch of correctness bugs in azure.ai.evaluations where a command exits 0 and writes or publishes something that cannot work. The failure then appears one or two commands later, attributed to the wrong step.
Three are in scope here. Each is a case where the CLI already had enough information to answer correctly at the time it acted.
ADO 5530209 (High/P1) — an evaluator loses its catalog metadata when a later version is published.
Editing a rubric and deploying publishes a version with a blank display name, no categories, and support narrowed from [turn, conversation] to [turn]. The narrowing is the damaging part: an evaluator that was valid for conversation evals looks incompatible afterwards, so an ordinary rubric edit silently changes what it can grade. The declaration records these fields — and its own comment says they are kept so that publishing a later version keeps them — but nothing ever put them in the request.
ADO 5631311 (High/P1) — init writes declarations for JSONL it cannot use.
A malformed line, an empty file, or an array where an object belongs all scaffold cleanly and exit 0. The config then references a file that can never form an evaluation row. Deploy already refuses exactly these files; init simply never checked.
ADO 5631310 (Medium, reopened) — init accepts a builtin. evaluator that does not exist.
The reference is only shape-checked, so --evaluator builtin.does_not_exist writes an eval that create and run cannot resolve. Nothing local can answer this: the four names init offers are deliberately the offered set, not a copy of the catalogue, and the service has many more valid built-ins that init does not offer.
Ask: fix all three, with tests, without changing behaviour beyond the named defects. In particular a fix must not start refusing input that works today — for 5631310 that means an unreachable project has to keep behaving exactly as it does now.
azd ai evalBuild-32 testing surfaced a batch of correctness bugs inazure.ai.evaluationswhere a command exits 0 and writes or publishes something that cannot work. The failure then appears one or two commands later, attributed to the wrong step.Three are in scope here. Each is a case where the CLI already had enough information to answer correctly at the time it acted.
ADO 5530209 (High/P1) — an evaluator loses its catalog metadata when a later version is published.
Editing a rubric and deploying publishes a version with a blank display name, no categories, and support narrowed from
[turn, conversation]to[turn]. The narrowing is the damaging part: an evaluator that was valid for conversation evals looks incompatible afterwards, so an ordinary rubric edit silently changes what it can grade. The declaration records these fields — and its own comment says they are kept so that publishing a later version keeps them — but nothing ever put them in the request.ADO 5631311 (High/P1) —
initwrites declarations for JSONL it cannot use.A malformed line, an empty file, or an array where an object belongs all scaffold cleanly and exit 0. The config then references a file that can never form an evaluation row. Deploy already refuses exactly these files; init simply never checked.
ADO 5631310 (Medium, reopened) —
initaccepts abuiltin.evaluator that does not exist.The reference is only shape-checked, so
--evaluator builtin.does_not_existwrites an eval that create and run cannot resolve. Nothing local can answer this: the four names init offers are deliberately the offered set, not a copy of the catalogue, and the service has many more valid built-ins that init does not offer.Ask: fix all three, with tests, without changing behaviour beyond the named defects. In particular a fix must not start refusing input that works today — for 5631310 that means an unreachable project has to keep behaving exactly as it does now.