Skip to content

fix(highwind): log run failures that happen before the first log line - #625

Merged
jaredlockhart merged 2 commits into
mozilla:mainfrom
JaredLockhart-Mozilla:highwind-log-run-failures
Nov 2, 2026
Merged

jaredlockhart merged 2 commits into
mozilla:mainfrom
JaredLockhart-Mozilla:highwind-log-run-failures

Conversation

@jaredlockhart

Copy link
Copy Markdown
Contributor

Because

  • A run that failed before its first log statement wrote nothing to the run log, so a crash in
    discovery left no trace in BigQuery.

This commit

  • Writes an ERROR row with the exception and flushes buffered rows before re-raising, on runs
    that write tables.
  • Adds tests for the failure row, the no-write runs and a failing log write.

Tracked by EXP-7916

Because

* A run that failed before its first log statement wrote nothing to the run log, so a crash in
  discovery left no trace in BigQuery.

This commit

* Writes an ERROR row with the exception and flushes buffered rows before re-raising, on runs
  that write tables.
* Adds tests for the failure row, the no-write runs and a failing log write.

Tracked by [EXP-7916](https://mozilla-hub.atlassian.net/browse/EXP-7916).
@jaredlockhart
jaredlockhart requested a review from a team as a code owner November 2, 2026 16:18
try:
yield run_log
except Exception:
logger.exception("the run failed and did not finish")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be helpful to log any additional info about the run?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, we can add more info later when fun new bugs crop up 🙏

@jaredlockhart
jaredlockhart enabled auto-merge (squash) November 2, 2026 18:12
@jaredlockhart
jaredlockhart merged commit 09e97f1 into mozilla:main Nov 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants