* ci: surface E2E screenshots in review comment
* ci: mark completed review commits in past tense
* ci: surface approved sensitive-file reviews
* ci: link sensitive files to reviewed changes
* ci: stage desktop E2E visual evidence
Track screenshots newly introduced against main and package visual diffs for a trusted publisher.
* fix(ci): pass E2E evidence output paths
Supply the manifest and staging-directory arguments required by the screenshot status helper.
* fix(ci): download the OSV SARIF artifact
Match the artifact name and result filename emitted by the pinned upstream reusable workflow.
Replace the static comment-pending + comment-results two-job pattern
with a live-updating comment system that polls the GitHub Actions API
every 15s, re-assembles the review comment from whatever results are
available, and upserts it via the <!-- hermes-ci-review-bot --> marker.
The comment updates in real time as each job finishes — no waiting for
the full pipeline.
Every CI job that wants to appear in the review comment emits a
review_status output — a JSON array of objects, each with a source
and a results array:
[
{
"source": "review-label-gate",
"results": [
{"kind": "action_required", "title": "...", "summary": "...",
"how_to_fix": "..."},
{"kind": "info", "title": "...", "summary": "..."}
]
},
{
"source": "ci timing",
"results": [
{"kind": "warning", "title": "CI timings", "summary": "...",
"detail": "...", "link": "..."}
]
}
]
One job can emit multiple results of different kinds. The source field
is used to exclude the corresponding job from the synthesized error
list (case-insensitive, hyphen-normalized matching against GitHub
Actions job display names).
| job | source | kind (on failure) | section |
|----------------------------|--------------------------|---------------------------|----------------------|
| review-labels | review label gate | action_required / info | Action required |
| lockfile-diff | lockfile-diff | action_required | Action required |
| ci-timings | ci timing | warning / info | Warnings |
| supply-chain scan | supply chain | error / (none) | Job failures |
| supply-chain dep-bounds | supply chain | action_required / (none) | Action required |
| osv-scanner | osv scan | warning / (none) | Warnings |
| uv-lockfile-check | uv.lock check | action_required / (none) | Action required |
| history-check | unrelated histories | action_required | Action required |
| contributor-check | contributor attribution | action_required | Action required |
Jobs that find nothing emit [] (empty array) — no noise info items.
A single comment-live job polls the GitHub Actions API every 15s,
classifies jobs into (completed, pending), assembles the comment, and
upserts it. Merges review_status outputs from all needs jobs via
toJSON(needs.*.outputs.review_status), and downloads the ci-timings
artifact when it becomes available. Shows commit SHA + message below
the header.
The assembler has ZERO job-specific knowledge. It just:
1. collect_from_statuses() — flattens all nested status objects into ReviewItems
2. collect_failed_jobs() — synthesizes errors for failed jobs with no declared status
3. _attach_job_urls() — fills in per-job log links for ALL items
4. render_comment() — groups by severity, renders with group headers
Each item shows links inline next to the title: View report (job-emitted
URL) and View job (auto-attached logs link). Each info item is its own
collapsible <details> block.
# ૮ >ﻌ< ა ci review
running on abc1234 — commit message first line
## ❌ Job failures
### {title} · [View job](url)
{summary}
## ⚠️ Action required
### {title} · [View job](url)
{summary}
**How to fix:**
{how_to_fix}
## ⚠️ Warnings
### {title} · [View report](url) · [View job](url)
{summary}
{detail}
<details><summary>{title}</summary>
{content}
</details>
Still running 3 jobs: ci-timings, docker
- test_assemble_review_comment.py (48 tests): collect_from_statuses,
collect_failed_jobs with exclude_sources, _attach_job_urls,
render_comment (group headers, inline links, commit info, per-item
details, pending footer), assemble integration
- test_live_comment.py (16 tests): classify_jobs pure function
- test_timings_report.py (10 tests): generate_review_status nested format
- test_lockfile_diff.py (6 tests)
- test_classify_changes.py (32 tests, pre-existing)
#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows and
#66577 restored the `|| github.token` fork fallback for detect-changes and
the label gates -- but it missed the ci-timings "Collect timings and
generate report" step, which still passes a bare AUTOFIX_BOT_PAT. On fork
PRs that PAT is empty, so timings_report.py hard-fails at
expect_env("GITHUB_TOKEN") before it can reach its own "degraded run must
never redden the PR" soft-fail path. Every fork PR gets a red run from this
advisory job (e.g. #66573).
- ci.yml: apply the same `secrets.AUTOFIX_BOT_PAT || github.token` fallback
to the timings step. github.token has `actions: read`, enough to read the
run's job/step durations on forks.
- timings_report.py: treat a missing/empty GITHUB_TOKEN as a degraded run
(TimingsUnavailable) instead of a hard ValueError, so this whole class of
failure can never redden a PR again even if a future workflow drops the
token. Still writes no JSON, so no empty baseline is ever cached.
Baseline bars were positioned at their absolute timeline offset, which
included the baseline run's own wait time — making duration comparisons
hard since the bars were visually offset. Now each baseline bar starts
at the same left position as its corresponding current job, so the two
bars directly overlap for at-a-glance duration comparison.
Also removed the now-unused bl_t0 / bl_max / bl_jobs_timed variables
that tracked the baseline timeline position.
Add wait time computation: for each job, wait_s = started_at - max(completed_at
of all jobs that finished before it started). This is a timestamp heuristic
(no workflow YAML dependency parse needed) that's accurate for pipeline-shaped
CI where the critical path is linear at each stage.
Shows up in:
- Job table: new Wait + Δ Wait columns
- Gantt chart: hatched bar segment before the run bar
- Step details: '(wait Xs)' annotation in the summary line
- Markdown summary: Total wait row with delta vs baseline
- Stats: total_wait / bl_total_wait in compute_stats
Also completes the skipped-jobs UI:
- Skipped stat card in stats cards
- Skipped row in markdown summary table
Backward compatible: old cached baselines without wait_s annotate on load.
Skipped jobs (conclusion == 'skipped') have null/zero-duration timestamps
that polluted every downstream computation: they counted as 'unchanged'
(0 vs 0) in faster/slower tallies, showed meaningless '0.0s (0%)' deltas
in the job table, rendered phantom gantt bars, and inflated wall/compute
totals.
Add is_skipped() helper and apply it consistently:
- compute_stats: exclude from wall/compute + faster/slower/unchanged;
add 'skipped' and 'bl_skipped' counts
- _gantt_bars: filter from current bars, baseline bars, and axis calc
- _job_table: show 'skipped' label instead of durations/deltas
- _step_details: skip entirely (no meaningful step data)
- _regressions: exclude from both current and baseline sides
The 'CI timing report' job is pure observability — it collects per-job/step
durations from the GitHub API after the run and publishes an HTML gantt
report + PR-vs-main timing diff. It gates nothing (all-checks-pass does not
include it), yet it could redden a PR: the script makes dozens of paginated
API calls with the shared repo GITHUB_TOKEN and had zero retry handling, so
a single 403 (rate-limit burst when several PRs run CI concurrently) failed
the job. Observed twice in a row on PR #59805.
- api_get(): retry 403/429/5xx and connection errors with exponential
backoff, honoring Retry-After / X-RateLimit-Reset (max 5 attempts, 120s
cap). Non-transient statuses (404 etc.) still fail fast.
- main(): exhausted retries raise TimingsUnavailable, caught to emit a
degraded summary line + placeholder HTML artifact and exit 0 — a metrics
collector must never fail the PR's checks. No timings JSON is written on
the degraded path so an empty baseline can never be cached.
- ci.yml: baseline-save steps on main skip gracefully when no JSON exists.
Verified with a mocked urlopen harness: retry-then-success (3 attempts),
exhausted-retries -> TimingsUnavailable, 404 fails fast without retry,
degraded main() exits 0 with summary + placeholder and no JSON, and the
--from-json happy path is unchanged.
