Print gh-image stdout and stderr on attachment failures, then replace the
pending inline-evidence marker in the PR review comment with an escaped
failure notice before preserving the failing workflow result.
Keep the live PR review comment polling for a short grace period after
visible jobs complete. Preserve composite-action jobs in timing and live
status collection, and split the timing HTML from its linked review-status
artifact.
* fix(ci): publish inline E2E evidence
Upload bounded screenshot evidence from E2E, then publish validated images
from a trusted workflow_run job to commit-pinned branches in the evidence repo.
Wait briefly for the live CI review comment marker before publishing, so
GitHub's read-after-write delay cannot leave an orphaned evidence branch.
* fix(ci): isolate privileged credentials from PR jobs
Keep App private keys and Docker Hub credentials out of PR-controlled
workflows. Use protected environments for trusted publishing and a public
repository variable for the App client ID.
* fix(ci): attach E2E evidence with restricted bot session
Replace the App-backed evidence repository publisher with gh-image uploads
from a dedicated bot session in the gh-image environment.
* fix(ci): publish validated E2E evidence from forks
Let the trusted default-branch publisher handle bounded, validated evidence
artifacts from fork PR CI without checking out or executing fork code.
* ci: surface E2E screenshots in review comment
* ci: mark completed review commits in past tense
* ci: surface approved sensitive-file reviews
* ci: link sensitive files to reviewed changes
* ci: stage desktop E2E visual evidence
Track screenshots newly introduced against main and package visual diffs for a trusted publisher.
* fix(ci): pass E2E evidence output paths
Supply the manifest and staging-directory arguments required by the screenshot status helper.
* fix(ci): download the OSV SARIF artifact
Match the artifact name and result filename emitted by the pinned upstream reusable workflow.
Let the scanner report critical findings without failing. The review-label
gate owns the action-required status and blocking result, allowing the
ci-reviewed label rerun to clear both CI and the PR comment.
Replace the static comment-pending + comment-results two-job pattern
with a live-updating comment system that polls the GitHub Actions API
every 15s, re-assembles the review comment from whatever results are
available, and upserts it via the <!-- hermes-ci-review-bot --> marker.
The comment updates in real time as each job finishes — no waiting for
the full pipeline.
Every CI job that wants to appear in the review comment emits a
review_status output — a JSON array of objects, each with a source
and a results array:
[
{
"source": "review-label-gate",
"results": [
{"kind": "action_required", "title": "...", "summary": "...",
"how_to_fix": "..."},
{"kind": "info", "title": "...", "summary": "..."}
]
},
{
"source": "ci timing",
"results": [
{"kind": "warning", "title": "CI timings", "summary": "...",
"detail": "...", "link": "..."}
]
}
]
One job can emit multiple results of different kinds. The source field
is used to exclude the corresponding job from the synthesized error
list (case-insensitive, hyphen-normalized matching against GitHub
Actions job display names).
| job | source | kind (on failure) | section |
|----------------------------|--------------------------|---------------------------|----------------------|
| review-labels | review label gate | action_required / info | Action required |
| lockfile-diff | lockfile-diff | action_required | Action required |
| ci-timings | ci timing | warning / info | Warnings |
| supply-chain scan | supply chain | error / (none) | Job failures |
| supply-chain dep-bounds | supply chain | action_required / (none) | Action required |
| osv-scanner | osv scan | warning / (none) | Warnings |
| uv-lockfile-check | uv.lock check | action_required / (none) | Action required |
| history-check | unrelated histories | action_required | Action required |
| contributor-check | contributor attribution | action_required | Action required |
Jobs that find nothing emit [] (empty array) — no noise info items.
A single comment-live job polls the GitHub Actions API every 15s,
classifies jobs into (completed, pending), assembles the comment, and
upserts it. Merges review_status outputs from all needs jobs via
toJSON(needs.*.outputs.review_status), and downloads the ci-timings
artifact when it becomes available. Shows commit SHA + message below
the header.
The assembler has ZERO job-specific knowledge. It just:
1. collect_from_statuses() — flattens all nested status objects into ReviewItems
2. collect_failed_jobs() — synthesizes errors for failed jobs with no declared status
3. _attach_job_urls() — fills in per-job log links for ALL items
4. render_comment() — groups by severity, renders with group headers
Each item shows links inline next to the title: View report (job-emitted
URL) and View job (auto-attached logs link). Each info item is its own
collapsible <details> block.
# ૮ >ﻌ< ა ci review
running on abc1234 — commit message first line
## ❌ Job failures
### {title} · [View job](url)
{summary}
## ⚠️ Action required
### {title} · [View job](url)
{summary}
**How to fix:**
{how_to_fix}
## ⚠️ Warnings
### {title} · [View report](url) · [View job](url)
{summary}
{detail}
<details><summary>{title}</summary>
{content}
</details>
Still running 3 jobs: ci-timings, docker
- test_assemble_review_comment.py (48 tests): collect_from_statuses,
collect_failed_jobs with exclude_sources, _attach_job_urls,
render_comment (group headers, inline links, commit info, per-item
details, pending footer), assemble integration
- test_live_comment.py (16 tests): classify_jobs pure function
- test_timings_report.py (10 tests): generate_review_status nested format
- test_lockfile_diff.py (6 tests)
- test_classify_changes.py (32 tests, pre-existing)
#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows and
#66577 restored the `|| github.token` fork fallback for detect-changes and
the label gates -- but it missed the ci-timings "Collect timings and
generate report" step, which still passes a bare AUTOFIX_BOT_PAT. On fork
PRs that PAT is empty, so timings_report.py hard-fails at
expect_env("GITHUB_TOKEN") before it can reach its own "degraded run must
never redden the PR" soft-fail path. Every fork PR gets a red run from this
advisory job (e.g. #66573).
- ci.yml: apply the same `secrets.AUTOFIX_BOT_PAT || github.token` fallback
to the timings step. github.token has `actions: read`, enough to read the
run's job/step durations on forks.
- timings_report.py: treat a missing/empty GITHUB_TOKEN as a degraded run
(TimingsUnavailable) instead of a hard ValueError, so this whole class of
failure can never redden a PR again even if a future workflow drops the
token. Still writes no JSON, so no empty baseline is ever cached.
git diff on a lockfile is unreadable: npm reorders entries, rewrites
integrity hashes, and moves packages between nesting levels, so a
one-line package.json bump produces a thousand-line textual diff.
scripts/ci/lockfile_diff.py instead parses the `packages` map out of
both versions of every tracked package-lock.json (via `git show`),
reduces each to {install path: version}, and set-diffs the maps —
reorder/hash churn vanishes, leaving only actual version movement
(added / removed / updated, with nested dedup copies tracked
separately).
The lockfile-diff workflow posts the result as a Markdown table in a
PR comment gated behind a hidden marker: subsequent pushes PATCH the
existing comment instead of stacking new ones, and a push that reverts
all lockfile changes updates the comment to say so. Advisory only —
never fails on findings; fork PRs (read-only token) degrade to a
warning.
Wired through the ci.yml orchestrator with a new npm_lock lane in
classify_changes.py (fails open on .github/ changes per the existing
contract).
Baseline bars were positioned at their absolute timeline offset, which
included the baseline run's own wait time — making duration comparisons
hard since the bars were visually offset. Now each baseline bar starts
at the same left position as its corresponding current job, so the two
bars directly overlap for at-a-glance duration comparison.
Also removed the now-unused bl_t0 / bl_max / bl_jobs_timed variables
that tracked the baseline timeline position.
Add wait time computation: for each job, wait_s = started_at - max(completed_at
of all jobs that finished before it started). This is a timestamp heuristic
(no workflow YAML dependency parse needed) that's accurate for pipeline-shaped
CI where the critical path is linear at each stage.
Shows up in:
- Job table: new Wait + Δ Wait columns
- Gantt chart: hatched bar segment before the run bar
- Step details: '(wait Xs)' annotation in the summary line
- Markdown summary: Total wait row with delta vs baseline
- Stats: total_wait / bl_total_wait in compute_stats
Also completes the skipped-jobs UI:
- Skipped stat card in stats cards
- Skipped row in markdown summary table
Backward compatible: old cached baselines without wait_s annotate on load.
Skipped jobs (conclusion == 'skipped') have null/zero-duration timestamps
that polluted every downstream computation: they counted as 'unchanged'
(0 vs 0) in faster/slower tallies, showed meaningless '0.0s (0%)' deltas
in the job table, rendered phantom gantt bars, and inflated wall/compute
totals.
Add is_skipped() helper and apply it consistently:
- compute_stats: exclude from wall/compute + faster/slower/unchanged;
add 'skipped' and 'bl_skipped' counts
- _gantt_bars: filter from current bars, baseline bars, and axis calc
- _job_table: show 'skipped' label instead of durations/deltas
- _step_details: skip entirely (no meaningful step data)
- _regressions: exclude from both current and baseline sides
The 'CI timing report' job is pure observability — it collects per-job/step
durations from the GitHub API after the run and publishes an HTML gantt
report + PR-vs-main timing diff. It gates nothing (all-checks-pass does not
include it), yet it could redden a PR: the script makes dozens of paginated
API calls with the shared repo GITHUB_TOKEN and had zero retry handling, so
a single 403 (rate-limit burst when several PRs run CI concurrently) failed
the job. Observed twice in a row on PR #59805.
- api_get(): retry 403/429/5xx and connection errors with exponential
backoff, honoring Retry-After / X-RateLimit-Reset (max 5 attempts, 120s
cap). Non-transient statuses (404 etc.) still fail fast.
- main(): exhausted retries raise TimingsUnavailable, caught to emit a
degraded summary line + placeholder HTML artifact and exit 0 — a metrics
collector must never fail the PR's checks. No timings JSON is written on
the degraded path so an empty baseline can never be cached.
- ci.yml: baseline-save steps on main skip gracefully when no JSON exists.
Verified with a mocked urlopen harness: retry-then-success (3 attempts),
exhausted-retries -> TimingsUnavailable, 404 fails fast without retry,
degraded main() exits 0 with summary + placeholder and no JSON, and the
--from-json happy path is unchanged.
ci: centralize path-gating behind single orchestrator + all-checks-pass
gate
Replace the scattered per-workflow detect-changes pattern with a single
ci.yml orchestrator that runs the classifier once, then conditionally
calls sub-workflows via workflow_call based on lane outputs. A final
all-checks-pass job (if: always()) aggregates all results so branch
protection only needs to require one check.
Changes:
- New .github/workflows/ci.yml orchestrator (detect + conditional calls
+ all-checks-pass gate)
- Extend classify_changes.py with scan/deps/mcp_catalog lanes, absorbing
supply-chain-audit's internal changes job
- Update detect-changes/action.yml to expose the new lane outputs
- Convert all 10 PR-gated sub-workflows to workflow_call-only triggers,
removing their push/pull_request triggers and per-step detect-changes
guards (gating now happens at the orchestrator level)
- lint.yml + supply-chain-audit.yml receive event_name as a
workflow_call
input to replace github.event_name (which is "workflow_call" inside
called workflows)
- supply-chain-audit.yml: remove internal changes job + *-gate jobs
(orchestrator handles gating, booleans arrive as inputs)
- contributor-check.yml: remove internal filter step
- Update test_classify_changes.py for 6-lane output + new supply-chain
test cases
Heavy PR checks run on every PR because the workflows deliberately avoid
`on.paths` filters — a path-gated workflow leaves its required check pending
forever when no matching file changes, blocking merge. So a docs-only PR
still spins up the TypeScript matrix, the full Python suite, and ruff/ty.
Keep every workflow triggering on every PR (checks always report) but gate
the expensive *steps* on what the PR touches. Skipping a step (not the job)
leaves the job green, so required checks never hang — the same idiom already
proven in contributor-check.yml.
A classifier (scripts/ci/classify_changes.py) maps the PR diff to three
lanes — python, frontend, site — surfaced as step outputs by a composite
action (.github/actions/detect-changes). Fail-open: an empty diff or any
.github/ change runs everything; python is a denylist (skipped only when
every file is provably prose or a frontend-only package); skills/**/SKILL.md
counts as python-relevant since the skill-doc tests read that tree. Non-PR
events always run the full pipeline.
