feat(report-ui): local report viewer for iterating on evals - #98
feat(report-ui): local report viewer for iterating on evals#98astahmer wants to merge 48 commits into
Conversation
Keep the selected case, drawer tab, and ledger filters shareable so a report link opens the same view.
Let inspectors order cases by status, name, score, duration, tokens, or tools without losing the original report order by default.
Turn passed/failed/skipped counts into real links so a click jumps straight to that slice of the report.
Give the Raw tab a Chrome-style inspector plus a text toggle so large case payloads are readable.
Make it obvious that the clipboard write succeeded without leaving the report.
Parse assertion headlines, diffs, and stacks so a failed eval is readable instead of one dumped string.
Jump from the drawer file path into VS Code, Cursor, or Zed at the failing location.
Bundle the failing case, judge evidence, and raw JSON into a paste-ready brief with short fix instructions.
|
@astahmer is attempting to deploy a commit to the Sentry Team on Vercel. A member of the Team first needs to authorize it. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d717760. Configure here.
| return null; | ||
| } | ||
|
|
||
| const { pricing, workspaceRoot } = useReportMeta(); |
There was a problem hiding this comment.
Drawer hook crashes on open
High Severity
useReportMeta runs only after the open / testCase early return, so opening or closing a case changes the hook count. React then throws and the ledger click path that should open the drawer takes down the report page.
Reviewed by Cursor Bugbot for commit d717760. Configure here.
| meta.workspaceRoot, | ||
| ) || "Eval report"; | ||
| const selectedCase = resolveSelectedCase(search.case, workspace.cases); | ||
| const isDrawerOpen = Boolean(search.case && selectedCase); |
There was a problem hiding this comment.
Rerun loses the open case
Medium Severity
Selection is stored as search.case (the case id), but a finished re-run replaces the row with the incoming dump identity keyed only by file + fullName. The new id does not match the URL, so selectedCase disappears and the drawer closes after the job the user was inspecting.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d717760. Configure here.
| (looksLikeEvalFile(request.file) ? EVAL_VITEST_CONFIGS[0] : undefined); | ||
| if (config) { | ||
| parts.push("--config", shellQuote(config)); | ||
| } |
There was a problem hiding this comment.
Confirm command invents eval config
Medium Severity
formatRerunCommand always appends --config vitest.evals.config.ts for eval files, while prepareRerun only adds a config that actually exists (and may pick .mts / .js / vitest.eval.config.ts). The confirm dialog and its copy button therefore show a command that is not the one the server runs, and a copied command can fail to find that config.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d717760. Configure here.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
| current.map((job) => | ||
| job.id === event.jobId | ||
| ? { ...job, log: `${job.log}${event.chunk}` } |
There was a problem hiding this comment.
Bug: The client-side log state in RerunSession grows without bounds by appending log chunks, potentially causing performance issues for long-running, verbose jobs.
Severity: LOW
Suggested Fix
Implement a client-side trimming mechanism for the job.log state within the Server-Sent Events handler. The log should be trimmed to a reasonable limit, such as the 32KB limit used by the server, to prevent unbounded memory growth.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: packages/report-ui/src/app/components/RerunSession.tsx#L82-L84
Potential issue: The client-side component `RerunSession` receives log chunks from the
server and appends them to the `job.log` state. While the server trims the log to the
last 32KB before sending, the client-side code continuously appends chunks without any
trimming mechanism. In the edge case of a long-running job with verbose output, this can
lead to unbounded growth of the log string in the browser's memory, causing performance
degradation and potential slowdowns. The log is reset to the trimmed version upon page
refresh or network reconnection.
Did we get this right? 👍 / 👎 to inform future reviews.
|
hey! I just recently migrated from evalite to vitest-evals and the experience seems nice here but I noticed I could make a bunch of small improvements, so I started a few prompts here & there would love for it to be merged but I have to say I haven't really bothered looking at the code at all; I figured if you want it written a certain way it would probably be easier/faster to do it on your side. and if you're limited on tokens, feel free to send me the prompt and I can just send grok 4.6 high on it again if that's fine with you |
1 participant
What
Turns the local
vitest-evals servereport from a thin result dump into something you can actually iterate on: scan failures, inspect a case, estimate cost, and re-run one eval without leaving the page.This is the combined tree vs
main(0.16.1). Screenshots are from a real 17-case eval dump (15 passed / 2 failed).Ledger and cost
Official report is a short case list. This PR keeps a compact header (pass rate, failed/passed links, runtime, estimated cost, average score) and a sortable ledger with Model / Score / Duration / Tokens / Cost / Tools. Failed cases sort first. Status, search, and removable chips stay in one toolbar.
Cost is estimated from models.dev rates (including cached-token splits when usage has them). The
?next to Cost shows the breakdown.Outcome counts in the header are links. Filtering updates the URL (
?status=failed), the ledger count (2/17), and a dismissible chip. Header totals stay on the full workspace, not the filtered subset.Also in this group (not every one is a separate screenshot):
/focuses the box)Case drawer
Click a row. The drawer keeps facts that exist (status, score, model, duration, tokens, cost) and drops empty chrome. Failures are listed as scan-able messages. Copy as Markdown for an agent, copy a failure without the whole JSON, and open the case file in a local editor (absolute path available too).
Transcript is unchanged in spirit — messages + tool calls — just sitting next to the new facts.
Raw is a collapsible JSON tree (opens three levels deep) instead of a single blob, with copy feedback.
Inspection state (selected case, tab, filters, sort) lives in the URL.
Command palette
⌘K/Ctrl+Kfor filters, jumping to a visible case (uncapped), and exports (Markdown, JUnit, pull-request comment).Keyboard:
j/kmove cases,Escclears filters / closes overlays.Re-run one case
From the ledger (or palette): confirm the exact
vitestcommand, copy it, then start it on the local report server (POST /api/rerun, logs over SSE).Eval suites (
*.eval(s).ts,__eval__/) spawn with--config vitest.evals.config.tswhen that file exists. Default Vitest include is*.speconly, so without the config the re-run says “No test files found”.A bottom-right task tray streams the job. Running rows show a spinner and a short clock (
4s,1m 04s) in both the tray and the ledger duration cell. Cancel sends SIGTERM then SIGKILL. When the job ends, the dump is re-read and merged back onto the previous workspace byfile + fullName, so one re-run does not drop the other cases in memory. (The Vitest JSON reporter still overwrites the file on disk.)Reload in the header/tray is user-owned — the page does not watch the dump file.
CLI / CI extras
vitest-evals serveis unchanged as the entry. Optional:pnpm exec vitest-evals serve results.json --workspace /repo --junit junit.xml --comment pr.mdJUnit + a Markdown PR comment are also exportable from the palette.
Notes for reviewers
feat/report-ui-megamerge) of small feature commits, all vs this repo’smain.@vitejs/plugin-reactbumped for Vite 8.pnpm exec tsc --noEmit -p packages/report-ui/tsconfig.jsonclean.Test plan
pnpm exec vitest-evals serve <dump.json> --workspace <repo>?popover wraps and matches the header total⌘K: jump to a case, export comment/junit