llm-evalkitin progress
TypeScript eval harness comparing Haiku / Sonnet / Opus on résumé tailoring: schema, hallucination, keyword-coverage and LLM-judge graders with a cost/latency report. In progress — repo and report not yet published.
- Repo · soon
- Report · soon
Problem
"Which model should I use for résumé tailoring?" is an empirical question, not a vibes question. I needed a harness that answers it with graders I can defend and a cost/latency report I can act on.
What I built
- A dataset of posting + profile pairs and the expected tailored output shape.
- Four graders: schema (does output validate?), hallucination (does it invent facts not in the source profile?), keyword coverage (are the posting's required keywords present?), and LLM judge (rubric-based quality score).
- A runner that executes the same cases across Haiku, Sonnet, and Opus and produces a cost/latency report.
Architecture
Rendering diagram…
Result
Status: comparison run in progress. The per-model table — schema pass rate, hallucination-free rate, keyword coverage, judge score, cost per case, p50 latency for Haiku / Sonnet / Opus — lands here when the run is finalised. The harness and all four graders are in the repo now.
Stack
TypeScript · @anthropic-ai/sdk · Zod · Vitest