Evaluating an agent that applies for jobs on my behalf

How I decided whether to trust an autonomous pipeline with my own applications, graders, traps, and what "good enough" meant.

In an early run, before I had the hallucination grader in place, the tailoring step rewrote a line from "familiar with Kubernetes" to "led Kubernetes migration for a 40-node cluster" because the posting mentioned Kubernetes twice. It read completely plausibly. I only caught it because I happened to know the posting's employer and knew that claim would come up in an interview I couldn't back up.

That's the problem with an agent that applies for jobs on my behalf: its failures ship under my name. A hallucinated skill on a tailored résumé isn't a bug ticket, it's me lying to an employer. So before I let the pipeline run without me watching every application, I needed an answer to a boring question: how do I know it's good enough?

What the agent does

The pipeline discovers postings from ATS boards (Greenhouse, Lever, Ashby), normalises them into one schema, scores each against my profile, tailors my résumé per posting using Claude tool use with schema-validated output, and fills the application form with browser automation. A small dashboard lets me review, approve, and audit each step. TypeScript monorepo, Nest.js, Next.js, Postgres, Docker.

The step that can hurt me is tailoring. Discovery and form-filling fail loudly, a missed posting or a broken selector is annoying but visible. Tailoring fails quietly: the output is always a plausible-looking résumé. You can't eyeball your way to trusting it at volume.

Four graders, ordered by cost

I grade every tailored résumé with four checks, cheapest first. Each one exists because it catches a failure the previous one can't.

1. Schema. Does the output validate against the expected structure, sections present, dates well-formed, required fields filled? Free to run, catches the dumbest failures, and a surprising share of them. roughly 15% of early failures were plain schema issues, malformed date ranges and empty required fields, mostly.

2. Hallucination. Every factual claim in the tailored résumé must trace back to my source profile. The grader diffs claims against the source: a technology I never used, an employer I never had, a metric I never produced, instant fail. This is the one that protects my name. It's also the one where "mostly right" is worthless; the pass bar is absolute.

3. Keyword coverage. Did the tailoring actually do its job? A résumé can be perfectly truthful and completely untargeted. This grader checks that the posting's required skills that I genuinely have appear in the output. Truthful-but-generic scores low.

4. LLM judge. A model scores the result against a rubric, does it read like a strong, coherent résumé for this posting, not a keyword dump? The judge is the most expensive and least trustworthy grader, so it runs last, and it can only lower confidence, never override a hallucination fail.

The ordering matters: deterministic checks gate before the judge ever runs, so the expensive, fuzzy opinion is only consulted about outputs that are already structurally valid and factually grounded.

Trap cases

A test set built only from normal postings tells you how the agent does on a good day. I added traps:

• postings that ask for skills I don't have, the correct output omits them, and a grader that only rewards keyword coverage would push the agent to lie;
• senior-titled postings with junior-level scope, to check the agent doesn't inflate my title or years of experience to match;
• near-duplicate postings from the same company reposted weeks apart, to check the agent doesn't apply twice under slightly different tailoring and trip an ATS dedup flag.

Traps are where the hallucination and coverage graders pull against each other, and that tension is the whole point: an agent optimised on coverage alone learns to fabricate.

What "good enough" meant

Schema and hallucination checks had to hit 100% pass over a 120-case test set, coverage had to average ≥ 85%, and the judge had to average ≥ 8/10, before I stopped reviewing every application. I compared Haiku, Sonnet, and Opus for the tailoring step itself, Haiku missed too many hallucinations upstream of the grader catching them, Opus was overkill for the cost, so Sonnet went to production.

The human stayed in the loop at one place: final submit. Every other step, discovery, scoring, tailoring, form-fill, runs unattended, but nothing actually goes out to an employer without my click, because that's the one action I can't undo.

Takeaways

• If an agent acts under your name, hallucination checks aren't one grader among several, they're the gate everything else sits behind.
• Order graders by cost. Deterministic checks first mean the LLM judge only sees candidates worth judging, and your eval run stays cheap enough to run on every change.
• Build traps where your graders conflict. That's where the agent's real incentives show.
• "Good enough" is a number you commit to in advance, not a feeling after a demo.