Ticket QA Scorecard & Sampling
Build an internal tool that fairly samples each agent's tickets, AI-pre-scores them against your quality rubric, and hands them to a reviewer to finalize - so support quality is measured systematically, not anecdotally.
A logged-in QA tool that draws a fair sample per agent, drafts pre-scores with evidence, lets a reviewer adjust and approve the final score, and reports per-agent quality trends over time.
Before you start
- A ticket export you can download as CSV or Google Sheet (agent, ticket content, outcome)
- Your written QA rubric and the weight of each criterion
- Free Vercel, Supabase, and Resend accounts (the plan walks you through each)
The problem this kills
Right now your quality program probably runs on vibes. Someone "spot-checks a few tickets" when they have a spare hour, the same loud agents keep getting reviewed, and the scores live in a spreadsheet nobody trusts. When a manager asks "is the team actually getting better?" you don't have a defensible answer - just anecdotes.
The real problems are sampling and consistency. You can't review every ticket, so you need a fair sample - a few from every agent, every cycle, across ticket types - not whatever happened to land in your inbox. And even when you do review, two reviewers score the same ticket differently because the rubric lives in their heads. The result: a QA program that feels political instead of objective.
What you'll build
A small, private web app for your QA team. You load a ticket export, and the tool draws a fair, configurable sample for each agent. For every sampled ticket, an AI reads the conversation and drafts a pre-score against your rubric - one line per criterion (accuracy, completeness, tone, policy, or whatever you use), each with a short piece of evidence quoted from the ticket. A human reviewer then reads the draft, adjusts any item, and approves the final score. Only approved scores are recorded against the agent, and the tool charts each agent's quality trend over time.
The AI never owns the score. It does the tedious first pass so your reviewers spend their time judging, not hunting.
What's inside the Implementation Plan
The plan opens by interviewing you about your business - your current QA process, the systems and exports you use, your exact rubric and criteria weights, your sampling rules, your typical and peak ticket volumes, and your messy edge cases. It reflects a short tailored spec back to you for a thumbs-up, then shapes the data model, the sampling logic, and every build step around your answers - so you get a tool fitted to your team, not a generic template.
From there it walks you, prompt by prompt, through building the whole thing: the sampling engine, the AI pre-scoring pass with evidence, the reviewer screen with the human approval gate, score storage, the per-agent trend report, and email digests. Every build step ends with a ready-to-paste prompt.
The governance it includes (this is the point)
This is a tool that touches people's performance reviews, so it's built to be trustworthy from day one:
- Login so only your QA team can open it.
- Row-level security so each organization only ever sees its own data.
- A complete audit trail - who pre-scored, who adjusted what, who approved, and when.
- A hard human-in-the-loop approval gate - the AI only ever drafts; no score is recorded against an agent until a reviewer approves it.
- Duplicate guards - the same ticket can't be scored twice in the same review cycle.
Who it's for
QA analysts, team leads, and support managers running a quality program who want measurement that holds up to scrutiny - and who'd rather build exactly the tool they need than wrestle a rigid SaaS QA platform into their workflow.
You've got this - paste the first prompt and let the interview tailor it to your team.