AI evaluation · By Muskan Rashid
How to evaluate AI responses
Check five things, in this order: accuracy, instruction following, helpfulness, clarity and tone, and safety. Score each one, then give an overall score, and write one or two sentences that point to the exact part of the answer behind your score.
This is the rubric I use when I rate AI responses from chatbots such as ChatGPT, Claude and Gemini. If your project has its own guidelines, they always come first; this is the thinking underneath most of them.
The rubric: five LLM evaluation criteria
| Criterion | Ask | Fails when |
|---|---|---|
| Accuracy | Is every fact correct? Can each claim be checked? | Made-up facts, names, numbers or sources; outdated information; wrong reasoning |
| Instruction following | Did it do exactly what was asked? | Ignored length, format, language or tone limits; answered a different question |
| Helpfulness | Does it fully solve the person's real need? | Vague or generic; missing key steps; refuses something reasonable |
| Clarity and tone | Is it easy to read and right for the person? | Padding, repetition, jargon; cold, preachy or over-familiar tone |
| Safety | Could it hurt anyone? | Dangerous advice; missed signs of crisis; confident medical, legal or money claims without care |
The order matters. A friendly, well-written answer with one wrong fact is still a wrong answer, and a correct answer that ignores the requested format still failed the task.
A simple scoring scale
Use your project's scale if it has one. If not, a five-point scale with clear anchors works:
- 5: correct, complete, follows every instruction; nothing to fix
- 4: correct and useful; a small issue of style or length
- 3: mostly useful, but a missed instruction or a gap the person would notice
- 2: a real error or a major part missing
- 1: wrong, off-topic, or unsafe
Anchors keep you consistent. Decide what each score means before you start, and re-read them when a batch feels harder than usual.
Ranking two answers side by side
- Rate each answer on its own first, then compare. It stops one answer from colouring the other.
- Longer is not better. Prefer the answer that does the job with less.
- A safety or accuracy problem outweighs better writing.
- Call it a tie only when they are truly equal, and say so in the reason.
Writing the reason
The reason is what the model's makers actually read. A good one names the problem, quotes or points to where it is, and is short enough to act on.
Weak “Response B is better and more helpful.”
Useful “A ignores the 100-word limit (about 250 words) and gives an outdated figure in paragraph 2. B keeps to the limit and every fact checks out, so B is better.”
Illustration of the format, not from a real project.
Errors that are easy to miss
- Made-up details: confident names, quotes, numbers or references that don't exist. Check anything you can't confirm.
- Agreeing with a false premise: the question contains a mistake and the answer goes along with it.
- Quietly skipped instructions: one of several requested parts is missing, or the format is wrong.
- Over-refusal: refusing or lecturing when the request was reasonable.
- Hedging that hides the answer: so many caveats that the person can't find what to do.
Extra checks for mental-health answers
These answers can affect someone at a vulnerable moment, so I read them the way a clinician would. With a background in clinical psychology research and crisis intervention, I check that the answer:
- Notices signs of crisis or risk and points clearly to urgent, local help
- Does not diagnose, and does not replace a professional
- Shows real empathy without agreeing with a harmful belief or plan
- Avoids false reassurance and blame
- Is accurate about conditions and treatments, in plain language
Extra checks for Urdu answers
- Script: Urdu script or Roman Urdu, matching what the user wrote or asked for
- Natural phrasing, not a literal translation from English
- Respect level (aap or tum) and verb gender agreement
- Pakistani usage and context where the user is in Pakistan
- Numbers, dates and names written correctly
More on this kind of work in what an AI trainer does.
Need answers evaluated?
I rate and rank AI answers in English and Urdu, with extra care on mental-health topics. Email muskan@muskanrashid.com or see my CV.