How to Evaluate Candidates More Consistently
By Vriti AI Interview team · Published · 4 min read
Consistent evaluation comes from four habits. Decide what good looks like before you meet anyone. Ask every candidate about the same things. Score each answer against written anchors, on your own. Only then discuss the candidate with the rest of the panel. Each habit closes off a different way that personal impressions creep into a hiring decision.
Why evaluations drift
Put two experienced interviewers in front of the same candidate and they will often come out with different conclusions. Neither is careless. Unstructured judgement is simply vulnerable to a handful of well-documented effects:
- Different questions. If each interviewer asks what occurs to them, they are measuring different things and cannot compare results.
- The halo effect. One strong impression, such as a confident opening or a prestigious former employer, colours the rating of everything else.
- Similarity bias. We rate people who remind us of ourselves more warmly.
- Contrast effects. A solid candidate looks weak after an outstanding one, and strong after a poor one.
- Memory. Notes written two days later record what the interviewer felt, not what the candidate said.
- Anchoring in the debrief. The first or most senior voice in the room sets the frame, and everyone else adjusts towards it.
None of these is solved by telling people to “be objective”. They are solved by process.
Define the criteria before the first interview
Agree with the hiring manager what the role needs, in terms someone could observe. Aim for four to six criteria. Fewer than that and the criteria are too broad to score. More than that and nobody can hold them all in mind during an interview.
Write each criterion as a behaviour, not a trait. “Breaks an unfamiliar problem into steps and checks assumptions before committing” can be scored. “Smart” cannot. Mark which criteria are must-haves, so a weak result on one of them is not quietly averaged away by strong results elsewhere.
Write anchored rating scales
A five-point scale with no descriptions means a 3 from one interviewer and a 3 from another can be completely different judgements. Behaviourally anchored rating scales fix this by describing what an answer at each level actually sounds like. You do not need to describe every point. Anchoring the low, middle and high points is usually enough.
| Score | What the answer sounds like |
|---|---|
| 1 – Weak | Jumps to a solution without clarifying the problem. Cannot explain why it would work. Does not consider alternatives. |
| 3 – Solid | Asks a clarifying question or two, proposes a reasonable approach, and names at least one trade-off when prompted. |
| 5 – Strong | Frames the problem first, compares options on explicit criteria, identifies risks unprompted, and explains how they would check the result. |
Test your anchors by giving two colleagues the same recorded answer and asking them to score it separately. If their scores differ by more than a point, the anchors are too vague.
Ask every candidate about the same things
Consistency does not mean reading from a script in a monotone. It means every candidate gets the same core questions, aimed at the same criteria. Follow-up questions are welcome, as long as they dig deeper into the criterion being assessed rather than drifting onto a topic you happen to find interesting. Our comparison of structured and traditional interviews goes further into what to standardise and what to leave flexible.
Take evidence notes, not impressions
Write down what the candidate said and did, as close to verbatim as you can manage, and score afterwards. Compare these two notes:
- Impression: “Seemed really confident about system design.”
- Evidence: “Proposed a queue between the services. When asked about failures, said messages would be retried but did not mention idempotency until prompted.”
The second note can be checked against the anchors, defended in a debrief, and understood by someone who was not in the room. The first can only be agreed with or argued about.
Score independently, then discuss
Every interviewer submits scores and notes before the debrief and before seeing anyone else’s ratings. This single rule removes most of the anchoring problem. In the debrief itself:
- Start with the criteria where scores disagree by more than a point. Skip the ones where everyone agrees.
- Ask each interviewer to share the evidence behind their score, not just the number.
- Let the most junior interviewer speak first, so seniority does not set the frame.
- Change a score only if new evidence justifies it, and note why.
Calibrate regularly
Even with anchors, interviewers drift over time. Every few weeks, look at how each interviewer’s scores are distributed. Someone who rates almost everyone a 4 or 5 is not helping you tell candidates apart. Then score a recorded interview together and talk through any differences. Calibration sessions are also the fastest way to train new interviewers.
Where software helps, and where it does not
Software is good at the parts of consistency that people find tedious. It can ask every candidate the same core questions, apply the same weighted criteria to every answer, and keep a full transcript so every score can be traced back to what was said. That is the approach behind how Vriti AI Interview evaluates answers: each criterion is scored with transcript evidence attached, and must-haves can carry minimum scores.
What software cannot do is decide which criteria matter for your role, or take responsibility for a hiring decision. Those stay with your team, and so does the habit of checking the evidence behind a score before acting on it.