Sooner or later, someone will ask you this. A client, a board member, a reviewer reading your report. The study was AI-moderated: an AI interviewer followed up on open answers, and now they want to know: how do you know it didn't put words in people's mouths?
It's the right question, and it isn't new. Researchers have always had to show their workings. A discussion guide is agreed before the groups. Clients watch from behind the glass. Recordings are kept. A good moderator can explain why they asked what they asked, and a bad one gets found out.
An AI moderator needs the same accountability as a human one, for the same reason, for the same reason. It's making judgement calls in the moment: whether to follow up, what to ask, when to stop. If all you can see at the end is a transcript, you're taking those judgements on trust, and asking your client to take them on trust from you.
This guide covers what "leading" looks like in practice, the six things you need to see to rule it out, and the questions to ask any platform before you rely on it.
What leading actually looks like
Leading is rarely one dramatic question. More often it's one of these:
- Assuming the premise. "What frustrates you most about the app?" when they haven't said anything frustrates them.
- Supplying the words. "Would you say it felt cheap?" The respondent agrees, and "cheap" becomes a theme they never actually raised.
- Chasing the hypothesis. Following up hard on answers that support what the researcher expected, and letting the others pass.
- Not knowing when to stop. Asking a third and fourth time until the respondent says something, anything, to move on.
- Probing a clear answer. Treating a complete answer as a vague one, which tells the respondent they got it wrong.
- Wandering off the question. Following up on something from three questions ago, so the answer ends up stored under a question it doesn't belong to.
Human moderators do all of these on a bad day. An AI interviewer can do them at scale, consistently, across hundreds of interviews, which is exactly why you need to be able to check.
The six things you need to see
1. The brief, written before fieldwork
A good probe starts with an instruction a researcher wrote on purpose, for that question. Not a generic "ask them to tell you more", but a brief: what to find out, what to leave alone, and when to stop.
For example, our own study on AI shopping assistants ended with an open question: "In a sentence or two, what would an AI shopping assistant have to do to earn your trust?" The brief for the follow-up was:
Only if the answer is vague or very short (e.g. "be honest", "don't know"), ask once for a concrete example of what that would look like when it is shopping for them. Keep it light and brief; do not probe a clear answer.
The objective was "a specific, quotable statement of what the assistant would need to do to be trusted", and the limit was one follow-up.
That brief is the equivalent of the discussion guide. It can be reviewed, challenged and agreed before a single respondent sees the study. If the platform can't show you the brief for each question, you can't tell whether the follow-up did what was intended.
2. Every follow-up, word for word, next to the answer that prompted it
Here's what that brief produced in a real interview. A woman aged 35 to 44 answered:
Be very honest with me.
That's exactly the kind of answer the brief anticipated, so the interviewer asked once:
What would 'very honest' look like when it's actually shopping for you — any example?
She answered:
Telling me everything they are doing to help.
You can judge that follow-up for yourself. It used her word, "honest", rather than offering one of its own. It asked for an example, and didn't suggest what one might be. It asked once, then stopped. "Telling me everything they are doing" came from her.
Now compare a man aged 45 to 54, who answered:
Get me what I want and verify before purchase, the first time.
That's already specific, so the interviewer didn't follow up at all. Leaving a clear answer alone is as much a part of good moderation as probing a vague one.
Both judgements are only possible because each follow-up, or the absence of one, was kept word for word next to the answer. A transcript would show the same words, but buried in a conversation that you'd have to read end to end.
3. The instruction exported with the data, not just the transcript
This is where most audit trails fail. The follow-ups exist somewhere, but the export is a wall of text, or the probe answers are merged into the original answer.
What you need is question-by-question output: the original answer, each follow-up question, each follow-up answer, and the instruction that produced it, in separate columns. Then you can:
- show a client exactly what was asked to get any quote in the report
- read the original answers on their own, before any follow-up, and compare
- filter to every follow-up on one question and read them together, which is the fastest way to spot a pattern of leading
It also means the qualitative depth arrives in the same shape as the quantitative data, so it can be analysed alongside it rather than sitting in a separate document.
4. A stopping rule you can see
Every follow-up should run to an objective and a limit. The objective says what counts as enough ("keep going until you have both the trigger and the mood, then stop"). The limit says how many times it can ask before it gives up.
Both should be visible in the study's settings and in the audit trail, so you can check that a respondent who gave a clear answer wasn't asked again, and that nobody was asked a fourth time just because the third answer was short.
5. The respondent's words, not yours
Read a sample of follow-ups and ask one question: where did the key word come from? If the theme in your report first appears in the interviewer's question, not the respondent's answer, you have a leading problem.
A useful habit is to brief the interviewer to build on the respondent's own words ("you said it felt rushed, what made it feel that way?") rather than introducing new ones. Then check that it did.
6. A human reading it early
None of the above replaces a researcher reading the interviews. The cheapest time to catch a leading follow-up is at soft launch, when thirty interviews are in and the brief can still be changed. Read every one. If the follow-ups are drifting, chasing or over-probing, fix the brief before the other few hundred interviews happen.
This is also where you'll catch the interviewer getting it wrong in ways that aren't about leading at all, such as asking about a different question or following up more often than its limit allows.
What we built, and why
When we built the interviewer into Redge, we built the audit trail first, because we wanted the data to be usable. It turned out that usable and accountable were the same thing.
In Redge:
- every question that has follow-ups switched on has its own brief, written by the researcher, with an objective and a maximum number of follow-ups
- the brief can tell the interviewer to acknowledge what the respondent said, build on their words, or draw on something they said earlier in the interview
- the original answer and every follow-up are stored separately, in order
- exports are question by question, with each follow-up and the instruction that produced it alongside the answer, not a single transcript
- the report's quotes can be traced back to the exact exchange they came from
None of this means the interviewer is never wrong. It means that when it is, you can see it, and so can your client.
Questions to ask any AI-moderated research platform
Whichever tool you use, ask these before you rely on its follow-ups:
- Can I write a different follow-up brief for each question?
- Can I set when it stops, and see that it did?
- Is the original answer stored separately from the follow-up answers?
- Does the export include each follow-up question and the instruction behind it, or only a transcript?
- Can I trace a quote in the report back to the exact exchange?
- Can I read and change the brief after soft launch, before full fielding?
If the answer to any of these is no, the honest answer to "how do you know the AI didn't lead the respondent?" is "we don't". That's worth knowing before your client asks.
Want to see the audit trail on a study of your own? Book a 30-minute walkthrough, or see what's in the platform. Buying a platform? Start with Buying an AI survey platform in Australia: twelve questions to ask.