Evaluate AI-moderated interview tools with a shared brief, complete calls, and evidence you can inspect. A feature list cannot tell you whether the moderator follows your methodology or whether the resulting findings survive review. The most useful test includes both interview quality and the work required to produce the final deliverable.
User Intuition provides AI-moderated qualitative research for agencies, consulting firms, and research teams. This guide is published by User Intuition; it is an evaluation framework, not an independent vendor ranking. Apply the same tests to User Intuition and every other platform on your shortlist.
Download the AI interview evaluation checklist (PDF) →
What should you prepare before evaluating platforms?
Prepare a live or recently completed brief, a discussion guide, participant criteria, and a description of the required output. Identify the decision the research will inform. Choose a brief your researchers understand well enough to recognize useful evidence, missed opportunities, and unsupported conclusions.
Separate essential requirements from preferences. If a project requires a specific language, participant source, data agreement, or interview format, failure on that requirement should not disappear inside a high average score. A convenient interface cannot compensate for a method that cannot answer the client’s question.
Use internal rehearsals to check setup and participant experience, then evaluate a small set of suitable participants. Record which conversations are rehearsals and which are research evidence. A colleague acting as a difficult respondent can expose workflow problems, but their invented answers should never enter the client findings.
For an agency evaluation, add the commercial context: who operates the platform, how client materials are handled, what the researcher reviews, and how the output becomes a presentation. The User Intuition agency workflow and sample study show concrete starting points for that assessment.
What are the ten checks that matter?
1. Guide adherence
Check whether complete interviews address the intended learning goals. Distinguish mentioning a topic from producing evidence that answers it. A participant may hear every planned question and still provide too little information to support the decision. Record those gaps instead of treating question completion as research success.
Confirm that your guide can be preserved where consistency matters. Where follow-ups are flexible, inspect whether they remain within the brief. Test changes to wording and review their effect before launching broadly. Keep the guide version alongside the resulting evidence.
2. Useful, neutral probing
Useful probing follows what the participant said and clarifies a concrete experience, reason, tradeoff, or contradiction. Read the exchange before and after an interesting quote. Check whether the moderator supplied an interpretation that the participant merely agreed with, or whether the participant introduced the meaning themselves.
There is no universal number of follow-ups that makes an interview qualitative. Ask whether each probe adds relevant evidence without repeatedly steering or exhausting the participant. Test short answers, uncertainty, and refusals as well as fluent, detailed answers. The right response to some questions is to move on.
3. Modality and participant experience
Choose voice, video, or chat according to what the research needs to capture. Test the actual device, connection, and stimulus flow your participants will use. Observe interruptions, recovery from errors, consent, and how easily someone can stop the session.
Chat is the lowest-friction option for participants and the easiest to analyze across many conversations, but it loses tone and pace. Voice captures hesitation and conviction. Video adds expression and context but raises participant drop-off and review effort. Ask whether moderation quality holds across modes or was built for one and adapted to the others.
Voice and video can provide context beyond the transcript, but inferred emotion should remain open to researcher scrutiny. A polished summary of tone is not a substitute for checking what was said. For usability work, inspect the task and screen-sharing experience rather than assuming every interview platform supports the same workflow.
4. Sample fit and quality
Ask how participants are sourced, screened, and checked for duplicates or unsuitable responses. Test your exact eligibility criteria. Panel size is a reach indicator, not evidence that a particular role or experience is available. Keep the source and exclusions documented in the study record.
User Intuition supports your participants or its 4M+ vetted panel. For client customer research, bringing the client’s sample may be appropriate. For specialty audiences, confirm feasibility and price before launch. Include your own recruitment effort and participant incentives in the comparison when you supply the sample.
5. Analysis and evidence traceability
Pick a finding and follow it back to the relevant interviews. Check whether the supporting quotes mean the same thing in context and whether contradictory cases were retained. Review the difference between participant statements, generated interpretation, and the recommendation written by the researcher.
Inspect denominators when the output reports theme counts. Were participants asked comparable questions? Was the topic raised spontaneously or prompted? A percentage within an interviewed sample should not silently become a population estimate. The agency guide to scaled qualitative research explains that distinction.
6. End-to-end timing
Measure time from the approved brief to a reviewed deliverable. Record setup, recruitment, interviews, processing, researcher review, and client revisions separately. A vendor’s fieldwork example may start after contracting and recruitment; your deadline often starts earlier.
Do not promise a client delivery date from a headline turnaround number alone. Ask what depends on sample availability, language, specialist recruitment, and client approval. During the pilot, record both elapsed time and staff time so you can see where delays and effort arise.
7. Complete project cost
Compare the cost of usable research at the same scope. Include the platform, sample, participant payments, required subscription, review labor, and reporting work. Check minimum commitments, included credits, expiry, replacement rules, and any paid services your study needs.
User Intuition Starter voice interviews cost $30 with your sample or $60 with standard panel recruitment, with no monthly fee. Specialty audiences are quoted separately; incentives you arrange for your own sample are additional. Subscription terms are available on the pricing page. See current pricing.
8. Language and local meaning
Test interviews in the languages you plan to use. Ask a fluent reviewer to inspect the original conversation and transcript, including specialist terms and local references. A translated summary can conceal a misunderstood question or an ambiguous response.
User Intuition supports interviews in 80+ languages. Treat that as a capability to test for your audience and guide. Agree who reviews translated material and how quotations will be presented in the client readout, particularly when a small wording difference changes the interpretation.
9. Data handling and client boundaries
Review the vendor’s current documentation for access controls, retention, deletion, data processing, subprocessors, and model-training policies. Involve the people responsible for your client requirements. Record answers and unresolved items rather than assuming a security badge covers every use case.
For agencies, test how projects and access are separated across clients. Confirm who can see recordings, transcripts, and exports. Plan what happens when a project ends or a researcher leaves. The relevant pass/fail criteria come from the data and agreements involved in the actual brief.
10. Reuse across projects
Ask the platform to retrieve an earlier finding and show its source. Then introduce a new study with a conflicting result and inspect whether that difference remains visible. Reuse is valuable when it preserves context, date, and sample; it is misleading when distinct studies become a single undifferentiated answer.
User Intuition’s Customer Intelligence Hub makes study evidence searchable across projects. Evaluate it using your own workflow and access requirements. For agency work, permission to access one client’s research does not automatically imply permission to reuse it for another client.
Why does a feature list not separate AI interview platforms?
Three problems make a demo or a feature list a weak basis for choosing a platform.
- The demo problem. A vendor demo controls the script, the participant and the topic. It shows the product at its smoothest, not on a participant who gives evasive, contradictory or emotionally charged answers, which is where moderation quality matters most.
- The feature-list problem. Adaptive questioning, multilingual support and fast turnaround appear on every vendor’s page, and none of these terms has an industry definition. One product interprets what the participant said and writes a new follow-up. Another selects from a branching tree. Ask what happens when two participants give different answers to the same question.
- The methodology gap. Research teams often lack the vocabulary to interrogate how a moderator works, so positioning language goes unchallenged. The questions below give you concrete, testable prompts.
How should you run a vendor demo?
Bring your own scenario, play a difficult participant and read the transcript afterward.
Before the demo
- Use a question you already understand. Choose a recent project where you know what the interviews should uncover, so you can judge whether the AI finds it.
- Prepare a difficult participant. Ask to play the respondent yourself: give short answers, contradict an earlier statement, say “I don’t know” three times, or offer a socially desirable answer that conflicts with the behavior you described.
- Write down the sequences you want to see. For example, how the moderator responds to a vague answer or to a participant who drifts off topic.
During the demo
Signs of useful moderation:
- Follow-ups could not have been written before the participant answered.
- The moderator notices a contradiction between an earlier and a later statement.
- It pursues hedging or enthusiasm, not only topical keywords.
- It redirects an off-topic answer without cutting the participant off.
- At least one topic reaches several levels of probing, from the event to its consequence to what it meant to the participant.
Warning signs:
- Every participant receives the same follow-up.
- The moderator moves on after a vague answer.
- Follow-ups repeat a template such as “Can you tell me more about that?”
- The vendor will not let you play the participant.
After the demo
Request anonymized transcripts from real studies and read them. Moderation quality is most visible in the transcript, not in the polished deck. Look for the moment a human moderator would have probed differently, and check whether follow-ups are specific to what the participant said.
User Intuition ladders 5–7 levels in 20–30 minute interviews, and its sample study shows complete transcripts you can read before any demo. A laddered exchange moves from a surface statement (“I stopped using the product”) to a functional reason (“the onboarding was confusing”), a personal consequence (“I missed a deadline”), and what that meant to the person (“I felt I had let my team down”). Apply the same reading to every vendor.
What methodology questions should you ask every vendor?
Ask about the moderator, the analysis and the validation, and record the answers in writing.
The moderator
- How does the system decide to probe deeper rather than move on? The answer shows whether depth is judged from the participant’s answer or set by a timer or question count.
- Does the system generate each follow-up from the response, or select from a library? This sets the ceiling on adaptation.
- Can you pin required questions verbatim and set rules for what the moderator may not do?
- How does it handle participant distress and sensitive topics? Health, money and loss need guardrails beyond changing the subject.
The analysis
- Can any finding be traced to the specific participant quotes that support it?
- How does the platform treat contradictory findings inside one study? Mature analysis surfaces them as findings instead of averaging them out.
- Does analysis separate what participants said from what the system inferred?
The validation
- What evidence compares the AI moderator with human moderation, what was measured and who reviewed it?
- What are the average interview completion rate and duration? Completion signals participant experience, and duration signals whether topics were explored or rushed.
- Can you speak to a team that moved from human-moderated work to the platform?
How should you score the shortlist?
Weight the ten checks to your brief, score after the demos, and require evidence for any score above three.
| Check | Suggested weight |
|---|---|
| Guide adherence | High |
| Useful, neutral probing | High |
| Modality and participant experience | Medium |
| Sample fit and quality | High |
| Analysis and evidence traceability | High |
| End-to-end timing | Medium |
| Complete project cost | Medium |
| Language and local meaning | Varies by brief |
| Data handling and client boundaries | Pass or fail |
| Reuse across projects | High for ongoing programs |
- Customize the weights. A domestic project may rank language low, and a continuous program may rank reuse high.
- Score after the demos. Take notes during the session and score afterward, comparing vendors side by side to avoid recency bias.
- Ask for evidence above a three. A four or five needs something you saw, read in a transcript or confirmed with a reference customer.
- Read the pattern, not only the total. A vendor with a five on probing and a two on cost suits high-stakes depth work. A steady three suits high-volume programs.
- Pilot the top one or two. No framework replaces a real study on a real question.
Avoid four common mistakes: crediting roadmap features, over-weighting cost per interview when cost per usable finding is the better measure, under-weighting reuse across projects, and treating data handling as partial credit.
How do you start the evaluation?
- Assemble the team. Include a researcher, someone from procurement or operations, and a stakeholder who will use the output.
- Shortlist three to five platforms. More creates fatigue without improving the decision.
- Prepare shared scenarios. Use the same brief, difficult-participant persona and rubric for every vendor.
- Score independently, then calibrate. Each reviewer scores alone before the group discusses, so the most senior voice does not anchor the result.
- Pilot the leader. Run a paid study on a live question before committing to an annual contract, and judge setup, analysis, deliverables and support as well as the interviews.
What does an annotated User Intuition call show?
The public sample study contains complete phone-purchase and grocery-shopping interviews, plus a presentation from a 43-participant shopper study. The examples below refer to the published phone-purchase transcript. Exchange numbers count each displayed speaker turn from one, including the opening greeting.
| Exchange | What happens in the phone call | Evaluation note |
|---|---|---|
| 5-8 | The moderator asks what triggered the purchase, then asks for the sequence; the participant describes a phone breaking while cooking | A general reason becomes a specific event and context that a researcher can inspect |
| 9-12 | The moderator asks about a specific phone versus getting online quickly, then explores the chosen device | Useful detail follows, but the either/or wording could suggest a frame; test a more open prompt |
| 13-20 | Store choice leads into proximity, urgency, and the staff interaction | Follow-ups connect the purchase goal to concrete experience rather than stopping at a broad satisfaction label |
| 27-28 | The moderator asks what could improve; the participant suggests digital signs | The interview creates an opportunity for criticism after positive comments |
This is a review of the visible conversation, not a controlled validation study. The transcript includes disfluencies and apparent transcription errors. Listen to the recording when wording matters; do not silently polish a quote into a stronger claim. The original full guide is not reproduced here, so this example alone cannot certify complete guide adherence.
The second call supplies a useful challenge to overgeneralization. The phone buyer values staff assistance; the grocery shopper prefers shopping independently. An analyst should preserve that difference rather than merge both interviews into a universal recommendation for more staff contact. Context changes the implication of a superficially similar shopping experience.
The public calls demonstrate inspectable output. They do not establish performance for every audience, the prevalence of these preferences among all shoppers, or agency profitability. Use them to form concrete evaluation questions, then run your own guide with appropriate participants.
How should you record the evaluation decision?
Use an evidence log rather than a score based on impressions. For each criterion, record the requirement, what you observed, the supporting call or artifact, and any unresolved question. Assign a reviewer and a decision: ready, revise and retest, or unsuitable for this brief.
| Decision field | What to record |
|---|---|
| Requirement | The condition the client brief needs the tool to meet |
| Evidence | Complete call, transcript exchange, output, or documented term |
| Result | Meets requirement, needs revision, or does not fit |
| Remaining work | Researcher hours, guide changes, sourcing, or manual review |
| Next step | Owner and action required before fieldwork expands |
Agree the essential requirements before comparing vendors. Review observations independently before discussing them as a team. If reviewers disagree, return to the evidence and identify whether they interpreted the requirement differently. A high total score should never override a failed requirement that makes the project unusable.
After the first paid study, add actual spend, setup time, evidence-review hours, client revisions, and whether a second brief fits the workflow. These observations show whether the platform is becoming useful in practice. A good demonstration is a reason to test; a repeatable delivery process is a reason to expand.
Bring a client brief to the evaluation. Download the checklist and annotated call example, explore User Intuition for agencies, or book a demo → to test your methodology and sample requirements.