# AI Exit Interview Guardrails What to automate, what to keep human, and the checks that stop an automated interview from producing confident nonsense. From saasfeedback.ai — https://saasfeedback.ai/blog/ai-exit-interviews --- ## The split | Stage | Automate? | Why | |---|---|---| | Detecting the churn event | **Yes, always** | Deterministic. Humans forget; webhooks do not. | | Scheduling and reminders | **Yes** | Pure logistics. | | Reaching out cold | **Partly** | Automated outreach works; automated *persuasion* to participate does not. | | Asking the first question | **Yes** | It is the same every time. | | Deciding the second question | **Careful** | This is where interviews are won or lost. See below. | | Handling an emotional or hostile answer | **No** | The moment a customer is angry, a bot converts a candid conversation into a complaint. | | Transcription | **Yes** | Solved problem. | | Theme coding | **Yes, with review** | Good first pass, unreliable on your specific domain vocabulary until corrected. | | Weighting by revenue | **Yes** | Arithmetic. | | Deciding what to build | **No** | Not a data problem. | --- ## The three things automation genuinely cannot do yet ### 1. Earn the disclosure The valuable half of an exit interview is the part the customer did not plan to say. That disclosure is a social act: it happens because a specific person sounded like they would do something with it. A form does not create that obligation, and neither does a synthetic voice that discloses it is synthetic — nor, more importantly, one that does not. **Guardrail:** measure *disclosure depth*, not completion rate. Count responses that contain a specific event with a date or a place. If under 30% of your automated responses clear that bar, you are collecting categories, not causes. ### 2. Know which thread to pull Interviewing skill is almost entirely the choice of follow-up. "We had a reorganisation" can lead to budget, to a champion leaving, or to a genuine change in need — and only one of those is your problem. A human hears the hesitation before "reorganisation" and asks about it. Models are getting better at this and are still worse than a trained person, particularly when the interesting signal is what the customer avoided saying. **Guardrail:** sample 10% of automated interviews for human review, specifically scoring "was there an obvious unpulled thread?" Track that rate over time. It is the single best measure of whether your automation is improving. ### 3. Distinguish the stated reason from the real one "Too expensive" is the most over-reported churn reason in SaaS and the least often true as stated. It is what people say when the real answer is "I never got it working and I don't want to explain why." Extracting the real reason requires gently disbelieving the first answer — which automated interviewers, tuned to be agreeable, are structurally bad at. **Guardrail:** flag every response whose primary theme is `price` and no supporting event. Route those to a human. In our experience the majority re-classify. --- ## Disclosure and consent Non-negotiable, regardless of jurisdiction: - [ ] The customer is told they are talking to an automated system, before the first substantive question. - [ ] Recording consent is captured explicitly and logged. - [ ] There is a working, obvious path to a human on request. - [ ] Responses are retained under your existing data policy, and the customer is told which one. - [ ] Nothing said in an exit interview is fed to a sales or win-back sequence without separate consent. The last one is where most programmes quietly break trust. A customer who gives you a candid answer and receives a discount offer three days later has learned that candour is a sales trigger, and will not repeat it. --- ## Quality checks to run monthly | Check | Threshold | If it fails | |---|---|---| | Specific-event rate | >30% of responses contain a dated event | Rewrite the follow-up prompts | | Price-theme audit | <25% of responses coded price-only | Route to human review | | Unpulled-thread rate (sampled) | <20% | Retrain or add a human tier | | Theme coding agreement with human review | >80% | Retrain the taxonomy | | Median response length | Rising or flat | Falling length means your questions have gone stale | --- ## The hybrid that works 1. **Automate detection, outreach, scheduling, transcription and first-pass coding** — all of it, without hesitation. This is the majority of the labour and none of the judgement. 2. **Put a human on the conversation itself** for anything above your MRR threshold, anything coded price-only, and a random 10% sample. 3. **Keep the decision human**, always. The output of an exit interview programme is not a report. It is an argument about what to build next, and arguments need someone accountable for them.