Before any engagement, we run every prospective client through five questions. These questions have killed projects that sounded exciting. They've also fast-tracked implementations that looked boring.
That's the point.
AI is not inherently valuable. It's valuable when it does a specific thing that creates measurable business impact at a cost lower than that impact.
Here are the five questions.
1. What's the specific decision or action this AI will change?
Vague answers are a red flag. "We want to use AI for customer service" is not an answer. "We want the AI to resolve tier-1 support tickets (password resets, order status, FAQ) without human involvement" is an answer.
The more specific you can get about the exact decision or action, the more accurately you can model the ROI — and the more likely the implementation is to succeed.
The test: Can you describe the moment the AI acts and what happens as a result? If you can't describe it in one sentence, you're not ready to build it.
2. How many times does this happen, and how much does it cost today?
This is where most ROI modeling falls apart. People estimate volume and cost vaguely. "We spend a lot of time on this" is not a number.
For any AI implementation to justify its cost, you need:
- Volume: How many times per day/week/month does this thing happen?
- Cost per instance: How much human time or direct cost does each instance represent?
- Total addressable cost: Volume × cost per instance
If the total addressable cost is $2,000/month and a good AI implementation costs $15,000 to build and $300/month to run, your payback period is over a year — and that's before accounting for imperfect automation rates.
The rule of thumb: if the total addressable cost isn't at least 3–5x the build cost on an annualized basis, the economics are marginal.
3. What's the realistic automation rate?
AI will not handle 100% of your use cases. Good AI systems handling well-scoped problems typically achieve 70–90% automation rates. The remaining 10–30% escalate to humans.
This matters for two reasons:
Cost modeling: Your real savings are (automation rate × total addressable cost) minus AI operating costs. If your automation rate is 75%, your realized savings from a $5,000/month addressable cost is $3,750/month — not $5,000.
System design: The escalation path for the 10–30% needs to be designed intentionally. A system that handles 85% of cases perfectly and fails badly on the remaining 15% is worse than a system with 75% automation that escalates gracefully.
4. What's the data you have vs. what you need?
The quality of your AI system is bounded by the quality and volume of data it learns from.
Three questions within this question:
- What historical data do you have that represents the task? (Call transcripts, support tickets, documents, etc.)
- Is it structured or unstructured? How clean is it?
- Do you have enough of it? (For fine-tuning: typically 500–2,000 labeled examples minimum)
If you don't have the data, you need to either collect it (delay), use synthetic data (risky), or use a general-purpose model that doesn't know your domain (limited performance).
Data availability is often the actual constraint on timeline and performance — not engineering complexity.
5. Who owns this after we ship?
This question kills the most projects.
Every AI system in production needs someone who:
- Checks performance metrics weekly
- Identifies edge cases and failures
- Has authority to approve prompt/logic changes
- Can communicate issues to the technical team
This person doesn't need to be technical. They need to care about the outcome and have bandwidth to stay close to it.
AI systems are not "set and forget." They drift. Edge cases accumulate. Language patterns change. A system that worked great at launch can quietly degrade over six months if nobody's watching.
The deal-breaker: If there's no designated owner, we won't start the project. We've seen too many well-built systems fail in production due to organizational neglect.
The Scoring Model
We score each project on a simple 1–3 scale across all five dimensions:
| Question | 1 (Weak) | 2 (Moderate) | 3 (Strong) |
|---|---|---|---|
| Specificity | Vague | Defined | Precise + measurable |
| Volume/cost | Estimated | Roughly quantified | Exactly modeled |
| Automation rate | Unknown | Benchmarked | Validated in pilot |
| Data | None | Partial | Sufficient + clean |
| Ownership | Nobody | Named but unclear | Named + committed |
Projects scoring 10–15 get a green light. Projects scoring 7–9 get conditional approval (with specific gaps to close). Projects scoring below 7 get deferred until they're ready.
Most projects that fail don't fail because the technology was wrong. They fail because the score was 6 and we (or someone else) started anyway.
Want to run your use case through this framework with someone who's done it 30+ times? Book a Free Audit. We'll score your project honestly — even if that means telling you to wait.