Building AI Features: The PM Playbook

AI is hype. Every founder wants "AI-powered features." But most fail because they add AI without solving a real problem. This playbook helps you spec and launch AI features that users love and that actually work.

Start with the Problem, Not the Technology

Bad: "Let's add an AI chatbot because everyone has one."

Good: "Users can't find answers in our knowledge base. A chatbot that understands questions and retrieves relevant docs would save them time."

Before you pick a model or technology, understand:

• The user job: What task are they trying to accomplish?
• Current friction: What makes it hard today?
• Success metric: How will you know the AI feature worked? (time saved, accuracy, user satisfaction)

This frames your feature spec and prevents "AI for AI's sake."

Model Selection and Design for Failure

Choose the model based on your constraints:

• Budget: Use Gemini Flash for cost-sensitive. Use GPT-4o if budget allows and reasoning is critical.
• Latency: Gemini is faster. If you need sub-200ms, prefer smaller models or cached responses.
• Accuracy: Test on real data. GPT-4o usually outperforms Gemini on reasoning; Gemini is better on multimodal.
• Data Privacy: If PII is involved, use RAG and never send raw data to external APIs.

Design for failure:

• Hallucinations Happen: Include disclaimers ("This is AI-generated and may be inaccurate"). Consider human review for high-stakes outputs.
• Fallbacks: If the AI can't answer confidently, fallback to a human agent or a pre-written response.
• Monitoring: Log outputs and user feedback. Monitor for bad outputs and retrain/adjust prompts.

Measuring Success and Iteration

Define metrics before launch:

• Accuracy: How often is the AI output correct? Sample outputs weekly and have humans grade.
• Latency: How fast does the feature respond? Aim for P95 under 1s for user-facing features.
• Cost: Track cost-per-user. If it's >$0.10/user/month, consider cost optimization (caching, model routing, fine-tuning).
• Adoption: Do users actually use the AI feature? If adoption is low, the problem you solved wasn't real.
• User Satisfaction: NPS or 1-5 rating. Real feedback beats guessing.

After launch, iterate fast. Weekly performance reviews. Adjust prompts, swap models, add guardrails based on real usage.

Key Takeaways

• Spec the user problem first, technology second.
• Design AI features for graceful failure (disclaimers, fallbacks, monitoring).
• Measure accuracy, latency, cost, and adoption from day one.
• Iterate based on real usage data, not gut feel.
• Don't ship AI features if the underlying problem isn't real. Users will reject them.