Fine-Tuning vs Prompting: When to Use Each

As a PM, you'll encounter the question: should we fine-tune our model or optimize our prompts? The answer depends on your use case, data, and cost constraints. This guide walks you through the decision framework.

Prompting: Start Here

Prompting is fast, cheap, and flexible. Techniques to maximize prompt quality:

• Chain-of-Thought: Ask the model to explain its reasoning step-by-step. Improves accuracy on complex tasks.
• Few-Shot Examples: Provide a few examples of the input-output format you want. The model learns from examples without retraining.
• System Prompts: Define the model's role and constraints upfront (e.g., "You are a customer support agent. Be polite but concise").
• Temperature and Top-P: Adjust randomness. Lower temperature for deterministic outputs; higher for creative generation.

With good prompting, you can achieve 80-90% of fine-tuning performance, with zero training cost and instant iterations.

When to Fine-Tune

Fine-tune when:

• Consistent Format Needed: Your output must always follow a strict schema (JSON, fixed fields). Fine-tuning enforces this better than prompting.
• Domain-Specific Vocabulary: Your domain has specialized jargon (medical, legal, technical). Fine-tuning helps the model understand context.
• Cost Optimization: You're running billions of tokens monthly. A smaller fine-tuned model costs less than a larger base model.
• Latency Sensitive: You need sub-100ms response times. A fine-tuned smaller model is faster than a large model.

Fine-Tuning Costs and Infrastructure

Fine-tuning requires:

• Labeled Training Data: 100–1,000s of examples. Collecting and labeling costs time and money.
• Compute: GPUs for training. 1–10 hours depending on dataset size.
• Inference Cost: Fine-tuned models on some providers (OpenAI, Anthropic) cost more to run than base models initially, but save money at scale.

Example: Fine-tuning GPT-3.5 with 1,000 examples costs $10–20 in training fees, plus ongoing usage costs. If you run 10M tokens/month, fine-tuning saves ~$20/month vs. GPT-4o. Payback is months out.

Key Takeaways

• Always start with prompting. It's fast and cheap.
• Graduate to fine-tuning only when you've exhausted prompting optimization.
• Fine-tuning ROI improves with scale (high volume, repeated use cases).
• Use OpenAI's fine-tuning for GPT-3.5; Anthropic for Claude if you prefer their models.
• Combine both: fine-tune for your primary use case, prompt for edge cases.