The default approach for customizing LLM behavior is prompt engineering — writing detailed instructions that shape the model's output. Fine-tuning takes a different approach: you update the model's weights directly using labeled examples. Both work. The question is which is right for your situation.
The core tradeoff
| Dimension | Prompting | Fine-Tuning |
|---|---|---|
| Setup cost | Hours (write the prompt) | Days–weeks (curate data, train, eval) |
| Data required | Zero (zero-shot) or few examples | Hundreds to thousands of examples |
| Inference cost | Higher (longer prompts = more tokens) | Lower (shorter prompts, smaller model) |
| Latency | Higher (more input tokens) | Lower (model already "knows" behavior) |
| Consistency | Variable (prompt-sensitive) | High (baked into weights) |
| Updateability | Instant (edit the prompt) | Slow (retrain on new data) |
| Knowledge cutoff | Works with any base model | Frozen to training data |
When prompting beats fine-tuning
Prompt engineering is the right default in almost every situation because it's faster to iterate and easier to update. Specifically, choose prompting when:
- You don't have hundreds of high-quality labeled examples yet
- The task requirements change frequently (fine-tuned models become stale)
- You need few-shot examples to communicate the format or style — a modern Opus 4 model follows these reliably
- Your production volume is low enough that extra prompt tokens don't add up to significant cost
- You're still in the experimentation phase and need to pivot quickly
When fine-tuning makes sense
Fine-tuning becomes worth the investment when you have a very specific, stable task at high volume. Specifically:
- High volume, stable task: You're running 1M+ completions/month on a specific task — the per-token cost of the long system prompt adds up to significant savings if you can compress it into weights
- Consistency requirement: You need extremely consistent output format — fine-tuning produces more reliable structured outputs than prompting
- Domain-specific language: Medical terminology, legal phrasing, or proprietary jargon that the base model doesn't handle well even with prompting
- Latency-critical path: You need to minimize input tokens on a latency-sensitive user-facing feature
Decision framework
Quick decision guide
The hybrid approach (most production agents)
Most mature agent deployments use both: prompt engineering for reasoning and task adaptation, and fine-tuning for the structured output layer. The reasoning model (Claude Opus 4, GPT-5) is used as-is with a detailed prompt; a smaller fine-tuned model handles the final output formatting step at a fraction of the cost.
Deploy optimized agents on MoltBot
Model routing, prompt management, and fine-tuned model support. 14-day free trial.
Start Free Trial →