You rewrote the prompt five times and the model still misses the mark. Before you rewrite it a sixth time, it's worth asking whether the problem is the prompt at all.
Your prompt isn't always the problem
When a model gives a weak answer, the instinct is to tweak the wording and try again. Sometimes that works. Often the real fix is somewhere else entirely: better examples, a cleaner dataset, or a small training run that teaches the model a skill no amount of phrasing can deliver.
This guide covers the practices that actually help when you're training a model with prompts, and when you should stop prompting and switch tools. It pulls from the guidance Weights & Biases, Google, and Unsloth publish for teams doing this work.
Prompting vs fine-tuning: pick the right lever first
Two ways exist to steer a model, and choosing the wrong one wastes weeks.
Prompting changes what you ask. You write instructions that shape the output, and the model's underlying weights never move. It's fast, reversible, and needs no training data.
Fine-tuning changes the model itself. You show it many examples of the task and let it adjust its internal behavior, which means the change persists across every future request once training finishes. It takes time and data, but it bakes in skills that prompting can't reach.
The decision comes down to one question. Is the problem about facts or about behavior?
If the model needs to know something new, like your company's refund rules, prompting plus retrieval handles it, not fine-tuning. If the model needs to adopt a consistent style, follow a rigid format, or perform a narrow task reliably, fine-tuning earns its cost. Weights & Biases puts the split simply: use prompting for tasks where the model already has the skill, and fine-tune where it needs to learn one.
Reach for prompting when
- The model already has the skill you need
- Requirements change often
- You have no labeled examples yet
- Speed of iteration matters most
Reach for fine-tuning when
- You need a strict output format, every time
- The task is narrow and repeated at scale
- You have clean, labeled examples
- Tone and style must stay consistent
What practices actually help during prompt training
A few habits consistently separate training that works from training that flounders.
Start with examples, not instructions alone. A model learns a pattern faster from a handful of worked examples than from a paragraph describing it. This is the core of few-shot prompting, where you show the task before asking for it.
Make the dataset the priority. Fine-tuning only reflects the quality of the data you feed it, and a small, clean set beats a large, messy one. Unsloth's guidance is blunt on this point: getting the dataset right matters more than anything downstream.
Avoid bare zero-shot requests when the task is hard. Research on prompting in social science tasks found that models with richer training and larger vocabularies did better, and that skipping straight to a plain zero-shot prompt underperformed approaches that used richer prompting and task-specific fine-tuning.
Match the evaluation to the job. If the task needs steady, reliable output, test the model on many similar cases rather than one clever example. A single good result means little.
Track what you change. Because both prompting and fine-tuning involve many small adjustments, teams that log each change and its outcome learn far faster than those that guess.
When fine-tuning doesn't pay off
Fine-tuning gets oversold. It's the right tool for behavior, and a poor one for facts.
A model fine-tuned on your FAQ won't reliably recall a new policy you added last week, because that fact wasn't in the training run. Keeping facts current belongs to retrieval, not training. Try to teach a model facts by fine-tuning and you'll end up retraining every time the facts change, which is exactly the cost you were trying to avoid in the first place.
Fine-tuning also struggles when your data is thin. A few dozen examples rarely justify a training run, and a model trained on unrepresentative data learns the wrong pattern with confidence. If you can't show the model a hundred solid examples, prompting is usually the better bet.
And it doesn't fix a task the model can't do. If the base model lacks the underlying capability, fine-tuning on a small dataset won't create it from nothing.
A short checklist before you train
Run through these before committing to a training run.
Decide whether the gap is facts or behavior, and let that pick your tool. Collect a clean set of labeled examples and check a sample by hand, because errors in the data become errors in the model. Write down what success looks like before you start, so you have something concrete to measure against later. Keep a simple log of what you changed and what happened. Then test on cases you didn't train on, since that's the only real test of whether it worked.
None of this is glamorous. All of it decides whether a training run produces a usable model or a week you'd rather forget.
The takeaway on training models with prompts
Prompts and training solve different problems. Use prompting when the model already knows the skill and you're shaping the request. Use fine-tuning when the skill itself has to be built in. Reach for retrieval when the issue is facts.
The mistake most teams make is rewriting the prompt when the task needs a different lever. Get the lever right first, put your effort into the dataset, and measure against cases the model hasn't seen. That order saves more time than any cleverly worded instruction.






