Practice · Established · Intermediate

Fine-Tuning

Continuing training of a pretrained model on a smaller, targeted dataset to specialise its behaviour, format or domain.

What Fine-Tuning is

Fine-tuning is best at teaching style, structure and task patterns. It is a poor and expensive way to inject facts, which is what retrieval is for.

How it works

Full fine-tuning updates all weights; parameter-efficient methods such as LoRA train small adapter matrices instead, cutting cost by orders of magnitude. Datasets of a few hundred to a few thousand high-quality examples often suffice.

Why it matters

Choosing correctly between prompting, retrieval and fine-tuning is one of the highest-leverage architecture decisions in an AI project.

Common uses

→Consistent output formats
→Brand voice and tone
→Domain jargon and classification
→Reducing prompt length and cost

Strengths

✓Reliable formatting and style
✓Shorter prompts at inference
✓Can beat larger models on narrow tasks

Watch for

✓Data preparation effort
✓Model must be retrained as needs change
✓Catastrophic forgetting of general ability

Continue exploring

Retrieval-Augmented Generation Prompt Engineering Direct Preference Optimization Model Distillation