Practice · Established · Advanced

LoRA (3)

Also known as: Low-Rank Adaptation, PEFT

A parameter-efficient fine-tuning method that trains small low-rank matrices while keeping the original weights frozen.

What LoRA (3) is

LoRA assumes the update needed to adapt a model is low-rank, so it can be represented by two thin matrices per layer instead of a full weight delta.

How it works

Adapters are injected into attention and feed-forward projections and trained on the target data. The result is a file of megabytes rather than gigabytes, which can be merged into the base weights or swapped at serving time. QLoRA combines this with quantised base weights to fine-tune large models on a single GPU.

Why it matters

It democratised fine-tuning, and makes serving many customer-specific variants from one base model economically viable.

Common uses

→Per-customer model variants
→Style adapters for image models
→Domain tuning on modest hardware

Strengths

✓Tiny artefacts
✓Cheap training
✓Hot-swappable adapters

Watch for

✓Slightly below full fine-tuning on hard tasks
✓Adapter management overhead

Continue exploring

Fine-Tuning Quantization Open-Weight Model