Learning Paradigms · Established · Advanced

Self-Supervised Learning

Training on unlabelled data by turning the data itself into a prediction task, such as predicting the next token or a masked patch.

What Self-Supervised Learning is

Self-supervision invents labels from raw data. Hide part of the input, ask the model to reconstruct it, and the model is forced to learn structure without any human annotation.

How it works

Common objectives include next-token prediction for text, masked-token prediction, contrastive matching of augmented views for images, and masked patch reconstruction. The pretrained representation is then adapted to real tasks with a much smaller labelled set.

Why it matters

This is the mechanism behind foundation models. It unlocked training on internet-scale corpora, which is what made large language and vision models possible.

Common uses

→Pretraining large language models
→Vision backbones trained without labels
→Speech representation learning

Strengths

✓Uses abundant unlabelled data
✓Produces reusable general representations

Watch for

✓Very compute-intensive
✓Pretraining objective may not match the end task

Continue exploring

Foundation Model Pretraining Large Language Model Contrastive Learning