Pretraining
The first and by far the most expensive stage: reading a very large body of text to learn language and world knowledge.
how it works · the vocabulary
base modelfoundation model
Costs tens to hundreds of millions of dollars and is done once. Everything a buyer interacts with — instruction tuning, alignment, fine-tuning — is comparatively cheap work layered on top of it.