Quantisation

Storing a model's numbers with less precision so it fits in less memory and runs faster.

what it is made of · the vocabulary

Weights are trained at 16 bits and can usually be squeezed to 8 or 4 with little measurable loss. FP8, INT4, AWQ and GGUF are all names for doing this in different ways. Two sellers offering the same model at different prices are often offering it at different precisions, which is why the lane is printed beside the rate here.

Nearby