Reranking search results models that run on 8 GB
7 models with published weights that fit in 8 GB — a phone, a base iPad, an Air. Counted at four-bit quantisation, weights only, leaving the machine about a third of its memory and a gigabyte for context: room for roughly 7 billion parameters. A mixture of experts is counted in full, because it is held in full even though only a few experts compute.
The memory of a phone, a base iPad or an entry-level laptop. What fits is small: models of a few billion parameters, quick and cheap to run, good at summarising, classifying and simple extraction, and out of their depth on long reasoning. This is also where on-device makes the most sense, because the alternative is a network round trip for something that takes a moment.
Models that take a handful of results a search already found and put them in the right order. They are the cheap second stage of retrieval: an embedding search casts a wide net fast, a reranker reads the candidates properly. Because they only ever see a few documents, they cost little per query, and they usually improve a weak search more than a better embedding model would.
- Qwen3-Embedding-4BAlibaba4.0B≈2.6 GB at 4-bit4 also selling it hosted
- Qwen3-Reranker-4BAlibaba4.0B≈2.6 GB at 4-bit1 also selling it hosted
- Qwen3-VL-Embedding-2BAlibaba2.1B≈1.4 GB at 4-bit
- llama-nemotron-rerank-vl-1b-v2NVIDIA1.0B≈0.7 GB at 4-bit1 also selling it hosted
- Qwen3-Embedding-0.6BAlibaba0.6B≈0.4 GB at 4-bit4 also selling it hosted
- Qwen3-Reranker-0.6BAlibaba0.6B≈0.4 GB at 4-bit1 also selling it hosted
- bge-reranker-largeBAAI0.6B≈0.4 GB at 4-bit