Apple silicon, and what it will run
Generating a token means reading every active weight out of memory once. On a Mac that makes memory bandwidth the ceiling, not the GPU: a 40 GB model on a 400 GB/s machine cannot pass ten tokens a second however fast the cores are. The figures below are that ceiling times two thirds, which is about what these machines reach in practice — and 'too big' means the model will not fit in memory at four-bit, whatever the bandwidth.
| Chip | Bandwidth | Memory | 8B≈4.7 GB | 32B≈18 GB | 70B≈40 GB | In |
|---|---|---|---|---|---|---|
| M1 | 68GB/s | 16GB max | 10tok/s | too big | too big | MacBook Air · MacBook Pro 13 · Mac mini · iMac 24 · iPad Pro · iPad Air |
| M1 Pro | 200GB/s | 32GB max | 28tok/s | 7tok/s | too big | MacBook Pro 14 · MacBook Pro 16 |
| M1 Max | 400GB/s | 64GB max | 56tok/s | 15tok/s | 7tok/s | MacBook Pro 14 · MacBook Pro 16 · Mac Studio |
| M1 Ultra | 800GB/s | 128GB max | 112tok/s | 29tok/s | 13tok/s | Mac Studio |
| M2 | 100GB/s | 24GB max | 14tok/s | 4tok/s | too big | MacBook Air 13 · MacBook Air 15 · MacBook Pro 13 · Mac mini · Vision Pro |
| M2 Pro | 200GB/s | 32GB max | 28tok/s | 7tok/s | too big | MacBook Pro 14 · MacBook Pro 16 · Mac mini |
| M2 Max | 400GB/s | 96GB max | 56tok/s | 15tok/s | 7tok/s | MacBook Pro 14 · MacBook Pro 16 · Mac Studio |
| M2 Ultra | 800GB/s | 192GB max | 112tok/s | 29tok/s | 13tok/s | Mac Studio · Mac Pro |
| M3 | 100GB/s | 24GB max | 14tok/s | 4tok/s | too big | MacBook Air 13 · MacBook Air 15 · MacBook Pro 14 · iMac 24 |
| M3 Pro | 150GB/s | 36GB max | 21tok/s | 6tok/s | too big | MacBook Pro 14 · MacBook Pro 16 |
| M3 Max | 300–400GB/s | 128GB max | 56tok/s | 15tok/s | 7tok/s | MacBook Pro 14 · MacBook Pro 16 |
| M3 Ultra | 800GB/s | 512GB max | 112tok/s | 29tok/s | 13tok/s | Mac Studio |
| M4 | 120GB/s | 32GB max | 17tok/s | 4tok/s | too big | MacBook Air 13 · MacBook Air 15 · MacBook Pro 14 · Mac mini · iMac 24 · iPad Pro |
| M4 Pro | 273GB/s | 64GB max | 38tok/s | 10tok/s | 5tok/s | MacBook Pro 14 · MacBook Pro 16 · Mac mini |
| M4 Max | 410–546GB/s | 128GB max | 77tok/s | 20tok/s | 9tok/s | MacBook Pro 14 · MacBook Pro 16 · Mac Studio |
| M5 | 153GB/s | 32GB max | 21tok/s | 6tok/s | too big | MacBook Pro 14 · iPad Pro |
| M5 Pro | 307GB/s | 48GB max | 43tok/s | 11tok/s | too big | MacBook Pro 14 · MacBook Pro 16 |
| M5 Max | 614GB/s | 128GB max | 86tok/s | 23tok/s | 10tok/s | MacBook Pro 14 · MacBook Pro 16 · Mac Studio |
| M5 Ultra | 1200GB/s | 512GB max | 169tok/s | 44tok/s | 20tok/s | Mac Studio |
Sizes are the common four-bit quantisations: about 4.7 GB for an 8B, 18 GB for a 32B, 40 GB for a 70B. Prompt reading is compute-bound and much faster than this; these are generation figures. Which models fit a given machine is on the size list.