#quantisation

Running Language Models on Your Own Machine: Apple Silicon, Quantisation, and What Actually Fits

Local language models are good enough for a large share of real work. How memory bandwidth sets the speed ceiling, what fits in the RAM you have, how much quality quantisation actually costs.