Running Language Models on Your Own Machine: Apple Silicon, Quantisation, and What Actually Fits

There is a specific moment when local language models stop being a curiosity and become useful. It arrives when you realise that a model running on your laptop, with no API key, no rate limit and no per token bill, is good enough for a task you were previously sending to a frontier model. That threshold has moved a long way. This is the hardware and sizing companion to Local AI in 2026, which covered the runtimes themselves. A great deal of the work developers actually do with language models, which is summarising, extracting, classifying, rewriting and answering questions about a document in front of them, now runs perfectly well on hardware you already own.

Why memory bandwidth decides everything

Generating a token with a transformer requires reading essentially every weight in the model. That is the fundamental operation, repeated for every single token you produce. If the model occupies 8GB in memory and your hardware can move 400GB per second, the arithmetic ceiling is around 50 tokens per second regardless of how fast the processor is, because the processor spends most of its time waiting for weights to arrive.

This single fact explains most of what looks strange about local inference performance. It explains why Apple Silicon performs so well despite modest raw compute figures, because unified memory offers high bandwidth to a large pool. It explains why a consumer GPU with fast but small memory outruns a workstation card with slower memory. It also explains why quantisation produces such dramatic speedups: halving the bytes per weight halves the data you must move, and the speed roughly doubles.

When you are choosing hardware for local inference, memory bandwidth and memory capacity are the two numbers to compare. Everything else is secondary.

What fits in the memory you have

Sizing is straightforward arithmetic. Take the parameter count in billions, multiply by the bytes per parameter implied by the quantisation, and add roughly 20 percent for the key value cache and runtime overhead. At 4 bit that is about 0.55 bytes per parameter in practice once you account for the metadata quantisation formats carry.

Note the context window is not free. Long contexts grow the key value cache substantially, and a 32k context on a mid sized model can add several gigabytes on its own. If a model loads fine and then fails partway through a long document, the cache is almost always the cause.

How much quality does quantisation cost

Quantisation reduces the precision of the weights. Sixteen bit is the usual full precision baseline, and 8, 5 and 4 bit variants trade accuracy for memory and speed. The reason 4 bit became the default is that the quality drop is small and unevenly distributed: it barely touches straightforward language tasks and shows up mainly in multi step reasoning, arithmetic and code generation.

The practical advice is to run the largest parameter count that fits at 4 bit rather than a smaller model at higher precision. A 14B model at 4 bit will generally outperform an 8B model at 8 bit despite occupying similar memory, because parameter count buys more capability than precision does across the range that matters.

Getting started without ceremony

The tooling has become genuinely pleasant. On Apple Silicon, frameworks built on Metal give you native GPU acceleration without configuration. On Linux and Windows with an NVIDIA card, the same runtimes work through CUDA. In both cases you can be generating tokens within a few minutes.

# Pull a quantised model and talk to it. Most local runtimes expose an
# OpenAI compatible endpoint, which means existing client code just works.

ollama pull llama3.1:8b-instruct-q4_K_M
ollama run llama3.1:8b-instruct-q4_K_M "Summarise this in three bullets: ..."

# Serve it on localhost with an OpenAI compatible API
ollama serve

# Point any existing client at it by changing the base URL
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b-instruct-q4_K_M",
    "messages": [{"role": "user", "content": "Extract every date from this text."}],
    "temperature": 0.2
  }'

That OpenAI compatible endpoint is the detail that makes local models practical rather than academic. Existing code that talks to a hosted API usually needs only a base URL change to run against a local model, which means you can A/B a local model against a frontier one on your real workload in an afternoon instead of rewriting an integration.

# Route by difficulty: local model for volume, frontier API for hard cases.
# The heuristic matters less than having one at all.

from openai import OpenAI

local    = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed")
frontier = OpenAI()   # reads the real key from the environment

def classify(text: str, hard: bool = False):
    client = frontier if hard else local
    model  = "gpt-5" if hard else "llama3.1:8b-instruct-q4_K_M"

    return client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": text}],
        temperature=0,
    )

# Escalate only when the local model is uncertain or the input is long
def needs_frontier(text: str, local_confidence: float) -> bool:
    return local_confidence < 0.7 or len(text) > 20_000

Where local models genuinely lose

It would be dishonest to present this as a straight replacement. Local models are meaningfully worse at long multi step reasoning, at holding very long contexts coherently, at generating correct code for unfamiliar libraries, and at following complicated multi constraint instructions without drifting. If your task involves reasoning across a hundred pages, or writing code that must be right the first time, the gap is real and you will feel it.

They also lose on sustained throughput. A single machine serving one user is fine. A single machine serving two hundred concurrent users is a different engineering problem, and at that point the economics of a hosted API start looking reasonable again.

The pragmatic position

Local inference is no longer a hobbyist exercise, but it is also not a wholesale replacement for hosted models. The useful framing is that you now have a fast, free, private tier available for the large volume of straightforward work, and a paid tier for the genuinely hard requests. Most teams discover that the split is heavily weighted towards the free tier once they measure it honestly. Spend an afternoon running your real prompts against an 8B model on the machine in front of you, and let the results rather than the marketing decide where each request should go.