What size model fits in 8 GB of memory?

At four-bit quantisation, comfortably up to about 9 billion parameters with usable context left over. In practice that means the 7B and 8B families — Llama, Qwen, Mistral, Gemma and the distilled reasoning models — plus smaller specialists like Whisper for speech and an embedding model for search.

How fast is local inference on this hardware?

Around 14 tok/s on a single stream for a 7B model at Q4_K_M. Faster than reading speed for conversation; slow enough that large batch jobs are best left to run overnight, which is what an always-on machine is for.

Can I load my own models?

Yes — anything Ollama or llama.cpp will run that fits in 8 GB, which in practice means GGUF up to about 9B at four-bit. The models it ships with are a starting point, not a walled garden.

Local AI hardware, and what it honestly runs

Most pages selling local AI hardware are vague about the one number that decides everything: how much memory the model has to fit in. This one is 8 GB, shared with the GPU, which puts a hard and knowable ceiling on what can run.

Memory is the constraint, not compute

A language model has to be resident in memory to be fast. At four-bit quantisation a model needs roughly half a gigabyte per billion parameters, plus room for the context window — so 8 GB lands you at the 7 to 9 billion parameter class with space left to work in.

The compute matters after that. 67 TOPS of accelerator throughput is what turns a model that fits into a model that answers at a usable speed rather than a word at a time.

What that class of model is good at

Summarising a long document. Pulling structured fields out of unstructured text. Classifying and routing. Drafting a first version of something you will edit. First-pass code review. Transcription and embeddings for local search. This is the everyday middle of knowledge work, and an 8B model at four-bit does it competently.

Expect around 14 tok/s on a single stream for a 7B model at Q4_K_M. That is faster than you read, which is the threshold that matters for anything conversational, and slow enough that a batch of a hundred documents is an overnight job rather than a coffee break.

What it is not good at, said plainly

It is not a frontier model and no amount of local enthusiasm makes it one. Long multi-step reasoning, hard novel code, anything needing a very large context — these are where a 7B model produces something confident and wrong, which is worse than slow.

It also will not train anything, will not do video, and will not hold two large models at once. If a page tells you a small box replaces your Claude subscription, it is selling you the disappointment as well as the hardware.

Which is why local is one of two options, not the only one

The useful arrangement is both. Work that cannot leave the building goes to the model on the box — no key, no bill, no packet off your network. Work that needs the big model goes to Anthropic, OpenAI, Google or OpenRouter on the key you already hold, billed by them at their rates.

You choose per task. Llama 3.1 8B, Qwen 2.5 7B, Mistral 7B, DeepSeek-R1 Distill 8B and more ship on the drive; the runtimes are Ollama and llama.cpp, with an OpenAI-compatible endpoint so anything that lets you set a base URL can point at it.

Common questions

What size model fits in 8 GB of memory?: At four-bit quantisation, comfortably up to about 9 billion parameters with usable context left over. In practice that means the 7B and 8B families — Llama, Qwen, Mistral, Gemma and the distilled reasoning models — plus smaller specialists like Whisper for speech and an embedding model for search.
How fast is local inference on this hardware?: Around 14 tok/s on a single stream for a 7B model at Q4_K_M. Faster than reading speed for conversation; slow enough that large batch jobs are best left to run overnight, which is what an always-on machine is for.
Can I load my own models?: Yes — anything Ollama or llama.cpp will run that fits in 8 GB, which in practice means GGUF up to about 9B at four-bit. The models it ships with are a starting point, not a walled garden.