Run a local LLM at home

Running a large language model on your own machine used to mean a rack of expensive GPUs. In 2026 it is a desktop-sized project: a single quiet box can run a capable model locally, privately, with no API bill and no data leaving your house. The whole thing hinges on one number, and once you understand it the hardware choice is easy.

1. It is a memory problem, not a compute problem

A local LLM needs enough fast memory to hold the model. That is the spec that decides whether it runs at all; everything else decides how fast. The size depends on the model's parameter count and its quantization (compression). At Q4 - the quality/size sweet spot most people run - the rough figures are:

• 7B model - about 6 to 8GB
• 13B - about 10 to 12GB
• 33B - about 20 to 24GB
• 70B - about 40 to 48GB

Pick the model you want to run, read off the memory, and buy for that. A 7B or 13B is plenty for a fast local assistant; a 70B is closer to the big cloud models but needs real memory.

2. Two ways to get the memory

There are two hardware paths, and the right one depends on how big you want to go and how fast.

• A discrete GPU with its own VRAM is the fastest option per gigabyte. A 16GB card runs a 13B nicely, a 24GB card a 33B. The full comparison of which cards bring how much VRAM is at ServerRigs - sort by VRAM.
• Unified memory on a modern AMD APU (a Ryzen AI Max+ 395, or a Ryzen AI 9 box) lets you hand most of the system RAM to the integrated GPU. A 128GB box can allocate roughly 96GB as graphics memory - enough to fit a 70B locally with no graphics card at all, in a machine that sips power on a desk. This is the cheapest and quietest way to run the biggest models. ServerRigs ranks these boxes in its best mini PC for local AI guide.

The honest trade: a discrete GPU runs a given model faster; unified memory fits bigger models cheaply, at lower tokens per second. For a personal assistant either is fine; for low-latency serving, lean GPU. Pick the box or card that matches your target model size.

3. The software is the easy part

You do not compile anything. The two front doors:

• Ollama - a one-line install that pulls and runs models from a simple command line or API. The default choice, and what most tutorials assume.
• LM Studio - a friendly desktop app with a model browser and a chat window, good if you would rather click than type.

Both use the same underlying runtimes and both hardware-accelerate on your GPU or APU automatically. Start with a small model (a 7B or 8B), confirm it runs at a comfortable speed, then step up until you find your machine's ceiling.

4. Storage and privacy

Models are large files - a single 70B at Q4 is tens of gigabytes, and you will collect several. Keep them on fast local storage (an NVMe drive), and if you hoard models, point the tool at a roomy NAS. None of it needs to touch the internet after the initial download, which is the whole point: your prompts and documents stay on hardware you own. Drives sized for that live at NASdisks.

The short version

Decide the model size, read off the memory it needs, then choose a discrete GPU for speed or a big unified-memory APU box to fit the largest models cheaply. Install Ollama or LM Studio, start small, and step up to your machine's limit. It is a private, no-subscription version of the tools everyone else rents. Start with the hardware comparison at ServerRigs.