Offline AI: How to Run AI Models Locally
August 12, 2026
Running an AI model locally is now a download and one command. The interesting question isn't whether you can — it's whether you should, and the answer depends almost entirely on your hardware and what you're using it for.
The short version:
- Memory is the binding constraint, not raw speed. A model has to fit in RAM or VRAM before anything else matters.
- Quantisation is what makes local practical. Compressing weights to 4-bit shrinks a model roughly fourfold with modest quality loss.
- Apple Silicon punches above its weight because unified memory lets the GPU address system RAM.
- Local wins on privacy, cost and availability. The cloud still wins on peak capability, and pretending otherwise leads to disappointment.
Why run models locally
Nothing leaves your machine. The strongest reason by far. Contracts, medical notes, client data, unreleased code — anything where sending text to a third party is a policy problem rather than a preference.
No per-token cost. Electricity instead of API bills. For high-volume, low-stakes work — bulk classification, draft generation, experimentation — the economics flip hard.
It works offline and doesn't change under you. A cloud model can be deprecated or silently updated; a local one behaves the same next year.
No rate limits. You're bounded by your own hardware, not someone's tier.
What you actually need
Memory decides everything. Roughly, at 4-bit quantisation:
| Model size | Memory needed | Runs comfortably on | Realistic use |
|---|---|---|---|
| 3B | ~2 GB | Almost any modern laptop | Autocomplete, simple classification |
| 7-8B | ~5 GB | 16 GB RAM, most Apple Silicon | Summarising, drafting, everyday chat |
| 13-14B | ~9 GB | 16-24 GB, mid-range GPU | Noticeably better reasoning |
| 30-34B | ~20 GB | 32 GB+ or 24 GB VRAM | Approaching useful for hard tasks |
| 70B+ | ~40 GB+ | 64 GB+ Mac, or multi-GPU | Serious work, serious hardware |
Two things surprise people. Apple Silicon does unusually well — unified memory means a 32 GB Mac can run models that would need a 32 GB dedicated GPU on a PC, which is expensive. And CPU-only inference works but is slow enough to be frustrating for anything interactive.
If a model doesn't fit in memory, it spills to disk and speed collapses. Fitting the model matters far more than having a fast chip.
The tools
Ollama is the easiest starting point. Install, run one command, and you have a model with an API endpoint. It handles downloads, quantisation and serving. Start here unless you have a reason not to.
LM Studio is the graphical option — browse models, adjust parameters, chat, with no terminal. Good for finding out which model suits you before committing.
llama.cpp is the engine much of this ecosystem sits on. Worth knowing if you want fine-grained control over quantisation and memory layout, or you're embedding inference in your own software.
Hugging Face is where the models live, and the place to check licences before using anything commercially. "Open weights" is not the same as "open source" — several popular models carry usage restrictions.
Where local genuinely loses
This deserves honest treatment, because the gap is real and most enthusiastic write-ups skip it.
Peak reasoning. The largest hosted models remain meaningfully better at hard multi-step reasoning, long-context work, and code. A 7B model running on your laptop is useful, not equivalent.
Long context. Local models advertise large context windows, but attention cost scales badly and memory fills fast. Practical local context is usually well below the headline number.
Setup and maintenance. You are now the operator. Model updates, quantisation choices, driver issues, and working out whether slowness is memory pressure or something else.
Speed on modest hardware. Cloud inference runs on accelerators you don't own. Local generation on a laptop is visibly slower, which matters most in interactive use.
The sensible position is to route by task: local for anything sensitive, high-volume or offline; hosted for the genuinely hard problems. The privacy constraint usually decides, not the benchmark.
Choosing a first setup
- Check your memory. RAM on Apple Silicon, VRAM on a discrete GPU. That number picks your model tier from the table above.
- Install Ollama and pull a 7-8B model. It's the best ratio of capability to hardware and will run almost anywhere with 16 GB.
- Use 4-bit quantisation unless you have memory to spare. The quality difference from 8-bit is smaller than people expect; the memory difference isn't.
- Test on your actual work, not benchmarks. The question is whether it's good enough for your task.
- Move up a size only when it fails you. Bigger models are slower and hungrier, and the jump is often smaller than the spec sheet implies.
Local models are one piece, not the whole workflow
A running model is a text endpoint. It doesn't know your files, doesn't remember last week, and doesn't do anything until something calls it. The gap between "I have a local model" and "this does useful work" is where most people stall — and it's a workflow problem, not a model problem.
Someone has usually already built the workflow you want. It doesn't run on your machine because sharing it means a repo, a runtime, environment variables and keys — the pattern behind why most AI tools end up in your bookmarks.
Taku is an AI-native desktop workspace built for that gap: mirror an AI app, skill or workflow that already works for someone, run it against your own files without reassembling the environment, then remix it. The app and workflow library is where you start rather than an empty prompt. It's in Beta, and it isn't a local inference engine — Ollama and LM Studio do that job well. For the automation layer above it, see agentic workflows.
FAQ
Can I run AI models offline with no internet at all?
Yes. Once the weights are downloaded, inference is entirely local. You need connectivity only to fetch models and updates.
How much RAM do I need to run AI locally?
16 GB is the practical floor for a genuinely useful 7-8B model at 4-bit. 8 GB limits you to small models. 32 GB or more opens up the 30B+ tier where quality improves noticeably.
Is a local model as good as ChatGPT or Claude?
Not at the top end. A good 7-8B model handles summarising, drafting, extraction and everyday questions well. Hard reasoning, long context and complex code still favour the largest hosted models by a clear margin.
Does running AI locally use a lot of electricity?
Less than people assume for intermittent use — comparable to gaming while generating, near-idle otherwise. Sustained heavy inference on a discrete GPU is a different story.
Is local AI actually more private?
Yes, in the meaningful sense: prompts and documents never leave your machine, so there's no third-party retention or training question. That's the reason most regulated teams look at local at all.