How to Run AI Agents Locally: The Four Pieces

Running a model locally and running an agent locally are different projects, and conflating them is why so many local agent setups don't work. A local model answers questions. An agent takes actions — which means it needs tools, a loop that decides which tool to call, and a model capable of deciding.

Four pieces have to be in place:

1. A runtime that serves a model on your machine — Ollama or LM Studio
2. A model that supports tool calling — not all local models do, and this is the step that silently fails
3. An agent loop that reads the model's tool calls and executes them
4. Tools for it to call, increasingly through MCP

Miss the second and you get an agent that talks about calling tools and never calls one. That's the most common failure, and it looks like a bug in the framework rather than a model choice.

This covers each layer, and what stays genuinely worse locally. If you only want to run models — chat, no actions — our guide to running AI models locally covers that smaller problem.

Layer 1: the runtime

Ollama is the usual answer. One command pulls a model, another serves it over a local API, and — the part that matters for agents — that API is OpenAI-compatible. Any agent framework built against a cloud provider generally works against Ollama by changing the base URL.

That compatibility is doing the heavy lifting in nearly every local agent setup you'll read about. It's why the instructions are usually three lines rather than a rewrite.

LM Studio also serves a local API and adds a GUI for browsing and downloading models, which makes the model-shopping part much less opaque. For an agent build either works; pick GUI or CLI by preference. The direct comparison covers the differences that aren't preference.

Layer 2: the model — this is where setups fail

An agent works by the model emitting a structured request to call a tool, then reading the result and deciding what's next. A model that wasn't trained for that will describe calling a tool in prose. The loop gets nothing it can parse, and nothing happens.

So the requirement is narrower than "a good local model." You need one with explicit tool-calling or function-calling support, and the model card says so. Hugging Face is where to check before downloading — the capability is listed, and guessing from the model's reputation doesn't work.

Size matters more for agents than for chat, and for a specific reason. Chat tolerates a mediocre answer. An agent chains decisions: a wrong tool choice at step two makes steps three through eight garbage. Small models degrade much faster under that compounding than their chat quality suggests.

Practical floor: a 7–8B model with tool-calling support, quantized, in about 16 GB of RAM will run and will make mistakes on anything with more than a few steps. Around 30B in 32 GB or more is where multi-step tasks start holding together. Below that, keep the tasks short and the tool list small.

Layer 3: the agent loop

The loop is simple in principle: send the prompt, read the response, if it contains a tool call then execute it and send the result back, repeat until done.

You can write this yourself in about fifty lines, and doing so once is genuinely instructive — it demystifies the whole category. For anything real, a framework handles retries, errors, parallel calls, and stopping conditions.

LangGraph is the most common choice for building custom agent flows with explicit control over state and branching; its source repository is worth reading regardless of whether you use it, since the control-flow patterns transfer.

For specific jobs, purpose-built agents beat a general framework. browser-use drives a web browser and works with local models. Coding agents run in your editor and can point at a local endpoint.

Open WebUI deserves a mention as the pragmatic middle: a full chat interface over Ollama that also supports tool integration, so you get agent behavior without building a loop.

Layer 4: tools, and MCP

Early local agents used bespoke tool definitions per framework. That's converging on MCP, which means a tool server you set up once works across clients rather than being rewritten for each.

For a local agent, the servers that matter most are the local ones: filesystem access scoped to directories you name, shell command execution, and a fetch server for reading web pages. Those three cover a large share of what people actually want an agent to do.

One thing to watch that hits local setups harder than cloud ones: on most runners, every tool definition occupies context before your prompt does. With a 1M-token cloud window that's a rounding error. With a local model's smaller window, connecting a dozen servers can consume a serious fraction of what you have.

Some hosts have started loading tool definitions only when the model searches for them rather than all at startup, which would relieve exactly this pressure — but that's a host feature, not something the protocol provides, and the local runners are generally behind the hosted products on it. Assume eager loading unless your setup documents otherwise, and keep the tool list tight either way. It's a bigger constraint locally than anywhere else.

Where local genuinely loses

Worth being direct about, because enthusiasm for local setups tends to skip it.

Multi-step reliability. The single biggest gap. Frontier cloud models hold a plan across many steps and recover from errors. Local models drift, forget constraints from earlier steps, and repeat failed actions. A ten-step task that a cloud agent completes may need supervision at every stage locally.

Speed. Tokens per second on consumer hardware is a fraction of an API's, and an agent generates many times more tokens than a chat does — every tool call, every result, every re-plan. A task that feels quick in a chat window can take a long, visible while as an agent.

Long context. Current cloud flagships carry around a million tokens. Local models typically have far less, and quality often degrades before the stated limit. Agents accumulate context fast, so this binds sooner than expected.

Setup cost. Real, and usually understated. Four layers, each with its own failure modes, and errors that don't say which layer broke.

What local wins: nothing leaves your machine, no per-token cost regardless of how much the agent runs, no rate limits, and it works with no connection. For anything involving data that can't go to a third party, those aren't preferences — they're requirements, and they make the trade-offs above worth paying.

A realistic first build

If you want the shortest path that actually works:

1. Install Ollama and pull a 7–8B model with tool-calling support
2. Install Open WebUI and point it at Ollama
3. Add one MCP server — filesystem, scoped to a single directory
4. Give it a two-step task and watch what it does

That's an afternoon, not a weekend, and it tells you honestly whether your hardware clears the bar before you invest in a framework.

The part that doesn't get easier

Every layer above is a place to get stuck, and the errors rarely name the layer. A silent no-op is a model without tool calling; a framework failure may be a transport mismatch; slow output may be a model that doesn't fit in memory and is spilling to disk.

That diagnostic loop is the actual work, and it's where most people stop — not because the idea is hard, but because reproducing someone else's working configuration is harder than it looks. Taku is built for that gap: you mirror an AI app, agent, or workflow that already runs into your own desktop workspace instead of assembling one from parts. The free app library is the fastest way to see whether that suits how you work. Taku is in Beta, and the Mac app is available now.

Key points

• A local agent needs four layers: runtime, tool-calling model, agent loop, and tools. A local model alone isn't an agent.
• Verify tool-calling support on the model card. A model without it fails silently rather than erroring.
• Ollama's OpenAI-compatible API is why most cloud agent frameworks work locally with a URL change.
• Agents compound errors across steps, so small models degrade faster as agents than as chatbots.
• Tool definitions eat context, and that constraint binds much harder on a local model's smaller window.
• Local wins on privacy, cost, and offline use. It loses on multi-step reliability, speed, and context length.

FAQ

Can I run an AI agent completely offline?

Yes, once the model is downloaded. The runtime, the loop, and local tools all work with no connection. Any tool that calls a web service obviously won't.

What hardware do I need?

About 16 GB of RAM for a 7–8B quantized model — workable for short tasks. 32 GB or more for something around 30B, which is where multi-step tasks start holding together. A GPU mainly buys speed.

Why does my local agent never call tools?

Almost always the model. Not all local models support tool calling, and one that doesn't will describe the call in prose instead of emitting a structured request. Check the model card.

Is running agents locally cheaper than using an API?

No per-token cost, so at high volume yes. The cost moves to hardware and your time. For occasional use, an API is usually cheaper overall once setup time is counted.

What's the difference between running a model locally and running an agent locally?

A model answers. An agent acts — which needs tools, a loop, and a model able to decide between them. The model is one of four pieces.

Can local agents use MCP servers?

Yes, and it's becoming the standard way to give them tools. The local servers — filesystem, shell, fetch — are the most useful for an offline setup.

Which local models are best for agents?

Look for explicit tool-calling support first, then size. Several open model families ship tool-calling variants; the capability is listed on the model card, which is a more reliable guide than a ranking that dates within weeks.