Run an SRE Agent on a Local LLM

No API keys. No hosted model. Once the model is pulled and the tool server is installed, nothing leaves the machine at request time.

That's the whole point of this example. For a lot of teams, the first question about an AI agent isn't "how smart is it?" It's "where does our data go?" The shortest answer is that it stays here.

This page shows the smallest fully local AURA setup, and what an engineer who runs AURA on local models recommends for getting good results.

TL;DR

• What it is: An AURA agent that runs on a local model through Ollama and reaches its tools through a local stdio MCP server. No cloud services are involved at request time.
• What this example does: It answers questions about files in one allowed directory, reading them through the filesystem MCP server and quoting the lines it used.
• Why it matters: It's the template for running any AURA agent with no outside network dependency. Swap in your own MCP servers.
• What you need:
  • Ollama with a tool-capable model (the example uses qwen3:30b-a3b).
  • Node.js 18+.
  • Enough GPU memory for the model you pick.
• Cost: AURA is free and open source (Apache 2.0). There are no per-token charges. You supply the hardware.
• Source: examples/complete/local-ollama-stdio.toml

What this example does

1. Local model. The agent uses Ollama at localhost:11434. Point base_url elsewhere if Ollama runs on another host or in Docker.
2. Local tools over stdio. AURA starts the filesystem MCP server as a child process and talks to it over stdin and stdout. There's no network port and no separate server to run.
3. Scoped access. The tool server can only see one directory (/tmp/aura-sandbox in the example).
4. Grounded answers. The agent is told to read files rather than guess, and to quote the lines and name the file.

Two settings make local models work better:

• num_ctx = 32000 raises Ollama's small default context window, so tool schemas and file contents fit.
• fallback_tool_parsing = true catches models that write tool calls as text instead of native tool calls, and executes them anyway.

Who it's for

Jobs to be done

• Engineer in a restricted environment: When our network policy blocks outside AI services, I want an agent that runs entirely on our hardware, so we can still use AI for operations.
• Security reviewer: When a team proposes an AI agent, I want to verify that prompts and data never leave our environment, so the review is short.
• Engineering leader: When we evaluate AI tooling, I want to start without signing a data processing agreement, so the pilot isn't blocked on legal review.
• Homelab practitioner: When I debug my own stack, I want a capable agent without per-token costs.

Pain points it removes

• Data residency blockers. No prompts, tool results, or documents go to a model provider.
• Long security reviews. There's no outside service in the request path to assess.
• Per-token cost. You're limited by your hardware, not a meter.
• Inconsistent tool calling in local models. Fallback parsing handles models that emit tool calls as text.
• Network setup for tools. stdio MCP servers run as child processes, with no ports to manage.

What it replaces, and why it's better

The value it drives

• AI where it was previously off-limits. Air-gapped, regulated, and sovereign environments get the same agent capabilities.
• Faster approval to pilot. With no outside data flow, the security conversation is simpler.
• Predictable cost. Hardware you already own instead of per-token billing.
• No lock-in. The same config moves between local and hosted models with a provider change.

What to measure in a pilot: whether answers match hosted-model answers on the same questions, response time on your hardware, and how often tool calls fail.

Why AURA

• Production-tested. AURA started as Mezmo's internal harness for operating our own SaaS. Our engineering and SRE teams still use it in production, and it powers thousands of agent sessions each month in Mezmo's hosted platform.
• Built for air-gapped deployments. AURA runs air-gapped when model providers and MCP servers are reachable locally.
• One config format, any model. Ollama, any OpenAI-compatible local server, or hosted providers.
• Every MCP transport. stdio for local child processes, plus Streamable HTTP and SSE.
• Traceable end to end. OpenTelemetry traces can go to a collector inside your network.

Getting good results from local models

The video on this page is from an engineer who runs AURA on local models day to day. His recommendations:

1. Start at roughly 30B parameters or larger. Smaller models struggle with multi-step debugging.
2. Prefer llama.cpp or LM Studio over Ollama. Both give more control and more reliable results. Ollama works, and it's what this example uses.
3. Use AURA's OpenAI provider for llama.cpp and LM Studio. It already handles reasoning settings. Point it at your host and port.
4. Reference the model by alias. Keep tuning in the model server's own config, and use agent.llm.additional_params only for what has to travel with the request.

His setup runs on 24 GB of GPU memory.

Start from where you are

• Already run Ollama? Use this example, or the even smaller minimal/ollama.toml.
• Run llama.cpp or LM Studio? Use the OpenAI provider with your local base_url, as shown in the video.
• Fully air-gapped host? Install the MCP server package ahead of time (npm install -g @modelcontextprotocol/server-filesystem) and change cmd to ["mcp-server-filesystem"].
• Need AWS-hosted but in-account instead? See the runbook knowledge base recipe, which covers Bedrock models and knowledge bases.
• Worried about small context windows? Turn on Scratchpad to keep large tool outputs out of context.

Look inside the config

[mcp.servers.filesystem]
transport = "stdio"
cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem"]
args = ["/tmp/aura-sandbox"]

[agent]
name = "Local File Assistant"
system_prompt = """
You answer questions about the files in /tmp/aura-sandbox. Use the
filesystem tools to read them; never guess file contents. Quote the
relevant lines and name the file they came from.
"""

[agent.llm]
provider = "ollama"
model = "qwen3:30b-a3b"
base_url = "http://localhost:11434"
temperature = 0.2
fallback_tool_parsing = true

[agent.llm.additional_params]
num_ctx = 32000

To make this an SRE agent, replace the filesystem server with your infrastructure's MCP servers, and keep the [agent.llm] block.

Try it in four steps

1. Install AURA (Linux or macOS):

curl -fsSL https://raw.githubusercontent.com/mezmo/aura/main/scripts/install.sh | bash

2. Install Ollama and pull a tool-capable model:

ollama pull qwen3:30b-a3b

3. Create a directory the agent may read:

mkdir -p /tmp/aura-sandbox && cp README.md /tmp/aura-sandbox/

4. Download the example and run it:

curl -fsSLO https://raw.githubusercontent.com/mezmo/aura/main/examples/complete/local-ollama-stdio.toml
aura --config local-ollama-stdio.toml

More detail: AURA Ollama guide.

Then ask: "Summarize the README and quote the install steps."

FAQ

Is anything sent over the network?

At request time, no. npx -y downloads the filesystem MCP server on first run, and Ollama downloads the model when you pull it. On an air-gapped host, install both ahead of time.

Which local models work best?

Tool-capable models of about 30B parameters or more. The video recommends families such as Qwen, Gemma, and GLM at that size.

Can I use local models in an agent team?

Yes, with one limitation: fallback_tool_parsing currently applies only to single-agent mode. Models with native tool calling work in orchestration.

What hardware do I need?

It depends on the model. The video's setup uses 24 GB of GPU memory.

Next steps

• Engineers: Clone the example, run it, and then swap in one of your own MCP servers.
• Security reviewers: Read the security and data-handling model alongside this config.
• Everyone: Join the AURA Slack community.