How to build your own agentic harness in Python

Most people use AI agents through prebuilt interfaces and frameworks, but you can actually build the core of an agent yourself in plain Python. Doing that once gives you a much deeper understanding of what agents really are, how they talk to models, how tool calling works, and how features like memory and model routing are wired together.

This guide walks through building a simple but capable agentic harness step by step. You’ll start with a basic chat loop, then add tools, an agentic loop, MCP servers, persistent memory, and finally model switching—all in pure Python.

Setting up your Python environment

You can use any code editor, but the example assumes a folder-based Python project (for example, in VS Code). Inside an empty project folder, you’ll create a virtual environment and install a few lightweight dependencies.

A convenient way to manage dependencies and virtual environments is uv. In your project folder, run:

uv init .

Then install the three core packages:

uv add mcp openai pydantic

These are not full agent frameworks. They simply provide:

openai – a Python client to talk to OpenAI-compatible LLM APIs (including local servers that mimic the OpenAI API)

mcp – utilities for working with Model Context Protocol (MCP) servers

pydantic – typed data models for Python (handy, but optional for the core logic)

Understanding the role of the LLM in an agent

A large language model (LLM) is fundamentally a text-in, text-out system. You send it a sequence of messages (the conversation history), and it predicts the next piece of text. That text might be natural language, JSON, or a structured instruction—still, it’s just text.

On its own, an LLM cannot:

• Read or write files

• Call APIs

• Browse the web

• Control your computer

What it can do is describe that it wants to take those actions. Your code—the harness—acts as the body around the LLM “brain”: it interprets those descriptions as tool calls, executes them, and feeds the results back into the model. That loop is what makes an “agent.”

Step 1: A basic chat completion in Python

The first building block is a function that sends messages to an LLM and returns the assistant’s reply. Using the OpenAI Python client, the pattern is:

• Configure the client (API key and base URL)

• Choose a model name

• Build a list of messages with roles: system, user, and assistant

• Call client.chat.completions.create(...)

• Extract response.choices[0].message.content

A minimal interactive loop looks like:

• Read user input from the terminal

• Append it as a user message

• Call the LLM

• Print the assistant’s content

At this stage, you have a basic chat assistant. Next, you’ll make it capable of calling tools.

Step 2: Using OpenAI vs local LLMs (Ollama)

The same client pattern works whether you’re using OpenAI’s hosted models or a local LLM that exposes an OpenAI-compatible API.

Using OpenAI-hosted models

To use OpenAI directly:

1. Create an API key at https://platform.openai.com/api-keys

2. Set client = OpenAI(api_key="YOUR_KEY")

3. Omit the base_url (the client will default to OpenAI’s servers)

4. Choose a model like gpt-4.1 or another supported name

This is usually the simplest path, especially on low-end hardware.

Using a local LLM with Ollama

If you prefer running models locally, you can use Ollama:

1. Install Ollama from https://ollama.com and ensure the ollama command works in your terminal.

2. Pull a model that fits your hardware, for example:

ollama pull qwen2.5:4b or ollama pull qwen2.5:2b

3. Point the OpenAI client to the Ollama server:

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

4. Set model="qwen2.5:4b" (or whatever you pulled)

The rest of your code stays the same. This is a key idea: your harness doesn’t care whether the model is remote or local, as long as it speaks the OpenAI-style API.

Step 3: Adding basic tool calling

Now you’ll give the agent “hands” by defining tools as Python functions. In the example harness, the first tools operate in a workspace directory:

• list_files: list all files in the workspace

• read_file: read the contents of a file in the workspace

Each tool is just a normal Python function. To make them callable by the LLM, you need two things:

1. A mapping from tool name to function, for example:

tools = {"list_files": list_files, "read_file": read_file}

2. A tool schema that describes each tool to the model: its name, description, and parameters. This is a JSON-like structure you pass to tools=... when creating a chat completion.

A schema entry for read_file might look like:

• type: "function"

• name: "read_file"

• description: what the function does

• parameters: JSON schema for an object with a required file_name string

Frameworks like LangChain or others generate these automatically, but when you’re writing a harness from scratch, you define them yourself. You can even ask an LLM to generate the schema for a given Python function.

Step 4: Executing tool calls from model output

Once tools are defined and their schemas are passed to the model, the LLM can respond with tool calls instead of plain text. The response structure usually includes:

• message.content – normal assistant text

• message.tool_calls – a list of requested tool invocations, each with a name and JSON arguments

Your harness must:

1. Inspect message.tool_calls

2. For each call, parse the JSON arguments

3. Look up the corresponding Python function in your tools dict

4. Call the function with those arguments

5. Capture the result (e.g., as a string)

6. Append a new message to the conversation with:

• role: "tool"

• tool_call_id: the ID from the LLM’s tool call

• content: the tool result

Then you send the updated messages back to the LLM so it can decide what to do next with the tool output.

Step 5: Introducing the agentic loop

In the first tool-calling version, the code only allowed a single round of tool use: the model could call a tool once, get a follow-up completion, and then stop. That isn’t enough for real tasks, where the agent might need to:

• List files

• Read several files

• Write a new file

• Possibly repeat until it reaches a goal

To support this, you wrap the interaction in an agentic loop:

• while True:

• Call the LLM with the current messages and tools

• If the assistant returns plain text with no tool calls, treat it as the final answer and break

• If there are tool calls, execute them, append tool results as messages, and loop again

This lets the agent chain multiple tool calls until it decides it has enough information to answer the user. For example, it can:

• List all files

• Read several of them

• Create a new combined.txt file with merged contents

• Then finally respond to the user explaining what it did

If you want a deeper conceptual overview of this pattern and related concepts like autonomy and orchestration, it pairs well with the article AI agent fundamentals: tools, autonomy, and how to build your own.

Step 6: Integrating MCP servers as dynamic tools

So far, your tools are hardcoded Python functions. The next step is to allow your harness to plug into external tool providers using the Model Context Protocol (MCP).

MCP defines a standard way for tools to expose capabilities to AI agents. Each MCP server can provide multiple tools (like “get current time” or “fetch a URL”), and clients can discover and call them at runtime.

In the harness, you:

1. Define MCP servers in a config structure (for example, mcp_servers.json), including how to start them. In the example, two stdio-based servers are used:

• A time server (returns current time for a given timezone)

• A fetch server (retrieves web pages)

2. Use the MCP client library and asyncio to:

• Start each server (via stdio or another transport)

• Wait for the session to initialize

• Discover the list of tools exposed by that server

• Convert each tool’s definition into a schema entry compatible with the LLM’s tools parameter

3. Merge these MCP-derived tool schemas with your local tool schemas so the model sees a unified tool list.

At call time, your generic call_tool function checks whether the requested tool name exists in your local tools or in the MCP sessions. If it’s an MCP tool, you call it through the MCP session API, then pass the result back to the LLM exactly like a local tool.

This means the agent can now do things like:

• “What time is it in Dubai?” → calls the MCP time tool

• “Fetch and summarize this website” → calls the MCP fetch tool and then uses local tools to process the content

For more on how MCP servers fit into broader agentic systems, you may also find this guide on building a personal agentic operating system helpful.

Step 7: Adding persistent memory

So far, the agent’s memory is limited to the current conversation. Once you restart the program, it forgets everything. To give it persistent memory across runs, you can store information in a file and inject it into the system prompt on every new session.

A simple pattern is to use a memory.md file in your project directory:

• load_memory(): read the contents of memory.md (or return an empty string if it doesn’t exist)

• save_memory(text): append or write new information into memory.md

Then you:

1. Add a save_memory tool to your tools list and schema so the LLM can decide what to store.

2. When starting a conversation, you load the memory and include it in the system message, for example:

“Here is what you know about the user from past sessions: ...”

Now the agent can:

• Learn “My name is Alex and I’m 26” via a tool call that writes to memory.md

• On the next run, answer “What is my name and age?” correctly, because that file’s contents are injected into the system prompt.

This is a very simple form of memory, but it demonstrates the core idea. More advanced systems use embeddings, vector databases, and retrieval strategies, but the harness pattern is similar.

Step 8: Supporting model routing and commands

The final step is turning all of this into a more polished harness script that can:

• Manage multiple models

• Switch models at runtime

• Connect to MCP servers from a config file

• Expose convenience commands in the terminal

Model configuration

You can define a model_config dictionary that lists available models, for example:

• "local" – a Qwen model via Ollama

• "local_small" – a smaller local model

• "gpt" – an OpenAI GPT model

• "claude" – an Anthropic Claude model

Each entry includes:

• The base URL (OpenAI or local)

• The model name

• Which client class to use (OpenAI, Anthropic, etc.)

The harness keeps track of the currently selected model name. When you run the agent, it looks up the config for that model and instantiates the appropriate client on the fly.

Slash commands in the terminal

To make the harness nicer to use, you can add simple commands that start with /:

• /models – list all available models from model_config

• /model <name> – switch to a different model (for example, /model local_small)

• /tools – list all tools (local + MCP) currently available

• /memory – print the contents of memory.md

• /quit – exit the harness

Any line that does not start with / is treated as normal user input and sent through the agentic loop.

Putting it all together

By the end of this build, your Python harness can:

• Talk to local or remote LLMs through a unified interface

• Expose local Python functions as tools

• Discover and use tools from MCP servers

• Run an agentic loop that chains multiple tool calls

• Maintain persistent memory across sessions

• Switch between multiple models on demand

From here, you can extend it with more powerful tools (APIs, databases, file systems), smarter memory (RAG, embeddings), and richer UIs (web, desktop, or chat integrations). But even this minimal harness gives you a clear, end-to-end view of how modern AI agents are really built—well beyond just calling a chat API.