What running my own AI system taught me about deploying AI
Eight ways of working from building a private, local-first AI system, and why they apply at enterprise scale
I advise organisations on how to make AI work. So I wanted to understand, from the inside, the problems they face: keeping data private, choosing the right model for the job, connecting AI to their own knowledge, and keeping it all current as the technology moves. The best way I know to understand a problem is to run it myself, every day.
So I built a private, local-first AI system on my own hardware. It is not a demo. I use it daily, and it has changed how I think about AI deployment.
What it is
At its heart are open-weight models running locally on a high-memory laptop, so nothing I work on has to leave my machine. The system is built almost entirely from open-source components. Around them sit several AI tools: a desktop assistant, a coding agent in my editor and a browser-based chat interface. Rather than connecting each tool separately to my data, they all connect through a single MCP gateway to a personal knowledge base of around 2,600 pages. The system also includes private web search and local image generation, and semantic search across the knowledge base is the next stage.
Building it taught me eight things.
1. Scan the landscape, then validate before you choose
The open-source AI landscape is crowded and moves fast, and many tools look similar on paper. Before adopting any component, I run a structured scan of the domain: what options exist, what each one actually does, where they tend to fail, and what is missing. The strongest candidates are then validated hands-on against my own requirements before anything is adopted.
It takes a little longer at the start, but it has saved me from several tools that looked impressive and would not have survived real use.
At enterprise scale: tool selection deserves the same discipline as any other procurement. A structured scan of the market, followed by hands-on validation against real requirements, avoids expensive commitments to tools that do not fit.
2. Connect your knowledge once, not once per tool
My first instinct was to connect each AI tool to my data separately. That quickly becomes a tangle: every tool configured differently, every change made several times. Routing everything through one MCP gateway changed that. The knowledge is connected once, and every tool benefits.
At enterprise scale: the same principle applies. Organisations adopting several AI tools should connect their knowledge and systems once, through a governed layer, rather than building a new integration for every tool that comes along. It is simpler to secure, simpler to audit and simpler to change.
3. Context is a budget, so spend it carefully
Connecting more tools to an AI model sounds like it should make it more capable. In practice, every tool the model can see takes up part of its context window, its working memory. Load too many and the model has less room for the actual task, and it gets worse at choosing the right tool.
So I built my own MCP tool selector, which loads only the tools each task needs. The difference was noticeable straight away: sharper answers, better tool choices and fewer wasted steps.
At enterprise scale: what the model sees should be designed, not accumulated. As organisations connect AI to more systems, deciding what goes into the context, and what stays out, becomes one of the most important design choices they make.
4. Choose the model for the task
Running models locally showed me exactly where they shine and where they do not. Local models are excellent for private material, everyday drafting and summarising, and work that has to stay offline or keep costs down. The most demanding reasoning and coding work still benefits from the leading frontier models.
The answer is not one model, but a deliberate mix, with each task routed to the model that fits it best.
At enterprise scale: a single-model strategy is rarely the right one. A portfolio approach, with clear rules about which data and which tasks go where, gives better results and better control of cost and risk.
5. Give the system a maintenance rhythm
The open-source components that hold an AI system together are updated constantly, sometimes to fix critical bugs and security issues. Without a rhythm, it is easy to end up running a component with a known flaw.
So I run a monthly scan of every component in the stack: current versions, changelogs and security notices, with a clear recommendation to upgrade, hold or defer. The aim is to catch critical problems in open-source software before they become my problem.
At enterprise scale: this is where change management earns its keep. Every component change should be assessed, tested, approved, scheduled and reversible, with a clear rollback plan. A regular review cycle, feeding a proper change process, keeps a live service secure without putting it at risk.
6. Test new models before you switch
New, more capable models arrive every few weeks, and it is easy to keep running yesterday's model when a better one is available. So a separate review looks at new models, to see whether one should replace what I use today.
Before I switch, I test the candidate against a golden set: a fixed collection of prompts I use regularly, with responses I already know are good. If the new model matches or beats those responses, it can be adopted. If it does not, I stay where I am. This kind of regression testing stops a "better" model from quietly making my everyday work worse.
At enterprise scale: a model change is a release, not a setting. Testing every candidate against a golden set of real prompts and known good responses, then releasing it through the same change process as any other update, lets an organisation adopt better models quickly and with confidence.
7. Write things down, for people and for agents
I keep a simple set of project documents: key facts, decisions, known bugs and open issues. I started doing this for my own benefit. I kept doing it because AI coding agents work far better with it. An agent that can read why a decision was made makes fewer wrong assumptions and needs less correction.
At enterprise scale: as more engineering work is done with AI agents, clear documentation stops being a nice-to-have. The way work is structured, documented and handed over decides how reliably agents can contribute.
8. Build in stages, and prove each one
I built the system in phases. First, getting every tool reading the knowledge base through the gateway. Only once that was working reliably did I start on semantic search. Each stage was proved working before the next began.
At enterprise scale: this is the same discipline that makes larger AI programmes succeed. Prove one step end to end, then extend. It is slower at the start and much faster overall.
Why this matters
None of these lessons is really about running AI on a laptop. Private data, choosing models, connecting knowledge, managing context and keeping everything current are the same problems organisations face, just at a larger scale. Running a system like this every day means that when I advise on them, I am speaking from experience, not from a slide.