When AI Runs Your Infrastructure: Three Lessons for a Production-Grade MCP Server

An AI chat agent you tell, in plain language, "clone the test VM, keep the SSH keys and password, just give it a new name and a new IP" — and it simply happens: that is the moment Model Context Protocol (MCP) turns from a gimmick into a real operations tool. In a live demo it looks like magic.

It only becomes production-ready through the work you never see in the demo. We built, hardened and wired an MCP server for managing a Proxmox cluster into a chat agent. Here are the three lessons that separate "works in demo mode" from "runs every day."

The killer feature: duplicate instead of provision

The most valuable command is not "create a new VM" but "duplicate this one." The reason is both mundane and decisive: cloning a known, working machine is the safest way to get an immediately functional system. SSH keys, passwords, installed packages, configuration — everything is preserved 1:1. Only the hostname and IP change.

The trick behind it is a full clone, not a linked clone: the copy is completely independent of the original. If the source container is running, a snapshot is taken first and the clone is made from that, so production stays undisturbed. Then only the network parameters are rewritten and the machine is started. Because provisioning disappears, so do its classic failure modes — no "forgotten password," no "key not installed," no half-configured system.

Lesson 1: Security — profiles, not full access

A language model wired directly to an infrastructure API is powerful — and precisely because of that, dangerous. The naive implementation gives the agent every tool and the token every right. That is an invitation to an accident.

We work with permission profiles instead:

The default profile is operator. Destructive tools — delete, destroy, roll back, prune backups — are not even visible in it. They do not surface even when the model explicitly searches for them. Only under the full profile are they unlocked, and there a separate confirmation gate requires an explicit confirm parameter before anything irreversible happens.

The second line of defence lives outside the model: the API token only gets the rights the use case needs. A token that may clone and take snapshots but can neither open a shell inside the guest nor change node settings does little damage even under a compromised prompt. Least privilege is not a bureaucratic ritual — it is the layer that still holds when the first one fails.

Lesson 2: Scale — semantic tool routing

A complete infrastructure MCP quickly grows to nearly 300 tools. Attach them all to the model at once and two things happen: the context overflows, and the hit rate drops — the model gets lost among variants and calls the wrong tool.

The answer is routing via embeddings. Instead of 300 definitions, the model sees only three meta-tools:

• route_tools — describe in natural language what you want to do and get the matching tools back
• call_routed_tool — invoke a discovered tool with arguments
• proxmox_api_raw — an escape hatch for anything without a dedicated tool

At startup a small, CPU-fast model (bge-small via fastembed) embeds every tool description into a vector. A request like "list LXC containers on a node" is embedded too, and cosine similarity surfaces the relevant tools in milliseconds. The model works with a handful of matching tools instead of an unwieldy catalogue — faster, cheaper, more accurate.

One detail we learned the hard way: routing must be built so a call still works even when the model skipped the search step. A tool that is only callable after an exact, immediately preceding search produces exactly the confusing "tool not found" loops you were trying to avoid.

Lesson 3: Reliability — the connection that dies

The third lesson is the least glamorous and the most important: the demo never fails on the happy path — it fails on the error cases.

Chat platforms hold a separate per-user connection to each MCP backend. As long as the backend is running, all is well. But restart the backend — after an update, say — and that connection loses its session. Many clients then retry a reconnect two or three times, give up, and leave behind a zombie connection: technically connected, but with not a single tool entry. The agent then reports "tool not available" — even though the server and the code are perfectly healthy.

The fault is not in the code but in the connection lifecycle. The right answer is therefore operational, not programmatic: a self-healing watchdog. A small service watches the chat platform's log and reacts to the exact line that marks the zombie state. The moment it appears, it triggers a clean re-initialization automatically — disconnect, rebuild, reload tools. In our tests the connection was back within two seconds of a backend restart, with no human action at all.

That is the real maturity test of an agent system: not whether it works in the best case, but whether it recovers on its own after a restart, a timeout or a network blip.

Takeaways

• Duplicating beats provisioning. A full clone of a known machine is the safest path to an immediately working system.
• Security in layers. Permission profiles in the server, a confirmation gate for destructive operations, and a token with minimal rights — each layer catches what the previous one lets through.
• Less is more at the context level. Semantic routing collapses hundreds of tools to three and makes the model faster and more accurate.
• The error case is the product. Self-healing after restarts is not a bonus but the condition for an agent to stay reliable in daily use.

Take these four points seriously and you move from an impressive demo to a tool you actually trust with your own infrastructure. That is exactly where the line runs between a toy and an operational asset.