The Missing Layer: Why Your AI Agents Keep Failing in Production
Prompt engineering is 5% of making AI agents work. The other 95% — memory, tools, safety, observability — is harness engineering. Here's what that means and why it matters.
Everyone's talking about prompt engineering.
You've seen the LinkedIn posts. The courses. The job titles. "Prompt Engineer, $150k base." Companies hiring people to write better instructions for ChatGPT.
Here's what nobody tells you: prompt engineering is the smallest part of making AI agents actually work.
We've spent the last year helping companies build agent systems that run in production — not demos, not pilots, not proofs of concept that die after the kickoff meeting. Real systems that do real work and don't hallucinate unauthorized discounts at your biggest customer. (That one actually happened at a company we know. The AI sales agent had Salesforce access, no guardrails, and a very creative interpretation of its authority.)
What separates the demos from the production systems is not the quality of the prompts. It's everything around the prompts — the scaffolding, the configuration, the architecture, the safety controls, the memory design, the monitoring. We call this harness engineering, and it's the discipline that the AI industry desperately needs and barely talks about.
What Harness Engineering Is
Think about it this way. An LLM is a powerful reasoning engine. But a reasoning engine with no operating system is useless.
You need memory management — what does the agent know, what does it forget, how does it retrieve the right context without flooding the model with noise? You need tool management — what tools does the agent have access to, how are they scoped, what happens when a tool fails? You need safety rails — what's the agent allowed to do, what requires human approval, what's completely off-limits? You need observability — when something goes wrong at 2am, can you trace exactly what the agent did and why?
None of that is prompt engineering. It's a different discipline entirely.
Harness engineering is the work of designing and maintaining the layer that makes AI agents reliable, safe, and useful in production. It covers:
- System architecture — how agents are structured, how they communicate, what patterns govern their behavior
- Context management — what information agents receive at each step and how that's curated (Anthropic calls this "context engineering" and it's harder than it sounds)
- Memory systems — short-term working memory, long-term retrieval, and the retrieval strategies that determine what the agent actually uses
- Tool design and configuration — building tools that fail gracefully, scope permissions correctly, and integrate with real systems
- Safety and guardrails — input validation, output filtering, human-in-the-loop design, prompt injection defenses
- Observability — tracing, logging, cost monitoring, and the ability to answer "why did it do that?"
- Multi-agent orchestration — supervisor patterns, handoffs, state management across a system of agents
A prompt is maybe 5% of this. The other 95% is the harness.
The Demo-to-Production Gap
Here's what the numbers look like right now.
Cleanlab surveyed 1,837 engineering and AI leaders across industries in 2025. Only 95 of them — 5% — had AI agents actually running in production.
Of those 95, 70% of regulated enterprises said they rebuild their AI agent stack every three months. Not because they want to, but because the stack keeps changing, things break, and there's no stable foundation underneath.
Less than one in three production teams are satisfied with their observability and guardrail solutions.
The failure modes are consistent across every company we've talked to:
Bad memory management. Teams dump all their documentation into a vector database, hope the model figures out what's relevant, and wonder why the agent gives confidently wrong answers. Context rot is real — as the context window fills up, accuracy degrades. You need a retrieval strategy, not just a vector store.
Brittle integrations. The agent has access to your CRM, your ticketing system, your Confluence. The demo works perfectly. Then you hit a rate limit, or an API schema change, or an auth token expiration, and the agent fails in a way it can't recover from. One failure cascades into the next.
No visibility. When an agent does something wrong — and it will — you need to understand why. What tools did it call? What did they return? What did it decide? Without proper tracing, you're debugging by asking the model to explain itself, which is like asking a confused employee why they did something three days later.
Security gaps. The OWASP GenAI Top 10 (2025) catalogs exactly what goes wrong: prompt injection via tool outputs, memory leakage between sessions, agents with over-permissioned tool access. Most demo-built agents ignore all of this.
These are engineering problems. Not prompt problems.
The Model Selection Problem Is a Harness Problem
Here's a real example of why the harness matters more than the model.
Kilo Code, an open-source coding assistant with over 1.5 million users, recently tested MiniMax M2.7 against Claude Opus 4.6 on three structured coding tasks: building a full-stack system from spec, investigating bugs from production logs, and running a security audit.
Both models found all 6 bugs. Both found all 10 security vulnerabilities. MiniMax M2.7 even produced a technically superior fix for a floating-point precision issue. The total cost for MiniMax: $0.27. For Claude Opus 4.6: $3.67.
The difference? Fix thoroughness. Opus added rollback logic, wrote 41 integration tests instead of 20 unit tests, and implemented defense-in-depth security patterns. MiniMax delivered 90% of the quality for 7% of the cost.
Now here's the harness question: which model should your agent use?
The answer depends on the task. Bug detection and diagnosis? MiniMax is arguably a waste of money to run on Opus. Generating production security fixes that need to ship without human review? You probably want Opus.
A well-designed harness routes different tasks to different models based on what the task requires — not based on which model topped a leaderboard. That routing logic, the cost-quality tradeoff, the confidence thresholds that determine when a cheaper model's output needs human review — that's harness engineering. A bad harness runs everything on the most expensive model and burns money. A worse harness runs everything on the cheapest model and ships broken security fixes.
Most companies don't even think about this. They pick one model and use it for everything. That's like using a sledgehammer for every job because it scored highest on a force benchmark.
Why This Matters Now
The market timing is real. Right now, most enterprises are trying to figure out why their AI pilots failed. They've spent six months and a lot of money on something that worked in the boardroom and broke in production.
They need someone who can look at their setup and identify whether the failure is in the model (usually not), the prompts (sometimes), or the harness (almost always). They need someone who can build the scaffolding correctly from the start, or diagnose and fix the one that's currently on fire.
The skills gap here is significant. Only 13% of enterprise respondents in the Teradata/NewtonX 2025 survey said they have the skills needed to implement AI agents at scale. Governance challenges and integration failures are the top barriers — both of which are harness problems.
The companies that are pulling ahead aren't the ones that found better prompts. They're the ones that invested in the infrastructure around their agents. The harness is what turns a clever demo into a system you'd trust with your business.
What We Do at GTA Labs
We've built agent systems across a range of industries — customer support automation, internal knowledge retrieval, multi-step workflow automation. The work is always the same at its core: figure out what the agent needs to do, design the harness that makes it do that reliably, and build the monitoring that tells you when it's not.
We audit existing agent setups and find the holes. We build new harnesses from scratch for companies starting fresh. We run retainers for organizations that want ongoing management and improvement.
If you're running AI agents that you're not fully confident in, or you're trying to get a pilot into production and hitting walls, that's the conversation we're built for.
The prompts are the easy part. Let's talk about the rest.
GTA Labs is an AI consulting firm based in Toronto. We specialize in building AI agent systems that work in production.