AgentHippo
AI agents in production · 15 September 2026 · 9 min read
What a company needs before its AI agents can act
An AI agent that sends, changes, and spends on your company’s behalf needs more than a good model. Taking one from demo to production takes governance that runs: eight layers of infrastructure that hold even when the model is wrong.
Most agent prototypes work. They answer the question, call the tool, draft the email. The hard part starts the day an agent is allowed to act for someone other than the person who built it: to send the email, change the record, move the money. From then on, the question is not whether the model is capable. It is what happens when it is wrong.
Most companies answer that question with policy: an acceptable-use document, a review board, a line in the system prompt that says “never show one customer another customer's data.” Policy matters, but an agent does not read it, and a model that has been confused, manipulated by text it read, or is simply mistaken will not obey it. The controls that matter have to live where the model cannot reach them: in the identity it is handed, the budget it draws from, the permissions of the database it queries, a lease it must keep renewing to stay alive, and a record it cannot edit.
We call this runtime governance: governance that runs, as infrastructure, on every action an agent takes. It is the difference between a policy that says nobody may wire more than $50,000 and a payments system that refuses the transfer. Security teams made this move once before, when zero trust pushed authorization out of the application and into checks every request has to pass. It takes eight layers. Most prototypes have none of them.
Enforce outside the agent. If the agent can talk its way around a control, it is not a control.
› The enterprise agent stack
Every agent a company runs sits on some version of this stack. The eight numbered layers are its runtime governance; the rest, from models to infrastructure, most companies already run.
1 CHANNELS Web · IDE · Slack / Teams · Email · API · schedule · events
│ message + who is asking
▼
┌─ 2 EDGE & IDENTITY ───────────────────────────────────────────────────────────────────┐
│ API gateway + WAF · SSO (OIDC / SAML: Okta, Entra ID) · tenant routing · rate limits │
│ every request stamped: user_id · tenant_id · agent_id · session_id · trace_id │
└───────────────────────────────────────────┬────────────────────────────────────────────┘
│ authenticated turn
▼
┌─ 5 CONTROL PLANE ───┐ ┌─ 3 AGENT RUNTIME ─────────────────┐ ┌─ GUARDRAILS ─────────┐
│ lease ⇄ runtime │ │ ┌─ YOUR AGENT HERE ──────────────┐ │ │ inline on every hop │
│ kill in under 1 min │◄►│ │ Agent Packs, LangChain, CrewAI │ │◄►│ prompt injection │
│ deny user · caps │ │ └────────────────────────────────┘ │ │ PII / DLP redaction │
│ pin version │ │ orchestrator · durable workflows │ │ policy (OPA, Cedar) │
│ alerts → on-call │ │ human-in-the-loop approvals │ │ tool allow-lists │
│ │ │ session state · retries · sandbox │ │ output schema check │
│ │ │ no keys, no DB creds on the host │ │ │
└──────────────────────┘ └─────────────────┬──────────────────┘ └──────────────────────┘
┌──────────────────────────────┼─────────────────────────────┐
│ think │ act │ ground
▼ ▼ ▼
┌─ 4 MODEL GATEWAY ───────┐ ┌─ 6 TOOLS & DATA ACCESS ─┐ ┌─ KNOWLEDGE & MEMORY ───────┐
│ LLM proxy (e.g. LiteLLM) │ │ MCP servers │ │ ingestion + chunking ETL │
│ routing & fallback │ │ internal APIs · SaaS │ │ vector DB + keyword search │
│ prompt / semantic cache │ │ CRM, ITSM, GitHub │ │ ACL-aware retrieval │
│ budget checked pre-call │ │ code sandbox (microVM) │ │ session / long-term memory │
│ over cap → refused │ │ browser automation │ │ warehouse, systems of rec. │
│ cost attributed per user │ │ acts AS the user (OBO) │ │ SharePoint, Drive, wikis │
│ provider keys held here │ │ wrong user → 0 rows │ │ │
└────────────┬─────────────┘ └──────────────────────────┘ └────────────────────────────┘
│ your keys, or our credits
▼
┌─ MODEL PROVIDERS ──────────────────────────────────────────────────────────────────────┐
│ Frontier APIs Cloud-hosted Self-hosted open weights │
│ Anthropic, OpenAI, AWS Bedrock, GCP Vertex, vLLM / TGI on a GPU pool, │
│ Google Azure AI Foundry fine-tuned small models │
└────────────────────────────────────────────────────────────────────────────────────────┘
══════════════════════════════ CROSS-CUTTING · every layer ═══════════════════════════════
┌─ 7 EVIDENCE & EVALS ────┐ ┌─ SECURITY & GOVERNANCE ──┐ ┌─ 8 BUILD & RELEASE ───────┐
│ OpenTelemetry traces │ │ agent workload identity │ │ IDE / CLI · templates │
│ cost · tokens · latency │ │ short-lived scoped creds │ │ signed, versioned packs │
│ per user/agent/version │ │ secrets vault (KMS/HSM) │ │ prompt + tool versioning │
│ tamper-evident (WORM) │ │ data residency · DLP │ │ offline eval suites │
│ eval harness, LLM judges │ │ SOC 2 · ISO 42001 │ │ eval-gated CI/CD │
│ regression/drift alerts │ │ EU AI Act mapping │ │ canary + one-cmd rollback │
│ trace replay for debug │ │ │ │ agent registry / catalog │
└──────────────────────────┘ └──────────────────────────┘ └────────────────────────────┘
┌─ INFRASTRUCTURE ───────────────────────────────────────────────────────────────────────┐
│ Kubernetes / VMs / Cloud Run · VPC + private link · Postgres · queues (Kafka / SQS) │
│ object storage · GPU capacity · IaC (Terraform) · multi-region DR · FinOps tagging │
└────────────────────────────────────────────────────────────────────────────────────────┘
LIFE OF ONE REQUEST
1. Alice asks in Slack [1] → the edge [2] checks SSO, stamps user + trace id
2. The runtime [3] holds a live lease from the control plane [5]; guardrails screen input
3. Agent loop: think via [4] → act as Alice via [6] → ground in knowledge (N steps)
4. Risky action (payment, prod deploy, customer email) → human approval in [3]
5. Guardrails check the output → reply; evidence [7] holds trace, version, cost, audit
WHERE A CTO SHOULD SPEND ATTENTION
Buy / adopt model providers, IdP, vector DB, Kubernetes, OTEL backend
Standardize one gateway [4], one tool protocol (MCP) [6], one trace schema [7]
Build (your edge) agents + prompts, integrations to YOUR systems, eval datasets
Non-negotiable per-user identity [2], kill switch [5], spend caps [4], audit trail [7]
Top risks prompt injection via tools/docs, over-privileged agents, runaway cost,
silent quality regressions after a model or prompt change Two things in the picture matter more than any single box. The agent sits in the middle holding nothing worth stealing: no provider keys, no database credentials, no standing permission. And every path out of it runs through a layer that can refuse.
› The eight layers of runtime governance
Each layer answers one question the model cannot be trusted to answer for itself, in the order a request meets them.
1 Channels Where does the request come from?
Agents earn their keep where work already happens: Slack, Teams, email, a scheduled job, an API another system calls. The hard part is not the chat integration. It is making sure every message arrives with the person behind it attached, so every layer below knows who is asking. An agent that lives in one developer's terminal never has this problem, which is also why it never becomes a workflow anyone else relies on.
2 Identity Who is asking?
The agent has to know who is asking, and it has to learn it from infrastructure, not from the conversation. A user who types “I am the CFO” has changed the prompt, not their permissions. The edge authenticates each person against the company's identity provider and hands the runtime a short-lived token for that turn. The token, not the transcript, is what every layer downstream checks.
3 Agent runtime What can the agent reach?
Treat the runtime as untrusted, because the model inside it takes instructions from whatever text it reads. Anything in its environment, a provider key or a database password, is one prompt injection (instructions hidden in a web page, email, or document the agent reads) away from being repeated to an attacker. So the runtime should hold nothing worth stealing. It reaches models through the gateway and data through a connector, both of which check the turn's identity rather than a secret the runtime owns, and it runs one signed build in a sandbox.
4 Model gateway How much may it spend?
An agent can call a model in a loop, and a loop that runs overnight is a bill nobody approved. Budgets have to be enforced before each call, not reconciled after the invoice arrives. That takes a gateway between every agent and every provider that knows which agent and which person each call belongs to, and refuses the call that would cross the cap. The same gateway turns the monthly bill from one number into a list of names.
5 Control plane How do we stop it?
An off switch has to work when the agent is misbehaving, which is exactly when a command pushed to a busy or compromised process is least likely to land. The reliable design inverts the direction: the runtime holds a lease and must keep renewing it to go on working. Revoke the lease and the agent stops at its next renewal, without anyone reaching into the machine it runs on. The same channel carries per-user denials, concurrency caps, and the version each deployment is pinned to.
6 Data access What may it see?
The data leak that matters most in agent systems is not exotic. The agent connects as one service account that can read every row, and one user's question returns another user's records. The fix is old: act on behalf of the user and let the database enforce it with the permission rules it already has, such as row-level security in Postgres. The model can ask for anything; the database answers as the person who asked. The test is one query: as the wrong user, it returns zero rows.
7 Evidence What did it do?
When an agent acts for a customer, someone will eventually ask what it did, and “we can check the logs” is not an answer a customer, an auditor, or a regulator accepts. Evidence is a record of who asked, which signed version ran, what it touched, and what it cost, written outside the runtime to storage that cannot be altered after the fact. Logs help you debug. Evidence settles the question.
8 Build and release Which version ran?
An agent is a prompt, a set of tools, a model, and an engine, and a change to any one of them changes its behavior. Version them together as one signed artifact, pin each deployment to a version, and make rollback restore the previous behavior, not just the previous code. Without that, nobody can say which agent ran last Tuesday, let alone bring it back.
› The reference deployment
The eight layers as AgentHippo deploys them. Two refusals carry the design: a call over budget never reaches a model, and a request for another user’s data returns nothing. Everything inside the boundary runs in your environment.
Alice · Bob via Slack · WhatsApp · Telegram · API · schedule
│ message + who is asking
▼
┌─ 1·2 EDGE & IDENTITY ─────────────────────────────────────────────────────────────────────────────────┐
│ TLS proxy → oauth2-proxy ⇄ your IdP (Okta · Entra · Google) → auth-broker │
│ out: the verified user + a short-lived token for this turn │
└─────────────────────────────────────────────────────────────┬──────────────────────────────────────────┘
│ authenticated turn
▼
┌─ 5 CONTROL PLANE ────────┐ ┌─ 3 AGENT RUNTIME ─────────────────────────────────────────────────────┐
│ agenthippo-control │ │ ┌─ YOUR AGENT HERE ────────────┐ serve · sandbox │
│ lease (TTL) ⇄ runtime │◄──│ │ signed Agent Pack │ on your VM · Cloud Run · │
│ kill · deny user · caps │──►│ │ agent.yaml · prompt · skills │ Databricks Apps · behind your proxy │
│ pin version · console │ │ └──────────────────────────────┘ no provider key, no DB credential │
│ alerts → PagerDuty / IRM │ │ a lease it keeps renewing │
└───────────────────────────┘ └──────────┬──────────────────────┬─────────────────────────┬────────────┘
▲ signed pack │ metered call │ as the user │ every action
┌─ 8 BUILD & RELEASE ──────┐ ▼ ▼ ▼
│ IDE / CLI → Agent Pack │ ┌─ 4 GATEWAY ───────┐ ┌─ 6 DATA (OBO) ─────┐ ┌─ 7 EVIDENCE ───────────┐
│ sign → GHCR │ │ agenthippo-gateway │ │ connector container │ │ OTEL → evidence-server │
│ image · store tag │ │ identity + budget │ │ Postgres RLS │ │ MinIO · S3 Object Lock │
│ one-command rollback │ │ → LiteLLM → model │ │ DynamoDB (STS tags) │ │ WORM · Spotlight │
└───────────────────────────┘ │ over cap → refused │ │ Unity Catalog │ │ who·version·touched·cost│
└────────────────────┘ │ wrong user → 0 rows │ └─────────────────────────┘
└─────────────────────┘ Nothing in this picture depends on the model behaving well. If a prompt is injected, the runtime has no secrets to leak. If the agent loops, the gateway stops paying for it. If it asks for another customer's data, the database returns nothing. If it has to stop, the lease stops it. That property is worth copying whatever components you choose.
Miss one layer and that is where it breaks. No spend cap, and a loop runs up the bill overnight. No per-user check, and one customer sees another's data. AgentHippo sets up all eight.
Components are defaults, not requirements: bring your own identity proxy, object storage, or provider keys. The control plane runs hosted by AgentHippo or on your own infrastructure.
› What you already have
Most of the enterprise stack is not new. You already choose model providers, run search and document stores, have security and model-risk teams, and operate Kubernetes or VMs; none of that needs replacing. Guardrails that screen prompts and outputs are worth having too, but they are advisory by construction: they make the model wrong less often. The eight layers decide what happens when it is wrong anyway.
› At a glance
| Layer | Without it | Building it yourself | With AgentHippo |
|---|---|---|---|
| 1 Channels and users | The agent lives in one developer's terminal. Nobody else can use it, so it never becomes a real workflow. | A bot per channel, each with its own auth, threading, and retry logic. Slack alone takes a week. | Agent Anywhere: Slack, WhatsApp, Telegram, and an API from one deployment. Scheduled runs from the fleet view. |
| 2 Identity edge | Every request looks the same to the agent. It cannot tell Alice from Bob, so it cannot treat them differently. | An OIDC integration, token exchange per turn, and a way to pass identity through every tool call without leaking it into the prompt. | Identity carried from the channel to the database on every turn as a short-lived token. Included in every production tier. |
| 3 Agent runtime | A provider key sits in an environment variable on a box someone forgot about. One prompt injection reads it out. | A hardened host, secret injection, a process supervisor, health checks, and a way to update the agent without SSH. | One command on any VM or cloud. No provider key ever lands on the host. Blueprints: single VM, Google Cloud Run, Databricks Apps, or behind your existing proxy. |
| 4 Model gateway | A loop burns for days before finance notices. The bill is one number with no name on it. | A proxy in front of every provider, per-agent counters, a pre-call check that actually blocks, and attribution to a user rather than a key. | Hard per-agent budgets checked before each call, refused at the cap, every dollar attributed to a person. Bring your own keys or use credits. |
| 5 Control plane | The prototype quietly became production, and the only off switch is finding the box and killing the process. | A lease protocol the agent obeys, a console, deny rules, concurrency caps, version pinning, and alerts wired into on-call. | Every deployment polls a lease. Kill one agent, one user, or the fleet in under a minute. Hosted by us, or on your VM with Self-Hosted Stack. |
| 6 Data access | The agent logs in as one service account that can read every row. Bob's question returns Alice's data. | Row-level security policies, per-user connection or token exchange, and proof that the prompt layer cannot bypass any of it. | Agents act as the requesting user; your database enforces it. Blueprints: Postgres RLS data plane, DynamoDB data plane, Databricks Unity Catalog. |
| 7 Evidence archive | "What did the agent do last Tuesday?" takes a week of log archaeology and ends a customer relationship. | Structured action logs tied to user, session, and version, hash-chained so they cannot be edited, stored where you control retention. | A tamper-evident record of who asked, which signed version ran, what it touched, and what it cost. Queryable on hosted tiers; WORM archive in your storage on self-hosted. |
| 8 Build and release | The agent is a prompt in a repo and a tool list in someone's head. Nobody can reproduce last month's behavior. | A packaging format, signing, a registry, version comparison with data, and rollback that actually restores the old behavior. | Agent Packs: prompt, tools, and engine as one signed, versioned artifact. Build in the IDE or CLI, roll back with one command. |
Every layer deploys from a plain bash script that buyers review in full before it runs, and every deploy verifies itself before reporting success. Container images are public.
› Where to start
A first agent in front of real users does not need all eight layers on day one. Four carry most of the risk, and they fit in one sentence: the agent acts for a verified person, within a budget, under an off switch, and on the record. Add data access the day it touches a system of record, and release discipline the day a second person edits the prompt.
Then pick one workflow where a mistake is visible and reversible, run it behind those layers for a month, and widen the agent's authority only as fast as the evidence supports.
› One workflow in production, in three weeks
The Production Sprint puts one real workflow into production on this stack, in your environment, in three weeks. To evaluate on your own, start free: the IDE, the CLI, and your first governed agent on the hosted control plane.