Every enterprise technology wave produces a familiar pattern: excitement outruns infrastructure. Cloud computing had it. Big data had it. Now agentic AI is having its turn — and the gap this time is unusually wide.
A recent CIO.com feature by senior writer Grant Gross captured the problem precisely: enterprises are busy deploying agents, but few understand the building blocks and blueprints necessary to engineer effective agent-enhanced business workflows. IDC predicts a tenfold increase in agent use by large enterprises by the end of this year. Yet many CIOs haven’t focused on the architecture needed to run a large fleet of agents responsibly.
This is not a hypothetical problem. It is happening across enterprise IT right now — and understanding why agentic architecture is so puzzling, and what it actually requires, has become one of the most consequential strategic questions a CIO will answer this year.
| The Core Problem“There’s a tendency to focus on what an agent can do and then move quickly to deployment before the underlying infrastructure is in place,” says Hilary Packer, EVP and Head of Enterprise Data and AI at American Express. “It is essential to first build the enterprise capabilities that allow agents to operate consistently across systems, data sources, and workflows.” |
1. The Deployment-First Trap
The most common failure pattern in enterprise agentic AI adoption is sequencing. Organisations see what a single, well-demoed agent can do — resolve a support ticket, draft a report, summarise a meeting — and move directly to scaling that capability across the business. The architecture that would let dozens or hundreds of those agents operate consistently, securely, and observably gets built reactively, project by project, agent by agent.
This is the deployment-first trap, and it produces a specific, predictable form of technical debt. Identity management gets rebuilt for every pilot. Access controls get reimplemented inconsistently across teams. Monitoring and observability tooling proliferates without a shared standard. What works for an isolated pilot becomes unmanageable the moment an organisation tries to scale across the enterprise.
Packer frames this as fundamentally a governance and operating model challenge — not merely a technology challenge. The tools to build individual agents are mature. What is immature, in most organisations, is the connective tissue: the shared infrastructure that lets agent number 47 inherit the same identity, access, and observability standards as agent number 3, instead of reinventing them.
| “As organizations experiment, teams often end up rebuilding the same capabilities repeatedly — for example, identity management, access controls, monitoring, and observability. That may be sufficient for isolated pilots, but it becomes difficult to manage and scale across the enterprise.”— Hilary Packer, EVP & Head of Enterprise Data and AI, American Express |
2. The Data Foundation Problem: You Can’t Automate What You Can’t Trust
Before any agent can act reliably, it needs confidence in the data it is acting on. This sounds obvious, but it is where a striking number of enterprise agentic AI initiatives quietly stall. Most large organisations still operate with fragmented systems, inconsistent data definitions across business units, and siloed ownership structures that predate any AI initiative by decades.
An agent that queries inconsistent data, or data owned by three different teams with three different definitions of the same metric, will produce confidently wrong answers — arguably worse than producing no answer at all, because the error is invisible until it causes a downstream business decision to go wrong.
The investment required here is not glamorous, and it does not show up in agent demos: governance, lineage, and modern data infrastructure that allows information to move efficiently and reliably across the enterprise. It is, however, the prerequisite that makes everything built on top of it trustworthy.
- Data governance: Clear ownership, consistent definitions, and access policies that travel with the data, not just the application
- Data lineage: Visibility into where data originated, how it has been transformed, and what depends on it downstream
- Unified data infrastructure: The technical capability for information to move across systems without manual reconciliation
3. What Agentic Architecture Actually Requires
One of the most persistent misconceptions among IT leaders is treating architecture as synonymous with the AI model powering the agent. Choose a strong model, the thinking goes, and the architecture problem is mostly solved. This significantly understates what agentic architecture requires.
Saurabh Pitkar, Director of Product Management for Agentic Commerce at Dell Technologies, identifies five distinct elements that make up a complete agentic architecture — and notes that many organisations have meaningfully underinvested in several of them:
The Orchestrator
Described by Pitkar as the brain that controls subagents, the orchestrator is the coordination layer that decomposes complex goals into tasks, routes those tasks to the right subagent, and assembles results into a coherent outcome. Without a well-designed orchestrator, an organisation ends up with a collection of disconnected point-solution agents rather than a coordinated system.
Subagents
Specialised agents, each built for a specific function — a billing subagent, a scheduling subagent, a research subagent. The discipline here is scope: subagents that try to do too much become harder to govern, test, and trust than a focused set of narrow, composable agents coordinated by the orchestrator.
APIs and Tools
The mechanisms through which agents actually act on the world — querying a database, updating a CRM record, triggering a workflow. This is where Model Context Protocol (MCP) standardisation becomes critical: without a consistent way for agents to discover and invoke tools, every new integration becomes a custom engineering project.
Memory
The capability to maintain context across a single agent session and, more demandingly, across longer-term interactions and behaviour patterns. Pitkar specifically flags memory as an area where many organisations have underinvested — and it shows. Agents without persistent memory cannot build on prior interactions, forcing users to re-explain context repeatedly, which is one of the fastest ways to erode trust and adoption.
Guardrails
The boundaries that define what an agent is permitted to do — and, just as importantly, what it is not. Guardrails are what make autonomous action safe enough to deploy without requiring human approval at every step. Without them, organisations are forced to choose between unsafe autonomy and approval-gated automation that defeats the purpose of agentic AI.
| “Many organizations are building agentic architecture, but those still struggling have failed to make the right investments in areas such as building memory and creating MCP tool standardizations. A lack of data access and integration can be a huge barrier to agent deployments despite heavy investments in data modernization.”— Saurabh Pitkar, Director of Product Management, Agentic Commerce, Dell Technologies |
4. Why “AI Doesn’t Care About Org Structure”
Perhaps the sharpest insight in the CIO.com analysis is Pitkar’s observation about organisational design: AI does not care about org structure when it comes to accessing data.
This single sentence explains a great deal of why agentic AI adoption stalls inside large enterprises. Most enterprise data architecture has been shaped, implicitly or explicitly, by organisational boundaries — finance owns its systems, marketing owns its systems, operations owns its systems, and the seams between them are managed through manual handoffs, scheduled reports, and human judgement about who needs to know what.
An agent tasked with answering a cross-functional business question doesn’t recognise those seams. It needs to traverse finance data, operational data, and customer data in a single reasoning chain — and if the organisation’s data architecture mirrors its org chart rather than its actual information needs, the agent hits walls that have nothing to do with its intelligence and everything to do with structural fragmentation that predates AI by years.
This is also why organisational change management is not a peripheral concern for agentic AI rollouts — it is central. Teams that aren’t ready to engage with agent-speed outputs slow adoption considerably, regardless of how capable the underlying agents are.
5. The Four-Layer Model of Agentic Architecture
To make the abstract elements of agentic architecture more concrete, a useful framework — referenced in the CIO.com analysis — organises agentic architecture into four distinct layers. Each layer addresses a different question, and weakness in any one layer limits what the system as a whole can reliably do.
| Layer | Name | Question It Answers |
| 1 | System of Context | What does the organisation know, and is it unified and semantically enriched enough for an agent to reason over reliably? |
| 2 | System of Work | What composable application services and workflows can agents actually invoke to get things done? |
| 3 | System of Coordination | How do orchestrators, subagents, and tools communicate, hand off tasks, and resolve conflicts? |
| 4 | System of Trust | What governance, guardrails, identity, and audit mechanisms make autonomous action safe to deploy? |
Most organisations that describe agentic architecture as “puzzling” are, on inspection, strong in one or two of these layers and substantially underdeveloped in the others. A company with excellent data infrastructure (System of Context) but no standardised orchestration layer (System of Coordination) will struggle to scale beyond isolated point solutions. A company with strong governance but weak data foundations will deploy agents that are safe but unreliable.
6. Use Case Determines the Architecture Priorities
A critical nuance that gets lost in generic agentic AI advice: the elements of agentic architecture are fairly standard, but the relative investment priority shifts significantly depending on the use case. There is no single correct weighting across orchestration, memory, guardrails, and data access — it depends on what the agents are actually being asked to do.
Customer Support Agents
Pitkar notes that customer support use cases demand higher investment in dynamic intent mapping, memory, and effective user experience design — because these are emotionally charged conversations with human users who need a problem solved to continue with their day. An agent that forgets context mid-conversation, or misreads intent in a frustrated customer’s message, causes damage disproportionate to the technical severity of the error.
Agentic Commerce
By contrast, agentic commerce use cases — agents that transact on a customer’s or business’s behalf — need to focus on deterministic APIs with machine-readable data, strict guardrails, and compliance mechanisms to securely process transactions involving real money. Here, the priority inverts: less emphasis on conversational nuance, far more emphasis on transactional certainty and auditability.
Internal Operations Agents
For agents handling internal operations — IT service management, finance reconciliation, HR processes — the priority often shifts toward integration breadth and data access across legacy systems, since these agents typically need to traverse more internal systems than either customer-facing or commerce use cases.
The practical implication for CIOs: a single, generic agentic architecture investment plan applied uniformly across every use case will misallocate resources. Architecture decisions should be use-case-aware from the outset, not retrofitted after a pilot reveals the gap.
7. A Practical Checklist for CIOs Building Agentic Architecture
Drawing on the building blocks outlined above, here is a practical sequence for CIOs evaluating where their organisation actually stands:
- Audit existing pilots for duplicated infrastructure: How many times has identity management, access control, or monitoring been rebuilt across different agent projects?
- Assess data foundation maturity: Are data definitions consistent across business units? Is lineage visible? Can data move across systems without manual reconciliation?
- Map current orchestration capability: Is there a shared orchestration layer, or does every team build its own coordination logic?
- Evaluate memory architecture: Can agents maintain context within a session? Across sessions? Across the same user’s history with the organisation?
- Review MCP and tool standardisation: Is there a consistent protocol for agents to discover and invoke tools, or is every integration bespoke?
- Stress-test guardrails against real failure scenarios: Are governance boundaries defined by action class and risk level, or applied uniformly regardless of stakes?
- Identify org structure barriers to data access: Where do organisational boundaries currently block agents from traversing data that the business question requires?
- Build use-case-specific investment plans: Weight architecture investment by the actual demands of each use case — support, commerce, internal operations — rather than a single template
Conclusion: Architecture Is the Unglamorous Work That Makes Agentic AI Real
The puzzle at the centre of agentic architecture is not technical sophistication — the orchestrators, subagents, APIs, memory systems, and guardrails that make up a complete architecture are all well-understood, buildable components. The puzzle is sequencing and discipline: resisting the pull toward rapid, isolated deployment in favour of the slower, less visible work of building shared infrastructure that scales.
IDC’s forecast of a tenfold increase in enterprise agent use by the end of this year makes this an urgent rather than theoretical concern. Organisations that have rebuilt the same identity, access, and monitoring capabilities for every pilot will find that the cost of that technical debt compounds precisely as the number of agents they are running multiplies.
The CIOs who get ahead of this will be the ones who treat agentic architecture as enterprise infrastructure from day one — with the same rigour applied to network architecture or core banking systems — rather than as a byproduct of whichever agent pilot happened to ship first. As Packer puts it, the foundational work is not optional groundwork before the real value begins. It is what determines whether the value materialises at all.
