Chander Damodaran is MD and Global CTO at Brillio, a global leader in Enterprise Digital Transformations.
getty
The next wave of enterprise AI will not arrive as a chatbot waiting for a prompt. It will arrive as software that can plan work, invoke tools, trigger workflows, negotiate with systems and act on behalf of people. That is the promise of agentic AI.
Boards and executives should be asking: When AI agents begin to act, can we prove that humans are still meaningfully in command?
Most large companies have already done the visible part of AI governance. They have written policies, assembled risk committees, mapped principles to frameworks and aligned to emerging regulation such as the EU AI Act and the NIST AI Risk Management Framework.
But policies do not stop a fast-moving agent in the middle of a workflow. Frameworks do not prove that a human had the authority, context and tools to intervene. And a governance charter does not answer the question that matters when something goes wrong: Who knew, who approved, who could have stopped it and what evidence proves it?
In the age of agentic AI, trust will not be built by intent. It will be built by proof.
The Oversight Paradox Is Becoming A Board-Level Risk
The comforting assumption behind most AI governance programs is that a human remains in control. But control is not a label. It is a capability that must be exercised, refreshed and tested. The more work we hand to agents, the fewer opportunities humans have to practice the judgment they are expected to use when the system fails.
The World Economic Forum calls this the oversight paradox: As systems improve, we delegate more to them, and the overseer gets fewer chances to keep their own skills sharp. This is not theoretical. It happens when agents gradually take over pattern recognition, escalation judgment or exception handling that once kept human experts close to work.
For executives, the implication is blunt: One of the most important governance events may not be a model release or data connection. It may be the moment a human's practical ability to challenge the system falls below the level the risk policy assumes. Few enterprises measure that today.
Accountability Without Authority Is Just Blame
The second uncomfortable truth is that assigning accountability is not the same as creating control.
Researchers use the phrase moral crumple zone to describe what happens when responsibility lands on the nearest human even though that person lacks the visibility, time or tools to intervene meaningfully. In an agentic enterprise, this risk becomes real. A manager may be accountable for an agent's actions, but if the agent moves faster than review cycles, if its reasoning is opaque or if intervention is buried behind poor tooling, accountability becomes theater.
For accountability to be real, the accountable person must be able to see what the agent is about to do, intervene at the speed of the agent and act without being punished by the operating rhythm of the business.
Miss any one of these, and the enterprise has not created accountability. It has created a scapegoat with a job title.
The Missing Link Is Identity
The least glamorous part of this debate may be the most important: identity. Many enterprises still cannot reliably trace an agent's action back to the human sponsor on whose behalf it acted. That is a dangerous gap. Nonhuman identities already outnumber human identities in many technology estates, while legacy identity systems were built for applications and service accounts, not autonomous agents.
A policy can say every agent action must be traceable to a human. Only architecture can make that true. Each agent needs its own identity, tightly scoped credentials, a named human sponsor and an auditable on-behalf-of chain for every downstream action.
Identity is not a back-office security detail. In the agentic enterprise, it is the control plane for accountability.
Govern The Boundary, Not Just The Model
Most recertification triggers focus on the machine: Did the model change, did the data source change, was a new tool added? Those questions are necessary, but incomplete. The safety profile of an AI agent is determined not by the model alone, but by the boundary between the agent and the people around it.
If an agent quietly absorbs three new decision types this quarter, last quarter's certification may already be stale, even if the model has not changed. When the human-agent boundary shifts, the system has changed.
That is why runtime observability matters. Not because executives need another dashboard, but because the enterprise needs to know when the assumptions behind a certification have expired.
Human Capability Is Part Of The Infrastructure
There is one more risk leaders tend to underestimate. When AI use becomes routine, the people expected to oversee it can gradually lose the underlying skills required to challenge it. Researchers call this distributed de-skilling. Executives should call it what it is: operational fragility.
Evidence only matters if people know how to interpret it. A company that instruments the agent but not the operator is only half-prepared. Role rotation, deliberate friction, refresher exercises, escalation drills and periodic re-anchoring are no longer soft training practices. They are part of the control environment.
The Next Phase Of AI Governance Is Proof
The first phase of AI governance was policy, second was process, third will be proof. Enterprises that want to earn trust in agentic AI must show, continuously and operationally, that control is real. That means:
• Certify agents before deployment against behavioral, security and confidence criteria.
• Instrument runtime actions so they are traceable to a delegated identity and named human sponsor.
• Recertify when the division of labor changes, not only when the model changes.
• Preserve the oversight competence of humans expected to intervene.
• Design accountable roles with genuine authority, not convenient blame.
Agentic AI will test more than enterprise technology; it will test enterprise control. The winners will not simply be the companies with the most capable agents but those that can prove those agents are accountable, observable, interruptible and anchored to human authority.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
>> Home