Agentic AI: What is it and how do you deal with the safety and compliance risks?

by Caroline Popper

As the AI tsunami hits the shore, it becomes especially important to segment “AI” into various categories. “AI” is not a monolith, especially not from the perspective of the need to implement appropriate governance and compliance.

Here, I consider Agentic AI, or AI Agents. I will describe them in a healthcare context, though the concepts are highly applicable in other possible uses. I have chosen healthcare as an example where everyone can personally identify with the concerns and the possible impact of mistakes.

Agentic AI: What is this?

An Agentic AI system makes decisions to accomplish a specific goal. It is task-driven and (using machine learning) mimics human behavior to achieve this goal, e.g., learning from watching lots and lots of surgeries, an autonomous Agentic AI robot can perform the surgery alone.

The element we are focusing on here is the agent’s decision autonomy, not it’s intelligence. A robot’s autonomy level is defined by the highest-impact decision it can make without asking a human. In healthcare, for example, the moment an AI Agent actually decides (rather than just offering advice) what happens next, it needs clinical governance — not just accuracy metrics. In other words, the AI robot needs to have appropriate credentials (and oversight) when it suggests to the human provider what to do (how to treat you) or takes the action by itself, irrespective of how good or appropriate that action is.

Autonomy and agency should be viewed on a continuum: how much autonomy and agency does the AI Agent have?

Let us consider 6 levels of autonomy ranging from 0 to 5 (the highest level being full autonomy where the agent defines the goals, adapts its behavior and operates without human involvement)).  At the total autonomy level, the Agentic AI is functioning entirely independently (like a robot doctor or robot surgeon).  At that point the compliance program’s elements need to be developed with this complete autonomy in mind. At level 5, there is no human-in-the-loop. At lower levels of autonomy, human doctor intervention may be the compliance line of defense. 

Agency determines who initiates, who decides, and who is accountable. To develop appropriate guardrails, dividing agency into 5 levels is helpful: Level 0 can be thought of as a passive tool in the hands of a human healthcare provider; Level 4 agents are capable of adapting and self-updating without human doctor involvement. 

Teaming

The degree of autonomy will vary, and reciprocally so will the extent of the role of the human healthcare provider in a particular episode of care. Therefore, we can think of the healthcare workforce in the future as a human and (AI Agent) machine team. The nature of this team, the authority of the human, the autonomy of the AI Agent, the extent of delegation by the human to the AI agent, will all directly impact the governance of the healthcare team. The compliance strategy and plan will need to take these variables into account.

All team members will need to be trained. More importantly the compliance team will need to establish new lines of defense and control points. Particularly critical, and vulnerable, will be the hand-over points between human and Agentic AI team members delivering care.

Risks and liability

Careful consideration must be given to the potential for AI Agents to malfunction. Mistakes may result from inappropriate training (e.g., not trained for particular procedure), inappropriate deployment (e.g., deployed in an operating room when trained for a clinic setting) or unanticipated meddling (hacking). Liability may be allocated to the human “master” who delegated authority to the agent, if the human was “in the loop”. In the future, in cases of full autonomy, liability may be allocated to the developer or supplier of the AI Agent or perhaps to the entity that credentialled (i.e., allowed, analogous to allowing a particular doctor to work in a particular hospital) the Agent’s use. And obviously appropriate cyber safeguards and insurance will need to be in place.

Noteworthy too, is that higher risks occur when an AI agent’s autonomy level is mismatched to the application, e.g., an agent is approved as Level 2 (requiring human execution) but deployed like Level 4 (no human in the loop) and governed at a lower risk level like Level 1. Consider for illustrative purposes, a surgical robot that is designed to be directed by a human surgeon (with full visibility and “hands on”) but is allowed to function in a particular operating room without a surgeon present there.

The challenges related to AI agents in healthcare (with application to other safety-critical domains)

In the healthcare context, one must determine for whom the agentic AI is an “agent”: the patient, physician, healthcare system or the developer. The answer obviously depends on what a particular AI Agent is doing; in the case of a fully autonomous AI surgical robot, the agentic AI system might not be a true agent for the human surgeon at all, and but is actually an independent team member. Agentic Ai in healthcare highlights the importance of the following issues when considering an appropriate compliance program:

• Understanding the specific application of the Agentic AI and the degree of autonomy of the agent (to allow ongoing monitoring of its impact), e.g., using a surgical robot for a specific procedure, e.g., removing a gallbladder
• Understanding the training, validation and accreditation of the agentic AI (to ensure fitness for purpose), e.g., how was it trained, tested and re-tested to make sure it did not lose its abilities?
• Understanding the risks associated with the activity and the specific risks associated with the AI Agent performing that activity, e.g., how risky is the procedure and how much more or less risk is associated with a robot performing it?
• Developing processes to mitigate those risks
• Training all team members, including compliance personnel, on human-machine teaming
• In healthcare, obtaining meaningful consent from the patient for the use of Agentic AI, i.e., making sure the patient really understand how the surgery is being performed
• Integrating consideration of ethics into the deployment of AI agents and monitoring ethical use, ensuring that such use stays within the guardrails that all parties have agreed to.
• Establishing a robust data collection, monitoring and auditing process in order to understand how the agentic AI performed in the real word setting. This will  enable retraining and modification as needed.

Making this real: Some examples in healthcare:

Healthcare AI fails when governance lags behind agency.

Here are 2 examples that can be found today

1. Automated clinical decision support may lead to dangerous over-trust

What can go wrong
The AI system may issue recommendations, prioritize alerts, and nudge clinician behavior

E.g., Sepsis detection systems that auto-trigger treatments (e.g. administration of fluids, antibiotics) with high false-positive rates (i.e., unnecessary or clinically inappropriate).

Here clinicians may follow alerts reflexively, leading to over-treatment, antibiotic misuse, or missed alternative diagnoses. This should be considered an AI Agent problem because the system here is not just predicting—it is driving action without escalation to a human decisionmaker to ensure the appropriate standard of care decision-making.

2. Early diagnostic chatbots and symptom checkers lead to over or under assessment of severity

What can go wrong:
Patient-facing AI Agents give confident advice without robust safety constraints.

E.g., A symptom checker Agent could downplay serious symptoms (dismissing them as “likely anxiety”) or over-escalate minor issues which may lead to ER over use and overcrowding. Here the AI Agents are substituting too much for triage nurses but do not have the necessary sophistication to achieve appropriate accuracy, and there is not an appropriate balance between the AI Agent and the human clinician.

Across cases, failures are not usually about “bad models” but about bad assessments of how much autonomy an agent should have and what the guardrails should be, e.g., treating predictions as decisions or having feedback loops that are too slow to detect harm.

These become the issues that governance and compliance need to address.

Universal guardrails must apply to all levels of agency

• Clear accountability must be in place (address the threshold question: “Who is accountable if this causes harm?”)
• Clear escalation triggers must be designed and implemented
• Humans must be in the optimal place in the loop and trained for the role

While the examples discussed here have focused on healthcare because of its universal nature, the same type of analysis applies for all uses of agentic AI.