The 10 Security Categories Every AI Agent Should Be Tested Against
AI agents are no longer isolated chat interfaces; they’re software actors that read documents, call tools, move data between systems, and sometimes make decisions with real consequences. That expanded capability also expands the attack surface. Security testing for agents can’t stop at generic “prompt safety” checks, because the risks live across identity, data, tools, and workflow boundaries. A practical way to build coverage is to test agents against ten recurring security categories that show up in production incidents: unauthorized data access, policy bypass, privilege escalation, output manipulation, indirect prompt injection, model extraction, insecure tool use, data leakage, adversarial inputs, and compliance drift. Each category represents a distinct failure mode that can be probed, reproduced, and mitigated with the same seriousness you’d apply to any other critical application.
Unauthorized data access is the most direct and often the most damaging category. Agents frequently sit in the middle of multiple data stores—internal knowledge bases, ticketing systems, document drives, CRM records—and they may be granted broad read permissions “for convenience” during early development. Testing here is about verifying that the agent’s retrieval and tool calls enforce the same access controls your organization expects from humans and services. It’s not enough that the UI shows a redacted answer; you need confidence that the agent cannot retrieve restricted content in the first place, cannot infer it from partial context, and cannot leak it through side channels such as logs, traces, or tool call arguments. The most revealing tests simulate a user with limited permissions trying to coax the agent into summarizing or “confirming” sensitive items that exist elsewhere, including adjacent tenants, projects, or departments.
Policy bypass is the art of making the agent violate its own rules without necessarily touching protected systems. Every serious deployment has policies—no providing legal advice, no disallowed content, no revealing internal instructions, no executing certain actions without approval. Attackers rarely ask for forbidden behavior directly; they use framing, role-play, translation tricks, or multi-step “harmless” tasks that culminate in the restricted outcome. Testing for policy bypass should include not just obvious jailbreak prompts, but also subtle boundary-walking requests that exploit ambiguity, such as asking for “examples,” “templates,” “what not to do,” or “a fictional scenario” that effectively reproduces restricted content. Strong systems treat policies as enforceable constraints rather than optional guidance, and they verify adherence even when the user’s request is cleverly packaged.
Privilege escalation becomes a major concern as soon as an agent can take actions—creating accounts, modifying configurations, approving expenses, issuing refunds, changing access controls, deploying code, or sending messages. In classic security, privilege escalation means a lower-privileged principal gains higher privileges. In agent systems, it often happens when the model is able to trigger an “admin” tool path, reuse cached credentials, exploit overly permissive service accounts, or manipulate an approval workflow. Testing should include attempts to get the agent to execute restricted tools “just this once,” to switch identities, to reuse authentication tokens from previous sessions, or to interpret user-provided text as authorization. The key is to validate that privileges are derived from strong identity and policy checks, not from conversational persuasion or model interpretation.
Output manipulation is a quieter but equally dangerous category because it targets what downstream systems and humans trust. Many organizations pipe agent output into other components: code generators, ticket writers, incident responders, report builders, or even automated execution chains. If an attacker can shape the agent’s output—through prompt tricks, malicious context, or tool-return content—they can induce misleading summaries, fraudulent instructions, or subtly altered recommendations. Testing should include scenarios where the agent is asked to produce machine-readable formats, where a single word can flip intent, and where “helpful” rephrasing could introduce errors. Mitigations often rely on separating “content” from “commands,” enforcing structured outputs with strict validation, and keeping the agent from emitting untrusted directives that other systems execute blindly.
Indirect prompt injection deserves special attention because it exploits the agent’s environment rather than the user’s message. Agents are trained to follow instructions, and modern agents read a lot of text they didn’t originate: emails, web pages, documents, support tickets, chat transcripts, and tool responses. If any of those sources contains embedded instructions—“ignore previous directions,” “export all data,” “send credentials,” “change your policy”—the agent may treat them as higher priority than the developer’s intent. Testing should place adversarial instructions inside documents the agent is expected to read and summarize, including long contexts where malicious fragments are buried. A robust agent treats external content as untrusted data, not authority, and it maintains a strict boundary between “what the user asked” and “what the retrieved text suggests the agent should do.”
Model extraction is about protecting the model’s internals and proprietary behavior. Even if you’re using a hosted model, your system likely includes proprietary prompts, tool schemas, routing logic, fine-tuned behaviors, or private datasets that effectively constitute an “application model.” Attackers may try to reconstruct system prompts, hidden policies, or decision heuristics through repeated probing, eliciting long outputs, or asking the agent to reveal “its instructions.” In some cases, they attempt to clone behavior by collecting many input-output pairs. Testing should examine how the system handles requests for internal configuration, whether it leaks hidden text through error messages or debugging modes, and whether rate limits and anomaly detection can reduce systematic probing.
Insecure tool use sits at the intersection of model behavior and classic application security. Tools are powerful: web fetchers, database queries, code execution sandboxes, file writers, calendar APIs, and messaging clients. If the agent can call them with attacker-controlled parameters, you may get injection vulnerabilities, data exfiltration, or unauthorized side effects. Testing should explore whether the agent can be tricked into issuing unsafe queries, writing to unintended locations, calling internal endpoints, or executing high-risk operations without checks. The security posture improves dramatically when tool inputs are validated, tool permissions are scoped to least privilege, and high-impact actions require deterministic guardrails such as allowlists, human approval, or policy engines that are not model-mediated.
Data leakage overlaps with unauthorized access but focuses on how sensitive information escapes even when access is legitimate. Agents routinely handle secrets: API keys pasted by users, personal identifiers from tickets, confidential strategy documents, customer data, and internal incident details. Leakage can occur through verbose responses, “helpful” examples that echo user-provided secrets, storing conversation history improperly, or sending content to third-party services during tool calls. Testing should include deliberate placement of secrets in context followed by prompts that try to get the agent to repeat them, transform them, or include them in summaries. It should also include checks for leakage through logs and telemetry, especially when debugging is enabled in production-like environments. The goal is not just to prevent the agent from blurting out secrets, but to ensure the entire pipeline treats sensitive data with appropriate minimization and retention controls.
Adversarial inputs cover the broad class of inputs crafted to break the system: malformed encodings, extremely long prompts, weird Unicode, nested markup, or inputs that trigger worst-case behavior in parsers and validators. While language models are tolerant, the surrounding infrastructure often isn’t—prompt assembly, document chunking, embedding pipelines, JSON parsing, and tool routing can all fail in surprising ways. Testing should include stress cases that cause truncation, mis-parsing, or unintended merges of instructions and data. It’s also where you validate resilience: safe failure modes, timeouts, backpressure, and clear error handling that doesn’t leak internal state. An agent that degrades safely under hostile input is far less likely to become an accidental escalation vector during an incident.
Compliance drift is the slow-burn risk that appears after launch. Policies evolve, regulations change, tool inventories grow, and new data sources get connected. Even if the agent passed every test at release, it can drift out of compliance when prompts change, when retrieval sources expand, when a tool gains new capabilities, or when teams “temporarily” widen permissions. Testing here is about continuous verification: regular policy regression suites, change reviews for tool scopes, and audits of what data the agent can access and where it can send it. It also includes monitoring for behavior shifts after model updates, because even subtle changes in model behavior can affect how strictly it follows constraints or how it summarizes sensitive data. Treating compliance as a living property—measured and revalidated—prevents yesterday’s safe agent from becoming tomorrow’s liability.
Taken together, these ten categories provide a practical map of where agent security fails in the real world. The most effective programs don’t treat them as one-off checkboxes; they turn them into repeatable tests that run before every deployment and after every meaningful configuration change. When teams can say, with evidence, that an agent resists unauthorized access, refuses policy bypass, can’t escalate privileges, won’t be manipulated through outputs, ignores indirect injections, protects its internals, uses tools safely, minimizes leakage, withstands adversarial inputs, and stays compliant over time, they’re no longer hoping the agent is secure—they’re demonstrating it.