Why Prohibited-Practice Findings Are Rising Faster Than High-Risk Findings
The compliance conversation around AI has spent the last few years orbiting the gravitational pull of “high-risk.” For many companies, that label felt like the only one worth worrying about: if you weren’t building medical diagnostics, critical infrastructure controls, or employment decision engines, you could assume you were somewhere on the safe side of the line. Yet the pattern emerging in audits, internal reviews, and regulator-facing readiness work is counterintuitive: prohibited-practice findings are rising faster than high-risk findings, and they’re catching organizations that genuinely believed they were operating at minimal risk. The reason isn’t that more companies are suddenly building obviously outlawed systems. It’s that Article 5-style prohibitions are easier to stumble into, harder to recognize in real time, and more tightly connected to everyday product decisions than many teams expected.
A major driver is the widespread misconception that “prohibited” means “rare.” In practice, prohibited practices are often defined not by the sector you’re in, but by the effect your system produces and the context in which it’s deployed. That makes them more like trapdoors than boundaries. A consumer app team can ship a seemingly benign personalization feature and inadvertently create a pattern of manipulation. A customer service chatbot can be tuned to “keep users engaged” and slip into behavior that edges toward exploiting vulnerability. A workplace tool that claims to “support productivity” can become, through incentives and rollout choices, a mechanism for undue pressure. Teams that rely on coarse risk labels miss the fact that prohibited-practice tests tend to be qualitative and contextual, and that context shifts as products evolve.
Another reason is simply that organizations have overfit their governance to “high-risk” because it feels more operational. High-risk regimes tend to come with checklists that map cleanly to existing compliance muscle: documentation, quality management, validation, monitoring, incident reporting. Even when demanding, these activities are familiar to regulated industries and increasingly standardized across enterprise AI programs. Prohibited practices, by contrast, are less about process maturity and more about design intent, behavioral outcomes, and power asymmetries—concepts that don’t always sit neatly in a risk register. The paradox is that a company can look impressive on AI governance artifacts and still fail the prohibited-practice test because the issue lies in what the system persuades people to do, not how neatly the model was validated.
The surge in findings also reflects how modern AI products are assembled. Many teams no longer “build a model” so much as orchestrate multiple components: third-party foundation models, in-house classifiers, data enrichment services, analytics pipelines, A/B testing frameworks, and growth experimentation. In these modular systems, harmful effects can emerge from the interaction of parts rather than any single component. A recommender system optimized for engagement may be paired with a prompt strategy that nudges behavior. Add a segmentation layer that identifies susceptibility signals, and you now have something that can look uncomfortably like exploitation—even if no one set out to do that. Prohibited practices thrive in gaps between ownership boundaries, where each team can plausibly say their part is harmless.
The same modularity creates a documentation illusion. Procurement and vendor management processes often focus on security, privacy, and model performance. They rarely probe the subtler questions that prohibited-practice assessments demand, such as: what user states does the system infer; what levers does it apply; how does it behave toward users in stress, confusion, or dependency; what happens when a user tries to disengage; and which incentives were set by product KPIs. When you buy components, you often inherit behaviors that weren’t apparent in a demo. If internal testing focuses on accuracy and latency, it may never surface the “dark corners” where a conversational system becomes coercive, or where a personalization engine begins steering choices in ways users don’t perceive.
Findings are rising faster, too, because prohibited-practice issues are often discovered late, when products are already in market and feedback loops are active. High-risk categorization is frequently done early—during scoping, procurement, or initial architecture—because organizations know to ask, “Is this a high-risk use case?” Prohibited practices are more likely to emerge during iteration: a tweak to reduce churn; a new onboarding flow; an update to monetize a free tier; a change in how the model responds to objections. The behavior becomes problematic not in the first release, but in the accumulation of optimization decisions. Each change looks minor; the overall system crosses a line.
A particularly common failure mode is treating prohibited practices as if they were only about “obvious bad actors.” Many teams assume that as long as they don’t target children, don’t do overt deception, and don’t call their feature “emotion recognition,” they are safe. But the boundary is rarely about labels; it’s about whether the system meaningfully impairs autonomy or takes advantage of vulnerability in ways users can’t reasonably resist. That can arise in ordinary settings: finance apps that push impulsive borrowing, games that pressure spending through hyper-personalized triggers, wellness tools that induce dependency through escalating claims, or enterprise software that nudges managers toward punitive actions without transparency. Even if your company’s intentions are positive, your system’s incentives can be misaligned with user welfare.
Another accelerant is the growing sophistication of internal audit and assurance teams. As AI governance matures, reviewers are learning that high-risk controls don’t automatically address prohibited-practice concerns. A well-documented model can still be deployed in a manipulative funnel; a fair classifier can still be part of a system that pressures users to consent; a robust monitoring plan can still miss harm if the metrics track engagement rather than user outcomes. Auditors are therefore widening the lens beyond model cards and bias tests to include product analytics, experimentation logs, UX flows, notification strategies, and incentive structures. That expanded scope surfaces more prohibited-practice flags, especially in consumer-facing products where persuasion is a feature, not a bug.
Legal and compliance functions are also encountering a translation problem: teams are comfortable mapping rules to artifacts, but prohibited practices require mapping rules to human experience. It’s not always clear who in the organization “owns” autonomy, dignity, or undue influence. Product owns growth; marketing owns messaging; design owns flows; data science owns optimization; compliance owns policies. When responsibility is diffuse, the default outcome is under-identification. Then, when a cross-functional review finally happens—often prompted by a launch in a new market or a major partnership—reviewers see the system as a whole for the first time, and prohibited-practice findings spike.
This helps explain why organizations that considered themselves minimal risk are being caught: many “low-risk” companies are in the business of attention, persuasion, or behavior change. Even B2B products can create the conditions for prohibited outcomes when they mediate workplace power dynamics. The issue isn’t that these businesses are inherently non-compliant; it’s that their day-to-day optimization culture is unusually capable of producing the kinds of effects prohibited-practice rules target. When success metrics reward conversion at all costs, and AI makes persuasion more precise, the distance between “smart personalization” and “undue influence” can shrink quickly.
The path forward is less about adding paperwork and more about changing the questions asked during design and iteration. Teams that are avoiding prohibited-practice findings tend to institutionalize early checks that focus on intent, leverage, and vulnerability. They also test systems in conditions that mirror real-world stress rather than idealized user journeys. Practical measures include:
- Reviewing user journeys specifically for points where the system applies pressure, creates urgency, or reduces perceived choice
- Stress-testing conversational behavior for escalation, dependency cues, and refusal-handling patterns
- Auditing what signals are inferred about a user’s state and whether those inferences are used to steer decisions
- Aligning KPIs so that “success” isn’t defined solely by engagement, conversion, or retention when those can conflict with autonomy
None of these steps require a company to label itself as “high-risk.” They require a company to accept that prohibited practices can be an emergent property of ordinary product work, especially when AI is used to optimize influence.
Ultimately, prohibited-practice findings are rising faster because they sit at the intersection of modern AI capabilities and modern product incentives. High-risk compliance is often a matter of classifying a use case and meeting defined controls. Prohibited-practice compliance is a matter of ensuring that, as your system learns what works, it doesn’t learn to cross lines you didn’t realize were there. Companies that assumed they were minimal risk are discovering that the riskiest thing isn’t always the model; it’s what the model is allowed to optimize, and what the business quietly rewards it for doing.