Context and challenge

A mid-sized regional water utility had invested in an operational monitoring system that used machine learning to spot anomalies consistent with pipe leaks. The system ingested pressure, flow, and telemetry signals from field sensors and produced ranked alerts for operations staff to investigate. Early pilots demonstrated value: faster identification of hidden leaks, fewer customer complaints, and better prioritization of field crews.

Then a new requirement reshaped the program’s direction. Under NIS2 evidence expectations, the utility needed to demonstrate—not merely claim—that it could manage cyber risk for essential services. The leak-detection system was operational technology adjacent: it touched supervisory data, influenced maintenance decisions, and could indirectly affect service continuity. That made it a natural focus for auditors and internal risk owners.

The challenge wasn’t “be secure.” It was prove it with evidence in a way that was:

• Repeatable (not a one-off scramble at audit time)
• Traceable (clear lineage from control intent to implemented measure to proof)
• Operationally realistic (usable by engineers and operators, not just compliance specialists)
• Resilient to change (new models, sensor rollouts, and software updates shouldn’t break the documentation)

The system had grown organically: a mix of scripts, dashboards, notebooks, and a few automated pipelines. Some controls existed informally (“we usually review changes,” “alerts are mostly right,” “access is limited”), but the evidence was scattered across tools and personal knowledge. NIS2 pushed the utility to transform that implicit trust into explicit, auditable assurance.

Approach and solution: building an evidence pack for operational monitoring

The utility adopted an “evidence pack” model: a structured set of artifacts mapped to security and resilience requirements, designed to answer common audit questions quickly. The guiding principle was simple: every key control should have an owner, an operational procedure, and a proof trail.

1) Define the system boundary and criticality

Before collecting evidence, the team clarified what exactly was in scope. They created a system boundary document that covered:

• Data sources: field sensors, telemetry brokers, historian exports
• Processing: feature generation, model inference, alert scoring
• Outputs: alert queues, dashboards, work-order triggers
• Dependencies: identity service, cloud/on-prem compute, logging, networking segments
• Operational interfaces: how operators acknowledge alerts and dispatch crews

This step prevented “evidence sprawl.” It also helped determine which parts were safety- and continuity-relevant and therefore needed stronger controls and logging.

2) Translate NIS2 expectations into control questions

Rather than treating NIS2 as a checklist, the team turned it into a set of practical questions aligned to incident prevention, detection, response, and recovery. Examples included:

• Who can access model outputs and configuration, and how is access reviewed?
• How are software and model changes authorized, tested, and rolled out?
• How is anomalous system behavior detected (not just leaks, but monitoring failures)?
• What evidence shows the team can respond and recover within an acceptable window?
• How is third-party and supply-chain risk assessed for core components?

Each question became a “control statement” with associated evidence requirements.

3) Build the evidence pack structure

The evidence pack was organized into a small number of folders (physical or digital), each with a consistent template:

• Control intent: why it exists and what risk it mitigates
• Implementation: what is actually configured or done
• Ownership and cadence: who does it and how often
• Evidence artifacts: screenshots, logs, exports, tickets, runbooks, approvals
• Exception handling: what happens when the process can’t be followed

The structure made it easy for operational teams to contribute without writing compliance narratives.

4) Engineer evidence-producing workflows (not screenshots-by-hand)

A major shift was to treat evidence as a system output. The team reduced manual collection by embedding evidence generation into day-to-day operations:

• Change management: model updates required a change record, peer review, test results, and an approval step. The pipeline automatically attached test summaries and model metadata to the change record.
• Model registry and provenance: each deployed model version was recorded with training data windows, feature set identifiers, performance metrics, and sign-off notes. This created a lineage trail from data to model to deployment.
• Access controls: role-based access policies were consolidated, and access reviews were scheduled. Evidence included exportable access lists and review acknowledgements.
• Logging and monitoring: system logs were centralized and time-synchronized. Alerts were created for pipeline failures, missing telemetry, unusual spikes in alert volume, and authentication anomalies—so the monitoring system itself was monitored.
• Backups and recovery tests: configuration and model artifacts were backed up with retention rules. Periodic restore tests produced a simple record: what was restored, who performed it, and whether integrity checks passed.

By turning evidence into a byproduct of normal work, the utility avoided the common trap of “documentation theater.”

5) Create operational runbooks that double as evidence

For incident management and continuity, the team wrote runbooks that were deliberately concrete:

• What counts as an incident affecting the leak-detection function
• Triage steps for false positives, sensor outages, and pipeline degradation
• Escalation paths to operations, security, and field leadership
• Containment options (e.g., disable auto-ticketing, revert to last known model)
• Recovery steps and post-incident review procedures

Each runbook included a section titled “Evidence to capture” listing the exact artifacts to retain: incident timeline notes, alert IDs, log queries, change rollback references, and communications records.

6) Validate with tabletop exercises and “evidence dry runs”

To ensure the pack would hold up under scrutiny, the team ran two rehearsals:

• A tabletop exercise simulating a telemetry manipulation attempt and a separate scenario of model drift causing widespread false alerts.
• An “evidence dry run” where someone unfamiliar with the system attempted to answer audit-style questions using only the pack.

These rehearsals exposed gaps—especially around dependency mapping, time synchronization proofs, and the difference between operational incidents and security incidents. Fixes were incorporated into the templates and runbooks.

Results

The most immediate result was reduced friction. Operational teams no longer had to interpret compliance requirements; they followed the same workflows they already needed for reliability, with evidence produced along the way.

Key outcomes included:

• Faster audit readiness: documentation was current by design, not through last-minute collection.
• Clear accountability: control owners and cadences were explicit, reducing ambiguity during incidents or staff changes.
• Improved system resilience: monitoring the monitoring system revealed silent failures—such as intermittent sensor feeds and pipeline delays—that previously went unnoticed.
• Higher trust in alerts: model provenance, change controls, and drift checks improved confidence that alerts were both explainable and stable over time.
• Better cross-team coordination: shared runbooks aligned operations, security, and data engineering around common incident language and response steps.

Where numerical metrics were discussed internally, they were treated as approximate and secondary to demonstrable process integrity. The primary win was the ability to show, with traceable artifacts, that the leak-detection capability could be governed as a critical operational function.

Key takeaways

• Start with boundaries, not paperwork. Evidence falls apart when the system scope is fuzzy. Define the data flows, dependencies, and operational touchpoints first.
• Turn requirements into questions. Practical control questions are easier for engineering and operations teams to implement and prove.
• Design evidence as an output of workflows. If evidence requires manual effort, it will drift out of date. Automate exports, attach artifacts to change records, and centralize logs.
• Provenance is the backbone for AI systems. For operational AI, the ability to trace from training data and features to deployed model versions is as important as accuracy.
• Monitor the monitoring. A leak-detection system can fail quietly—through missing telemetry, pipeline delays, or model drift. Treat these as first-class risks with alerts and runbooks.
• Runbooks should specify what to save. Incident documentation is often incomplete because teams don’t know what evidence will be needed later. Make “evidence to capture” explicit.
• Practice with dry runs. A pack that reads well can still fail under real questioning. Tabletop scenarios and evidence rehearsals reveal gaps early.

A leak-detection AI can improve operational performance, but under NIS2-style evidence expectations, success depends on more than model quality. The durable advantage comes from a defensible operating model—where controls are real, repeatable, and continuously provable.