Zuletzt aktualisiert am:
AI monitoring is the continuous observation of AI systems in production to detect degradation, drift, unsafe behavior and control failures. It covers model performance, input and output distributions, fairness metrics, cost and latency, and for agentic systems the actions the system takes.
What should be monitored?
Four layers, and most programs cover only the first. Performance tracks accuracy, error rates and the business outcome the system exists to produce. Data tracks the inputs, watching for distribution shift, schema changes and upstream pipeline breaks, which cause more incidents than model decay does. Behavior tracks what the system produces or does, covering output quality, refusal and hallucination rates for generative systems, and the actual actions taken for agents. Control tracks whether the governance controls still function: that human review is happening, that escalations fire, that logging is complete.
What is drift and which kind matters?
Drift is the divergence between the conditions a model was built for and the conditions it now operates in. Data drift is a change in the input distribution, which is straightforward to detect because it needs no outcome labels. Concept drift is a change in the underlying relationship between inputs and the correct answer, which is harder because it can only be confirmed once outcomes are known, and outcomes often arrive months later in credit, insurance and clinical settings. The practical approach is to alert on data drift as a leading indicator and validate against outcomes on a slower cycle.
How does monitoring differ for generative and agentic systems?
Accuracy metrics do not transfer. A generative system has no single correct output, so monitoring moves to sampled evaluation against rubrics, groundedness checks against source material, refusal and safety rates, and user feedback signals. For agents the important series is behavioral: which tools were called, how often escalation thresholds were crossed, how many actions were reversed or corrected afterwards, and whether the agent attempted operations outside its permitted set. A rising rate of blocked actions is a governance signal well before it becomes an incident.
Why do regulators care?
Post-deployment monitoring is an explicit obligation, not good practice. The EU AI Act requires providers of high-risk systems to operate a post-market monitoring system and to report serious incidents. The NIST AI Risk Management Framework places continuous measurement in its Measure and Manage functions. Model risk supervision has expected ongoing performance monitoring for years. In each case what is asked for at examination is evidence: what was monitored, what thresholds applied, what fired, and what was done about it.
What makes a monitoring program work?
Thresholds defined before deployment and recorded against the system, so that an alert has an agreed meaning. Ownership assigned to a named person, not to a rota. Alerts routed to somebody who can act, since the common failure is a dashboard nobody opens. Coverage proportionate to risk tier, because monitoring every system equally exhausts the team and leaves the important ones under-watched. And retention of the monitoring record itself, since the evidence a supervisor wants is the history, not the current state of the dashboard.
Praxisbeispiel:
A lender monitors a credit decisioning model on three cycles. Daily, it checks input distributions against the training baseline and alerts when any feature shifts beyond an agreed threshold, which catches an upstream bureau file format change within a day. Monthly, it reviews approval rates by protected characteristic against the fairness thresholds set at approval. Quarterly, it validates predictions against realised defaults as outcomes mature, which is the only way concept drift becomes visible. Each threshold, alert and response is recorded against the system in the inventory, so when the regulator asks how the lender knew the model was still fit for use, the answer is a dated record and not an assertion.
