A risk management system for a high-risk AI system has to be established, implemented, documented and maintained, which is governance by another name.
Why it matters when the plan changes
A model is not a fixed object. It is retrained, its inputs drift, its provider changes it, and the population it is applied to shifts. Governance is what lets an organisation say which version produced which output on which data, and whether the system still performs as it did when it was assessed. The European Union's Artificial Intelligence Act requires a risk management system to be established, implemented, documented and maintained for high-risk systems, under Article 9. Without it every explanation is about a system that no longer exists.
The tension is between control and pace. Every governance step slows a release, and a provider under commercial pressure feels each one. The discipline is that release gates are evaluated on evidence quality rather than presentation, so a model ships when it passes rather than when the roadmap says it should. Article 17 of the same Act requires providers of high-risk systems to operate a documented quality management system, the same discipline as a legal duty.
In practice
A provider improves its model and the new version scores a construct differently. Reports issued before and after the change are compared by a client as though they were on one scale. Nobody recorded which version produced which report. The client's comparison is meaningless and nobody can prove it, which is worse than knowing.
Evidence
A risk management system must be established, implemented, documented and maintained for high-risk AI systems.
Article 9, European Union Artificial Intelligence Act (2024)Providers of high-risk systems must operate a documented quality management system.
Article 17, European Union Artificial Intelligence Act (2024)
What it cannot tell you
Model governance records whether a model was versioned, tested, monitored and reviewed; it does not tell you whether the model's judgement is sound or whether the decision made on its output was the right one. A fully governed model can still be a poor one. Governance is a record of process, not a guarantee of quality.
Questions
Versioning of models and data, testing before release, monitoring of drift in use, documentation of changes, human review of outputs, and a route for retiring a model that stops performing. Article 9 of the European Union's Artificial Intelligence Act requires exactly this: a risk management system, established, implemented, documented and maintained.
Because a decision has to be explainable later against the system that produced it. If the model has changed since, the explanation has to be about the earlier version. Without a record of which version produced which output, no honest explanation is possible and no comparison across time is valid.
A repeatable test of one part of the system: whether it retrieves the right evidence, interprets it correctly, follows the workflow, or produces outputs of the required quality. Evals run before a release and gate it. A model that fails an eval does not ship, whatever the roadmap says.
Closely. Article 17 of the European Union's Artificial Intelligence Act (2024) requires providers of high-risk systems to operate a documented quality management system. That is one component of model governance written as a legal duty; an organisation with real governance already has much of the substance the Act asks for.
The provider for how the model is built and changed; the deployer for how it is used and monitored in its own setting. Each needs the other's records. A provider that changes a model without telling deployers, or a deployer that never reads what the provider sends, has left a gap neither can close alone.