I’ve read a lot of corporate AI policies this year. Most of them are careful, well-intentioned and written for a single imagined system: a chatbot that answers questions. Then the same policy is applied to a forecasting model, a document classifier and an agent that can send emails on someone’s behalf.

A policy that treats those four the same way governs none of them properly. The problem isn’t the effort that went into the policy. It’s the word “AI.”

For the rest of us: why “AI” is a useless label for governance

“AI” has never had a stable meaning. Alan Turing, who opened the field in 1950, refused to define whether machines could “think” and proposed a behavioural test instead. Seventy-five years later, a much-cited management paper by Kaplan and Haenlein opened by noting that people use “AI” for things as different as expert systems, statistical prediction and science-fiction robots.

Berente, Gu, Recker and Santhanam, writing an editorial for MIS Quarterly in 2021, made the sharpest version of the point. AI, they argue, is a moving frontier: whatever computing is newest and most impressive gets the label, until it becomes routine and gets renamed. Search was once AI. So were recommendation engines and fraud detection. Today we just call them features.

For governance, that has a practical consequence. Knowing that something is “AI” tells you almost nothing about what controls it needs. A hand-coded rules engine is deterministic and can be audited line by line. A trained classifier gives repeatable answers but can be hard to explain. A generative model can give different answers to the same question. An agent can act on those answers. One label, four very different governance problems.

The three facets

Berente and colleagues propose a better handle. They describe contemporary AI by three facets, and each one maps to a different governance burden. I use them in my doctoral research as a test for any governance framework.

Autonomy: the system acts with less direct human direction. The governance burden is decision rights. When a system approves a refund, routes a case or sends an email, a decision that used to belong to a named person now belongs to a running system. Peter Weill and Jeanne Ross defined IT governance twenty years ago as the allocation of decision rights and accountabilities. Autonomy is that exact question, asked about software. Test question: for each action this system can take, who is accountable, and who could have stopped it?

Learning: the system’s behaviour changes with data and feedback. The governance burden is re-approval. Most governance is point-in-time: a review board approves a design, and the approval stands until someone changes the system. With learning systems, and with models updated by their providers, the system that passed review last quarter isn’t necessarily the system running today. I’ve written about how this happens silently in model drift. Test question: what event triggers a re-review, and would you notice it happening?

Inscrutability: the outputs resist explanation, even by the people who built the system. The governance burden is an audit ceiling. Documentation, model cards and review can all be done diligently and still not tell you why a particular output came out the way it did. Inscrutability doesn’t make governance pointless. It means some assurance has to come from watching behaviour, not reading specifications. Test question: where your documentation runs out, what behavioural evidence do you collect instead?

Applying the test

Take whatever AI policy your organisation has and mark each control against the three facets. In my experience, the pattern is fairly consistent.

Autonomy is usually half-covered. Policies say “a human must remain in the loop” without saying which human, with what authority, and whether they can realistically review what they approve. That’s a slogan, not a decision-rights allocation. (I’ve argued elsewhere that the useful question is who the right human is, not whether one exists.)

Learning is usually the gap. Almost every policy has an approval step. Very few define what should trigger re-approval: a provider model update, a change in the retrieved data, a new tool permission, a drop in measured quality. If nothing triggers re-review, the approval becomes stale the moment the system changes.

Inscrutability is usually treated as a documentation problem. The instinct is to demand more documentation. But if the outputs can’t be explained from the documentation, more of it doesn’t help. What helps is monitoring behaviour in production: sampling outputs, tracking error rates, and watching for shifts.

A note on the words themselves

Melanie Mitchell, in a 2021 paper called Why AI Is Harder Than We Think, describes a trap she calls “wishful mnemonics.” The words we use for AI systems, like learn, understand and decide, are borrowed from human cognition and quietly carry over capabilities the systems don’t have.

It matters for governance, because the words end up in policies. A system that “understands” customer intent sounds like it can be trusted with judgment. A system that “classifies text into intent categories with a measured error rate” sounds like what it is. The second description is less exciting and much easier to govern.

What this means

None of this requires a new committee. Autonomy, learning and inscrutability map onto questions that architecture and governance functions already know how to ask: decision rights, change control and monitoring. What’s different is that all three now apply inside the technical layer, to systems that don’t behave the same way twice.

People were never deterministic either, and organisations have always found ways to govern them through delegation, supervision and accountability. What’s new is that unpredictability now ships as software. Ask the three questions of your own policy. If it only answers one of them, you know where to start.

References