AI governance does not start with AI

AI has made many organisations look again at their existing IT governance practices. Kaimar Karu considers how ready we are for AI governance, and how we should prepare for its challenges.

I have been spending a lot of time recently with senior IT leaders, working through their organisations’ AI-related challenges and often landing on discussions about governance. These leaders have usually known, or at least suspected, that their governance-as-designed and their governance-as-practised had drifted apart even before AI debates entered the scene, and that business-led requests for wider AI adoption have made the distance between the two impossible to ignore. It is clear to them that AI governance cannot be just added onto this wobbly foundation, but the task of tackling deep-rooted governance structures at the same time as devising a plan for AI adoption can feel overwhelming. Let’s look at why the usual approach keeps stalling and then at a few starting points that can help.

The state of AI adoption

Figuring out what and how to do with AI has been near the top of many organisations’ priority lists for the past few years. Some organisations have done an exemplary job already – usually not without (significant) challenges, of course – but for most the path ahead is far from clear.

Unlike ‘traditional’ enterprise applications, AI (in its currently most widespread LLM-based form) often does not behave like software at all. It can be an add-on to existing software (e.g. how Copilot is usually introduced) or indeed an application you can buy a subscription for (e.g. various analytics and planning tools), but even then it feels different.

Sometimes, AI tools are purchased specifically to aid decision-making, but more often than that the decision-making aspect emerges from how these tools are used. An operational decision about workforce scheduling. A tactical decision about where to focus next. A strategic decision about investment opportunities. These all include a component of analysing and interpreting data – and then doing something about it. Making a choice and acting on it, including situations where one accepts the suggested wording for an email reply.

Much more than before

Enterprise software users are quite familiar with clicking a button (or tapping the screen) with a clear expectation for that software to perform a specific task. True, it can feel like magic when Excel cleverly adjusts the forecast details, or when the monitoring software produces a dashboard that is so easy to look at you don’t even mind everything being in the red, but you still kind of know how that happens and you also know what you are supposed to do next.

In contrast, the functionality of AI tools often goes well beyond just completing the task. The way they imply meaning behind the detailed information presented on the screen, suggest a few concrete next steps from a potential list of thousands, provide recommendations for follow-up tasks (using that same tool), allow the person in front of the screen to delegate action to automated workflows, and then decisions, too…

While it might feel unfamiliar at times, it can also feel… welcome. So much more can be done, and so much quicker. The amount of toil removed from everyday work can feel like the equivalent of Qomolangma being lifted off one’s shoulders.

Also, not quite like before

From an enterprise point of view, employees being able to do so much more with so much less (not counting the AI tools’ licence fees) is great news, of course. Yet it is also cause for concern as the usual constraints around enterprise applications do not hold anymore. Some approved tools can now do more than they were approved for, and using non-approved tools is as easy as opening a web browser. But what about information security? Data access rights? Regulations?

For that reason, the topic of AI governance has been near the top of many organisations’ priority lists for the past few years. We discuss this in strategy meetings and include it in our annual plans; and in principle, we agree that we should probably get this sorted. Yet, when we get to doing, we still struggle with who should be responsible for governing AI in the enterprise, or how that responsibility connects to the governance of technology already in place.

A common thread through these discussions in organisations is how the starting point itself is creating significant issues. “How do we govern AI?” is a fair question, it is the question, but as a starting point it also assumes that we are quite happy with our existing technology governance, and that for AI, we just need a fresh coat of paint for the south-facing wall.

In reality, though, how much confidence do we actually have in that existing governance? How certain are we that it is working well in 2026, having survived multiple industry and organisational transformations (Agile, Lean, DevOps, cloud computing, COVID-forced remote working and the first true digital transformation for many organisations etc.)? Are the foundations solid enough to deal with situations where a foreign nation’s sponsored software accessing our sensitive data is one “Browse and upload” button away on every employee’s screen?

Everything is fine. Probably.

Many of the leaders I’ve spoken to have struggled to answer these questions with strong confidence. They also recognise that this is not a case of FUD – the risks are real, and the impact can already be felt by most.

In any large organisation, we do have formal governance structures in place, and we do pass the mandated annual audits successfully, so the organisation tends to assume things are working. But those audits verify whether documented procedures, based on expectations from assumptions as understood several years ago, are being followed.

The questions being answered are different from truly knowing whether those procedures are producing the outcomes governance is supposed to deliver: confidence that decisions are being made by the right people, with appropriate permissions, at the right time, and under proper oversight.

We do spot the occasional signs that something is off – perhaps accountability turns out to be unclear after a major incident, or a new critical risk surfaces unexpectedly in an area thought to have been safe and standardised. In many organisations – who are all dealing with strategic challenges and new technologies and once-in-a-lifetime disruptions on an annual basis – these often get resolved individually and we move on. AI is making that moving on much harder now.

What AI walked into

Why? Because our governance of technology was built around two assumptions that were, for the most part, relatively reliable in most organisations. We could see the decisions being made, and we could trace them back to a person who made them. Neither was guaranteed, but common enough and governance procedures could work around the edges.

The governance we are used to was designed for decisions that came through recognisable channels and followed approved workflows. For digital solutions, someone requested a change, a budget was approved, and the IT department knew what was supposed to be deployed.

Individual AI adoption activities can often bypass all of this. A single employee can sign up for a free AI tool in minutes. A department can make a decision about subscribing to a new browser-based AI tool “just to test it out, you know” in a single meeting, and execute immediately without ever involving the IT department. In these situations, there is no purchase order and no deployment request. Technology governance never engages because it never sees the decision.

And there is another challenge, even in situations where the decision was spotted by the governance in place. In pre-AI workflows, we could trace decisions back to people who made them, and these individuals could be requested to explain their reasoning if needed. We had someone to check things with.

Now, in situations where an AI tool recommends a course of action and a human follows that recommendation, the accountability chain is disrupted. It still exists on paper, sort of, but its essence has changed fundamentally.

The human-in-the-loop, sometimes mandated as a position due to legislation, may not have had the expertise to evaluate the recommendation. They may not have even known they were supposed to. And when someone does push back on an AI recommendation, there is rarely a framework in place for how to justify the override. How can a single person think they are smarter than almighty AI? The audacity!

The black box that got a seat at the table

We expect decision-makers to explain their reasoning. That is how accountability works. LLM-based AI does not reason. It processes patterns and produces outputs, and in most cases, no one involved can fully reconstruct why a particular recommendation appeared. We have introduced a black box into the workflow, and this is indeed a problem with a major societal impact.

When someone follows an AI recommendation, or overrides one, the existing governance mechanisms in most organisations have no good way of accounting for how the initial decision was actually made. Anyone can test this by trying to trace the last AI-influenced decision in their organisation back through the established and documented accountability chain. Most of us would run out of answers way before we run out of questions.

Invisible adoption and ungoverned decision-making are already the reality in many organisations, as both the approved and the covert use of AI in the enterprise have exploded. The organisation’s governance is often failing catastrophically in identifying these situations, let alone preventing them. It is becoming quite clear that AI governance is not just ‘a coat of paint’ and requires changes to existing governance – which we need to understand first.

Know thyself (the governance edition)

The “how” of governance varies enormously between organisations, and often even within them, at different levels and in different teams. Without a better understanding of existing governance and without a shared frame of reference, any conversation about what needs to change re: governance tends to stall before it starts.

So how do you assess your existing governance and understand its characteristics? When we were working on ITIL (Version 5) and describing the governance-related aspects of managing products, services, and the whole system of value co-creation, we discussed the practical difficulties and acknowledged the need for an easily adoptable model for this assessment.

(For those interested, most of the governance-related content in ITIL (Version 5) can be found in the Strategy and Transformation publications, and for AI governance specifically, in the AI Governance white paper.)

This model works along two axes, the first of which is authority. For example, is decision-making centralised at the top of the organisation, or distributed across teams and business units? How, and following which rules? Are these rules known and followed? Enforced? Useful? This is where many governance structures start to diverge from reality, because the formal authority map and the way decisions actually get made are often quite different, and the organisation needs to understand how.

The second axis is assurance. Again, for example, is oversight achieved through formal structures and documented procedures, or does it emerge more organically, e.g. through peer review and shared ways of working? How well is this supported and how much of this is real rather than performative? Do we expect one approach, but battle the realities of another on a daily basis?

Where an organisation sits on these two axes provides insight into which of the four BAU governance patterns it most closely follows. Directive governance operates through top-down control and formal hierarchies. Guided governance sets central direction while leaving execution to local teams. Federated governance distributes authority across units that coordinate through formal structures. Autonomous governance relies on self-organising teams and peer-based accountability. All with their nuances, of course, and no two organisations looking the same.

Even though most organisations have a pretty clear idea of which pattern their governance is supposed to follow, significantly fewer have a good understanding of which pattern it actually follows in practice.

Formal, documented and assumed governance might look like Directive on paper, with centralised approval chains and desired structured oversight, while the reality on the ground looks much closer to (uncoordinated) Federated or even (unsanctioned) Autonomous, with teams making their own decisions because the formal process is too slow or too disconnected from how work actually gets done.

These gaps between assumed and real governance hide risks that are hard to spot, and the workarounds people have built around them can amplify those risks once AI enters the picture.

Start where you are

We need to resist the common temptation to treat AI governance purely as a new layer, a fresh coat of paint; as just a new policy and a new committee added to a structure whose load-bearing capacity is, at best, unclear.

That approach will fail for the same reason the existing governance has been struggling, as it assumes the foundations are solid. So, before we can govern AI, we need to sort out how we make decisions now, and how those decisions are governed.

If you can say right now, with confidence, that the governance around (technology-related) decisions is functioning the way it has been documented, you’re good to go and design your AI governance on that basis. It will require you to dismantle some of that foundation and rebuild some of the walls, but you are operating in a predictable world, right?

If, on the other hand, you do wincingly recognise some of the situations described throughout this article, the aforementioned AI white paper can help you figure out what to do next. Not make decisions for you, not dazzle you with easy answers, but help you start the journey confidently.