The panicked push for AI governance reveals why AI is so hard to implement and get value from at scale.
The Real Cost of AI Inference#
Cost is the one thing that always gets mentioned. Inference is expensive. Not just token cost, but the fact that we have to rerun the same prompts and contexts over and over, essentially running trial and error until we get the right result.
By now, we all know AI never gives perfect outputs. We accept error as a cost of doing business with Mr. AI. That understanding is baked into everything governance-related, including guardrails and compliance.
Every public-facing AI-enabled widget has a disclaimer that “AI makes mistakes!” You don’t see a medical records system pop up a message saying, “This EMR system makes mistakes!” You can imagine how that would go over.
And that is why there’s a panicked push for governance around AI. It’s not just that AI isn’t perfect. It’s not even that AI isn’t good enough, because often it is.
The problem is we don’t know when its outputs will fall within the acceptable error thresholds. That uncertainty means we have to adjust our workflows to handle the cases where AI is really bad and human review is required.
Why AI Needs a Human in the Loop#
I’ve always said, and I stand by this, that any critical AI-enabled workflow must have a human in the loop. What “critical” means depends on the context, but at a minimum, anything affecting human life.
That means no outsourcing diagnoses solely to AI. It also means not letting AI be a standalone air traffic controller, a driver, or a pilot. We can use AI in those cases, but there must be a human to make decisions where a bad output could be catastrophic.
So what’s the problem with just human-in-the-looping everything? If you design things right from the start, it isn’t a problem.
But what happens is that people let AI dictate the workflow (something else I’ve railed against from the beginning). It’s fine to ask AI for advice, and it might even be good advice that you end up taking. But ultimately a human is comprehending and approving the design first.
At that point, where to place manual human-in-the-loop approvals becomes a business decision.
Who Should Design the Workflow?#
This is exactly why so many companies are hiring architects to design custom agentic AI frameworks. Businesses want AI, but they aren’t used to letting the devs, architects, and engineers be involved in creating or critiquing the high-level workflows.
This is why the forward deployed engineer (FDE) role has suddenly appeared seemingly out of nowhere. I’ve always been a generalist, and this is why.
A decade before AI, it was obvious to me that businesses were missing out because they didn’t invite the tech nerds to the strategy meetings. It took fear of missing out (FOMO) about AI to make them finally see the value in having a seasoned technology expert involved throughout the whole process, from high-level workflow design to implementation to day-two operations.
Governance Creates Silos#
Okay, so now that we’re shelling out to implement AI, we have another problem. Everywhere AI receives an input or generates an output in a workflow, there’s some nexus to a governance concern: data privacy, compliance, risk, finance, you name it.
So as we walk through the workflow, the rules change. And that’s how silos are born. Governance requires AI silos.
That’s not as bad as it sounds. One team can manage all of the AI silos. But in most organizations, a team with a holistic understanding of the entire workflow doesn’t exist below the manager level.
Now we have to have a team of AI engineers who understand the governance requirements for AI at every stage of the workflow. Note that this is different from a software engineer writing rules that differ depending on what part of the code he or she is working on.
The AI engineers have to know and implement not just the rules, but also babysit AI’s inputs and outputs so that the workflow diverges accordingly when AI violates the rules (which it inevitably will). So it’s not just about wrapping guardrails around AI. It’s about detecting when AI has slammed into a guardrail, and responding in a way that yields a deterministic output that the downstream workflow stages can use.
Detecting When AI Fails#
To get a better feel for this, think about how AI might answer a particular request. It can (1) do what you asked, (2) do the opposite, or (3) do something else entirely, like actively refuse, babble, or give irrelevant output.
How do you detect when AI has done this? There’s no automated way to detect that with 100% certainty. This is why critical tasks need a human in the loop.
If you think about it, this makes a lot of sense. If you could reliably detect when AI fulfilled a request properly, then why would you need AI to start with?
For example, if you gave it a collection of documents and asked for a complete list of names and roles that appear in those documents, how would you know if it gave you a full list, or left some out? Or hallucinated names or roles? Or mixed them up?
If you had a reliable way to detect that, you wouldn’t need AI. And there’s the rub.
Trading Accuracy for Speed#
We use AI for tasks that humans do with accuracy, but slowly. We’re trading accuracy for speed. In some cases, we can make that sacrifice. It’s a calculated risk.
But when it comes to some things, we don’t want to sacrifice accuracy or speed. So we use AI, but with a human check to “put back” the accuracy that AI took out.
There’s no free lunch. It’s a question of optimization.
The Bicycle Analogy#
In an interview, Steve Jobs once pointed out that a human on a bicycle is more energy efficient than any other animal. I’ve said that LLMs are the bicycles of the information age.
A bicycle is a series of machines optimized for rolling along. To make my analogy a little tighter, LLMs are the machines, and if we piece them together right, we can build our own “information bicycles.”
If you found this post useful, you might also be interested in:
- AI Engineering: Building Applications with Foundation Models by Chip Huyen
- The LLM Engineer's Handbook by Paul Iusztin and Maxime Labonne
- Designing Multi-Agent Systems: Principles, Patterns and Implementation by Victor Dibia
Recommended Reading#
- AI Engineering: Building Applications with Foundation Models by Chip Huyen
- Designing Multi-Agent Systems: Principles, Patterns and Implementation by Victor Dibia
- The LLM Engineer's Handbook by Paul Iusztin and Maxime Labonne