We’re missing something.
Everyone is building and using AI agents. But something doesn’t seem right. We’re missing something, and it feels like it should be obvious, but it’s not. It seems like there’s a way to use LLMs differently and more effectively, but we haven’t quite figured it out yet. And when we try to figure it out, we just end up with a bunch of disconnected or loosely connected observations without a cohesive theory to tie them all together.
The Determinism Problem#
flowchart TB
subgraph Traditional["Traditional System"]
A1[Input] --> A2[Deterministic Function]
A2 --> A3[Predictable Output]
end
subgraph Agentic["LLM-Based System"]
B1[Input] --> B2[LLM]
B2 --> B3{Non-Deterministic Layer}
B3 --> B4[Output A]
B3 --> B5[Output B]
B3 --> B6[Output C]
end
LLMs are not deterministic. Well, they kinda are, because they’re running on computers, which are deterministic machines. But they’re not deterministic in the way regular functions are, where the same set of inputs always yields the same outputs.
LLMs are just predicting the next token. Yet once these models get to a certain size, they suddenly gain this emergent capability to produce human-like text that actually makes sense (usually). What they produce is hardly profound, but it’s still useful. LLMs can produce knowledge-driven artifacts such as code, images, music, and video far faster than ever before.
As we use these LLMs, a theme starts to emerge, and a realization sets in: they don’t actually comprehend anything. They can’t reason. They can’t actually make decisions, or understand what’s best or even good.
Fake Reasoning as a Constraint#
LLMs have been gifted with the ability to “fake reason.” They mimic thinking out loud, complete with filler words like “Okay, let me see… hmmm.” But this is programmed behavior. “Reasoning” is just a programmed execution path.
The implementation of fake “reasoning” was intended to limit the LLM and constrain it to a finite, terminable path. If you used the smaller, older LLMs like GPT 2.5, they could go on and on and just keep spitting out gibberish. The larger models produced less junk, but they could still go off on a tangent and get lost and confused. Constraining models with faux human-like reasoning mimicry redirected them back to a preprogrammed set of rules for execution.
To use an analogy: without reasoning, models are like a young child trying to draw a picture from scratch. It might be mostly decipherable, but it’s going to be messy. With reasoning, models are given an outline and allowed to choose the colors. They might go outside the lines, but they’re programmed to detect this, and they can self-correct.
This is why models excel at coding. Programming requires a rigid set of rules. It’s not a free for all. When trained on code, models infer the rules, and hence make pretty good decisions within that framework.
The Tension Between Freedom and Control#
As we think about this shift from non-reasoning to reasoning, our dilemma with these models becomes clear, and so does the reason why using them always feels a little off. We’re trying to have it both ways. What we’re feeling when we use LLMs is a tension between letting the model be free and creative but wrong, and constraining it with rules that help it perform better, while potentially sacrificing creativity and insight.
Free and creative is the route people tend to take when they first use LLMs. They just want to dump things on it and watch it go. That doesn’t work, so we slowly start to creep toward constraint instead.
Prompt and context engineering is just constraining the model with rules so that it doesn’t mess up as much. But determining those rules, writing them, and making sure we haven’t missed anything is fundamentally design work. We have to do a lot of setup with prompts and context to get the LLM to a place where it can execute without being continually, manually steered.
But as we write more and more rules into our prompts, we start to see that some of what we’re asking the LLM to do can be done programmatically. So now we have to split the programmatic, deterministic rules from the looser, natural language prompts. This is where the idea of multiple agents came from.
flowchart LR
A["Free and Creative
(Easy, but Wrong)"] --> B[Prompt Engineering]
B --> C[Context Engineering]
C --> D[Agent Skills]
D --> E["Fully Constrained
(Hard, but Right)"]
The Rise of Multi-Agent Systems#
Now we have a proliferation of agents and functions. One agent passes its output to another agent, which passes its output to a function, which passes its output to an agent, and so on. We’re now having to do more than just task decomposition. We’re having to figure out which tasks should be done by an agent and which should be done with old-fashioned deterministic code.
The agent represents a weak link in the chain. If it feeds garbage output to, say, a function, the function might not produce valid output. So we have to add more complexity: a feedback loop to detect and remedy bad agent outputs. That’s why we have things like LangChain and LangGraph now. When implemented, those feedback loops essentially become functional wrappers around the agents.
Agents as Approximation Functions#
And here’s where we feel more tension. Anytime we’re doing something with an agent, we have to protect it with functional rules for its inputs and outputs. This isn’t a bad thing. We already do this with typing, and that’s the point. The more we do with agents, the more the agent feels like a strange, unpredictable function.
Well, that’s exactly what it is. Agents behave like approximation functions. This makes them good at quickly creating “close enough” or “good enough” artifacts such as code, images, sound, and video. But these artifacts are usually used as inputs into a workflow whose output must be accurate and precise.
That accuracy requires iterating with other agents and wrapping the inputs and outputs with functions that bound the insanity an agent might produce. And even with all that, this really doesn’t work. Put simply, you can’t use approximations or best guesses as inputs and expect an accurate, precise, predictable output every time.
The Iron Triangle of Agent Tradeoffs#
So why do we use agents at all? It’s a tradeoff. We could have a human deal with the ambiguous tasks that can’t be written as deterministic code, but humans take time to work. An LLM can do many of those tasks much faster, but at the cost of accuracy.
The old iron triangle rears its head again: good, fast, cheap. Pick only two. If you choose human, you get good and cheap. If you choose LLM, you get fast and cheap. This is why companies are struggling with AI. They wrongly believe they can use LLMs to get good, fast, and cheap. Alas, there’s no free lunch here.
I’ve heard the sentiment that in critical workflows, such as those supporting life, human review of AI is mandatory. That’s because the output of such applications must be good. But most businesses would probably agree that the must-be-good requirement applies to many areas, not just those supporting life.
Agent skills were one answer to the problem of agents being too loose with their reasoning and outputs. Skills are nothing more than constraints on a model. Evaluations (evals) are an attempt to reify how well a particular skill steers the agent. Notice a pattern here? Everything we do with these agents is trying to force them to act like deterministic functions.
Pushing Rules Into the Model#
But wait a minute. If we’re using context engineering, prompt engineering, and skills to make agents act more deterministically, are we operating at the wrong layer? Perhaps it makes more sense to push this rule-making to the model itself.
Of course, this would defeat the purpose of the huge, multimodal, multipurpose foundation models. The whole idea behind those was to have a “brain” that could achieve a variety of tasks. But that’s not how we actually use these models. We break tasks down into things for specialized agents.
If we follow this approach to its end, and especially if you buy into Buckminster Fuller’s concept of ephemeralization (the idea that technology always becomes smaller and more invisible), then the next step is specialized, purpose-built, miniature models. In other words, smaller approximation functions that are better at approximating than their huge, billion- or trillion-parameter cousins.
We already have models specialized for medicine (e.g. internlm2, medllama, etc). But this can and will be broken down further, into models for gastroenterology, neurology, and so on. Those will then be broken down even further into models for specific ailments, diagnoses, and procedures.
Of course, this requires careful curation and work that only humans can do. We can’t trust a generalized foundation model to properly classify and train a smaller model with close to 100% accuracy. That is a human task.
(This is one reason artificial general intelligence, or AGI, is impossible. A huge approximation function cannot generate something more accurate than itself without human input. I digress.)
Why Models Excel at Code#
We all know models are great at coding because, as I said earlier, code has a lot of rules that are very easy for the model to learn. Not only are there implicit rules the model learns just from being trained on code, it also learns from the many books written on programming.
Many people have asked why models can’t just directly produce machine-readable executable code and skip the human-readable programming language. Why write Go or Rust instead of just producing a binary executable?
After our discussion, the reason is obvious: models haven’t been trained on raw binary executables. They don’t know the rules for those, at least not to the extent they know code. A model might have been trained on some hex dumps it can translate to binary, but the chances of it producing a safe, runnable binary are extremely low.
Recommended Reading#
- The LLM Engineer's Handbook by Paul Iusztin and Maxime Labonne
- Designing Multi-Agent Systems: Principles, Patterns and Implementation by Victor Dibia
- AI Engineering: Building Applications with Foundation Models by Chip Huyen
Featured image by Brett Jordan on Unsplash