What Actually Makes Something an Agent#
Most vendor content blurs the line between “agent” and “RAG pipeline with a tool call bolted on” on purpose. The ambiguity sells more product than precision would.
Here’s the distinction that matters. A chatbot with a tool call executes a fixed sequence: user asks a question, model decides to call a function, function returns data, model formats a response. One decision point. One tool. The control flow is written by you, not the model.
An agent is different because the model owns the loop. It decides what to do next based on the result of what it just did, and it keeps deciding until it hits a stopping condition, a timeout, or a wall it can’t get past. That means the number of steps, which tools get called, and in what order, are not fixed at design time. They’re determined at runtime by the model’s own reasoning.
That distinction is the whole ballgame for infrastructure purposes. A fixed-sequence tool call has a predictable cost and a predictable failure mode. An agent loop does not. You can’t cap “one function call” the same way you cap “the model decides how many function calls it needs,” because the second one can be zero or it can be forty, and the model won’t tell you in advance which.
The Infrastructure Problems Nobody Puts in the Demo#
Demos work because the task is narrow and the failure modes are hidden. Production doesn’t get that luxury. Three problems show up immediately once you move an agent past a proof of concept.
State management across long-running sessions. A chatbot request-response cycle lives and dies in a few seconds. An agent session might run for minutes, sometimes longer, across multiple tool calls, multiple model invocations, and possibly multiple retries. You need to persist intermediate state somewhere durable, because if your process restarts, gets rescheduled, or times out mid-loop, you need to resume without re-running side effects that already happened. If step three of your agent’s plan sent an email or wrote a database record, you cannot safely retry from step one. This is the same idempotency problem distributed systems have always had, except now the thing deciding whether to retry is a language model instead of your own retry logic.
Cost control when agents loop or retry. A model call has a token cost. An agent loop has a token cost multiplied by however many iterations it decides to run. If your agent gets stuck in a pattern where it keeps calling the same tool, gets an ambiguous result, and calls it again, you’re paying for every one of those calls. Without a hard iteration cap and a hard cost ceiling per task, an agent in a bad loop can burn through a budget that a human would have noticed and stopped after the second try. This isn’t hypothetical. It’s the most common way agent bills surprise people.
Observability when the decision path is non-deterministic. With a normal service, you can trace a request through your code and know exactly why it took the path it took. With an agent, the “why” lives inside a model’s reasoning, and that reasoning can change between two runs with identical inputs. You need to log the full decision trace: what the model saw, what it decided, what tool it called, what came back, and what it decided next. Without that trace, debugging a bad outcome means guessing. Tools built for this, like LangSmith and similar tracing platforms, exist specifically because “add a print statement” doesn’t work when the thing you’re debugging is a chain of model decisions instead of a call stack.
RAG Isn’t Dead. Naive RAG Is Just Bad RAG#
RAG became unfashionable the moment agents became fashionable, which says more about hype cycles than it does about RAG’s actual utility. RAG still solves a real problem: grounding a model’s output in your own current data instead of whatever it memorized during training.
Where naive RAG breaks down is well understood at this point. A stale index means the model is confidently wrong about something that changed last week. Poor chunking means the retriever hands the model half a paragraph missing the context that would have changed its interpretation. And a system with no feedback loop means you have no way to know retrieval is degrading until someone complains that the answers got worse.
Agentic retrieval patterns try to fix this by letting the model decide it needs to search again, rephrase the query, or pull from a different source if the first retrieval doesn’t look sufficient. That’s a real improvement over “retrieve once, stuff it in the prompt, hope for the best.” But it’s not magic. It’s still a retrieval system, just with a loop wrapped around it, and it inherits the same cost and observability problems described above. If your chunking strategy is bad, letting the model retry the search five times doesn’t fix bad chunks. It just costs five times as much to fail.
Edge Agent Platforms vs Rolling Your Own#
Cloudflare’s pitch with Workers AI and its agent tooling is convenience at the edge: low latency, integrated durable objects for state, and a billing model tied to Cloudflare’s usage-based pricing. That’s a legitimate value proposition if your traffic patterns and data already live in Cloudflare’s ecosystem.
The tradeoff is lock-in. Durable Objects, Cloudflare’s KV store, and its Vectorize offering are all proprietary primitives. Building your agent’s state management and retrieval layer directly on top of them means a future migration off Cloudflare is a rewrite, not a redeploy.
Rolling your own on AWS with Bedrock, Lambda, and a vector store like OpenSearch or a managed Postgres with pgvector gives you more portability and more control, at the cost of assembling more pieces yourself. You’re managing IAM policies, cold start latency on Lambda, and a vector store that you provision and scale independently rather than getting for free with your compute. The complexity is real, but so is the portability. If you already run infrastructure on AWS, extending it with Bedrock agents keeps everything under one billing account and one set of operational tooling your team already knows.
Neither option is objectively better. The real question is whether your organization’s existing infrastructure gravity points toward the edge or toward a hyperscaler, and whether the convenience of an integrated platform is worth trading for the flexibility of assembling your own stack.
Do You Even Need an Agent#
Most demos work because the task is narrow: one clear goal, a small number of tools, a controlled environment where failure is unlikely and, if it happens, invisible to the audience. Production tasks are rarely that narrow, and failure is never invisible to the person whose job depends on the output being right.
Before reaching for an agent framework, ask what actually fails in your current process and how often. If your workflow is a fixed sequence of steps that rarely changes, you don’t need an agent. You need an orchestrated pipeline with deterministic control flow and a model call or two where judgment is genuinely required. That’s cheaper to build, cheaper to run, and infinitely easier to debug.
If your workflow requires retrieval but the retrieval is well-scoped and the acceptable answer space is narrow, you need a solid RAG system with good chunking and a feedback loop, not an agent. Agentic retrieval adds cost and non-determinism to solve a problem that better chunking and indexing would have solved for less money.
Reach for a real agent framework only when the number of steps and the choice of tools genuinely cannot be known ahead of time, and when a human is available to intervene when the loop goes somewhere you didn’t expect. That’s a narrower set of use cases than the marketing suggests.
A Checklist for Evaluating Agent Frameworks#
Before adopting any agent framework, whether it’s Cloudflare’s, AWS’s, or an open source one, run it through these questions:
- Latency budget. How long can a user or downstream system wait for a task to complete, and what’s the worst-case latency if the agent needs six iterations instead of two?
- Cost per task ceiling. What’s the maximum dollar amount you’re willing to spend on a single task before you cut it off, and does the framework let you enforce that hard limit?
- Failure recovery behavior. When a tool call fails or returns garbage, does the agent retry sanely, escalate to a human, or silently produce a bad answer with high confidence?
- Auditability. Can you reconstruct, after the fact, exactly what the agent decided and why, well enough to explain it to a customer or a compliance officer?
If a framework can’t give you clear answers to all four, it’s not ready for anything you’d call production, no matter how good the demo looked during Agents Week.
Recommended Reading#
- AWS Certified Solutions Architect Study Guide: Associate SAA-C03 Exam, 4th Edition by Ben Piper & David Clinton
- AI Agents in Action by Micheal Lanham
- AWS Certified Cloud Practitioner Study Guide: CLF-C01 Exam by Ben Piper & David Clinton