Building block: The augmented LLM#

The basic building block of agentic systems is an LLM enhanced with augmentations such as retrieval, tools, and memory. Most models can actively use these capabilities—generating their own search queries, selecting appropriate tools, and determining what information to retain.

In practice, it is recommended to focus on 2 key aspects of the implementation:

1. tailoring these capabilities to specific use case, and

2. ensuring they provide an easy, well-documented interface for LLM.

   One approach is through Model Context Protocol, which allows developers to integrate with a growing ecosystem of third-party tools with a simple client implementation.

   

   Having Model Context Protocol (MCP) as the bridge between LLMs and external tools, the system changes from a model trying to handle individual tool integrations manually, to a streamlined client-server model.

Agents#

Agents are emerging in production as LLMs mature in key capabilities:

• understanding complex inputs
• engaging in reasoning and planning
• using tools reliably
• recovering from errors

Agents begin their work with either a command from, or interactive discussion with, the human user. Once the task is clear, agents plan and operate independently, potentially returning to the human for further information or judgement. During execution, it’s crucial for the agents to gain “ground truth” from the environment at each step (such as tool call results or code execution) to assess its progress. Agents can then pause for human feedback at checkpoints or when encountering blockers. The task often terminates upon completion, but it’s also common to include stopping conditions (such as a maximum number of iterations) to maintain control.

Agents can handle sophisticated tasks, but their implementation is often straightforward. They are typically just LLMs using tools based on environmental feedback in a loop. It is therefore crucial to design toolsets and their documentation clearly and thoughtfully.

When to use agents

Agents can be used for open-ended problems where it’s difficult or impossible to predict the required number of steps, and where we can’t hardcode a fixed path. The LLM will potentially operate for many turns, and we must have some level of trust in its decision-making. Agents’ autonomy makes them ideal for scaling tasks in trusted environments.

The autonomous nature of agents means higher costs, and the potential for compounding errors. It is recommended to extensively test in sandboxed environments, along with the appropriate guardrails.

Examples where agents are useful:

• A coding Agent to resolve SWE-bench tasks, which involve edits to many files based on a task description
• Anthropic’s “computer use” reference implementation, where Claude uses a computer to accomplish tasks.

High-level flow of a coding agent

Workflow: Prompt Chaining#

Explore in Jupyter notebook

Prompt chaining decomposes a task into a sequence of steps, where each LLM call processes the output of the previous one. We can add programmatic checks (see “gate” in the diagram below) on any intermediate steps to ensure that the process is still on track.

When to use this workflow

This workflow is ideal for situations where the task can be easily and cleanly decomposed into fixed subtasks. The main goal is to trade off latency for higher accuracy, by making each LLM call an easier task. For example:

• Generating Marketing copy, then translating it into a different language.
• Writing an outline of a document, checking that the outline meets certain criteria, then writing the document based on the outline.

Workflow: Routing#

Explore in Jupyter notebook

Routing classifies an input and directs it to a specialized followup task. This workflow allows for separation of concerns, and building more specialized prompts. Without this workflow, optimizing for one kind of input can hurt performance on other inputs.

When to use this workflow

Routing works well for complex tasks where there are distinct categories that are better handled separately, and where classification can be handled accurately, either by an LLM or a more traditional classification model/algorithm. For example:

• Directing different types of customer service queries (general questions, refund requests, technical support) into different downstream processes, prompts, and tools.
• Routing easy/common questions to smaller, cost-efficient models and hard/unusual questions to more capable models to optimize for best performance.

Workflow: Parallelization#

Explore in Jupyter notebook

LLMs can sometimes work simultaneously on a task and have their outputs aggregated programmatically. This workflow, parallelization, manifests in 2 key variations:

• Sectioning: Breaking a task into independent subtasks run in parallel.
• Voting: Running the same task multiple times to get diverse outputs.

When to use this workflow

Parallelization is effective when the divided subtasks can be parallelized for speed, or when multiple perspectives or attempts are needed for higher confidence results. For complex tasks with multiple considerations, LLMs generally perform better when each consideration is handled by a separate LLM call, allowing focused attention on each specific aspect.

Examples where parallelization is useful:

• Sectioning

  • Implementing guardrails where one model instance processes user queries while another screens them for inappropriate content or requests. This tends to perform better than having the same LLM call handle both guardrails and the core response.
  • Automating evals for evaluating LLM performance, where each LLM call evaluates a different aspect of the model’s performance on a given prompt.

• Voting

  • Reviewing a piece of code for vulnerabilities, where several different prompts review and flag the code if they find a problem.
  • Evaluating whether a given piece of content is inappropriate, with multiple prompts evaluating different aspects or requiring different vote thresholds to balance false positives and negatives.

Workflow: Orchestrator-Workers#

In the orchestrator-workers workflow, a central LLM dynamically breaks down tasks, delegates them to worker LLMs, and synthesizes their results.

When to use this workflow

This workflow is well-suited for complex tasks where we can’t predict the subtasks needed (in coding, for example, the number of files that need to be changed and the nature of the change in each file likely depend on the task). Whereas it’s topographically similar, the key difference from parallelization is its flexibility - subtasks aren’t pre-defined, but determined by the orchestrator based on the specific input. For example:

• Coding products that make complex changes to multiple files each time.
• Search tasks that involve gathering and analyzing information from multiple sources for possible relevant information.

Workflow: Evaluator-Optimizer#

Explore in Jupyter notebook

In the evaluator-optimizer workflow, one LLM call generates a response while another provides evaluation and feedback in a loop.

When to use this workflow

This workflow is particularly effective when we have clear evaluation criteria, and when iterative refinement provides measurable value. The two signs of good fit are, first, that LLM responses can be demonstrably improved when a human articulates their feedback; and second, that the LLM can provide such feedback. This is analogous to the iterative writing process a human writer might go through when producing a polished document.

Examples where evaluator-optimizer is useful:

• Literary translation where there are nuances that the translator LLM might not capture initially, but where an evaluator LLM can provide useful critiques.
• Complex search tasks that require multiple rounds of searching and analysis to gather comprehensive information, where the evaluator decides whether further searches are warranted.

Combining and Customizing These Patterns#

These building blocks aren’t prescriptive. They’re common patterns that developers can shape and combine to fit different use cases. The key to success, as with any LLM features, is measuring performance and iterating on implementations. To repeat: we should consider adding complexity only when it demonstrably improves outcomes.

When to Use Agents#

When building applications with LLMs, it is recommended to find the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all. Agentic systems often trade latency and cost for better task performance, and we should consider when this tradeoff makes sense.

When more complexity is warranted, workflows offer predictability and consistency for well-defined tasks, whereas agents are the better option when flexibility and model-driven decision-making are needed at scale. For many applications, however, optimizing single LLM calls with retrieval and in-context examples is usually enough.

When to Use Agent Workflows#

There are many frameworks that make agentic systems easier to implement, including:

• Graph & State-Machine Orchestration

  • LangGraph: Conceptualizes workflows as formal graphs where nodes act as arbitrary transforms and edges represent deterministic branches based on state. It is heavily utilized for production systems requiring state checkpointing, durable execution, and state rollback (time travel) patterns independently of specific model providers.

• Lightweight / Handoff-Driven: These frameworks prioritize minimal abstraction, avoiding heavy, rigid DAG (Directed Acyclic Graph) architectures in favor of letting the model’s native ReAct loop drive the process.

  • The Claude Agent SDK: Anthropic’s library focuses heavily on the core autonomous loop (Reason $\rightarrow$ Act $\rightarrow$ Observe) and tool execution. Rather than enforcing strict state machines, it provides the base primitives to give the model a “computer” and lets the LLM dynamically decide its tool choices and step counts.
  • Strands Agents SDK by AWS: Similar to the Claude Agent SDK and OpenAI Swarm, this open-source AWS framework is fundamentally model-driven. It is designed to minimize boilerplate orchestration code by allowing the LLM itself to handle reasoning and tool selection, rather than relying on deterministic, hardcoded control flows.

• Event-Driven & Data-Centric

  • LlamaIndex Workflows: Utilizes an event-driven workflow architecture that shifts away from rigid DAG (Directed Acyclic Graph) abstractions. Workflow steps emit typed events that trigger downstream handlers asynchronously, allowing for detailed loop control and state isolation without requiring proprietary vendor infrastructure.

• Role-Based & Conversational Multi-Agent

  • CrewAI: Designed around role-based multi-agent collaboration. Developers define individual agents with specific personas, goals, and backstories, grouping them into “crews” that execute tasks either sequentially or hierarchically. It is highly optimized for fast prototyping and collaborative delegation.

• Enterprise & Type-Safe Integration

  • Pydantic AI: A developer-centric, type-first Python framework built by the Pydantic team. It prioritizes strict type safety, predictable structured data outputs, and robust observability (via native tracing tools like Logfire) while remaining completely provider-agnostic.

• Open-Source Visual Builders (GUI Tools): These platforms shift orchestration away from backend scripting and into visual interfaces, allowing developers to map out pipelines intuitively.

  • Rivet, a drag and drop GUI LLM workflow builder; and
  • Vellum, another GUI tool for building and testing complex workflows

These frameworks make it easy to get started by simplifying standard low-level tasks like calling LLMs, defining and parsing tools, and chaining calls together. However, they often create extra layers of abstraction that can obscure the underlying prompts and responses, making them harder to debug. They can also make it tempting to add complexity when a simpler setup would suffice.

It is suggested that developers start by using LLM APIs directly: many patterns can be implemented in a few lines of code. If we do use a framework, ensure we understand the underlying code. Incorrect assumptions about what’s under the hood are a common source of customer error.