An open-source, model-agnostic agentic framework that helps dependency injection
Ideally, you’ll be able to consider agentic purposes at the same time as you might be creating them, as an alternative of analysis being an afterthought. For this to work, although, you want to have the ability to mock each inside and exterior dependencies of the agent you might be creating. I’m extraordinarily excited by PydanticAI as a result of it helps dependency injection from the bottom up. It’s the first framework that has allowed me to construct agentic purposes in an evaluation-driven method.
On this article, I’ll discuss in regards to the core challenges and display creating a easy agent in an evaluation-driven manner utilizing PydanticAI.
Challenges when creating GenAI purposes
Like many GenAI builders, I’ve been ready for an agentic framework that helps the complete growth lifecycle. Every time a brand new framework comes alongside, I strive it out hoping that this would be the One — see, for instance, my articles about DSPy, Langchain, LangGraph, and Autogen.
I discover that there are core challenges {that a} software program developer faces when creating an LLM-based utility. These challenges are sometimes not blockers if you’re constructing a easy PoC with GenAI, however they may come to chunk you if you’re constructing LLM-powered purposes in manufacturing.
What challenges?
(1) Non-determinism: In contrast to most software program APIs, calls to an LLM with the very same enter may return completely different outputs every time. How do you even start to check such an utility?
(2) LLM limitations: Foundational fashions like GPT-4, Claude, and Gemini are restricted by their coaching information (e.g., no entry to enterprise confidential data), functionality (e.g., you cannot invoke enterprise APIs and databases), and cannot plan/motive.
(3) LLM flexibility: Even in case you resolve to stay to LLMs from a single supplier similar to Anthropic, it’s possible you’ll discover that you simply want a special LLM for every step — maybe one step of your workflow wants a low-latency small language mannequin (Haiku), one other requires nice code-generation functionality (Sonnet), and a 3rd step requires glorious contextual consciousness (Opus).
(4) Charge of Change: GenAI applied sciences are transferring quick. Not too long ago, lots of the enhancements have come about in foundational mannequin capabilities. Not are the foundational fashions simply producing textual content based mostly on person prompts. They’re now multimodal, can generate structured outputs, and might have reminiscence. But, in case you attempt to construct in an LLM-agnostic manner, you usually lose the low-level API entry that can activate these options.
To assist handle the primary downside, of non-determinism, your software program testing wants to include an analysis framework. You’ll by no means have software program that works 100%; as an alternative, you have to to have the ability to design round software program that’s x% right, construct guardrails and human oversight to catch the exceptions, and monitor the system in real-time to catch regressions. Key to this functionality is evaluation-driven growth (my time period), an extension of test-driven growth in software program.
The present workaround for all of the LLM limitations in Problem #2 is to make use of agentic architectures like RAG, present the LLM entry to instruments, and make use of patterns like Reflection, ReACT and Chain of Thought. So, your framework might want to have the power to orchestrate brokers. Nevertheless, evaluating brokers that may name exterior instruments is tough. You want to have the ability to inject proxies for these exterior dependencies so that you could take a look at them individually, and consider as you construct.
To deal with problem #3, an agent wants to have the ability to invoke the capabilities of several types of foundational fashions. Your agent framework must be LLM-agnostic on the granularity of a single step of an agentic workflow. To deal with the speed of change consideration (problem #4), you wish to retain the power to make low-level entry to the foundational mannequin APIs and to strip out sections of your codebase which might be not vital.
Is there a framework that meets all these standards? For the longest time, the reply was no. The closest I may get was to make use of Langchain, pytest’s dependency injection, and deepeval with one thing like this (full instance is right here):
from unittest.mock import patch, Mock
from deepeval.metrics import GEvalllm_as_judge = GEval(
identify="Correctness",
standards="Decide whether or not the precise output is factually right based mostly on the anticipated output.",
evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
mannequin='gpt-3.5-turbo'
)
@patch('lg_weather_agent.retrieve_weather_data', Mock(return_value=chicago_weather))
def eval_query_rain_today():
input_query = "Is it raining in Chicago?"
expected_output = "No, it's not raining in Chicago proper now."
outcome = lg_weather_agent.run_query(app, input_query)
actual_output = outcome[-1]
print(f"Precise: {actual_output} Anticipated: {expected_output}")
test_case = LLMTestCase(
enter=input_query,
actual_output=actual_output,
expected_output=expected_output
)
llm_as_judge.measure(test_case)
print(llm_as_judge.rating)
Basically, I’d assemble a Mock object (chicago_weather within the above instance) for each LLM name and patch the decision to the LLM (retrieve_weather_data within the above instance) with the hardcoded object each time I wanted to mock that a part of the agentic workflow. The dependency injection is in every single place, you want a bunch of hardcoded objects, and the calling workflow turns into extraordinarily laborious to comply with. Word that in case you don’t have dependency injection, there isn’t any method to take a look at a operate like this: clearly, the exterior service will return the present climate and there’s no method to decide what the proper reply is for a query similar to whether or not or not it’s raining proper now.
So … is there an agent framework that helps dependency injection, is Pythonic, gives low-level entry to LLMs, is model-agnostic, helps constructing it one eval-at-a-time, and is straightforward to make use of and comply with?
Virtually. PydanticAI meets the primary 3 necessities; the fourth (low-level LLM entry) will not be attainable, however the design doesn’t preclude it. In the remainder of this text, I’ll present you learn how to use it to develop an agentic utility in an evaluation-driven manner.
1. Your first PydanticAI Utility
Let’s begin out by constructing a easy PydanticAI utility. This can use an LLM to reply questions on mountains:
agent = llm_utils.agent()
query = "What's the tallest mountain in British Columbia?"
print(">> ", query)
reply = agent.run_sync(query)
print(reply.information)
Within the code above, I’m creating an agent (I’ll present you ways, shortly) after which calling run_sync passing within the person immediate, and getting again the LLM’s response. run_sync is a method to have the agent invoke the LLM and watch for the response. Different methods are to run the question asynchronously, or to stream its response. (Full code is right here if you wish to comply with alongside).
Run the code above, and you’re going to get one thing like:
>> What's the tallest mountain in British Columbia?
The tallest mountain in British Columbia is **Mount Robson**, at 3,954 metres (12,972 toes).
To create the agent, create a mannequin after which inform the agent to make use of that Mannequin for all its steps.
import pydantic_ai
from pydantic_ai.fashions.gemini import GeminiModeldef default_model() -> pydantic_ai.fashions.Mannequin:
mannequin = GeminiModel('gemini-1.5-flash', api_key=os.getenv('GOOGLE_API_KEY'))
return mannequin
def agent() -> pydantic_ai.Agent:
return pydantic_ai.Agent(default_model())
The thought behind default_model() is to make use of a comparatively cheap however quick mannequin like Gemini Flash because the default. You possibly can then change the mannequin utilized in particular steps as vital by passing in a special mannequin to run_sync()
PydanticAI mannequin assist seems sparse, however probably the most generally used fashions — the present frontier ones from OpenAI, Groq, Gemini, Mistral, Ollama, and Anthropic — are all supported. By way of Ollama, you may get entry to Llama3, Starcoder2, Gemma2, and Phi3. Nothing vital appears to be lacking.
2. Pydantic with structured outputs
The instance within the earlier part returned free-form textual content. In most agentic workflows, you’ll need the LLM to return structured information so that you could use it instantly in packages.
Contemplating that this API is from Pydantic, returning structured output is kind of simple. Simply outline the specified output as a dataclass (full code is right here):
from dataclasses import dataclass@dataclass
class Mountain:
identify: str
location: str
peak: float
Whenever you create the Agent, inform it the specified output sort:
agent = Agent(llm_utils.default_model(),
result_type=Mountain,
system_prompt=(
"You're a mountaineering information, who gives correct data to most of the people.",
"Present all distances and heights in meters",
"Present location as distance and course from nearest massive metropolis",
))
Word additionally the usage of the system immediate to specify models and many others.
Operating this on three questions, we get:
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(identify='Mount Robson', location='130km North of Vancouver', peak=3999.0)
>> Is Mt. Hood simple to climb?
Mountain(identify='Mt. Hood', location='60 km east of Portland', peak=3429.0)
>> What is the tallest peak within the Enchantments?
Mountain(identify='Mount Stuart', location='100 km east of Seattle', peak=3000.0)
However how good is that this agent? Is the peak of Mt. Robson right? Is Mt. Stuart actually the tallest peak within the Enchantments? All of this data may have been hallucinated!
There isn’t any manner so that you can know the way good an agentic utility is except you consider the agent towards reference solutions. You can’t simply “eyeball it”. Sadly, that is the place numerous LLM frameworks fall brief — they make it actually laborious to guage as you develop the LLM utility.
3. Consider towards reference solutions
It’s while you begin to consider towards reference solutions that PydanticAI begins to point out its strengths. Every thing is kind of Pythonic, so you’ll be able to construct customized analysis metrics fairly merely.
For instance, that is how we’ll consider a returned Mountain object on three standards and create a composite rating (full code is right here):
def consider(reply: Mountain, reference_answer: Mountain) -> Tuple[float, str]:
rating = 0
motive = []
if reference_answer.identify in reply.identify:
rating += 0.5
motive.append("Appropriate mountain recognized")
if reference_answer.location in reply.location:
rating += 0.25
motive.append("Appropriate metropolis recognized")
height_error = abs(reference_answer.peak - reply.peak)
if height_error < 10:
rating += 0.25 * (10 - height_error)/10.0
motive.append(f"Top was {height_error}m off. Appropriate reply is {reference_answer.peak}")
else:
motive.append(f"Fallacious mountain recognized. Appropriate reply is {reference_answer.identify}")return rating, ';'.be a part of(motive)
Now, we are able to run this on a dataset of questions and reference solutions:
questions = [
"Tell me about the tallest mountain in British Columbia?",
"Is Mt. Hood easy to climb?",
"What's the tallest peak in the Enchantments?"
]reference_answers = [
Mountain("Robson", "Vancouver", 3954),
Mountain("Hood", "Portland", 3429),
Mountain("Dragontail", "Seattle", 2690)
]
total_score = 0
for l_question, l_reference_answer in zip(questions, reference_answers):
print(">> ", l_question)
l_answer = agent.run_sync(l_question)
print(l_answer.information)
l_score, l_reason = consider(l_answer.information, l_reference_answer)
print(l_score, ":", l_reason)
total_score += l_score
avg_score = total_score / len(questions)
Operating this, we get:
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(identify='Mount Robson', location='130 km North-East of Vancouver', peak=3999.0)
0.75 : Appropriate mountain recognized;Appropriate metropolis recognized;Top was 45.0m off. Appropriate reply is 3954
>> Is Mt. Hood simple to climb?
Mountain(identify='Mt. Hood', location='60 km east of Portland, OR', peak=3429.0)
1.0 : Appropriate mountain recognized;Appropriate metropolis recognized;Top was 0.0m off. Appropriate reply is 3429
>> What is the tallest peak within the Enchantments?
Mountain(identify='Dragontail Peak', location='14 km east of Leavenworth, WA', peak=3008.0)
0.5 : Appropriate mountain recognized;Top was 318.0m off. Appropriate reply is 2690
Common rating: 0.75
Mt. Robson’s peak is 45m off; Dragontail peak’s peak was 318m off. How would you repair this?
That’s proper. You’d use a RAG structure or arm the agent with a device that gives the proper peak data. Let’s use the latter strategy and see learn how to do it with Pydantic.
Word how evaluation-driven growth reveals us the trail ahead to enhance our agentic utility.
4a. Utilizing a device
PydanticAI helps a number of methods to offer instruments to an agent. Right here, I annotate a operate to be referred to as each time it wants the peak of a mountain (full code right here):
agent = Agent(llm_utils.default_model(),
result_type=Mountain,
system_prompt=(
"You're a mountaineering information, who gives correct data to most of the people.",
"Use the offered device to search for the elevation of many mountains."
"Present all distances and heights in meters",
"Present location as distance and course from nearest massive metropolis",
))
@agent.device
def get_height_of_mountain(ctx: RunContext[Tools], mountain_name: str) -> str:
return ctx.deps.elev_wiki.snippet(mountain_name)
The operate, although, does one thing unusual. It pulls an object referred to as elev_wiki out of the run-time context of the agent. This object is handed in once we name run_sync:
class Instruments:
elev_wiki: wikipedia_tool.WikipediaContent
def __init__(self):
self.elev_wiki = OnlineWikipediaContent("Record of mountains by elevation")instruments = Instruments() # Instruments or FakeTools
l_answer = agent.run_sync(l_question, deps=instruments) # observe how we're in a position to inject
As a result of the Runtime context will be handed into each agent invocation or device name , we are able to use it to do dependency injection in PydanticAI. You’ll see this within the subsequent part.
The wiki itself simply queries Wikipedia on-line (code right here) and extracts the contents of the web page and passes the suitable mountain data to the agent:
import wikipediaclass OnlineWikipediaContent(WikipediaContent):
def __init__(self, subject: str):
print(f"Will question on-line Wikipedia for data on {subject}")
self.web page = wikipedia.web page(subject)
def url(self) -> str:
return self.web page.url
def html(self) -> str:
return self.web page.html()
Certainly, once we run it, we get right heights now:
Will question on-line Wikipedia for data on Record of mountains by elevation
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(identify='Mount Robson', location='100 km west of Jasper', peak=3954.0)
0.75 : Appropriate mountain recognized;Top was 0.0m off. Appropriate reply is 3954
>> Is Mt. Hood simple to climb?
Mountain(identify='Mt. Hood', location='50 km ESE of Portland, OR', peak=3429.0)
1.0 : Appropriate mountain recognized;Appropriate metropolis recognized;Top was 0.0m off. Appropriate reply is 3429
>> What is the tallest peak within the Enchantments?
Mountain(identify='Mount Stuart', location='Cascades, Washington, US', peak=2869.0)
0 : Fallacious mountain recognized. Appropriate reply is Dragontail
Common rating: 0.58
4b. Dependency injecting a mock service
Ready for the API name to Wikipedia every time throughout growth or testing is a nasty concept. As an alternative, we’ll wish to mock the Wikipedia response in order that we are able to develop rapidly and be assured of the outcome we’re going to get.
Doing that could be very easy. We create a Faux counterpart to the Wikipedia service:
class FakeWikipediaContent(WikipediaContent):
def __init__(self, subject: str):
if subject == "Record of mountains by elevation":
print(f"Will used cached Wikipedia data on {subject}")
self.url_ = "https://en.wikipedia.org/wiki/List_of_mountains_by_elevation"
with open("mountains.html", "rb") as ifp:
self.html_ = ifp.learn().decode("utf-8")def url(self) -> str:
return self.url_
def html(self) -> str:
return self.html_
Then, inject this pretend object into the runtime context of the agent throughout growth:
class FakeTools:
elev_wiki: wikipedia_tool.WikipediaContent
def __init__(self):
self.elev_wiki = FakeWikipediaContent("Record of mountains by elevation")instruments = FakeTools() # Instruments or FakeTools
l_answer = agent.run_sync(l_question, deps=instruments) # observe how we're in a position to inject
This time once we run, the analysis makes use of the cached wikipedia content material:
Will used cached Wikipedia data on Record of mountains by elevation
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(identify='Mount Robson', location='100 km west of Jasper', peak=3954.0)
0.75 : Appropriate mountain recognized;Top was 0.0m off. Appropriate reply is 3954
>> Is Mt. Hood simple to climb?
Mountain(identify='Mt. Hood', location='50 km ESE of Portland, OR', peak=3429.0)
1.0 : Appropriate mountain recognized;Appropriate metropolis recognized;Top was 0.0m off. Appropriate reply is 3429
>> What is the tallest peak within the Enchantments?
Mountain(identify='Mount Stuart', location='Cascades, Washington, US', peak=2869.0)
0 : Fallacious mountain recognized. Appropriate reply is Dragontail
Common rating: 0.58
Look fastidiously on the above output — there are completely different errors from the zero-shot instance. In Part #2, the LLM picked Vancouver because the closest metropolis to Mt. Robson and Dragontail because the tallest peak within the Enchantments. These solutions occurred to be right. Now, it picks Jasper and Mt. Stuart. We have to do extra work to repair these errors — however evaluation-driven growth a minimum of provides us a course of journey.
Present Limitations
PydanticAI could be very new. There are a few locations the place it could possibly be improved:
- There isn’t any low-level entry to the mannequin itself. For instance, completely different foundational fashions assist context caching, immediate caching, and many others. The mannequin abstraction in PydanticAI doesn’t present a method to set these on the mannequin. Ideally, we are able to work out a kwargs manner of doing such settings.
- The necessity to create two variations of agent dependencies, one actual and one pretend, is kind of frequent. It might be good if we had been in a position to annoate a device or present a easy method to swap between the 2 sorts of providers throughout the board.
- Throughout growth, you don’t want logging as a lot. However while you go to run the agent, you’ll normally wish to log the prompts and responses. Generally, it would be best to log the intermediate responses. The way in which to do that appears to be a industrial product referred to as Logfire. An OSS, cloud-agnostic logging framework that integrates with the PydanticAI library could be very best.
It’s attainable that these exist already and I missed them, or maybe they may have been carried out by the point you might be studying this text. In both case, depart a remark for future readers.
General, I like PydanticAI — it affords a really clear and Pythonic method to construct agentic purposes in an evaluation-driven method.
Instructed subsequent steps:
- That is a kind of weblog posts the place you’ll profit from really operating the examples as a result of it describes a technique of growth in addition to a brand new library. This GitHub repo incorporates the PydanticAI instance I walked by means of on this publish: https://github.com/lakshmanok/lakblogs/tree/essential/pydantic_ai_mountains Comply with the directions within the README to strive it out.
- Pydantic AI documentation: https://ai.pydantic.dev/
- Patching a Langchain workflow with Mock objects. My “earlier than” answer: https://github.com/lakshmanok/lakblogs/blob/essential/genai_agents/eval_weather_agent.py