The spine of this utility are the brokers and their interactions. Total, we had two various kinds of brokers :
- Consumer Brokers: Brokers hooked up to every consumer. Primarily tasked with translating incoming messages into the consumer’s most well-liked language
- Aya Brokers: Varied brokers related to Aya, every with its personal particular position/job
Consumer Brokers
The UserAgent class is used to outline an agent that shall be related to each consumer a part of the chat room. A few of the capabilities carried out by the UserAgent class:
1. Translate incoming messages into the consumer’s most well-liked language
2. Activate/Invoke graph when a consumer sends a message
3. Preserve a chat historical past to assist present context to the interpretation activity to permit for ‘context-aware’ translation
class UserAgent(object):def __init__(self, llm, userid, user_language):
self.llm = llm
self.userid = userid
self.user_language = user_language
self.chat_history = []
immediate = ChatPromptTemplate.from_template(USER_SYSTEM_PROMPT2)
self.chain = immediate | llm
def set_graph(self, graph):
self.graph = graph
def send_text(self,textual content:str, debug = False):
message = ChatMessage(message = HumanMessage(content material=textual content), sender = self.userid)
inputs = {"messages": [message]}
output = self.graph.invoke(inputs, debug = debug)
return output
def display_chat_history(self, content_only = False):
for i in self.chat_history:
if content_only == True:
print(f"{i.sender} : {i.content material}")
else:
print(i)
def invoke(self, message:BaseMessage) -> AIMessage:
output = self.chain.invoke({'message':message.content material, 'user_language':self.user_language})
return output
For essentially the most half, the implementation of UserAgent is fairly customary LangChain/LangGraph code:
- Outline a LangChain chain ( a immediate template + LLM) that’s chargeable for doing the precise translation.
- Outline a send_text operate thats used to invoke the graph at any time when a consumer needs to ship a brand new message
For essentially the most half, the efficiency of this agent relies on the interpretation high quality of the LLM, as translation is the first goal of this agent. And LLM efficiency can fluctuate considerably for translation, particularly relying on the languages concerned. Sure low useful resource languages don’t have good illustration within the coaching information of some fashions and this does have an effect on the interpretation high quality for these languages.
Aya Brokers
For Aya, we even have a system of separate brokers that every one contributes in direction of the general assistant. Particularly, now we have
- AyaSupervisor : Management agent that supervises the operation of the opposite Aya brokers.
- AyaQuery : Agent for working RAG primarily based query answering
- AyaSummarizer : Agent for producing chat summaries and doing activity identification
- AyaTranslator: Agent for translating messages to English
class AyaTranslator(object):def __init__(self, llm) -> None:
self.llm = llm
immediate = ChatPromptTemplate.from_template(AYA_TRANSLATE_PROMPT)
self.chain = immediate | llm
def invoke (self, message: str) -> AIMessage:
output = self.chain.invoke({'message':message})
return output
class AyaQuery(object):
def __init__(self, llm, retailer, retriever) -> None:
self.llm = llm
self.retriever = retriever
self.retailer = retailer
qa_prompt = ChatPromptTemplate.from_template(AYA_AGENT_PROMPT)
self.chain = qa_prompt | llm
def invoke(self, query : str) -> AIMessage:
context = format_docs(self.retriever.invoke(query))
rag_output = self.chain.invoke({'query':query, 'context':context})
return rag_output
class AyaSupervisor(object):
def __init__(self, llm):
immediate = ChatPromptTemplate.from_template(AYA_SUPERVISOR_PROMPT)
self.chain = immediate | llm
def invoke(self, message : str) -> str:
output = self.chain.invoke(message)
return output.content material
class AyaSummarizer(object):
def __init__(self, llm):
message_length_prompt = ChatPromptTemplate.from_template(AYA_SUMMARIZE_LENGTH_PROMPT)
self.length_chain = message_length_prompt | llm
immediate = ChatPromptTemplate.from_template(AYA_SUMMARIZER_PROMPT)
self.chain = immediate | llm
def invoke(self, message : str, agent : UserAgent) -> str:
size = self.length_chain.invoke(message)
attempt:
size = int(size.content material.strip())
besides:
size = 0
chat_history = agent.chat_history
if size == 0:
messages_to_summarize = [chat_history[i].content material for i in vary(len(chat_history))]
else:
messages_to_summarize = [chat_history[i].content material for i in vary(min(len(chat_history), size))]
print(size)
print(messages_to_summarize)
messages_to_summarize = "n ".be part of(messages_to_summarize)
output = self.chain.invoke(messages_to_summarize)
output_content = output.content material
print(output_content)
return output_content
Most of those brokers have the same construction, primarily consisting of a LangChain chain consisting of a customized immediate and a LLM. Exceptions embrace the AyaQuery agent which has an extra vector database retriever to implement RAG and AyaSummarizer which has a number of LLM capabilities being carried out inside it.
Design concerns
Position of AyaSupervisor Agent: Within the design of the graph, we had a hard and fast edge going from the Supervisor node to the consumer nodes. Which meant that every one messages that reached the Supervisor node had been pushed to the consumer nodes itself. Due to this fact, in instances the place Aya was being addressed, we had to make sure that solely a single remaining output from Aya was being pushed to the customers. We didn’t need intermediate messages, if any, to achieve the customers. Due to this fact, we had the AyaSupervisor agent that acted as the one level of contact for the Aya agent. This agent was primarily chargeable for decoding the intent of the incoming message, direct the message to the suitable task-specific agent, after which outputting the ultimate message to be shared with the customers.
Design of AyaSummarizer: The AyaSummarizer agent is barely extra complicated in comparison with the opposite Aya brokers because it carries out a two-step course of. In step one, the agent first determines the variety of messages that must be summarized, which is a LLM name with its personal immediate. Within the second step, as soon as we all know the variety of messages to summarize, we collate the required messages and go it to the LLM to generate the precise abstract. Along with the abstract, on this step itself, the LLM additionally identifies any motion objects that had been current within the messages and lists it out individually.
So broadly there have been three duties: figuring out the size of the messages to be summarized, summarizing messages, figuring out motion objects. Nonetheless, provided that the primary activity was proving a bit troublesome for the LLM with none express examples, I made the selection to have this be a separate LLM name after which mix the 2 final two duties as their very own LLM name.
It could be attainable to remove the extra LLM name and mix all three duties in a single name. Potential choices embrace :
- Offering very detailed examples that cowl all three duties in a single step
- Producing lot of examples to truly finetune a LLM to have the ability to carry out properly on this activity
Position of AyaTranslator: One of many objectives with respect to Aya was to make it a multilingual AI assistant which might talk within the consumer’s most well-liked language. Nonetheless, it could be troublesome to deal with totally different languages internally throughout the Aya brokers. Particularly, if the Aya brokers immediate is in English and the consumer message is in a unique language, it may probably create points. So in an effort to keep away from such conditions, as a filtering step, we translated any incoming consumer messages to Aya into English. Because of this, all the inside work throughout the Aya group of brokers was accomplished in English, together with the output. We didnt should translate the Aya output again to the unique language as a result of when the message reaches the customers, the Consumer brokers will care for translating the message to their respective assigned language.