Tables, Photographs, figures or equations usually are not drawback anymore! Full Code supplied.
Automated doc processing is without doubt one of the largest winners of the ChatGPT revolution, as LLMs are capable of sort out a variety of topics and duties in a zero-shot setting, which means with out in-domain labeled coaching information. This has made constructing AI-powered functions to course of, parse, and robotically perceive arbitrary paperwork a lot simpler. Although naive approaches utilizing LLMs are nonetheless hindered by non-text context, similar to figures, photos, and tables, that is what we are going to attempt to tackle on this weblog put up, with a particular give attention to PDFs.
At a fundamental stage, PDFs are only a assortment of characters, photos, and features together with their actual coordinates. They don’t have any inherent “textual content” construction and weren’t constructed to be processed as textual content however solely to be considered as is. That is what makes working with them troublesome, as text-only approaches fail to seize all of the format and visible components in some of these paperwork, leading to a big lack of context and knowledge.
One approach to bypass this “text-only” limitation is to do heavy pre-processing of the doc by detecting tables, photos, and format earlier than feeding them to the LLM. Tables may be parsed to Markdown or JSON, photos and figures may be represented by their captions, and the textual content may be fed as is. Nevertheless, this strategy requires customized fashions and can nonetheless lead to some lack of data, so can we do higher?
Most up-to-date massive fashions are actually multi-modal, which means they’ll course of a number of modalities like textual content, code, and pictures. This opens the way in which to a less complicated resolution to our drawback the place one mannequin does all the pieces directly. So, as a substitute of captioning photos and parsing tables, we are able to simply feed the web page as a picture and course of it as is. Our pipeline will be capable to load the PDF, extract every web page as a picture, break up it into chunks (utilizing the LLM), and index every chunk. If a bit is retrieved, then the total web page is included within the LLM context to carry out the duty. In what follows, we are going to element how this may be carried out in follow.
The pipeline we’re implementing is a two-step course of. First, we phase every web page into vital chunks and summarize every of them. Second, we index chunks as soon as then search the chunks every time we get a request and embrace the total context with every retrieved chunk within the LLM context.
Step 1: Web page Segmentation and Summarization
We extract the pages as photos and move every of them to the multi-modal LLM to phase them. Fashions like Gemini can perceive and course of web page format simply:
- Tables are recognized as one chunk.
- Figures kind one other chunk.
- Textual content blocks are segmented into particular person chunks.
- …
For every factor, the LLM generates a abstract than may be embedded and listed right into a vector database.
Step 2: Embedding and Contextual Retrieval
On this tutorial we are going to use textual content embedding just for simplicity however one enchancment can be to make use of imaginative and prescient embeddings immediately.
Every entry within the database consists of:
- The abstract of the chunk.
- The web page quantity the place it was discovered.
- A hyperlink to the picture illustration of the total web page for added context.
This schema permits for native stage searches (on the chunk stage) whereas holding observe of the context (by linking again to the total web page). For instance, if a search question retrieves an merchandise, the Agent can embrace the complete web page picture to offer full format and additional context to the LLM so as to maximize response high quality.
By offering the total picture, all of the visible cues and vital format data (like photos, titles, bullet factors… ) and neighboring objects (tables, paragraph, …) can be found to the LLM on the time of producing a response.
We’ll implement every step as a separate, re-usable agent:
The primary agent is for parsing, chunking, and summarization. This includes the segmentation of the doc into vital chunks, adopted by the era of summaries for every of them. This agent solely must be run as soon as per PDF to preprocess the doc.
The second agent manages indexing, search, and retrieval. This consists of inserting the embedding of chunks into the vector database for environment friendly search. Indexing is carried out as soon as per doc, whereas searches may be repeated as many occasions as wanted for various queries.
For each brokers, we use Gemini, a multimodal LLM with robust imaginative and prescient understanding skills.
Parsing and Chunking Agent
The primary agent is accountable for segmenting every web page into significant chunks and summarizing every of them, following these steps:
Step 1: Extracting PDF Pages as Photographs
We use the pdf2image library. The photographs are then encoded in Base64 format to simplify including them to the LLM request.
Right here’s the implementation:
from document_ai_agents.document_utils import extract_images_from_pdf
from document_ai_agents.image_utils import pil_image_to_base64_jpeg
from pathlib import Pathclass DocumentParsingAgent:
@classmethod
def get_images(cls, state):
"""
Extract pages of a PDF as Base64-encoded JPEG photos.
"""
assert Path(state.document_path).is_file(), "File doesn't exist"
# Extract photos from PDF
photos = extract_images_from_pdf(state.document_path)
assert photos, "No photos extracted"
# Convert photos to Base64-encoded JPEG
pages_as_base64_jpeg_images = [pil_image_to_base64_jpeg(x) for x in images]
return {"pages_as_base64_jpeg_images": pages_as_base64_jpeg_images}
extract_images_from_pdf: Extracts every web page of the PDF as a PIL picture.
pil_image_to_base64_jpeg: Converts the picture right into a Base64-encoded JPEG format.
Step 2: Chunking and Summarization
Every picture is then despatched to the LLM for segmentation and summarization. We use structured outputs to make sure we get the predictions within the format we anticipate:
from pydantic import BaseModel, Subject
from typing import Literal
import json
import google.generativeai as genai
from langchain_core.paperwork import Docclass DetectedLayoutItem(BaseModel):
"""
Schema for every detected format factor on a web page.
"""
element_type: Literal["Table", "Figure", "Image", "Text-block"] = Subject(
...,
description="Sort of detected merchandise. Examples: Desk, Determine, Picture, Textual content-block."
)
abstract: str = Subject(..., description="An in depth description of the format merchandise.")
class LayoutElements(BaseModel):
"""
Schema for the checklist of format components on a web page.
"""
layout_items: checklist[DetectedLayoutItem] = []
class FindLayoutItemsInput(BaseModel):
"""
Enter schema for processing a single web page.
"""
document_path: str
base64_jpeg: str
page_number: int
class DocumentParsingAgent:
def __init__(self, model_name="gemini-1.5-flash-002"):
"""
Initialize the LLM with the suitable schema.
"""
layout_elements_schema = prepare_schema_for_gemini(LayoutElements)
self.model_name = model_name
self.mannequin = genai.GenerativeModel(
self.model_name,
generation_config={
"response_mime_type": "software/json",
"response_schema": layout_elements_schema,
},
)
def find_layout_items(self, state: FindLayoutItemsInput):
"""
Ship a web page picture to the LLM for segmentation and summarization.
"""
messages = [
f"Find and summarize all the relevant layout elements in this PDF page in the following format: "
f"{LayoutElements.schema_json()}. "
f"Tables should have at least two columns and at least two rows. "
f"The coordinates should overlap with each layout item.",
{"mime_type": "image/jpeg", "data": state.base64_jpeg},
]
# Ship the immediate to the LLM
outcome = self.mannequin.generate_content(messages)
information = json.masses(outcome.textual content)
# Convert the JSON output into paperwork
paperwork = [
Document(
page_content=item["summary"],
metadata={
"page_number": state.page_number,
"element_type": merchandise["element_type"],
"document_path": state.document_path,
},
)
for merchandise in information["layout_items"]
]
return {"paperwork": paperwork}
The LayoutElements schema defines the construction of the output, with every format merchandise sort (Desk, Determine, … ) and its abstract.
Step 3: Parallel Processing of Pages
Pages are processed in parallel for velocity. The next technique creates an inventory of duties to deal with all of the web page picture directly for the reason that processing is io-bound:
from langgraph.varieties import Shipclass DocumentParsingAgent:
@classmethod
def continue_to_find_layout_items(cls, state):
"""
Generate duties to course of every web page in parallel.
"""
return [
Send(
"find_layout_items",
FindLayoutItemsInput(
base64_jpeg=base64_jpeg,
page_number=i,
document_path=state.document_path,
),
)
for i, base64_jpeg in enumerate(state.pages_as_base64_jpeg_images)
]
Every web page is shipped to the find_layout_items operate as an unbiased process.
Full workflow
The agent’s workflow is constructed utilizing a StateGraph, linking the picture extraction and format detection steps right into a unified pipeline ->
from langgraph.graph import StateGraph, START, ENDclass DocumentParsingAgent:
def build_agent(self):
"""
Construct the agent workflow utilizing a state graph.
"""
builder = StateGraph(DocumentLayoutParsingState)
# Add nodes for picture extraction and format merchandise detection
builder.add_node("get_images", self.get_images)
builder.add_node("find_layout_items", self.find_layout_items)
# Outline the stream of the graph
builder.add_edge(START, "get_images")
builder.add_conditional_edges("get_images", self.continue_to_find_layout_items)
builder.add_edge("find_layout_items", END)
self.graph = builder.compile()
To run the agent on a pattern PDF we do:
if __name__ == "__main__":
_state = DocumentLayoutParsingState(
document_path="path/to/doc.pdf"
)
agent = DocumentParsingAgent()# Step 1: Extract photos from PDF
result_images = agent.get_images(_state)
_state.pages_as_base64_jpeg_images = result_images["pages_as_base64_jpeg_images"]
# Step 2: Course of the primary web page (for example)
result_layout = agent.find_layout_items(
FindLayoutItemsInput(
base64_jpeg=_state.pages_as_base64_jpeg_images[0],
page_number=0,
document_path=_state.document_path,
)
)
# Show the outcomes
for merchandise in result_layout["documents"]:
print(merchandise.page_content)
print(merchandise.metadata["element_type"])
This ends in a parsed, segmented, and summarized illustration of the PDF, which is the enter of the second agent we are going to construct subsequent.
RAG Agent
This second agent handles the indexing and retrieval half. It saves the paperwork of the earlier agent right into a vector database and makes use of the outcome for retrieval. This may be break up into two seprate steps, indexing and retrieval.
Step 1: Indexing the Cut up Doc
Utilizing the summaries generated, we vectorize them and save them in a ChromaDB database:
class DocumentRAGAgent:
def index_documents(self, state: DocumentRAGState):
"""
Index the parsed paperwork into the vector retailer.
"""
assert state.paperwork, "Paperwork ought to have at the least one factor"
# Test if the doc is already listed
if self.vector_store.get(the place={"document_path": state.document_path})["ids"]:
logger.data(
"Paperwork for this file are already listed, exiting this node"
)
return # Skip indexing if already completed
# Add parsed paperwork to the vector retailer
self.vector_store.add_documents(state.paperwork)
logger.data(f"Listed {len(state.paperwork)} paperwork for {state.document_path}")
The index_documents technique embeds the chunk summaries into the vector retailer. We preserve metadata such because the doc path and web page quantity for later use.
Step 2: Dealing with Questions
When a person asks a query, the agent searches for essentially the most related chunks within the vector retailer. It retrieves the summaries and corresponding web page photos for contextual understanding.
class DocumentRAGAgent:
def answer_question(self, state: DocumentRAGState):
"""
Retrieve related chunks and generate a response to the person's query.
"""
# Retrieve the top-k related paperwork based mostly on the question
relevant_documents: checklist[Document] = self.retriever.invoke(state.query)# Retrieve corresponding web page photos (keep away from duplicates)
photos = checklist(
set(
[
state.pages_as_base64_jpeg_images[doc.metadata["page_number"]]
for doc in relevant_documents
]
)
)
logger.data(f"Responding to query: {state.query}")
# Assemble the immediate: Mix photos, related summaries, and the query
messages = (
[{"mime_type": "image/jpeg", "data": base64_jpeg} for base64_jpeg in images]
+ [doc.page_content for doc in relevant_documents]
+ [
f"Answer this question using the context images and text elements only: {state.question}",
]
)
# Generate the response utilizing the LLM
response = self.mannequin.generate_content(messages)
return {"response": response.textual content, "relevant_documents": relevant_documents}
The retriever queries the vector retailer to search out the chunks most related to the person’s query. We then construct the context for the LLM (Gemini), which mixes textual content chunks and pictures so as to generate a response.
The total agent Workflow
The agent workflow has two levels, an indexing stage and a query answering stage:
class DocumentRAGAgent:
def build_agent(self):
"""
Construct the RAG agent workflow.
"""
builder = StateGraph(DocumentRAGState)
# Add nodes for indexing and answering questions
builder.add_node("index_documents", self.index_documents)
builder.add_node("answer_question", self.answer_question)
# Outline the workflow
builder.add_edge(START, "index_documents")
builder.add_edge("index_documents", "answer_question")
builder.add_edge("answer_question", END)
self.graph = builder.compile()
Instance run
if __name__ == "__main__":
from pathlib import Path# Import the primary agent to parse the doc
from document_ai_agents.document_parsing_agent import (
DocumentLayoutParsingState,
DocumentParsingAgent,
)
# Step 1: Parse the doc utilizing the primary agent
state1 = DocumentLayoutParsingState(
document_path=str(Path(__file__).dad and mom[1] / "information" / "docs.pdf")
)
agent1 = DocumentParsingAgent()
result1 = agent1.graph.invoke(state1)
# Step 2: Arrange the second agent for retrieval and answering
state2 = DocumentRAGState(
query="Who was acknowledged on this paper?",
document_path=str(Path(__file__).dad and mom[1] / "information" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
agent2 = DocumentRAGAgent()
# Index the paperwork
agent2.graph.invoke(state2)
# Reply the primary query
result2 = agent2.graph.invoke(state2)
print(result2["response"])
# Reply a second query
state3 = DocumentRAGState(
query="What's the macro common when fine-tuning on PubLayNet utilizing M-RCNN?",
document_path=str(Path(__file__).dad and mom[1] / "information" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
result3 = agent2.graph.invoke(state3)
print(result3["response"])
With this implementation, the pipeline is full for doc processing, retrieval, and query answering.
Let’s stroll by a sensible instance utilizing the doc LLM & Adaptation.pdf , a set of 39 slides containing textual content, equations, and figures (CC BY 4.0).
Step 1: Parsing and summarizing the Doc (Agent 1)
- Execution Time: Parsing the 39-page doc took 29 seconds.
- End result: Agent 1 produces an listed doc consisting of chunk summaries and base64-encoded JPEG photos of every web page.
Step 2: Questioning the Doc (Agent 2)
We ask the next query:
“Clarify LoRA, give the related equations”
End result:
Retrieved pages:
