Contextual Retrieval for Multimodal RAG on Slide Decks

Construct and Persist the Index

We’ll now embed these summarized chunks and retailer them in a vector retailer for retrieval. LlamaIndex can persist indices regionally or combine with 40+ exterior vector databases.

import os
from llama_index.core import (
    StorageContext,
    VectorStoreIndex,
    load_index_from_storage,
)

if not os.path.exists("storage_nodes_iconiq"):
    index = VectorStoreIndex(new_text_nodes, embed_model=embed_model)
    index.set_index_id("vector_index")
    index.storage_context.persist("./storage_nodes_iconiq")
else:
    storage_context = StorageContext.from_defaults(persist_dir="storage_nodes_iconiq")
    index = load_index_from_storage(storage_context, index_id="vector_index")

retriever = index.as_retriever()

Baseline Index (With out Summaries)

We’ll additionally construct a “baseline” index on the unique textual content nodes (with out the contextual summaries) to check the distinction in retrieval high quality.

if not os.path.exists("storage_nodes_iconiq_base"):
    base_index = VectorStoreIndex(text_nodes, embed_model=embed_model)
    base_index.set_index_id("vector_index")
    base_index.storage_context.persist("./storage_nodes_iconiq_base")
else:
    storage_context = StorageContext.from_defaults(
        persist_dir="storage_nodes_iconiq_base"
    )
    base_index = load_index_from_storage(storage_context, index_id="vector_index")

Construct a Multimodal Question Engine

We would like a RAG pipeline that:

• Retrieves related chunks of textual content.
• Additionally hundreds the web page photos.
• Sends each textual content chunks and pictures to a multimodal LLM (right here we illustrate utilizing an OpenAI-like GPT-4 multimodal endpoint, labeled gpt-4o).

import base64
import openai
import os
from typing import Non-compulsory, Checklist

from llama_index.core.query_engine import CustomQueryEngine
from llama_index.core.base.response.schema import Response
from llama_index.core.retrievers import BaseRetriever
from llama_index.core.prompts import PromptTemplate
from llama_index.core.schema import NodeWithScore, MetadataMode

QA_PROMPT_TMPL = """
Under we give parsed textual content from slides, in addition to photos.

---------------------
{context_str}
---------------------

Given the context data and no prior data, please reply the question:

Question: {query_str}
Reply:
"""

QA_PROMPT = PromptTemplate(QA_PROMPT_TMPL)

def encode_image(image_path: str) -> str:
    """If you wish to inline a neighborhood picture in base64."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.learn()).decode("utf-8")


class MultimodalQueryEngine(CustomQueryEngine):
    """
    Customized multimodal Question Engine that retrieves textual content nodes,
    then sends them + picture(s) to the brand new Imaginative and prescient-capable API as documented.
    """

    def __init__(
        self,
        retriever: BaseRetriever,
        model_name: str = "gpt-4o",
        qa_prompt: Non-compulsory[PromptTemplate] = None,
    ) -> None:
        tremendous().__init__(qa_prompt=qa_prompt or QA_PROMPT)
        self.retriever = retriever
        self.model_name = model_name

    def custom_query(self, query_str: str) -> Response:
        # 1) Retrieve textual content nodes
        node_with_scores: Checklist[NodeWithScore] = self.retriever.retrieve(query_str)

        # 2) Construct context
        context_str = "nn".be a part of(
[nws.node.get_content(metadata_mode=MetadataMode.LLM) for nws in node_with_scores]

        )

        # 3) Format the ultimate immediate
        formatted_prompt_text = self._qa_prompt.format(
            context_str=context_str,
            query_str=query_str,
        )

        # 4) Construct the person message with textual content + photos
        user_message_content = [
            {
                "type": "text",
                "text": formatted_prompt_text,
            }
        ]

        for nws in node_with_scores:
            image_path = nws.node.metadata.get("image_path", "")
            if image_path:
                base64_data = encode_image(image_path)
                image_url = f"information:picture/jpeg;base64,{base64_data}"
                user_message_content.append(
                    {
                        "kind": "image_url",
                        "image_url": {
                            "url": image_url,
                            "element": "auto"
                        },
                    }
                )

        messages = [
            {
                "role": "user",
                "content": user_message_content,
            }
        ]

        # 5) Name your Imaginative and prescient mannequin
        response = openai.ChatCompletion.create(
            mannequin=self.model_name,
            messages=messages,
            max_tokens=500,
        )

        # 6) Return a Response object
        return Response(
            response=response.decisions[0].message.content material,
            source_nodes=node_with_scores,
            metadata={},
        )
        
        # 2) Create a question engine
query_engine = MultimodalQueryEngine(
    retriever=index.as_retriever(similarity_top_k=3),
    model_name="gpt-4o",   # or "gpt-4o-mini", "gpt-4-turbo", and so forth.
)

base_query_engine = MultimodalQueryEngine(
    retriever=base_index.as_retriever(similarity_top_k=3),
    model_name="gpt-4o",
)

Making an attempt Out Queries

Let’s question our new pipeline about AI utilization by division.

response = query_engine.question(
    "Which departments use GenAI essentially the most and the way are they utilizing it?"
)
print(str(response))

A typical response may appear like this:

Based mostly on the parsed markdown textual content supplied, the departments/groups that use 
generative AI essentially the most are:
1. **AI, Machine Studying, and Knowledge Science** with a rating of 4.5.
2. **IT** with a rating of 4.0.
3. **Engineering / R&D** with a rating of three.9.
These scores are derived from a survey the place respondents rated the extent of
generative AI utilization on a scale of 1-5.
When it comes to how these departments are utilizing generative AI:
- **AI, Machine Studying, and Knowledge Science**: Whereas particular use circumstances for this
division usually are not detailed within the supplied textual content, it may be inferred that they're
seemingly utilizing generative AI for superior information evaluation, mannequin growth, and
enhancing AI capabilities throughout the group.
- **IT**: The IT division is utilizing generative AI for a number of impactful use circumstances,
together with:
- Ticket administration
- Chatbots
- Buyer help and troubleshooting
- Data administration
- Case summarization
The details about the departments and their use circumstances comes from the parsed
markdown textual content. There are not any discrepancies between the parsed markdown and the
context supplied, because the markdown textual content clearly outlines each the departments with
the best utilization scores and the precise use circumstances for the IT division.

Comparatively, if we run the identical question on the baseline index:

base_response = base_query_engine.question(
    "Which departments use GenAI essentially the most and the way are they utilizing it?"
)
print(str(base_response))

You’ll see the baseline may need fewer particulars or barely totally different retrieval outcomes. Contextual retrieval offers extra exact context across the IT utilization particularly. The response would appear like:

Based mostly on the parsed markdown textual content supplied, the departments that use Generative AI
(GenAI) essentially the most are:
1. **AI, Machine Studying, and Knowledge Science** - This division has the best 
weighted common rating of 4.5 for GenAI utilization, indicating vital adoption. The
particular use circumstances usually are not detailed within the parsed textual content, however given the character of the
division, it's seemingly concerned in growing and refining AI fashions and
algorithms.
2. **IT** - With a rating of 4.0, the IT division can be a number one person of GenAI.
The use circumstances for IT embrace inner productiveness enhancements and IT operations,
as indicated by the 61% adoption charge for inner productiveness and 42% ROI point out
in IT use circumstances.
3. **Engineering / R&D** - This division has a rating of three.9. Whereas particular use
circumstances usually are not detailed within the parsed textual content, it's affordable to deduce that GenAI is
used for product growth and analysis functions, as advised by the 69% 
adoption charge for core product efficiency enhancements and 50% for pure language
interfaces.
The data is derived from the parsed markdown textual content, which offers an in depth
breakdown of GenAI utilization by division and particular use circumstances. There are not any 
discrepancies between the parsed markdown and the uncooked textual content, because the markdown seems
to be a structured illustration of the identical information. The picture was not supplied, so
it was not utilized in forming the reply.

Observing the Advantages of Contextual Retrieval

Right here’s one other instance question, On this subsequent query, the identical sources are retrieved with and with out contextual retrieval, and the reply is appropriate for each approaches. That is thanks for LlamaParse Premium’s capability to grasp graphs.

question = "what are related insights from the 'deep dive on infrastructure' part when it comes to mannequin preferences, price, deployment environments?"

response = query_engine.question(question)
print(str(response))

Output

The "Deep Dive on Infrastructure" part from the ICONIQ Progress report offers
insights into the infrastructure elements obligatory for deploying AI options. 
Nevertheless, the parsed markdown textual content doesn't explicitly point out mannequin preferences or
prices on this part. As a substitute, it focuses on infrastructure tooling and deployment
environments.
From the parsed markdown textual content, we are able to collect the next insights associated to 
deployment environments:
1. **Deployment Environments**: Enterprises are primarily internet hosting generative AI
workloads on the cloud or utilizing a hybrid method. The popular deployment strategies
are:
- Cloud: 56%
- Hybrid: 42%
- On-prem: 2%
2. **Cloud Service Suppliers**: Essentially the most utilized cloud service suppliers for
internet hosting AI workloads are:
- Amazon Net Companies (AWS): 68%
- Microsoft Azure: 61%
- Google Cloud (GCP): 40%
These insights are derived from the parsed markdown textual content, particularly from the 
sections discussing "Cloud Deployment Technique" and "Infrastructure Tooling." There's
no point out of mannequin preferences or price concerns within the supplied textual content. If
there have been any discrepancies or further particulars within the picture or uncooked textual content, they
usually are not out there right here, so the reply is predicated solely on the parsed markdown textual content
supplied.

Now, lets strive with the baseline method:

base_response = base_query_engine.question(question)
print(str(base_response))

Output

The parsed textual content from the slides doesn't present particular insights relating to mannequin preferences, price, or deployment environments within the 'deep dive on infrastructure' part. The slide titled "Deep Dive on Infrastructure" (web page 24) solely comprises the title, the ICONIQ Progress branding, and confidentiality and copyright notices. There isn't any detailed data or information offered within the parsed textual content for this part.

Subsequently, based mostly on the parsed markdown textual content supplied, there are not any related insights out there from the ‘deep dive on infrastructure’ part relating to mannequin preferences, price, or deployment environments. If there have been any photos related to this part, they weren’t supplied, and thus no further insights might be derived from them.

This conclusion is drawn from the parsed markdown textual content, which lacks any particular data on mannequin preferences, price, or deployment environments in that part. The picture confirms this, because it solely exhibits the title and a graphic with out further particulars.

In the event you want insights on these subjects, you may wish to confer with different sections or slides that particularly handle mannequin preferences, prices, or deployment environments.

• Contextual Retrieval may fetch the pages that debate cloud deployment strategies, infrastructure tooling, and price references, resulting in a extra thorough response.
• The baseline method may (in some circumstances) fail to retrieve the right chunk or present much less element.

Evaluating each solutions helps exhibit that these quick “contextual summaries” in your metadata usually result in extra related retrieval.

An enormous due to Jerry Liu from LlamaIndex for creating this wonderful pipeline.

Conclusion

On this tutorial, we explored the method of parsing a PDF slide deck utilizing LlamaParse to extract each textual content and pictures, enriching every textual content chunk with contextual summaries to reinforce retrieval accuracy. We demonstrated learn how to construct a Multimodal RAG pipeline with LlamaIndex, integrating each textual and visible information into a robust mannequin like GPT-4, showcasing the potential of multimodal LLMs. Lastly, we in contrast outcomes from a baseline index to a contextual index, highlighting the numerous enhancements in retrieval precision and relevance achieved by the contextual method. This complete information equips you with the instruments and strategies to construct efficient multimodal AI options.

Key Takeaways

• Contextual retrieval improves chunk matching for queries that may not have a direct key phrase overlap.
• Multimodal RAG can incorporate not simply textual content but additionally photos, charts, or diagrams from slides.
• Immediate caching is important when chunk sizes are giant and also you’re producing a context abstract for every chunk—this may cut back price considerably.
• When you have web-based content material (like retailer listings, giant units of HTML pages), you should use ScrapeGraphAI to fetch that information, then feed it into the identical pipeline.

With these steps, you may adapt the method to any PDF or exterior information supply—whether or not it’s an enormous enterprise data base, advertising supplies, or your organization’s inner documentation.

Steadily Requested Questions

This text was revealed as part of the Knowledge Science Blogathon.

Hello! I am Adarsh, a Enterprise Analytics graduate from ISB, at the moment deep into analysis and exploring new frontiers. I am tremendous keen about information science, AI, and all of the progressive methods they will rework industries. Whether or not it is constructing fashions, engaged on information pipelines, or diving into machine studying, I really like experimenting with the newest tech. AI is not simply my curiosity, it is the place I see the longer term heading, and I am all the time excited to be part of that journey!