Enhancing RAG Programs with Nomic Embeddings

The intersection of synthetic intelligence and information processing has developed considerably with the rise of multimodal Retrieval-Augmented Era methods. Multimodal RAG goes past conventional fashions that focus solely on textual content. It integrates numerous information varieties like textual content, photographs, audio, and video. This enables for extra nuanced and context-aware responses. A key innovation is Nomic imaginative and prescient embeddings. They create a unified house for each visible and textual information. This permits seamless interplay throughout totally different codecs. By utilizing superior fashions to generate high-quality embeddings, multimodal RAG improves data retrieval. It bridges the hole between totally different content material varieties. The result’s richer and extra informative person experiences.

Studying Targets

• Perceive the basics of multimodal Retrieval-Augmented Era methods and their benefits over conventional RAG.
• Discover the position of Nomic Imaginative and prescient Embeddings in making a unified embedding house for textual content and pictures.
• Examine Nomic Imaginative and prescient Embeddings with CLIP fashions and analyze their efficiency benchmarks.
• Implement a multimodal RAG system in Python utilizing Nomic Imaginative and prescient and Textual content Embeddings.
• Discover ways to extract and course of textual and visible information from PDFs for multimodal retrieval.

This text was printed as part of the Information Science Blogathon.

What’s Multimodal RAG?

Multimodal RAG represents a major development in synthetic intelligence. It’s construct upon conventional RAG methods by incorporating various information varieties akin to textual content, photographs, audio, and video. Not like standard RAG methods that primarily course of textual data, multimodal RAG is designed to deal with and combine a number of types of information concurrently. This functionality permits for extra complete understanding and technology of responses which are context-aware throughout totally different modalities.

Key Elements of Multimodal RAG

• Information Ingestion: The method begins with ingesting numerous sorts of information by means of specialised processors for every format. This ensures that the system can validate, clear, and normalize incoming information whereas preserving its important traits
• Vector Illustration: Totally different modalities are processed utilizing respective neural networks (e.g., CLIP for photographs or BERT for textual content) to generate unified vector representations or embeddings. These embeddings keep semantic relationships throughout totally different modalities.
• Vector Database Storage: The generated embeddings are saved in vector databases optimized with indexing methods like HNSW or FAISS for environment friendly retrieval
• Question Processing: Incoming queries are analyzed and reworked into the identical vector house because the saved information to find out related modalities and generate acceptable embeddings for search

Nomic Imaginative and prescient Embeddings

A major innovation on this discipline of multimodal embeddings is the incorporation of Nomic imaginative and prescient embeddings, which create a cohesive embedding house for each visible and textual information. 

Nomic Embed Imaginative and prescient v1 and v1.5 are each high-quality imaginative and prescient embedding fashions developed by Nomic AI, designed to share the identical latent house as their corresponding textual content embedding fashions, Nomic Embed Textual content v1 and v1.5, respectively. It operates throughout the identical house as Nomic Embed Textual content, making it well-suited for multimodal duties akin to text-to-image retrieval. With a imaginative and prescient encoder comprising solely 92M parameters, Nomic Embed Imaginative and prescient is well-suited for high-volume manufacturing functions, complementing the 137M parameters of Nomic Embed Textual content.

CLIP fashions undergo in unimodal duties

Multimodal fashions akin to CLIP show exceptional zero-shot capabilities throughout totally different modalities. Nevertheless, CLIP’s textual content encoders wrestle with duties past picture retrieval, as seen in benchmarks like MTEB, which evaluates the effectiveness of textual content embedding fashions. Nomic Embed Imaginative and prescient goals to handle these limitations by aligning a imaginative and prescient encoder with the prevailing Nomic Embed Textual content latent house.

To deal with the difficulty of underperformance on unimodal duties, akin to semantic similarity, Nomic Embed Imaginative and prescient, a imaginative and prescient encoder, was educated alongside Nomic Embed Textual content, a long-context textual content encoder. The coaching technique concerned freezing the textual content encoder and coaching the imaginative and prescient encoder on image-text pairs. This strategy not solely produced optimum outcomes but in addition ensured backward compatibility with the embeddings from Nomic Embed Textual content.

Efficiency Benchmarks of Nomic Imaginative and prescient Embeddings

As talked about earlier, present multimodal fashions akin to CLIP exhibit spectacular zero-shot capabilities throughout totally different modalities. Nevertheless, the efficiency of CLIP’s textual content encoders is subpar outdoors of duties like picture retrieval, as evidenced by benchmarks like MTEB, which evaluates the standard of textual content embedding fashions. Nomic Embed Imaginative and prescient is particularly designed to handle these shortcomings by aligning a imaginative and prescient encoder with the prevailing Nomic Embed Textual content latent house. This alignment ends in a unified multimodal latent house that delivers robust efficiency on picture, textual content, and multimodal duties, as demonstrated by the Imagenet Zero-Shot, MTEB, and Datacomp benchmarks.

Arms on Python Implementation of MultiModal RAG with Nomic Imaginative and prescient Embeddings

On this tutorial, we’ll construct a multimodal RAG system that may effectively retrieve data from a PDF containing each textual and visible content material. We are going to construct this on Google Colab utilizing T4 GPU (Free tier).

Step 1: Putting in Essential Libraries

Set up all required Python libraries, together with OpenAI, Qdrant, Transformers, Torch, and PyMuPDF.

!pip set up openai==1.55.3 httpx==0.27.2 
!pip set up qdrant_client
!pip set up transformers
!pip set up transformers torch pillow
!pip set up --upgrade nltk
!pip set up sentence-transformers
!pip set up --upgrade qdrant-client fastembed Pillow
!pip set up PyMuPDF

Step 2: Setting OpenAI API key and Importing Essential Libraries

Arrange the OpenAI API key and import important libraries like PyMuPDF, PIL, LangChain, and OpenAI.

from openai import ChatCompletion
import openai
import os
from openai import AzureOpenAI
from PIL import Picture
import torch
import numpy as np
import fitz  # PyMuPDF
import os
import time
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.messages import HumanMessage
from langchain_openai import ChatOpenAI
from openai import ChatCompletion
import openai
import base64
from base64 import b64decode

os.environ["OPENAI_API_KEY"] = ''

Arrange the OpenAI API key and import important libraries like PyMuPDF, PIL, LangChain, and OpenAI.

#photographs

def extract_images_from_pdf(pdf_path, output_folder):
    pdf_document = fitz.open(pdf_path)
    os.makedirs(output_folder, exist_ok=True)
    #Iterating throught the pages within the PDF
    for page_number in vary(len(pdf_document)):
        web page = pdf_document[page_number]
        #Perform For Getting Photos From the PDF Pages
        photographs = web page.get_images(full=True)

        for image_index, img in enumerate(photographs):
            xref = img[0]
            base_image = pdf_document.extract_image(xref)
            image_bytes = base_image["image"]
            image_ext = base_image["ext"]
            image_filename = f"page_{page_number+1}_image_{image_index+1}.{image_ext}"
            image_path = os.path.be a part of(output_folder, image_filename)
            with open(image_path, "wb") as image_file:
                image_file.write(image_bytes)
    pdf_document.shut()

Use PyMuPDF to extract textual content from all pages of the PDF and retailer it in an inventory.

def extract_text_pdf(path):
    """Extracts textual content from a PDF utilizing PyMuPDF."""
    doc = fitz.open(path)
    text_results = []
    for web page in doc:
        textual content = web page.get_text()
        text_results.append(textual content)
    return text_results

Step 5: Saving Extracted Textual content and Photos From PDF

Save photographs within the “check” listing and extract textual content for additional processing.

def get_contents(pdf_path, output_directory):
  """Extracts textual content and pictures from a PDF, saves photographs, and returns textual content and elapsed time."""

  extract_images_from_pdf(pdf_path, output_directory)
  text_results=extract_text_pdf(pdf_path)
  return(text_results)
  
pdf_path = "/content material/retailcoffee.pdf"
output_directory = "/content material/check"
text_results=get_contents(pdf_path, output_directory)

We use this PDF that has each textual content and pictures or charts to check the multimodal RAG. 

We save the pictures extracted from the PDF utilizing the PyMuPDF library within the “check” listing. Within the subsequent steps, create embeddings of those photographs in order to have the ability to retrieve data from them in future based mostly on a person question.

Step 6. Chunking Textual content Information For RAG

Break up extracted textual content into smaller chunks utilizing LangChain’s RecursiveCharacterTextSplitter.

text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=2048,
        chunk_overlap=50,
        length_function=len,
        is_separator_regex=False,
        separators=[
            "nn",
            "n",
            " ",
            ".",
            ",",
            "u200b",  # Zero-width space
            "uff0c",  # Fullwidth comma
            "u3001",  # Ideographic comma
            "uff0e",  # Fullwidth full stop
            "u3002",  # Ideographic full stop
            "",
        ],
    )

doc_texts = text_splitter.create_documents(text_results)

Step 7: Loading Nomic Textual content Embedding Mannequin and Nomic Imaginative and prescient Embedding Mannequin

Load Nomic’s textual content and imaginative and prescient embedding fashions utilizing Hugging Face’s Transformers library.

from transformers import AutoTokenizer, AutoModel

# Load the tokenizer and mannequin
text_tokenizer = AutoTokenizer.from_pretrained("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
text_model = AutoModel.from_pretrained("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)

def text_embeddings(textual content):
    inputs = text_tokenizer(textual content, return_tensors="pt", padding=True, truncation=True)
    outputs = text_model(**inputs)
    embeddings = outputs.last_hidden_state.imply(dim=1)
    return embeddings[0].detach().numpy()
    
from transformers import AutoModel, AutoProcessor
from PIL import Picture
import torch
mannequin = AutoModel.from_pretrained("nomic-ai/nomic-embed-vision-v1.5", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("nomic-ai/nomic-embed-vision-v1.5")

Step 8: Producing Textual content and Picture Embeddings For Our Information

Convert textual content and pictures into vector embeddings for environment friendly retrieval.

#Textual content Embeddins
texts_embeded = [text_embeddings(document.page_content) for document in doc_texts]

#Picture Embeddings
image_embeddings = []
for img in image_files:
    strive:
        picture = Picture.open(os.path.be a part of(output_directory, img))
        inputs = processor(photographs=picture, return_tensors="pt")
        with torch.no_grad():
            outputs = mannequin(**inputs)
        embeddings = outputs.last_hidden_state
        if embeddings.measurement(0) > 0:  # Make sure the batch measurement is non-zero

            image_embedding = embeddings.imply(dim=1).squeeze().cpu().numpy()
            image_embeddings.append(image_embedding)
        else:
            print(f"No Embeddings For {img}")

    besides Exception as e:
        print(e)

#SIZE OF Textual content & Picture Embeddings
text_embeddings_size=len(texts_embeded[0])
image_embeddings_size=len(image_embeddings[0])

Step 9: Storing Textual content Embeddings in Qdrant 

Qdrant is an open-source vector database and search engine designed to effectively retailer, handle, and question high-dimensional vectors.We save our embeddings on this vector DB.

from qdrant_client import QdrantClient, fashions

shopper = QdrantClient(":reminiscence:")

if not shopper.collection_exists("text1"): #making a Assortment
 shopper.create_collection(
        collection_name ="text1",
      vectors_config=fashions.VectorParams(
        measurement=text_embeddings_size,  # Vector measurement is outlined by used mannequin
        distance=fashions.Distance.COSINE,
    ),
 )
 
 shopper.upload_points(
    collection_name="text1",
    factors=[
        models.PointStruct(
            id=str(uuid.uuid4()),
            vector=np.array(texts_embeded[idx]),
            payload={
                "metadata": doc.metadata,
                "content material": doc.page_content
            }
        )
        for idx, doc in enumerate(doc_texts)
    ]
)

Step 10: Storing Picture Embeddings in Qdrant 

Save picture embeddings in a separate Qdrant assortment for multimodal retrieval.

if not shopper.collection_exists("images1"):
    shopper.create_collection(
        collection_name="images1",
        vectors_config=fashions.VectorParams(
        measurement=image_embeddings_size,  # Vector measurement is outlined by used mannequin
        distance=fashions.Distance.COSINE,
    ),
  )
  
# Be certain that image_embeddings will not be empty
if len(image_embeddings) > 0:
    shopper.upload_points(
        collection_name="images1",
        factors=[
            models.PointStruct(
                id=str(uuid.uuid4()),  # unique id
                vector= np.array(image_embeddings[idx])  ,
                payload={"image_path": output_directory+'/'+str(image_files[idx])}  # Picture path as metadata
            )
            for idx in vary(len(image_embeddings))  
    )
else:
    print("No embeddings discovered")

Step 11: Making a MultiModal Retriever For Retrieving Photos and Textual content

Retrieve essentially the most related textual content and picture embeddings based mostly on a person question.

def MultiModalRetriever(question):

    question = text_embeddings(question)

    # Retrieve textual content hits
    text_hits = shopper.query_points(
        collection_name="text1",
        question=question,
        restrict=3,3
    ).factors
    # Retrieve picture hits
    Image_hits = shopper.query_points(
        collection_name="images1",
        question=question,
        restrict=5,
    ).factors

    return text_hits, Image_hits

Step 12: Making a MultiModal RAG utilizing LangChain

Use LangChain to course of retrieved textual content and pictures, producing context-aware responses utilizing GPT-4o.

def MultiModalRAG(context,photographs,user_query,mannequin):  
    # Helper perform to encode a picture as a base64 string
    def encode_image(image_path):
        if image_path:
            with open(image_path, "rb") as image_file:
                return base64.b64encode(image_file.learn()).decode()
        return None


    image_paths = photographs   
    #three photographs based mostly on retrived photographs
    img_base64 = encode_image(image_paths[0])        
    img_base641 = encode_image(image_paths[1])  
    img_base642 = encode_image(image_paths[2])  

    message = HumanMessage(
            content material=[
                {"type": "text", "text": "BASED ON RETRIEVED CONTEXT %s ONLY, ANSWER THE FOLLOWING QUERY %s. Context can be tables, texts or Images"%(context,user_query)},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{img_base64}"},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{img_base641}"},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{img_base642}"},
                },
            ],)

    mannequin = ChatOpenAI(mannequin=mannequin)    
    response = mannequin.invoke([message])
    return response.content material


def RAG(question):
  text_hits, Image_hits=MultiModalRetriever(question)

  retrieved_images=[i.payload['image_path'] for i in Image_hits]
  print(retrieved_images)
  reply=MultiModalRAG(text_hits,retrieved_images,question,"gpt-4o")
  return(reply)

Querying the Mannequin

Allow us to now question our multimodal RAG system with totally different queries to check its multimodal functionality, 

RAG("Income of Starbucks in billion {dollars} of Meals in 2020?")

Output:

'Primarily based on the chart exhibiting Starbucks' income by product for 2020, the income from
meals is roughly $3 billion.'

The response to this question is simply current within the following chart (Fig 4) within the PDF and never in any textual content. So our, multimodal RAG is ready to retrieve this data precisely.

RAG("Clarify what the Ansoff Matrix is for Starbucks.")

Output:

'The Ansoff Matrix is a strategic instrument that helps companies like Starbucks analyze
their development methods. For Starbucks, it may be damaged down as follows: 
1. **Market Penetration:** Starbu cks focuses on growing gross sales of present
merchandise in present markets. This contains enhancing the client expertise, leveraging their cell app for comfort, and selling present choices.
2. **Product Improvement:** Starbucks introduces new merchandise for present markets. Examples embody launching new beverage choices or introducing meatless breakfast
gadgets to adapt to altering shopper preferences.
3. **Market Improvement:** This includes Starbucks increasing into new geographical
places or market segments with present merchandise. It selects high-traffic
places and creates a constant model picture and retailer expertise to draw prospects.
4. **Diversification:** Introducing totally new merchandise to new markets. This might
contain Starbuck s exploring areas like providing alcoholic drinks to draw 
totally different buyer demographics. 
General, the Ansoff Matrix helps Starbucks strategically plan the best way to develop and adapt
in numerous market situations by specializing in both present or new merchandise and 
markets.

The response to this question as properly is simply current within the following diagram (Fig 3) within the PDF and never in any textual content. So our, multimodal RAG is ready to retrieve this data precisely.

RAG("International espresso consumption in 2017")

Output:

'The worldwide espresso consumption in 2017 was 161.37 million luggage.'

The response to this question as properly is simply current within the following chart (Fig 1) within the PDF and never in any textual content. So our, multimodal RAG is ready to retrieve this data precisely.

Conclusion

The combination of Nomic imaginative and prescient embeddings into multimodal RAG methods represents a serious leap in AI, permitting seamless interplay between visible and textual information for enhanced understanding and response technology. By overcoming limitations seen in fashions like CLIP, Nomic Embed Imaginative and prescient affords a unified embedding house, boosting efficiency on multimodal duties. This improvement paves the way in which for richer, extra context-aware person experiences in high-volume manufacturing environments.

Key Takeaways

• Multimodal Retrieval-Augmented Era (RAG) methods combine numerous information varieties, akin to textual content, photographs, audio, and video, enabling extra context-aware and nuanced outputs in comparison with conventional RAG methods centered on textual content alone.
• Nomic imaginative and prescient embeddings play a key position by unifying visible and textual information right into a single embedding house, enhancing the system’s means to retrieve and synthesize data throughout a number of modalities.
• The multimodal RAG system processes information by means of specialised ingestion, vector illustration, and storage methods, guaranteeing environment friendly retrieval and significant responses throughout various content material codecs.
• Whereas CLIP fashions excel in zero-shot capabilities, they wrestle with unimodal duties like semantic similarity. Nomic Embed Imaginative and prescient addresses this by aligning imaginative and prescient and textual content encoders, enhancing efficiency on a variety of duties.

Steadily Requested Questions

The media proven on this article shouldn’t be owned by Analytics Vidhya and is used on the Creator’s discretion.