Tips on how to Use Hybrid Seek for Higher LLM RAG Retrieval | by Dr. Leon Eversberg | Aug, 2024

Constructing a complicated native LLM RAG pipeline by combining dense embeddings with BM25

Code snippet from the hybrid search we’re going to implement on this article. Picture by writer

The fundamental Retrieval-Augmented Technology (RAG) pipeline makes use of an encoder mannequin to seek for related paperwork when given a question.

That is additionally known as semantic search as a result of the encoder transforms textual content right into a high-dimensional vector illustration (known as an embedding) wherein semantically related texts are shut collectively.

Earlier than we had Giant Language Fashions (LLMs) to create these vector embeddings, the BM25 algorithm was a extremely popular search algorithm. BM25 focuses on essential key phrases and appears for actual matches within the obtainable paperwork. This strategy is named key phrase search.

If you wish to take your RAG pipeline to the subsequent stage, you would possibly wish to strive hybrid search. Hybrid search combines the advantages of key phrase search and semantic search to enhance search high quality.

On this article, we’ll cowl the speculation and implement all three search approaches in Python.

Desk of Contents

· RAG Retrieval
Key phrase Search With BM25
Semantic Search With Dense Embeddings
Semantic Search or Hybrid Search?
Hybrid Search
Placing It All Collectively
·…