This text explains the way to use an LLM (Massive Language Mannequin) to carry out the chunking of a doc primarily based on idea of “concept”.
I take advantage of OpenAI’s gpt-4o mannequin for this instance, however the identical method will be utilized with every other LLM, resembling these from Hugging Face, Mistral, and others.
Everybody can entry this article at no cost.
Concerns on Doc Chunking
In cognitive psychology, a bit represents a “unit of data.”
This idea will be utilized to computing as properly: utilizing an LLM, we are able to analyze a doc and produce a set of chunks, sometimes of variable size, with every chunk expressing an entire “concept.”
Which means the system divides a doc into “items of textual content” such that every expresses a unified idea, with out mixing completely different concepts in the identical chunk.
The objective is to create a data base composed of unbiased parts that may be associated to at least one one other with out overlapping completely different ideas throughout the identical chunk.
In fact, throughout evaluation and division, there could also be a number of chunks expressing the identical concept if that concept is repeated in numerous sections or expressed in another way throughout the identical doc.
Getting Began
Step one is figuring out a doc that shall be a part of our data base.
That is sometimes a PDF or Phrase doc, learn both web page by web page or by paragraphs and transformed into textual content.
For simplicity, let’s assume we have already got a listing of textual content paragraphs like the next, extracted from Across the World in Eighty Days:
paperwork = [
"""On October 2, 1872, Phileas Fogg, an English gentleman, left London for an extraordinary journey.
He had wagered that he could circumnavigate the globe in just eighty days.
Fogg was a man of strict habits and a very methodical life; everything was planned down to the smallest detail, and nothing was left to chance.
He departed London on a train to Dover, then crossed the Channel by ship. His journey took him through many countries,
including France, India, Japan, and America. At each stop, he encountered various people and faced countless adventures, but his determination never wavered.""","""However, time was his enemy, and any delay risked losing the bet. With the help of his faithful servant Passepartout, Fogg had to face
unexpected obstacles and dangerous situations.""",
"""Yet, each time, his cunning and indomitable spirit guided him to victory, while the world watched in disbelief.""",
"""With one final effort, Fogg and Passepartout reached London just in time to prove that they had completed their journey in less than eighty days.
This extraordinary adventurer not only won the bet but also discovered that the true treasure was the friendship and experiences he had accumulated along the way."""
]
Let’s additionally assume we’re utilizing an LLM that accepts a restricted variety of tokens for enter and output, which we’ll name input_token_nr and output_token_nr.
For this instance, we’ll set input_token_nr = 300 and output_token_nr = 250.
Which means for profitable splitting, the variety of tokens for each the immediate and the doc to be analyzed should be lower than 300, whereas the end result produced by the LLM should eat not more than 250 tokens.
Utilizing the tokenizer offered by OpenAI we see that our data base paperwork consists of 254 tokens.
Due to this fact, analyzing all the doc directly isn’t attainable, as regardless that the enter will be processed in a single name, it might’t match within the output.
So, as a preparatory step, we have to divide the unique doc into blocks no bigger than 250 tokens.
These blocks will then be handed to the LLM, which is able to additional cut up them into chunks.
To be cautious, let’s set the utmost block dimension to 200 tokens.
Producing Blocks
The method of producing blocks is as follows:
- Think about the primary paragraph within the data base (KB), decide the variety of tokens it requires, and if it’s lower than 200, it turns into the primary aspect of the block.
- Analyze the scale of the following paragraph, and if the mixed dimension with the present block is lower than 200 tokens, add it to the block and proceed with the remaining paragraphs.
- A block reaches its most dimension when trying so as to add one other paragraph causes the block dimension to exceed the restrict.
- Repeat from the 1st step till all paragraphs have been processed.
The blocks technology course of assumes, for simplicity, that every paragraph is smaller than the utmost allowed dimension (in any other case, the paragraph itself should be cut up into smaller parts).
To carry out this job, we use the llm_chunkizer.split_document_into_blocks perform from the LLMChunkizerLib/chunkizer.py library, which will be discovered within the following repository — LLMChunkizer.
Visually, the end result appears like Determine 1.
When producing blocks, the one rule to comply with is to not exceed the utmost allowed dimension.
No evaluation or assumptions are made concerning the which means of the textual content.
Producing Chunks
The subsequent step is to separate the block into chunks that every categorical the identical concept.
For this job, we use the llm_chunkizer.chunk_text_with_llm perform from the LLMChunkizerLib/chunkizer.py library, additionally present in the identical repository.
The end result will be seen in Determine 2.
This course of works linearly, permitting the LLM to freely determine the way to type the chunks.
Dealing with the Overlap Between Two Blocks
As beforehand talked about, throughout block splitting, solely the size restrict is taken into account, with no regard for whether or not adjoining paragraphs expressing the identical concept are cut up throughout completely different blocks.
That is evident in Determine 1, the place the idea “bla bla bla” (representing a unified concept) is cut up between two adjoining blocks.
As you may see In Determine 2, the chunkizer processes just one block at a time, which means the LLM can’t correlate this data with the next block (it doesn’t even know a subsequent block exists), and thus, locations it within the final cut up chunk.
This drawback happens regularly throughout ingestion, notably when importing an extended doc whose textual content can’t all match inside a single LLM immediate.
To handle it, llm_chunkizer.chunk_text_with_llm works as proven in Determine 3:
- The final chunk (or the final N chunks) produced from the earlier block is faraway from the “legitimate” chunks record, and its content material is added to the following block to be cut up.
- The New Block2 is handed to the chunking perform once more.
As proven in Determine 3, the content material of chunk M is cut up extra successfully into two chunks, conserving the idea “bla bla bla” collectively
The concept behind this resolution is that the final N chunks of the earlier block characterize unbiased concepts, not simply unrelated paragraphs.
Due to this fact, including them to the brand new block permits the LLM to generate comparable chunks whereas additionally creating a brand new chunk that unites paragraphs that have been beforehand cut up with out regard for his or her which means.
Results of Chunking
On the finish, the system produces the next 6 chunks:
0: On October 2, 1872, Phileas Fogg, an English gentleman, left London for a unprecedented journey. He had wagered that he might circumnavigate the globe in simply eighty days. Fogg was a person of strict habits and a really methodical life; every thing was deliberate right down to the smallest element, and nothing was left to likelihood.
1: He departed London on a prepare to Dover, then crossed the Channel by ship. His journey took him by many nations, together with France, India, Japan, and America. At every cease, he encountered numerous individuals and confronted numerous adventures, however his dedication by no means wavered.
2: Nevertheless, time was his enemy, and any delay risked shedding the wager. With the assistance of his devoted servant Passepartout, Fogg needed to face surprising obstacles and harmful conditions.
3: But, every time, his crafty and indomitable spirit guided him to victory, whereas the world watched in disbelief.
4: With one closing effort, Fogg and Passepartout reached London simply in time to show that they'd accomplished their journey in lower than eighty days.
5: This extraordinary adventurer not solely gained the wager but additionally found that the true treasure was the friendship and experiences he had gathered alongside the best way.
Concerns on Block Dimension
Let’s see what occurs when the unique doc is cut up into bigger blocks with a most dimension of 1000 tokens.
With bigger block sizes, the system generates 4 chunks as a substitute of 6.
This habits is anticipated as a result of the LLM might analyzed a bigger portion of content material directly and was ready to make use of extra textual content to characterize a single idea.
Listed here are the chunks on this case:
0: On October 2, 1872, Phileas Fogg, an English gentleman, left London for a unprecedented journey. He had wagered that he might circumnavigate the globe in simply eighty days. Fogg was a person of strict habits and a really methodical life; every thing was deliberate right down to the smallest element, and nothing was left to likelihood.
1: He departed London on a prepare to Dover, then crossed the Channel by ship. His journey took him by many nations, together with France, India, Japan, and America. At every cease, he encountered numerous individuals and confronted numerous adventures, however his dedication by no means wavered.
2: Nevertheless, time was his enemy, and any delay risked shedding the wager. With the assistance of his devoted servant Passepartout, Fogg needed to face surprising obstacles and harmful conditions. But, every time, his crafty and indomitable spirit guided him to victory, whereas the world watched in disbelief.
3: With one closing effort, Fogg and Passepartout reached London simply in time to show that they'd accomplished their journey in lower than eighty days. This extraordinary adventurer not solely gained the wager but additionally found that the true treasure was the friendship and experiences he had gathered alongside the best way.
Conclusions
It’s necessary to try a number of chunking runs, various the block dimension handed to the chunkizer every time.
After every try, the outcomes must be reviewed to find out which method most closely fits the specified final result.
Coming Up
Within the subsequent article, I’ll present the way to use an LLM to retrieve chunks — LLMRetriever .
You may discover all of the code and extra instance in my repository — LLMChunkizer.