Construct a Tokenizer for the Thai Language from Scratch | by Milan Tamang | Sep, 2024

The first process of the Tokenizer is to translate the uncooked enter texts (Thai in our case however will be in any overseas language) into numbers and go them to the mannequin’s transformers. The mannequin’s transformer then generates output as numbers. Once more, Tokenizer interprets these numbers again to texts which is comprehensible to finish customers. The excessive stage diagram beneath describes the circulation defined above.

Typically, many people are solely enthusiastic about studying how the mannequin’s transformer structure works beneath the hood. We regularly overlook studying some necessary parts equivalent to tokenizers intimately. Understanding how tokenizer works beneath the hood and having good management of its functionalities offers us good leverage to enhance our mannequin’s accuracy and efficiency.

Just like Tokenizer, a few of the most necessary parts of LLM implementation pipelines are Information preprocessing, Analysis, Guardrails/Safety, and Testing/Monitoring. I might extremely suggest you research extra particulars on these subjects. I noticed the significance of those parts solely after I used to be engaged on the precise implementation of my foundational multilingual mannequin ThaiLLM in manufacturing.

Why do you want a Thai tokenizer or another overseas language tokenizer?

• Suppose you might be utilizing generic English-based tokenizers to pre-train a multilingual massive language mannequin equivalent to Thai, Hindi, Indonesian, Arabic, Chinese language, and many others. In that case, your mannequin may unlikely give an acceptable output that makes good sense to your particular area or use circumstances. Therefore, constructing your individual tokenizer in your alternative of language definitely helps make your mannequin’s output way more coherent and comprehensible.
• Constructing your individual tokenizer additionally offers you full management over how complete and inclusive vocabulary you wish to construct. Throughout the consideration mechanism, due to complete vocabulary, the token can attend and be taught from extra tokens inside the restricted context size of the sequence. Therefore it makes studying extra coherent which finally helps in higher mannequin inference.

The excellent news is that after you end constructing Thai Tokenizer, you may simply construct a tokenizer in another language. All of the constructing steps are the identical besides that you just’ll have to coach on the dataset of your alternative of language.

Now that we’ve all the great cause to construct our personal tokenizer. Under are steps to constructing our tokenizer within the Thai language.

1. Construct our personal BPE algorithm
2. Prepare the tokenizer
3. Tokenizer encode and decode operate
4. Load and check the tokenizer

Step 1: Construct our personal BPE (Byte Pair Encoding) algorithm:

The BPE algorithm is utilized in many fashionable LLMs equivalent to Llama, GPT, and others to construct their tokenizer. We are able to select certainly one of these LLM tokenizers if our mannequin is predicated on the English language. Since we’re constructing the Thai Tokenizer, the most suitable choice is to create our personal BPE algorithm from scratch and use it to construct our tokenizer. Let’s first perceive how the BPE algorithm works with the assistance of the easy circulation diagram beneath after which we’ll begin constructing it accordingly.

The examples within the circulation diagram are proven in English to make it simpler to know.

Let’s write code to implement the BPE algorithm for our Thai Tokenizer.

# A easy observe instance to get familiarization with utf-8 encoding to transform strings to bytes. 
textual content = "How are you คุณเป็นอย่างไร"      # Textual content string in each English and Thai
text_bytes = textual content.encode("utf-8")
print(f"Textual content in byte: {text_bytes}")
text_list = listing(text_bytes)          # Converts textual content bytes to a listing of integer
print(f"Textual content listing in integer: {text_list}")

# As I do not wish to reinvent the wheel, I will likely be referencing many of the code block from Andrej Karpathy's GitHub (https://github.com/karpathy/minbpe?tab=readme-ov-file).
# Nonetheless, I will be modifying code blocks particular to constructing our Thai language tokenizer and likewise explaining the codes as a way to perceive how every code block works and make it straightforward while you implement code to your use case later.
# This module supplies entry to the Unicode Character Database (UCD) which defines character properties for all Unicode characters.
import unicodedata
# This operate returns a dictionary with consecutive pairs of integers and their counts within the given listing of integers.
def get_stats(ids, stats=None):
stats = {} if stats is None else stats
# zip operate permits to iterate consecutive gadgets from given two listing
for pair in zip(ids, ids[1:]): 
# If a pair already exists within the stats dictionary, add 1 to its worth else assign the worth as 0.
stats[pair] = stats.get(pair, 0) + 1
return stats
# As soon as we discover out the listing of consecutive pairs of integers, we'll then substitute these pairs with new integer tokens.
def merge(ids, pair, idx):
newids = []
i = 0
# As we'll be merging a pair of ids, therefore the minimal id within the listing needs to be 2 or extra.
whereas i < len(ids):
# If the present id and subsequent id(id+1) exist within the given pair, and the place of id is just not the final, then substitute the two consecutive id with the given index worth.
if ids[i] == pair[0] and that i < len(ids) - 1 and ids[i+1] == pair[1]:
newids.append(idx)
i += 2  # If the pair is matched, the following iteration begins after 2 positions within the listing.
else:
newids.append(ids[i])
i += 1  # For the reason that present id pair did not match, so begin iteration from the 1 place subsequent within the listing.
# Returns the Merged Ids listing
return newids
# This operate checks that utilizing 'unicodedata.class' which returns "C" as the primary letter if it's a management character and we'll have to interchange it readable character.
def replace_control_characters(s: str) -> str:
chars = []
for ch in s:
# If the character is just not distorted (which means the primary letter would not begin with "C"), then append the character to chars listing.
if unicodedata.class(ch)[0] != "C":
chars.append(ch) 
# If the character is distorted (which means the primary letter has the letter "C"), then substitute it with readable bytes and append to chars listing.
else:
chars.append(f"u{ord(ch):04x}") 
return "".be a part of(chars)
# Among the tokens equivalent to management characters like Escape Characters cannot be decoded into legitimate strings. 
# Therefore these have to be substitute with readable character equivalent to �
def render_token(t: bytes) -> str:    
s = t.decode('utf-8', errors='substitute')
s = replace_control_characters(s)
return s

The 2 features get_stats and merge outlined above within the code block are the implementation of the BPE algorithm for our Thai Tokenizer. Now that the algorithm is prepared. Let’s write code to coach our tokenizer.

Step 2: Prepare the tokenizer:

Coaching tokenizer entails producing a vocabulary which is a database of distinctive tokens (phrase and sub-words) together with a novel index quantity assigned to every token. We’ll be utilizing the Thai Wiki dataset from the Hugging Face to coach our Thai Tokenizer. Similar to coaching an LLM requires an enormous information, you’ll additionally require an excellent quantity of information to coach a tokenizer. You might additionally use the identical dataset to coach the LLM in addition to tokenizer although not obligatory. For a multilingual LLM, it’s advisable to make use of each the English and Thai datasets within the ratio of two:1 which is a typical method many practitioners observe.

Let’s start writing the coaching code.

# Import Common Expression
import regex as re    
# Create a Thai Tokenizer class.
class ThaiTokenizer():
def __init__(self):
# The byte pair needs to be performed inside the associated phrases or sentences that give a correct context. Pairing between unrelated phrases or sentences might give undesirable output.
# To stop this habits, we'll implement the LLama 3 common expression sample to make significant chunks of our textual content earlier than implementing the byte pair algorithm.
self.sample = r"(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^rnp{L}p{N}]?p{L}+|p{N}{1,3}| ?[^sp{L}p{N}]+[rn]*|s*[rn]+|s+(?!S)|s+"                
self.compiled_pattern = re.compile(self.sample)
# Particular tokens are used to offer coherence within the sequence whereas coaching.
# Particular tokens are assigned a novel index quantity and saved in vocabulary. 
self.special_tokens = end_header_id
# Initialize merges with empty dictionary
self.merges = {}
# Initialize the vocab dictionary by calling the operate _build_vocab which is outlined later on this class.
self.vocab = self._build_vocab()
# Tokenizer coaching operate
def practice(self, textual content, vocab_size):
# Be sure that the vocab measurement should be at the least 256 because the utf-8 encoding for the vary 0-255 are similar because the Ascii character.
assert vocab_size >= 256
# Complete variety of merges into the vocabulary.
num_merges = vocab_size - 256
# Step one is to ensure to separate the textual content up into textual content chunks utilizing the sample outlined above.
text_chunks = re.findall(self.compiled_pattern, textual content)
# Every text_chunks will likely be utf-8 encoded to bytes after which transformed into an integer listing.
ids = [list(ch.encode("utf-8")) for ch in text_chunks]
# Iteratively merge the most typical pairs to create new tokens
merges = {} # (int, int) -> int
vocab = {idx: bytes([idx]) for idx in vary(256)} # idx -> bytes
# Till the whole num_merges is reached, discover the widespread pair of consecutive id within the ids listing and begin merging them to create a brand new token
for i in vary(num_merges):
# Rely the variety of occasions each consecutive pair seems
stats = {}
for chunk_ids in ids:
# Passing in stats will replace it in place, including up counts
get_stats(chunk_ids, stats)
# Discover the pair with the very best depend
pair = max(stats, key=stats.get)
# Mint a brand new token: assign it the following accessible id
idx = 256 + i
# Substitute all occurrences of pair in ids with idx
ids = [merge(chunk_ids, pair, idx) for chunk_ids in ids]
# Save the merge
merges[pair] = idx
vocab[idx] = vocab[pair[0]] + vocab[pair[1]]
# Save class variables for use later throughout tokenizer encode and decode
self.merges = merges 
self.vocab = vocab   
# Perform to return a vocab dictionary combines with merges and particular tokens
def _build_vocab(self):
# The utf-8 encoding for the vary 0-255 are similar because the Ascii character. 
vocab = {idx: bytes([idx]) for idx in vary(256)}
# Iterate by means of merge dictionary and add into vocab dictionary
for (p0, p1), idx in self.merges.gadgets():
vocab[idx] = vocab[p0] + vocab[p1]
# Iterate by means of particular token dictionary and add into vocab dictionary
for particular, idx in self.special_tokens.gadgets():
vocab[idx] = particular.encode("utf-8")
return vocab
# After coaching is full, use the save operate to save lots of the mannequin file and vocab file.
# Mannequin file will likely be used to load the tokenizer mannequin for additional use in llm
# Vocab file is only for the aim of human verification
def save(self, file_prefix):        
# Writing to mannequin file 
model_file = file_prefix + ".mannequin"           # mannequin file identify
# Mannequin write begins
with open(model_file, 'w') as f:            
f.write("thai tokenizer v1.0n")          # write the tokenizer model
f.write(f"{self.sample}n")              # write the sample utilized in tokenizer            
f.write(f"{len(self.special_tokens)}n")  # write the size of particular tokens
# Write every particular token within the particular format like beneath
for tokens, idx in self.special_tokens.gadgets():
f.write(f"{tokens} {idx}n")
# Write solely the keys half from the merges dict
for idx1, idx2 in self.merges:
f.write(f"{idx1} {idx2}n")
# Writing to the vocab file
vocab_file = file_prefix + ".vocab"       # vocab file identify
# Change the place of keys and values of merge dict and retailer into inverted_merges 
inverted_merges = {idx: pair for pair, idx in self.merges.gadgets()}        
# Vocab write begins
with open(vocab_file, "w", encoding="utf-8") as f:
for idx, token in self.vocab.gadgets():
# render_token operate processes tokens and prevents distorted bytes by changing them with readable character
s = render_token(token)
# If the index of vocab is current in merge dict, then discover its baby index, convert their corresponding bytes in vocab dict and write the characters
if idx in inverted_merges:                    
idx0, idx1 = inverted_merges[idx]
s0 = render_token(self.vocab[idx0])
s1 = render_token(self.vocab[idx1])
f.write(f"[{s0}][{s1}] -> [{s}] {idx}n")
# If index of vocab is just not current in merge dict, simply write it is index and the corresponding string
else:                    
f.write(f"[{s}] {idx}n")
# Perform to load tokenizer mannequin. 
# This operate is invoked solely after the coaching is full and the tokenizer mannequin file is saved.
def load(self, model_file):
merges = {}             # Initialize merge and special_tokens with empty dict
special_tokens = {}     # Initialize special_tokens with empty dict
idx = 256               # Because the vary (0, 255) is already reserved in vocab. So the following index solely begins from 256 and onwards.
# Learn mannequin file
with open(model_file, 'r', encoding="utf-8") as f:
model = f.readline().strip()          # Learn the tokenizer model as outlined throughout mannequin file writing            
self.sample = f.readline().strip()     # Learn the sample utilized in tokenizer            
num_special = int(f.readline().strip()) # Learn the size of particular tokens
# Learn all of the particular tokens and retailer in special_tokens dict outlined earlier
for _ in vary(num_special):
particular, special_idx = f.readline().strip().cut up()
special_tokens[special] = int(special_idx)
# Learn all of the merge indexes from the file. Make it a key pair and retailer it in merge dictionary outlined earlier. 
# The worth of this key pair can be idx(256) as outlined above and carry on enhance by 1.            
for line in f:
idx1, idx2 = map(int, line.cut up())
merges[(idx1, idx2)] = idx
idx += 1
self.merges = merges                  
self.special_tokens = special_tokens  
# Create a remaining vocabulary dictionary by combining merge, special_token and vocab (0-255). _build_vocab operate helps to do exactly that.
self.vocab = self._build_vocab()

Step 3: Tokenizer encode and decode operate:

• Tokenizer Encode: The tokenizer encoding operate seems to be into vocabulary and interprets the given enter texts or prompts into the listing of integer IDs. These IDs are then fed into the transformer blocks.
• Tokenizer Decode: The tokenizer decoding operate seems to be into vocabulary and interprets the listing of IDs generated from the transformer’s classifier block into output texts.

Let’s check out the diagram beneath to have additional readability.

Let’s write code to implement the tokenizer’s encode and decode operate.

# Tokenizer encode operate takes textual content as a string and returns integer ids listing
def encode(self, textual content):      
# Outline a sample to determine particular token current within the textual content
special_pattern = "(" + "|".be a part of(re.escape(okay) for okay in self.special_tokens) + ")"        
# Break up particular token (if current) from the remainder of the textual content
special_chunks = re.cut up(special_pattern, textual content)        
# Initialize empty ids listing 
ids = []                                                
# Loop by means of every of components within the particular chunks listing.
for half in special_chunks:
# If the a part of the textual content is the particular token, get the idx of the half from the particular token dictionary and append it to the ids listing.
if half in self.special_tokens:                
ids.append(self.special_tokens[part])            
# If the a part of textual content is just not a particular token 
else:                
# Break up the textual content into a number of chunks utilizing the sample we have outlined earlier.
text_chunks = re.findall(self.compiled_pattern, textual content)
# All textual content chunks are encoded individually, then the outcomes are joined                
for chunk in text_chunks:
chunk_bytes = chunk.encode("utf-8")   # Encode textual content to bytes                    
chunk_ids = listing(chunk_bytes)         # Convert bytes to listing of integer  
whereas len(chunk_ids) >= 2:    # chunks ids listing should be at the least 2 id to kind a byte-pair
# Rely the variety of occasions each consecutive pair seems
stats = get_stats(chunk_ids)
# Some idx pair may be created with one other idx within the merge dictionary. Therefore we'll discover the pair with the bottom merge index to make sure we cowl all byte pairs within the merge dict.
pair = min(stats, key=lambda p: self.merges.get(p, float("inf")))
# Break the loop and return if the pair is just not current within the merges dictionary                        
if pair not in self.merges:
break 
# Discover the idx of the pair current within the merges dictionary
idx = self.merges[pair]
# Substitute the occurrences of pair in ids listing with this idx and proceed
chunk_ids = merge(chunk_ids, pair, idx)                    
ids.lengthen(chunk_ids)                
return ids
# Tokenizer decode operate takes a listing of integer ids and return strings
def decode(self, ids):
# Initialize empty byte listing
part_bytes = []
# Change the place of keys and values of special_tokens dict and retailer into inverse_special_tokens 
inverse_special_tokens = {v: okay for okay, v in self.special_tokens.gadgets()}
# Loop by means of idx within the ids listing
for idx in ids:
# If the idx is present in vocab dict, get the bytes of idx and append them into part_bytes listing
if idx in self.vocab:
part_bytes.append(self.vocab[idx])
# If the idx is present in inverse_special_tokens dict, get the token string of the corresponding idx, convert it to bytes utilizing utf-8 encode after which append it into part_bytes listing
elif idx in inverse_special_tokens:
part_bytes.append(inverse_special_tokens[idx].encode("utf-8"))
# If the idx is just not present in each vocab and particular token dict, throw an invalid error
else:
increase ValueError(f"invalid token id: {idx}")
# Be part of all the person bytes from the part_byte listing
text_bytes = b"".be a part of(part_bytes)
# Convert the bytes to textual content string utilizing utf-8 decode operate. Be sure that to make use of "errors=substitute" to interchange distorted characters with readable characters equivalent to �.
textual content = text_bytes.decode("utf-8", errors="substitute")
return textual content

Step 4: Load and check the tokenizer:

Lastly, right here comes the very best a part of this text. On this part, we’ll carry out two attention-grabbing duties.

• First, practice our tokenizer with the Thai Wiki Dataset from the Hugging Face. Now we have chosen a small dataset measurement (2.2 MB) to make coaching sooner. Nonetheless, for real-world implementation, it’s best to select a a lot bigger dataset for higher outcomes. After the coaching is full, we’ll save the mannequin.
• Second, we’ll load the saved tokenizer mannequin and carry out testing the tokenizer’s encode and decode operate.

Let’s dive in.

# Prepare the tokenizer
import time   # To caculate the period of coaching completion
# Load coaching uncooked textual content information (thai_wiki dataset) from huggingface. thai_wiki_small.textual content: https://github.com/tamangmilan/thai_tokenizer
texts = open("/content material/thai_wiki_small.txt", "r", encoding="utf-8").learn()
texts = texts.strip()
# Outline vocab measurement
vocab_size = 512
# Initialize a tokenizer mannequin class
tokenizer = ThaiTokenizer()
# Begin practice a tokenizer
start_time = time.time()
tokenizer.practice(texts, vocab_size)
end_time = time.time()
# Save tokenizer: you may change path and filename.
tokenizer.save("./fashions/thaitokenizer")
print(f"Complete time to finish tokenizer coaching: {end_time-start_time:.2f} seconds")
# Output: Complete time to finish tokenizer coaching: 186.11 seconds (3m 6s) [Note: Training duration will be longer if vocab_size is bigger and lesser for smaller vocab_size]

# Take a look at the tokenizer
# Initialize a tokenizer mannequin class
tokenizer = ThaiTokenizer()
# Load tokenizer mannequin. This mannequin was saved throughout coaching.
tokenizer.load("./fashions/thaitokenizer.mannequin")
# Invoke and confirm the tokenizer encode and decode operate for English Language
eng_texts = "When society advanced in several lands"
print(f"English Textual content: {eng_texts}")
encoded_ids = tokenizer.encode(eng_texts)
print(f"Encoded Ids: {encoded_ids}")
decoded_texts = tokenizer.decode(encoded_ids)
print(f"Decoded Texts: {decoded_texts}n")
# Invoke and confirm the tokenizer encode and decode operate for Thai Language
thai_texts = "เมื่อสังคมมีวิวัฒนาการขึ้นในดินแดนต่าง"
print(f"Thai Textual content: {thai_texts}")
thai_encoded_ids = tokenizer.encode(thai_texts)
print(f"Encoded Ids: {thai_encoded_ids}")
thai_decoded_texts = tokenizer.decode(thai_encoded_ids)
print(f"Decoded Texts: {thai_decoded_texts}")

Excellent. Our Thai Tokenizer can now efficiently and precisely encode and decode texts in each Thai and English languages.

Have you ever seen that the encoded IDs for English texts are longer than Thai encoded IDs? It’s because we’ve solely educated our tokenizer with the Thai dataset. Therefore the tokenizer is barely in a position to construct a complete vocabulary for the Thai language. Since we didn’t practice with an English dataset, the tokenizer has to encode proper from the character stage which ends up in longer encoded IDs. As I’ve talked about earlier than, for multilingual LLM, it’s best to practice each the English and Thai datasets with a ratio of two:1. This will provide you with balanced and high quality outcomes.

And that’s it! Now we have now efficiently created our personal Thai Tokenizer from scratch solely utilizing Python. And, I believe that was fairly cool. With this, you may simply construct a tokenizer for any overseas language. This will provide you with a variety of leverage whereas implementing your Multilingual LLM.

Thanks so much for studying!

Hyperlink to Google Colab pocket book

References

[1] Andrej Karpathy, Git Hub: Karpthy/minbpe