Leveraging GPT-3.5 and unstructured APIs for translations
This weblog submit particulars how I utilised GPT to translate the non-public memoir of a household pal, making it accessible to a broader viewers. Particularly, I employed GPT-3.5 for translation and Unstructured’s APIs for environment friendly content material extraction and formatting.
The memoir, a heartfelt account by my household pal Carmen Rosa, chronicles her upbringing in Bolivia and her romantic journey in Paris with an Iranian man through the vibrant Seventies. Initially written in Spanish, we aimed to protect the essence of her narrative whereas increasing its attain to English-speaking readers via the applying of LLM applied sciences.
Beneath you’ll be able to learn the interpretation course of in additional element or you’ll be able to entry right here the Colab Pocket book.
I adopted the subsequent steps for the interpretation of the guide:
- Import E book Knowledge: I imported the guide from a Docx doc utilizing the Unstructured API and divided it into chapters and paragraphs.
- Translation Approach: I translated every chapter utilizing GPT-3.5. For every paragraph, I supplied the most recent three translated sentences (if accessible) from the identical chapter. This method served two functions:
- Type Consistency: Sustaining a constant type all through the interpretation by offering context from earlier translations.
- Token Restrict: Limiting the variety of tokens processed directly to keep away from exceeding the mannequin’s context restrict.
3. Exporting translation as Docx: I used Unstructured’s API as soon as once more to avoid wasting the translated content material in Docx format.
1. Libraries
We’ ll begin with the set up and import of the mandatory libraries.
pip set up --upgrade openai
pip set up python-dotenv
pip set up unstructured
pip set up python-docx
import openai# Unstructured
from unstructured.partition.docx import partition_docx
from unstructured.cleaners.core import group_broken_paragraphs
# Knowledge and different libraries
import pandas as pd
import re
from typing import Listing, Dict
import os
from dotenv import load_dotenv
2. Connecting to OpenAI’s API
The code beneath units up the OpenAI API key to be used in a Python undertaking. It is advisable save your API key in an .env file.
import openai# Specify the trail to the .env file
dotenv_path = '/content material/.env'
_ = load_dotenv(dotenv_path) # learn native .env file
openai.api_key = os.environ['OPENAI_API_KEY']
3. Loading the guide
The code permits us to import the guide in Docx format and divide it into particular person paragraphs.
parts = partition_docx(
filename="/content material/libro.docx",
paragraph_grouper=group_broken_paragraphs
)
The code beneath returns the paragraph within the tenth index of parts.
print(parts[10])# Returns: Destino sorprendente, es el título que la autora le puso ...
4. Group guide into titles and chapters
The subsequent step includes creating an inventory of chapters. Every chapter will likely be represented as a dictionary containing a title and an inventory of paragraphs. This construction simplifies the method of translating every chapter and paragraph individually. Right here’s an instance of this format:
[
{"title": title 1, "content": [paragraph 1, paragraph 2, ..., paragraph n]},
{"title": title 2, "content material": [paragraph 1, paragraph 2, ..., paragraph n]},
...
{"title": title n, "content material": [paragraph 1, paragraph 2, ..., paragraph n]},
]
To attain this, we’ll create a operate known as group_by_chapter. Listed below are the important thing steps concerned:
- Extract Related Data: We are able to get every narrative textual content and title by calling
factor.class. These are the one classes we’re all in favour of translating at this level. - Establish Narrative Titles: We recognise that some titles must be a part of the narrative textual content. To account for this, we assume that italicised titles belong to the narrative paragraph.
def group_by_chapter(parts: Listing) -> Listing[Dict]:
chapters = []
current_title = Nonefor factor in parts:
text_style = factor.metadata.emphasized_text_tags # checks whether it is 'b' or 'i' and returns listing
unique_text_style = listing(set(text_style)) if text_style will not be None else None
# we think about a component a title if it's a title class and the type is daring
is_title = (factor.class == "Title") & (unique_text_style == ['b'])
# we think about a component a story content material if it's a narrative textual content class or
# if it's a title class, however it's italic or italic and daring
is_narrative = (factor.class == "NarrativeText") | (
((factor.class == "Title") & (unique_text_style is None)) |
((factor.class == "Title") & (unique_text_style == ['i'])) |
((factor.class == "Title") & (unique_text_style == ['b', 'i']))
)
# for brand new titles
if is_title:
print(f"Including title {factor.textual content}")
# Add earlier chapter when a brand new one is available in, except present title is None
if current_title will not be None:
chapters.append(current_chapter)
current_title = factor.textual content
current_chapter = {"title": current_title, "content material": []}
elif is_narrative:
print(f"Including Narrative {factor.textual content}")
current_chapter["content"].append(factor.textual content)
else:
print(f'### No must convert. Factor sort: {factor.class}')
return chapters
Within the instance beneath, we are able to see an instance:
book_chapters[2] # Returns
{'title': 'Proemio',
'content material': [
'La autobiografía es considerada ...',
'Dentro de las artes literarias, ...',
'Se encuentra más próxima a los, ...',
]
}
5. E book translation
To translate the guide, we comply with these steps:
- Translate Chapter Titles: We translate the title of every chapter.
- Translate Paragraphs: We translate every paragraph, offering the mannequin with the most recent three translated sentences as context.
- Save Translations: We save each the translated titles and content material.
The operate beneath automates this course of.
def translate_book(book_chapters: Listing[Dict]) -> Dict:
translated_book = []
for chapter in book_chapters:
print(f"Translating following chapter: {chapter['title']}.")
translated_title = translate_title(chapter['title'])
translated_chapter_content = translate_chapter(chapter['content'])
translated_book.append({
"title": translated_title,
"content material": translated_chapter_content
})
return translated_book
For the title, we ask GPT a easy translation as follows:
def translate_title(title: str) -> str:
response = consumer.chat.completions.create(
mannequin="gpt-3.5-turbo",
messages= [{
"role": "system",
"content": f"Translate the following book title into English:n{title}"
}]
)
return response.selections[0].message.content material
To translate a single chapter, we offer the mannequin with the corresponding paragraphs. We instruct the mannequin as follows:
- Establish the position: We inform the mannequin that it’s a useful translator for a guide.
- Present context: We share the most recent three translated sentences from the chapter.
- Request translation: We ask the mannequin to translate the subsequent paragraph.
Throughout this course of, the operate combines all translated paragraphs right into a single string.
# Perform to translate a chapter utilizing OpenAI API
def translate_chapter(chapter_paragraphs: Listing[str]) -> str:
translated_content = ""for i, paragraph in enumerate(chapter_paragraphs):
print(f"Translating paragraph {i + 1} out of {len(chapter_paragraphs)}")
# Builds the message dynamically based mostly on whether or not there's earlier translated content material
messages = [{
"role": "system",
"content": "You are a helpful translator for a book."
}]
if translated_content:
latest_content = get_last_three_sentences(translated_content)
messages.append(
{
"position": "system",
"content material": f"That is the most recent textual content from the guide that you have translated from Spanish into English:n{latest_content}"
}
)
# Provides the consumer message for the present paragraph
messages.append(
{
"position": "consumer",
"content material": f"Translate the next textual content from the guide into English:n{paragraph}"
}
)
# Calls the API
response = consumer.chat.completions.create(
mannequin="gpt-3.5-turbo",
messages=messages
)
# Extracts the translated content material and appends it
paragraph_translation = response.selections[0].message.content material
translated_content += paragraph_translation + 'nn'
return translated_content
Lastly, beneath we are able to see the supporting operate to get the most recent three sentences.
def get_last_three_sentences(paragraph: str) -> str:
# Use regex to separate the textual content into sentences
sentences = re.break up(r'(?<!w.w.)(?<![A-Z][a-z].)(?<=.|?)s', paragraph)# Get the final three sentences (or fewer if the paragraph has lower than 3 sentences)
last_three = sentences[-3:]
# Be part of the sentences right into a single string
return ' '.be part of(last_three)
6. E book export
Lastly, we cross the dictionary of chapters to a operate that provides every title as a heading and every content material as a paragraph. After every paragraph, a web page break is added to separate the chapters. The ensuing doc is then saved regionally as a Docx file.
from docx import Docdef create_docx_from_chapters(chapters: Dict, output_filename: str) -> None:
doc = Doc()
for chapter in chapters:
# Add chapter title as Heading 1
doc.add_heading(chapter['title'], degree=1)
# Add chapter content material as regular textual content
doc.add_paragraph(chapter['content'])
# Add a web page break after every chapter
doc.add_page_break()
# Save the doc
doc.save(output_filename)
Whereas utilizing GPT and APIs for translation is quick and environment friendly, there are key limitations in comparison with human translation:
- Pronoun and Reference Errors: GPT did misread pronouns or references in few circumstances, doubtlessly attributing actions or statements to the fallacious particular person within the narrative. A human translator can higher resolve such ambiguities.
- Cultural Context: GPT missed refined cultural references and idioms {that a} human translator may interpret extra precisely. On this case, a number of slang phrases distinctive to Santa Cruz, Bolivia, had been retained within the unique language with out further context or rationalization.
Combining AI with human overview can stability velocity and high quality, guaranteeing translations are each correct and genuine.
This undertaking demonstrates an method to translating a guide utilizing a mixture of GPT-3 and Unstructured APIs. By automating the interpretation course of, we considerably diminished the handbook effort required. Whereas the preliminary translation output might require some minor human revisions to refine the nuances and make sure the highest high quality, this method serves as a robust basis for environment friendly and efficient guide translation
When you’ve got any suggestions or solutions on how you can enhance this course of or the standard of the translations, please be at liberty to share them within the feedback beneath.