The important thing steps within the workflow lie in structuring the transcript in paragraphs (step 2) earlier than grouping the paragraphs into chapters from which a desk of contents is derived (step 4). Observe that these two steps could depend on completely different LLMs: A quick and low cost LLM corresponding to LLama 3 8B for the straightforward process of textual content modifying and paragraph identification, and a extra refined LLM corresponding to GPT-4o-mini for the era of the desk of contents. In between, TF-IDF is used so as to add again timestamp info to the structured paragraphs.
The remainder of the submit describes every step in additional element.
Try the accompanying Github repository and Colab pocket book to discover by yourself!
Allow us to use for example the primary lecture of the course ‘MIT 6.S191: Introduction to Deep Studying’ (IntroToDeepLearning.com) by Alexander Amini and Ava Amini (licensed underneath the MIT License).
Observe that chapters are already offered within the video description.
This supplies us with a baseline to qualitatively evaluate our chaptering later on this submit.
YouTube transcript API
For YouTube movies, an mechanically generated transcript is often made accessible by YouTube. A handy solution to retrieve that transcript is by calling the get_transcript technique of the Python youtube_transcript_api library. The tactic takes the YouTube video_id library as argument:
# https://www.youtube.com/watch?v=ErnWZxJovaM
video_id = "ErnWZxJovaM" # MIT Introduction to Deep Studying - 2024# Retrieve transcript with the youtube_transcript_api library
from youtube_transcript_api import YouTubeTranscriptApi
transcript = YouTubeTranscriptApi.get_transcript(video_id, languages=["en"])
This returns the transcript as an inventory of textual content and timestamp key-value pairs:
[{'text': '[Music]', 'begin': 1.17},
{'textual content': 'good afternoon everybody and welcome to', 'begin': 10.28},
{'textual content': 'MIT sus1 191 my title is Alexander amini', 'begin': 12.88},
{'textual content': "and I will be one in all your instructors for", 'begin': 16.84},
...]
The transcript is nevertheless poorly formatted: it lacks punctuation and accommodates typos (‘MIT sus1 191’ as a substitute of ‘MIT 6.S191′, or ‘amini’ as a substitute of ‘Amini’).
Speech-to-text with Whisper
Alternatively, a speech-to-text library can be utilized to deduce the transcript from a video or audio file. We advocate utilizing faster-whisper, which is a quick implementation of the state-of-the-art open-source whisper mannequin.
The fashions come in numerous measurement. Essentially the most correct is the ‘large-v3’, which is ready to transcribe about quarter-hour of audio per minute on a T4 GPU (accessible without cost on Google Colab).
from faster_whisper import WhisperModel# Load Whisper mannequin
whisper_model = WhisperModel("large-v3",
gadget="cuda" if torch.cuda.is_available() else "cpu",
compute_type="float16",
)
# Name the Whisper transcribe perform on the audio file
initial_prompt = "Use punctuation, like this."
segments, transcript_info = whisper_model.transcribe(audio_file, initial_prompt=initial_prompt, language="en")
The results of the transcription is offered as segments which will be simply transformed in an inventory of textual content and timestamps as with the youtube_transcript_api library.
Tip: Whisper could generally not embrace the punctuation. The initial_prompt argument can be utilized to nudge the mannequin so as to add punctuation by offering a small sentence containing punctuation.
Under is an excerpt of the transcription of the our video instance with whisper large-v3:
[{'start': 0.0, 'text': ' Good afternoon, everyone, and welcome to MIT Success 191.'},
{'start': 15.28, 'text': " My name is Alexander Amini, and I'll be one of your instructors for the course this year"},
{'start': 19.32, 'duration': 2.08, 'text': ' along with Ava.'}
...]
Observe that in comparison with the YouTube transcription, the punctuation is added. Some transcription errors nevertheless nonetheless stay (‘MIT Success 191’ as a substitute of ‘MIT 6.S191′).
As soon as a transcript is on the market, the second stage consists in modifying and structuring the transcript in paragraphs.
Transcript modifying refers to adjustments made to enhance readability. This includes, for instance, including punctuation whether it is lacking, correcting grammatical errors, eradicating verbal tics, and many others.
The structuring in paragraphs additionally improves readability, and additionnally serves as a preprocessing step for figuring out chapters in stage 4, since chapters can be shaped by grouping paragraphs collectively.
Paragraph modifying and structuring will be carried out in a single operation, utilizing an LLM. We illustrated under the anticipated results of this stage:
This process doesn’t require a really refined LLM because it principally consists in reformulating content material. On the time of writing this text, first rate outcomes may very well be obtained with for instance GPT-4o-mini or Llama 3 8B, and the next system immediate:
You’re a useful assistant.
Your process is to enhance the person enter’s readability: add punctuation if wanted and take away verbal tics, and construction the textual content in paragraphs separated with ‘nn’.
Preserve the wording as trustworthy as doable to the unique textual content.
Put your reply inside <reply></reply> tags.
We depend on OpenAI suitable chat completion API for LLM calling, with messages having the roles of both ‘system’, ‘person’ or ‘assistant’. The code under illustrates the instantiation of an LLM shopper with Groq, utilizing LLama 3 8B:
# Connect with Groq with a Groq API key
llm_client = Groq(api_key=api_key)
mannequin = "llama-8b-8192"# Extract textual content from transcript
transcript_text = ' '.be a part of([s['text'] for s in transcript])
# Name LLM
response = shopper.chat.completions.create(
messages=[
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": transcript_text
}
],
mannequin=mannequin,
temperature=0,
seed=42
)
Given a bit of uncooked ‘transcript_text’ as enter, this returns an edited piece of textual content inside <reply> tags:
response_content=response.decisions[0].message.content materialprint(response_content)
"""
<reply>
Good afternoon, everybody, and welcome to MIT 6.S191. My title is Alexander Amini, and I will be one in all your instructors for the course this 12 months, together with Ava. We're actually excited to welcome you to this unbelievable course.
It is a fast-paced and intense one-week course that we're about to undergo collectively. We'll be protecting the foundations of a quickly altering area, and a area that has been revolutionizing many areas of science, arithmetic, physics, and extra.
Over the previous decade, AI and deep studying have been quickly advancing and fixing issues that we did not assume have been solvable in our lifetimes. At this time, AI is fixing issues past human efficiency, and annually, this lecture is getting more durable and more durable to show as a result of it is presupposed to cowl the foundations of the sphere.
</reply>
"""
Allow us to then extract the edited textual content from the <reply> tags, divide it into paragraphs, and construction the outcomes as a JSON dictionary consisting of paragraph numbers and items of textual content:
import re
sample = re.compile(r'<reply>(.*?)</reply>', re.DOTALL)
response_content_edited = sample.findall(response_content)
paragraphs = response_content_edited.strip().cut up('nn')
paragraphs_dict = [{'paragraph_number': i, 'paragraph_text': paragraph} for i, paragraph in enumerate(paragraphs)print(paragraph_dict)
[{'paragraph_number': 0,
'paragraph_text': "Good afternoon, everyone, and welcome to MIT 6.S191. My name is Alexander Amini, and I'll be one of your instructors for the course this year, along with Ava. We're really excited to welcome you to this incredible course."},
{'paragraph_number': 1,
'paragraph_text': "This is a fast-paced and intense one-week course that we're about to go through together. We'll be covering the foundations of a rapidly changing field, and a field that has been revolutionizing many areas of science, mathematics, physics, and more."},
{'paragraph_number': 2,
'paragraph_text': "Over the past decade, AI and deep learning have been rapidly advancing and solving problems that we didn't think were solvable in our lifetimes. Today, AI is solving problems beyond human performance, and each year, this lecture is getting harder and harder to teach because it's supposed to cover the foundations of the field."}]
Observe that the enter shouldn’t be too lengthy because the LLM will in any other case ‘neglect’ a part of the textual content. For lengthy inputs, the transcript have to be cut up in chunks to enhance reliability. We seen that GPT-4o-mini handles effectively as much as 5000 characters, whereas Llama 3 8B can solely deal with as much as 1500 characters. The pocket book supplies the perform transcript_to_paragraphs which takes care of splitting the transcript in chunks.
The transcript is now structured as an inventory of edited paragraphs, however the timestamps have been misplaced within the course of.
The third stage consists in including again timestamps, by inferring which section within the uncooked transcript is the closest to every paragraph.
We rely for this process on the TF-IDF metric. TF-IDF stands for time period frequency–inverse doc frequency and is a similarity measure for evaluating two items of textual content. The measure works by computing the variety of comparable phrases, giving extra weight to phrases which seem much less often.
As a preprocessing step, we alter the transcript segments and paragraph beginnings in order that they include the identical variety of phrases. The textual content items needs to be lengthy sufficient in order that paragraph beginnings will be efficiently matched to a novel transcript section. We discover that utilizing 50 phrases works effectively in follow.
num_words = 50transcript_num_words = transform_text_segments(transcript, num_words=num_words)
paragraphs_start_text = [{"start": p['paragraph_number'], "textual content": p['paragraph_text']} for p in paragraphs]
paragraphs_num_words = transform_text_segments(paragraphs_start_text, num_words=num_words)
We then depend on the sklearn library and its TfidfVectorizer and cosine_similarity perform to run TF-IDF and compute similarities between every paragraph starting and transcript section. under is an instance of code for locating the most effective match index within the transcript segments for the primary paragraph.
from sklearn.feature_extraction.textual content import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity# Paragraph for which to search out the timestamp
paragraph_i = 0
# Create a TF-IDF vectorizer
vectorizer = TfidfVectorizer().fit_transform(transcript_num_words + paragraphs_num_words)
# Get the TF-IDF vectors for the transcript and the excerpt
vectors = vectorizer.toarray()
# Extract the TF-IDF vector for the paragraph
paragraph_vector = vectors[len(transcript_num_words) + paragraph_i]
# Calculate the cosine similarity between the paragraph vector and every transcript chunk
similarities = cosine_similarity(vectors[:len(transcript_num_words)], paragraph_vector.reshape(1, -1))
# Discover the index of probably the most comparable chunk
best_match_index = int(np.argmax(similarities))
We wrapped the method in a add_timestamps_to_paragraphs perform, which provides timestamps to paragraphs, along with the matched section index and textual content:
paragraphs = add_timestamps_to_paragraphs(transcript, paragraphs, num_words=50)#Instance of output for the primary paragraph:
print(paragraphs[0])
{'paragraph_number': 0,
'paragraph_text': "Good afternoon, everybody, and welcome to MIT 6.S191. My title is Alexander Amini, and I will be one in all your instructors for the course this 12 months, together with Ava. We're actually excited to welcome you to this unbelievable course.",
'matched_index': 1,
'matched_text': 'good afternoon everybody and welcome to',
'start_time': 10}
Within the instance above, the primary paragraph (numbered 0) is discovered to match the transcript section number one that begins at time 10 (in seconds).
The desk of content material is then discovered by grouping consecutive paragraphs into chapters and figuring out significant chapter titles. The duty is usually carried out by an LLM, which is instructed to remodel an enter consisting in an inventory of JSON paragraphs into an output consisting in an inventory of JSON chapter titles with the beginning paragraph numbers:
system_prompt_paragraphs_to_toc = """You're a useful assistant.
You might be given a transcript of a course in JSON format as an inventory of paragraphs, every containing 'paragraph_number' and 'paragraph_text' keys.
Your process is to group consecutive paragraphs in chapters for the course and determine significant chapter titles.
Listed below are the steps to observe:
1. Learn the transcript fastidiously to know its common construction and the principle matters lined.
2. Search for clues {that a} new chapter is about to begin. This may very well be a change of subject, a change of time or setting, the introduction of latest themes or matters, or the speaker's specific point out of a brand new half.
3. For every chapter, preserve monitor of the paragraph quantity that begins the chapter and determine a significant chapter title.
4. Chapters ought to ideally be equally spaced all through the transcript, and talk about a particular subject.
Format your lead to JSON, with an inventory dictionaries for chapters, with 'start_paragraph_number':integer and 'title':string as key:worth.
Instance:
{"chapters":
[{"start_paragraph_number": 0, "title": "Introduction"},
{"start_paragraph_number": 10, "title": "Chapter 1"}
]
}
"""
An vital ingredient is to particularly ask for a JSON output, which will increase the possibilities to get a accurately formatted JSON output that may later be loaded again in Python.
GPT-4o-mini is used for this process, as it’s more cost effective than OpenAI’s GPT-4o and usually supplies good outcomes. The directions are offered via the ‘system’ position, and paragraphs are offered in JSON format via the ‘person’ position.
# Connect with OpenAI with an OpenAI API key
llm_client_get_toc = OpenAI(api_key=api_key)
model_get_toc = "gpt-4o-mini-2024-07-18"# Dump JSON paragraphs as textual content
paragraphs_number_text = [{'paragraph_number': p['paragraph_number'], 'paragraph_text': p['paragraph_text']} for p in paragraphs]
paragraphs_json_dump = json.dumps(paragraphs_number_text)
# Name LLM
response = client_get_toc.chat.completions.create(
messages=[
{
"role": "system",
"content": system_prompt_paragraphs_to_toc
},
{
"role": "user",
"content": paragraphs_json_dump
}
],
mannequin=model_get_toc,
temperature=0,
seed=42
)
Et voilà! The decision returns the record of chapter titles along with the beginning paragraph quantity in JSON format:
print(response){
"chapters": [
{
"start_paragraph_number": 0,
"title": "Introduction to the Course"
},
{
"start_paragraph_number": 17,
"title": "Foundations of Intelligence and Deep Learning"
},
{
"start_paragraph_number": 24,
"title": "Course Structure and Expectations"
}
....
]
}
As in step 2, the LLM could battle with lengthy inputs and dismiss a part of the enter. The answer consists once more in splitting the enter into chunks, which is carried out within the pocket book with the paragraphs_to_toc perform and the chunk_size parameter.
This final stage combines the paragraphs and the desk of content material to create a structured JSON file with chapters, an instance of which is offered within the accompanying Github repository.
We illustrate under the ensuing chaptering (proper), in comparison with the baseline chaptering that was accessible from the YouTube description (left):