Learn how to change into an LLM Scientist and Engineer from scratch
The Massive Language Mannequin (LLM) course is a group of subjects and academic assets for folks to get into LLMs. It options two primary roadmaps:
- 🧑🔬 The LLM Scientist focuses on constructing the absolute best LLMs utilizing the most recent methods.
- 👷 The LLM Engineer focuses on creating LLM-based functions and deploying them.
For an interactive model of this course, I created an LLM assistant that may reply questions and take a look at your information in a personalised method on HuggingChat (advisable) or ChatGPT.
This part of the course focuses on studying how one can construct the absolute best LLMs utilizing the most recent methods.
An in-depth information of the Transformer structure shouldn’t be required, but it surely’s necessary to know the principle steps of recent LLMs: changing textual content into numbers by tokenization, processing these tokens by layers together with consideration mechanisms, and at last producing new textual content by varied sampling methods.
- Architectural Overview: Perceive the evolution from encoder-decoder Transformers to decoder-only architectures like GPT, which kind the premise of recent LLMs. Give attention to how these fashions course of and generate textual content at a excessive degree.
- Tokenization: Be taught the ideas of tokenization — how textual content is transformed into numerical representations that LLMs can course of. Discover completely different tokenization methods and their influence on mannequin efficiency and output high quality.
- Consideration mechanisms: Grasp the core ideas of consideration mechanisms, significantly self-attention and its variants. Perceive how these mechanisms allow LLMs to course of long-range dependencies and keep context all through sequences.
- Sampling methods: Discover varied textual content technology approaches and their tradeoffs. Examine deterministic strategies like grasping search and beam search with probabilistic approaches like temperature sampling and nucleus sampling.
📚 References:
- Visible intro to Transformers by 3Blue1Brown: Visible introduction to Transformers for full inexperienced persons.
- LLM Visualization by Brendan Bycroft: Interactive 3D visualization of LLM internals.
- nanoGPT by Andrej Karpathy: A 2h-long YouTube video to reimplement GPT from scratch (for programmers). He additionally made a video about tokenization.
- Consideration? Consideration! by Lilian Weng: Historic overview to introduce the necessity for consideration mechanisms.
- Decoding Methods in LLMs by Maxime Labonne: Present code and a visible introduction to the completely different decoding methods to generate textual content.