A sensible information on utilizing cutting-edge optimization strategies to hurry up inference
Intro
Massive language fashions are extraordinarily power-hungry and require a major quantity of GPU assets to carry out nicely. Nonetheless, the transformer structure doesn’t take full benefit of the GPU.
GPUs, by design, can course of issues in parallel, however the transformer structure is auto-regressive. To ensure that the following token to get generated it has to take a look at the entire earlier tokens that got here earlier than it. Transformers don’t let you predict the following n tokens in parallel. Finally, this makes the technology section of LLMs fairly sluggish as every new token should be produced sequentially. Speculative decoding is a novel optimization approach that goals to resolve this difficulty.
There are just a few totally different strategies for speculative decoding. The approach described on this article makes use of the 2 mannequin strategy.
Speculative Decoding
Speculative decoding works by having two fashions, a big fundamental mannequin and a…