CS 16:198:671 - Recent Advances in Large Language Models

Description

This graduate-level course offers an in-depth exploration of the cutting-edge research and engineering principles behind large language models (LLMs). We will cover the foundational architectures that power modern AI, such as the Transformer, and delve into the complex systems required to train and serve these massive models at scale. Topics include advanced parallel computing paradigms (data, tensor, pipeline), specialized hardware like GPUs and accelerators, modern model architectures like Mixture-of-Experts (MoE), and the latest techniques in model optimization and alignment. A significant portion of the course is dedicated to student-led presentations and a final project, allowing students to engage directly with state-of-the-art literature and apply their knowledge to a practical problem.

Prerequisites

A strong background in machine learning, deep learning, and proficiency in Python programming are expected. Familiarity with deep learning frameworks like PyTorch or TensorFlow is highly recommended.

Efficient Attention

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022)
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (Dao, 2023)

Efficient Inference & Serving

• vLLM: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)
• Speculative Decoding (Leviathan et al., 2022)

Efficient Training & Fine-Tuning

• Mixed Precision Training (Micikevicius et al., 2017)
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)

Model Alignment

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023)

Alternative Architectures

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Gu & Dao, 2023)

Scientific Applications

• Highly accurate protein structure prediction with AlphaFold (Jumper et al., 2021)