讲座标题（中文）：可扩展线性RNN的进展：DeltaNet及其变体
讲座摘要：Linear recurrent models are gaining significant attention for their efficiency in large-scale training compared to nonlinear counterparts. Notable examples include Mamba, RWKV, GLA, and xLSTM. In this talk, I will introduce DeltaNet, a linear RNN that is strictly more expressive while retaining hardware-efficient training properties. I will motivate DeltaNet from an in-context learning perspective and outline strategies for scaling its training effectively. The talk will also explore DeltaNet's connections to recent advances such as TTT and Titans, along with emerging extensions such as Gated DeltaNet, RWKV7, DeltaProduct, and Mixture of Memory (MoM). 线性循环模型因其在大规模训练中的高效性，正在受到越来越多的关注，相较于非线性模型尤为如此。代表性方法包括 Mamba、RWKV、GLA 和 xLSTM。在本次报告中，我将介绍 DeltaNet——一种线性 RNN，其在保持硬件友好训练特性的同时，拥有更强的表达能力。我将从 in-context learning 的视角出发，阐述 DeltaNet 的动机，并介绍其可扩展训练的策略。报告还将探讨 DeltaNet 与近期进展（如 TTT 和 Titans）之间的联系，以及其衍生扩展方法，如 Gated DeltaNet、RWKV7、DeltaProduct 、 Mixture of Memory (MoM).
讲者信息：杨松琳是麻省理工学院计算机科学与人工智能实验室（MIT CSAIL）的一名二年级博士生，导师是 Prof. Yoon Kim。她的研究方向是面向硬件优化的高效序列建模算法，尤其专注于线性注意力模型。她是 Flash Linear Attention 库的主要贡献者，也是 “从算法视角推进序列建模前沿（ASAP）” 研讨会的发起人之一：https://asap-seminar.github.io/
相关工作：
Linear Transformers Are Secretly Fast Weight Programmers
https://arxiv.org/abs/2102.11174
Parallelizing Linear Transformers with the Delta Rule over Sequence Length
https://arxiv.org/abs/2406.06484
Gated Delta Networks: Improving Mamba2 with Delta Rule
https://arxiv.org/abs/2412.06464
RWKV-7 "Goose" with Expressive Dynamic State Evolution
https://arxiv.org/abs/2503.14456
Longhorn: State Space Models are Amortized Online Learners
https://arxiv.org/abs/2407.14207
Learning to (Learn at Test Time): RNNs with Expressive Hidden States
https://arxiv.org/abs/2407.04620
DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
https://arxiv.org/abs/2502.10297
MoM: Linear Sequence Modeling with Mixture-of-Memories
https://arxiv.org/abs/2502.13685
