Scientific Paper Reading Club: Vector Quantization & KV Caching in LLMs

This is part of the scientific paper reading club. Next Wednesday, we are taking a deep dive into the mechanics of making Large Language Models faster and more memory-efficient! We will be exploring the fascinating world of Vector Quantization, specifically focusing on how it is used for KV caching in LLMs. What to expect: Our presenter, Anier Velasco Sotomayor, has recently gone deep into vector quantization for his work and will be sharing his insights with us. If you've ever wondered how models manage massive context windows without running out of RAM, this session is for you. We will be covering two highly relevant papers that are pushing the boundaries of vector compression: 1. "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate" (https://arxiv.org/abs/2504.19874) – A method from Google (presented at ICLR) that combines Polar Quant and QJL. 2. "RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search" (https://arxiv.org/abs/2405.12497) – Covering the 1-bit quantization method (QJL). Additionally, Anier will walk us through his own educational Python implementation of Turbo Quant (https://github.com/aniervs/vq-algos). He built this from scratch (without the complexity of GPU code) to unpack the underlying math and algorithmic details, making it a great learning resource for the group! Meeting Details: 🗣 Presenter: Anier Velasco Sotomayor 💬 Language of the meeting: English 🎟 Entrance donation: no donation, but please be responsible to university peroperty 📍 Location: Harbour Space Institute of Technology Looking forward to a great discussion this Wednesday!