Leiden.dev • Reliability of Streaming Pipelines

October edition; Reliability of our Streaming Pipelines; How we took Flink CDC completeness from 92% to 99.78% by Abhishek Jain In data infrastructure, streaming pipelines capture every database change (inserts, updates, deletes) in real time to keep downstream systems in sync. We thought ours were 92% reliable, until we realized our measurement was double-counting some changes, hiding actual data loss. Using Global Transaction Identifiers (GTIDs), unique tags for each database transaction, we achieved 99.78% completeness. This talk covers the pitfalls of duplicate counting, how GTIDs exposed the gaps, and the steps we took to harden our Apache Flink pipelines. Doors open at 19:00 Talk starts at 19:30 📅 Thursday, 15 October 2026 🕒 19:00 - 22:00 📍 Café De Keyzer, Kaiserstraat 2-4, Leiden (city center) ❣️ Upstairs