Scaling Analytical Benchmarks to Production Complexity

Speaker: Jan Vincent Szlang, University of Cambridge
Series: Computer Laboratory Systems Research Group Seminar

Abstract:

Benchmarks drive analytical database research and engineering. They guide system design, shape optimization priorities, and provide shared baselines for comparing engines. Academia and industry have long relied on TPC‑DS as a proxy for production data‑warehousing challenges, and it captures many dominant analytical patterns well. However, recent workload studies from Snowflake, Amazon Redshift, and Tableau show that TPC‑DS underrepresents important production characteristics, such as string‑heavy processing, deep join graphs, sparse and skewed data, and large operator parameter scales.

To close these gaps, we introduce Prod‑DS, an extension of TPC‑DS grounded in recent production evidence. Prod‑DS allows scaling query complexity through configurable stressors on the existing schema and tooling. Using WorkloadLens, a coverage‑analysis tool, we show that Prod‑DS tracks published production distributions more closely than common analytical benchmarks. Evaluating three popular analytical engines, we find that production‑aligned complexity can increase runtime by up to ≈91 × compared with TPC‑DS, while a moderate ≈8 % of queries fail on modern SQL syntax that the engines do not fully support. Engines differ sharply in how they handle the added complexity, revealing qualitative performance and feasibility differences not visible under traditional benchmarks.

Bio:

Jan Vincent Szlang is a PhD student at the University of Cambridge, working with Prof Evangelia Kalyvianaki and Prof Martin Kleppmann. Before joining Cambridge, he worked on workload characterisation and query execution at Snowflake and Huawei. His PhD examines how real production workloads should inform the design of analytical systems.

Online meeting link:

Microsoft Teams