Data Science Distinguished Lecture: Agostino Capponi - Computer Science, Columbia University

This talk is co-sponsored with the ORIE Colloquium and the Center for Data Science for Enterprise & Society.

TALK: Data-Driven Learning of Latent Dynamics

Abstract: High-dimensional time series often exhibit complex dependencies that can be explained by a few latent factors. Uncovering these factors and modeling their dynamics is a fundamental challenge in data science, with applications ranging from signal processing and forecasting to asset pricing and financial risk management.

We introduce a data-driven framework for modeling and predicting a multivariate response y(t) from a high-dimensional covariate process x(t), allowing for nonlinear dependencies without imposing a parametric model on the covariate dynamics. We leverage diffusion maps to uncover low-dimensional representations of the covariate process that retain explanatory power for the response time series (y(t)). We establish concentration inequalities that provide convergence guarantees for time averages and show that, under mild assumptions, the resulting diffusion map embeddings exhibit approximately linear dynamics. This property enables us to develop a geometric Kalman filter that operates in the learned embedding space to estimate latent states and predict the response variable.

We demonstrate the effectiveness of our framework in financial stress testing. Using historical data spanning three major financial crises, we show that our data-driven approach reduces the mean absolute error of scenario-based portfolio return predictions by up to 52% and 57%, relative to standard scenario analysis and dynamic principal component analysis, respectively.

Beyond financial stress testing, we highlight the broader applicability of our framework to problems such as nowcasting, inverse problems, and data-driven portfolio selection.

Bio: Agostino Capponi is a Professor in the Department of Industrial Engineering and Operations Research at Columbia University, where he also holds a courtesy appointment at Columbia Business School. He is a member of Columbia's Data Science Institute and the founding Director of the Center for Digital Finance and Technologies.

His research lies at the intersection of financial engineering, artificial intelligence, and economics, with particular interests in financial technology, machine learning in finance, market microstructure, and economic networks. His research has received funding from major government agencies, including the NFS, DARPA, the Department of Energy (DOE), as well as leading industry organizations such as IBM, JPMorgan, Ripple, and the Ethereum Foundation. His contributions have been recognized with numerous honors, including the NSF CAREER Award, the JPMorgan AI Research Faculty Award, the University Blockchain Research Innovator Award, and the Presidential Early Career Award for Scientists and Engineers (PECASE). His research has also received media coverage from Bloomberg, the Financial Times, Vox, and Politico.

He previously served as Department Editor for Finance at Management Science and currently serves as an Area Editor of Operations Research and Co-Editor of Mathematics and Financial Economics. He has also held editorial appointments at several leading journals, including Review of Finance, Operations Research Letters, Stochastic Systems, Stochastic Models, SIAM Journal on Financial Mathematics, Mathematical Finance, and Finance and Stochastics.

Agostino is President-Elect of the Bachelier Finance Society. He is a former Chair of the SIAM Activity Group on Financial Mathematics and Engineering and of the INFORMS Finance Section. He is also co-editor of the book Machine Learning and Data Sciences for Financial Markets: A Guide to Contemporary Practices, published by Cambridge University Press in 2023.

Capponi advises government agencies on digital assets, and financial markets. His advisory and consulting engagements have included work with the Commodity Futures Trading Commission (CFTC) and the Securities and Exchange Commission (SEC).