Conversation Modelling: The Curse of Multi-Modality

Speaker: Nora Raud, Aalto University, Finland
Series: CUED Speech Group Seminars
Also open online: the joining details are on the talk's page.

Conversational data opens new avenues of language modelling for the wide field of human communications, but it also challenges many existing assumptions and paradigms conventional to linguists, educators, and engineers. What does it mean to be a fluent communicator? How to process complex and multi-faceted data? And where to look for meaning in non-verbal communications? We will cover how new data modalities beyond speech and text may be included in language modelling of second language spontaneous, conversations. We find that a multivariate time-series representation of speech enables linguists to comprehend conversational dynamics at a glance and feature extractors to create interpretable features for separating speakers into proficiency levels. Despite complexity and drawbacks, our findings reveal great potential in including new modalities in speech research.