Multimodal representation learning of group interactions

Overview & Synopsis

The transition toward a low-carbon economy requires the implementation of complex socio-technical projects involving multiple stakeholders and often generating socio-environmental controversies. In this context, the Critical Minerals in Orogens (CMiO) project aims to support the sustainable exploration and valorization of critical mineral resources while fostering dialogue and collaboration with territorial stakeholders. Although psychosocial sciences have extensively investigated the mechanisms that promote citizen participation and the quality of public debate, the interaction dynamics underlying these processes remain largely unexplored from a computational perspective. This research focuses on the role of non-verbal interpersonal synchrony in group interactions and collective decision-making processes. We hypothesize that the synchronization of non-verbal behaviors constitutes a meaningful indicator of interaction quality and mutual understanding among participants. To investigate this phenomenon, we propose an unsupervised multimodal representation learning framework based on Dynamical Variational Autoencoders (DVAEs). The objective is to learn a latent representation that disentangles interaction-related dynamics associated with interpersonal synchrony from participant-specific behavioral characteristics. The proposed model relies on latent representations extracted from temporal sequences of 3D human body meshes derived from the Seamless Interaction dataset, one of the largest currently available corpora of dyadic interactions. A shared latent variable captures common interaction dynamics, while individual latent variables model participant-specific behaviors. This factorized representation is designed to provide a more interpretable characterization of interpersonal synchrony and to facilitate the analysis of relational mechanisms emerging during discussions and debates. Ultimately, the learned representations will support the study of interaction patterns, the assessment of synchrony's influence on collective dynamics, and the development of computational tools for analyzing public deliberation processes. More broadly, this work contributes to the interdisciplinary objectives of the CMiO project by advancing our understanding of the social mechanisms that foster constructive dialogue, mutual understanding, and socially acceptable socio-technical transitions.