Single Camera based Simulation and Try-on Ready Avatar Modeling

Overview & Synopsis

Reconstructing and rendering high-fidelity, animatable, and try-on-ready virtual avatars using only a single camera is crucial for many applications. However, existing try-on-ready human avatar reconstruction methods struggle to achieve this from monocular videos or single images, primarily due to two issues: (1) existing approaches typically use a single layer to represent both garments and the underlying body, ignoring their distinct motion characteristics; and (2) input captured from a single camera often fails to fully cover both the garments and the underlying body. To address these challenges, we propose a physically integrated, double-layer GS-Mesh representation for reconstructing decoupled 3D clothed human avatars from monocular videos or even single images by incorporating 2D generated data. While 2D generation methods can produce video sequences from given pose sequences, they struggle to maintain inter-frame consistency. We decompose the inter-frame inconsistencies in the generated data into geometric and appearance inconsistencies. Based on our 3D representation, we ensure that the final reconstruction and rendering results are both 3D-consistent and temporally coherent. Compared with existing 3D garment transfer methods that rely on multi-view reconstruction and most 2D approaches that use diffusion models for image-based garment editing without novel-view or novel-pose synthesis, our method requires only a monocular video or even a single image while producing visually plausible cloth dynamics during virtual try-on and novel-pose animation. This represents a significant advancement toward virtual try-on with visually plausible cloth dynamics using only a single camera.