Learning garment synthesis through a shared multimodal approach

Abstract

Designing real and virtual garments has become extremely demanding due to the increased need for synthesising realistically dressed digital humans for video games and movies. The traditional workflow involves a trial-and-error procedure in which a mannequin is draped to judge the resultant folds, a process which is carried out iteratively until the desired look has been achieved. This work presents a garment synthesis pipeline without the need for simulating the final garment by using a multimodal dataset that consists of garment sketches, body parameters and 3D garment meshes in which each domain consists of a different number of dimensions. The pipeline begins by synthesising garment sketches using a Generative Adversarial Network (GAN), followed by filtering out the dissimilar generated garment sketches using an Autoencoder (AE) with anomaly detection. A quantitative evaluation was carried out between a variety of GAN and AE models against the training dataset which showed that a Wasserstein GAN with Gradient Penalty (WGAN-GP) and Angle Based Outlier Detection (ABOD) produced the best results. A Variational Autoencoder (VAE) model was also used to analyse the distribution similarity between the real and generated garment sketches. A multimodal AE with a shared embedding is then trained using the multimodal dataset that can predict across these domains by knowing the garment representation in at least one of the domains. Finally, a fitting algorithm is developed to dress the 3D mannequin mesh with the 3D garment mesh based on the associated body parameters. These dressed mannequins are checked using a simple geometric criterion to discard invalidly dressed mannequins. A qualitative evaluation of garment sketch synthesis, multimodal garment design, and the proposed pipeline quality was carried out using a rating and preference judgment survey. The 32 participants highlighted that (i) WGAN-GP inliers performed closely to the training set, (ii) WGAN-GP outliers performed significantly worse than the WGAN-GP inliers, (iii) characters within animated crowds were perceived similar but different, and (iv) a crowd without clones achieved better quality than a crowd with clones.