ViGA-FLAME:
Aligning Latent Visual and Geometric Representations for Robust 3D Head Expression Estimation

Wen Hao He, Jiacheng Wu, Jie Chen†
Hong Kong Baptist University

Abstract

Estimating 3D head expression from a single image requires linking visual appearance to facial geometry. We propose ViGA-FLAME, a two-stage framework that aligns semantic visual features with a structured geometric latent space. First, a Parametric–Geometric Variational Autoencoder (PG-VAE) learns a unified parametric-geometric representation of canonical head surfaces and FLAME parameters, supporting both detailed surface reconstruction and interpretable parameter recovery. Supervision on geometry and parameters, together with consistency between the two decoding branches, connects surface detail with parametric structure. Second, a trainable Visual-Geometric Adapter (ViGA) maps features from a pretrained semantic visual encoder into this latent space. Asymmetric attention allows the adapter to read contextual features while preserving the encoder’s original semantic outputs. ViGA-FLAME brings together complementary cues from visual semantics and geometric structure to support coherent 3D head expression estimation under challenging conditions, including extreme expressions and viewing angles, outperforming competing methods.

Pipeline

ViGA-FLAME method overview.

Method overview. Our framework has two stages. PG-VAE first learns a shared representation of canonical FLAME surfaces and decodes it into both ordered 3D geometry and interpretable FLAME parameters. ViGA then uses asymmetric attention to map features from a frozen Sapiens2 image encoder into the same geometric latent space. Finally, the frozen PG-VAE decoder reconstructs the head surface and FLAME parameters from a single image.

Qualitative Comparisons

Qualitative comparison visualization.

Qualitative face reconstruction comparisons. Each row shows one example: (a) Input image; (b) frontal-view display; (c) Optimized / GT mesh; (d) DECA [3]; (e) SMIRK [4]; (f) TEASER [5]; (g) SHeaP [6]; (h) Our ViGA-FLAME.

Long-sequence comparisons on RenderMe360

Video 1 of 8

Long-sequence results on the RenderMe360 dataset. Each video shows seven items in order: (1) Input image; (2) Optimized mesh; (3) DECA [3]; (4) SMIRK [4]; (5) TEASER [5]; (6) SHeaP [6]; (7) Our ViGA-FLAME.

Quantitative Comparisons

Cross-view and cross-distribution face reconstruction robustness comparisons.

Face reconstruction robustness. (a) Cross-view robustness shows face-only CD-L1 across five camera yaw angles; each point gives equal weight to the synthetic FLAME and RenderMe360 results at that angle. (b) Cross-distribution robustness compares five synthetic FLAME parameter regimes with the real RenderMe360 dataset. Both panels report face-only CD-L1 in millimeters after rigid alignment; lower is better.

Face-only reconstruction across synthetic FLAME subsets and the RenderMe360 dataset. The synthetic subsets separate typical cases from challenging shape, expression, and pose settings, including a combined extreme subset. ViGA-FLAME is trained on images rendered from FLAME2023, while the baseline methods use FLAME2020-based decoders. All five synthetic test subsets are generated with FLAME2020, making this a cross-version evaluation for ViGA-FLAME. The RenderMe360 reference meshes are reconstructed from the dataset’s FLAME2020 fitting annotations. Each result averages the five camera views after rigid ICP alignment. CD-L1 measures surface error in millimeters (lower is better), while F@5 measures agreement within 5 mm (higher is better). Best results are in bold. Δ denotes ViGA-FLAME Param. minus the strongest baseline for each metric; negative CD-L1 and positive F@5 indicate improvements.
Method All typical Shape extreme Expression extreme Pose extreme All extreme RenderMe360
CD-L1↓F@5↑ CD-L1↓F@5↑ CD-L1↓F@5↑ CD-L1↓F@5↑ CD-L1↓F@5↑ CD-L1↓F@5↑
DECA 3.1810.864 3.4680.822 3.2850.849 6.4600.650 6.6420.608 3.9720.747
SMIRK 3.2670.848 3.4890.819 3.3410.835 6.0160.653 6.2600.618 4.1840.723
TEASER 3.4430.821 3.6620.792 3.5170.812 5.6200.642 5.8170.615 4.4920.693
SHeaP 4.2690.823 4.9880.769 4.7860.788 8.0230.613 8.1130.585 3.7540.780
ViGA-FLAME Recon. 2.6930.937 2.7740.923 2.7270.930 3.5810.835 3.7680.798 4.4300.696
ViGA-FLAME Param. 2.5560.949 2.7100.929 2.6590.938 3.2120.868 3.4450.833 4.0740.731
Δ vs. best baseline -0.625+0.085 -0.758+0.107 -0.626+0.089 -2.408+0.215 -2.372+0.215 +0.320-0.049

The parametric branch achieves the best results across all five synthetic FLAME subsets, with particularly clear gains under extreme pose and combined extreme conditions. RenderMe360 provides a complementary evaluation on real multiview captures, using reference meshes reconstructed from the dataset’s optimization-fitted FLAME annotations. Together, these experiments assess reconstruction across controlled parameter variations and real captured faces.

Face-only reconstruction across camera viewpoints. The same synthetic evaluation is regrouped by viewing angle: frontal (0°), oblique (±45°), and profile (±90°). Each group pools all five FLAME parameter subsets after rigid ICP alignment, isolating the effect of viewpoint while retaining variation in shape, expression, and pose. CD-L1 is in millimeters; lower CD-L1 and higher F@5 are better. Best results are in bold.
Method Frontal (0°) Oblique (±45°) Profile (±90°)
CD-L1↓F@5↑ CD-L1↓F@5↑ CD-L1↓F@5↑
DECA 5.2630.756 4.3870.775 4.4990.744
SMIRK 4.3940.779 4.3730.774 4.6160.723
TEASER 4.1160.772 4.4070.745 4.5640.710
SHeaP 5.6420.736 5.4080.743 6.8600.678
ViGA-FLAME Recon. 3.0190.902 3.0110.896 3.2510.864
ViGA-FLAME Param. 2.7230.930 2.8200.915 3.1090.879

Both ViGA-FLAME branches retain their lead across the three angle groups. Their errors increase and F@5 scores decrease at profile views, showing that side views remain harder than frontal views even when the global rigid pose is aligned.

Ablation studies of PG-VAE variations. These results use randomly sampled FLAME states with varied orientations. The three columns report mean corresponding-vertex errors in millimeters for the direct geometry reconstruction, the decoded parametric mesh, and the disagreement between those two outputs. Lower is better. Unlike the face-only comparisons above, these errors are measured without rigid alignment.
Method Recon. ↓ Param. ↓ Recon.↔Param. ↓
Base 1.02810.94721.2707
+ λgLpar-gt 0.99420.81521.1726
++ λcLpar-geo 1.03470.76431.2344
+++ Condquery 0.86470.49750.9713
+++ Condadd 0.98310.58301.1605

Starting from direct reconstruction and FLAME parameter supervision, the variants add supervision for the decoded parametric surface and then encourage agreement between the two decoding branches. The last two rows compare replacing the learned semantic queries with FLAME-conditioned embeddings against adding those embeddings to the queries. Query replacement yields the lowest error in all three measures.