Expression retargeting from images to three‐dimensional face models represented in texture space
Source abstract
Abstract
Abstract Facial expressions play a very important role in creating a vivid and realistic virtual character. This paper proposes a convolutional neural network‐based facial expression retargeting framework named EXPUV‐Net, which can extract expression information from an image and transfer it to a specified 3D face. With the help of the nonlinear generator, the proposed framework can generate rich facial expressions and does not rely on the blendshape model. A data augmentation approach is adopted in this paper, which can improve the adaptability of the proposed framework. The framework can realize expression retargeting for face models with different topologies. This paper also demonstrates the effectiveness of the proposed framework through some experiments and compares it with some linear face model‐based methods for facial component extraction and face reconstruction. The results show that the proposed framework has better performance than the linear methods. The proposed framework is very helpful for facial expression generation for virtual characters.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face recognition and analysis · Generative Adversarial Networks and Image Synthesis · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics Beijing Institute of Technology Beijing China
Funding
Special Project for Research and Development in Key areas of Guangdong Province · National Natural Science Foundation of China · Higher Education Discipline Innovation Project
References (43)
Show source-provided references
1. BlanzV VetterT.A morphable model for the synthesis of 3D faces.Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques;1999. p.187–194.10.1145/311535.311556 ↗
2. 10.1111/cgf.1338210.1111/cgf.13382 ↗
3. 10.1145/3272127.327509310.1145/3272127.3275093 ↗
4. 10.1145/2601097.260120410.1145/2601097.2601204 ↗
5. 10.1145/3072959.307367410.1145/3072959.3073674 ↗
6. ThiesJ ZollhoferM StammingerM TheobaltC NiessnerM.Face2Face: real‐time face capture and reenactment of RGB videos.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR);2016. p.2387–2395.10.1109/CVPR.2016.262 ↗
7. WuW ZhangY LiC QianC LoyCC.Reenactgan: learning to reenact faces via boundary transfer.Proceedings of the European Conference on Computer Vision (ECCV);2018. p.603–619.10.1007/978-3-030-01246-5_37 ↗
8. 10.1145/3272127.327504310.1145/3272127.3275043 ↗
9. FLNet: landmark driven fetching and learning network for faithful talking facial animation synthesis
10. ZhaoR WuT GuoG.Sparse to dense motion transfer for face image animation.Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops;2021. p.1991–2000.10.1109/ICCVW54120.2021.00226 ↗
11. 10.1109/TVCG.2020.301387610.1109/TVCG.2020.3013876 ↗
12. JiangZ‐H WuQ ChenK ZhangJ.Disentangled representation learning for 3D face shape.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR);2019. p.11957–11966.10.1109/CVPR.2019.01223 ↗
13. 10.1111/cgf.1440010.1111/cgf.14400 ↗
14. 10.1145/3450626.345993610.1145/3450626.3459936 ↗
15. ChaudhuriB VesdapuntN ShapiroL WangB.Personalized face modeling for improved face reconstruction and motion retargeting.European Conference on Computer Vision Springer;2020. p.142–160.10.1007/978-3-030-58558-7_9 ↗
16. DengY YangJ XuS ChenD JiaY TongX.Accurate 3D face reconstruction with weakly‐supervised learning: from single image to image set.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops;2019.10.1109/CVPRW.2019.00038 ↗
17. LeeG‐H LeeS‐W.Uncertainty‐aware mesh decoder for high fidelity 3D face reconstruction.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR);2020. p.6100–6109.10.1109/CVPR42600.2020.00614 ↗
18. ZhuJ‐Y ParkT IsolaP EfrosAA.Unpaired image‐to‐image translation using cycle‐consistent adversarial networks.Proceedings of the IEEE International Conference on Computer Vision (ICCV);2017. p.2223–2232.10.1109/ICCV.2017.244 ↗
19. IsolaP ZhuJ‐Y ZhouT EfrosAA.Image‐to‐image translation with conditional adversarial networks.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR);2017. p.1125–1134.10.1109/CVPR.2017.632 ↗
20. MengQ ZhaoS HuangZ ZhouF.MagFace: A universal representation for face recognition and quality assessment.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR);2021. p.14225–14234.10.1109/CVPR46437.2021.01400 ↗
21. DengJ GuoJ XueN ZafeiriouS.Arcface: Additive angular margin loss for deep face recognition.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR);2019. p.4690–4699.10.1109/CVPR.2019.00482 ↗
22. HuangY WangY TaiY LiuX ShenP LiS et al.Curricularface: adaptive curriculum learning loss for deep face recognition.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR);2020. p.5901–5910.10.1109/CVPR42600.2020.00594 ↗
23. ChandranP ZossG GotardoP GrossM BradleyD.Adaptive convolutions for structure‐aware style transfer.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR);2021. p.7972–7981.10.1109/CVPR46437.2021.00788 ↗
24. 10.1109/TIP.2020.301650210.1109/TIP.2020.3016502 ↗
25. 10.1145/3450626.345977110.1145/3450626.3459771 ↗
26. AbdalR QinY WonkaP.Image2StyleGAN++: How to edit the embedded images?Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR);2020. p.8296–8305.10.1109/CVPR42600.2020.00832 ↗
27. JoY ParkJ.SC‐FEGAN: face editing generative adversarial network with user's sketch and color.Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV);2019. p.1745–1753.10.1109/ICCV.2019.00183 ↗
28. RonnebergerO FischerP BroxT.U‐Net: convolutional networks for biomedical image segmentation;2015. p.234–241.10.1007/978-3-319-24574-4_28 ↗
29. RanjanA BolkartT SanyalS BlackMJ.Generating 3D faces using convolutional mesh autoencoders.Proceedings of the European Conference on Computer Vision (ECCV);2018. p.704–720.10.1007/978-3-030-01219-9_43 ↗
30. YanY LuK XueJ GaoP LyuJ.Feafa: A well‐annotated dataset for facial expression analysis and 3D facial animation.2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) IEEE;2019. p.96–101.10.1109/ICMEW.2019.0-104 ↗
31. ZhuX LeiZ LiuX ShiH LiSZ.Face alignment across large poses: a 3D solution.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR);2016. p.146–155.10.1109/CVPR.2016.23 ↗
32. 10.1109/TVCG.2013.24910.1109/TVCG.2013.249 ↗
33. PaysanP KnotheR AmbergB RomdhaniS VetterT.A 3D face model for pose and illumination invariant face recognition.2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance IEEE;2009. p.296–301.10.1109/AVSS.2009.58 ↗
34. 10.1145/3306346.332295910.1145/3306346.3322959 ↗
35. QiCR SuH MoK GuibasLJ.Pointnet: deep learning on point sets for 3D classification and segmentation.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR);2017. p.652–660.
36. SuH MajiS KalogerakisE Learned‐MillerE.Multi‐view convolutional neural networks for 3D shape recognition.Proceedings of the IEEE International Conference on Computer Vision (ICCV);2015. p.945–953.10.1109/ICCV.2015.114 ↗
37. JacksonAS BulatA ArgyriouV TzimiropoulosG.Large pose 3D face reconstruction from a single image via direct volumetric cnn regression.Proceedings of the IEEE International Conference on Computer Vision (ICCV);2017. p.1031–1039.10.1109/ICCV.2017.117 ↗
38. FengY WuF ShaoX WangY ZhouX.Joint 3d face reconstruction and dense alignment with position map regression network.Proceedings of the European Conference on Computer Vision (ECCV);2018. p.534–551.10.1007/978-3-030-01264-9_33 ↗
39. Deepfakes.https://github.com/deepfakes/faceswap
40. Paszke A · 2019 · 8024
41. Acquiring identity and expression information from monocular face image
42. GuoJ ZhuX YangY YangF LeiZ LiSZ.Towards fast accurate and stable 3D dense face alignment;2020. p.152–168.10.1007/978-3-030-58529-7_10 ↗
43. 10.1145/3130800.313081310.1145/3130800.3130813 ↗