journal article·2024-09Open access

Metaverse‐based remote support system with smooth combination of free viewpoint observation and hand gesture instruction

Takashi Numata, Yuya Ogi, Keiichi Mitani, Kazuyuki Tajima, Yusuke Nakamura, Naohito Ikeda, Kenichi Shimada

Abstract

Abstract In field operations with large machines, onsite large task spaces have been typically required. In these cases, remote skilled workers have to provide a step‐by‐step guide by observing the onsite situation around the large task spaces in three dimensions, instructing onsite unskilled workers on how to perform the tasks with the correct hand gestures at the right position, and switching between observation and instruction frequently during the remote support. In this study, we developed a Metaverse‐based remote support system with a seamless user interface for switching between free viewpoint observation and hand gesture instruction. The proposed system enables remote skilled workers to observe the onsite field from any viewpoint, transfer hand gesture instruction to onsite workers, and seamlessly switch between free viewpoint observation and free hand gesture instruction without having to change devices. We compared the time efficiency of the proposed system and a conventional system through experiments with 28 users and found that our system improved the observation time efficiency by 65.7%, the instruction time efficiency by 27.9%, and the switching time efficiency between observation and instruction by 14.6%. These results indicate that the proposed system enables remote skilled workers to support onsite workers quickly and efficiently.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Virtual Reality Applications and Impacts · Tactile and Sensory Interactions · Teleoperation and Haptic Systems

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Research & Development Group, Hitachi, Ltd. Kokubunji Japan

References (17)

Show source-provided references
1. 10.1016/j.procir.2016.10.00110.1016/j.procir.2016.10.001 ↗
2. 10.1016/j.cmpb.2013.12.01810.1016/j.cmpb.2013.12.018 ↗
3. OyamaE andShiromaN.Behavior navigation system for use in harsh environments. Proceedings of the 2011 IEEE International Symposium on Safety Security and Rescue Robotics.2011;272–277.10.1109/SSRR.2011.6106799 ↗
4. 10.1080/01691864.2015.111388810.1080/01691864.2015.1113888 ↗
5. RobertK ZhuD HuangW andAlemL.MobileHelper: remote guiding using smart mobile devices hand gestures and augmented reality. Siggraph Asia 2013 Symposium on Mobile Graphics and Interactive Applications.2013;38 1–5.10.1145/2543651.2543664 ↗
6. GauglitzS NuernbergerB TurkM HöllererT.In touch with the remote world: remote collaboration with augmented reality drawings and virtual navigation. In Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology.2014;197–205.10.1145/2671015.2671016 ↗
7. The stretchable arms for collaborative remote guiding
8. TeoT LawrenceL LeeGA BillinghurstM andAdcockM.Mixed reality remote collaboration combining 360 video and 3D reconstruction. In Proceedings of the 2019 CHI conference on human factors in computing systems.2019;201 1–14.10.1145/3290605.3300431 ↗
9. 10.3389/frobt.2019.0010610.3389/frobt.2019.00106 ↗
10. 10.1007/s12193-020-00335-x10.1007/s12193-020-00335-x ↗
11. 10.1016/j.procir.2015.08.02610.1016/j.procir.2015.08.026 ↗
12. 10.1016/j.jclepro.2016.07.12310.1016/j.jclepro.2016.07.123 ↗
13. 10.3390/en1419629810.3390/en14196298 ↗
14. 10.3390/app1205266710.3390/app12052667 ↗
15. SuzukiS KanematsuH BarryDM OgawaN YajimaK NakahiraKT ShiraiT KawaguchiM KobayashiT andYoshitakeM.Virtual experiments in metaverse and their applications to collaborative projects: the framework and its significance. 24th International Conference on Knowledge‐Based and Intelligent Information & Engineering Systems.2020;176 2125–2132.10.1016/j.procs.2020.09.249 ↗
16. MitaniK TajimaK andNakamuraY.Real‐time 3D space reconstruction for OT metaverse as interactive virtual site. 29th International Display Workshops (IDW’22).2022; PRJ1–2.10.36463/idw.2022.0722 ↗
17. NumataT OgiY MitaniK TajimaK NakamuraY IkedaN andShimadaK.Development of a remote field operation support system with functions of free viewpoint observation and free hand gesture instruction via Metaverse. 29th International Display Workshops (IDW’22).2022; MVS2–2.10.36463/idw.2022.1186 ↗