journal article·2022-12

Deep‐Learning‐Assisted Noncontact Gesture‐Recognition System for Touchless Human‐Machine Interfaces

Hao Zhou, Wei Huang, Zhuo Xiao, Shichuan Zhang, Wangzhan Li, Jinhui Hu, Tianxing Feng, Jing Wu, Pengcheng Zhu, Yanchao Mao

Advanced Functional Materials· Volume 32· Issue 49
Publisher page

Abstract

Abstract Human‐machine interfaces (HMIs) play important role in the communication between humans and robots. Touchless HMIs with high hand dexterity and hygiene hold great promise in medical applications, especially during the pandemic of coronavirus disease 2019 (COVID‐19) to reduce the spread of virus. However, current touchless HMIs are mainly restricted by limited types of gesture recognition, the requirement of wearing accessories, complex sensing platforms, light conditions, and low recognition accuracy, obstructing their practical applications. Here, an intelligent noncontact gesture‐recognition system is presented through the integration of a triboelectric touchless sensor (TTS) and deep learning technology. Combined with a deep‐learning‐based multilayer perceptron neural network, the TTS can recognize 16 different types of gestures with a high average accuracy of 96.5%. The intelligent noncontact gesture‐recognition system is further applied to control a robot for collecting throat swabs in a noncontact mode. Compared with present touchless HMIs, the proposed system can recognize diverse complex gestures by utilizing charges naturally carried on human fingers without the need of wearing accessories, complicated device structures, adequate light conditions, and achieves high recognition accuracy. This system could provide exciting opportunities to develop a new generation of touchless medical equipment, as well as touchless public facilities, smart robots, virtual reality, metaverse, etc.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Advanced Sensor and Energy Harvesting Materials · Tactile and Sensory Interactions · Gaze Tracking and Assistive Technology

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Key Laboratory of Materials Physics Ministry of Education School of Physics and Microelectronics Zhengzhou University Zhengzhou 450001 P. R. China

Funding

National Natural Science Foundation of China · China Postdoctoral Science Foundation

References (47)

Show source-provided references
1. 10.1126/scirobotics.abn049510.1126/scirobotics.abn0495 ↗
2. 10.1007/s12274-022-4622-x10.1007/s12274-022-4622-x ↗
3. 10.1002/adfm.20220129210.1002/adfm.202201292 ↗
4. 10.1002/adfm.20210943010.1002/adfm.202109430 ↗
5. 10.1126/sciadv.abo520110.1126/sciadv.abo5201 ↗
6. 10.1002/adfm.20210511010.1002/adfm.202105110 ↗
7. 10.1002/advs.20220105610.1002/advs.202201056 ↗
8. 10.1002/adfm.20211300510.1002/adfm.202113005 ↗
9. 10.1002/adma.20210021810.1002/adma.202100218 ↗
10. 10.1002/smll.20210054210.1002/smll.202100542 ↗
11. 10.1038/s41467-022-29197-510.1038/s41467-022-29197-5 ↗
12. 10.1021/acsnano.1c0395010.1021/acsnano.1c03950 ↗
13. 10.1007/s12274-021-3831-z10.1007/s12274-021-3831-z ↗
14. 10.1038/s41467-020-19531-010.1038/s41467-020-19531-0 ↗
15. 10.1002/adfm.20210108910.1002/adfm.202101089 ↗
16. 10.1002/advs.20210505610.1002/advs.202105056 ↗
17. 10.1126/scirobotics.abe131510.1126/scirobotics.abe1315 ↗
18. 10.1002/adfm.20190789310.1002/adfm.201907893 ↗
19. 10.1016/j.nanoen.2020.10571910.1016/j.nanoen.2020.105719 ↗
20. 10.1016/j.nanoen.2021.10607110.1016/j.nanoen.2021.106071 ↗
21. 10.1038/s41467-022-29083-010.1038/s41467-022-29083-0 ↗
22. 10.1126/sciadv.abl987410.1126/sciadv.abl9874 ↗
23. 10.1021/acsnano.2c0260910.1021/acsnano.2c02609 ↗
24. 10.1038/s41565-020-00779-y10.1038/s41565-020-00779-y ↗
25. 10.1002/aisy.20210022810.1002/aisy.202100228 ↗
26. 10.1021/acsnano.1c0757910.1021/acsnano.1c07579 ↗
27. 10.1002/adma.20220025210.1002/adma.202200252 ↗
28. 10.1002/adfm.20220480310.1002/adfm.202204803 ↗
29. An S. · ACS Nano · 2022 · 16 · 9395
30. 10.1002/adma.20220307310.1002/adma.202203073 ↗
31. 10.1038/s41467-021-27672-z10.1038/s41467-021-27672-z ↗
32. 10.1038/s41928-021-00558-010.1038/s41928-021-00558-0 ↗
33. 10.1126/scirobotics.abc880110.1126/scirobotics.abc8801 ↗
34. 10.1063/5.001648510.1063/5.0016485 ↗
35. 10.1002/adma.20220079310.1002/adma.202200793 ↗
36. 10.1038/s41467-021-25637-w10.1038/s41467-021-25637-w ↗
37. 10.1002/advs.20200026110.1002/advs.202000261 ↗
38. 10.1038/s41467-022-30224-810.1038/s41467-022-30224-8 ↗
39. 10.1002/adma.20210541610.1002/adma.202105416 ↗
40. 10.1002/adma.20220343110.1002/adma.202203431 ↗
41. 10.1002/adfm.20210842310.1002/adfm.202108423 ↗
42. 10.1016/0304-3886(90)90010-S10.1016/0304-3886(90)90010-S ↗
43. 10.1016/j.elstat.2005.04.00210.1016/j.elstat.2005.04.002 ↗
44. 10.1021/acsnano.1c1109610.1021/acsnano.1c11096 ↗
45. 10.1038/s41467-020-17842-w10.1038/s41467-020-17842-w ↗
46. 10.1007/s12274-022-4339-x10.1007/s12274-022-4339-x ↗
47. 10.1038/s41563-020-00902-310.1038/s41563-020-00902-3 ↗