journal article·2024-05Open access

Microwave Speech Recognizer Empowered by a Programmable Metasurface

Hongrui Zhang, Hengxin Ruan, Hanting Zhao, Zhuo Wang, Shengguo Hu, Tie Jun Cui, Philipp del Hougne, Lianlin Li

Advanced Science· Volume 11· Issue 17

Abstract

Abstract Speech recognition becomes increasingly important in the modern society, especially for human–machine interactions, but its deployment is still severely thwarted by the struggle of machines to recognize voiced commands in challenging real‐life settings: oftentimes, ambient noise drowns the acoustic sound signals, and walls, face masks or other obstacles hide the mouth motion from optical sensors. To address these formidable challenges, an experimental prototype of a microwave speech recognizer empowered by programmable metasurface is presented here that can remotely recognize human voice commands and speaker identities even in noisy environments and if the speaker's mouth is hidden behind a wall or face mask. The programmable metasurface is the pivotal hardware ingredient of the system because its large aperture and huge number of degrees of freedom allows the system to perform a complex sequence of sensing tasks, orchestrated by artificial‐intelligence tools. Relying solely on microwave data, the system avoids visual privacy infringements. The developed microwave speech recognizer can enable privacy‐respecting voice‐commanded human–machine interactions is experimentally demonstrated in many important but to‐date inaccessible application scenarios. The presented strategy will unlock new possibilities and have expectations for future smart homes, ambient‐assisted health monitoring, as well as intelligent surveillance and security.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Indoor and Outdoor Localization Technologies · Metamaterials and Metasurfaces Applications · Speech and Audio Processing

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

State Key Laboratory of Advanced Optical Communication Systems and Networks School of Electronics Peking University Beijing 100871 China

Peng Cheng Laboratory Shenzhen Guangdong 518000 China

State Key Laboratory of Millimeter Waves Southeast University Nanjing 210096 China

Pazhou Laboratory (Huangpu) Guangzhou Guangdong 510555 China

Univ Rennes CNRS, IETR—UMR 6164 Rennes F‐35000 France

Funding

National Key Research and Development Program of China · National Natural Science Foundation of China

References (57)

Show source-provided references
1. 10.1016/j.specom.2009.08.00210.1016/j.specom.2009.08.002 ↗
2. 10.1016/j.medengphy.2007.05.00310.1016/j.medengphy.2007.05.003 ↗
3. 10.1109/TASLP.2017.274000010.1109/TASLP.2017.2740000 ↗
4. Zhang R. · Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. · 2021 · 5 · 192
5. N.Kimura T.Gemicioglu J.Womack R.Li Y.Zhao A.Bedri Z.Su A.Olwal J.Rekimoto T.Starner in:Proc. CHI ACM Press New York 2022 pp.1–19.10.1145/3491102.3502015 ↗
6. 10.1109/TASLP.2018.286560910.1109/TASLP.2018.2865609 ↗
7. 10.1038/s41467-022-32231-110.1038/s41467-022-32231-1 ↗
8. Gao Y. · Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. · 2020 · 4 · 80
9. Y.Zhang Y.‐C.Chen H.Wang X.Jin in:Proc. UbiComp ACM Press New York 2021 580–585.10.1145/3460418.3480163 ↗
10. 10.1109/LAWP.2010.204057410.1109/LAWP.2010.2040574 ↗
11. 10.3390/s1611181210.3390/s16111812 ↗
12. Movellan J. R. · Proc. NIPS · 1994 · 7 · 8
13. 10.1109/TMM.2009.203063710.1109/TMM.2009.2030637 ↗
14. 10.1109/6046.86547910.1109/6046.865479 ↗
15. 10.1109/MAP.2016.256944710.1109/MAP.2016.2569447 ↗
16. 10.1109/TMC.2016.251763010.1109/TMC.2016.2517630 ↗
17. 10.1038/lsa.2014.9910.1038/lsa.2014.99 ↗
18. 10.1063/5.007602210.1063/5.0076022 ↗
19. 10.1038/s41377-019-0209-z10.1038/s41377-019-0209-z ↗
20. A.Graves N.Jaitly in:Proc. ICML PMLR Beijing2014 pp.1764–1772.
21. A.Vaswani N.Shazeer N.Parmar J.Uszkoreit L.Jones A. N.Gomez L.Kaiser I.Polosukhin in:Proc. NIPS Curran Associates New York2017 pp.6000–6010.
22. 10.1063/1.493594110.1063/1.4935941 ↗
23. 10.1137/11082057910.1137/110820579 ↗
24. 10.1038/srep0154510.1038/srep01545 ↗
25. 10.1126/sciadv.160178210.1126/sciadv.1601782 ↗
26. 10.1002/advs.20190191310.1002/advs.201901913 ↗
27. 10.1016/j.patter.2020.10000610.1016/j.patter.2020.100006 ↗
28. 10.1038/s41467-019-09103-210.1038/s41467-019-09103-2 ↗
29. 10.1038/s41467-017-00164-910.1038/s41467-017-00164-9 ↗
30. CST Bio Models Library Extension 3.2.
31. 10.1364/OSAC.2.00349910.1364/OSAC.2.003499 ↗
32. Hougne P. · Phys. Rev. · 2020 · 2
33. 10.1038/s41928-023-01011-010.1038/s41928-023-01011-0 ↗
34. 10.1093/nsr/nwac26610.1093/nsr/nwac266 ↗
35. 10.1038/srep3569210.1038/srep35692 ↗
36. 10.1038/srep4230210.1038/srep42302 ↗
37. 10.1002/adma.20190406910.1002/adma.201904069 ↗
38. 10.1038/srep2373110.1038/srep23731 ↗
39. 10.1103/PhysRevApplied.6.05401910.1103/PhysRevApplied.6.054019 ↗
40. 10.1364/JOSAB.34.00171310.1364/JOSAB.34.001713 ↗
41. 10.1364/AO.57.00412310.1364/AO.57.004123 ↗
42. 10.1109/TAP.2020.298665310.1109/TAP.2020.2986653 ↗
43. H.Ruan M.Wei H.Zhao H.Li L.Li in:Proc. NEMO.IEEE New York 2020 pp.1–4.10.1109/NEMO49486.2020.9343444 ↗
44. 10.1109/TCOMM.2018.287689910.1109/TCOMM.2018.2876899 ↗
45. 10.1038/s41928-018-0190-110.1038/s41928-018-0190-1 ↗
46. 10.1186/s13638-019-1438-910.1186/s13638-019-1438-9 ↗
47. 10.1109/ACCESS.2020.297777210.1109/ACCESS.2020.2977772 ↗
48. 10.1038/s41467-020-17808-y10.1038/s41467-020-17808-y ↗
49. 10.1109/MWC.001.200026710.1109/MWC.001.2000267 ↗
50. 10.1038/s41928-021-00554-410.1038/s41928-021-00554-4 ↗
51. 10.1002/advs.20227016710.1002/advs.202270167 ↗
52. Hougne P. · Phys. Rev. X · 2018 · 8
53. 10.1038/s41467-022-29354-w10.1038/s41467-022-29354-w ↗
54. 10.1038/s41928-022-00719-910.1038/s41928-022-00719-9 ↗
55. 10.1038/s41566-020-0604-210.1038/s41566-020-0604-2 ↗
56. Z.Wang H.Zhao H.Zhang M.Wei S.Jiang L.Li in:Proc. IMWS‐AMP IEEE New York 2021 pp.379–381.10.1109/IMWS-AMP53428.2021.9643950 ↗
57. D. P.Kingma J. L. A.Ba A method for stochastic optimization.arXiv:1412.6980 2014.