journal article·2024-09-11Open access

Stable Jumping Control Based on Deep Reinforcement Learning for a Locust-Inspired Robot

Qijie Zhou, Gangyang Li, Rui Tang, Yi Xu, Hao Wen, Qing Shi

Biomimetics· Volume 9· Issue 9

Abstract

Biologically inspired jumping robots exhibit exceptional movement capabilities and can quickly overcome obstacles. However, the stability and accuracy of jumping movements are significantly compromised by rapid changes in posture. Here, we propose a stable jumping control algorithm for a locust-inspired jumping robot based on deep reinforcement learning. The algorithm utilizes a training framework comprising two neural network modules (actor network and critic network) to enhance training performance. The framework can control jumping by directly mapping the robot’s observations (robot position and velocity, obstacle position, target position, etc.) to its joint torques. The control policy increases randomness and exploration by introducing an entropy term to the policy function. Moreover, we designed a stage incentive mechanism to adjust the reward function dynamically, thereby improving the robot’s jumping stability and accuracy. We established a locus-inspired jumping robot platform and conducted a series of jumping experiments in simulation. The results indicate that the robot could perform smooth and non-flip jumps, with the error of the distance from the target remaining below 3%. The robot consumed 44.6% less energy to travel the same distance by jumping compared with walking. Additionally, the proposed algorithm exhibited a faster convergence rate and improved convergence effects compared with other classical algorithms.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Robotic Locomotion and Control · Reinforcement Learning in Robotics · Biomimetic flight and propulsion mechanisms

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Intelligent Robotics Institute, School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China

Key Laboratory of Biomimetic Robots and Systems, Beijing Institute of Technology, Ministry of Education, Beijing 100081, China

School of Mathematics and Computational Science, Xiangtan University, Xiangtan 411105, China

Yangtze Delta Region Academy, Beijing Institute of Technology, Jiaxing 314000, China

Funding

National Natural Science Foundation of China · Science and Technology Innovation Program of Beijing Institute of Technology

References (33)

Show source-provided references
1. A Survey of Sim-to-Real Transfer Techniques Applied to Reinforcement Learning for Bioinspired Robots10.1109/TNNLS.2021.3112718 ↗
2. Research status of bionic amphibious robots: A review10.1016/j.oceaneng.2021.108862 ↗
3. Design and technical development of wall-climbing robots: A review10.1007/s42235-022-00189-x ↗
4. Biologically inspired jumping robots: A comprehensive review10.1016/j.robot.2019.103362 ↗
5. Development of a fast-swimming dolphin robot capable of leaping10.1109/TMECH.2016.2572720 ↗
6. Design and Optimization of a Miniature Locust-Inspired Stable Jumping Robot10.1109/LRA.2023.3287795 ↗
7. Flea-inspired catapult mechanism for miniature jumping robots10.1109/TRO.2012.2198510 ↗
8. Robotic vertical jumping agility via series-elastic power modulation10.1126/scirobotics.aag2048 ↗
9. Locust-inspired jumping mechanism design and improvement based on takeoff stability10.1115/1.4063406 ↗
10. Jung, G.-P., and Cho, K.-J. (2016). Froghopper-inspired direction-changing concept for miniature jumping robots. Bioinspiration Biomim., 11.10.1088/1748-3190/11/5/056015 ↗
11. Bai, X., Kong, D., Wang, Q., Yu, X., and Xie, X. (2023). Bionic Design of a Miniature Jumping Robot. Appl. Sci., 13.10.3390/app13074534 ↗
12. Designing minimal and scalable insect-inspired multi-locomotion millirobots10.1038/s41586-019-1388-8 ↗
13. The Continuous Jump Control of a Locust-inspired Robot with Omnidirectional Trajectory Adjustment10.1109/LRA.2024.3349808 ↗
14. Chen, Z., Liang, Q., Wei, Z., Chen, X., Shi, Q., Yu, Z., and Sun, T. (2023). An Overview of In Vitro Biological Neural Networks for Robot Intelligence. Cyborg. Bionic. Syst., 4.10.34133/cbsystems.0001 ↗
15. Perceptive locomotion through nonlinear model-predictive control10.1109/TRO.2023.3275384 ↗
16. Trajectory optimization of 5-link biped robot using beetle antennae search
17. Robust jumping with an articulated soft quadruped via trajectory optimization and iterative learning10.1109/LRA.2023.3331288 ↗
18. ANYmal parkour: Learning agile navigation for quadrupedal robots10.1126/scirobotics.adi7566 ↗
19. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
20. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv.
21. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016, January 20–22). Asynchronous methods for deep reinforcement learning. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
22. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
23. Smith, L., Kew, J.C., Li, T., Luu, L., Peng, X.B., Ha, S., Tan, J., and Levine, S. (2023). Learning and adapting agile locomotion skills by transferring experience. arXiv.10.15607/RSS.2023.XIX.051 ↗
24. Ye, L., Li, J., Cheng, Y., Wang, X., Liang, B., and Peng, Y. (2023). From Knowing to Doing: Learning Diverse Motor Skills through Instruction Learning. arXiv.
25. From motor control to team play in simulated humanoid football10.1126/scirobotics.abo0235 ↗
26. Zhou, Q., Li, G., Tang, R., Xu, Y., and Shi, Q. (2023, January 13–15). Adaptive Formation Tracking of Swarm Jumping Robots Using Multiagent Deep Reinforcement Learning. Proceedings of the 2023 IEEE International Conference on Unmanned Systems (ICUS), Hefei, China.10.1109/ICUS58632.2023.10318255 ↗
27. Control of high-speed jumps: The rotation and energetics of the locust (Schistocerca gregaria)10.1007/s00360-023-01524-2 ↗
28. Jump takeoff in a small jumping spider10.1007/s00359-021-01473-7 ↗
29. Take-off speed in jumping mantises depends on body size and a power-limited mechanism
30. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10–15). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
31. Todorov, E., Erez, T., and Tassa, Y. (2012, January 7–12). Mujoco: A physics engine for model-based control. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal.10.1109/IROS.2012.6386109 ↗
32. Jumping efficiency of small creatures and its applicability in robotics10.1016/j.promfg.2018.02.117 ↗
33. Snelling, E.P., Becker, C.L., and Seymour, R.S. (2013). The effects of temperature and body mass on jump performance of the locust Locusta migratoria. PLoS ONE, 8.10.1371/journal.pone.0072471 ↗