Multi-AGV Route Planning for Unmanned Warehouses Based on Improved DQN Algorithm

doi:10.3969/j.issn.1007-7375.220247

Abstract

Abstract: To solve the problem of multi-AGV route planning and conflicts in unmanned warehouses, with the objective of minimizing the total travel time, a multi-AGV route planning model is established, and an improved DQN algorithm based on dynamic decision-making is proposed. An empirical knowledge model based on static route planning of a single AGV is designed to guide the learning and exploration direction of AGVs. It avoids conflicts and obstacles for AGVs in advance, and accelerates the convergence of the proposed algorithm. Also, a conflict resolution strategy based on the shortest total travel time is proposed to fundamentally solve the problem of multi-AGV route conflicts and deadlocks. Finally, a grid map of an unmanned warehouse is established for simulation experiments. Results show that, compared with other DQN algorithms, the convergence speed of the proposed model and algorithm is increased by 13.3%, and the average loss value is reduced by 26.3%. This result indicates that the model and algorithm are conducive to avoiding and resolving the conflicts of multi-AGV route planning in unmanned warehouses, reducing the total travel time of multiple AGVs and having important guiding significance to improve the efficiency of unmanned warehouse operations.

Key words: multiple AGVs, route planning, DQN algorithm, empirical knowledge, conflict resolution

CLC Number:

F406.2
TP24

XIE Yong, ZHENG Suijun, CHENG Niansheng, ZHU Hongjun. Multi-AGV Route Planning for Unmanned Warehouses Based on Improved DQN Algorithm[J]. Industrial Engineering Journal, 2024, 27(1): 36-44,53.

References

[1] 余娜娜, 李铁克, 王柏琳, 等. 自动化分拣仓库中多AGV调度与路径规划算法[J]. 计算机集成制造系统, 2020, 26(1): 171-180
YU Nana, LI Tieke, WANG Bailin, et al. Multi-AGVs scheduling and path planning algorithm in automated sorting warehouse[J]. Computer Integrated Manufacturing Systems, 2020, 26(1): 171-180
[2] 王秀红, 刘雪豪, 王永成. 基于改进A*算法的仓储物流移动机器人任务调度和路径优化研究[J]. 工业工程, 2019, 22(6): 34-39
WANG Xiuhong, LIU Xuehao, WANG Yongcheng. A research on task scheduling and path planning of mobile robot in warehouse logistics based on improved A* algorithm[J]. Industrial Engineering Journal, 2019, 22(6): 34-39
[3] YANG L, FU L, LI P, et al. An effective dynamic path planning approach for mobile robots based on ant colony fusion dynamic windows[J]. Machines, 2022, 10(1): 50
[4] ZHONG X, TIAN J, HU H, et al. Hybrid path planning based on safe A* algorithm and adaptive window approach for mobile robot in large-scale dynamic environment[J]. Journal of Intelligent & Robotic Systems, 2020, 99(1): 65-77
[5] YANG Y, LI Juntao, PENG Lingling. Multi-robot path planning based on a deep reinforcement learning DQN algorithm[J]. CAAI Transactions on Intelligence Technology, 2020, 5(3): 177-183
[6] GUO S, ZHANG X, ZHENg Y, et al. An autonomous path planning model for unmanned ships based on deep reinforcement learning[J]. Sensors, 2020, 20(2): 426
[7] GAO P, LIU Z, WU Z, et al. A global path planning algorithm for robots using reinforcement learning[C/OL]//2019 IEEE International Conference on Robotics and Biomimetics (ROBIO). Dali: IEEE, 2019: 1693-1698 (2019-12-01). https:// doi.org/10.1109/ROBIO49542.2019.8961753
[8] WATKINS C J C H, DAYAN P. Q-learning[J]. Machine Learning, 1992, 8(3): 279-292
[9] 周飞燕, 金林鹏, 董军. 卷积神经网络研究综述[J]. 计算机学报, 2017, 40(6): 1229-1251
ZHOU Feiyan, JIN Linpeng, DONG Jun. Review of convolutional neural network[J]. Chinses Journal of Computers, 2017, 40(6): 1229-1251
[10] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518: 529-533
[11] JIANG L, HUANG H, DING Z. Path planning for intelligent robots based on deep Q-learning with experience replay and heuristic knowledge[J]. IEEE/CAA Journal of Automatica Sinica, 2019, 7(4): 1179-1189
[12] 朱霸坤, 朱卫纲, 李伟, 等. 基于先验知识的多功能雷达智能干扰决策方法[J]. 系统工程与电子技术, 2022, 44(12): 3685-3695
ZHU Bakun, ZHU Weigang, LI Wei, et al. Multi-function radar intelligent jamming decision method based on prior knowledge[J]. Systems Engineering and Electronics, 2022, 44(12): 3685-3695
[13] 邹裕吉, 宋豫川, 王馨坤, 等. 自动导向小车与加工设备多目标集成调度的聚类遗传算法[J]. 中国机械工程, 2022, 33(1): 97-108
ZOU Yuji, SONG Yuchuan, WANG Xinkun, et al. Clustering genetic algorithm for multi-objective integrated scheduling of AGVs and machine[J]. China Mechanical Engineering, 2022, 33(1): 97-108
[14] MATIGNON L, LAURENT G J, LE FORT-PIAT N L. Hysteretic Q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams[C]// 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. San Diego: IEEE, 2007: 64-69.