Real-time Scheduling of Wafer Photolithography Area Based on Reinforcement Learning with Gated Recurrent Unit

doi:10.3969/j.issn.1007-7375.240100

Abstract

Abstract: To address the scheduling problem of wafer photolithography area, characterized by dynamic nature, real-time requirements, multiple constraints, and multiple objectives, a real-time scheduling method based on gated recurrent unit (GRU) reinforcement learning is proposed. This method incorporates GRU to learn the temporal information of historical scheduling decisions and states in the photolithography area, providing auxiliary decision-making information for the double deep reinforcement learning (DDRL) model. The input state space and output action set of the DDRL model are designed, and a multi-objective reward function is established with the objective of minimizing the maximum completion time of wafers and maximizing the on-time delivery rate, optimizing the scheduling output by intelligent agents. Additionally, constraint relaxation rules and scheduling methods are proposed combining equipment-specific constraints and mask constraints, to enhance the practicality of scheduling strategies. Through empirical evaluation using real-world cases from a wafer manufacturing enterprise, this method is compared with traditional double deep reinforcement learning and heuristic rule methods for photolithography area, demonstrating its superiority and verifying its effectiveness in solving this problem.

Key words: wafer fabrication system, scheduling of photolithography area, deep reinforcement learning, gated recurrent unit (GRU), multi-objective

CLC Number:

F426.4
TP391

WU Lihui, SHI Jinming, JIN Keshan, ZHANG Jie. Real-time Scheduling of Wafer Photolithography Area Based on Reinforcement Learning with Gated Recurrent Unit[J]. Industrial Engineering Journal, 2024, 27(3): 12-21,30.

References

[1] CHEN H, GUO P, JIMENEZ J, et al. Unrelated parallel machine photolithography scheduling problem with dual resource constraints[J]. IEEE Transactions on Semiconductor Manufacturing, 2022, 36(1): 100-112
[2] LI X L, HUANG Y L, TAN Q, et al. Scheduling unrelated parallel batch processing machines with non-identical job sizes[J]. Computers & Operations Research, 2013, 40(12): 2983-2990
[3] MAECKER S, SHEN L, MÖNCH L. Unrelated parallel machine scheduling with eligibility constraints and delivery times to minimize total weighted tardiness[J]. Computers & Operations Research, 2023, 149: 105999
[4] HAM A. Scheduling of dual resource constrained lithography production: using CP and MIP/CP[J]. IEEE Transactions on Semiconductor Manufacturing, 2017, 31(1): 52-61
[5] 张朋, 张洁, 王卓君, 等. 基于分解多目标进化算法的光刻区调度方法[J]. 华中科技大学学报 (自然科学版), 2022, 50(4): 26-32
ZHANG Peng, ZHANG Jie, WANG Zhuojun, et al. Decomposition multi-objective evolutionary algorithm based photolithography area scheduling method[J]. Journal of Huazhong University of Science and Technology (Natural Science Edition), 2022, 50(4): 26-32
[6] ZHANG P, ZHAO X, SHENG X, et al. An imperialist competitive algorithm incorporating remaining cycle time prediction for photolithography machines scheduling[J]. IEEE Access, 2018, 6: 66787-66797
[7] ZHANG Z, ZHENG L, LI N, et al. Minimizing mean weighted tardiness in unrelated parallel machine scheduling with reinforcement learning[J]. Computers & Operations Research, 2012, 39(7): 1315-1324
[8] ZHANG T, XIE S, ROSE O. Real-time batching in job shops based on simulation and reinforcement learning[C]//Proceedings of the 2018 Winter Simulation Conference (WSC) . Gothenburg: IEEE Press, 2018: 3331-3339.
[9] VAN HASSELT H, GUEZ A, SILVER D. Deep reinforcement learning with double q-learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Phoenix: AAAI Press, 2016.
[10] MENG L, GORBET R, KULIĆ D. The effect of multi-step methods on overestimation in deep reinforcement learning[C]//Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR) . Milan: IEEE Press, 2021: 347-353.
[11] LIU R, PIPLANI R, TORO C. Deep reinforcement learning for dynamic scheduling of a flexible job shop[J]. International Journal of Production Research, 2022, 60(13): 4049-4069
[12] WANG M, ZHANG J, ZHANG P, et al. Independent double DQN-based multi-agent reinforcement learning approach for online two-stage hybrid flow shop scheduling with batch machines[J]. Journal of Manufacturing Systems, 2022, 65: 694-708
[13] LEE D, KANG H, LEE D, et al. Deep reinforcement learning-based scheduler on parallel dedicated machine scheduling problem towards minimizing total tardiness[J]. Sustainability, 2023, 15(4): 2920
[14] PAENG B, PARK I B, PARK J. Deep reinforcement learning for minimizing tardiness in parallel machine scheduling with sequence dependent family setups[J]. IEEE Access, 2021, 9: 101390-101401
[15] GAO S, HUANG Y, ZHANG S, et al. Short-term runoff prediction with GRU and LSTM networks without requiring time step optimization during sample generation[J]. Journal of Hydrology, 2020, 589: 125188
[16] ZHANG K, CHEN N, LIU J, et al. A GRU-based ensemble learning method for time-variant uncertain structural response analysis[J]. Computer Methods in Applied Mechanics and Engineering, 2022, 391: 114516
[17] OUKIL A, EL-BOURI A. Ranking dispatching rules in multi-objective dynamic flow shop scheduling: a multi-faceted perspective[J]. International Journal of Production Research, 2021, 59(2): 388-411