Physics-Informed Reinforcement Learning Framework for Real-Time Coordination of EV Charging with Renewable Energy Sources
DOI:
https://doi.org/10.13052/dgaej2156-3306.41411Keywords:
Electric vehicle (EV), EV charging management, physics-informed reinforcement learning (PI-RL), renewable energy sources (RES)Abstract
We made a Physics-Informed Reinforcement Learning (PI-RL) framework to coordinate the charging stations for electric vehicles (EVs) in real time. These stations are powered by different renewable energy sources (RES), like wind and photovoltaic (PV). Our methodology explicitly integrates energy conservation laws, state-of-charge (SOC) dynamics, and inverter limitations into the training process, unlike previous reinforcement learning (RL)-based methodologies that are confined to single-source renewable energy systems (RES) and do not incorporate physical system constraints. We changed the Soft Actor-Critic (SAC) algorithm by adding domain-informed reward shaping and adaptive Lagrangian multipliers in order to make sure that constraints were met. We subsequently structured the EV-RES coordination issue as a physics-constrained Markov Decision Process (MDP). We evaluated the proposed PI-RL approach using actual datasets of solar irradiance, synthetic wind generation profiles, and electric vehicle arrival patterns. In terms of operational profit, safety (constraint violation rate), and use of renewable energy, our approach worked better than traditional SAC and model-based rolling optimization. Also, our model made it much less common for charge-discharge switching to happen, which led to control strategies that are easier to understand and that help the battery last longer. These results show that the PI-RL framework is a reliable and widely applicable way to manage energy in real time in EV charging infrastructures that are getting more and more complicated as they use renewable energy.
Downloads
References
Q. Peng, W. Liu, Y. Zhang, S. Zeng, and B. Graham, “Generation planning for power companies with hybrid production technologies under multiple renewable energy policies,” Renewable and Sustainable Energy Reviews, vol. 176, p. 113209, 2023.
O. Almughram, S. Abdullah ben Slama, and B. A. Zafar, “A reinforcement learning approach for integrating an intelligent home energy management system with a vehicle-to-home unit,” Applied Sciences, vol. 13, no. 9, p. 5539, 2023.
F. Sun et al., “Prediction-based EV-PV coordination strategy for charging stations using reinforcement learning,” IEEE Trans Ind Appl, vol. 60, no. 1, pp. 910–919, 2023.
Q. Huang, Q.-S. Jia, and X. Guan, “Robust scheduling of EV charging load with uncertain wind power integration,” IEEE Trans Smart Grid, vol. 9, no. 2, pp. 1043–1054, 2016.
S. F. Aliabadi, S. A. Taher, and M. Shahidehpour, “Smart deregulated grid frequency control in presence of renewable energy resources by EVs charging control,” IEEE Trans Smart Grid, vol. 9, no. 2, pp. 1073–1085, 2016.
M. F. M. Arani and Y. A.-R. I. Mohamed, “Cooperative control of wind power generator and electric vehicles for microgrid primary frequency regulation,” IEEE Trans Smart Grid, vol. 9, no. 6, pp. 5677–5686, 2017.
J. Zhang, L. Che, X. Wan, and M. Shahidehpour, “Distributed hierarchical coordination of networked charging stations based on peer-to-peer trading and EV charging flexibility quantification,” IEEE Transactions on Power Systems, vol. 37, no. 4, pp. 2961–2975, 2021.
Y. Zheng, Y. Song, D. J. Hill, and K. Meng, “Online distributed MPC-based optimal scheduling for EV charging stations in distribution systems,” IEEE Trans Industr Inform, vol. 15, no. 2, pp. 638–649, 2018.
X. Chen, G. Qu, Y. Tang, S. Low, and N. Li, “Reinforcement learning for selective key applications in power systems: Recent advances and future challenges,” IEEE Trans Smart Grid, vol. 13, no. 4, pp. 2935–2958, 2022.
F. Sun, X. Kong, J. Wu, B. Gao, K. Chen, and N. Lu, “DSM pricing method based on A3C and LSTM under cloud-edge environment,” Appl Energy, vol. 315, p. 118853, 2022.
Y. Zheng, Y. Song, D. J. Hill, and K. Meng, “Online distributed MPC-based optimal scheduling for EV charging stations in distribution systems,” IEEE Trans Industr Inform, vol. 15, no. 2, pp. 638–649, 2018.
F. Sun et al., “Prediction-based EV-PV coordination strategy for charging stations using reinforcement learning,” IEEE Trans Ind Appl, vol. 60, no. 1, pp. 910–919, 2023.
N. Sadeghianpourhamami, J. Deleu, and C. Develder, “Definition and evaluation of model-free coordination of electrical vehicle charging with reinforcement learning,” IEEE Trans Smart Grid, vol. 11, no. 1, pp. 203–214, 2019.
Z. Ye, Y. Gao, and N. Yu, “Learning to operate an electric vehicle charging station considering vehicle-grid integration,” IEEE Trans Smart Grid, vol. 13, no. 4, pp. 3038–3048, 2022.
S. Wang, S. Bi, and Y. A. Zhang, “Reinforcement learning for real-time pricing and scheduling control in EV charging stations,” IEEE Trans Industr Inform, vol. 17, no. 2, pp. 849–859, 2019.
F. L. Da Silva, C. E. H. Nishida, D. M. Roijers, and A. H. R. Costa, “Coordination of electric vehicle charging through multiagent reinforcement learning,” IEEE Trans Smart Grid, vol. 11, no. 3, pp. 2347–2356, 2019.
L. Yan, X. Chen, Y. Chen, and J. Wen, “A cooperative charging control strategy for electric vehicles based on multiagent deep reinforcement learning,” IEEE Trans Industr Inform, vol. 18, no. 12, pp. 8765–8775, 2022.
T. Qian, C. Shao, X. Li, X. Wang, Z. Chen, and M. Shahidehpour, “Multi-agent deep reinforcement learning method for EV charging station game,” IEEE Transactions on Power Systems, vol. 37, no. 3, pp. 1682–1694, 2021.
L. Yan, X. Chen, J. Zhou, Y. Chen, and J. Wen, “Deep reinforcement learning for continuous electric vehicles charging control with dynamic user behaviors,” IEEE Trans Smart Grid, vol. 12, no. 6, pp. 5124–5134, 2021.
Z. Wan, H. Li, H. He, and D. Prokhorov, “Model-free real-time EV charging scheduling based on deep reinforcement learning,” IEEE Trans Smart Grid, vol. 10, no. 5, pp. 5246–5257, 2018.
H. Li, Z. Wan, and H. He, “Constrained EV charging scheduling based on safe deep reinforcement learning,” IEEE Trans Smart Grid, vol. 11, no. 3, pp. 2427–2439, 2019.
H. Li et al., “Constrained large-scale real-time EV scheduling based on recurrent deep reinforcement learning,” International Journal of Electrical Power & Energy Systems, vol. 144, p. 108603, 2023.
G. Guo and Y. Gong, “Energy management of intelligent solar parking lot with EV charging and FCEV refueling based on deep reinforcement learning,” International Journal of Electrical Power & Energy Systems, vol. 140, p. 108061, 2022.
M. Dorokhova, Y. Martinson, C. Ballif, and N. Wyrsch, “Deep reinforcement learning control of electric vehicle charging in the presence of photovoltaic generation,” Appl Energy, vol. 301, p. 117504, 2021.
