Application of Deep Reinforcement Learning Control of an Inverted Hydraulic Pendulum
DOI:
https://doi.org/10.13052/ijfp1439-9776.2429Keywords:
Reinforcement Learning Control, Hydraulic Control, Learning Control, Inverted Hydraulic Pendulum, Deep Reinforcement LearningAbstract
Deep reinforcement learning (RL) control is an emerging branch of machine learning focusing on data-driven solutions to complex nonlinear optimal control problems by trial-and-error learning. This study aims to apply deep reinforcement learning control to a hydromechanical system. The investigated system is an inverted pendulum on a cart with a hydraulic drive. The focus lies on implementing a comprehensive framework for the deep RL controller, which allows for training a control strategy in simulation and solving the tasks of swinging the pendulum up and balancing it. The RL controller can solve these challenges successfully; therefore, reinforcement learning presents a possibility for novel data-driven control approaches for hydromechanical systems.
Downloads
References
J. Mohieddine, Hydraulic Servo-systems: modeling, identification, and control, In: Springer-Verlag London Limited, doi: 10.1007/978-1-4471-0099-7, 2003.
R. Sutton, A. Barto, Reinforcement Learning: An Introduction, In: The MIT Press, Second edition, 2018.
R. Sutton, A. Barto, R. Williams, Reinforcement Learning is Direct Adaptive Optimal Control, In: IEEE Control Systems Magazine, vol. 12, no. 2, pp. 19–22, doi: 10.1109/37.126844, 1992.
S. Adam, L. Busoniu, and R. Babuska, “Experience replay for real-time reinforcement learning control,” Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, vol. 42, pp. 201–212, 2012.
F. Brumand-Poor, Control of a Hydromechanical Pendulum with a Reinforcement Learning Agent, In: Proceedings of the 13th International Fluid Power Conference, Aachen, 2022.
D. Bertsekas, Reinforcement Learning and Optimal Control, In: MIT, February 2019.
S. Gottschalk, Differences and similarities between reinforcement learning and the classical optimal control framework, in PAMM. 19. doi: 10.1002/pamm.201900390, 2019.
V. Mnih, Human-level control through deep reinforcement learning, Nature 518, 529–533, https://doi.org/10.1038/nature14236, 2015.
D. Silver, Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484, 2016.
D. Hafner, Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018.
P. Egli, A general approach for the automation of hydraulic excavator arms using reinforcement learning, IEEE Robotics and Automation Letters, 2021.
J. Andersson, Reinforcement learning control of a forestry crane manipulator, arXiv, 2021.
M. Karpenko, Coordination of hydraulic manipulators by reinforcement learning, in 2006 American Control Conference, pp. 6, 2006.
M. Karpenko, Decentralized Coordinated Motion Control of Two Hydraulic Actuators Handling a Common Object, Journal of Dynamic Systems, Measurement, and Control, vol. 129, no. 5, pp. 729–741, 2007.
S. Wang, Reinforcement learning control for biped robot walking on uneven surfaces, in The 2006 IEEE International Joint Conference on Neural Network Proceedings, pp. 4173–4178, 2006.
S. Satheeshbabu, Open loop position control of soft continuum arm using deep reinforcement learning, in 2019 International Conference on Robotics and Automation (ICRA), pp. 5133–5139, 2019.
K. Schmitz, and H. Murrenhoff,, Grundlagen der Fluidtechnik, Shaker Verlag, ISBN: 978-3-8440-6246-5, 2018
Translational friction: Friction in contact between moving bodies. [Online]. Available: https://de.mathworks.com/help/simscape/ref/translationalfriction.html, 2022.
C. D. Green. Equations of motion for the cart and pole control task. [Online]. Available: https://sharpneat.sourceforge.io/research/cart-pole/cart-pole-equations.html, 2020.
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” arXiv, 2017.
J. Schulman et al., “Proximal Policy Optimization Algorithms”, in CoRR, arXiv: 1707.06347, 2017.
X. Steenbrugge. An introduction to policy gradient methods - deep reinforcement learning. [Online]. Available: https://www.youtube.com/watch?v=5P7I-xPq8u8, 2018.
J. Achiam, “Spinning up in deep reinforcement learning,” GitHub repository, 2018.
S. (https://stackoverflow.com/users/754136/simon), “How to choose the reward function for the cart-pole inverted pendulum task,”, RL: https://stackoverflow.com/revisions/51684009/2, 2018.
