Low-Trust Trajectory Design from LEO to GEO Using Reinforcement Learning

Document Type : Original Article

Authors
Hamed Soleymani 1
Majid Bakhtiari 2
Kamran Daneshjou 3
1 PhD student in aerospace engineering at Iran University of Science and Technology
2 School of Advanced Technologies - Iran University of Science and Technology
3 Faculty member of Iran University of Science and Technology
10.22034/jssta.2024.446359.1153
Abstract
In this paper, In the preliminary phase of space mission design, the selection of the spacecraft's trajectory is critical. This study simulates the dynamics of low-thrust orbital transfers within a two-dimensional orbital plane, employing a set of ordinary differential equations to represent the continuum of the spacecraft's orbital elements. These elements are encapsulated by six orbital parameters, manipulated under a defined thrust vector strategy within the action space, adhering to a specified policy framework. An agent, trained via a reinforcement learning algorithm within an actor-critic network architecture, is tasked with executing a low-thrust transfer between Low Earth Orbit (LEO) and Geostationary Orbit (GEO). The algorithm dynamically adjusts the spacecraft's trajectory, informed by initial orbital conditions and mission-specific constraints, to derive an optimal thrust angle trajectory and corresponding adjustments in orbital elements for the maneuver. To validate the algorithm's efficacy and robustness, a comparative analysis is conducted by implementing an alternative transfer mode at a varied orbital altitude. Additionally, the study explores the impact of adjusting the algorithm's degradation coefficient hyperparameter on the learning efficacy. Conclusively, the findings suggest that the agent, once adequately trained within the specified dynamical model, is capable of autonomously executing analogous orbital transfers. This is achieved without necessitating reiteration of the dynamical simulations, contingent solely upon the stipulation of initial and terminal orbital parameters
Keywords
Low
Thrust# Equinoctial orbital elements# Reinforcement Learning# Actor# Critic networks# Agent
Subjects
[1]    Robert E. Pritchett, “Numerical methods for low-thrust trajectory optimization”, Purdue University West Lafayette, Indiana, August 2016.
[2]    J. B. Caillau, J. Gergaud, and J. Noailles. 3D geosynchronous transfer of a satellite: Continuation on the thrust. Journal of Optimization Theory and Applications, 118 (3):541–565, September 2003.
[3]   Mischa Kim, “Continuous Low-Thrust Trajectory Optimization: Techniques and Applications”,Doctor of Philosophy Dissertation in Aerospace Engineering ,April 18, 2005
[4]   Ryan P. Russell, “Primer Vector Theory Applied to Global Low-Thrust Trade Studies”, California Institute of Technology, Pasadena, California 91109 , JOURNAL OF GUIDANCE, CONTROL, AND DYNAMICS, 2007 , DOI: 10.2514/1.22984
[5]   Weipeng Li Hai Huang,"Optimization of low-thrust transfers to libration point orbits", Aircraft Engineering and Aerospace Technology: An International Journal (2016), Vol. 88 Iss 1 pp. 16 – 23
[6]    Mauro Pontani · Bruce Conway, “Optimal Low-Thrust Orbital Maneuvers via Indirect Swarming Method” Accepted: 29 October 2013, Springer Science+Business Media New York 2013, DOI 10.1007/s10957-013-0471-9
[7]    P. Enright and B. A. Conway, “Discrete approximations to optimal trajectories using direct transcription and Nonlinear Programming,” Journal of Guidance Control and Dynamics, Vol. 15, 09 1992, 10.2514/3.20934.
[8]    Jonathan D. Aziz, Jeffrey S. Parker, Daniel J. Scheeres, Jacob A. Englander, “Low-Thrust Many-Revolution Trajectory Optimization via Differential Dynamic Programming and a Sundman Transformation”, American Astronautical Society, part of Springer Nature 2018
[9]    S. N. Williams and V. Coverstone-Carroll. Mars missions using solar electric propulsion. Journal of Spacecraft and Rockets, 37(1):71–77, January–February 2000.
[10]    Zhenbo Wang and Michael J. Grant, “Minimum-Fuel  Low-Thrust Transfers for Spacecraft: A Convex Approach” School of Aeronautics and Astronautics, Purdue University, West Lafayette, Indiana, 47907-2045, USA, DOI 10.1109/TAES.2018.2812558, IEEE Transactions on Aerospace and Electronic Systems 2018
[11]   Daniel S. Kolosa, “A Reinforcement Learning Approach to Spacecraft Trajectory Optimization”, Western Michigan University, Dissertations. 3542,2019
[12]    M.P. van Hoorn, ”OPTIMIZING AIR-TO-AIR MISSILE GUIDANCE USING REINFORCEMENT LEARNING”, Master of Science thesis in Aerospace Engineering, Delft University of TechnologyMarch 26, 2019
[13]   T.A.H. Kranen, “Low-Thrust Gravity Assist Trajectory Optimisation using Evolutionary Neurocontrol”, Dissertation TU Delft Aerospace Engineering, 2019.
[14]   Howard D. Curtis, “Orbital Mechanics for Engineering Students”, Third Edition, season 6, p.g 344, 2014
[15]    Richard S. Sutton and Andrew G. Barto,Reinforcement Learning:An Introduction, MIT Press, Cambridge, MA, 1998A Bradford Book
[16]   M. Navabi, E. Meshkinfam, "Space low-thrust trajectory optimization utilizing numerical techniques, a comparative study," 2013 6th International Conference on Recent Advances in Space Technologies (RAST), IEEE, 2013, pp. 303-307, doi: 10.1109/RAST.2013.6581222.
[17] Brian Gaudet, Roberto Furfaro, Richard Linares, “Reinforcement Learning for Angle-Only Intercept Guidance of Maneuvering Targets”, Aerospace Science and Technology,Volume 99, April 2020
[18] Navabi, M., & Sabatifar, M. (2010). Optimal Impulsive Maneuver Between Elliptical Coplanar-Noncoaxial Orbits. Space Science and Technology, 3(1), 67-74.
[19] Navaee, M., & Sanati, M. (2013). Optimal Impulsive Orbital 3D Maneuver with or without Time Constraint. Amirkabir Journal of Mechanical Engineering, 44(2), 53-69.
[20] Fakoor, M., Sadeghi, S., & Bakhtiari, M. (2020). Investigation of low thrust optimal orbital transfer from LEO to GEO considering circular orbits. The Journal of the Astronautical Sciences, 67, 77-97.

  • Receive Date 03 March 2024
  • Revise Date 04 May 2024
  • Accept Date 18 May 2024