Hostname: page-component-cd9895bd7-dzt6s Total loading time: 0 Render date: 2025-01-03T16:17:04.415Z Has data issue: false hasContentIssue false
  • English
  • Français

NMPC-based visual path following control with variable perception horizon

Published online by Cambridge University Press:  02 May 2023

Tiago T. Ribeiro Open the ORCID record for Tiago T. Ribeiro [Opens in a new window] ,
Iago José P. B. Franco  and
André Gustavo S. Conceição Open the ORCID record for André Gustavo S. Conceição [Opens in a new window]
Tiago T. Ribeiro*
Affiliation:
LaR - Robotics Laboratory, Department of Electrical and Computer Engineering, Federal University of Bahia, Salvador, Bahia, Brazil
Iago José P. B. Franco
Affiliation:
LaR - Robotics Laboratory, Department of Electrical and Computer Engineering, Federal University of Bahia, Salvador, Bahia, Brazil
André Gustavo S. Conceição
Affiliation:
LaR - Robotics Laboratory, Department of Electrical and Computer Engineering, Federal University of Bahia, Salvador, Bahia, Brazil
*
Corresponding author: Tiago Ribeiro; Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

For greater autonomy of visual control-based solutions, especially applied to mobile robots, it is necessary to consider the existence of unevenness in the navigation surface, an intrinsic characteristic of several real applications. In general, depth information is essential for navigating three-dimensional environments and for the consistent parameter calibration of the visual models. This work proposes a new solution, including depth information in the visual path-following (VPF) problem, which allows the variation of the perception horizon at runtime while forcing the coupling between optical and geometric quantities. A new NMPC (nonlinear model predictive control) framework considering the addition of a new input to an original solution for the constrained VPF-NMPC allows the maintenance of low computational complexity. Experimental results in an outdoor environment with a medium-sized commercial robot demonstrate the correctness of the proposal.

Keywords

vision-based controlrobot navigationnonlinear model predictive controlcomputer visionpath followingoptimal control
Type
Research Article
Information
Robotica , Volume 41 , Issue 8 , August 2023 , pp. 2552 - 2570
Copyright
© The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

R. Ghosh, R. Pragathi, S. Ullas and S. Borra, “Intelligent transportation systems: A survey,” 2017 International Conference on Circuits, Controls, and Communications (CCUBE), Bangalore, India, 2017, pp. 160–165, doi: 10.1109/CCUBE.2017.8394167.CrossRefGoogle Scholar
Zhu, F., Lv, Y., Chen, Y., Wang, X., Xiong, G. and Wang, F.-Y., “Parallel transportation systems: Toward IOT-enabled smart urban traffic control and management,” IEEE Trans. Intell. Transp. Syst. 21(10), 4063–4071 (2020).CrossRefGoogle Scholar
Baranzadeh, A. and Savkin, A. V., “A distributed control algorithm for area search by a multi-robot team,” Robotica 35(6), 1452–1472 (2017).CrossRefGoogle Scholar
Wang, H., Zhang, C., Song, Y., Pang, B. and Zhang, G., “Three-dimensional reconstruction based on visual SLAM of mobile robot in search and rescue disaster scenarios,” Robotica 38(2), 350–373 (2020).CrossRefGoogle Scholar
Costa, F. S., Nassar, S. M., Gusmeroli, S., Schultz, R., Conceição, A.é G. S., Xavier, M., Hessel, F. and Dantas, M. A. R., “FASTEN IIoT: An open real-time platform for vertical, horizontal and end-to-end integration,” Sensors 20(19), 5499 (2020).CrossRefGoogle ScholarPubMed
Yekkehfallah, M., Yang, M., Cai, Z., Li, L. and Wang, C., “Accurate 3D localization using RGB-TOF camera and IMU for industrial mobile robots,” Robotica 39(10), 1816–1833 (2021).CrossRefGoogle Scholar
Qi, R., Tang, Y. and Zhang, K., “An optimal visual servo trajectory planning method for manipulators based on system nondeterministic model,” Robotica 40(6), 1665–1681 (2022).CrossRefGoogle Scholar
Jin, Z., Wu, J., Liu, A., Zhang, W.-A. and Yu, L., “Policy-based deep reinforcement learning for visual servoing control of mobile robots with visibility constraints,” IEEE Trans. Ind. Electron. 69(2), 1898–1908 (2022).CrossRefGoogle Scholar
Huang, C., Xu, T., Liu, J., Manamanchaiyaporn, L. and Wu, X., “Visual servoing of miniature magnetic film swimming robots for 3-D arbitrary path following,” IEEE Robot. Autom. Lett. 4(4), 4185–4191 (2019).CrossRefGoogle Scholar
Aldana-Murillo, N. G., Sandoval, L., Hayet, J.-B., Esteves, C. and Becerra, H. M., “Coupling humanoid walking pattern generation and visual constraint feedback for pose-regulation and visual path-following,” Robot. Auton. Syst. 128, 103497 (2020).CrossRefGoogle Scholar
Martinez, E. A. R., Caron, G., Pégard, C. and Lara-Alabazares, D., “Photometric-planner for visual path following,” IEEE Sens. J. 21(10), 11310–11317 (2021).CrossRefGoogle Scholar
Wang, Y., Sun, Q., Liu, Z. and Gu, L., “Visual detection and tracking algorithms for minimally invasive surgical instruments: A comprehensive review of the state-of-the-art,” Robot. Auton. Syst. 149, 103945 (2021).Google Scholar
Allan, D. A., Bates, C. N., Risbeck, M. J. and Rawlings, J. B., “On the inherent robustness of optimal and suboptimal nonlinear MPC,” Syst. Control Lett. 106, 68–78 (2017).CrossRefGoogle Scholar
Husmann, R. and Aschemann, H., “Comparison and benchmarking of NMPC for swing-up and side-stepping of an inverted pendulum with underlying velocity control,” IFAC-PapersOnLine 54(14), 263–268 (2021), 3rd IFAC Conference on Modelling, Identification and Control of Nonlinear Systems MICNON 2021.CrossRefGoogle Scholar
Chacko, K., Sivaramakrishnan, J. and Kar, I., “Computationally efficient nonlinear MPC for discrete system with disturbances,” Int. J. Control Autom. Syst. 20(6), 1–10 (2022).CrossRefGoogle Scholar
Reinhold, J., Baumann, H. and Meurer, T., “Constrained-differential-kinematics-decomposition-based NMPC for online manipulator control with low computational costs,” Robotics 12(1), 7 (2023).CrossRefGoogle Scholar
Diosi, A., Remazeilles, A., Segvic, S. and Chaumette, F.. "Outdoor Visual Path Following Experiments," 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA (2007) pp. 4265–4270, doi: 10.1109/IROS.2007.4399247.CrossRefGoogle Scholar
Safia, F. and Fatima, C.. "Visual Path Following by an Omnidirectional Mobile Robot Using 2D Visual Servoing," 2017 5th International Conference on Electrical Engineering - Boumerdes (ICEE-B), Boumerdes, Algeria (2017), pp. 1–7, doi: 10.1109/ICEE-B.2017.8192097.CrossRefGoogle Scholar
Bai, G., Liu, L., Meng, Y., Luo, W., Gu, Q. and Ma, B., “Path tracking of mining vehicles based on nonlinear model predictive control,” Appl. Sci. 9(7), 1372 (2019).CrossRefGoogle Scholar
Grigorescu, S., Ginerica, C., Zaha, M., Macesanu, G. and Trasnea, B., “LVD-NMPC: A learning-based vision dynamics approach to nonlinear model predictive control for autonomous vehicles,” Int. J. Adv. Robot. Syst. 18(3), 17298814211019544 (2021).CrossRefGoogle Scholar
Zakaria, N. J., Shapiai, M. I., Ghani, R. A., Yassin, M. N. M., Ibrahim, M. Z. and Wahid, N., “Lane detection in autonomous vehicles: A systematic review,” IEEE Access 11, 3729–3765 (2023).CrossRefGoogle Scholar
Hu, Y., Chen, Z. and Lin, W.. RGB-D Semantic Segmentation: A Review. In: 2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Los Alamitos, CA, USA: IEEE Computer Society (2018) pp. 1–6.Google Scholar
Zhang, Y., Yang, Y., Xiong, C., Sun, G. and Guo, Y., Attention-Based Dual Supervised Decoder for RGBD Semantic Segmentation, ArXiv, abs/2201.01427, (2022).Google Scholar
Ostafew, C. J., Schoellig, A. P. and Barfoot, T. D., “Robust constrained learning-based NMPC enabling reliable mobile robot path tracking,” Int. J. Robot. Res. 35(13), 1547–1563 (2016).CrossRefGoogle Scholar
Kumar, A., Gupta, S., Fouhey, D., Levine, S. and Malik, J., “Visual Memory for Robust Path Following,” In: Advances in Neural Information Processing Systems (Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N. and Garnett, R., eds.), Vol. 31 (Curran Associates, Inc., Red Hook, NY, USA, 2018) pp. 765–774.Google Scholar
Maldonado-Ramirez, A., Rios-Cabrera, R. and Lopez-Juarez, I., “A visual path-following learning approach for industrial robots using DRL,” Robot. Comput. Integr. Manuf. 71, 102130 (2021).CrossRefGoogle Scholar
Ribeiro, T. T. and Conceição, A. G. S., “Nonlinear model predictive visual path following control to autonomous mobile robots,” J. Intell. Robot. Syst. 95(2), 731–743 (2019).CrossRefGoogle Scholar
Franco, I. J. P. B., Ribeiro, T. T. and Conceição, A. G. S., “A novel visual lane line detection system for a NMPC-based path following control scheme,” J. Intell. Robot. Syst. 101(1), 12 (2021).CrossRefGoogle Scholar
Arrais, R., Veiga, G., Ribeiro, T. T., Oliveira, D., Fernandes, R., Conceição, A. G. S. and Farias, P. C. M. A., “Application of the Open Scalable Production System to Machine Tending of Additive Manufacturing Operations by a Mobile Manipulator,” In: Progress in Artificial Intelligence, (Oliveira, P. M., Novais, P. and Reis, L. P., eds.), (Springer International Publishing, Cham, 2019) pp. 345–356.CrossRefGoogle Scholar
Spellucci, P., “An SQP Method for General Nonlinear Programs Using Only Equality Constrained Subproblems,” In: Mathematical Programming, Vol. 82, (1998) pp. 413–448. https://doi.org/10.1007/BF01580078.Google Scholar

Ribeiro et al. supplementary material

Ribeiro et al. supplementary material 1

Download Ribeiro et al. supplementary material(Video)
Video 20 MB
Supplementary material: File

Ribeiro et al. supplementary material

Ribeiro et al. supplementary material 2

Download Ribeiro et al. supplementary material(File)
File 49.2 MB