Hostname: page-component-5cf477f64f-h6p2m Total loading time: 0 Render date: 2025-04-03T18:15:46.037Z Has data issue: false hasContentIssue false
  • English
  • Français

Reinforcement-learning-based control of confined cylinder wakes with stability analyses

Published online by Cambridge University Press:  14 December 2021

Jichao Li Open the ORCID record for Jichao Li [Opens in a new window]  and
Mengqi Zhang Open the ORCID record for Mengqi Zhang [Opens in a new window]
Jichao Li
Affiliation:
Department of Mechanical Engineering, National University of Singapore, 117575 Republic of Singapore
Mengqi Zhang*
Affiliation:
Department of Mechanical Engineering, National University of Singapore, 117575 Republic of Singapore
*
†Email address for correspondence: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

This work studies the application of a reinforcement learning (RL)-based flow control strategy to the flow past a cylinder confined between two walls to suppress vortex shedding. The control action is blowing and suction of two synthetic jets on the cylinder. The theme of this study is to investigate how to use and embed physical information of the flow in the RL-based control. First, global linear stability and sensitivity analyses based on the time-mean flow and the steady flow (which is a solution to the Navier–Stokes equations) are conducted in a range of blockage ratios and Reynolds numbers. It is found that the most sensitive region in the wake extends itself when either parameter increases in the parameter range we investigated here. Then, we use these physical results to help design RL-based control policies. We find that the controlled wake converges to the unstable steady base flow, where the vortex shedding can be successfully suppressed. A persistent oscillating control seems necessary to maintain this unstable state. The RL algorithm is able to outperform a gradient-based optimisation method (optimised in a certain period of time) in the long run. Furthermore, when the flow stability information is embedded in the reward function to penalise the instability, the controlled flow may become more stable. Finally, according to the sensitivity analyses, the control is most efficient when the probes are placed in the most sensitive region. The control can be successful even when few probes are properly placed in this manner.

JFM classification

Mathematical Foundations: Machine learning Wakes/Jets: Wakes Flow Control: Flow Control
Type
JFM Papers
Information
Journal of Fluid Mechanics , Volume 932 , 10 February 2022 , A44
Check for updates
Copyright
© The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Akervik, E., Brandt, L., Henningson, D.S., Hoepffner, J., Marxen, O. & Schlatter, P. 2006 Steady solutions of the Navier–Stokes equations by selective frequency damping. Phys. Fluids 18 (6), 068102.CrossRefGoogle Scholar
Anagnostopoulos, P., Iliadis, G. & Richardson, S. 1996 Numerical study of the blockage effects on viscous flow past a circular cylinder. Intl J. Numer. Meth. Fluids 22 (11), 1061–1074.3.0.CO;2-Q>CrossRefGoogle Scholar
Arnoldi, W.E. 1951 The principle of minimized iterations in the solution of the matrix eigenvalue problem. Q. Appl. Maths 9 (1), 17–29.CrossRefGoogle Scholar
Barkley, D. 2006 Linear analysis of the cylinder wake mean flow. Europhys. Lett. (EPL) 75 (5), 750–756.CrossRefGoogle Scholar
Barkley, D., Blackburn, H.M. & Sherwin, S.J. 2008 Direct optimal growth analysis for timesteppers. Intl J. Numer. Meth. Fluids 57 (9), 1435–1458.CrossRefGoogle Scholar
Beintema, G., Corbetta, A., Biferale, L. & Toschi, F. 2020 Controlling Rayleigh–Bénard convection via reinforcement learning. J. Turbul. 21 (9-10), 585–605.CrossRefGoogle Scholar
Belus, V., Rabault, J., Viquerat, J., Che, Z., Hachem, E. & Reglade, U. 2019 Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film. AIP Adv. 9 (12), 125014.CrossRefGoogle Scholar
Boujo, E. & Gallaire, F. 2014 Controlled reattachment in separated flows: a variational approach to recirculation length reduction. J. Fluid Mech. 742, 618–635.CrossRefGoogle Scholar
Brandt, L., Sipp, D., Pralits, J.O. & Marquet, O. 2011 Effect of base-flow variation in noise amplifiers: the flat-plate boundary layer. J. Fluid Mech. 687, 503–528.CrossRefGoogle Scholar
Brunton, S.L., Noack, B.R. & Koumoutsakos, P. 2020 Machine learning for fluid mechanics. Annu. Rev. Fluid Mech. 52 (1), 477–508.CrossRefGoogle Scholar
Brunton, S.L., Proctor, J.L. & Kutz, J.N. 2016 Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl Acad. Sci. 113 (15), 3932–3937.CrossRefGoogle ScholarPubMed
Budanur, N.B. & Cvitanović, P. 2017 Unstable manifolds of relative periodic orbits in the symmetry-reduced state space of the Kuramoto–Sivashinsky system. J. Stat. Phys. 167 (3), 636–655.CrossRefGoogle Scholar
Chen, J.-H., Pritchard, W.G. & Tavener, S.J. 1995 Bifurcation for flow past a cylinder between parallel planes. J. Fluid Mech. 284, 23–41.CrossRefGoogle Scholar
Corbett, P. & Bottaro, A. 2001 Optimal control of nonmodal disturbances in boundary layers. Theor. Comput. Fluid Dyn. 15 (2), 65–81.CrossRefGoogle Scholar
Coutanceau, M. & Bouard, R. 1977 Experimental determination of the main features of the viscous flow in the wake of a circular cylinder in uniform translation. Part 1. Steady flow. J. Fluid Mech. 79 (2), 231–256.CrossRefGoogle Scholar
Delaunay, Y. & Kaiktsis, L. 1999 Active control of cylinder wakes: use of base suction and blowing. ESAIM: Proc. 7, 104–119.CrossRefGoogle Scholar
Duraisamy, K., Iaccarino, G. & Xiao, H. 2019 Turbulence modeling in the age of data. Annu. Rev. Fluid Mech. 51 (1), 357–377.CrossRefGoogle Scholar
Eriksson, L.E. & Rizzi, A. 1985 Computer-aided analysis of the convergence to steady state of discrete approximations to the euler equations. J. Comput. Phys. 57 (1), 90–128.CrossRefGoogle Scholar
Fan, D., Yang, L., Wang, Z., Triantafyllou, M.S. & Karniadakis, G.E. 2020 Reinforcement learning for bluff body active flow control in experiments and simulations. Proc. Natl Acad. Sci. 117 (42), 26091–26098.CrossRefGoogle ScholarPubMed
Fischer, P.F., Lottes, J.W. & Kerkemeier, S.G. 2017 Nek5000 Version 17.0. Argonne National Laboratory, Illinois. Available at: https://nek5000.mcs.anl.gov.Google Scholar
Flinois, T.L.B. & Colonius, T. 2015 Optimal control of circular cylinder wakes using long control horizons. Phys. Fluids 27 (8), 087105.CrossRefGoogle Scholar
Gao, C., Zhang, W., Kou, J., Liu, Y. & Ye, Z. 2017 Active control of transonic buffet flow. J. Fluid Mech. 824, 312–351.CrossRefGoogle Scholar
Garnier, P., Viquerat, J., Rabault, J., Larcher, A., Kuhnle, A. & Hachem, E. 2021 A review on deep reinforcement learning for fluid mechanics. Comput. Fluids 225, 104973.CrossRefGoogle Scholar
Giannetti, F. & Luchini, P. 2007 Structural sensitivity of the first instability of the cylinder wake. J. Fluid Mech. 581, 167–197.CrossRefGoogle Scholar
Glezer, A. 2011 Some aspects of aerodynamic flow control using synthetic-jet actuation. Phil. Trans. R. Soc. A: Math. Phys. Engng Sci. 369 (1940), 1476–1494.CrossRefGoogle ScholarPubMed
Hervé, A., Sipp, D., Schmid, P.J. & Samuelides, M. 2012 A physics-based approach to flow control using system identification. J. Fluid Mech. 702, 26–58.CrossRefGoogle Scholar
Hill, D. 1992 A theoretical approach for analyzing the restabilization of wakes. In 30th Aerospace Sciences Meeting and Exhibit. AIAA Paper 1992-67.CrossRefGoogle Scholar
Huerre, P. & Monkewitz, P.A. 1990 Local and global instabilities in spatially developing flows. Annu. Rev. Fluid Mech. 22, 473–537.CrossRefGoogle Scholar
Jackson, C.P. 1987 A finite-element study of the onset of vortex shedding in flow past variously shaped bodies. J. Fluid Mech. 182 (-1), 23.CrossRefGoogle Scholar
Jameson, A. 1988 Aerodynamic design via control theory. J. Sci. Comput. 3 (3), 233–260.CrossRefGoogle Scholar
Jonsson, E., Riso, C., Lupp, C.A., Cesnik, C.E., Martins, J.R. & Epureanu, B.I. 2019 Flutter and post-flutter constraints in aircraft design optimization. Prog. Aerosp. Sci. 109, 100537.CrossRefGoogle Scholar
Kanaris, N., Grigoriadis, D. & Kassinos, S. 2011 Three dimensional flow around a circular cylinder confined in a plane channel. Phys. Fluids 23 (6), 064106.CrossRefGoogle Scholar
Kim, J. & Bewley, T.R. 2007 A linear systems approach to flow control. Annu. Rev. Fluid Mech. 39 (1), 383–417.CrossRefGoogle Scholar
Kuhnle, A., Schaarschmidt, M. & Fricke, K. 2017 Tensorforce: a tensorflow library for applied reinforcement learning. Web page.Google Scholar
Leontini, J.S., Thompson, M.C. & Hourigan, K. 2010 A numerical study of global frequency selection in the time-mean wake of a circular cylinder. J. Fluid Mech. 645, 435–446.CrossRefGoogle Scholar
Ling, J., Kurzawski, A. & Templeton, J. 2016 Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. J. Fluid Mech. 807, 155–166.CrossRefGoogle Scholar
Loiseau, J.-C., Noack, B.R. & Brunton, S.L. 2018 Sparse reduced-order modelling: sensor-based dynamics to full-state estimation. J. Fluid Mech. 844, 459–490.CrossRefGoogle Scholar
Luchini, P. & Bottaro, A. 2014 Adjoint equations in stability analysis. Annu. Rev. Fluid Mech. 46 (1), 493–517.CrossRefGoogle Scholar
Maceda, G.Y.C., Li, Y., Lusseyran, F., Morzyński, M. & Noack, B.R. 2021 Stabilization of the fluidic pinball with gradient-enriched machine learning control. J. Fluid Mech. 917, A42.Google Scholar
Marquet, O., Sipp, D. & Jacquin, L. 2008 Sensitivity analysis and passive control of cylinder flow. J. Fluid Mech. 615, 221–252.CrossRefGoogle Scholar
Maurel, A., Pagneux, V. & Wesfreid, J.E. 1995 Mean-flow correction as non-linear saturation mechanism. Europhys. Lett. (EPL) 32 (3), 217–222.CrossRefGoogle Scholar
Noack, B.R., Afanasiev, K., Morzyński, M., Tadmor, G. & Thiele, F. 2003 A hierarchy of low-dimensional models for the transient and post-transient cylinder wake. J. Fluid Mech. 497, 335–363.CrossRefGoogle Scholar
Noack, B.R. & Eckelmann, H. 1994 A global stability analysis of the steady and periodic cylinder wake. J. Fluid Mech. 270, 297–330.CrossRefGoogle Scholar
Paris, R., Beneddine, S. & Dandois, J. 2021 Robust flow control and optimal sensor placement using deep reinforcement learning. J. Fluid Mech. 913, A25.CrossRefGoogle Scholar
Pastoor, M., Henning, L., Noack, B.R., King, R. & Tadmor, G. 2008 Feedback shear layer control for bluff body drag reduction. J. Fluid Mech. 608, 161–196.CrossRefGoogle Scholar
Peplinski, A., Schlatter, P., Fischer, P.F. & Henningson, D.S. 2014 Stability tools for the spectral-element code NEK5000: application to jet-in-crossflow. In Spectral and High Order Methods for Partial Differential Equations - ICOSAHOM 2012 (ed. M. Azaïez, H. El Fekih & J.S. Hesthaven), pp. 349–359. Springer International Publishing.CrossRefGoogle Scholar
Perez, R.E., Jansen, P.W. & Martins, J.R.R.A. 2012 pyOpt: a Python-based object-oriented framework for nonlinear constrained optimization. Struct. Multidiscipl. Optim. 45 (1), 101–118.CrossRefGoogle Scholar
Pier, B. 2002 On the frequency selection of finite-amplitude vortex shedding in the cylinder wake. J. Fluid Mech. 458, 407–417.CrossRefGoogle Scholar
Pier, B. & Huerre, P. 2001 Nonlinear self-sustained structures and fronts in spatially developing wake flows. J. Fluid Mech. 435, 145–174.CrossRefGoogle Scholar
Provansal, M., Mathis, C. & Boyer, L. 1987 Bénard-von Kármán instability: transient and forced regimes. J. Fluid Mech. 182, 1–22.CrossRefGoogle Scholar
Rabault, J., Kuchta, M., Jensen, A., Réglade, U. & Cerardi, N. 2019 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. J. Fluid Mech. 865, 281–302.CrossRefGoogle Scholar
Rabault, J. & Kuhnle, A. 2019 Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Phys. Fluids 31 (9), 094105.CrossRefGoogle Scholar
Rabault, J., Ren, F., Zhang, W., Tang, H. & Xu, H. 2020 Deep reinforcement learning in fluid mechanics: a promising method for both active flow control and shape optimization. J. Hydrodyn. 32 (2), 234–246.CrossRefGoogle Scholar
Raissi, M., Babaee, H. & Givi, P. 2019 a Deep learning of turbulent scalar mixing. Phys. Rev. Fluids 4, 124501.CrossRefGoogle Scholar
Raissi, M., Perdikaris, P. & Karniadakis, G. 2019 b Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686–707.CrossRefGoogle Scholar
Raissi, M., Wang, Z., Triantafyllou, M.S. & Karniadakis, G.E. 2018 Deep learning of vortex-induced vibrations. J. Fluid Mech. 861, 119–137.CrossRefGoogle Scholar
Ren, F., Rabault, J. & Tang, H. 2021 Applying deep reinforcement learning to active flow control in weakly turbulent conditions. Phys. Fluids 33 (3), 037121.Google Scholar
Rowley, C.W., Mezić, I., BAGHERI, S., SCHLATTER, P. & HENNINGSON, D.S. 2009 Spectral analysis of nonlinear flows. J. Fluid Mech. 641, 115–127.CrossRefGoogle Scholar
von Rueden, L., et al. 2021 Informed machine learning - a taxonomy and survey of integrating prior knowledge into learning systems. IEEE Trans. Knowledge Data Engng. https://doi.org/10.1109/TKDE.2021.3079836CrossRefGoogle Scholar
Saad, Y. 1980 Variations on arnoldi method for computing eigenelements of large unsymmetric matrices. Linear Algebra Appl. 34, 269–295.CrossRefGoogle Scholar
Sahin, M. & Owens, R.G. 2004 A numerical investigation of wall effects up to high blockage ratios on two-dimensional flow past a confined circular cylinder. Phys. Fluids 16 (5), 1305–1320.CrossRefGoogle Scholar
Schäfer, M., Turek, S., Durst, F., Krause, E. & Rannacher, R. 1996 Benchmark computations of laminar flow around a cylinder. In Notes on Numerical Fluid Mechanics, pp. 547–566. Vieweg+Teubner Verlag.CrossRefGoogle Scholar
Schmid, P.J. 2010 Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 656, 5–28.CrossRefGoogle Scholar
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. 2017 Proximal policy optimization algorithms. Preprint. arXiv:1707.06347.Google Scholar
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D. & Riedmiller, M. 2014 Deterministic policy gradient algorithms. In International Conference on Machine Learning, pp. 387–395. PMLR.Google Scholar
Sipp, D. & Lebedev, A. 2007 Global stability of base and mean flows: a general approach and its applications to cylinder and open cavity flows. J. Fluid Mech. 593, 333–358.CrossRefGoogle Scholar
Sreenivasan, K., Strykowski, P. & Olinger, D. 1987 Hopf bifurcation, landau equation, and vortex shedding behind circular cylinders. In Forum on Unsteady Flow Separation, vol. 1, pp. 1–13. ASME.Google Scholar
Strykowski, P.J. & Sreenivasan, K.R. 1990 On the formation and suppression of vortex ‘shedding’ at low Reynolds numbers. J. Fluid Mech. 218, 71–107.CrossRefGoogle Scholar
Tang, H., Rabault, J., Kuhnle, A., Wang, Y. & Wang, T. 2020 Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning. Phys. Fluids 32 (5), 053605.CrossRefGoogle Scholar
Tartakovsky, A.M., Marrero, C.O., Perdikaris, P., Tartakovsky, G.D. & Barajas-Solano, D. 2020 Physics-informed deep neural networks for learning parameters and constitutive relationships in subsurface flow problems. Water Resour. Res. 56 (5), e2019WR026731.CrossRefGoogle Scholar
Tezuka, A. & Suzuki, K. 2006 Three-dimensional global linear stability analysis of flow around a spheroid. AIAA J. 44 (8), 1697–1708.CrossRefGoogle Scholar
Theofilis, V. 2011 Global linear instability. Annu. Rev. Fluid Mech. 43 (1), 319–352.CrossRefGoogle Scholar
Turton, S.E., Tuckerman, L.S. & Barkley, D. 2015 Prediction of frequencies in thermosolutal convection from mean flows. Phys. Rev. E 91 (4), 043009.CrossRefGoogle ScholarPubMed
Verma, S., Novati, G. & Koumoutsakos, P. 2018 Efficient collective swimming by harnessing vortices through deep reinforcement learning. Proc. Natl Acad. Sci. 115 (23), 5849–5854.CrossRefGoogle ScholarPubMed
Viquerat, J., Meliga, P. & Hachem, E. 2021 A review on deep reinforcement learning for fluid mechanics: an update. arXiv:2107.12206v2.Google Scholar
Williamson, C.H.K. 1996 Vortex dynamics in the cylinder wake. Annu. Rev. Fluid Mech. 28 (1), 477–539.CrossRefGoogle Scholar
Wu, N., Kenway, G., Mader, C.A., Jasa, J. & Martins, J.R.R.A. 2020 pyoptsparse: a Python framework for large-scale constrained nonlinear optimization of sparse systems. J. Open Source Softw. 5 (54), 2564.CrossRefGoogle Scholar
Xu, H., Zhang, W., Deng, J. & Rabault, J. 2020 Active flow control with rotating cylinders by an artificial neural network trained by deep reinforcement learning. J. Hydrodyn. 32 (2), 254–258.CrossRefGoogle Scholar
Yang, X. & Zebib, A. 1989 Absolute and convective instability of a cylinder wake. Phys. Fluids A: Fluid Dyn. 1 (4), 689–696.CrossRefGoogle Scholar
Zeng, K. & Graham, M.D. 2021 Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics. Phys. Rev. E 104, 014210.CrossRefGoogle ScholarPubMed