Hostname: page-component-5cf477f64f-xc2pj Total loading time: 0 Render date: 2025-04-02T21:37:36.532Z Has data issue: false hasContentIssue false
  • English
  • Français

Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control

Published online by Cambridge University Press:  20 February 2019

Jean Rabault Open the ORCID record for Jean Rabault [Opens in a new window] ,
Miroslav Kuchta ,
Atle Jensen ,
Ulysse Réglade  and
Nicolas Cerardi
Jean Rabault*
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway
Miroslav Kuchta
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway
Atle Jensen
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway
Ulysse Réglade
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway CEMEF, Mines ParisTech, 06904 Sophia-Antipolis, France
Nicolas Cerardi
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway CEMEF, Mines ParisTech, 06904 Sophia-Antipolis, France
*
†Email address for correspondence: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

We present the first application of an artificial neural network trained through a deep reinforcement learning agent to perform active flow control. It is shown that, in a two-dimensional simulation of the Kármán vortex street at moderate Reynolds number ($Re=100$), our artificial neural network is able to learn an active control strategy from experimenting with the mass flow rates of two jets on the sides of a cylinder. By interacting with the unsteady wake, the artificial neural network successfully stabilizes the vortex alley and reduces drag by approximately 8 %. This is performed while using small mass flow rates for the actuation, of the order of 0.5 % of the mass flow rate intersecting the cylinder cross-section once a new pseudo-periodic shedding regime is found. This opens the way to a new class of methods for performing active flow control.

JFM classification

Flow Control: Control theory Flow Control: Drag reduction
Type
JFM Papers
Information
Journal of Fluid Mechanics , Volume 865 , 25 April 2019 , pp. 281 - 302
Copyright
© 2019 Cambridge University Press 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Rudlur, M., Lovenberg, J., Monga, R., Moore, S., Steiner, B., Tuchu, P., Vasudejon, V., Warden, P., Wicke, M., Yu, Y. & Zheng, X. 2016 Tensorflow: a system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16), vol. 16, pp. 265–283.Google Scholar
Barbagallo, A., Dergham, G., Sipp, D., Schmid, P. J. & Robinet, J.-C. 2012 Closed-loop control of unsteadiness over a rounded backward-facing step. J. Fluid Mech. 703, 326–362.10.1017/jfm.2012.223Google Scholar
Barbagallo, A., Sipp, D. & Schmid, P. J. 2009 Closed-loop control of an open cavity flow using reduced-order models. J. Fluid Mech. 641, 1–50.10.1017/S0022112009991418Google Scholar
Brunton, S. L. & Noack, B. R. 2015 Closed-loop turbulence control: progress and challenges. Appl. Mech. Rev. 67 (5), 050801.10.1115/1.4031175Google Scholar
Davis, T. A. 2004 Algorithm 832: UMFPACK v4.3 – an unsymmetric-pattern multifrontal method. ACM Trans. Math. Softw. 30 (2), 196–199.10.1145/992200.992206Google Scholar
Dean, B. & Bhushan, B. 2010 Shark-skin surfaces for fluid-drag reduction in turbulent flow: a review. Phil. Trans. R. Soc. Lond. A 368 (1929), 4775–4806.10.1098/rsta.2010.0201Google Scholar
Duan, Y., Chen, X., Houthooft, R., Schulman, J. & Abbeel, P. 2016 Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329–1338.Google Scholar
Duriez, T., Brunton, S. L. & Noack, B. R. 2016 Machine Learning Control – Taming Nonlinear Dynamics and Turbulence. Springer.Google Scholar
Erdmann, R., Pätzold, A., Engert, M., Peltzer, I. & Nitsche, W. 2011 On active control of laminar–turbulent transition on two-dimensional wings. Phil. Trans. R. Soc. Lond. A 369 (1940), 1382–1395.10.1098/rsta.2010.0364Google Scholar
Fransson, J. H. M., Talamelli, A., Brandt, L. & Cossu, C. 2006 Delaying transition to turbulence by a passive mechanism. Phys. Rev. Lett. 96 (6), 064501.10.1103/PhysRevLett.96.064501Google Scholar
Gautier, N., Aider, J.-L., Duriez, T., Noack, B. R., Segond, M. & Abel, M. 2015 Closed-loop separation control using machine learning. J. Fluid Mech. 770, 442–457.10.1017/jfm.2015.95Google Scholar
Geuzaine, C. & Remacle, J.-F. 2009 Gmsh: a 3-D finite element mesh generator with built-in pre- and post-processing facilities. Intl J. Numer. Meth. Engng 79 (11), 1309–1331.10.1002/nme.2579Google Scholar
Glezer, A. 2011 Some aspects of aerodynamic flow control using synthetic-jet actuation. Phil. Trans. R. Soc. Lond. A 369 (1940), 1476–1494.10.1098/rsta.2010.0374Google Scholar
Goda, K. 1979 A multistep technique with implicit difference schemes for calculating two- or three-dimensional cavity flows. J. Comput. Phys. 30 (1), 76–95.10.1016/0021-9991(79)90088-3Google Scholar
Goodfellow, I., Bengio, Y., Courville, A. & Bengio, Y. 2016 Deep Learning, vol. 1. MIT Press.Google Scholar
Gu, S., Lillicrap, T., Sutskever, I. & Levine, S. 2016 Continuous deep Q-learning with model-based acceleration. In Intl Conference on Machine Learning, pp. 2829–2838.Google Scholar
Guéniat, F., Mathelin, L. & Hussaini, M. Y. 2016 A statistical learning strategy for closed-loop control of fluid flows. Theor. Comput. Fluid Dyn. 30 (6), 497–510.10.1007/s00162-016-0392-yGoogle Scholar
He, K., Zhang, X., Ren, S. & Sun, J. 2016 Deep residual learning for image recognition. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 770–778.Google Scholar
Hornik, K., Stinchcombe, M. & White, H. 1989 Multilayer feedforward networks are universal approximators. Neural Networks 2 (5), 359–366.10.1016/0893-6080(89)90020-8Google Scholar
Kober, J., Bagnell, J. A. & Peters, J. 2013 Reinforcement learning in robotics: a survey. Intl J. Robotics Res. 32 (11), 1238–1274.10.1177/0278364913495721Google Scholar
Krizhevsky, A., Sutskever, I. & Hinton, G. E. 2012 Imagenet classification with deep convolutional neural networks. Adv. Neural Inform. Proc. Syst. pp. 1097–1105.Google Scholar
Kutz, J. N. 2017 Deep learning in fluid dynamics. J. Fluid Mech. 814, 1–4.10.1017/jfm.2016.803Google Scholar
LeCun, Y., Bengio, Y. & Hinton, G. 2015 Deep learning. Nature 521, 436–444.10.1038/nature14539Google Scholar
Li, R., Noack, B. R., Cordier, L., Borée, J. & Harambat, F. 2017 Drag reduction of a car model by linear genetic programming control. Exp. Fluids 58 (8), 103.10.1007/s00348-017-2382-2Google Scholar
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. & Wierstra, D.2015 Continuous control with deep reinforcement learning. arXiv:1509.02971.Google Scholar
Logg, A., Mardal, K.-A. & Wells, G. 2012 Automated Solution of Differential Equations by the Finite Element Method: The FEniCS Book, vol. 84. Springer.10.1007/978-3-642-23099-8Google Scholar
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. & Riedmiller, M.2013 Playing Atari with deep reinforcement learning. arXiv:1312.5602.Google Scholar
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Peterson, S., Beatie, C., Sadih, A., Antonoglou, I., King, H., Rumanron, D., Wiersta, D., Legg, S. & Hassabis, D. 2015 Human-level control through deep reinforcement learning. Nature 518 (7540), 529.10.1038/nature14236Google Scholar
Pastoor, M., Henning, L., Noack, B. R., King, R. & Tadmor, G. 2008 Feedback shear layer control for bluff body drag reduction. J. Fluid Mech. 608, 161–196.10.1017/S0022112008002073Google Scholar
Rabault, J., Kolaas, J. & Jensen, A. 2017 Performing particle image velocimetry using artificial neural networks: a proof-of-concept. Meas. Sci. Technol. 28 (12), 125301.10.1088/1361-6501/aa8b87Google Scholar
Rauber, P. E., Fadel, S. G., Falcao, A. X. & Telea, A. C. 2017 Visualizing the hidden activity of artificial neural networks. IEEE Trans. Vis. Comput. Graphics 23 (1), 101–110.10.1109/TVCG.2016.2598838Google Scholar
Schaarschmidt, M., Kuhnle, A. & Fricke, K.2017 Tensorforce: a tensorflow library for applied reinforcement learning. https://github.com/tensorforce/tensorforce.Google Scholar
Schäfer, M., Turek, S., Durst, F., Krause, E. & Rannacher, R. 1996 Benchmark computations of laminar flow around a cylinder. In Flow Simulation with High-Performance Computers II (ed. Hirschel, E. H.), pp. 547–566. Springer.10.1007/978-3-322-89849-4_39Google Scholar
Schmidhuber, J. 2015 Deep learning in neural networks: an overview. Neural Networks 61, 85–117.10.1016/j.neunet.2014.09.003Google Scholar
Schoppa, W. & Hussain, F. 1998 A large-scale control strategy for drag reduction in turbulent boundary layers. Phys. Fluids 10 (5), 1049–1051.10.1063/1.869789Google Scholar
Schulman, J., Levine, S., Moritz, P., Jordan, M. I. & Abbeel, P.2015 Trust region policy optimization. CoRR abs/1502.05477, arXiv:1502.05477.Google Scholar
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O.2017 Proximal policy optimization algorithms. arXiv:1707.06347.Google Scholar
Shahinfar, S., Sattarzadeh, S. S., Fransson, J. H. & Talamelli, A. 2012 Revival of classical vortex generators now for transition delay. Phys. Rev. Lett. 109 (7), 074501.10.1103/PhysRevLett.109.074501Google Scholar
Siegelmann, H. T. & Sontag, E. D. 1995 On the computational power of neural nets. J. Comput. Syst. Sci. 50 (1), 132–150.10.1006/jcss.1995.1013Google Scholar
Sipp, D. & Schmid, P. J. 2016 Linear closed-loop control of fluid instabilities and noise-induced perturbations: a review of approaches and tools. Appl. Mech. Rev. 68 (2), 020801.10.1115/1.4033345Google Scholar
Valen-Sendstad, K., Logg, A., Mardal, K.-A., Narayanan, H. & Mortensen, M. 2012 A comparison of finite element schemes for the incompressible Navier–Stokes equations. In Automated Solution of Differential Equations by the Finite Element Method, pp. 399–420. Springer.10.1007/978-3-642-23099-8_21Google Scholar
Verma, S., Novati, G. & Koumoutsakos, P. 2018 Efficient collective swimming by harnessing vortices through deep reinforcement learning. Proc. Natl Acad. Sci. USA; http://www.pnas.org/content/early/2018/05/16/1800923115.full.pdf.10.1073/pnas.1800923115Google Scholar
Vernet, J., Örlü, R., Alfredsson, P. H., Elofsson, P. & Scania, A. B. 2014 Flow separation delay on trucks a-pillars by means of dielectric barrier discharge actuation. In First International Conference in Numerical and Experimental Aerodynamics of Road Vehicles and Trains (Aerovehicles 1), Bordeaux, France, pp. 1–2.Google Scholar
Wang, Z., Xiao, D., Fang, F., Govindan, R., Pain, C. C. & Guo, Y. 2018 Model identification of reduced order fluid dynamics systems using deep learning. Intl J. Numer. Meth. Fluids 86 (4), 255–268.10.1002/fld.4416Google Scholar