Hostname: page-component-745bb68f8f-lrblm Total loading time: 0 Render date: 2025-01-08T08:44:07.972Z Has data issue: false hasContentIssue false
  • English
  • Français

Machine Learning Pipeline for Segmentation and Defect Identification from High-Resolution Transmission Electron Microscopy Data

Published online by Cambridge University Press:  06 May 2021

Catherine K. Groschner ,
Christina Choi  and
Mary C. Scott
Catherine K. Groschner
Affiliation:
Department of Materials Science and Engineering, University of California Berkeley, Berkeley, CA94720, USA
Christina Choi
Affiliation:
Department of Materials Science and Engineering, University of California Berkeley, Berkeley, CA94720, USA
Mary C. Scott*
Affiliation:
Department of Materials Science and Engineering, University of California Berkeley, Berkeley, CA94720, USA Molecular Foundry, Lawrence Berkeley National Laboratory, Berkeley, CA94720, USA
*
*Author for correspondence: Mary C. Scott, E-mail: [email protected]
Get access

Abstract

In the field of transmission electron microscopy, data interpretation often lags behind acquisition methods, as image processing methods often have to be manually tailored to individual datasets. Machine learning offers a promising approach for fast, accurate analysis of electron microscopy data. Here, we demonstrate a flexible two-step pipeline for the analysis of high-resolution transmission electron microscopy data, which uses a U-Net for segmentation followed by a random forest for the detection of stacking faults. Our trained U-Net is able to segment nanoparticle regions from the amorphous background with a Dice coefficient of 0.8 and significantly outperforms traditional image segmentation methods. Using these segmented regions, we are then able to classify whether nanoparticles contain a visible stacking fault with 86% accuracy. We provide this adaptable pipeline as an open-source tool for the community. The combined output of the segmentation network and classifier offer a way to determine statistical distributions of features of interest, such as size, shape, and defect presence, enabling the detection of correlations between these features.

Keywords

automated analysisdeep learningHRTEMsegmentationstructure classification
Type
Software and Instrumentation
Information
Microscopy and Microanalysis , Volume 27 , Issue 3 , June 2021 , pp. 549 - 556
Check for updates
Copyright
Copyright © The Author(s), 2021. Published by Cambridge University Press on behalf of the Microscopy Society of America

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Buseck, PR, Epelboin, Y & Rimsky, A (1988). Signal processing of high-resolution transmission electron microscope images using Fourier transforms. Acta Cryst A44, 975–986.CrossRefGoogle Scholar
Chollet, F (2015–2019). Keras. Available at https://keras.io/.Google Scholar
Dan, J, Zhao, X & Pennycook, SJ (2019). A machine perspective of atomic defects in scanning transmission electron microscopy. InfoMat 1(3), 359–375. doi:10.1002/inf2.12026.CrossRefGoogle Scholar
DeCost, BL, Francis, T & Holm, E (2019). High throughput quantitative metallography for complex microstructures using deep learning: A case study in ultrahigh carbon steel. Microsc Microanal 25(1), 21–29. doi:10.1017/S1431927618015635.CrossRefGoogle ScholarPubMed
De Jong, AF, Coene, W & Van Dyck, D (1989). Image processing of HRTEM images with non-periodic features. Ultramicroscopy 27(1), 53–65. doi:10.1016/ 0304-3991(89)90200-3.CrossRefGoogle Scholar
Dennler, N, Foncubierta-Rodriguez, A, Neupert, T & Sousa, M ( (2020). Learning-based defect recognition for quasi-periodic microscope images,.CrossRefGoogle Scholar
Förster, GD, Castan, A, Loiseau, A, Nelayah, J, Alloyeau, D, Fossard, F, Bichara, C & Amara, H (2020). A deep learning approach for determining the chiral indices of carbon nanotubes from high-resolution transmission electron microscopy images. Carbon 169, 465–474. doi:10.1016/j.carbon.2020.06.086. arXiv: 2003.11505.CrossRefGoogle Scholar
Frei, M & Kruis, FE (2018). Fully automated primary particle size analysis of agglomerates on transmission electron microscopy images via artificial neural networks. Powder Technol 332, 120–130. doi:10.1016/j.powtec.2018. 03.032. arXiv: 1806.04010.CrossRefGoogle Scholar
Geron, A (2017). Hands-On Machine Learning with Scikit-Learn & TensorFlow, 1st ed. Sebatopol: O'Reilly Media.Google Scholar
Goodfellow, I, Bengio, Y & Courville, A (2016). Deep Learning. MIT Press. Available at http://www.deeplearningbook.org.Google Scholar
Groom, DJ, Yu, K, Rasouli, S, Polarinakis, J, Bovik, AC & Ferreira, PJ (2018). Automatic segmentation of inorganic nanoparticles in BF TEM micrographs. Ultramicroscopy 194, 25–34. doi:10.1016/j.ultramic.2018.06.002.CrossRefGoogle ScholarPubMed
Groschner, CK, Choi, C & Scott, MC (2020). Methodologies for successful segmentation of HRTEM images via neural network. arXiv: 2001.05022. Available at http://arxiv.org/abs/2001.05022.Google Scholar
Güven, G & Oktay, AB (2018). Nanoparticle detection from TEM images with deep learning. In 26th IEEE Signal Processing and Communications Applications Conference, SIU 2018, pp. 1–4.CrossRefGoogle Scholar
Jain, A, Ong, SP, Hautier, G, Chen, W, Richards, WD, Dacek, S, Cholia, S, Gunter, D, Skinner, D, Ceder, G & Persson, KA (2013). Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL Mater 1(1), 011002. doi:10.1063/1.4812323.CrossRefGoogle Scholar
Jany, BR, Janas, A & Krok, F (2020). Automatic microscopic image analysis by moving window local Fourier transform and machine learning. Micron 130, 102800. doi:10.1016/j.micron.2019.102800.CrossRefGoogle ScholarPubMed
Kalinin, SV, Lupini, AR, Dyck, O, Jesse, S, Ziatdinov, M & Vasudevan, RK (2019). Lab on a beam—Big data and artificial intelligence in scanning transmission electron microscopy. MRS Bulletin 44(7), 12742–12752. doi:10.1557/mrs.2019.159.CrossRefGoogle Scholar
Kingma, DP & Ba, JL (2015). Adam: A Method for Stochastic Optimization, pp. 1–15. arXiv: 1412.6980v9.Google Scholar
Laramy, CR (2015). High-throughput, algorithmic determination of nanoparticle structure from electron microscopy images. ACS Nano 9(12), 12488–12495. doi:10.1021/acsnano.5b05968.CrossRefGoogle ScholarPubMed
LeBeau, J, Kumar, A & Hauwiller, M (2020). A universal scripting engine for transmission electron microscopy. Microsc Microanal 26(S2), 2958–2959. doi:10.1017/S1431927620023338.CrossRefGoogle Scholar
Li, W, Field, KG & Morgan, D (2018). Automated defect analysis in electron microscopic images. npj Comput Mater 4(1), 1–9. doi:10.1038/ s41524-018-0093-8.CrossRefGoogle Scholar
Madsen, J, Liu, P, Kling, J, Wagner, JB, Hansen, TW, Winther, O & Schiøtz, J (2018). A deep learning approach to identify local structures in atomic-resolution transmission electron microscopy images. Adv Theory Simul 1, 8. doi:10.1002/adts.201800037.CrossRefGoogle Scholar
Maksov, A, Dyck, O, Wang, K, Xiao, K, Geohegan, DB, Sumpter, BG, Vasudevan, RK, Jesse, S, Kalinin, SV & Ziatdinov, M (2019). Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2. npj Comput Mater 5(1), 12. doi:10. 1038/s41524-019-0152-9.CrossRefGoogle Scholar
MATLAB & Simulink (2019). Get Started with the Image Labeler. Available at https://www.mathworks.com/help/vision/ug/get-started-with-the-image-labeler.html (retrieved May 30, 2019).Google Scholar
Orfield, NJ, Mcbride, JR, Keene, JD & Davis, LM (2015). Correlation of atomic structure and photoluminescence of the same quantum dot: Pinpointing surface and internal defects that inhibit photoluminescence. ACS Nano 9(1), 831–839. doi:10.1021/nn506420w.CrossRefGoogle ScholarPubMed
Otsu, N (1979). A threshold selection method from gray-level histograms. IEEE Trans Syst, Man, and Cybern SMC 9(1), 62–66. doi:10.1109/TSMC.1979.4310076.CrossRefGoogle Scholar
Pedregosa, F, Varoquaux, G, Gramfort, A, Michel, V, Thirion, B, Grisel, O, Blondel, M, Prettenhofer, P, Weiss, R, Dubourg, V, Vanderplas, J, Passos, A, Cournapeau, D, Brucher, M, Perrot, M & Duchesnay, E (2011). scikit-learn: Machine learning in python. J Mach Lear Res 12, 2825–2830.Google Scholar
Peng, X, Manna, L, Yang, W, Wickham, J, Scher, E, Kadavanich, A & Alivisatos, AP (2000). Shape control of CdSe nanocrystals. Nature 404(6773), 59–61. doi:10.1038/35003535.CrossRefGoogle ScholarPubMed
Rasool, HI, Ophus, C, Klug, WS, Zettle, A & Gimzewski, JK (2013). Measurement of the intrinsic strength of crystalline and polycrystalline graphene. Nat Commun 4, 1. doi:10.1038/ncomms3811.CrossRefGoogle Scholar
Roberts, G, Haile, SY, Sainju, R, Edwards, DJ, Hutchinson, B & Zhu, Y (2019). Deep learning for semantic segmentation of defects in advanced STEM images of steels. Sci Rep 9(1), 1–12. doi:10.1038/s41598-019-49105-0.CrossRefGoogle ScholarPubMed
Ronneberger, O, Fischer, P & Brox, T (2015). U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, Navab, N (Ed.), Vol. 9351, pp. 234–241. Cham: Springer International Publishing. doi:10.1007/978-3-319-24574-4_28.CrossRefGoogle Scholar
Schiøtz, J, Madsen, J, Liu, P, Winther, O, Kling, J, Wagner, J & Hansen, T (2018). Identifying atoms in high resolution transmission electron micrographs using a deep convolutional neural net. Microsc Microanal 24(S1), 512–513. doi:10.1017/S1431927618003057.CrossRefGoogle Scholar
Schiøtz, J, Madsen, J, Østergaard, BJG, Dreisig, AS, Liu, P, Helveg, S, Winther, O, Kling, J, Wagner, JB & Hansen, TW (2019). Using neural networks to identify atoms in HRTEM images. Microsc Microanal 25(S2), 216–217. doi:10.1017/S1431927619001818.CrossRefGoogle Scholar
Schorb, M, Haberbosch, I, Wim, JHH, Schwab, Y & Mastronarde, DN (2019). Software tools for automated transmission electron microscopy. Nat Methods. doi:10.1038/s41592-019-0396-9.CrossRefGoogle ScholarPubMed
Szegedy, C, Toshev, A & Erhan, D (2013). Deep neural networks for object detection. In Advances in Neural Information Processing Systems 26, Burges, CJC (Ed.), pp. 2553–2561. Lake Tahoe: Curran Associates, Inc.Google Scholar
Taha, AA & Hanbury, A (2015). Metrics for evaluating 3D medical image segmentation: Analysis, selection, and tool. BMC Med Imaging 15(1). doi:10.1186/s12880-015-0068-x.CrossRefGoogle ScholarPubMed
Van der Walt, S, Schönberger, JL, Nunez-Iglesias, J, Boulogne, F, Warner, JD, Yagner, N, Gouillart, E & Yu, T (2014). scikit-image: Image processing in python. PeerJ 2, e453. doi:10.7717/peerj.453.CrossRefGoogle ScholarPubMed
Yeom, J, Stan, T, Hong, S & Voorhees, PW (2020). Segmentation of experimental datasets via convolutional neural networks trained on phase field simulations. SSRN Electron J. doi:0.2139/ssrn.3680400.CrossRefGoogle Scholar
Zhao, S, Austin, N, Li, M, Song, Y, House, SD, Bernhard, S, Yang, JC, Mpourmpakis, G & Jin, R (2018). Influence of atomic-level morphology on catalysis: The case of sphere and rod-like gold nanoclusters for CO2 electroreduction. ACS Catal 8(6), 4996–5001. doi:10.1021/acscatal.8b00365.CrossRefGoogle Scholar
Ziatdinov, MA, Dyck, O, Jesse, S & Kalinin, SV (2019). Atomic mechanisms for the Si atom dynamics in graphene: Chemical transformations at the edge and in the bulk.CrossRefGoogle Scholar
Ziatdinov, M, Dyck, O, Maksov, A, Li, X, Sang, X & Xiao, K (2017). Deep learning of atomically resolved scanning transmission electron microscopy images: Chemical identification and tracking local transformations. ACS Nano 11(12), 12742–12752. doi:10.1021/acsnano.7b07504.CrossRefGoogle ScholarPubMed
Supplementary material: PDF

Groschner et al. supplementary material

Groschner et al. supplementary material

Download Groschner et al. supplementary material(PDF)
PDF 2.3 MB