Hostname: page-component-586b7cd67f-r5fsc Total loading time: 0 Render date: 2024-11-30T07:48:18.208Z Has data issue: false hasContentIssue false
  • English
  • Français

Automating material image analysis for material discovery

Published online by Cambridge University Press:  24 April 2019

Chiwoo Park  and
Yu Ding Open the ORCID record for Yu Ding [Opens in a new window]
Chiwoo Park
Affiliation:
Department of Industrial and Manufacturing Engineering, Florida State University, Tallahassee, FL 32310, USA
Yu Ding*
Affiliation:
Department of Industrial and Systems Engineering, Texas A&M University, 3131 TAMU, College Station, TX 77843, USA
*
Address all correspondence to Yu Ding at [email protected]
Get access

Abstract

Advancements in temporal and spatial resolutions of microscopes promise to expand the frontiers of understanding in materials science. Imaging techniques produce images at a high-frame rate, streaming out a tremendous amount of data. Analysis of all these images is time-consuming and labor intensive, creating a bottleneck in material discovery that needs to be overcome. This paper summarizes recent progresses in machine learning and data science for expediting and automating material image analysis. The discussion covers both static image and dynamic image analyses, followed by remarks concerning ongoing efforts and future needs in automated image analysis that accelerates material discovery.

Type
Artificial Intelligence Prospectives
Information
MRS Communications , Volume 9 , Issue 2 , June 2019 , pp. 545 - 555
Copyright
Copyright © Materials Research Society 2019 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

1.Basic Research Needs for Innovation and Discovery of Transformative Experimental Tools; available at http://science.energy.gov, 2017.Google Scholar
2.Crewe, A.V.: Scanning transmission electron microscopy. J. Microsc. 100, 247–259 (1974).Google Scholar
3.Salapaka, S.M. and Salapaka, M.V.: Scanning probe microscopy. IEEE Control Syst. 28, 65–83 (2008).Google Scholar
4.Abellan, P., Mehdi, B.L., Parent, L.R., Gu, M., Park, C., Xu, W., Zhang, Y., Arslan, I., Zhang, J.-G., and Wang, C.-M.: Probing the degradation mechanisms in electrolyte solutions for Li-ion batteries by in situ transmission electron microscopy. Nano Lett. 14, 1293–1299 (2014).Google Scholar
5.Chien, M.P., Thompson, M.P., Barback, C.V., Ku, T.H., Hall, D.J., and Gianneschi, N.C.: Enzyme-directed assembly of a nanoparticle probe in tumor tissue. Adv. Mater. 25, 3599–3604 (2013).Google Scholar
6.Evans, J.E., Jungjohann, K.L., Browning, N.D., and Arslan, I.: Controlled growth of nanoparticles from solution with in situ liquid transmission electron microscopy. Nano Lett. 11, 2809–2813 (2011).Google Scholar
7.Kim, J.S., LaGrange, T., Reed, B.W., Taheri, M.L., Armstrong, M.R., King, W.E., Browning, N.D., and Campbell, G.H.: Imaging of transient structures using nanosecond in situ TEM. Science 321, 1472–1475 (2008).Google Scholar
8.LaGrange, T., Campbell, G.H., Reed, B., Taheri, M., Pesavento, J.B., Kim, J.S., and Browning, N.D.: Nanosecond time-resolved investigations using the in situ of dynamic transmission electron microscope (DTEM). Ultramicroscopy 108, 1441–1449 (2008).Google Scholar
9.Woehl, T.J., Evans, J.E., Arslan, I., Ristenpart, W.D., and Browning, N.D.: Direct in situ determination of the mechanisms controlling nanoparticle nucleation and growth. ACS Nano 6, 8599–8610 (2012).Google Scholar
10.Woehl, T.J., Park, C., Evans, J.E., Arslan, I., Ristenpart, W.D., and Browning, N.D.: Direct observation of aggregative nanoparticle growth: kinetic modeling of the size distribution and growth rate. Nano Lett. 14, 373–378 (2013).Google Scholar
11.Patterson, J. P., Abellan, P., Denny, M. S. Jr., Park, C., Browning, N. D., Cohen, S. M., Evans, J. E., and Gianneschi, N. C.: Observing the growth of metal–organic frameworks by in situ liquid cell transmission electron microscopy. J. Am. Chem. Soc. 137, 7322–7328 (2015).Google Scholar
12.Jesse, S. and Kalinin, S.V.: Band excitation in scanning probe microscopy: sines of change. J. Phys. D: Appl. Phys. 44, 464006 (2011).Google Scholar
13.Rodriguez, B.J., Callahan, C., Kalinin, S.V., and Proksch, R.: Dual-frequency resonance-tracking atomic force microscopy. Nanotechnology 18, 475504 (2007).Google Scholar
14.Kalinin, S.V., Strelcov, E., Belianinov, A., Somnath, S., Vasudevan, R.K., Lingerfelt, E.J., Archibald, R.K., Chen, C., Proksch, R., Laanait, N., and Jesse, S.: Big, deep, and smart data in scanning probe microscopy. ACS Nano 10, 9068–9086 (2016).Google Scholar
15.Roco, M.C.: The Long View of Nanotechnology Development: The National Nanotechnology Initiative at 10 Years. Journal of Nanoparticles 13, 427–445 (2011).Google Scholar
16.Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–698 (1986).Google Scholar
17.Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9, 62–66 (1979).Google Scholar
18.Jiang, X. and Mojon, D.: Adaptive local thresholding by verification-based multithreshold probing with application to vessel detection in retinal images. IEEE Trans. Pattern Anal. Mach. Intell. 25, 131–137 (2003).Google Scholar
19.Vo, G. and Park, C.: Robust regression for image binarization under heavy noises and nonuniform background. Pattern Recognit. 81, 224–239 (2018).Google Scholar
20.Park, C., Huang, J.Z., Huitink, D., Kundu, S., Mallick, B.K., Liang, H., and Ding, Y.: A multistage, semi-automated procedure for analyzing the morphology of nanoparticles. IIE Trans. 44, 507–522 (2012).Google Scholar
21.Park, C., Huang, J.Z., Ji, J.X., and Ding, Y.: Segmentation, inference and classification of partially overlapping nanoparticles. IEEE Trans. Pattern Anal. Mach. Intell. 35, 669–681 (2013).Google Scholar
22.Beucher, S. and Meyer, F.: The morphological approach to segmentation: the watershed transformation. Optical Engineering 34, 433–433 (1992).Google Scholar
23.Qian, Y., Huang, J.Z., Li, X., and Ding, Y.: Robust nanoparticles detection from noisy background by fusing complementary image information. IEEE Trans. Image Process. 25, 5713–5726 (2016).Google Scholar
24.Zafari, S., Eerola, T., Sampo, J., Kälviäinen, H., and Haario, H.: Segmentation of overlapping elliptical objects in silhouette images. IEEE Trans. Image Process. 24, 5942–5952 (2015).Google Scholar
25.Meng, X.-L., and Rubin, D.B.: Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika 80, 267–278 (1993).Google Scholar
26.Bubenik, P.: Statistical topological data analysis using persistence landscapes. J. Mach. Learn. Res. 16, 77–102 (2015).Google Scholar
27.Konomi, B.A., Dhavala, S.S., Huang, J.Z., Kundu, S., Huitink, D., Liang, H., Ding, Y., and Mallick, B.K.: Bayesian object classification of gold nanoparticles. Ann. Appl. Stat. 7, 640–668 (2013).Google Scholar
28.Frank, J.: Electron Tomography: Three-Dimensional Imaging with the Transmission Electron Microscope (New York, NY: Springer Science & Business Media, 2013).Google Scholar
29.Mu, C. and Park, C.: Optimal filtered backprojection for fast and accurate tomography reconstruction. Pattern Recognition Submitted (2019).Google Scholar
30.Li, X., Belianinov, A., Dyck, O., Jesse, S., and Park, C.: Two-level structural sparsity regularization for identifying lattices and defects in noisy images. Ann. Appl. Stat. 12, 348–377 (2018).Google Scholar
31.Dong, L., Li, X., Qian, Y., Yu, D., Zhang, H., Zhang, Z., and Ding, Y.: Quantifying nanoparticle mixing state to account for both location and size effects. Technometrics 59, 391–403 (2017).Google Scholar
32.Belianinov, A., He, Q., Kravchenko, M., Jesse, S., Borisevich, A., and Kalinin, S.V.: Identification of phases, symmetries and defects through local crystallography. Nat. Commun. 6, 7801 (2015).Google Scholar
33.Bright, D.S. and Steel, E.B.: Two-dimensional top hat filter for extracting spots and spheres from digital images. J. Microsc. 146, 191–200 (1987).Google Scholar
34.Sage, D., Neumann, F.R., Hediger, F., Gasser, S.M., and Unser, M.: Automatic tracking of individual fluorescence particles: application to the study of chromosome dynamics. IEEE Trans. Image Process. 14, 1372–1383 (2005).Google Scholar
35.Hughes, J., Fricks, J., and Hancock, W.: Likelihood inference for particle location in fluorescence microscopy. Ann. Appl. Stat. 4, 830–848 (2010).Google Scholar
36.Laanait, N., Ziatdinov, M., He, Q., and Borisevich, A.: Identifying local structural states in atomic imaging by computer vision. Adv. Struct. Chem. Imaging 2, 14 (2017).Google Scholar
37.Ripley, B.D.: The second-order analysis of stationary point processes. J. Appl. Probab. 13, 255–266 (1976).Google Scholar
38.Li, X., Zhang, H., Jin, J., Huang, D., Qi, X., Zhang, Z., and Yu, D.: Quantifying dispersion of nanoparticles in polymer nanocomposites through transmission electron microscopy micrographs. J. Micro Nano-Manufacturing 2, 021008 (2014).Google Scholar
39.De Jonge, N. and Ross, F.M.: Electron microscopy of specimens in liquid. Nat. Nanotechnol. 6, 695 (2011).Google Scholar
40.Kalinin, S.V., Sumpter, B.G., and Archibald, R.K.: Big-deep-smart data in imaging for guiding materials design. Nat. Mater. 14, 973 (2015).Google Scholar
41.Zheng, H., Meng, Y.S., and Zhu, Y.: Frontiers of in situ electron microscopy. MRS Bull. 40, 12–18 (2015).Google Scholar
42.Grzelczak, M., Vermant, J., Furst, E.M., and Liz-Marzán, L.M.: Directed self-assembly of nanoparticles. ACS Nano 4, 3591–3605 (2010).Google Scholar
43.Mikhailov, A. and Gundersen, G.: Relationship between microtubule dynamics and lamellipodium formation revealed by direct imaging of microtubules in cells treated with nocodazole or taxol. Cytoskeleton 41, 325–340 (1998).Google Scholar
44.Bergen, L.G. and Borisy, G.G.: Head-to-tail polymerization of microtubules in vitro. Electron microscope analysis of seeded assembly. J. Cell Biol. 84, 141–150 (1980).Google Scholar
45.Park, C.: Estimating multiple pathways of object growth using nonlongitudinal image data. Technometrics. 56, 186–199 (2014).Google Scholar
46.Park, C. and Shrivastava, A.K.: Multimode geometric-profile monitoring with correlated image data and its application to nanoparticle self-assembly processes. J. Qual. Technol. 46, 216–233 (2014).Google Scholar
47.Park, C., Woehl, T.J., Evans, J.E., and Browning, N.D.: Minimum cost multi-way data association for optimizing multitarget tracking of interacting objects. IEEE Trans. Pattern Anal. Mach. Intell. 37, 611–624 (2015).Google Scholar
48.Qian, Y., Huang, J. Z.; Park, C., and Ding, Y.: Fast dynamic nonparametric distribution tracking in electron microscopic data. Ann. Appl. Stat. in press (2019).Google Scholar
49.Qian, Y., Huang, J.Z., and Ding, Y.: Identifying multi-stage nanocrystal growth using in situ TEM video data. IISE Trans. 49, 532–543 (2017).Google Scholar
50.Zheng, H., Smith, R.K., Jun, Y.-W., Kisielowski, C., Dahmen, U., and Alivisatos, A.P.: Observation of single colloidal platinum nanocrystal growth trajectories. Science 324, 1309–1312 (2009).Google Scholar
51.Rodriguez, A. and Ter Horst, E.: Bayesian dynamic density estimation. Bayesian Anal. 3, 339–365 (2008).Google Scholar
52.Mena, R.H. and Ruggiero, M.: Dynamic density estimation with diffusive Dirichlet mixtures. Bernoulli. (Andover) 22, 901–926 (2016).Google Scholar
53.Jiang, H., Fels, S., and Little, J. J.: In A linear programming approach for multiple object tracking, 2007 IEEE Conference on Computer Vision and Pattern Recognition, IEEE: 2007; pp. 1–8.Google Scholar
54.Okuma, K., Taleghani, A., De Freitas, N., Little, J. J., and Lowe, D. G.: In A boosted particle filter: Multitarget detection and tracking, 2004 European Conference on Computer Vision, Springer, 2004; pp. 28–39.Google Scholar
55.Pirsiavash, H., Ramanan, D., and Fowlkes, C. C.: In Globally-optimal greedy algorithms for tracking a variable number of objects, 2011 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2011; pp. 1201–1208.Google Scholar
56.Jaqaman, K., Loerke, D., Mettlen, M., Kuwata, H., Grinstein, S., Schmid, S.L., and Danuser, G.: Robust single-particle tracking in live-cell time-lapse sequences. Nat. Methods 5, 695 (2008).Google Scholar
57.Henriques, J. F., Caseiro, R., and Batista, J.: In Globally optimal solution to multi-object tracking with merged measurements, 2011 IEEE International Conference on Computer Vision, IEEE, 2011; pp. 2470–2477.Google Scholar
58.Welch, D.A., Woehl, T.J., Park, C., Faller, R., Evans, J.E., and Browning, N.D.: Understanding the role of solvation forces on the preferential attachment of nanoparticles in liquid. ACS Nano 10, 181–187 (2015).Google Scholar
59.Esmaieeli Sikaroudi, A., Welch, D.A., Woehl, T.J., Faller, R., Evans, J.E., Browning, N.D., and Park, C.: Directional statistics of preferential orientations of two shapes in their aggregate and Its application to nanoparticle aggregation. Technometrics 60, 332–344 (2018).Google Scholar
60.Mehdi, B.L., Qian, J., Nasybulin, E., Park, C., Welch, D.A., Faller, R., Mehta, H., Henderson, W.A., Xu, W., and Wang, C.M.: Observation and quantification of nanoscale processes in lithium batteries by operando electrochemical (S) TEM. Nano Lett. 15, 2168–2173 (2015).Google Scholar
61.Touve, M.A., Figg, C.A., Wright, D.B., Park, C., Cantlon, J., Sumerlin, B.S., and Gianneschi, N.C.: Polymerization-induced self-assembly of micelles observed by liquid cell transmission electron microscopy. ACS Cent. Sci. 4, 543–547 (2018).Google Scholar
62.Stevens, A., Luzi, L., Yang, H., Kovarik, L., Mehdi, B., Liyu, A., Gehm, M., and Browning, N.: A sub-sampled approach to extremely low-dose STEM. Appl. Phys. Lett. 112, 043104 (2018).Google Scholar
63.Kovarik, L., Stevens, A., Liyu, A., and Browning, N.D.: Implementing an accurate and rapid sparse sampling approach for low-dose atomic resolution STEM imaging. Appl. Phys. Lett. 109, 164102 (2016).Google Scholar
64.Stevens, A., Kovarik, L., Abellan, P., Yuan, X., Carin, L., and Browning, N.D.: Applying compressive sensing to TEM video: a substantial frame rate increase on any camera. Adv. Struct. Chem. Imaging 1, 10 (2015).Google Scholar
65.Castro, R., Haupt, J., and Nowak, R.: In Compressed sensing vs. active learning, 2006 IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2006; p. III.Google Scholar
66.Edgeworth, R. and Wilhelm, R.G.: Adaptive sampling for coordinate metrology. Prec. Eng. 23, 144–154 (1999).Google Scholar
67.Park, C. and Qiu, P.: Sequential Adaptive Design for Jump Regression Estimation. Submitted (IEEE Transactions on Pattern Analysis and Machine Intelligence 2019). Also available at https://arxiv.org/abs/1904.01648Google Scholar
68.Zewail, A. H. and Thomas, J. M.: 4D Electron Microscopy: Imaging in Space and Time. (Imperial College Press: London, 2009).Google Scholar
69.Sreehari, S., Venkatakrishnan, S., Bouman, K. L., Simmons, J. P., Drummy, L. F., and Bouman, C. A.: In Multi-resolution data fusion for super-resolution electron microscopy, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017; pp. 1084–1092.Google Scholar
70.Xia, H., Ding, Y., and Mallick, B.K.: Bayesian hierarchical model for combining misaligned two-resolution metrology data. IIE Trans. 43, 242–258 (2011).Google Scholar
71.Ezzat, A.A., Pourhabib, A., and Ding, Y.: Sequential design for functional calibration of computer models. Technometrics. 60, 286–296 (2018).Google Scholar
72.Pourhabib, A., Huang, J.Z., Wang, K., Zhang, C., Wang, B., and Ding, Y.: Modulus prediction of buckypaper based on multi-fidelity analysis involving latent variables. IIE Trans. 47, 141–152 (2015).Google Scholar