Introduction
The scanning transmission electron microscope (STEM) is a powerful instrument for probing materials down to the atomic scale. However, reliably imaging fragile samples require the operator to consider the risk of electron-beam damage introducing artifacts into the data. Many approaches have been introduced to reduce the electron-fluence, measured in e−/Å2 (often referred to as electron dose in the electron microscopy community, which is the term used hereafter) or dose rate (e−/Å2 s−1) imposed on the sample (Chen et al., Reference Chen, Dwyer, Sheng, Zhu, Li, Zheng and Zhu2020), or to increase a sample's durability with either fixation or cryogenic cooling (Glaeser, Reference Glaeser1971; Elbaum, Reference Elbaum2018). Recently, novel approaches such as compressed sensing (CS) have been proposed to reduce the fraction of pixels illuminated and so reduce the electron exposure (Anderson et al., Reference Anderson, Ilic-Helms, Rohrer, Wheeler and Larson2013; Stevens et al., Reference Stevens, Yang, Carin, Arslan and Browning2014, Reference Stevens, Yang, Hao, Jones, Ophus, Nellist and Browning2018). However, CS requires additional fast (and expensive) hardware to be introduced that may not be available to all microscopists (Béché et al., Reference Béché, Goris, Freitag and Verbeeck2016; Kovarik et al., Reference Kovarik, Stevens, Liyu and Browning2016). Moreover, where readout noise is not the limiting factor, such as with digital read-out approaches (Jones & Downing, Reference Jones and Downing2018; Mullarkey et al., Reference Mullarkey, Downing and Jones2020), CS may not offer any appreciable benefit over conventional (Shannon) scanning (Sanders & Dwyer, Reference Sanders and Dwyer2018; Van den Broek et al., Reference Van den Broek, Reed, Beche, Velazco, Verbeeck and Koch2019).
Instead, approaches which rely only on reducing beam-current (Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010), reducing scanning density (more coarse scans) (Yankovich et al., Reference Yankovich, Berkels, Dahmen, Binev and Voyles2015), or reducing dwell-time (Mittelberger et al., Reference Mittelberger, Kramberger and Meyer2018) may be attractive as they are available to all STEM operators without the purchase of additional hardware. With these approaches, individual frames may then exhibit a relatively poor signal noise ratio (SNR), but a final image is reconstructed through frame alignment and averaging (Kimoto et al., Reference Kimoto, Asaka, Yu, Nagai, Matsui and Ishizuka2010). By acquiring the data as a series of sometimes several hundred low-dose frames over a period of what may be some minutes, the electron dose rate is further lowered. Moreover, the intrinsically redundant nature of the data allows for unbiased correction of time-varying scanning-distortion/drift while preserving time-invariant sample data (Jones & Nellist, Reference Jones and Nellist2013; Sang & LeBeau, Reference Sang and LeBeau2014; Jones et al., Reference Jones, Yang, Pennycook, Marshall, Van Aert, Browning, Castell and Nellist2015; Ophus et al., Reference Ophus, Ciston and Nelson2016). These techniques also lend themselves to extension into areas including digital super-resolution (Bárcena-González et al., Reference Bárcena-González, Guerrero-Lebrero, Guerrero, Fernández-Reyes, González, Mayoral, Utrilla, Ulloa and Galindo2016, Reference Bárcena-González, Guerrero-Lebrero, Guerrero, Yañez, Fernández-Reyes, González and Galindo2017) and simultaneous spectrum image recording and alignment (Yankovich et al., Reference Yankovich, Zhang, Oh, Slater, Azough, Freer, Haigh, Willett and Voyles2016; Jones et al., Reference Jones, Varambhia, Beanland, Kepaptsoglou, Griffiths, Ishizuka, Azough, Freer, Ishizuka, Cherns, Ramasse, Lozano-Perez and Nellist2018; Wang et al., Reference Wang, Eren Suyolcu, Salzberger, Hahn, Srot, Sigle and van Aken2018), but discussion of these areas is left for later work.
While appealing, reducing pixel dwell-time alone rapidly becomes insufficient as line flyback quickly dominates the electron exposure time. For conventional STEM scan rates, a relatively cautious flyback waiting time of 1,000–1,500 μs may be set as the default by the installing engineer (Sang et al., Reference Sang, Lupini, Unocic, Chi, Borisevich, Kalinin, Endeve, Archibald and Jesse2016). When this flyback time is reduced, appreciable distortions may become visible at the start of every line (typically the left half of the image), where the image appears compressed by up to 30% (Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010). Fortunately, the profile of this artifact is highly reproducible and for times greater than around 10 μs appears to closely follow an exponential decay profile (Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010; Anderson et al., Reference Anderson, Ilic-Helms, Rohrer, Wheeler and Larson2013; Kovarik et al., Reference Kovarik, Stevens, Liyu and Browning2016; Sang et al., Reference Sang, Lupini, Unocic, Chi, Borisevich, Kalinin, Endeve, Archibald and Jesse2016); this suggests a route then to calibrate and correct for this in postprocessing (Sanchez et al., Reference Sanchez, Galindo, Kret, Falke, Beanland and Goodhew2006; Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010).
In this work, we explore the diagnosis and compensation of these flyback artifacts whose successful mitigation allows for greatly reduced flyback times to be used, delivering a commensurate increase in STEM frame rate and reduction in electron exposure. First, we present a numerical study exploring the potential for dose reduction. Second, methods for modeling and correcting the flyback compression artifact are discussed through comparison with generated, artifact-free reference frames. Finally, we experimentally apply our corrections to evaluate the approach and discuss the implications for fast multiframe image averaging, and the practicalities of the real-world usage of this technique (Sang & LeBeau, Reference Sang and LeBeau2014; Ophus et al., Reference Ophus, Ciston and Nelson2016).
Background
Electron Dose and Scanning Efficiency
Before developing strategies to reduce electron dose, we will first revisit the current approach to dose calculation. The most common approach to calculate sample exposure requires the user to know their beam current, and the spacing and dwell-time of each probe position. This leads to the expression in Equation (1):
$${\rm Dose} = I\cdot C\;( \delta _t) \;\displaystyle{1 \over {{( dx) }^2}},$$where I is the beam current in amps, C is the Coulomb number, δt is the dwell-time at each pixel in seconds, and dx is the pixel width (assuming a square pixel), this equation calculates the dose in units of electrons per unit area. For CS acquisitions, where only a fraction of pixels are illuminated, this number is then multiplied by some fraction less than one. From this equation then, we see the motivation to pursue approaches with both low beam currents and reduced dwell-times (Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010).
However, this equation represents a significant oversimplification of the practical operation of a STEM instrument. Figure 1a shows a schematic representation of the useful information-yielding electron exposure during a series of successive scanned image frames.
Fig. 1. Schematic illustration of sample electron-exposure with time. Plot (a) shows an illustration of a series of three image frames recorded in succession over a characteristic time of several seconds. Plot (b) shows an expanded illustration of the individual scan lines within a single frame. Useful sample exposure which yields information is illustrated in green, while line flyback is shown in orange, and frame flyback time is shown in red.
A conventional approach to producing images of high SNR is to capture a single frame with a very long dwell-time to allow the accumulation of a large amount of signal, however this leaves the data vulnerable to scanning noise and distortion (Muller et al., Reference Muller, Kirkland, Thomas, Grazul, Fitting and Weyland2006). A more contemporary approach is to capture multiple fast image frames for later averaging, as this is more robust to these environmental effects (Sang & LeBeau, Reference Sang and LeBeau2014; Jones et al., Reference Jones, Yang, Pennycook, Marshall, Van Aert, Browning, Castell and Nellist2015). One might assume that an image derived from of 100 fast frames, and one of a single frame but with a 100 times longer dwell-time, expose the sample to equal amounts of electron radiation. However, this would only be true if the effects of two flyback times are ignored. These are the line flyback time it takes for the electron beam to travel from the end of one image line to the beginning of the next, and the frame flyback time, the time it takes for the beam to travel from the final pixel of an image frame to the first of the next.
For large dwell-times, these flyback times represent a small fraction of the overall frame time. However, Figure 1b shows the detail within a single scan frame captured with a lower dwell-time and a characteristic time frame of milliseconds. In this scenario, not only is a significant amount of time expended during flyback time, but the electron probe still impinges on the sample. During this time, the probe may be just to the side of the area being imaged, but secondary electrons can still cause damage within the region of interest and beam-heating also persists (Egerton, Reference Egerton2017). These effects lead to potential sample damage while not delivering any useful information. Fast blanking during these times may be one solution, but again, this would require expensive hardware (Béché et al., Reference Béché, Goris, Freitag and Verbeeck2016).
Knowing this motivated the creation of an expression for the efficiency with which our beam generates useful data, η:
$$\eta = \displaystyle{{\delta _t\,\cdot\, n_p} \over {( {\delta_t\,\cdot\, n_p} ) + ( {T_{{\rm LFB}}\,\cdot\, n_L} ) + T_{{\rm FFB}}}},$$where δt is the pixel dwell-time, np is the number of image pixels, nL is the number of scan lines, T LFB is the line flyback time, and T FFB is the frame flyback time. Equation (2) can be understood intuitively as the ratio of useful, information collecting time, to the total frame time (Mullarkey et al., Reference Mullarkey, Downing and Jones2020), and as η is always less than 1 we can see that equation (1) always underestimates the dose. Of the variables in this equation (δt, np, nL, T LFB, and T FFB) generally only the first four out of five can be readily changed by the operator. Although increasing the pixel dwell-time and the number of pixels or decreasing the number of scan lines would increase the efficiency, it would also increase the dose as per equation (1). The route to increase scanning efficiency then, without increasing the dose, is to lower the line flyback time, if not eliminate it completely. This is of particular importance when using the aforementioned low-dose conditions of short dwell-times; the shorter each line is, the larger the fraction of it is occupied by the flyback time.
As an example, for a 512 × 512 pixel image captured with a 2 μs dwell-time, a 400 μs line flyback time, and a 0.25 s frame flyback time, has an efficiency of just over 50%. This means the dose predicted by equation (1) is approximately half of the actual dose the sample received. For a more general overview, we evaluate equation (2) for dwell-times in the range 0.167–38 μs as this encompasses the entire range from the fastest Nion (6 MHz) and Gatan (2 MHz) scan-generators up to the line-sync speeds for 50 Hz mains power and 512 × 512 images. Figure 2 shows the electron-dose efficiency across this range, frame flyback times were experimentally found on our system to be in the range of 0.2–0.3 s and so 0.25 s was chosen as a representative value. Equation (2) was also evaluated for the same range of values for 256 × 256 images, resulting in overall lower efficiencies (Supplementary Fig. S1).
Fig. 2. Plots of dose efficiency as a function of pixel dwell-time for varying flyback settings. The uppermost line represents the efficiencies that could be achieved with the elimination of frame flyback time. The points labeled L50 and L60 represent the efficiency of scans using a 600 μs line flyback time and which are line-synced to a 50 and 60 Hz mains frequency, respectively.
At dwell-times around 38 μs, typical for traditional single frame imaging, the dose efficiency is above 90% for all cases and perhaps explains why the wasted flyback dose has to date received little attention. For dwell-times of 2 μs (Fig. 2, red vertical line), and where a 600 μs flyback time is maintained, the dose efficiency falls to just under 50%, indicating that more than half the total number of electrons passing through/near the image region are causing damage but yielding no useful information. Alternatively, the dose efficiency improves to nearly 70% for the same 2 μs dwell-time but for a flyback time of 20 μs, which we later show is usable without introducing distortions with the method introduced in this article. This jump in efficiency is represented by the points A and B in the figure and is strong motivation for using shorter flyback times. To jump again to point C requires the frame flyback time to be eliminated or reduced to a very low value. Approaches to achieve this may include custom scan generators or the implementation of a different pair of scan coils with low inductance, such as in Ishikawa et al. (Reference Ishikawa, Jimbo, Terao, Nishikawa, Ueno, Morishita, Mukai, Shibata and Ikuhara2020). These approaches are potentially costly or require dramatic modification to the instrument being used, which is generally not possible in a shared-user facility.
Figure 2 also shows the case for line-synced captures at both 50 and 60 Hz with a line flyback time of 600 μs, marked L50 and L60, respectively. In this case, the start of each scan line is locked into the phase of the mains power supply, and when using a DigiScan the dwell-time is increased such that sequential scan lines begin at the same phase. In both of these scenarios, the efficiencies are very high at over 90%, and the common use of line-syncing may explain why this topic has received little attention to date.
There have been approaches that seek to eliminate line flyback time entirely using either serpentine or spiral scans, but these approaches are somewhat complex to implement, require additional hardware, or result in circular images less convenient for onward processing (Wang et al., Reference Wang, Wang, Hou and Lu2010; Sang et al., Reference Sang, Lupini, Unocic, Chi, Borisevich, Kalinin, Endeve, Archibald and Jesse2016). Here, we focus on square (Shannon) scans available to all existing operators without the need to purchase new hardware. What remains then is to decrease the line flyback time as much as possible, but doing so begins to introduce artifacts into images due to the unavoidable hysteresis of the scan coils.
Scan Coil Hysteresis
As the scan coils are electromagnets of finite inductance, there is a non-zero response time of the coil current to changes in the driving voltage (Anderson et al., Reference Anderson, Ilic-Helms, Rohrer, Wheeler and Larson2013). Due to this, there is an appreciable lag between the expected position of the electron beam and its achieved position until the beam reaches a constant velocity (Velazco et al., Reference Velazco, Nord, Béché and Verbeeck2020). This is analogous to inertia in a physical system and leads to distortions in images captured at low flyback times (Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010; Sang et al., Reference Sang, Lupini, Unocic, Chi, Borisevich, Kalinin, Endeve, Archibald and Jesse2016).
These distortions can be understood by considering the physical path the beam tracks during the flyback time. After reaching the end of a scan line, the beam initially travels from right-to-left (for a typical scan) and overshoots for first pixel of the next line. It must then travel from left-to-right, ready for the next line's start. At long line flyback times, the beam has enough time to reach a constant velocity as it is traveling to the right to begin the next line. However, with short flyback times, the beam is still lagging behind the correct position when the next scan line begins. This causes an overly large area of the sample to be captured and fitting this extra area into the image pixels results in the appearance of compressed data at the start of each line. Although this may vary by instrument, this is indeed observed in this article on two different microscopes (Supplementary Fig. S2). Conventionally, the electron dose is highest at the left-hand side of an image due to the flyback time. However, at the conditions used here where the beam is traveling faster at the start of each line, the electron dose may be lower in this region as a larger area is scanned in the same number of pixel dwell-times, as evidenced by the compression artifact.
It has been observed that the scan coil rise time follows a characteristic exponential shape, as to be expected from a system with inductance effects (Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010; Anderson et al., Reference Anderson, Ilic-Helms, Rohrer, Wheeler and Larson2013). In extreme cases where the flyback time is eliminated, and direct or indirect measures of the deflection rise-time can be measured, 90% rise-time values between 10 and 100 μs have been obtained (Anderson et al., Reference Anderson, Ilic-Helms, Rohrer, Wheeler and Larson2013; Kovarik et al., Reference Kovarik, Stevens, Liyu and Browning2016). At typical scanning speeds, this rise-time would be confined in only the first few pixels of a line. However, at shorter dwell-times, this artifact becomes more apparent and may cover half the width of the frame (Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010; Sang et al., Reference Sang, Lupini, Unocic, Chi, Borisevich, Kalinin, Endeve, Archibald and Jesse2016).
Methods
The experimental results presented here were recorded using a Nion UltraSTEM 200 at 200 kV controlled through a Gatan DigiScan II, unless stated otherwise. Similar, though not identical, effects were seen on our ThermoFisher Titan STEM with its own PIA scan-generator, but for clarity we focus on results from the common DigiScan controlled system.
Automated Flyback Diagnosis
Various approaches may be followed to measure the flyback distortion when using a single-crystal reference sample. For example, geometric-phase analysis (GPA) can be used to diagnose distortions, where the exx term from a perfect single-crystal image natively captures flyback distortion (Hÿtch et al., Reference Hÿtch, Snoeck and Kilaas1998; Zhu et al., Reference Zhu, Ophus, Ciston and Wang2013). However, owing to Fourier truncation, this technique always exhibits some edge effects around the image in the measured distortion fields (unfortunately the region of most interest). Moreover, depending on the radius, x–y position, and edge-smoothing of the manually selected Fourier masks (minimum two masks leads to a minimum of six adjustable parameters), other systematic artifacts may be introduced (Peters et al., Reference Peters, Beanland, Alexe, Cockburn, Revin, Zhang and Sanchez2015).
Alternatively, a manual analysis extracting the position of fiducial features (Anderson et al., Reference Anderson, Ilic-Helms, Rohrer, Wheeler and Larson2013), or the relative compression of crystal unit-cells (Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010) may be performed. However, this again requires manual intervention and/or special sample orientation.
What is ideally needed then, for this to be readily deployable to a general STEM user, is some fully automated method (e.g., without manual selection of Fourier masks), and which does not intrinsically have additional image-edge artifacts. The solution to both these problems is to work in real-space (rather than reciprocal-space) and use a readily available single-crystal reference material (e.g., Si, SrTiO3). The regular atomic spacings of such a material can be used to optimize any parameters to define the exponential decay.
Using the prior knowledge that the flyback hysteresis causes each scan line to be compressed, which can be approximated with an exponential form, we can then write the distortion as a function of just two unknown parameters (Buban et al., Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010):
$$x_{{\rm corrected}} = x-Ae^{{-}t/b},$$where x are the measured x-positions and A and b are to be determined. Calculating the exponential compression and subtracting it from the measured positions will correct this compression artifact, restoring the true positions, x corrected [equation (3)]. Working with units of time in the exponential has the advantage that we can deal explicitly with the flyback time:
$$t = x\cdot \delta _t + T_{{\rm LFB}}.$$This simple distortion then is a function described by two parameters, A and b, and can be removed from an image using bilinear interpolation. The instrument-specific parameters can be obtained from nonlinear least squared fitting with respect to a distortion free reference image, sometimes referred to as a quasi-static reference image (Rečnik et al., Reference Rečnik, Möbus and Šturm2005; Sanchez et al., Reference Sanchez, Galindo, Kret, Falke, Beanland and Goodhew2006). This reference can be generated in two ways. Either an image is acquired with large flyback (e.g., during calibration when the microscope is first installed) or we can use the fact that the furthest right half of an image will have negligible distortion due to flyback (Supplementary Fig. S3). In either case, the reference needs to be made with the same crystallographic phase as the image to be diagnosed. The method here uses autocorrelation to perform self-same tiling and generate an oversized image, from which an appropriately positioned reference can be extracted (Supplementary Fig. S4).
The fitting process can be repeated for images of varying line flyback and dwell-time combinations with the calculated values for A and b tabulated for each. With the observation that these parameters remain constant for the same imaging settings at the same operating voltages, even across a timespan of weeks (Supplementary Fig. S5), these values can then be used to correct any future images taken at the same settings.
An expansion of this is to generalize the recorded values of A and b and to use these values to generate an empirical equation to describe the behavior of the scan coils. This instrument-specific equation can then be used to determine the correct values of A and b which eliminate the hysteresis artifact in an image for all combinations of flyback-time and dwell-time. This is the approach taken in this article as a given instrument can be characterized (say by core staff) and then be available to all users. It should be noted that, in the same way, that magnification is calibrated on a per-instrument basis, so too this inductance descriptor equation should be done on a per-instrument basis.
The model here assumes a single time constant b for the system. Although this is unlikely to be the case due to different elements of the scan system such as the waveform generator, filter boards, and coil magnetization each having their own time constants, it is seen here that this approximation leads to satisfactory results for a practical range of line flyback and pixel dwell-times.
Measuring T FFB
Although the line flyback time can be set by the user using a DigiScan II, the frame flyback time, T FFB, cannot. This is the time taken for the electron beam to travel from the final pixel of an image frame to the first pixel of the next. To measure this, an acquisition was run for 60 s with known imaging conditions. Assuming a T FFB of zero, the frame time, and therefore the expected number of frames in one minute can be calculated. This can then be compared to the experimentally achieved number of frames, with the difference being explained by any unaccounted-for time, in this case the frame flyback time.
Samples
The samples used in this article are lamellae of strontium titanate (SrTiO3) and silicon. These were chosen as they are readily available single crystals whose real-space periodicity is exploited in our method as described above. Any sample can be used if a long flyback quasi-static image is chosen as a reference, however this requires minimal sample drift for the duration of the data-capture for best performance. In practice, a sample with periodic features that can be tiled, as shown in Supplementary Figure S4, is more suitable and is recommended.
Results and Discussion
Averaging a series of frames of an object is a common means of improving image SNR. Recently, more advanced tools for correcting for nonrigid distortions before this averaging have been shown to improve performance further (Berkels et al., Reference Berkels, Binev, Blom, Dahmen, Sharpley and Vogt2014; Jones et al., Reference Jones, Yang, Pennycook, Marshall, Van Aert, Browning, Castell and Nellist2015; Ophus et al., Reference Ophus, Ciston and Nelson2016). The algorithms in these previous studies work in slightly different ways, but all rely on one assumption; that the multiple observations provide a mathematically redundant set from which the time-varying distortions can be separated from the time-invariant ground-truth. Additional redundancy can be encoded into the dataset by varying the scan-orientation during the acquisition, either as perpendicular pairs (Ophus et al., Reference Ophus, Ciston and Nelson2016) or with images recorded with a 90° increment between successive frames (Sang & LeBeau, Reference Sang and LeBeau2014; Jones et al., Reference Jones, Yang, Pennycook, Marshall, Van Aert, Browning, Castell and Nellist2015). Data from both single-orientation series and rotating-scan frame series are presented in our results, with the former presented first.
Hysteresis Correction of a Single-Orientation Series
For a qualitative visual comparison, excerpts from images of strontium titanate (STO) with increasingly shorter flyback waiting times are shown in Figure 3a.
Fig. 3. (a) A montage of cropped panels from increasing faster line flyback time acquisitions. At faster acquisitions, the unit cell appears increasingly compressed. (b) A montage of the same images post-correction. For flyback times as low as 20 μs, visually fidelity is retained across the entire field of view. Some artifact remains in the leftmost region of the 0 μs image.
The image captured with the 1,000 μs (1 ms) line flyback time is taken to be a quasi-static reference frame for visual comparison with the distorted images. Across the entire width of the quasi-static frame, the unit cells are uniform in width (Supplementary Fig. S3). Equally, at all flyback-delays, the latter half of each scan line (right-hand half) is also in alignment with the 1 ms frame, which is good justification for using this region to generate a reference image to compare against.
At shorter flyback-delays, the leftmost region of the image is increasingly distorted (compressed). Using the determined hysteresis parameters, the flyback distortion is corrected in all raw-frames prior to any alignment and nonrigid registration. Figure 3b shows the same data to Figure 3a but after flyback correction.
As shown in Supplementary Figure S6, this is reduced to below 0.1%. After applying this correction, these distortions are eliminated for flyback times as low as 20 μs. In the case of 0 μs, line flyback time only a very small region at the beginning of the line still retains some distortion. This is because the beam cannot achieve a 0 μs flyback time as it takes a finite amount of time. The remaining distortion then is what is captured while the beam is still flying back.
For a quantitative comparison, the diagnosis of the compression using the proposed automated method and also manually following the method in Buban et al. (Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010), before and after correction are presented in Figure 4.
Fig. 4. Example flyback hysteresis diagnosis for a 512 px wide image with a 20 μs flyback and 0.5 μs dwell-time. The black round points represent a manual analysis after Buban et al. (Reference Buban, Ramasse, Gipson, Browning and Stahlberg2010), whereas the solid line represents the output of our automated analysis method introduced here. The blue square points represent a manual analysis after flyback hysteresis correction.
Figure 4 shows an example analysis for the 20 μs flyback-delay acquisition with both the results of the automated parametric diagnosis and the manual unit-cell-based analysis. While both methods give nominally identical results, the manual method is limited in its precision because of human intervention and a small random scatter due to the finite pixilation of the peak position identification. The automated approach, however, uses a smoothly varying mesh as opposed to pixels, uses the entire field-of-view, and does not rely on any particular scan orientation. Due to this, the automated approach is used going forward.
Diagnosing the Hysteresis Compression Artifact
The method described above was used to then diagnose the compression for a wide range of dwell-times and line flyback times with a fixed scan rotation angle. Figure 5 shows the experimentally determined A and b values for a range of dwell-times and line flyback times. Clear trends with dwell-time are present for both parameters, but noticeable the use of equation (4) has removed any flyback dependence, allowing the diagnosis to depend only on the dwell-time and not the line flyback time.
Fig. 5. (a,b) Graphs of A and b from equation (3) which describes the compression of each scan line due to flyback hysteresis. These are presented as a function of dwell-time for varying line flyback times. Error estimates for the fit parameters and goodness-of-fit estimates can be found in Supplementary Table S1.
The A parameter in Figure 5a clearly shows an inverse decay with dwell-time, whereas Figure 5b shows a linear increase of b as the dwell-time. This is understandable, as the slower scans exhibit a reduced magnitude of compression (lower A value), and the longer dwell-times exhibit a more rapid decay of the compression across the image (larger b value).
From this, we can see that a relatively simple set of equations (one linear, one inverse) can be used to determine the parameters needed in equation (3) to correct flyback distortions for all line flybacks and dwell-times. As the equations do not depend on the flyback time, this means only a small set of images is required and with only dwell-time varying to fully diagnose the hysteresis behavior. Discussion on the scatter of the points in Figure 5b is included in Supplementary Figure S5, while the mean and standard deviation of the data are plotted in Supplementary Figure S7.
The physical origin of these functions is obfuscated by each manufacturer's choice of the microscope control hardware and the scan generator itself. For example, the DigiScan II produces a pure sawtooth waveform, even during line flyback wait time, that modifies the coil current of the left side of the image dependent on the scan speed (and therefore dwell-time). To avoid unnecessary discussion that is specific to our particular microscope, and to maintain generality, we do not attempt to reduce the number of fit parameters further. Nevertheless, though we might expect the exact form of the trends to vary between microscopes, the observed trends are consistent can be used to diagnose and correct the compression.
Due to the limited range of fields-of-view which can be used on the Nion UltraSTEM 200, a full characterization of the effects of magnification on this process could not be carried out. However, magnification is linearly related to the current in the scan coils, and we predict that the effect of the visible hysteresis within the image will also scale linearly. Therefore, we expect that fractionally, the compression artifact will appear the same within any image at different magnifications. However, when changing from very high to very low magnifications, the current in the scan coils can change by an order of magnitude and this assumption may not hold, and so we recommend operators investigate this on their own instruments where necessary. Similarly, as most operators choose between a small set of fixed voltages, we do not fit the voltage dependence and instead, for simplicity, recommending characterizing each voltage separately.
Correction of Rotating Scan Frame Series
Where flyback hysteresis is present in each frame of a scan-rotated series, one edge will be corrupted for each of the principal scan-orientations; this has significant effects for both rigid and nonrigid registration. For rigid registration (correction of simple stage drift translation), the differing artifact profiles in each scan-orientation reduces the quality of the cross-correlation (or other correlation) used to align the data and may lead to poorer drift diagnosis. For nonrigid registration algorithms, where the average frame is used as a starting estimate (Jones et al., Reference Jones, Yang, Pennycook, Marshall, Van Aert, Browning, Castell and Nellist2015), the degraded starting average frame may reduce the quality and speed of the registration convergence. Instead, by pre-treating the multiframe data to first correct for the now calibrated flyback hysteresis, the speed and precision of the nonrigid correction can be improved. After pre-treatment, the fast-flyback multiframe series can be processed as if a conventional series and corrected for rigid-offsets (Bárcena-González et al., Reference Bárcena-González, Guerrero-Lebrero, Guerrero, Fernández-Reyes, González, Mayoral, Utrilla, Ulloa and Galindo2016), affine-shears (Sang & LeBeau, Reference Sang and LeBeau2014; Dycus et al., Reference Dycus, Harris, Sang, Fancher, Findlay, Oni, Chan, Koch, Jones, Allen, Irving and LeBeau2015), and nonlinear distortions (Jones et al., Reference Jones, Yang, Pennycook, Marshall, Van Aert, Browning, Castell and Nellist2015; Ophus et al., Reference Ophus, Ciston and Nelson2016; Wang et al., Reference Wang, Eren Suyolcu, Salzberger, Hahn, Srot, Sigle and van Aken2018) as necessary.
A series of 100 images of SrTiO3, each with a 20 μs line flyback time and a scan rotation of 90° between consecutive frames was captured to demonstrate the ability of this technique to correct flyback hysteresis artifacts for rotating image series. This image series was aligned and then averaged with, and without, hysteresis correction applied as the first step, with the results shown in Figure 6. The image comparison in Figure 6 shows the same image series both with and without flyback hysteresis correction applied as the first postprocessing step before the usual aligning and averaging. While the flyback-uncorrected average image (Fig. 6a) shows the distortions on all edges, the flyback-pretreated average image (Fig. 6b) maintains fidelity across the entire field of view.
Fig. 6. Aligned average of 100 ADF frames recorded with a 20 μs flyback time and a scan-rotation increment of 90° between frames. Average frame is shown without hysteresis correction (a) and with the proposed correction (b). (c,d) Show the variation of A and b, respectively, with rotation angle. The images analyzed were captured with a dwell-time of 2 μs and line flyback time of both 0 s and 20 μs with the rotation angle increasing in steps of 15° from 0° to 360°.
Averaging multiple frames is an important approach to increase the SNR where it is too low in a single frame, and also to encode redundancies into our datasets to aid in future processing. One reason the SNR can be too low in a single frame is when low dwell-times are being used, which is the region where the largest jumps in scanning efficiency are seen when the line flyback time is reduced (Fig. 2). It is therefore of key importance that the method presented here is compatible with multiframe imaging, as is shown.
Figure 6b is produced by applying hysteresis correction as the first step to each captured image frame individually. Although this works on a case-by-case basis, if we wish to construct a comprehensive model of the hysteresis, the general effects of scan rotation on the compression artifact should be investigated. To do so, a series of images of varying dwell-times and line flyback times were captured while also varying the scan rotation angle through a full 360° (a total of 50 images were recorded). The hysteresis diagnosis step was performed for each, with the calculated values for A and b plotted as a function of rotation angle in Figures 6c and 6d.
Figures 6c and 6d show that while b does not appear to vary in any meaningful way with the rotation angle, A varies sinusoidally with the rotation angle. The variation could be explained by factors such as different coil windings in the x and y scan coils. Another feature to note is that the cosine function is not at a maximum at a rotation angle of zero, that is, there is an offset. This was allowed as a parameter in the fit and could be explained by a physical assembly misalignment or a manufacturer firmware offset (e.g., for the orientation of the stage to the Ronchigram camera). Although the exact physical origins of these effects are not exactly known, reassuringly it can be seen that there are two cycles of the sinusoid in a full 360° rotation, corresponding to the two-fold rotation symmetry of the scan coils.
Having established the variations of A and b with rotation angle, dwell-time, and line flyback time, we now have all the empirical coefficients to fully describe the flyback hysteresis effects in our microscope. Simply knowing these three imaging parameters will allow the automatic correction of any future image captured and could even be applied retroactively to any previously captured data.
Conclusion
In this paper, we introduce the concept of scanning efficiency and lay bare the unfortunate reality that it can be surprisingly low for increasingly commonplace fast-frame imaging conditions. To increase the efficiency without also increasing the electron dose, the path to take is to decrease the line flyback time as much as is possible. Unfortunately, at low flyback times, a compression artifact due to hysteresis in the scan coils appears.
We have demonstrated the simple characterization of flyback distortions and the subsequent correction for this on single image frames, and multiframe datasets with or without scan rotation on subsequent frames. Not only is precision retained across the full field-of-view, ensuring techniques such as strain-mapping remains precise even when using low flyback times, but the scanning efficiency is increased (Supplementary Fig. S6). This paper describes the jump in efficiency from 48% (Fig. 2, point “A”) to 67% (Fig. 2, point “B”), an increase of almost 20%. This figure also indicates that to achieve very high efficiencies, such as labeled “C,” we need a separate approach to control the frame flyback time, such as a custom scan generator.
Furthermore, we have shown that it is possible to form a semi-empirical model of the hysteresis effect to be created, whereby any image captured with any arbitrary combination of line flyback time, dwell-time, and scan rotation can be corrected. Although this model is expected to vary by instrument, the method presented in this manuscript can, and should be, repeated by operators for their own instruments. This will enable the entire community of STEM operators to reduce electron dose, sample damage, and increase frame rates without any additional hardware purchase.
Code and Data Availability
The code which is used to measure and correct flyback distortions can be found at our Github repository https://github.com/TCD-Ultramicroscopy/STEM-Flyback-correction. The data used in this paper can be found at doi:10.5281/zenodo.5713336.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1431927621013908.
Acknowledgments
The authors would like to acknowledge the Centre for Research on Adaptive Nanostructures and Nanodevices (CRANN) and the Advanced Materials and BioEngineering Research (AMBER) Network for financial and infrastructural support for this work. J.J.P.P. and L.J. acknowledge SFI grant 19/FFP/6813, T.M. acknowledges the SFI & EPSRC Centre for Doctoral Training in the Advanced Characterisation of Materials (award references 18/EPSRC-CDT-3581 and EP/S023259/1).
Conflict of interest
The authors declare no competing interests.
Open access
