Remote microscopy

To meet the objective of providing wide-ranging access to top-of-the-line instrumentation, CEMAS has implemented a microscopy program that is having far-reaching ramifications in the way advanced microscopy capabilities are made available to a broader range of university and industrial clients, in both efficiency and economics. Furthermore, remote microscopy is having a major impact on the way microscopy and other characterization techniques are taught and the way both industry and academia collaborate. CEMAS continues to expand the program throughout Ohio and across the nation, and it is exploring the possibility of international links. The first phase of the program was instituted with funding from the Ohio Third Frontier Program and FEI Company, now ThermoFisherScientific, in partnership with the University of Dayton and the Air Force Research Laboratory at Wright Patterson Air Force Base [1].

Earlier remote operations

The idea of remote operation of electron microscopes is not new. There have been many “demonstrations” at microscopy conferences of data collection on a vendor’s exhibition booth from a microscope that is hundreds or even thousands of miles away. However, these are often a long way from being “live and real-time” operation; rather, they demonstrate that “over the shoulder” or “remote desktop” viewing is possible, but “live interactive control” of the microscope is much more challenging. Earlier efforts to develop “telemicroscopy” in universities and research centers were often limited either by the network bandwidth/latency or could only give limited control of the microscope functions [Reference Voelkl2–Reference Leyva-Porrasa9]. For example, while at Imperial College London, the lead author demonstrated the use of a microscope at Oakridge National Laboratory in Tennessee to the UK science minister during a visit to the Materials Department in London but with only limited microscope functionality [Reference Perkins10]. To our knowledge, the first implementation of a classroom environment with “live and interactive” microscope control was at Carnegie-Mellon University where De Graef et al. designed and implemented a remote laboratory where microscopes were “hardwired” to multiple stations to allow groups of students to be trained on SEM techniques [Reference De Graef11]. While this approach was effective locally, the approach was not scalable beyond the host institution. The topic has remained of interest to the community, and Sinclair and others have organized series of international meetings to discuss this topic and identify best practice [Reference Sinclair12].

Time delay

At CEMAS the first step was to identify the technical challenges that needed to be overcome. It quickly became apparent that reliable, high-speed communication between CEMAS and the remote station was an absolute requirement. Many researchers have experienced an international call where there is a time delay in the connection. Such a delay, especially in the midst of detailed technical conversation, can be very frustrating. Now imagine that same delay introduced into the control of a microscope. When the operator adjusts the focus or makes an alignment correction while looking at the output of the detector on the computer screen, he or she expects the response on the screen to be instantaneous. If the operator is sitting in a room 200 miles away and there is any delay in the response, operating the instrument quickly becomes inefficient and execution of a complex experiment becomes almost impossible. In this scenario, most CEMAS users will eventually conclude, “Forget it, I’m going to jump in the car and drive to Columbus.”

Network latency

The network communication required for remote microscopy must have two characteristics. First, it needs relatively high bandwidth because each image may contain many megabytes of data. But it needs more than just simple bandwidth. Networks make connections through switches, routers, and firewalls. The network path length and each device introduce delay. This delay is referred to as the network’s switching latency. Other factors such as congestion can cause jitter, where the latency is dynamic or, in the worst case, causes packet loss. The key to delivering the speed and reliability needed for remote microscopy is a high-quality connection, with low latency and minimal jitter and packet loss. If the latency is consistently low enough, then the user can sit in a remote site anywhere in the world and feel a suitable response to operator commands.

High-speed connections

The Ohio Academic Resources Network (OARnet, www.oar.net) and Internet2 (www.internet2.edu) provided the solution to this challenge (Figure 2). CEMAS has a direct connection to the OARnet, a 100-gigabit-per-second backbone that runs throughout the state of Ohio. OARnet was designed to move big data quickly and efficiently with minimal latency. It provides high-speed/low-latency connections to every major city in Ohio. Beyond Ohio, OARnet connects to Internet2, a nationwide high-speed network.

Figure 2 By utilizing the bandwidth and low latency of the OARnet and Internet2 networks, CEMAS has been able to install -control stations in locations hundreds of miles from Columbus, Ohio, that allow users to operate the microscopes without any perceptible delay.

First remote access partners

Having identified an appropriate network solution, the next step in developing a remote microscopy capability was to prove the concept through a practical demonstration with a remote partner willing to work through the development process. The University of Dayton (about 90 miles from Columbus) and the Air Force Research Laboratory at Wright Patterson Air Force Base, near Dayton, agreed to participate.

Working with FEI Company (ThermoFisherScientific), the development team designed a remote-control station that replicates a set of microscope controls, without the microscope. There are a variety of video encoders and decoders and other hardware and software that maximize the speed of the signal and handle all the different detectors. After a thorough process of debugging and optimization, the first remote-control station was installed at the Air Force Research Laboratory (Figure 3), and a few weeks later the second went to the University of Dayton [1]. Both remote-control stations are currently in use. Users send their specimens by courier to the microscopy facility in Columbus. When the sample is in the microscope, the user receives an indication on the network that the system is ready to go, and then they can use the microscope in the same manner as if they were sitting in front of it.

Figure 3 Image of the operation station installed in the Air Force Research Laboratory at Wright-Patterson Air Force Base in Dayton, Ohio.

Expanding the program

Having successfully proven the concept of functional remote microscopy, researchers at CEMAS are now discussing expansion of the project with federal funding agencies. The National Science Foundation (NSF) is very familiar with the challenges facing programs such as the Major Research Instrumentation scheme and is keen to see how the capabilities at CEMAS can be developed and extended. Ultimately, this approach could help NSF deliver higher research impact with a limited capital budget. For example, with NSF support a remote system was successfully installed at North Carolina Agricultural and Technical University (N.C. A&T) in June of 2018. This system, which is the first to be installed outside of the OARnet network, is able to control the instruments at CEMAS with the same high-quality operational standards as the direct OARnet connected systems at the University of Dayton and the Air Force Research Laboratory.

Ohio State University and N.C. A&T, an HBCU institution with an outstanding track record of addressing diversity in STEM education, have signed a strategic research partnership agreement that covers many common areas of activity, including electron and ion beam characterization. N.C. A&T, the first university partner outside Ohio, is home to the NSF Engineering Research Center for Revolutionizing Metallic Biomaterials (ERC-RMB). The link with CEMAS will allow ERC-RMB students to connect to our facility remotely, take part in interactive courses in advanced microscopy, and use CEMAS instruments in their research. The partnership with N.C.A&T is an excellent example of remote microscopy’s potential to bring together two world-leading research organizations to maximize the impact of research activities and federal funding.

Teaching microscopy

This is not only about providing researchers with access to advanced instrumentation for scientific research, this is about changing the way electron microscopy is taught and, ultimately, how the operation of any complex system is taught. Most electron microscopy teaching labs feature an electron microscope in a confined space with an “expert” operator sitting at the controls in front of the microscope, and four or five students crowded together peering over the operator’s shoulder. The expert demonstrates something, and then the students take turns at the controls duplicating the exercise. That’s how microscopists are trained today, and that’s how they have been trained for the past 50 years. It is tremendously inefficient. It limits the number of students to those that can clearly see over the instructor’s shoulder, and the presence of so many bodies close to the instrument can actually interfere with the instrument’s performance. Advanced microscopes are typically controlled remotely from a neighboring room because humans perturb the environment enough to interfere with the microscope’s ability to perform at the highest level. If a TEM can be operated remotely from the room next door, why not make that room larger and set it up specifically to enhance the training experience?

Classroom-laboratory

A digital lecture theater at CEMAS, completed in 2016, seats 33 students and has a video wall at the front that is 6 feet tall and 20 feet wide (Figure 4). An instructor can deliver a lecture and present examples from the microscope on the screen at the same time. Also visible are all the control panels of microscope, and the outputs of the detectors and the imaging system (Figure 5). The instructor has a set of digital microscope controls at the lectern, and all of the students in the theater have access to full microscope controls, in some instances from their own laptop or tablet. So rather than describing the theory and practice of operating the microscope to the students in the lecture hall and moving to the microscope in another location to demonstrate, the instructor can take control of the microscope and show the students the use of the instrument in real time. In turn the instructor can “flip” from the classroom to the laboratory environment, for example, inviting Francesca in seat 14: “You now have control. Will you please repeat what I have just shown you?” This not only overcomes the space limitations of microscope rooms, by creating an environment that is conducive to engaged learning, it changes the way in which students learn advanced characterization techniques. Trainees are sharing in the peer-group experience, not just relying on the one-to-one interaction with the microscope expert. The opportunities for education research on how students learn most effectively in this hybrid environment are currently being explored at CEMAS in collaboration with colleagues in the Department of Engineering Education at OSU.

Figure 4 Image of the OSU-FEI Electron Microscopy Collaboratory space at CEMAS. Multiple operation stations allow every microscope—TEM, SEM, FIB—to be controlled from this hybrid laboratory.

Figure 5 Operation of the aberration-corrected microscope in the OSU-FEI collaboratory space allows all members of the audience to learn microscopy and interact with the microscope in an environment that is conducive to active and engaged learning.

Remote students

The concept of the OSU-FEI Electron Microscopy Collaboratory goes beyond this. The lecture space has world-class audio and video conferencing capabilities, and with the OARnet connection instruction doesn’t need to be restricted to students in that room. The students could just as easily be at the University of Akron or in Cincinnati at some industrial or government facility. They can all have the same connection. All are able to see and interact, not only with the instructor, but also with the other users in the lecture theatre or anywhere in the world. Each student can see exactly what’s going on with the microscope. Most importantly, the students get an intimate feel for the operation of the microscope that cannot be gained in any way other than by turning the knobs and watching the result. This has the potential to revolutionize the way these kinds of skills are taught at the university level and provides new opportunities for highly effective workforce development and lifelong learning for engineers and scientists in technologically driven industrial sectors such as materials, manufacturing, pharmaceuticals, and medicine.

The entry level cost of the equipment necessary to create and install a remote operation station is relatively modest (<$100K) compared with the cost of advanced electron microscopes. This provides an opportunity for educators and researchers at universities and colleges with limited resources to gain access to over $40M of advanced instrumentation at CEMAS to support their teaching and advance their research programs. This is the democratization of electron microscopy, which leads the way to the democratization of science.

A new model for collaboration

The OSU-FEI electron microscopy collaboratory and our developments in remote microscopy provide a new and powerful platform for collaboration. We are transforming teleconferences to “telecollaborations.” The real-time operation of the microscope allows researchers to connect from different locations, not just to learn and work on their own, but to collaborate. Users can be in one location or in multiple locations; they can all talk to each other and see the screen, and anyone can take control.

Multidisciplinary research collaboration is essential in order to progress in many fields of research and to tackle the grand challenges facing the world in the 21st century. Whether it is to address drug development, clean water, sustainable manufacturing, or climate change, researchers from a wide range of disciplines and from an even greater number of organizations are trying to find efficient ways to work together. Our facility presented its vision at the 2015 Materials Research Society fall meeting as part of a symposium entitled “Engaged Learning of Science and Engineering in the 21st Century.” Many researchers are embracing the use of technologies that we take for granted in our regular lives—cell phones and tablet computers, for example—in teaching, learning, and research. Taking advantage of these technologies for collaboration and education is the vision behind the development of remote microscopy at CEMAS, and remote microscopy is only the beginning.