Scanning transmission electron microscopy (STEM) has become the main stay for materials characterization on atomic level, with applications ranging from visualization of localized and extended defects to mapping order parameter fields. In recent years, attention has focused on the potential of STEM to explore beam induced chemical processes and especially manipulating atomic motion, enabling atom-by-atom fabrication. These applications, as well as traditional imaging of beam sensitive materials, necessitate increasing the dynamic range of STEM in imaging and manipulation modes, and increasing the absolute scanning speed which can be achieved by combining sparse sensing methods with nonrectangular scanning trajectories. Here we have developed a general method for real-time reconstruction of sparsely sampled images from high-speed, noninvasive and diverse scanning pathways, including spiral scan and Lissajous scan. This approach is demonstrated on both the synthetic data and experimental STEM data on the beam sensitive material graphene. This work opens the door for comprehensive investigation and optimal design of dose efficient scanning strategies and real-time adaptive inference and control of e-beam induced atomic fabrication.