Upscaling X-ray nanoimaging to macroscopic specimens

In the era of diffraction-limited storage rings, can X-ray nanoimaging be extended to millimetre- or even centimetre-sized specimens such as whole mouse brains? The authors believe so and provide size-dependent imaging time and resource estimates based on calculated flux requirements and recent method developments in related disciplines.

geometric magnification onto a scintillator-based detector (Mokso et al., 2007), but the scintillator detectors used often have quantum efficiencies of 20% or lower (Martin & Koch, 2006). If one can instead place inefficient components upstream of the specimen being imaged, the inefficiencies increase the overall imaging time. However, they do not lead to the specimen receiving a higher radiation dose. This is the case of scanning microscopes, which require spatially coherent illumination in order to obtain diffraction-limited resolution from the optic (Jacobsen et al., 1991;Jacobsen, 2020).
Scanning and full-field imaging make different demands on instrument stability.
Because full-field imaging in absorption or Zernike phase contrast can be done with incoherent sources and with all pixels collected in parallel, it can be done with much shorter exposure times so that there is less sensitivity to position drift between specimen and optic. In x-ray nanoholography, drift can lead to blurring of the image but one does not have to quickly move beam positions as in scanned beam methods.
With scanned beam methods, drift during data collection can lead to distortions in the image or resolution degradation in the case of ptychography; however, one can correct for these scanned beam position errors in ptychography (Guizar-Sicairos & Fienup, 2008;Zhang et al., 2013;Dwivedi et al., 2018).

Scaling of previous experiments
Scaling from previous experiments can add confidence to the estimates made in the main text.
Two examples of high-throughput 2D ptychographic imaging of the integrated circuit layer within a silicon wafer provide insights on what can be achieved. Using 6.2 keV X rays at the Swiss Light Source to image in a 10 µm thick silicon wafer, an effective imaging rate of T p = 40 µs for a resolution of ∆ p = 41 nm was achieved . At the Advanced Photon Source at Argonne, 8.8 keV X rays were used to image in a 130 µm thick silicon wafer ; a double-multilayer monochromator was used to obtain a spatially coherent flux of about 1 × 10 10 photons/s in 1% spectral bandpass, with an imaging rate of T p = 11 µs for a resolution of ∆ p = 10 nm. This latter example involved scanning the focus of a Fresnel zone plate with 50 nm outermost zone width (and thus 49 nm theoretical probe FWHM size, given that the central 33% of the optic was obstructed with a central stop; see Fig. 4.32 of (Jacobsen, 2020)) at a linear speed of 150 µm/s while collecting 3000 diffraction patterns per second. Both of these examples scale reasonably well to the pixel time estimates shown in Fig. 7 of the main text when one accounts for sample thickness, spatial resolution, photon energy used, and coherent flux values obtainable in the future as shown in Fig. 6. In the latter case, the main limitation was the maximum "frame rate" of the detector used. While there are direct-detection x-ray pixel array detectors that use per-pixel analog charge storage to record up to 352 images at a burst frame rate of 4.5 MHz, the sustained frame rate of these detectors is limited to 16 kHz, or a sustained per-frame time of 63 µs (Allahgholi et al., 2019). The Swiss Light Source example  can be scaled to larger sample imaging as follows: • This example was done at the present-day Swiss Light Source with a brightness about 600 times lower than that expected from the APS-U (the APS-U brightness was used in calculating the spatially coherent flux shown in Fig. 6 of the main text; it should be noted that the Swiss Light Source is also planning an upgrade to a diffraction-limited storage ring). Thus one can divide the exposure time by 600 to account for this.
• The experiment at the C-SAXS beamline used a double-silicon-crystal monochromator with about 0.01% spectral bandpass. In fact, one can carry out ptychography using a much broader bandwidth (Enders et al., 2014;Yao et al., 2019), IUCr macros version 2.1.11: 2019/01/14 such as the full 1% bandwidth available from typical undulator sources. Thus one can divide the exposure time by 100 for (1% bandpass/0.01% bandpass).
• The experiment at the C-SAXS beamline used a Fresnel zone plate focusing optic with 100 nm thick gold zones with a theoretical diffraction efficiency (Kirz, 1974) of about 0.6%, whereas broad bandpass multilayer-coated Kirkpatrick-Baez mirror optics have demonstrated 45% focusing efficiency (da Silva et al., 2017).
These optics were used in connection with a multilayer-coated monochromator with 4% bandpass so as to limit exposure to other photon energies, so it is reasonable to assume a combined efficiency for monochromator and optics of about 20%. Thus one can divide the exposure time by (20%/0.6%), or a factor of 33.
• The C-SAXS experiment was done at 41 nm spatial resolution, whereas the pixel time calculation of Fig. 7 of the main text was for 20 nm resolution. We use the scaling ofn pixel ∝ t −2 f of Eq. 2 of the main text to translate the results to higher spatial resolution. Thus one must multiply the exposure time by (41/20) 2 = 4.2.
This leads to an estimated per-pixel imaging time of 8.4 × 10 −11 seconds using 6.2 keV photon energy. In fact, the pixel time calculation shown in Fig. 7 of the main text suggests that the optimum photon energy to use for 10 µm thick silicon background is 1.45 keV rather than 6.2 keV, so let us also consider the scaling of this result to 1.45 keV photon energy: • From the data used in Fig. 6  from λ 1 to λ 2 goes as where we have used tabulated values (Henke et al., 1993) for the refractive index of copper and silicon at 1.45 keV versus 6.2 keV. Thus one can divide the exposure time by 18.7.
• Also from Eq. 2 of the main text, we must correct for the increased absorption in the 10 µm silicon background thickness when moving from λ 1 to λ 2 . This absorption scaling s a goes as using tabulated values (Henke et al., 1993) for the linear absorption coefficient µ of silicon. Thus one must multiply the exposure time by 2.9 to account for the absorption scaling s a .
Taken together, the scaling of this example to 14.5 keV photon energy leads to an estimate of a per-pixel imaging time of 6.5 × 10 −13 seconds, whereas the calculation results shown in Fig These examples indicate that significant advances in ptychography methods and instrumentation will be required to scale x-ray nanoimaging up to macroscopic objects: • One of the adjustable parameters in a ptychographic imaging experiment is the ptychographic resolution gain which is the ratio of the diameter d of the scanned coherent illumination spot relative to the target spatial resolution δ r (for simplicity, we assume a circular illumination spot, though in practice one might have an Airy illumination pattern from a lens). In order to maximize the ptychographic imaging throughput, small values of G p are preferred . With a smaller ptychographic gain G p , one can relaxed the spectral bandwidth ∆λ/λ ≤ 2/G p and thus make better use of the full ∼ 1% bandwidth of the output of undulators at diffraction-limited storage rings (Fig. 6 of the main text). The minimum number of pixels N det,min required along one edge of the detector is while the exposure time per illumination spot T p can be written as Thus the ideal detector for high throughput ptychography is one with a relatively modest number of pixels N det,min (for example, 128×128 or 256×256 pixels) but the highest possible frame rate, ideally approaching the inverse of the per-pixel illumination time shown in Fig. 7 of the main text.
• A standard ptychographic dataset with a 2D diffraction pattern per 2D probe position provides information compartmentalized in 4 dimensions, so that it is highly redundant for reconstructing a 2D complex exit wave from the specimen (Edo et al., 2013;da Silva & Menzel, 2015). One can therefore use sparse sampling and regularizers in optimization-based image recovery approaches, as discussed in Sec. 5.1 of the main text, to obtain successful ptychographic image reconstructions with compressed data (Stevens et al., 2018b).
• The method originally employed for continuous-probe-motion ptychography was a simple raster scan with a sawtooth position-versus-time scan in one direction, and a discrete position step in the orthogonal transverse direction (Pelz et al., 2014;Deng et al., 2015;Huang et al., 2015). However, other scanning approaches have been developed, including spiral scans for reducing artifacts with reduced probe overlap (Dierolf et al., 2010;Huang et al., 2014). Archimedean spiral scans have been used for high-speed scanning in atomic force and scanning tunneling microscopy (Sang et al., 2016;Ziegler et al., 2017), and Hilbert "snake" scans have been used in scanning transmission electron microscopy (Velazco et al., 2020). Lissajous trajectories offer another approach to minimize sharp accelerations in both orthogonal scanning directions (Sullivan et al., 2014). One can also use specimen rotation as the fast axis for ptychography (Gürsoy, 2017).
Therefore, one can use some flexibility in choosing a scanning approach that best exploits the capabilities, and avoids imposed limitations, of the scanning stage and control system at hand.
• Because x-ray mirrors operate at grazing incidence, it is difficult to use methods such as the resonant galvanometric mirror pairs of confocal light microscopy (Wilke, 1983) to scan the illumination beam. Instead, one usually physically scans either the specimen or the optic. If the optic is scanned, one risks translating the coherent illumination variations on the optic into variations on the complex illuminating probe. While one can account for probe variations in the reconstruction (Thibault & Menzel, 2013), one can also avoid this by using rapid optic scanning over small areas and slower specimen scanning over larger areas . While rapid, precise scanning is technically challenging, the challenges can be met if sufficient resources are brought to bear. This is most    figure), one can annotate features in the reconstructed image and use them to train a neural network to identify diffraction patterns that are obtained when the beam crosses a desired feature (such as a synapse in brain tissue). When scanning the entire sample at low fluence (with beam positions shown as blue dots in the lower figure), the trained neural network allows one to identify regions which are likely to contain the desired features, and thus increase the x-ray fluence used (shown as orange or red illumination spots in the lower figure) either on this scan or on a subsequent scan at a slightly different angle in tomographic data collection.

Smart scanning approaches
As noted briefly in the main text, one can improve imaging speed using "smart scanning" to collect more data from crucial regions (such as synapses in brain tissue), and less data from homogeneous regions, or regions which do not contain information that is as crucial for understanding the material under study.
For some types of specimens, unsupervised machine learning can be applied directly on incrementally acquired data. This allows one to extract distinct features in the sample, and thus guide scanning trajectories to regions of scientific interest. Techniques such as Bayesian compressive sensing (Donoho, 2006;Candès et al., 2006;Ji et al., 2008) have been successfully applied to image acquisition (Trampert et al., 2018;Stevens et al., 2018a) and demonstrated real-time feedback during scanning.
In subsampled ptychography, one first learns a "dictionary" of textures present in the specimen (Kreutz-Delgado et al., 2003;Aharon et al., 2006) and then uses this dictionary to "inpaint" the most likely combination of textures into image regions that have sparsely sampled actual data. This capability is particularly beneficial to applications such as integrated circuits that have numerous copies of near-identical structures. However, such approach will not work when an axon and dendrite are in close proximity without having an actual synaptic connection (Kasthuri et al., 2015); that is, one may have regions which look very similar in undersampled data so that the act of inpainting could potentially lead to an unacceptably high number of false (connection) positives in the reconstructed connectome. Therefore, a "smart" scanning is desired which can adaptively improve its own scanning strategy.
One candidate to achieve the high-speed and dose-efficient scan is through the so-called "active learning" approach (Cohn et al., 1996) which enables an adaptive x-ray experimental design that optimally distributes resources (time, tolerable dose, etc.) and acquires the "useful" data at minimum cost. Active learning frameworks have shown successes in many fields (Tong, 2001) including microbiology (Hajmeer & Basheer, 2003), neurophysiology (Lewi et al., 2009), and manufacturing (Jones et al., 2010). Their capability to automatically identify and classify sample features will allow real-time informative feedback, and can be the key to studies of small features in a large sample volume. Furthermore, an active learning approach can provide a structured and amendable framework to incorporate a priori morphological knowledge into the data acquisition protocol.
Another application is the detection of the cracks in battery materials, since these cracks indicate degradation of the material leading to lower energy density and higher product-life-cycle cost (Kamaya et al., 2011). Here one wishes to catch rare events such as crack formation and propagation that occur over sub-second timescales, with nanometer-scale detail yet with cracks that may extend to lengths of millimeters. Traditional full-specimen raster scanning schemes will miss the details of crack formation during the charging process. Instead, intelligent scanning schemes that dynamically guide data collection along the crack as it forms in operando experiments can allow one to trace crack formation and evolution, hopefully leading to an understanding of the underlying mechanisms and contributing to the development of improved battery materials. In this context, initial random coarse scanning of a large region can first provide a refined region of interest, and rapidly train a neural network to recognize crack features in either direct (real-space) or indirect (e.g., ptychographic) imaging.
This can then guide fine scanning along the developing crack, with additional data being used to further refine the learned model and possibly incorporate additional evolving features.
Similarly, one could devise a different type of "smart" scanning to optimize the fluence. For example, one can use local tomography to image a small volume at uniform full fluence, and carry out a detailed analysis on that volume to identify the subregions where full fluence is actually required for unambiguous feature detection. In the con-text of ptychography, one could only acquire the coherent diffraction patterns obtained from illuminating the previously identified critical features, and use these data to train a neural network to recognize the regions where higher fluence is required to recognize more structural detail. When scanning the entire specimen, one could use the 2D scan taken at one rotational angle to inform the scanning hardware at the next rotation angle on how to modulate the scanning speed to adjust the fluence (see Fig. 1). In a rotation-as-fast-axis ptychographic approach (Gürsoy, 2017), one could use fast modulation of the transverse probe position during rotation to achieve a similar result.
In the case of connectomics, this could let one go from lower fluence data telling one where synapses might be, to acquire sufficient signal for high confidence in knowing that a specific region is (or is not) a synapse.
A challenge that this "smart" scanning scheme may face is that when a thick specimen is imaged, each 2D diffraction pattern encodes the projected information through the sample, and the feature overlay along the beam axis, as mentioned at the beginning of Sec. 3 of the main text, may hinder the neural network from determining exactly whether the diffraction pattern contains the desired feature. A possible workaround is to use a stack of diffraction patterns collected from multiple (preferably largely spaced) viewing angles in both training and prediction, so that the multi-angle input can hopefully disambiguate the projection overlay. In practice, this solution would be more compatible with the rotation-as-fast-axis ptychographic approach if multi-angle diffraction patterns are acquired in an interleaving fashion (Kaestner et al., 2011): for each lateral probe position, one could first collect a few diffraction patterns covering the full range of rotation, and pass the data to the neural network. This could allow the neural network to predict whether the ring-shaped region contains the desired feature, and if not, subsequent diffraction patterns at interleaving angles can be acquired with lower fluence.

To stain, or not to stain: examples in the literature
As noted in the main text, tissue staining methods offer improved contrast but reduced penetration. Several en bloc staining protocols have been developed for osmium and uranium staining of membranes deep into brain tissue (Mikula & Denk, 2015;Hua et al., 2015;Genoud et al., 2018), as required for serial sectioning and serial blockface imaging in electron microscopy connectomics. These protocols typically require significant time to allow for stain infiltration, and they can show variability in staining depth and staining density (Fera et al., 2020). Both challenges increase with specimen size.
An alternative approach is to use immunohistochemical (IHC) methods engineered to target different receptors of the neurons and extraneuronal tissue based on antibody binding selectivity. Their recent use for macroscopic x-ray imaging demonstrates protocol scalability to the entire mouse brain (Depannemaecker et al., 2019;Massimi et al., 2019), though ultrastructural integrity has yet to be evaluated in detail. Molecular tags developed for correlative cellular imaging (Clarke & Royle, 2019; Victor et al., 2020) might also prove useful for staining brain specimens. Alternatively, using genetic labeling approaches, specific heavy or exogenous elements can also be transported to target location via adeno-associated virus following in vivo injection .
From an imaging perspective, the staining and labeling methods described above modify the spatially-resolved absorptive (β) and phase-shifting (δ) parts of the x-ray refractive index simultaneously (Eq. 1 in the main text), so as to increase contrast.
Unfortunately, the chemical modification of stains and fixatives can lead to alteration of the specimen morphology at the subcellular level (O'Toole et al., 1993). In light of this, we may ask the question whether absorption or phase contrast of unstained brain specimens alone is sufficient for delineating structural boundaries for connectomics. There are already examples of x-ray imaging of unstained brain tissue where certain neuronal features can be identified, including examples with paraffin embedding Hieber et al., 2016;Ren et al., 2016;Töpperwien et al., 2017;Khimchenko et al., 2018).
Because the radiation dose necessarily imparted to a specimen increases significantly with improvements in spatial resolution (Sayre et al., 1977;Howells et al., 2009), cryogenic specimen preparation and imaging conditions are preferred for preserving morphology (O'Toole et al., 1993) and chemistry (Perrin et al., 2015;Jin et al., 2017).
Present-day cryogenic sample preparation aims mainly at the smaller specimen sizes amenable to electron microscopy. High-pressure freezing has been developed mainly for sub-millimeter-size specimens (Moor, 1987) but approaches based on gas pressurization followed by cooling (Kim et al., 2005) are being explored for cryogenic sample preparation of larger specimens. Cryogenic imaging has used for studies of chemically fixed but unstained brain tissue (Shahmoradian et al., 2017), and also paraffinembedded, stained brain tissue (Kuan et al., 2020). The latter study compared phase contrast x-ray imaging against serial section electron microscopy, and found similar contrast from metal staining, implying a similar level of biological interpretability between two modalities. Additional GABAergic neuronal nuclei labeling with peroxidase APEX2 (Lam et al., 2014) provides cell type information. The multi-contrast imaging strategy they demonstrated (Kuan et al., 2020) can guide the design of x-ray imaging experiments to maximize their structural information. Another approach is to use the natural distribution of metals in the brain to add functional information in unstained specimens, whether through atomic spectrometry (McAllum & Hare, 2019) or x-ray induced x-ray fluorescence (Bourassa & Miller, 2012), which can also be combined with the staining and labeling methods described above.
To image function-induced changes in dense neuroanatomy, one can rapidly freeze a specimen after sensory stimulus. One can then look for synaptic junctions that were undergoing neurotransmitter release, as evidenced by fusing vesicles containing neurotransmitters captured "in action" at the release sites of the neuronal axon. This has been done decades ago using transmission electron microscopy, where electrical stimulation and slam freezing was followed either by freeze fracture (Heuser et al., 1979), or freeze-substitution and uranyl acetate staining followed by serial sectioning (Heuser & Reese, 1981). A more modern approach (Watanabe et al., 2013;Kavalali & Jorgensen, 2014;Watanabe et al., 2014) has used cultured neurons infected with lentivirus expressing a rhodopsin derivative that can be triggered using blue light, followed by rapid high-pressure freezing, freeze-substitution, uranyl acetate staining, plastic embedding, and sectioning (Watanabe et al., 2013). When combined with genetic targeting, in principle this type of approach could lead to a very sparse set of target synapses to search for using the adaptive scanning approaches discussed in section S3.