research communications
On crosscorrelations, averages and noise in electron microscopy
^{a}Department of Molecular Physiology and Biophysics, University of Vermont, 149 Beaumont Avenue, Burlington, VT 05405, USA
^{*}Correspondence email: michael.radermacher@uvm.edu
Biological samples are radiationsensitive and require imaging under lowdose conditions to minimize damage. As a result, images contain a high level of noise and exhibit signaltonoise ratios that are typically significantly smaller than 1. Averaging techniques, either implicit or explicit, are used to overcome the limitations imposed by the high level of noise. Averaging of 2D images showing the same molecule in the same orientation results in highly significant projections. A highresolution structure can be obtained by combining the information from many singleparticle images to determine a 3D structure. Similarly, averaging of multiple copies of macromolecular assembly subvolumes extracted from tomographic reconstructions can lead to a virtually noisefree highresolution structure. Crosscorrelation methods are often used in the alignment and classification steps of averaging processes for both 2D images and 3D volumes. However, the high noise level can bias alignment and certain classification results. While other approaches may be implicitly affected, sensitivity to noise is most apparent in multireference alignments, 3D referencebased projection alignments and projectionbased volume alignments. Here, the influence of the image signaltonoise ratio on the value of the crosscorrelation coefficient is analyzed and a method for compensating for this effect is provided.
Keywords: image processing; signaltonoise ratio; crosscorrelation; multireference alignment; 3D referencebased projection alignment.
1. Introduction
Cryoelectron microscopy of biological samples has made large strides towards achieving close to atomic resolution et al., 2017; Nobel Foundation, 2017). Since biological samples are radiationsensitive, micrographs are recorded under lowdose conditions, resulting in images with signaltonoise ratios (SNRs) substantially lower than 1 (or negative decibels; dB). In contrast, the detection of image details by the human eye requires SNRs with values ranging between 4 and 5 (Rose, 1973). To overcome this problem in biological images of single particles are aligned, classified and averaged to obtain clear projection images. 3D reconstructions are calculated from many images showing the molecule or macromolecular assembly in multiple orientations, using a much larger number than would be required by the sampling conditions. For 2D averaging, images are aligned rotationally and translationally, classified with multivariate statistical methods combined with multireference alignments, and subsequently the images corresponding to each class are averaged separately (Frank, 1975, 1978; Frank et al., 1978; van Heel & Frank, 1981; van Heel & StöfflerMeilicke, 1985). For 3D reconstructions a number of methods are used, all of which include a variation of a crosscorrelation process. If tomographic reconstructions are used as a starting point, they are often followed by 3D alignments and averaging. Crosscorrelations are also present when angular reconstitution or random conical tilt methods are applied to obtain first references, or when random conical tilt methods are applied for resolving multiple structures representing different conformations of a highly heterogeneous sample (Radermacher et al., 1987; van Heel, 1987; Radermacher, 1988; Bartesaghi & Subramaniam, 2009; Yu et al., 2010, 2013; Schmid, 2011; Asano et al., 2016; Wan & Briggs, 2016). Most of these techniques are followed by 3D referencebased projection alignments, in which the projection angles and xy positions are refined using crosscorrelation methods between a 3D reference structure and single 2D projections, or in which additional 2D projections are first aligned with a 3D reference and subsequently added to the 3D reconstruction (see, for example, Radermacher & Ruiz, 2006; Scheres et al., 2007).
(ChengAll of the above averaging approaches either explicitly or implicitly use crosscorrelation methods, and the very low SNR of the data may adversely affect the image processing and bias the results. The effect of noise on the crosscorrelation coefficient has been described previously for correlations between two images with the same SNR (Bershad & Rockmore, 1974; Frank & AlAli, 1975). Many steps in the processing of singleparticle data sets, however, include crosscorrelation procedures of images with different SNRs. These include, but are not limited to, the correlation of a single 2D image with a 2D average image, or the correlation of a 2D projection with a 3D volume reconstructed from a 2D projection set that is not evenly distributed, thus exhibiting different SNRs in different directions, which are apparent along the radial lines of the polar 3D Fourier transform or the 3D Radon transform. Since 3D projection alignments utilize comparisons of the projection transform with the central sections of the 3D transform of the structure, the varying SNR may bias the alignment results.
In 2D and 3D multireference alignment procedures, crosscorrelation coefficients are explicitly used when deciding the assignment of a test image or volume to a specific reference. Here, we analyze the effect of noise on the value of the crosscorrelation coefficient in 2D and 3D applications.
2. Theory
The value of the crosscorrelation maximum, when two images are crosscorrelated, depends on the agreement between the motifs in each image and on their SNR. In
image processing all 2D images and 3D structures typically originate from projections with similar noise content. This allows the calculation of the influence of noise on the crosscorrelation coefficient not only when correlating two images with the same SNR, but also when correlating single images to an average image from the same data set, or when correlating two averages, again derived from the same data set. The following calculations are estimates and use approximations. They will aid, however, in judging the effect of variations in the SNR on the outcome of a calculation. For simplicity we use the following assumptions: (i) the noise is white, additive and Gaussiandistributed with an average equal to 0, (ii) the signal and the noise are uncorrelated and (iii) the average of the signal is 0. The latter assumption is used to simplify the calculations but does not affect the results.From assumption (i) it follows that the
of the noise crosscorrelation is 0 andwhere n_{i}^{1} and n_{i}^{2} are two independent realizations of Gaussiandistributed white noise and M is the number of pixels in an image.
From assumption (ii), stating that the signal and noise are uncorrelated, it follows that the crosscorrelation between signal and noise also vanishes,
where s_{i} is the signal and n_{i}^{k} is the kth realization of Gaussiandistributed white noise.
In the following, the SNR α is defined as the ratio of the variances:
Under these assumptions, the well known equation for the value of the crosscorrelation in the presence of noise can be derived (Bershad & Rockmore, 1974; Frank & AlAli, 1975). Let C be the normalized crosscorrelation coefficient and (s_{i} + n_{i}^{x}) and (s_{i} + n_{i}^{y}) two images with the same motif but different noise. When no noise is present, the crosscorrelation coefficient C is 1.
Developing this equation, we obtain
Using the approximations stated above, the mixed signal and noise terms yield 0, the crosscorrelation of noise terms yield 0, and since n_{i}^{x} and n_{i}^{y} are two realizations of Gaussian distributed white noise
Defining
and using the definition of the SNR in (3), α = (s^{2}/n^{2}), which allows the substitution of s^{2} by s^{2} = α · n^{2}, (6) yields
The relation described in (7) can be used to determine the SNR of a series of images by calculating their pairwise crosscorrelations (Frank & AlAli, 1975):
(7) describes the crosscorrelation coefficient of two images with the same SNR. However, more often crosscorrelations are used for the alignment of noisy images with an average reference with a larger SNR. If the reference represents an average of images with the same signal and with the same SNR, then the crosscorrelation is also noisedependent and has a value smaller than 1. The crosscorrelation coefficient depends on the SNR of each single image and on the number of images used to calculate the average image.
It is well known that averaging improves the SNR. When averaging N images the standard deviation of the noise is reduced by N^{1/2}. Thus, if α is the SNR of a single image, defined as above as the ratio of signal and noise variances, the SNR of an average image containing N images is β = α · N. The crosscorrelation between a single image and the average image can then be calculated.
From (3), the SNR of a single image is α = (s^{2}/n^{2}) and the SNR of the averaged image results in
Defining the superscript x in (4) to indicate the single image and the superscript y to indicate the average, and using
and substituting s by the SNR α using the formula
The crosscorrelation coefficient C in (10) represents the value obtained when a single image is crosscorrelated with an average image containing N images of the same image set with the same SNR.
Equation (10) can be used to determine the SNR from the crosscorrelation of single images to an average image obtained fom the same data set:
For N = 1, (11) simplifies to (8) for determining the SNR from crosscorrelations of single images.
The calculations can easily be extended to the crosscorrelation between two average images calculated from different numbers of images. We can assume that both images are average images of the single images with noise n^{x}, the first average image contains L single images and the second average image contains N single images. (5) can then be rewritten as
Performing the same calculations as above and substituting s by the SNR α using the equation
yields
The crosscorrelation coefficient C in (13) represents the value obtained when an average image containing L single images is crosscorrelated with an average image containing N single images of the same image set with the same SNR per image.
3. Test calculations and a cryoEM data example
The theory was initially tested with model data for both pairwise correlations and correlations towards an average. We created model images with an SNR of 0.5 (Fig. 1). The motif was generated by first creating a random noise image, lowpass filtering and thresholding it to reintroduce higher frequencies; it was then rotated and masked. To introduce noise, a large image containing Gaussiandistributed white noise was first created. Subsequently, rows of nonoverlapping images with the same size as the motif were boxed out of the large noise image and the motif was added to obtain the final images. Both the motif and the noise image were scaled such that the variances of signal to noise had a ratio of 0.5. A total of 400 images were created. In a first experiment, all 400 images were crosscorrelated in unique pairs, excluding autocorrelations. The average crosscorrelation coefficient was determined to be 0.334, with a standard deviation of 0.0084. The SNR for each crosscorrelation coefficient, calculated using (8), yielded an average SNR of 0.501 with a standard deviation of 0.011. In a second experiment 50 images were averaged. Each of the 400 single images was crosscorrelated with the average and yielded normalized crosscorrelation coefficients with an average of 0.557 and a standard deviation of 0.007. The resulting SNR calculated as an average of individual SNRs using (11) was 0.508 with a standard deviation of 0.016, which is in good agreement with the SNR determined by pairwise correlation. All calculations were carried out using the Environment for Modular Image Reconstruction Algorithms (EMIRA; Radermacher, 2013).
The theory was subsequently applied to a set of experimental images for both pairwise correlations and correlations with an average. The data set used contained images of mitochondrial complex I purified from the yeast Yarrowia lipolytica (Radermacher et al., 2006) and prepared for microscopy in vitreous ice (Fig. 2a). Images were recorded on an FEI Tecnai 12 electron microscope equipped with an LaB_{6} filament operated in point mode at 100 kV (Ruiz et al., 2003; Ruiz & Radermacher, 2006). The typical defocus was approximately 1.8 µm and the nominal magnification was 52 000×. The micrographs were digitized on an SCAI flatbed scanner with a pixel size of 7 µm and subsequently binned down by a factor of three, resulting in a calibrated pixel size of 4 Å. A total of 1750 images of single particles were boxed out and subjected to multiple rounds of alignment, correspondence analysis and classification using Diday's method of moving centers (Diday, 1971), and ten final classes were obtained. Calculations to test the theory presented in this paper were carried out using the class with the largest number of particles, class 8, which contained 462 images (Figs. 2b and 2c). The original images had dimensions of 160 × 160; however, for this crosscorrelation experiment the images were boxed down to 80 × 80 to exclude most of the image background areas. No additional mask was applied, since the crosscorrelation program used normalizes over the whole image and does not allow normalization restricted to a specified mask. All images were lowpass filtered to 33 Å, the resolution of the average image determined by Fourier ring correlation with a cutoff of 0.5.
SNRs were determined using the same approach as for the model data for both pairwise correlations and correlations to an average. First all 462 images were crosscorrelated pairwise (using only unique pairs and excluding autocorrelations) and SNR values were calculated from each crosscorrelation coefficient. For pairwise crosscorrelation the average crosscorrelation value was 0.279 with a standard deviation of 0.053 and the averaged SNR was 0.394 with a standard deviation of 0.104. When the average of all 462 images was used as a reference, the average crosscorrelation coefficient was 0.530 with a standard deviation of 0.047 and the resulting averaged SNR was 0.404 with a standard deviation of 0.099. The SNR value found relative to the average is slightly higher, since in the crosscorrelation of individual images variations in the motif are expected to lower the average crosscorrelation coefficients.
The negative effect of the SNR on the crosscorrelation can be compensated for. We generated a data set containing 1000 images with an SNR of 0.5. The signal image was a pattern of 99 squares, and for each of the ten motifs created a different square was missing (Fig. 3a). The final data set contained 100 copies of each motif with noise added as before (Fig. 3b). Ten references for crosscorrelation were calculated, the first one by averaging ten images containing the first motif, the second by averaging 20 images containing the second motif etc., until the tenth containing the tenth motif was calculated by averaging 100 images (Fig. 3c). The 1000 images were recreated using a different noise image to avoid any noise correlation between the averages and single images. The images were correlated in a multireference (translational) alignment using a normalized crosscorrelation. During this test no filters or masks were applied since this would have changed the SNR of the data. In a conventional multireference correlation, no images were assigned to averages 1–3 and the plurality of images were assigned to reference 10 (Fig. 3d). The multireference alignment process was repeated using the inverse of (10) to correct the crosscorrelation values. After applying the correction the procedure assigned each image to its correct reference (Fig. 3e, Table 1).

4. Discussion
The calculations show the strong dependence of crosscorrelation coefficients on the image noise.
Equation (10) describes the value of the crosscorrelation coefficient when a single image with SNR = α is correlated with an average image calculated from N single images with the same SNR. In crosscorrelation alignment processes the crosscorrelation coefficient corresponds to the maximum of the correctly normalized crosscorrelation function. While the specific value of the crosscorrelation is of minor importance during rotational and translational alignments against a single reference, it can bias the results in multireference alignments when the references are averages of only a few and different numbers of images (van Heel & StöfflerMeilicke, 1985). In this situation the crosscorrelation with the reference with the highest number of images averaged will show a higher valued crosscorrelation, and if used in an iterative procedure, images may tend to be assigned to the reference that starts with the highest number of images averaged. Fig. 4 illustrates the influence of the number of images used to calculate the reference on the crosscorrelation coefficient. While for an SNR of >0.5 the change in the crosscorrelation is minimal when more than 40 images are averaged, for lower values of the SNR the dependence on the number of images averaged to calculate the reference is still strong.
The specific value of the crosscorrelation coefficient is also affected when images exhibiting a low SNR are correlated with an almost noisefree reference. Under these conditions, the ). For this particular calculation the value N in (10) was set to 1000. The abscissa shows the SNR and the ordinate the crosscorrelation coefficient. Its value is always smaller than 1.
manifests an asymptotic behavior (Fig. 5Multireference alignments are often used as classification tools. The extent of signal differences between the images determines whether the bias introduced by differences in the SNR of the references will have a significant effect on the outcome. If there are large differences among the signals in the images, the signal differences will dominate the selection process. However, if the differences are small then the bias can be significant. The effect is dominated by an asymptotic behavior, as shown in the curves in Figs. 4 and 5. The possible bias diminishes when either the SNR of the images increases or the SNR of the references increases. The effect is obvious in our model calculations, where in one single round of correlations a plurality of images were assigned to the average calculated from the most images, while none of the images were assigned to the averages calculated from 30 or fewer images.
The crosscorrelation value can be corrected by multiplication by the inverse of either (10) or (13) and will result in a crosscorrelation coefficient of close to 1 if identical motifs exist in the reference and in the image (Fig. 3e). The correction defined in (10) is implemented in our 3D referencebased projection alignment (Radermacher, 1994), where the angular distribution of the projections used in the calculation of the 3D reference is often uneven. In this alignment procedure 2D Radon transforms of the projections are crosscorrelated with the reference 3D Radon transform. The 3D Radon transform is created by an algorithm that averages the 2D Radon transforms of the projections into a 3D Radon transform. Accordingly, each radial line is an average of multiple radial lines from each of the projections. A counter is maintained indicating how many projection lines contribute to the average in each radial line at any specific angle (φ_{i}, θ_{j}). The crosscorrelations are carried out by crosscorrelation of each radial line in the 2D transform of the projection with each radial line in the 3D transform of the reference. The algorithm as implemented allows the (optional) normalization of each line correlation with the inverse of (10), using the counter of each line for N. The Radon inversion of the linebyline crosscorrelations at every angle provides the crosscorrelation function. For normalization, the SNR of either the projection or the 2D Radon transform of the projection needs to be known. If the SNR of the image is not known a priori, it can be estimated by performing the crosscorrelation between a single image and an average image. The best results are obtained when the calculation is carried out in the asymptotic range of (10).
One of the main sources of noise in lowdose electron micrographs is shot noise, which is approximately constant throughout the whole spectrum. Thus, the SNR strongly depends on the image resolution. The signal energy typically weakens towards higher frequencies or better resolutions. Therefore, besides using masks to exclude background noise, lowpass filters are extensively applied in alignment procedures by crosscorrelation. Above a certain radius lowpass filters remove more noise energy than signal, thus increasing the SNR. Any SNR value used for correction of the influence of noise on the crosscorrelation must be estimated taking into account any mask and bandpass filter applied in the crosscorrelation process. One should be careful not to overcompensate for the SNR; the crosscorrelation coefficient must not exceed a value of 1. The statistics of the crosscorrelation coefficients in the alignment process can be used as a safeguard against overcompensation.
The recently introduced method of projectionbased volume alignment (Yu et al., 2013) for aligning 3D volumes of macromolecular structures could easily take advantage of the theoretical principles described here. In this method, a set of projections with known orientations calculated from a reference volume are aligned with the volumes whose orientations are to be determined. If the reference volume is calculated by averaging the 2D Radon transforms of the projections into a 3D Radon transform, the extracted projections can also provide an index for each radial line that specifies how many projections contributed to each line. A correction for the influence of noise can now be implemented in this algorithm using (13) by making use of the occupancy counters of both the reference 3D Radon transform and the 3D Radon transform of the volumes being aligned.
5. Conclusion
Early literature has described the value of the crosscorrelation coefficient when two images with the same noise level are correlated. Here, we extended this equation to include crosscorrelations between images and their averages, and between image averages, all obtained from the same original noisy data. The results provide a method to correct for the major influence of noise, and we hope that they will help to increase the awareness of possible bias in multireference alignments and classification.
Funding information
This work was supported in part by NSF grant DBI 1660908.
References
Asano, S., Engel, B. D. & Baumeister, W. (2016). J. Mol. Biol. 428, 332–343. CrossRef CAS Google Scholar
Bartesaghi, A. & Subramaniam, S. (2009). Curr. Opin. Struct. Biol. 19, 402–407. CrossRef CAS Google Scholar
Bershad, N. & Rockmore, A. (1974). IEEE Trans. Inf. Theory, 20, 112–113. CrossRef Google Scholar
Cheng, Y., Glaeser, R. M. & Nogales, E. (2017). Cell, 171, 1229–1231. CrossRef CAS Google Scholar
Diday, E. (1971). Rev. Stat. Appl. 19, 19–34. Google Scholar
Frank, J. (1975). Ultramicroscopy, 1, 159–162. CrossRef CAS Google Scholar
Frank, J. (1978). Ann. NY Acad. Sci. 306, 112–120. CrossRef CAS Google Scholar
Frank, J. & AlAli, L. (1975). Nature (London), 256, 376–379. CrossRef PubMed CAS Web of Science Google Scholar
Frank, J., Goldfarb, W., Eisenberg, D. & Baker, T. S. (1978). Ultramicroscopy, 3, 283–290. CrossRef CAS Google Scholar
van Heel, M. (1987). Ultramicroscopy, 21, 111–123. CrossRef PubMed CAS Google Scholar
van Heel, M. & Frank, J. (1981). Ultramicroscopy, 6, 187–194. CAS PubMed Google Scholar
van Heel, M. & StöfflerMeilicke, M. (1985). EMBO J. 4, 2389–2395. CrossRef CAS PubMed Google Scholar
Nobel Foundation (2017). The Nobel Prize in Chemistry 2017. https://www.nobelprize.org/prizes/chemistry/2017/advancedinformation/. Google Scholar
Radermacher, M. (1988). J. Electron Microsc. Tech. 9, 359–394. CrossRef CAS PubMed Web of Science Google Scholar
Radermacher, M. (1994). Ultramicroscopy, 53, 121–136. CrossRef CAS PubMed Web of Science Google Scholar
Radermacher, M. (2013). Microsc. Microanal. 19, 762–763. CrossRef Google Scholar
Radermacher, M. & Ruiz, T. (2006). Methods Mol. Biol. 319, 427–461. CrossRef Google Scholar
Radermacher, M., Ruiz, T., Clason, T., Benjamin, S., Brandt, U. & Zickermann, V. (2006). J. Struct. Biol. 154, 269–279. CrossRef CAS Google Scholar
Radermacher, M., Wagenknecht, T., Verschoor, A. & Frank, J. (1987). J. Microsc. 146, 113–136. CrossRef CAS PubMed Web of Science Google Scholar
Rose, A. (1973). Vision: Human and Electronic. New York: Plenum Press. Google Scholar
Ruiz, T., Mechin, I., Bär, J., Rypniewski, W., Kopperschläger, G. & Radermacher, M. (2003). J. Struct. Biol. 143, 124–134. CrossRef CAS Google Scholar
Ruiz, T. & Radermacher, M. (2006). Methods Mol. Biol. 319, 403–425. CrossRef Google Scholar
Scheres, S. H. W., Gao, H., Valle, M., Herman, G. T., Eggermont, P. P. B., Frank, J. & Carazo, J.M. (2007). Nature Methods, 4, 27–29. CrossRef CAS Google Scholar
Schmid, M. F. (2011). Adv. Protein Chem. Struct. Biol. 82, 37–65. CrossRef CAS Google Scholar
Wan, W. & Briggs, J. A. G. (2016). Methods Enzymol. 579, 329–367. CrossRef CAS Google Scholar
Yu, L., Snapp, R. R., Ruiz, T. & Radermacher, M. (2010). J. Struct. Biol. 171, 18–30. CrossRef Google Scholar
Yu, L., Snapp, R. R., Ruiz, T. & Radermacher, M. (2013). J. Struct. Biol. 182, 93–105. CrossRef CAS Google Scholar
This is an openaccess article distributed under the terms of the Creative Commons Attribution (CCBY) Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are cited.