Better models by discarding data?

Making the most of hard-won data in protein crystallography: to keep or not to keep, that is the question.


Introduction
Since the number of reflections in a crystallographic experiment is high, indicators of aggregated statistical properties are needed. For decades, the 'merging' R value, ; has been used almost exclusively for this purpose. This normalized linear residual was defined (Arndt et al., 1968) ad hoc to measure the consistency of measurements made with the first two-dimensional detectors by utilizing the multiplicity (also called redundancy) of observations of the unique reflections. The formula sums the absolute deviations of intensities of n i observations of unique reflections from their averages and normalizes using the sum of the intensities. As a relative measure of deviation, it can be calculated as an overall quantity for a data set, but also as a function of resolution. Later, it was shown (Diederichs & Karplus, 1997) that each term of the numerator has to be modified by a factor of [n i /(n i À 1)] 1/2 to give a result that is independent of the average multiplicity. The resulting quantity is called R meas (or the redundancy-independent merging R, R r.i.m. ; Weiss, 2001) and reports on the consistency of the measured observations. It was also realised that a higher multiplicity of observations results in the merged data being of higher quality than the individual measurements, so a distinct statistic, R mrgd , was introduced to assess the merged data quality (Diederichs & Karplus, 1997). Thereafter, it was shown that the quality of the merged data could also be estimated by introducing an additional factor of 1/n i 1/2 into each term of the R meas numerator (Weiss, 2001), as this accounts for the expected increase in accuracy associated with averaging n i measurements. The resulting quantity is called R p.i.m. (the precision-indicating merging R; Weiss, 2001).
Recently, we (Karplus & Diederichs, 2012) and Evans (2011) have suggested that the Pearson correlation coefficient of two 'half' data sets (i.e. each derived by averaging half of the observations for a given reflection) might be better suited than merging R factors for assessing data quality. In our work, we designated this quantity CC 1/2 and conclusively showed that in many ways it has a better statistical foundation than merging R values (Karplus & Diederichs, 2012) and that in particular it provides a direct assessment of the relative proportions to which signal and noise contribute to the variation in the data in a given resolution shell. Furthermore, we introduced a quantity termed CC*, defined as CC* (with an associated uncertainty) is an experimental estimate of what could be called CC true , the correlation of the final merged data with the underlying true values of these quantities. CC* is thus an upper limit for CC work and CC free from a properly refined model, where the latter are correlation coefficients between intensities calculated from the model and those obtained from the experiment. CC* is limiting because if the intensities calculated from a model match the experimental data better than the (unknown) true intensities do, this implies that the model is overfitted, fitting not just the signal but also the noise that is in the data. The relationships between these correlation coefficients are schematically summarized in Fig. 1.
Any scientific experiment entails data processing, which can include outlier detection and rejection. In crystallography, common practices include systematically rejecting certain subsets of data in order to 'improve' the resulting merged data set. The kinds of data that are sometimes rejected are whole data sets, high-resolution shells, unique reflections in the final reduced data set and single observations before merging or combinations thereof. While these practices do serve to improve the merging R-value statistics associated with the data, as far as we are aware little effort has been made to assess how these procedures impact the quality of the model that is produced by refinement against the data, which in our view is the single outcome that matters.
The only such study of which we are aware is our recent work using a novel 'paired-refinement' strategy to show that the inclusion of often-rejected weak data in high-resolution shells significantly improves the quality of the refined models (Karplus & Diederichs, 2012). Paired refinement means that a starting model is refined using the same refinement protocol against both a full data set and an (in some way) truncated version of the full data set. The resulting two models are then compared in terms of R values (R work , R free ) to judge which model is better. Importantly, the comparison of R values is only meaningful when the truncated data set is also used to calculate R work and R free of the model that was refined against the full data set.
For our test cases, paired refinements indicated that including data to the resolution at which CC 1/2 was in the range 0.1-0.2 led to an improved model, even though at these resolutions the traditionally used statistics were well beyond the conventional limits (the limiting signal-to-noise ratios were near 0.3-0.6 and R meas was in excess of 300%). Interestingly, this CC 1/2 range is a reasonable match to the value of 0.143 which was proposed in the field of electron microscopy for a quantity (FSC; Fourier shell correlation) related to CC 1/2 as an appropriate limit for discarding data because they correspond to a CC* value of 0.5 (Rosenthal & Henderson, 2003). Although this value indeed seems to be a reasonable cutoff for X-ray data for the test cases we studied, we resist generalizing this point and suggest that the high-resolution cutoff is in general better decided using the 'paired-refinement' strategy (Karplus & Diederichs, 2012  data sets may behave differently, and secondly, it allows for the possibility that as refinement programs improve they may be able to more fully extract structural information from weak data. Given the paired-refinement technique and the novel correlation coefficient-based data-quality indicators, it is now possible to systematically investigate the impacts of other data-selection practices mentioned above. Here, we document further properties of CC 1/2 and CC* and also explore how the various debatable 'data-filtering' procedures impact these new and the conventional R-value-based statistics, as well as model quality.

Data sets
Crystals of cysteine dioxygenase (CDO) were grown and soaked as described previously (Simmons et al., 2008). Data frames were collected on beamline 5.0.1 at the Advanced Light Source.
The data frames were processed and scaled with XDS (v. December 6, 2010; Kabsch, 2010a,b) and data sets were merged with XSCALE using default parameters. The space group is P4 3 2 1 2, with average unit-cell parameters a = b = 57.5, c = 122.2 Å .

Quality indicators
In addition to data-quality R values, it is conventional to quantify the signal-to-noise ratio of the merged intensities, given as hI/i. We used a custom program HIRESCUT to evaluate R meas , hI/i, CC 1/2 and CC* as a function of resolution, a feature that has since been merged into XDS and XSCALE. Furthermore, we added functionality to HIRESCUT that allows the rejection of single observations or unique reflections according to their I/ ratio.
The CCP4  program SFTOOLS was used to obtain correlation coefficients between experimental intensities and F 2 calc from the model.

Refinement
To obtain a model suitable for refinement, we used the PDB entry representing a related structure (PDB entry 2b5h; Simmons et al., 2006); the S-cysteine-persulfenate ligand was added guided by a difference Fourier map. H-atom positions were constructed and the resulting PDB file was refined in phenix.refine (v.1.7.1; Adams et al., 2010) using default parameters for the weights and number of macrocycles and updating the solvent model in every cycle.

Results and discussion
The main goal of this work is to consider three common questions that arise as part of data reduction and explore how the new correlation-coefficient-based data-quality measures behave and what the paired-refinement strategy indicates about the choices that will deliver the best refined models. The first of these scenarios is that of having a strong data set and a weaker data set, and we ask whether it is wise to merge the two data sets or to just use the stronger one. The other two scenarios have to do with practices that are not considered good practice by experts but are sometimes used to improve data-reduction statistics by deleting selected weak reflections.

Can strong data be improved by merging with a weaker data set?
This set of analyses uses three data sets (CDO3, CDO4 and CDO5, each corresponding to 90 of rotation) collected from different crystals that were grown and handled equivalently. According to all data-quality indicators [CC 1/2 , R meas and hI/(I)i] CDO3 is the best data set (Table 1), with CDO4 and CDO5 being similar to each other and of lower quality than CDO3. These data sets were merged in all possible combinations, and refinement in phenix.refine, both isotropically and anisotropically, was carried out against the resulting data sets using the same test set of reflections and the same highresolution limit of 1.57 Å . Diederichs & Karplus Better models by discarding data? 1217 Table 1 Statistics of single and merged CDO data sets.
The resolution range is 50-1.57 Å ; values in parentheses are for the highest shell (1.61-1.57 Å ). CC statistics are only given for the highest resolution shell because the overall CC values are always close to 1 and thus are uninformative. For CC 1/2 , CC* and CC work (based upon $2000 reflection pairs per shell) the values in the lower resolution shells are always higher than those in the highest resolution shell. This is not always true for CC free , which owing to the smaller set of reflections ($100 reflection pairs in each shell) has a standard error ($0.1) that is much larger than that of CC work ($0.02).
The increased value of R meas , if taken as an indicator of data quality, would suggest that merging of CDO3 (R work /R free = 0.186/ 0.219) with either CDO4 or CDO5 should significantly decrease the quality of the resulting data set, but in fact, based on the overall R work /R free , the model quality slightly decreases for CDO3+4 (R work /R free = 0.185/0.221) but slightly improves for CDO3+5 (R work /R free = 0.185/0.216). However, the comparison of model R values in this way is not really meaningful, since they are calculated against different data sets. This problem is overcome by the paired-refinement technique (Table 2), which unambiguously confirms that the model obtained by refinement against CDO3+5 fits data set CDO3 better than the original model obtained by refinement against CDO3. Conversely, the paired-refinement technique confirms that refinement against a merged CDO3+4 data set does not produce a model that better fits CDO3 than the original model.
In both cases, CC 1/2 /CC* correctly predict this result: the value of CC* in the highest resolution shell is increased relative to CDO3 for the CDO3+5 data set but not for the CDO3+4 data set.
In contrast, in the case of hI/(I)i the overall value decreases but the value in the highest resolution shell increases in both cases, so the influence of data set merging upon model quality is difficult to anticipate.
Merging of the two weak CDO4 and CDO5 data sets yields a better data set, as revealed by the decreased R free (R work / R free = 0.199/0.237) of the model refined against it. However, this model only fits CDO5 better than the original model refined against CDO5; it does not fit the CDO4 data set better despite the increased CC*. The explanation in this case is most likely that CDO4 is less complete, in particular in the highest resolution shell, than CDO3 and CDO5. Therefore, the caveat mentioned above, namely that comparison of crystallographic indicators should be performed against the same data, applies. In addition, slight non-isomorphism between CDO4 versus CDO3 and CDO5 is conceivable. In this case in particular, it is important to use the paired-refinement technique.
We note that in all cases the isotropic CC work (Table 1) is significantly worse than the anisotropic value and that the anisotropic R free is lower by 0.5-1.3% than the isotropic R free (Table 1), which indicates that there is some justification for anisotropic refinement. However, we decided not to show the detailed results of anisotropic refinement since it is not completely justified: compared with isotropic refinement, anisotropic refinement reduced R work by about 3%, thus widening the R free -R work gap; likewise, a wider CC work -CC free gap is observed for anisotropic refinement than for isotropic refinement. This is consistent with the observation that the anisotropic CC work reaches or slightly exceeds CC*, indicating overfitting. In summary, R meas is not a useful indicator of merged data-set quality, whereas CC 1/2 , and to a lesser extent hI/(I)i, correctly indicate the direction of quality change upon merging. CC* is a meaningful upper limit of CC work ; CC work of isotropic models is found to be significantly lower than CC*, whereas CC work of anisotropic models is slightly higher than CC*. The latter finding also suggests that sphericity restraints in anisotropic refinement are

Does discarding weak observations to improve conventional data-quality statistics actually lead to better models?
It is obvious that by discarding the weakest data it is possible to create a data set with better conventional statistics at high resolution [i.e. lower R meas and higher hI/(I)i], so that a higher resolution cutoff appears to be justified. While we did not find publications about these practices, we know from discussion with students and colleagues that they are being applied, in particular to prevent possible criticism by reviewers. To assess the impact of such 'data-filtering' practices on the CC 1/2 statistic and on model quality, we reprocessed the CDO3 data set to reject either all negative unique reflections (data set CDO3b) or all negative observations (data set CDO3c). The high-resolution statistics for the three data sets (Fig. 2) shows a few interesting features. Firstly, as expected, both CDO3b and CDO3c have much lower R meas and higher hI/(I)i in the high-resolution bins. Secondly, more reflections are rejected in CDO3b, which makes sense because any reflections having at least a single positive observation will be included in CDO3c, while even reflections with positive observations will be deleted from CDO3b whenever the positive observations are more than offset by negative observations of the same reflection. Thirdly, the highresolution CC 1/2 values of both CDO3b and CDO3c are lower than those of CDO3, with CDO3b showing the greater decrease.
To assess the relative quality of the models resulting from refinements against these three data sets applying the pairedrefinement strategy, the same starting model was refined against each data set and the resulting models were used, without further refinement, to obtain R work and R free against the other two data sets. These R values (Table 3) show that the model resulting from refinement against all reflections (data set CDO3) is unambiguously the best model, giving both the Example demonstrating the possibility of negative CC 1/2 when rejecting reflections with negative intensities from a data set. The plots show " 1 versus " 2 for simulated data having Gaussian noise and no signal ( = 0). (a) 1000 unique reflections, each represented by two observations; no rejections. The correlation of " 1 and " 2 is near zero. (b) From the 1000 unique reflections, those with negative intensity (" 1 + " 2 < 0) were rejected. The resulting correlation between " 1 and " 2 is about À0.47. (c) From the 1000 unique reflections, those with negative " 1 or negative " 2 were rejected, also resulting in positive (merged) intensity. The resulting correlation between " 1 and " 2 is near zero. lowest R free and the least overfitting (as judged from the reduced R free -R work gap) with all three data sets. Furthermore, the model refined against data set CDO3b is consistently better than that resulting from refinement against CDO3c, even though data set CDO3b has fewer unique reflections than data set CDO3c. We conclude that data 'massaging' or 'filtering' by rejecting negative unique reflections, or -even worse -negative observations, with the purpose of enhancing R meas or hI/(I)i values is counterproductive and leads to worse models. This conclusion is entirely consistent with the concept that the inclusion of weak data (even so weak as to be negative), when appropriately weighted, improves the resulting model and that they should not be discarded.
In contrast to the behaviour of R meas and hI/(I)i, the CC 1/2 values at high resolution of the CDO3b and CDO3c data sets actually decrease, paralleling the changes in model quality even though CC 1/2 might be expected to also increase given that the remaining data are stronger. As is illustrated in Fig. 3, the data-filtering practices are not producing a typical data set with a higher signal-to-noise ratio, but are introducing large systematic errors into the data by skewing the distribution of reflection intensities from what would be expected for a data set that has a certain level of signal and random errors. In the next section, we present an analysis of how systematic errors such as these can influence the CC 1/2 and CC* values.

Theory of the impact of systematic errors on CC 1/2 and CC*
Here, we use the terminology and definitions of the work that introduced CC 1/2 (Karplus & Diederichs, 2012). To calculate the intra-data-set correlation coefficient CC 1/2 , the measurements belonging to each unique reflection of the experimental data set are randomly assigned to two half data sets and the observations belonging to each half data set are averaged to give I 1 and I 2 , respectively. We observe that by choosing observations randomly, none of the two half data sets is preferred in any way; thus, their variances 2 "1 and 2 "2 are the same ( " 2 ). We may thus define J À hJi = for 'true' measurements with mean zero and variance 2 , " 1 as the errors in random half data set 1, with an expectation value of zero and variance " 2 , and " 2 as the errors in random half data set 2, with an expectation value of zero and variance " 2 . CC* as given by (1) is an estimate of CC true , the correlation between the arithmetic average of the half data set intensities I 1 and I 2 and the true intensities J. This estimate should be accurate since no approximations are involved in deriving (1). When deriving (1), we assumed that , " 1 and " 2 are mutually independent. However, when systematic errors in the measurement or data processing are present , " 1 and " 2 may no longer be independent of each other.
An example of systematic error that leaves , " 1 nd " 2 mutually independent is the case of an error in a dataprocessing program that results in a positive offset to all intensities. This is clearly a systematic error, but it does not affect , " 1 and " 2 (since any offset is subtracted when constructing , " 1 and " 2 , which have an expectation value of zero) and thus has no influence on CC 1/2 or CC*. This type of error would, however, lower R merge but increase R work /R free . In this 'Gedankenexperiment', data and model R values are thus anticorrelated, whereas a CC-based data-quality indicator is unchanged. A simple way to detect such a problem would be to monitor the scale factors between I obs and I calc [note that comparing I obs and I calc rather than F obs and F calc avoids artifacts introduced by the French-Wilson (French & Wilson, 1978) procedure for converting intensities to amplitudes].
Three conceptually simple examples for systematic errors that do invalidate one or more of the assumptions of independence are the following.
(i) Owing to an error in space-group assignment that can, for instance, occur in special cases of pseudosymmetric translational symmetry, only every second reflection is processed; the missed reflections are wrongly assigned an intensity of zero. If we consider all reflections, including the missed ones, " 1 and " 2 are (positively) correlated (i.e. nonindependent); in particular, they are negative for the missed reflections. is (negatively) correlated with " 1 and " 2 .
(ii) Owing to overflow or nonlinearity of the detector hardware, the intensity of strong reflections may be underestimated. Also in this case the true signal is (negatively) correlated with " 1 and " 2 , and " 1 and " 2 are (positively) correlated with each other (e.g. if one of them is negative, the other is often negative as well).
(iii) Inadequate scaling or radiation damage may yield intensities that are too low or too high for half of the observations of each unique reflection, respectively. If two observations are available for each unique reflection (which may, for example, happen when the data collection covers the asymmetric unit of reciprocal space two times in a row), then " 1 and " 2 are (negatively) correlated, but is not correlated with " 1 or " 2 .
In all cases of systematic error we can still assume that E(" 1 ) = E(" 2 ), since the random assignment of measurements to half-data sets on average prevents any particular of the two expectation values being larger than the other.
The difference CC* 2 -CC 2 true is zero if the above assumptions about the mutual independence of , " 1 and " 2 hold. It is interesting that even after dropping these assumptions we can calculate CC* 2 À CC 2 true . This offers a way to predict whether the estimate CC* overestimates or underestimates CC true .  Table 3 Application of the pairwise refinement technique to the data sets specified.
Within each row of the table, the same sets of reflections are used to calculate R work and R free . Each model (top row) was obtained by refinement against one data set. Its model R values (R work , R free ) against the other data sets are also given. For each data set, the model that gives the best R free is marked in bold.  204, 0.235 0.199, 0.239 Collecting and rearranging terms as in the Supplementary Material of Karplus & Diederichs (2012), we obtain The denominator of the right-hand side is positive. It is noteworthy that no matter whether the true signal is negatively or positively correlated with " 1 and " 2 , the second term of the numerator is always negative. Overall, the numerator may be positive or negative, or its two terms could cancel. Cases where " 1 and " 2 are positively correlated, as in the first two of the three examples above, seem to have practical relevance; for these, the first term E(" 1 " 2 ) is larger than zero, which favours CC* > CC true (i.e. overestimation of CC true ). However, the second term may cancel the first term, or if it is larger than the first term the overall result may be an underestimation of CC true . The third example yields CC* 2 À CC 2 true < 0, which means that CC* is an underestimate of CC true . The deviation from the truth occurs in opposite directions in the two half data sets, so that their average is close to the truth (high CC true ) but their agreement may be poor (low CC 1/2 ) such that CC* < CC true .
These examples demonstrate that even if the assumption that CC* is an accurate estimate of CC true may not be completely fulfilled, CC* is often a conservative estimate of CC true owing to the safeguarding effect of the second term of the numerator. This clearly is a desirable property of a dataquality indicator.
We also note that in the case of vanishing signal (!0) only the first term of the numerator remains. Thus, at high resolution the value of CC* 2 À CC 2 true is dominated by the expectation value of " 1 " 2 .

Application of the theory to the specific data-filtering test cases
Applying these theoretical considerations to the two datafiltering practices explored, we can understand that the decreases in CC 1/2 do not in this case occur owing to there being less signal in these data. Rather, the reduction of CC 1/2 is owing to systematic error being introduced: if negative unique reflections are rejected (data set CDO3b) then in highresolution shells where the signal vanishes, " 1 and " 2 become negatively correlated (Fig. 3b). This, according to (2), is augmented by the correlation between and " 1 , " 2 , leading to a substantial reduction of CC* below CC true . In other words, the systematic error causes CC 1/2 to be an underestimate of the level of signal in the data, meaning that in turn CC* is no longer a valid upper limit for CC work and CC free . Indeed, consideration of the refinement results demonstrates that at high resolution CC work and CC free are both higher than CC* for data set CDO3b (Table 4).
Considering data set CDO3c, the CC 1/2 (and thus the CC*) value is lower than that of CDO3, but to a lesser extent than CDO3b (Fig. 2). Since rejection of negative observations does not necessarily result in a correlation between " 1 and " 2 ( Fig. 3c), the reason for the decrease is not the first term of the numerator, as for CDO3b, but rather the second term. Here, the rejection of negative observations increases the intensity of the merged data over that of the true data, which leads to a correlation between and " 1 , " 2 . Again, according to (2), this decreases CC* below CC true . The fact that CC work and CC free are much reduced for CDO3c compared with models refined against CDO3 or CDO3b (Table 4) is owing to the fact that at high resolution the rejection of negative observations leads to systematically increased intensities for the affected reflections, but not for reflections without negative observations. This is a systematic error that the model cannot fit; i.e. it reduces CC work and CC free . This effect does not happen for CDO3b, which just has its weak reflections discarded; in this latter case, CC work and CC free are higher than for the model refined against CDO3 since the model refined against CDO3b fits the subset of stronger, less noisy intensities.
The rejection of negative intensities in this section serves as an example for a broader class of data-massaging practices, namely all those employing a positive cutoff. Employing such a cutoff can be expected to result in even worse models than those obtained with a cutoff of zero, as shown here. Negative cutoffs of about À3 and below, on the other hand, may be expected to reject true outliers and to affect very few reflections. In addition, it should be noted that the artificial increase of average intensities at high resolution brought about by rejection of weak reflections invalidates the French-Wilson procedure for estimating amplitudes from intensities and may result in a model with unrealistically low temperature factors.
Unfortunately, non-isomorphism between data sets cannot be treated using the theory laid out above, since the concept of 'truth' is undefined when two data sets from different crystals are merged: if the data sets are best described by different structures, then which is the 'true' one? Obviously, further research is needed to obtain meaningful prediction of the model quality resulting from the merging of slightly nonisomorphous data sets (Giordano et al., 2012).

Summary
Since the introduction of data R values, decisions based on these have been influencing protocols dealing with rejection of research papers Acta Cryst. (2013). D69, 1215-1222 Diederichs & Karplus Better models by discarding data? 1221 Table 4 Comparison of CC* with CC work , CC free in the highest resolution shell (1.61-1.57 Å ).
All of the data from CDO3 were used or only positive unique reflections or only positive observations were used. The number of unique reflections is given in parentheses. complete data sets, resolution shells with weak data and weak (e.g. negative) reflections. Procedures such as those analyzed here that discard, filter or massage data in order to minimize data R values need to be abandoned since they lead to suboptimal atomic models. They should be replaced by evaluations of data quality using better suited correlationcoefficient-based criteria, together with unambiguous identification of the best models, which can be performed using a paired-refinement strategy.