Citations in supplementary material

On 21 February 2008, a letter in the correspondence section of Nature drew attention to systematic undercounting of citations that are published only in supplementary material (Seeber, 2008). The principal concern expressed in the letter was that references in supplementary material are invisible to services or search engines such as PubMed, Scopus, Web of Science or Google Scholar. Clearly, this problem will have a negative effect on all numbers that are based on counting citations. Journals will end up with lower impact factors than they deserve and the H-indices of individual scientists will be lower than they should be. Since current science evaluation systems are largely based on citation counts, scientists and journals alike are disadvantaging themselves and others by placing references into the supplementary material of their published articles. Journals and scientists that focus on publishing methodologically oriented papers are particularly affected, because in most cases there is only space for the results of the study in the main part of the paper and the methodological part is hidden away in the supplementary material.

In order to gauge the severity of the situation, we have carried out a systematic study. The target journals (those publishing references in supplementary material) were chosen as Nature, Science, Cell and the Proceedings of the National Academy of Sciences (USA), and the affected journals (those impacted by references in supplementary material in the target journals) were journals published by the Inter­national Union of Crystallog­raphy (IUCr) – Acta Crystallographica Sections A to F, Journal of Applied Crystallography and Journal of Syn­chrotron Radiation. All 2009 issues of the four target journals were examined for publications that report a protein or nucleic acid structure determination. Typically, such papers are easily identified by the accompanying Protein Data Bank entry code. The reason that these papers were chosen by us is that most of the published structures were determined by X-ray crystallographic methods, which are typically described in one of the IUCr journals. The citations in the publications and in their respective supplementary material were counted and grouped into two categories: references to all journals and references to IUCr journals. For Nature, preliminary investigations showed that references in their so-called extended methods section, which is published online only, are covered by indexing services; these references were therefore counted as main-text references. The results of the study are summarized in Tables 1(a) and 1(b) and illustrated in Fig. 1. With a little over 300 articles and more than 15 000 references examined in total, the sample is large enough to be statistically significant.

Table 1
Results of the citation survey of the 2009 issues of Nature, Science, Cell and the Proceedings of the National Academy of Sciences (USA) (PNAS)

(a) Distribution of all references between main text and supplementary material.

	Total number of articles	Total number of references	Number of references in the main text	Number of references in supplementary material	Fraction of references in supplementary material (%)
Nature	74	3810	3242	568	15
Science	34	1823	1107	716	39
Cell	29	1579	1326	253	16
PNAS	167	8078	6563	1515	19
All	304	15290	12238	3052	20

(b) Distribution of IUCr references between main text and supplementary material.

	Total number of articles	Total number of IUCr references	Number of references to IUCr journals in the main text	Number of references to IUCr journals in supplementary material	Fraction of IUCr references in supplementary material (%)
Nature	74	347	279	68	20
Science	34	147	8	139	95
Cell	29	128	67	61	48
PNAS	167	655	338	317	48
All	304	1277	692	585	46

Figure 1 Pie chart showing the distribution of references between main text (blue) and supplementary material (red) for all journals (on the left) and for the IUCr journals (on the right).

From the numbers shown in Table 1(a) it is evident that on average about 20% of all references (main-text and supplementary material) are published in the supplementary material. In the case of Science the numbers are significantly higher. Here, about 40% of all references are published in the supplementary material. We would also like to note here that a small proportion of references are published in both the main text and the supplementary material. However, since these numbers are small, they do not alter the overall trends and can thus safely be ignored. For the citations of publications in IUCr journals a striking imbalance exists. References to IUCr publications amount to about 8% of all references, but 19% of all references in supplementary material. Further, almost half of all references to publications in IUCr journals end up being published in the supplementary material only. This is bad news for the crystallographic community. Not only does this mean that the impact factor of IUCr journals should be higher, but also that the real overall numbers of citations of methods papers are much higher than what is reported, for instance, by the Web of Science.

Although only a small number of target journals and affected journals were analysed, it must be anti­cipated that the problem is much more widespread. For instance, a similar negative impact to that for the affected IUCr journals should be expected for authors and journals publishing NMR structure determinations.

While the possibility of supporting scientific results by publishing supplementary material is clearly beneficial to authors, reviewers and readers alike, the problem of undercounting the references calls for a solution. In principle, there are several ways to solve this problem, and the scientific community and the publishers should agree on one. The first option would be that the publishers, as part of the publishing procedure, systematically provide the indexing services with all references, including those in the supplementary material. A second option would be that all references should appear in the main text of the article so that they would all be properly indexed. This would require some journals to abandon limits on the number of references. We can ask publishers to be fair to all authors, because in the end, they depend on them. Also, we can only make a plea to authors to be fair to other authors in citing their work because, in the end, methods need to be continuously developed and refined in order to ensure progress.