editorial\(\def\hfill{\hskip 5em}\def\hfil{\hskip 3em}\def\eqno#1{\hfil {#1}}\)

Journal logoJOURNAL OF
ISSN: 1600-5767

Citations in supplementary material

aSection Editor of Acta Crystallographica Section F, Helmholtz-Zentrum Berlin für Materialien und Energie, Macromolecular Crystallography (BESSY-MX), Albert-Einstein-Strasse 15, D-12489 Berlin, Germany, bSection Editor of Acta Crystallographica Section F, PO Box 6483, Lawrenceville, NJ 08648-0483, USA, cSection Editor of Acta Crystallographica Section D, Past-President of the International Union of Crystallography, School of Biological Sciences, University of Auckland, Private Bag 92-019, Auckland, New Zealand, dSection Editor of Acta Crystallographica Section D, Argonne National Laboratory, Biosciences Division, Building 202, Room Q142, Argonne, IL 60439, USA, eEditor of Journal of Applied Crystallography, Helmholtz-Zentrum Berlin für Materialien und Energie, Wissenschaftliche Geschäftsführung, Glienicker Strasse 100, D-14109 Berlin, Germany, fEditor-in-chief of IUCr journals, ETH Zürich, Wolfgang-Pauli-Strasse 16, CH-8093 Zürich, Switzerland, and gPresident of the IUCr, Department of Chemistry, University of Copenhagen, Universitetsparken 5, DK-2100 Copenhagen, Denmark
*Correspondence e-mail: msweiss@helmholtz-berlin.de

(Received 14 October 2010; accepted 15 October 2010; online 16 November 2010)

On 21 February 2008, a letter in the correspondence section of Nature drew attention to systematic undercounting of citations that are published only in supplementary material (Seeber, 2008[Seeber, F. (2008). Nature (London), 451, 887.]). The principal concern expressed in the letter was that references in supplementary material are invisible to services or search engines such as PubMed, Scopus, Web of Science or Google Scholar. Clearly, this problem will have a negative effect on all numbers that are based on counting citations. Journals will end up with lower impact factors than they deserve and the H-indices of individual scientists will be lower than they should be. Since current science evaluation systems are largely based on citation counts, scientists and journals alike are disadvantaging themselves and others by placing references into the supplementary material of their published articles. Journals and scientists that focus on publishing methodologically oriented papers are particularly affected, because in most cases there is only space for the results of the study in the main part of the paper and the methodological part is hidden away in the supplementary material.

In order to gauge the severity of the situation, we have carried out a systematic study. The target journals (those publishing references in supplementary material) were chosen as Nature, Science, Cell and the Proceedings of the National Academy of Sciences (USA), and the affected journals (those impacted by references in supplementary material in the target journals) were journals published by the Inter­national Union of Crystallog­raphy (IUCr) – Acta Crystallographica Sections A to F, Journal of Applied Crystallography and Journal of Syn­chrotron Radiation. All 2009 issues of the four target journals were examined for publications that report a protein or nucleic acid structure determination. Typically, such papers are easily identified by the accompanying Protein Data Bank entry code. The reason that these papers were chosen by us is that most of the published structures were determined by X-ray crystallographic methods, which are typically described in one of the IUCr journals. The citations in the publications and in their respective supplementary material were counted and grouped into two categories: references to all journals and references to IUCr journals. For Nature, preliminary investigations showed that references in their so-called extended methods section, which is published online only, are covered by indexing services; these references were therefore counted as main-text references. The results of the study are summarized in Tables 1[link](a) and 1[link](b) and illustrated in Fig. 1[link]. With a little over 300 articles and more than 15 000 references examined in total, the sample is large enough to be statistically significant.

Table 1
Results of the citation survey of the 2009 issues of Nature, Science, Cell and the Proceedings of the National Academy of Sciences (USA) (PNAS)

(a) Distribution of all references between main text and supplementary material.

  Total number of articles Total number of references Number of references in the main text Number of references in supplementary material Fraction of references in supplementary material (%)
Nature 74 3810 3242 568 15
Science 34 1823 1107 716 39
Cell 29 1579 1326 253 16
PNAS 167 8078 6563 1515 19
All 304 15290 12238 3052 20

(b) Distribution of IUCr references between main text and supplementary material.

  Total number of articles Total number of IUCr references Number of references to IUCr journals in the main text Number of references to IUCr journals in supplementary material Fraction of IUCr references in supplementary material (%)
Nature 74 347 279 68 20
Science 34 147 8 139 95
Cell 29 128 67 61 48
PNAS 167 655 338 317 48
All 304 1277 692 585 46
[Figure 1]
Figure 1
Pie chart showing the distribution of references between main text (blue) and supplementary material (red) for all journals (on the left) and for the IUCr journals (on the right).

From the numbers shown in Table 1[link](a) it is evident that on average about 20% of all references (main-text and supplementary material) are published in the supplementary material. In the case of Science the numbers are significantly higher. Here, about 40% of all references are published in the supplementary material. We would also like to note here that a small proportion of references are published in both the main text and the supplementary material. However, since these numbers are small, they do not alter the overall trends and can thus safely be ignored. For the citations of publications in IUCr journals a striking imbalance exists. References to IUCr publications amount to about 8% of all references, but 19% of all references in supplementary material. Further, almost half of all references to publications in IUCr journals end up being published in the supplementary material only. This is bad news for the crystallographic community. Not only does this mean that the impact factor of IUCr journals should be higher, but also that the real overall numbers of citations of methods papers are much higher than what is reported, for instance, by the Web of Science.

Although only a small number of target journals and affected journals were analysed, it must be anti­cipated that the problem is much more widespread. For instance, a similar negative impact to that for the affected IUCr journals should be expected for authors and journals publishing NMR structure determinations.

While the possibility of supporting scientific results by publishing supplementary material is clearly beneficial to authors, reviewers and readers alike, the problem of undercounting the references calls for a solution. In principle, there are several ways to solve this problem, and the scientific community and the publishers should agree on one. The first option would be that the publishers, as part of the publishing procedure, systematically provide the indexing services with all references, including those in the supplementary material. A second option would be that all references should appear in the main text of the article so that they would all be properly indexed. This would require some journals to abandon limits on the number of references. We can ask publishers to be fair to all authors, because in the end, they depend on them. Also, we can only make a plea to authors to be fair to other authors in citing their work because, in the end, methods need to be continuously developed and refined in order to ensure progress.


The authors would like to thank Martin Bommer (Helmholtz-Zentrum Berlin) for carrying out the journals and citations survey. Thanks are also due to Sarah Froggatt, Louise Jones and Peter Strickland from the IUCr Editorial Office for checking the citation information and suggesting improvements to the manuscript.


First citationSeeber, F. (2008). Nature (London), 451, 887.  Web of Science CrossRef PubMed Google Scholar

© International Union of Crystallography. Prior permission is not required to reproduce short quotations, tables and figures from this article, provided the original authors and source are cited. For more information, click here.

Journal logoJOURNAL OF
ISSN: 1600-5767
Follow J. Appl. Cryst.
Sign up for e-alerts
Follow J. Appl. Cryst. on Twitter
Follow us on facebook
Sign up for RSS feeds