The success story of crystallography†

Diffractionists usually place the birth of crystallography in 1912 with the first X-ray diffraction experiment of Friedrich, Knipping and Laue. This discovery propelled the mathematical branch of mineralogy to global importance and enabled crystal structure determination. Knowledge of the geometrical structure of matter at atomic resolution had revolutionary consequences for all branches of the natural sciences: physics, chemistry, biology, earth sciences and material science. It is scarcely possible for a single person in a single article to trace and appropriately value all of these developments. This article presents the limited, subjective view of its author and a limited selection of references. The bulk of the article covers the history of X-ray structure determination from the NaCl structure to aperiodic structures and macromolecular structures. The theoretical foundations were available by 1920. The subsequent success of crystallography was then due to the development of diffraction equipment, the theory of the solution of the phase problem, symmetry theory and computers. The many structures becoming known called for the development of crystal chemistry and of data banks. Diffuse scattering from disordered structures without and with partial long-range order allows determination of short-range order. Neutron and electron scattering and diffraction are also mentioned.

1. Introduction

The observation of X-ray diffraction by Friedrich, Knipping & Laue (Friedrich et al., 1912) is one of the most important discoveries in the history of science, with monumental consequencies. It opened the path for the development of modern solid-state physics and materials science, including mineralogy, chemistry and molecular biology, i.e. all the science describing the material world around us. It led to the discovery of some of the most fundamental concepts taught to first-year students in introductory chapters of textbooks – so fundamental that every educated person is familiar with them while the science `crystallography' is much less broadly known or has become rather irrelevant to many.

Notably, X-ray diffraction by crystals is a continuation of the invention and perfection of microscopes, which have been indispensable tools for the development of science from the Renaissance on. The ever-increasing resolution and perfection of these instruments have given ever-more detailed insights into the structure of matter, of dead and living objects, driven by the belief that in order to understand the properties of matter we must know its structure with ever-finer details. Up to 1912, the world of atoms was hopelessly out of reach for direct observation. Atoms figured, of course, in scientific theories such as the kinetic theory of gases, the lattice theory of crystals and in Avogadro's number, and the size of `air molecules' had been estimated by Loschmidt in 1865 as roughly 1 nm. Chemists talked in terms of molecules com­posed of atoms, but the physical nature of such atoms was mysterious. As late as in the 1950s, respectable chemists might still wonder whether atoms and molecules had more reality than being just `classification schemes', and talk of NaCl molecules (at ambient conditions) 40 years after W. L. Bragg's structure determination of rocksalt. X-ray diffraction (and diffraction of other radiations) ushered in a new age, because it opened the door to the realm of atoms which could now be seen and became real physical objects. Of course, up to quite recently one might object that crystallographers solved the phase problem by supposing that the structure is composed of atoms of known shape (given by the scattering factor), and that their `microscope' shows only what they have supposed to be there. However, this model is so fantastically successful, even for quite approximate scattering factors, that we may accept that X-ray crystallography provides a `microscope' with atomic resolution.

Evidently, Friedrich, Knipping & Laue's discovery (Fried­rich et al., 1912) verified the lattice theory of crystals, which had been formulated 100 years earlier but was considered doubtful by many scientists. The periodicity of crystal structures is of such fundamental importance for structure determination that the new `microscopy' with atomic resolution was called crystallography. However, the `microscope' covers a much broader field comprising many kinds of disordered materials with partial long-range or even only short-range order. The latest developments aim at diffraction from single molecules and at the direct observation of the dynamics of structures.

The discovery of X-ray diffraction opened the way not only to structure determination. The analysis with a crystal, using Bragg's law, of the frequencies present in primary radiation emitted by an X-ray tube, or in secondary radiation excited in a material by the primary radiation, marks the start of X-ray spectroscopy. Moseley (1913, 1914) established that the square root of the frequency ν of spectra of different elements is a linear function of the atomic number Z,

k₁ and k₂ being constants, and k₂ = 1 for K spectra. Moseley connected this law with Bohr's quantum theory of the hydrogen atom. It gave a physical foundation to Mendelejew's periodic system of the elements, and allowed one to predict and identify missing elements. From the K, L and M spectra, the atomic shell model and emission selection rules were experimentally established. When sealed X-ray tubes became available, X-ray spectroscopy became a very powerful analytical tool. Ever-increasing spectral resolution led to the development of EXAFS and XANES for the study of interatomic bonding.

Excellent detailed accounts of the earlier history of X-ray diffraction, absorption, spectroscopy and structure determination have been published by the witnesses of, and scientists involved in, the original work [see Ewald (1962), Bragg (1975), Bacon (1966) and Hildebrandt (1993)]. The present work, although based in part on these sources, is not intended to duplicate them, and the reader is encouraged to consult them.

2. Getting ready

The development of the theory of crystal structure determination following the work of Friedrich, Knipping & Laue (June 1912) was remarkably fast. In many countries, scientists immediately started to perform their own experiments, and to come up with interpretations of diffraction pictures and with theories. The geometrical representation of Laue's interference equations with the reciprocal lattice and the sphere of reflection was published by Ewald (1913). Bragg's equation was presented to the Cambridge Philosophical Society in November 1912 (Bragg, 1913a) and the first complete crystal structures of NaCl, KCl, KBr and KI were published in July 1913 (Bragg, 1913b). More such simple structures with atomic positions fixed by symmetry, including that of diamond, and the more complicated structures of calcite and pyrite with a free positional parameter followed in the same year. The lengths of the translation periods could be obtained from the observed density and Avogadro's number, quite accurate values of which were available since the determination of the charge of the electron by Millikan in 1910. Hence the wavelengths of the X-rays could be calibrated.

The kinematical model of X-ray diffraction assuming a single reflection event of the X-ray beam by a set of lattice planes was soon realized to be unsatisfactory, since it neglects weakening of the primary beam by diffraction and diffraction of the secondary beam back into the primary beam. Darwin (1914) published a two-beam dynamical theory of diffraction derived with back-and-forth Bragg reflections. He defined the ideal mosaic crystal and formulated the theory of extinction, which is still a basic ingredient of crystal structure determination up to the present day. At the same time, Moseley (1913, 1914) studied X-ray absorption and emission spectra, which at the time were essential for atomic theory.

Sommerfeld had feared that the thermal movement of the atoms, by destroying the periodicity of the structure, would mar any diffraction effect. This was of course shown by the experiment not to be true. The very essential problem of diffraction by a temperature-disturbed crystal was solved in 1913–1914 by Debye (1914), who derived the isotropic displacement factor B still used today (with a correction by Waller in 1923) and the Debye temperature characterizing the elastic properties of crystals. The temperature dependence of B led the way to an experimental verification of the quantum-mechanical zero-point vibration. W. H. Bragg understood the need to obtain integrated diffraction intensities, rather than peak intensities. H. A. Lorentz derived the geometrical factor known as the Lorentz factor (see Ewald, 1962, p. 78). Hence­forth, X-ray structure factors could be determined from integrated intensities as is still done today. The first simple structures could be determined without knowledge of atomic scattering curves. Subsequently, experimental estimates of scattering factors were obtained from the measured integrated intensities of these structures.

Some basic diffraction apparatus was also developed early on. The Braggs built the first single-crystal X-ray spectrometer in 1913, which allowed a crystal to be oriented in the primary beam into a reflection condition; the diffracted beam was measured in an ionization chamber. Debye & Scherrer (1916), and independently Hull (1917), invented the powder method that became immensely useful for metallurgical and many other problems.

Finally, the prime importance of space-group symmetry for structure determination was realized. Niggli (1919), in his book Geometrische Kristallographie des Diskontinuums, published the 230 space groups, including all equivalent positions and many diagrams much as we know them from present-day International Tables. He also discussed methods for space-group determination by systematic absences. Indeed, the first Internationale Tabellen zur Bestimmung von Kristallstrukturen (1935) edited by C. Hermann incorporated much of Niggli's work. Somewhat later, R. G. Wyckoff in 1922 and W. T. Astbury in 1924 published their space-group tables.

Therefore, by 1920 all the basic knowledge about X-ray diffraction still used today for crystal structure determination was available. The stage was set. But more efficient diffraction equipment needed to be developed. Ever-more complicated structures required methods for the solution of the phase problem. And the calculations were extremely tiresome, if not impossible, with the means then available.

3. The success of crystal structure determination

3.1. Equipment

In the early 1920s, W. D. Coolidge constructed the first sealed X-ray tube, which facilited enormously the production of X-rays with different types of anode materials. X-ray photographic film cameras, in particular the rotating crystal and Weissenberg cameras, were invented in the 1920s and became standard equipment in crystallography laboratories for many decades. Somewhat later in 1932, the Sauter and Schiebold cameras were invented, but never became important. Undistorted photographs of reciprocal-lattice planes were obtained first with the retigraph of W. F. de Jong & J. Bouman in 1938, and later with the hugely popular precession camera invented by M. J. Buerger in 1944. Diffracted intensities for structure determination were recorded and visually measured with multiple-film methods on Weissenberg cameras. These were favored because they use cylindrical films and record intensities up to the resolution given by the wavelength; in addition, small inaccuracies in the alignment of the crystal are harmless because each spot on the film records a single reflection, and not a superposition of two reflections from the same lattice plane hkl, as is the case for the precession method. Some laboratories had rows of Weissenberg cameras turning day and night. Weissenberg cameras were also used with high- and low-temperature equipment. The mechanics are so simple that the camera performed well inside enclosures cooled to moderately low temperatures. The first simple general-purpose diffractometers were rotating crystal cameras equipped with a scintillation counter.

Automatic diffractometers with diverse geometries were designed from the late 1950s on, with setting angles being supplied on paper tape or punched cards. Such instruments were pioneered at neutron sources: neutron intensities are measured with proportional counters rather than with films, and the cost of the production of neutrons calls for efficient measurement strategies. The nascent macromolecular crystallography with its need for rapid recording of a great number of reflection intensities was another major driving force. The first diffractometer capable of measuring automatically, without human intervention or precalculated setting angles, a complete reciprocal-lattice plane was the linear diffractometer developed by Arndt & Phillips (1961), the setting angles for an oriented crystal being generated by an analogue mechanism. It was capable of measuring 800 reflections of myoglobin in 16 h. However, the future belonged to Eulerian geometries. An instrument with a quarter Eulerian cradle (0 ≤ χ ≤ 90°) was marketed in Germany, but by the 1970s four-circle diffractometers had full Eulerian cradles (or the κ-geometry roughly equivalent to a half-circle cradle). They were driven by mini-computers and could function fully automatically in all of three-dimensional space. They became standard equipment for structure determination. Film cameras fell into disuse. The enormous gain in efficiency of intensity measurement was unfortunately accompanied by a potential loss of information because point counters of diffractometers measure only at the points they are positioned to. Weak superstructure reflections, satellites and diffuse scattering, which could readily be detected on film, might go unobserved, and resulting structural models might show disorder where there is none. From 1970 to 1990, very few people observed diffuse scattering. This changed only with the advent of two-dimensional detectors, first image plates and then charge-coupled device (CCD) detectors, which not only added another steep increase in efficiency, but also reminded crystallographers that reciprocal space contains more than just Bragg peaks.

The first dedicated synchrotron X-ray sources were com­missioned in the early 1980s. Insertion devices at synchrotrons became available in the early 1990s. Synchrotron light sources delivering very intense, highly monochromatic, coherent and wavelength-tunable X-rays have become indispensable for many applications, and in particular for macromolecular crystallography and structure determination by powder diffraction. The next boost in intensity and coherence of the radiation is imminent with the construction of free-electron lasers.

3.2. Solution of the phase problem

Early crystal structure determination proceeded by trial and error, i.e. educated guesses based on a deep knowledge of diffraction, symmetry, geometry and known structures. Lovely examples can be found in the textbook X-ray Analysis of Crystals by Bijvoet, Kolkmeyer and MacGillavry (Bijvoet et al., 1951). By 1930, many inorganic and alloy structures had been determined. In the 1930s, Fourier methods came into general use, but it was realized much earlier that structure factors are coefficients of Fourier series. The Patterson function was proposed in 1934 (see Patterson, 1935), while the auto-correlation function was already a familiar concept for mathematicians. The theory of Patterson methods has been described exhaustively by M. J. Buerger (1959). Many organic molecular structures were determined, predominantly with the heavy-atom method. However, the summation of Fourier series was very labor intensive without computers, and much effort went into facilitating this chore. Beevers & Lipson (1934) designed the Beevers–Lipson strips imprinted with the values of the required cosine and sine terms. They estimated for their example CuSO₄·5H₂O, space group , with 89 hk0 reflections that `the longest double Fourier synthesis can be accomplished by two workers in about two days.' An optical device for producing two-dimensional Fourier summations was proposed by Bragg (1929): a slide with light and dark fringes representing a Fourier wave is projected on photographic film; the image is obtained by superimposing exposures with different orientations and periods that can be simulated by rotating and translating the slide. Several mechanical and electro-mechanical machines were devised that reduced the time needed for the summation of a one-dimensional Fourier series, e.g. to under an hour for 16 terms. R. Pepinsky's analogue computer XRAC was the non-plus-ultra machine [Pepinsky (1947): `the summation of a two-dimensional series is accomplished within seconds after the data is fed into the machine'], but fast became obsolete with the widespread availability of digital computers. Henceforth, crystallographers would trace isodensity lines on grid points of a computer printout to obtain somewhat distorted Fourier and Patterson maps. This called for the development of peak-search programs.

It was realized quite early that it ought to be possible to determine phases directly from the observed absolute values of structure factors, since the standard spherical-atom model comprises far fewer parameters (atomic coordinates) to be determined than there are observations. An algebraic method by Ott (1927) to directly solve the structure-factor equations for the atom coordinates could not be developed into a practically useful method. More successful were the Harker–Kasper (Harker & Kasper, 1948) and Karle–Hauptman (Karle & Hauptman, 1950) inequalities. The foundations of modern direct methods were laid with Sayre's equation (Sayre, 1952a), and by Hauptman & Karle (1953). We see once again that the theoretical basis was known before the calculations could be carried out efficiently. Sign relationships were initially exploited by hand. In the 1960s, with the proliferation of computers, the theories were developed into useful programs: Symbolic Addition (Karle & Karle, 1966) and multisolution methods such as MULTAN (Germain et al., 1970). They still underlie the highly successful, nearly automatic direct-methods programs we know today.

Charge flipping (Oszlányi & Sütő, 2008) is the latest method of structure determination, first proposed in 2004. It is conceptually new in that it is not based on the assumption that the structure consists of atoms of known electron density. It only supposes that the scattering density (electron density) is concentrated in a few regions, most of the unit cell being essentially empty. No symmetry information is required, the structure is solved in the triclinc space group P1; constraining the symmetry appears to hinder structure solution; knowledge of the chemical composition is not required. The diffraction data do not need to be normalized and the scale factor need not be known. Charge flipping belongs to a class of methods that alternate between direct space and reciprocal space, applying iteratively constraints in both spaces. Another method in this class is Elser's difference map (Elser, 2003). The number of iterations needed to solve a structure is unpredictable, the process is intrinsically chaotic. Symmetry is determined and imposed by averaging only after convergence of the iterations. The method works well for difficult problems, such as superstructures. It also works for aperiodic structures, i.e. modulated structures and quasicrystals in their higher-dimensional representations where the scattering density is not given by atoms of known shape, but is still concentrated in small portions of the unit cell. Charge flipping is also one of the methods used for solving structures from powder data.

3.3. Symmetry

The solution of many problems in physics and chemistry is greatly facilitated by exploiting the symmetry of the system under investigation. It is noteworthy that symmetry theory in crystallography was developed very early and has a special flavor quite distinct from symmetry developed for other branches of science. Outside the crystallographic community, space groups appear to be exceedingly complicated entities, and Hermann–Mauguin symbols do not seem to be very popular. Space-group symmetry played a decisive role in structure determination. The first collaborative work, Internationale Tabellen zur Bestimmung von Kristallstrukturen (1935), edited by C. Hermann, was followed by International Tables for X-ray Crystallography (1952), starting with Volume I on space groups edited by K. Lonsdale. For four decades, this work accompanied every crystal structure determination. It was succeeded by International Tables for Crystallography (1983), starting with Volume A (IT-A), which provides properties of space groups such as maximal subgroups and minimal supergroups useful beyond conventional structure determination. The usual approach to crystallographic symmetry in tables and textbooks is geometrical and limited to three-dimensional space. A new algebraic approach valid for any dimension of space and well suited for computer applications was developed by Zassenhaus (1947). Modern space-group theory reformulated by mathematicians and crystallographers as presented in IT-A has profoundly modified the use of symmetry by crystallographers. Much of what older crystallographers looked up in the International Tables is now available in software for (semi-)automatic structure determination. More advanced group theory, such as lattices of maximal subgroups or space-group representations, has become available on the Internet (e.g. https://www.cryst.ehu.es/ ).

3.4. Computers

Crystal structure determination was a tedious undertaking until the early 1960s, since many calculations were hardly feasible by hand or with simple mechanical machines. It owes its success to the ever-increasing availability of digital computers. Before the advent of high-level languages such as ALGOL and Fortran, computer programming was awkward, but absolute-address machine-language Fourier summation programs were written in the 1950s that had to run through a whole night on an error-prone machine, with sometimes dubious results. The computers of the 1960s were large mainframes operated behind closed doors by specialized personnel, suffering from turn-around times of sometimes several days. The first very useful computer I worked with which was not operated by a computer center and to which researchers had direct access was the IBM-1620 (first marketed in 1959). This was a decimal machine with a memory of 20 K decimal digits. Numbers of variable lengths (at least two digits and at most all of the memory) were delimited by flags. It was programmed in a symbolic machine language, or in Fortran. For the mainframe machines, besides structure determination programs (Fourier–Patterson and direct methods), least-squares refinement programs were written by Friedlander et al. (1955), by Sparks et al. (1956) at UCLA, by Lavine & Rollett (1956) at Oxford, and by Busing et al. (1962) at Oak Ridge (ORFLS). These codes were very influential and inspired many later programs. Thus, buried in more modern code one might find the number-crunching loops of ORFLS. Also in the 1960s, the famous thermal-ellipsoid program ORTEP was developed by Johnson (1965), and ellipsoids have henceforth become the hallmark of crystallographers.

The numerous stand-alone programs for structure determination, refinement and geometrical properties were then assembled into integrated systems of crystallographic programs. XRAY-67 and its ever-more complete successors XRAY-69, XRAY-72 and XTAL (Hall et al., 1980) were a collaborative effort with many crystallographers contributing and debugging programs obeying certain restrictive rules in the use of Fortran. In contrast to this, Sheldrick's system of programs (see Sheldrick, 2008) was first written in separate public and commercial versions and maintained exclusively by the author with a consistent philosophy. It has become today's most popular structure-determination package. Another excellent system, with a long tradition, available today is CRYSTALS from Oxford (see Betteridge et al., 2003). Several other very impressive and successful structure determination and multipurpose refinement programs are not cited here and I ask their authors to pardon this omission.

Mini-computers, such as the PDP-8 that became available first in 1965, were programmed in machine language and were used by diffractometer manufacturers to drive their apparatus from the late 1960s on. The software was based on a very influential paper by Busing & Levy (1967). The mini-computers became rapidly larger and more efficient. By 1974, the source code of the diffractometer program by SYNTEX at the University of Lausanne was written in Fortran, probably the first such commercial program written in a high-level language, to which the producer gave free access. Steering programs became ever-more potent, with ever-more proprietorial restrictions, and access to the source code was no longer given. Diffractometers became efficient black boxes. Initially, the mainframe computer centers wished to centralize all computer resources and tried to limit the use of these mini-computers exclusively to driving the instruments. However, very soon commerical structure determination and refinement programs were written for the mini-computers, or somewhat larger versions of them, and gave crystallographers access to attractive in-laboratory computing. Already in the 1970s, R. A. Spark's programs for SYNTEX and G. M. Sheldrick's programs became available on Data General NOVA and ECLIPSE computers. Today, with big mainframe computers running for much more demanding problems, crystallographic calculations have become comparatively small and run on in-laboratory machines, but they, too, are usually used as black boxes.

3.5. Macromolecular crystallography

The development of macromolecular crystallography is a success story quite analogous to, but 20 to 30 years later than, the development of `small-structure' crystallography described above. Right after the discovery of Friedrich et al. (1912), scientists X-rayed all kinds of materials; not only crystals, but also fibrous, lamellar and granular substances, glasses, polycrystalline metals, rolled zinc, beeswax – almost anything, even if it seemed hopeless at the time to interpret some of the diffraction pictures. In 1934, J. D. Bernal got the first successful diffraction pictures of a hydrated protein. Some 25 years later, the first protein structures, myoglobin and haemoglobin, were successfully determined by M. F. Perutz and J. C. Kendrew. The story of the discovery of the double helix of DNA in 1953 need not be recounted here. I remember D. Harker presenting the first macromolecular structure solved in the USA at the ACA 1967 winter meeting in Atlanta, Georgia (Kartha et al., 1967). Some ten years later, American friends advised me to switch to macromolecular crystallography, since `small-structure' crystallography would soon be practiced only by service crystallographers. I did not heed the advice and am therefore ill-qualified to write a history of what has become one of the most important methods of structural biology. The two crystallographic communities, small and macromolecular structure, tended to be some­what separate. In addition to the fundamental importance of biochemistry, crystallization and cryo-crystallographic methods, the techniques of macromolecular structure determination developed according to their own needs: single and multiple isomorphic replacement, rotation and translation functions, resonance scattering, to name just a few topics. Dedicated synchrotron beamlines and fast, large CCD detectors made possible the exponential increase in the number of known structures that required the establishment of data banks, exactly as has been the case for `small' structures. Modern computing power and software development have led towards a convergence of `small' and `large' structure determination with algorithms useful for both. It appears that any structure can be solved with ab initio methods applied to a single diffraction data set that extends to atomic resolution of better than 1 Å. Structures with much lower resolution are solved using the tools developed for macromolecular structure determination.

3.6. Crystal chemistry and data banks

By 1930, the basic rules of inorganic crystal chemistry had been formulated by Madelung, Kossel, Born and Haber. The first sets of atomic and ionic radii were compiled, allowing the representation and prediction of interatomic distances. A definitive and widely used set of radii was established by Shannon & Prewitt (1969). L. Pauling's extremely influential book The Nature of the Chemical Bond, first published in 1939, could not have been written without the results of crystallographic structure determinations. Symbols for structure types (such as A1, A2,…, for elements; B1, B2,…, for binary compounds) were designed that are still sometimes used by physicists and materials scientists. However, the definition of a structure type, providing criteria for a classification of crystal structures, is in general still quite hazy, except for the simplest structures. Requiring the same space group and the same occupied Wyckoff positions is too restrictive and does not allow one to find more general geometrical relationships. Hellner (1965) applied the theory of lattice complexes to find geometrical relations between structures with different space groups. Symmetry relations between space groups, subgroups and supergroups, are exploited for describing related structures as derivatives of a highest symmetric variant, the aristo­type (International Tables for Crystallography, Volume A1, see https://it.iucr.org/ ). Modern research directions are the study of topology, tilings, nets and minimal surfaces; many publications on these topics can be found in recent issues of Acta Crystallographica Section A.

By the 1930s, the number of known structures had already become so large that a repository had to be created. Struktur­bericht, published by Zeitschrift für Kristallographie from 1929 to 1939, and its successor Structure Reports, published by the International Union of Crystallography (IUCr) from 1947, listed, classified and discussed the structures determined starting from 1913. This effort was discontinued in 1985 for organic structures and in 1990 for inorganic and metal structures, as the rapidly increasing number of known structures could no longer be stored in books. Henceforth, crystal structures are collected in databases which are searched and mined for numerous purposes (Allen, 1998): the powder diffraction file of the International Centre for Diffraction Data (ICDD), the Cambridge Structural Database (CSD), the Protein Data Bank (PDB), the Nucleic Acid Database (NDB), the Inorganic Crystal Structure Database (ICSD), and the Metals Data File (CRYSTMET until 1995) and its successor, the Material Phases Data System (MPDS, see Villars et al., 2004).

The ever-increasing number of structures published in many scientific journals not only calls for efficient presentation and storage. Many errors may enter publications, from occasional wrong numbers to entirely incorrect structures. It is therefore important that structural data be checked for consistency of lattice parameters, coordinates, symmetry, distances and angles. For this purpose, the IUCr has developed the Crystallographic Information Framework (CIF). Further information on CIF can be found at https://www.iucr.org/resources/cif .

3.7. Thermal motion

As mentioned above, the problem of the influence of thermal motion on the diffraction intensities had been solved very early by Debye (1914). Once computing power allowed structures to be refined by least-squares methods, isotropic and anisotropic Debye–Waller factors, or more generally displacement parameters, were produced routinely. Fifty years ago, these were sometimes regarded as parameters of minor importance, as a garbage dump for model imperfections. This opinion has been shown to be excessively pessimistic. In 1956, D. W. J. Cruickshank expressed anisotropic displacement parameters in terms of translational and librational motion of a rigid molecule, and derived a corresponding bond-length correction. In 1964, W. R. Busing and H. A. Levy published a method for bond-length correction due to the riding motion of an atom upon another atom. The definitive theory of the rigid molecule was published in 1968 (Schomaker & Trueblood, 1968). Since then, much work has been published on theories of segmented rigid-body motion, and of the effect of internal molecular motion added to the overall motion of the complete molecule (see e.g. Bürgi & Capelli, 2000). Such theories may also be capable of sorting out structural disorder from thermal motion. Thus, displacement parameters of accurately determined structures have become very valuable information and should be taken seriously.

3.8. Aperiodic structures

Incommensurately modulated crystals, composite crystals and quasicrystals are aperiodic in three dimensions, but they are perfectly ordered structures that diffract X-rays into narrow Bragg peaks. An example of a one-dimensional modulated crystal is the classical optical diffraction grating used for spectroscopic analysis, whose rulings may exhibit periodic variations of the distances between the lines due to imperfections of the ruling engine. It was known before the advent of X-ray diffraction that such imperfections produce additional interference maxima that were called lattice ghosts (`Gittergeister'). Such additional satellite reflections are also produced in incommensurate, displacively modulated and composite crystal structures, and scientists were aware of this before World War II. James (1948, p. 205) thanks R. Peierls for explaining thermal diffuse scattering (TDS) as an assemblage of a very great number of lattice ghosts produced by the elastic waves traversing the thermally agitated crystal. By the 1960s, diffraction from periodic distortions of crystal structures was well understood. Korekawa (1967) classified in detail the satellite reflections due to longitudinal and transverse waves.

Quasicrystals have revolutionized the most fundamental assumption concerning the classical symmetry of crystals and consequently the notion of `crystal': the atomic arrangement in a crystal is periodic. Since a periodic two- or three-dimensional lattice in Euclidean space admits only one-, two-, three-, four- and sixfold rotation axes, these are the only rotation symmetries a crystal may possess. The observation by D. Shechtman in 1982 (Shechtman et al., 1984) of diffraction pictures showing sharp Bragg reflections together with icosahedral symmetry (thus containing fivefold axes) has shown that crystals can possess macroscopic fivefold symmetry. Later, quasicrystals with eight-, ten- and 12-fold rotation symmetry have been found.

Aperiodic structures can be represented by periodic structures in higher-dimensional Euclidian spaces (superspace). The corresponding symmetry theory was developed in the 1970s for modulated and composite crystals (de Wolff, 1974, 1977; Janner & Janssen, 1977), and first applied to quasi­periodic structures by N. G. de Bruijn when defining matching rules for the Penrose tiling (de Bruijn, 1981). Determination of such structures has gradually become much easier. Charge flipping (Palatinus, 2004) is the newest, very successful method for the determination of incommensurate and quasicrystal structures in superspace.

3.9. Powder diffraction

Up to the 1970s, powder diffraction was not a method of structure determination, except for very simple structures. It was immensely useful for the identification of substances and for accurate measurement of lattice constants. Focusing film cameras such as the Guinier (1937) camera achieved very high precision. The powder method also served to study particle-size effects and preferred orientation of, and strains in, the crystallites. Its simple geometry made it suitable for investigations of phase transitions at high and low temperatures, and at high pressures. An excellent account of the use of the powder method 50 years ago is found in Buerger & Azaroff (1958). The International Centre for Diffraction Data (https://www.icdd.com ) maintains a database of powder diffraction data. Full structure determination became feasible with the development of high-resolution powder diffractometers. Ever-more complicated structures were successfully solved, in particular when synchrotron radiation became available. X-ray powder diffraction, together with electron diffraction, is today an important source of data for structure determination when single crystals are difficult or impossible to prepare (Xie et al., 2008). Recently, it has also become useful for macromolecular structures (Basso et al., 2010).

4. Beyond the free-atom model

Crystal structure determination is based on the free-atom model: the structure is supposed to be composed of rigid atoms with a spherically averaged theoretical electron density whose Fourier transform is the atomic scattering factor. These rigid atoms undergo harmonic thermal motions expressed by the anisotropic Debye–Waller factor (displacement parameters). This hugely successful model does not take into account chemical bonding between atoms or anharmonic motions. In contrast, X-ray structure factors are the Fourier transform of the thermally averaged true electron density. However, they are measured only to a limited accuracy and a limited resolution. Systematic errors in the observations and series-termination ripples mar the bonding features that in principle should be observable by summing a Fourier series with the observed structure factors and properly estimated phases. For centrosymmetric structures, the free-atom model is expected to give the correct phases of all but the very weakest structure factors, which is not exactly the case for non­centrosymmetric structures. It is to be expected that bonding effects consist of only minor deviations from the spherical free-atom densities, since they are caused by the diffusely distributed outermost valence electrons. This explains the success of the free-atom model for structure determination. The heavier the atom, the less significant is the effect on the observations of the modification of its electron density due to bonding. In an impressive, ground-breaking work, interatomic bonding was first successfully demonstrated by Brill et al. (1939) for the simplest high-symmetry inorganic and organic structures, NaCl, diamond and hexamethylenetetramine, with very carefully measured diffraction data. These data were extrapolated with the best scattering factors and displacement parameters available to very high resolution and density maps were computed by summing the Fourier series. The results showed for the first time the difference between an ionic bond and a covalent bond. Later, H. Witte, E. Wölfel and S. Göttlicher measured very accurate intensities for some simple structures using a gold-leaf electrometer as detector. An experimental extinction correction was obtained by extrapolating intensities from progressively thinner crystal plates to zero thickness. Witte & Wölfel (1955) obtained density maps of NaCl with somewhat cube-shaped atoms. Much effort was spent on trying to interpret these pictures in terms of Na⁺ and Cl⁻ ions. However, the evaluation of ionic charges depends on the way the continuous electron density is partitioned into atomic constituents. Charge became an elusive concept. The choice of simple inorganic compounds (with the exception of diamond with coordination number of only 4) for early electron-density studies was probably not the easiest way to success, since their bonding features are unspectacular and difficult to observe. This is due to the fact that spherical ions differ from neutral atoms only by the radial electron distribution function. Electron distributions of ions or atoms overlap in the structure and cannot be observed individually.

It is easier to study electron densities in lower-symmetry organic structures composed of C, H, N and O, where the maps are also more spectacular. The atomic positions in these structures are not fixed by symmetry. Therefore, atomic positions and displacement parameters found with the free-atom model are expected to deviate somewhat from those of the nuclei. This is the so-called asphericity shift. The calculation of difference-electron-density maps showing the deviations from the spherical-atom model due to bonding, or equivalently the extrapolation of structure factors to high resolution, require the nuclear parameters, which in principle can be measured with neutron diffraction. The result is the X − N difference map. A spectacular early result shows the bonding in symmetric triazine, C₃H₃N₃ (Coppens, 1967). Alternatively, X − X maps were obtained with structural parameters from free-atom refinements against high-angle X-ray data.

The idea of defining aspherical pseudo-atoms whose superposition gives the true electron density and whose parameters are refined against X-ray data arose quite early. Aspherical pseudo-atoms were represented by multipole expansions of their electron densities (or equivalent expansions using different basis functions) whose electron populations and radial functions are refineable parameters. Such models were implemented by Hirshfeld (1971), Stewart (1976) and Hansen & Coppens (1978). From the results, many kinds of geometric, electrostatic and electronic properties were derived. Compounds studied by many research groups included organic molecules, metal-organic compounds, minerals and compounds of rather heavy atoms. Much effort went into measurement strategies, more recently also with CCD detectors and synchrotron radiation. From the beginning of charge-density studies, a fruitful interaction between theoreticians performing quantum-chemical calculations and experimentalists produced progress in both domains, each refining its methods by attempting to reproduce the results of the other.

It turned out that bonding electron densities are reasonably transferable between different organic compounds. This called for the establishment of data banks containing the parameters of the most important pseudo-atoms, defined with respect to well chosen local coordinate systems. The data banks available at present are ELMAM2 (Domagała & Jelsch, 2008) with pseudo-atoms derived from high-resolution diffraction data, UBDB (Koritsanszky et al., 2002) with pseudo-atoms obtained from single-point calculations, and Invariom (Dittrich et al., 2006) with pseudo-atoms derived from geometry-optimized calculations. These data banks can be used to obtain more precise atomic coordinates and displacement parameters by taking care of the asphericity shifts. In subsequent refinements of large structures against relatively low resolution data, these coordinates and displacement parameters may then be kept invariant.

Anharmonic motions are today usually parameterized by a Gram–Charlier series expansion about the harmonic approximation of the movement of an atom. This series is favored for mathematical reasons and is not based on a physical model. The theoretical and experimental aspects of generalized atomic displacements have been presented by Kuhs (1992). Simultaneous refinement of bonding density and anharmonicity has been repeatedly attempted with various successes and criticisms.

5. Beyond periodic structures

5.1. Absence of long-range order

As was the case for crystalline substances, the theoretical foundations for diffraction from amorphous substances were worked out in the first years after the discovery of Friedrich et al. (1912). Debye (1915) first published the formula giving the intensity of the diffuse scattering by an isotropic gas, liquid or glass without long-range order, as a sum of sin x/x functions,

with f_m the scattering factor of atom m, d_mn the distance between atoms m and n, 2θ the angle between incident and diffracted beams and λ the wavelength. The scattering from a diluted monoatomic gas of N atoms is close to Nf², and the scattering factor can thus be observed experimentally. Well defined interatomic distances of molecules in a gas, e.g. C–C, Cl–Cl and C–Cl distances in CCl₄, create intensity waves superimposed on the scattering-factor curve . Interestingly, this formula predates the invention of the Patterson function by 20 years! It is the basis of molecular structure determination by gas electron diffraction. From Debye's formula, Zernicke & Prins (1927) developed the theory of radial distribution functions, which are obtained by Fourier inversion of the observed, scaled and normalized scattered diffuse intensity and show the probability of finding inter­atomic distances. Radial distribution functions were first obtained for monoatomic liquids such as Hg and Na, and showed preferred distances similar to the crystalline structures out to about 10 Å. An example of a polyatomic substance is SiO₂ glass, where radial distribution functions suggest the presence of linked SiO₄ tetrahedra. This is the theoretical basis of modern pair distribution functions (p.d.f.'s) in the study of nanomaterials (Farrow & Billinge, 2009).

One of the most fascinating and active current research topics is the attempt to realize single-particle and single-molecule structure determination. This requires radiation sources delivering extremely high intensity monochromatic X-rays with ultra-high spatial resolution, and it is expected that free-electron lasers will be a great step forward towards this goal. There are still many technical problems to be solved. However, the theoretical foundations were laid more than 50 years ago. The first reference is usually made to a short communication by Sayre (1952b) demonstrating that for centrosymmetric structures of limited size the phase problem is nonexistent if the diffraction pattern is oversampled, i.e. recorded in sufficiently fine intervals. However, in the same issue of Acta Crystallographica, Hosemann & Bagchi (1952) published independently, in German, a complete theory of the unique solution of a centrosymmetric structure with finite dimensions from its X-ray scattering intensities. This was followed immediately by two additional exhaustive publications, also in German (Hosemann & Bagchi, 1953a,b). Again, the theory was available long before any practical application could be envisaged.

5.2. Disordered structures with partial long-range order

Diffraction showing more or less sharp reflections accompanied by diffuse scattering had been observed quite early on. It is due to deviations of the packing of atoms from exact lattice symmetry, while there exists a periodic average structure. Such deviations are called `disorder' and may be due to `dynamic' thermal disorder and/or to other structural imperfections that are referred to as `static'. For X-ray Bragg intensities, this distinction is of little importance, since Bragg intensities represent a time-averaged structure or a space-averaged structure for dynamic and static disorder, respectively. Ground-breaking work on this topic was published in the 1930s and 1940s (Lonsdale, 1942). Many references to early work are also found in Wooster (1962). When working with film methods, crystallographers often observed diffuse scattering. When working with point-detector diffractometers, they were less aware of diffuse scattering, until the modern two-dimensional detectors showed again its ubiquity. It is well known that thermal disorder leads to a decrease in Bragg intensities due to the Debye–Waller factor, and concomitant thermal diffuse scattering (TDS). Early studies of TDS of X-rays in simple structures were carried out by Laval (1939). TDS is mainly attributed to long-wavelength elastic acoustic waves that can be calculated with the elasticity tensor of the material. Conversely, elastic constants of simple metals such as Pb were determined from TDS intensities. The study of phonon dispersion curves with inelastic scattering of slow neutrons started in the 1950s.

Early work on static disorder was carried out on diamond and on disordered alloy structures such as Cu–Au alloys. Fourier transformation of diffuse-scattering intensities gives probabilities of coordination of an atom by atoms of the same and of a different kind. From the Bragg intensities (i.e. the relatively sharp reflections with integral Miller indices), the Bragg–Williams order parameter is computed, giving a measure of long-range order. Antiphase domains are also indicated by diffuse scattering. Today, modern structure determination from diffuse intensities in all kinds of substances, organic and inorganic, relies on high-intensity X-rays, two-dimensional detectors, and computer-intensive Monte Carlo and genetic algorithms. It has again become an active field of research.

The simplest kind of diffuse scattering results from one-dimensional disorder of layer structures, where the layers are perfectly two-dimensional periodic. These layers may be stacked with several alternative displacements (such as the layers in closest sphere packings) that give the same closest contacts to atoms in adjacent layers, but differ in the distances to atoms in more distant layers. This results in rods of diffuse intensity perpendicular to the layers. The basic theory was first given by Hendricks & Teller (1942) and Wilson (1942). A definitve formulation in terms of stacking probabilities dependent on the range of interaction between layers was given by Jagodzinski (1949 and successive publications). The geometric classification of such structures, called OD (order–disorder) structures, is due to Dornberger-Schiff (Dornberger-Schiff & Grell-Niemann, 1961, and later work). Research on alloy structures and sphere packings exhibiting simultaneously stacking and twinning faults is still active today.

6. Time-resolved crystallography

Routine crystal structure determination produces essentially static images, where atomic time-dependent displacements are averaged and represented by ellipsoids from which some aspects of the dynamics of the structure may sometimes be inferred. The modern high-intensity pulsed synchrotron X-ray sources have enabled the development of dynamic non-routine crystallography, which aims to follow atomic motions in real time, thus adding time as a fourth structural coordinate. Visualizing the actual sequence of atomic motions in chemical reactions, phase transitions and biological functions is part of the wide modern field of structural dynamics comprising microscopy, diffraction and spectroscopy in crystalline and non-crystalline phases. With the latest radiation sources and lasers, femtosecond resolution is attained today, of the order of, or finer than, a single atomic vibration period.

Crystallographic methods have been applied to the investigation of geometry changes on molecular excitation. With lasers, a fraction of the molecules in the crystal, typically a few per cent, may be excited without breaking the crystal. The result is a disordered crystal where the overall long-range order is assured by non-excited molecules, and structure determination shows additionally the generally minor population of excited molecules, their shapes, inter­atomic distances and induced molecular environment. If the photo-excitation is irreversible, the structures of the non-irradiated and the irradiated crystals are compared. Reversible excitations are investigated with pump–probe experiments. Before each X-ray pulse a laser pulse excites the molecules of interest. Coppens et al. (2005) describe pico­second powder diffraction experiments on molecular excitations to a singlet state, and microsecond experiments on inorganic complexes. The field is rapidly developing and will be an important application of free-electron lasers. The interested reader is referred to the special issue of Acta Crystallographica Section A, Volume A66, Part 2 (2010), Dynamical Structural Science.

7. Dynamical theory of X-ray diffraction

As already mentioned, Laue's kinematical theory of X-ray diffraction is the basis of crystal structure determination and led to its huge success. However, its deficiencies were realized very early. Diffracted intensities are often lower than predicted by the simple kinematical theory. This effect is called extinction and may be dramatic for strong low-order reflections. Darwin's (1914) two-beam dynamical theory explained this decrease of observed intensities, distinguished between primary and secondary extinction, and introduced the notion of a mosaic crystal. Even today, corrections of diffraction intensities for secondary extinction are derived with Darwin's transfer equations expressing the energy balance between the primary and the diffracted beam. This is the main impact dynamical theory has had on X-ray and neutron structure determination. Extinction theory is still approximate today and unsatisfactory for large extinction effects.

A complete n-beam dynamical theory was first published by P. P. Ewald in 1917. M. von Laue's version of this theory followed in 1931. The reader is referred to Ewald (1962) for a more detailed account. On this basis, much additional work has been accomplished since and continues today, often motivated by the design of monochromators for synchrotron radiation that are fabricated from perfect crystals. Applied to electron diffraction, the dynamical theory has led to the development of high-resolution electron microscopy and to electron crystallography.

The interaction of X-rays with crystals is just one of many examples of the behavior of waves in periodic structures (Brillouin, 1946). This may be the most fascinating aspect of the dynamical theory for the history of science. A periodic structure always constitutes a band-pass filter for waves. For a given propagation direction, only certain wavelengths may propagate. This is the case for macroscopic systems such as periodic arrangements of electrical resistances and impedances in electric lines, or of balls connected by springs. It applies to lattice vibrations in crystals (Born & von Karman, 1912), and to electron waves as in the band theory of electronic states in solids (Bloch, 1928). Waves forbidden in the crystal are ejected as is the case in X-ray diffraction, and the breadth of total reflection is proportional to the energy band gap between allowed waves, which in turn is proportional to the absolute value of the structure factor. It seems that in 1912 many physicists were aware of the lattice hypothesis for crystals, as the first theory of lattice vibrations preceeded by a few months the discovery of X-ray diffraction by crystals (Born & von Karman, 1912).

8. Neutron and electron diffraction

The discovery of the particle–wave duality of quantum mechanics suggested that light particles such as electrons might also be diffracted by crystals. The first successful electron diffraction experiments were carried out by Davisson & Germer (1927) with low-energy electrons, and by Thomson & Reid (1927) with high-energy electrons. With the discovery of the neutron by Chadwick in 1932, neutron diffraction became feasible, but was difficult to implement since no sufficiently strong neutron sources were available. The existence of Bragg reflections of thermal neutrons produced by large single crystals of MgO was first reported by Mitchell & Powers (1936). Around these radiations, distinctive scientific com­munities developed, usually with their own scientific societies and journals, and their specific diffraction theories.

Electron diffraction led to three applications: electron microscopy, gas electron diffraction and quantitative structure determination. The ability of electron microscopes to form physical-space images of ever-increasing resolution gave extremely important insights into the structure of matter. Examples are the direct visualization of structural imperfections such as dislocations and their movements, grain boundaries, antiphase domains, twinning interfaces, complicated mixtures of phases in minerals such as feldspars, and more. Gas electron diffraction enabled the determination of inter­atomic distances in molecules and radial distribution functions of distances essentially by applying Debye's equation [equation (2)], scattered intensities being much larger than for gas X-ray diffraction. Early quantitative structure determination from electron diffraction data was developed mainly in Moscow during the 1950s and 1960s. Many scientists believed that it should be nearly impossible to obtain meaningful structural results, since electron diffraction is multiple-beam dynamic and structure factors cannot be simply extracted from diffraction intensities. This difficulty is now reduced with the invention of precession electron diffraction (PED). Electron crystallography, with its ability to study very small specimens, is an increasingly important structure determination tool, also in combination with X-ray powder diffraction (Xie et al., 2008). In addition, convergent beam electron diffraction (CBED) has a growing impact on crystallography, since it allows the measurement of very accurate structure factors on an absolute scale, to be used for example in charge-density studies, or in the study of Debye–Waller factors.

Neutron crystallography became feasible only with the availability after World War II of nuclear reactors delivering sufficiently intense neutron beams. Structure determination techniques are analogous to the X-ray methods. The interest of structure determination with thermal neutrons lies in the fact that the widely varying neutron scattering lengths of atoms allow one to distinguish between neighboring atoms in the periodic table of the elements, and between different isotopes of an atomic species. In contrast to X-ray diffraction, hydrogen (which is unfortunately a strong spin-incoherent scatterer) and deuterium can be precisely located by neutron diffraction. Neutron scattering is unique in its application to magnetic structures and to the determination of phonon dispersion curves. Modern applications of thermal and cold neutrons are very diverse. As is the case with X-rays, there is a great demand for the ever-increasing beam intensities that are delivered by nuclear reactors and spallation sources.

9. Conclusions

Crystallography is considered to be a mature science, in the sense that its theoretical foundations are very well established, and the scientific drive is towards applications. This is certainly true for crystal structure determination, which has become a well established high-throughput method of chemical analysis. Macromolecular structure determination appears to evolve in the same direction. However, it is hazardous to write off crystallography as `mature' in the sense that nothing new is to be expected. Modern crystallographic research covers a very broad scientific domain, it is very innovative, reaches far beyond crystal structure determination, and is not always referred to explicitly as crystallography. It will remain influential for a long time. To appreciate the scope of modern crystallography, the reader is referred to the Special Issues of Acta Crystallographica Section A, Crystallography Across the Sciences, Volumes A54, part 6(1) (1998) and A64, part 1 (2008). State-of-the-art presentations of modern crystallography are found in International Tables for Crystallography, Volumes A to G (see https://it.iucr.org/ ).

As a rule, theories of scattering and diffraction of short-wavelength radiations by matter, and theories enabling structure determination were available decades before they could be practically applied. Today, the capabilities of technical resources such as computing hardware, detectors and radiation sources are growing fast. This will enable new research and realizations deemed very difficult or impossible today.

Footnotes

†This Laue centennial article has also been published in Zeitschrift für Kristallographie [Schwarzenbach (2012). Z. Kristallogr. 227, 52–62].