A semi-supervised deep-learning approach for automatic crystal structure classification

The structural solution problem can be a daunting and time-consuming task. Especially in the presence of impurity phases, current methods, such as indexing, become more unstable. In this work, the novel approach of semi-supervised learning is applied towards the problem of identifying the Bravais lattice and the space group of inorganic crystals. The reported semi-supervised generative deep-learning model can train on both labeled data, i.e. diffraction patterns with the associated crystal structure, and unlabeled data, i.e. diffraction patterns that lack this information. This approach allows the models to take advantage of the troves of unlabeled data that current supervised learning approaches cannot, which should result in models that can more accurately generalize to real data. In this work, powder diffraction patterns are classified into all 14 Bravais lattices and 144 space groups (the number is limited due to sparse coverage in crystal structure databases), which covers more crystal classes than other studies. The reported models also outperform current deep-learning approaches for both space group and Bravais lattice classification using fewer training data.

1. Introduction

The first step towards understanding the properties of a crystalline material at a microscopic level is identifying the crystal structure. However, this is nontrivial. The first part of crystal structure determination is indexing. There are several programs which can be used, such as DICVOL06 (Boultif & Louër, 1991), TOPAS (Coelho, 2018), GSAS-II (Toby & Von Dreele, 2013) or N-TREOR (Werner et al., 1985; Altomare et al., 2000). These programs output a set of space groups and lattice parameters that could represent the crystal. Using Le Bail (Le Bail et al., 1988) and Pawley (Pawley, 1981) refinements, the space group that fits the diffraction pattern the best can be identified. Rietveld (Rietveld, 1967, 1969) refinement can then be applied to profile the lattice parameters and check the space group. In the presence of impurity phases, this approach becomes more expensive as peaks must be selected manually or tolerance levels must be tuned to discard a certain number of peaks.

One of the approaches for identifying the positions of atoms in a crystal is the charge-flipping algorithm (CFA) (Oszlányi & Sütő, 2008; Palatinus, 2013; Baerlocher et al., 2007). CFA is an iterative approach that relies on fast Fourier transforms to determine the crystal structure of a material (Nussbaumer, 1981; Palatinus & Chapuis, 2007). For CFA, the unit cell and Bravais lattice have to be known, that is, we must have already been successful with some degree of indexing. CFA also cannot handle impurity phases, which are prevalent in many real-world samples.

Data science methods are being used increasingly in materials development (Balachandran, 2020; Vandermause et al., 2020; Reyes & Maruyama, 2019; Karigerasi et al., 2018). An example of this is the use of supervised neural networks (NNs) to analyze diffraction patterns. Supervised learning is an approach that seeks to learn a functional mapping between data and their labels. The benefit of NNs is that they, unlike CFA, do not require additional parameters, such as the Bravais lattice or lattice parameters. Although some approaches use NNs to aid in the Rietveld refinement (Ozaki et al., 2020; Chang et al., 2020; Schmidt et al., 2019; Schleder et al., 2019), others use NNs to classify diffraction patterns on the basis of the crystal structure. These classifiers can be trained to identify impurity phases and can be tailored towards specific detectors or parameters. For example, Ryu et al. (2019) trained an NN to classify the diffraction patterns of crystals that had defects. Liu et al. (2019) used the pair-density function with powder neutron diffraction data for space-group classification. Ziletti et al. (2018) used a convolutional neural network to classify simulated single-crystal diffraction X-ray image data into eight space groups.

A number of studies represent powder diffraction patterns as 2D images. However, the information is inherently one dimensional. Previous groups probably used the image approach to take easy advantage of trained models developed by the machine-learning community. Unfortunately, this could introduce more complexities to the model. Garcia-Cardona et al. (2019) carried out one of the only studies to examine neutron scattering data and used a 1D approach with simulated powder diffraction data to both differentiate perovskites into five crystal systems and tune the lattice parameters using regression. This study only looked at a small subset of crystals.

A significant challenge with NNs is that they struggle to generalize to new data sets. Most models that predict the space group of a material use less than 100 space groups in their training data set, which limits their application to new diffraction patterns. However, large labeled diffraction data sets are often rare, as labeling them is an expensive task. For this reason, we use a semi-supervised model, which takes advantage of both labeled and unlabeled data during training (Odena, 2016; Zhu & Goldberg, 2009; Kingma et al., 2014; Kipf & Welling, 2016). We employ a generative network that can extract features from the unlabeled data distribution and match these features with the corresponding crystal structure. This allows semi-supervised learning to be used on more data sets, especially ones where labels are not available.

In this study, we propose a 1D semi-supervised model for Bravais lattice and space-group classification using powder neutron diffraction data. Our NNs are trained with data spanning 144 space groups and 14 Bravais lattices. The models used in this study are freely available and can be downloaded (Lolla & Liang, 2021).

2. Methods

2.1. Data

To test our approach under conditions where we know the correct answer, we worked with simulated data sets. Our data were taken from the Inorganic Crystal Structure Database (ICSD; https://www.psds.ac.uk/icsd), which contains structural information about more than 210 000 crystals (Bergerhoff et al., 1983). A total of 138 362 diffraction patterns were simulated using TOPAS (Coelho, 2018). For the Bravais lattices, we combine the rhombohedral and tetragonal classes for a total of 14 classes with `F', `I', `P' and `C' representing the face-centered, body-centered, primitive and base-centered lattices, respectively. We note that there is an inherent class imbalance in the ICSD, as shown in Fig. 1. The most prevalent classes in this data set were the primitive hexagonal, the face-centered cubic and the primitive orthorhombic lattices. The least represented lattices were the face-centered ortho­rhombic and body-centered orthorhombic lattices.

Figure 1 The (a) Bravais lattice and (b) space-group class distribution of the simulated data.

For the space groups, we used 136 454 of the simulated diffraction patterns. We used only the space groups that had more than 50 diffraction patterns, leaving us with 144 out of the 225 space groups present in the ICSD. The most frequent space group is No. 62 (Pnma), which is orthorhombic, and accounts for the disproportionately large number of ortho­rhombic (P) diffraction patterns in the ICSD data set. A complete list of the 144 space groups used is shown in Table 1. A complete list of the ICSD IDs used in this study can be found in the GitHub repository at the URL https://github.com/usnistgov/semi-supervised-neutron (Lolla & Liang, 2021).

Table 1
List of space groups sorted by crystal system

Crystal system	Space groups
Triclinic	1, 2
Monoclinic	4–15
Orthorhombic	18–20, 26, 29, 31, 33, 34, 36, 38, 40–47, 51–53, 55–67, 69–74
Tetragonal	82, 85–88, 92, 96, 99, 100, 107, 109, 113, 114, 119 121, 122, 123, 125–131, 135–137, 139–142
Trigonal	143, 144, 146–148, 150, 152, 154–157, 159–167
Hexagonal	173–176, 180, 182, 185–187, 189–194
Cubic	197–201, 203–206, 212, 213, 215–218, 220, 221, 223–227, 229, 230

In this study, we use a 1D approach rather than the traditional 2D image approach. Our data set consists of diffraction patterns of powders. Examples of the 1D diffraction patterns are shown in Fig. 2. To normalize these diffraction patterns, we divided all intensities in each diffraction pattern by the maximum intensity. This ensures that the new maximum intensity is equal to 1 and the minimum is equal to 0.

Figure 2 Example 1D diffraction patterns for (a) hexagonal (P) crystals and (b) orthorhombic (P) crystals.

2.2. Models

We use two approaches to classify the diffraction patterns: a supervised approach using convolutional neural networks (CNNs) and a semi-supervised approach using a semi-supervised generative adversarial network (SGAN).

2.3. Supervised model

We used a 1D ResNet-18, a residual network (He et al., 2016), model to identify the crystal structure of the diffraction patterns. ResNets are examples of CNNs which are commonly used for image classification network algorithms. CNNs consist of convolutional layers, which are responsible for extracting high-level features, such as edges and colors, from images. These layers are used to create a feature map consisting of the most relevant characteristics of the image. To create this map using 1D data, a filter of size n is applied to a larger sequence with size m, and the dot product of every n consecutive values and the filter is computed. This generates a smaller matrix that only includes the relevant features (LeCun & Bengio, 1995).

A ResNet was used in this study to overcome the degradation problem, which occurs when neural networks are too dense so that the accuracy saturates and then quickly degrades (He et al., 2016). ResNets are characterized by their residual blocks, which contain convolutional layers with an identity function. Fig. 3 shows the model architecture used for the ResNet-18, and includes an example of a ResNet block used in this model. During training, we randomly selected 90% of the data to use as the training set and the remaining 10% of the data were used to test the model. This testing data set was distinct from the training one, so the model did not learn from the testing data. These models and the associated training scripts are available on GitHub (Lolla & Liang, 2021).

Figure 3 (a) The ResNet block used in this study and (b) the ResNet-18 architecture. In both parts, the orange, purple, green, red, gray, blue and yellow layers represent 1D Convolutional layers, Dropout layers, Layer Normalization layers, Gaussian error linear unit (GELU) activation functions, 1D Max Pooling layers, 1D Adaptive Max Pooling layers and Fully Connected layers, respectively. The white circles with `+' signs in them represent the addition of two layers. In part (b), each ResNet block was repeated twice, shown by the `2×' next to each block. `nc' and `ks' represent the number of channels and the kernel stride for each convolutional layer.

2.4. Semi-supervised model

We also used an SGAN (Odena, 2016; Goodfellow et al., 2020; Salimans et al., 2016). The SGAN consists of two models: a Generator and a Discriminator. The Generator tries to fool the Discriminator with fake diffraction patterns, while the Discriminator aims to differentiate between real and fake diffraction patterns. The Discriminator also classifies the real labeled data into the corresponding crystal structure class.

2.4.1. Generator

The purpose of the Generator is to sample the latent space, a high-dimensional feature space, to generate realistic diffraction patterns. The inputs to the Generator were sampled from a random normal distribution with a mean of 0 and a standard deviation of 1. The Generator consists of 1D Convolutional Transpose layers, 1D Batch Normalization layers and a Leaky rectified linear unit (ReLU) activation function. The Convolutional Transpose layers are used to upsample the data (Radford et al., 2015; Dumoulin & Visin, 2016). The Batch Normalization layers standardize the output of each layer, which reduces error when the model tries to generalize to new inputs (Ioffe & Szegedy, 2015) and has also been shown to reduce mode collapse, a major problem in GANs (Radford et al., 2015). Mode collapse occurs when the Generator only produces a few distinct diffraction patterns despite the latent space input. The Leaky ReLU (Xu et al., 2015) activation with α = 0.2 is used rather than ReLU to reduce the vanishing gradients problem (Radford et al., 2015). Graphs of the ReLU and the Leaky ReLU activation functions are shown in Fig. 4. For negative values, the derivative for the Leaky ReLU function is equal to α, but for the ReLU function, it is equal to 0. By having a nonzero derivative for all values, the Leaky ReLU is used to combat the sparse gradient problem that occurs while training GANs. Due to our normalization method, which was dividing all values in a diffraction pattern by the maximum intensity, the Discriminator's inputs were in the range from 0 to 1. For this reason, a sigmoid activation was applied to the last layer of our Generator, rather than the hyperbolic tangent function recommended by Salimans et al. (2016). Fig. 5 shows the model architecture of the Generator.

Figure 4 (a) ReLU, (b) Leaky ReLU with α = 0.2, (c) derivative of ReLU and (d) derivative of Leaky ReLU with α = 0.2.

Figure 5 The architecture for the Generator. The blue, green, orange and red boxes represent 1D Convolutional Transpose layers, Leaky ReLU activations, Batch Normalization layers and sigmoid activations, respectively. `nc' and `ks' in the blue boxes represent the number of channels and the kernel size, respectively, for each 1D Convolutional Transpose layer.

2.4.2. Discriminator

The Discriminator has two objectives: to differentiate between real and generated data, and to classify the real data into the correct class. To do this, we used the same 1D ResNet-18 model described in Section 2.3, but an activation function to the last fully connected layer, as proposed by Salimans et al. (2016). This activation function is shown in equation (1) and is a version of the softmax activation:

In this equation, l_k(x) represents the logit for class k with data x. By doing this, we eliminate the need for a second output layer and instead use only the logits from the classification layer. By applying this activation function, diffraction patterns with larger logits, which signify more confident predictions, will be classified as `real', whereas diffraction patterns with smaller logits will be classified as `fake'. This encourages the Discriminator to be more confident in its predictions, which sharpens the decision boundary between classes. The Discriminator's architecture is shown in Fig. 6.

Figure 6 The model architecture for the ResNet-18 Discriminator. The orange, purple, green, red, gray, blue and yellow layers represent 1D Convolutional layers, Dropout layers, Layer Normalization layers, GELU activation functions, 1D Max Pooling layers, 1D Adaptive Max Pooling layers and Fully Connected layers, respectively. The white circles with `+' signs in them represent the addition of two layers. In part (b), each ResNet block was repeated twice. `nc' and `ks' represent the number of channels and the kernel stride for each convolutional layer, respectively.

While training the Discriminator, there are two modes: supervised and unsupervised. During unsupervised training, the Discriminator acts the same way it would in a regular GAN as it tries to determine that the generated diffraction patterns are fake and the data drawn from the unsupervised set is real. In the supervised mode, the Discriminator is trained to predict the class label for real samples. Training in the unsupervised mode can help the Discriminator extract features from the data, and training on the supervised data will allow the Discriminator to use those extracted features for classification.

2.4.3. Loss functions and objective functions

The modified min–max loss proposed by Goodfellow et al. (2020) was used for the adversarial loss between the networks. The objective function that the Generator tries to maximize is shown in equation (2):

represents the parameters in the Generator and z⁽ⁱ⁾ represents the random values in the latent space. G(z⁽ⁱ⁾) is the generated diffraction pattern from the Generator and is the probability that the Discriminator predicts that the generated pattern is real.

For the Discriminator, the objective function to be maximized is shown in equation (3):

represents the parameters in the Discriminator, x⁽ⁱ⁾ is the unsupervised real data, and maximizing D(x⁽ⁱ⁾) implies that the model can identify real data. Like the Generator's objective function, z⁽ⁱ⁾ represents the random values in the latent space and G(z⁽ⁱ⁾) is the generated diffraction pattern from the latent space. Increasing the value of shows that the Discriminator can determine that the generated patterns are fake. Equation (3) also includes the categorical cross entropy loss, which is shown in the term . Here, C represents the number of classes, t_i shows whether the ith class is the label of the diffraction pattern and s_i is the Discriminator's prediction.

2.4.4. Training details

Fig. 7 shows the training pipeline used in the SGAN. During SGAN training, the Discriminator has three inputs: a generated sequence from the Generator, a powder diffraction pattern that is labeled with either the space group or the Bravais lattice, and an unlabeled powder diffraction pattern that does not include the crystal structure. We train our SGAN using four different amounts of labeled training data. In all scenarios, we randomly select 10% of the data as testing data, which is distinct from the labeled training data and the unlabeled training data. In the first scenario, we use 5% of the data as labeled training data and 85% as unlabeled training data. In the second, we use 10% of the data as labeled training data and 80% as unlabeled training data. In the third, we use 25% of the data as labeled training data and 65% of the data as unlabeled training data, and finally we use 50% of the data as labeled training data with 40% of the data as unlabeled training data. We also train our supervised ResNet with the same 5, 10, 25 and 50% of the data to compare the accuracy of the SGAN with that of the purely supervised approach.

Figure 7 The architecture for the semi-supervised GAN.

To train a supervised classifier, we use only the percentage of labeled training data. The model uses a powder diffraction pattern as input and aims to differentiate between the various crystal structure classes.

Table 2 shows the hyperparameters used in the ResNet and the SGAN.

Table 2
Hyperparameters used in the supervised ResNet, Generator and Discriminator

Hyperparameter	Supervised	Generator	Discriminator
Optimizer	Adam	Adam	Adam
Learning rate	1 × 10^−4	1 × 10^−6	5 × 10^−6
Dropout rate	0.1	None	0.1
Batch size	64	32	32
Nonlinear activations	GELU	Leaky ReLU & sigmoid	GELU

We used PYTORCH (Paszke et al., 2017) as a deep-learning framework. To accelerate training, each model was trained on eight NVIDIA Tesla V100 Tensor Cores.

3. Results and discussion

3.1. Supervised model

Our supervised ResNet trained on 90% of the data set had an accuracy of 88%. The confusion matrix for the Bravais lattice model is shown in Fig. 8. By plotting the predicted Bravais lattice against the actual Bravais lattice, the confusion matrix provides more information about the sets of classes that the network misclassified. If the model had a perfect testing accuracy, the values along the principal diagonal would sum to 100% as the network would have classified every diffraction pattern correctly. Again, there is a clear imbalance in the sampled ICSD data set, with orthorhombic (F) and orthorhombic (I) having the least samples. From the confusion matrix, we can see that, despite the fact that orthorhombic (P) is the most prevalent class, the model misclassifies some of these as monoclinic (P) crystals. The network also has trouble differentiating between triclinic (P) and monoclinic (P) diffraction patterns, as both of these classes have low symmetries, agreeing with previous studies (Garcia-Cardona et al., 2019). Similarly to Suzuki et al. (2020), we believe that this result was caused by undersampling the triclinic crystals.

Figure 8 The confusion matrix for the supervised ResNet-18.

For the space-group identification, our model had a top-1 accuracy of 80.6% and a top-5 accuracy of 90.27% across all 144 space groups. We trained our model on all 230 space groups and found that the model had a top-1 accuracy of 74% and a top-5 accuracy of 85%. We decided to investigate further the model on only the 144 most prevalent space groups within the data set, due to a major class imbalance. Some space groups had less than 50 diffraction patterns, less than 0.03% of our data set. Accuracy was measured by dividing the number of correctly classified diffraction patterns in the testing set by the total number of patterns in the testing set. Top-5 accuracy is the percentage of samples for which the actual space group was one of the model's top five predictions. This outperforms most current models of which we are aware. Liu et al. (2019) used machine learning with a pairwise distribution function with a top-1 accuracy of 71% and a top-5 accuracy of 90% across 45 space groups. Tiong et al. (2020) classified X-ray diffraction data into 8, 20, 49 and 72 space groups (Table 3). Their accuracy decreased from 99 to 80% for 8 and 72 space groups, respectively, implying that this accuracy would decrease further if their model was trained on more space groups. Aguiar et al. (2019) had a top-2 accuracy greater than 80% across all space groups, but used a data set consisting of 650 000 diffraction patterns, more than five times the size of the data set used in this study. However, they used a 1D network, suggesting that a 1D approach can lead to more accurate predictions. We note that we did not take advantage of data augmentation.

Table 3
Comparison with other supervised space-group classifiers

Study	Number of space groups	Type of data	Accuracy (%)
Our model	144	Powder diffraction	80
Liu et al. (2019)	45	Pair distribution function	71
Ziletti et al. (2018)	8	2D X-ray diffraction	96.3
Tiong et al. (2020)	72	2D X-ray diffraction	80.2

3.2. Semi-supervised model

We compare the accuracy of the SGAN with the accuracy of the supervised model in Table 4. The SGAN consistently out­performs the purely supervised model, showing that the semi-supervised approach has the potential to be more applicable in the real world. A graph comparing the accuracy of the supervised and semi-supervised models is shown in Fig. 9. This graph shows that although the accuracy of the SGAN is impacted by a lack of data, the difference between the accuracy of the SGAN and the accuracy of the supervised model is greatest when only 5% of the data are used.

Table 4
Comparing the accuracy (%) of the SGAN to the purely supervised approach with different quantities of labeled training data for both Bravais lattice and space-group classification

Percentage of data	Supervised Bravais lattice	SGAN Bravais lattice	Supervised space group	SGAN space group
5	61	70	54	60
10	68	74	61	65
25	76	80	68	72
50	82	85	75	78

Figure 9 Graph comparing the accuracies of the SGAN and supervised ResNet for Bravais lattice and space-group identification.

4. Conclusion

In this study, we use both CNNs and a semi-supervised GAN to investigate supervised and semi-supervised approaches for crystal structure classification. We demonstrate that SGANs can prove to be more accurate with limited quantities of labeled data for Bravais lattice and space-group classification. Further, we explore a 1D approach rather than a traditional 2D one. Our 1D model is more accurate than 2D image models, which agrees with previous results in the literature. Our semi-supervised model is also more applicable to real data sets which will lack large quantities of labeled data.

In the future, we would like to train the SGAN to identify impurity phases and to test the method on real data sets.