research papers\(\def\hfill{\hskip 5em}\def\hfil{\hskip 3em}\def\eqno#1{\hfil {#1}}\)

Journal logoSTRUCTURAL SCIENCE
CRYSTAL ENGINEERING
MATERIALS
ISSN: 2052-5206

The seventh blind test of crystal structure prediction: structure ranking methods

crossmark logo
Lily M. Hunnisett,a* Nicholas Francia,a Jonas Nyman,a Nathan S. Abraham,b Srinivasulu Aitipamula,c Tamador Alkhidir,d Mubarak Almehairbi,d Andrea Anelli,e Dylan M. Anstine,f John E. Anthony,g Joseph E. Arnold,h Faezeh Bahrami,i Michael A. Bellucci,j Gregory J. O. Beran,k Rajni M. Bhardwaj,b Raffaello Bianco,l Joanna A. Bis,m A. Daniel Boese,n James Bramley,h Doris E. Braun,o Patrick W. V. Butler,h Joseph Cadden,c,h Stephen Carino,m Ctirad Červinka,p Eric J. Chan,q Chao Chang,r Sarah M. Clarke,s Simon J. Coles,h Cameron J. Cook,k Richard I. Cooper,t Tom Darden,u Graeme M. Day,h Wenda Deng,v Hanno Dietrich,w Antonio DiPasquale,x Bhausaheb Dhokale,d,y Bouke P. van Eijck,z Mark R. J. Elsegood,aa Dzmitry Firaha,w Wenbo Fu,r Kaori Fukuzawa,bb,cc Nikolaos Galanakis,q Hitoshi Goto,dd,ee Chandler Greenwell,j Rui Guo,ff Jürgen Harter,a Julian Helfferich,w Johannes Hoja,n John Hone,gg Richard Hong,b,q Michal Hušák,hh Yasuhiro Ikabata,dd Olexandr Isayev,f Ommair Ishaque,ii Varsha Jain,u Yingdi Jin,r Aling Jing,ii Erin R. Johnson,s Ian Jones,gg K. V. Jovan Jose,jj Elena A. Kabova,kk Adam Keates,gg Paul F. Kelly,aa Jiří Klimeš,ll Veronika Kostková,p He Li,r Xiaolu Lin,r Alexander List,n Congcong Liu,r Yifei Michelle Liu,w Zenghui Liu,r Ivor Lončarić,l Joseph W. Lubach,x Jan Ludík,p Alexander A. Maryewski,mm Noa Marom,f,v,nn Hiroyuki Matsui,oo Alessandra Mattei,b R. Alex Mayo,s John W. Melkumov,ii Bruno Mladineo,l Sharmarke Mohamed,d,pp Zahrasadat Momenzadeh Abardeh,mm Hari S. Muddana,u Naofumi Nakayama,dd Kamal Singh Nayal,f Marcus A. Neumann,w Rahul Nikhar,ii Shigeaki Obata,dd,ee Dana O'Connor,v Artem R. Oganov,mm Koji Okuwaki,cc Alberto Otero-de-la-Roza,qq Sean Parkin,g Antonio Parunov,l Rafał Podeszwa,rr Alastair J. A. Price,s Louise S. Price,ff Sarah L. Price,ff Michael R. Probert,ss Angeles Pulido,a Gunjan Rajendra Ramteke,jj Atta Ur Rehman,ii Susan M. Reutzel-Edens,a,tt Jutta Rogal,q,uu Marta J. Ross,kk Adrian F. Rumson,s Ghazala Sadiq,a Zeinab M. Saeed,d Alireza Salimi,i Kiran Sasikumar,w Sivakumar Sekharan,j Kenneth Shankland,kk Baimei Shi,r Xuekun Shi,r Kotaro Shinohara,oo A. Geoffrey Skillman,u Hongxing Song,q Nina Strasser,n Jacco van de Streek,w Isaac J. Sugden,a Guangxu Sun,r Krzysztof Szalewicz,ii Lu Tan,r Kehan Tang,v Frank Tarczynski,m Christopher R. Taylor,h Alexandre Tkatchenko,vv Petr Touš,p Mark E. Tuckerman,q,ww,xx Pablo A. Unzueta,k Yohei Utsumi,cc Leslie Vogt-Maranto,q Jake Weatherston,ss Luke J. Wilkinson,aa Robert D. Willacy,a Lukasz Wojtas,yy Grahame R. Woollam,zz Yi Yang,v Zhuocen Yang,r Etsuo Yonemochi,cc Xin Yue,r Qun Zeng,r Tian Zhou,r Yunfei Zhou,r Roman Zubatyukf and Jason C. Colea

aThe Cambridge Crystallographic Data Centre, 12 Union Road, Cambridge CB2 1EZ, UK, bAbbVie Inc., Research & Development, 1 N Waukegan Road, North Chicago, IL 60064, USA, cCrystallization and Particle Sciences, Institute of Chemical and Engineering Sciences, 1, Pesek Road, Singapore, 627833, Singapore, dGreen Chemistry and Materials Modelling Laboratory, Khalifa University of Science and Technology, PO Box 127788, Abu Dhabi, United Arab Emirates, eRoche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd, Grenzacherstrasse 124, 4070 Basel, Switzerland, fDepartment of Chemistry, Carnegie Mellon University, 4400 Fifth Avenue, Pittsburgh, PA 15213, USA, gDepartment of Chemistry, University of Kentucky, Lexington, KY 40506, USA, hSchool of Chemistry, University of Southampton, Southampton SO17 1BJ, UK, iDepartment of Chemistry, Faculty of Science, Science Boulevard, Ferdowsi University of Mashhad, Mashhad, Iran, jXtalPi Inc, 245 Main Street, Cambridge, MA 02142, USA, kDepartment of Chemistry, University of California, Riverside, CA 92521, USA, lRuđer Bošković Institute, Bijenička cesta 54, Zagreb, Croatia, mCatalent Pharma Solutions, 160 Pharma Drive, Morrisville, NC 27560, USA, nDepartment of Chemistry, University of Graz, Heinrichstrasse 28, Graz, Austria, oUniversity of Innsbruck, Institute of Pharmacy, Innrain 52c, A-6020 Innsbruck, Austria, pDepartment of Physical Chemistry, University of Chemistry and Technology, Technická 5, 16628 Prague, Czech Republic, qDepartment of Chemistry, New York University, New York, NY 10003, USA, rXtalPi Inc., International Biomedical Innovation Park II 3F 2 Hongliu Road, Futian District, Shenzhen, Guangdong, China, sDepartment of Chemistry, Dalhousie University, 6274 Coburg Road, Dalhousie, Halifax, Canada, tDepartment of Chemistry, University of Oxford, 12 Mansfield Road, Oxford OX1 3TA, UK, uOpenEye Scientific Software, 9 Bisbee Court, Santa Fe, NM 87508, USA, vDepartment of Materials Science and Engineering, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA, wAvant-garde Materials Simulation, Alte Strasse 2, 79249 Merzhausen, Germany, xGenentech, Inc., 1 DNA Way, South San Francisco, CA 94080, USA, yDepartment of Chemistry, University of Wyoming, Laramie, Wyoming 82071, USA, zUniversity of Utrecht (Retired), Department of Crystal and Structural Chemistry, Padualaan 8, 3584 CH Utrecht, The Netherlands, aaChemistry Department, Loughborough University, Loughborough LE11 3TU, UK, bbGraduate School of Pharmaceutical Sciences, Osaka University, 1-6 Yamadaoka, Suita, Osaka 656-0871, Japan, ccSchool of Pharmacy and Pharmaceutical Sciences, Hoshi University, 2-4-41 Ebara, Shinagawa-ku, Tokyo 142-8501, Japan, ddInformation and Media Center, Toyohashi University of Technology, 1-1 Hibarigaoka, Tempaku-cho, Toyohashi, Aichi 441-8580, Japan, eeCONFLEX Corporation, Shinagawa Center building 6F, 3-23-17 Takanawa, Minato-ku, Tokyo 108-0074, Japan, ffDepartment of Chemistry, University College London, 20 Gordon Street, London WC1H 0AJ, UK, ggSyngenta Ltd., Jealott's Hill International Research Station, Berkshire, RG42 6EY, UK, hhDepartment of Solid State Chemistry, University of Chemistry and Technology, Technická 5, 16628 Prague, Czech Republic, iiDepartment of Physics and Astronomy, University of Delaware, Newark, DE 19716, USA, jjSchool of Chemistry, University of Hyderabad, Professor C.R. Rao Road, Gachibowli, Hyderabad, 500046 Telangana, India, kkSchool of Pharmacy, University of Reading, Whiteknights, Reading, RG6 6AD, UK, llDepartment of Chemical Physics and Optics, Faculty of Mathematics and Physics, Charles University, Ke Karlovu 3, 121 16 Prague, Czech Republic, mmSkolkovo Institute of Science and Technology, Bolshoy Boulevard 30, 121205 Moscow, Russia, nnDepartment of Physics, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA, ooGraduate School of Organic Materials Science, Yamagata University, 4-3-16 Jonan, Yonezawa 992-8510, Yamagata, Japan, ppCenter for Catalysis and Separations, Khalifa University of Science and Technology, PO Box 127788, Abu Dhabi, United Arab Emirates, qqDepartment of Analytical and Physical Chemistry, Faculty of Chemistry, University of Oviedo, Julián Clavería 8, 33006 Oviedo, Spain, rrInstitute of Chemistry, University of Silesia in Katowice, Szkolna 9, 40-006 Katowice, Poland, ssSchool of Natural and Environmental Sciences, Newcastle University, Kings Road, Newcastle NE1 7RU, UK, ttSuRE Pharma Consulting, LLC, 7163 Whitestown Parkway - Suite 305, Zionsville, IN 46077, USA, uuFachbereich Physik, Freie Universität, Berlin, 14195, Germany, vvDepartment of Physics and Materials Science, University of Luxembourg, 1511 Luxembourg City, Luxembourg, wwCourant Institute of Mathematical Sciences, New York University, New York, NY 10012, USA, xxNYU-ECNU Center for Computational Chemistry at NYU Shanghai, 3663 Zhongshan Road North, Shanghai 200062, China, yyDepartment of Chemistry, University of South Florida, USF Research Park, 3720 Spectrum Blvd, IDRB 202, Tampa, FL 33612 USA, and zzNovartis Pharma AG, Basel 4002, Switzerland
*Correspondence e-mail: lhunnisett@ccdc.cam.ac.uk

Edited by A. Nangia, CSIR–National Chemical Laboratory, India (Received 23 May 2024; accepted 3 September 2024; online 17 October 2024)

This article is part of a collection of articles covering the seventh crystal structure prediction blind test.

A seventh blind test of crystal structure prediction has been organized by the Cambridge Crystallographic Data Centre. The results are presented in two parts, with this second part focusing on methods for ranking crystal structures in order of stability. The exercise involved standardized sets of structures seeded from a range of structure generation methods. Participants from 22 groups applied several periodic DFT-D methods, machine learned potentials, force fields derived from empirical data or quantum chemical calculations, and various combinations of the above. In addition, one non-energy-based scoring function was used. Results showed that periodic DFT-D methods overall agreed with experimental data within expected error margins, while one machine learned model, applying system-specific AIMnet potentials, agreed with experiment in many cases demonstrating promise as an efficient alternative to DFT-based methods. For target XXXII, a consensus was reached across periodic DFT methods, with consistently high predicted energies of experimental forms relative to the global minimum (above 4 kJ mol−1 at both low and ambient temperatures) suggesting a more stable polymorph is likely not yet observed. The calculation of free energies at ambient temperatures offered improvement of predictions only in some cases (for targets XXVII and XXXI). Several avenues for future research have been suggested, highlighting the need for greater efficiency considering the vast amounts of resources utilized in many cases.

1. Introduction

1.1. Background

The Cambridge Crystallographic Data Centre (CCDC) has been organizing a set of blind tests to assess the predictive ability of existing methods for molecular crystal structure prediction (CSP), and to stimulate the development of novel approaches. The results of the seventh blind test are reported in two articles. Part one (Hunnisett et al., 2024[Hunnisett, L. M., Nyman, J., Francia, N., Abraham, N. S., Adjiman, C. S., Aitipamula, S., Alkhidir, T., Almehairbi, M., Anelli, A., Anstine, D. M., Anthony, J. E., Arnold, J. E., Bahrami, F., Bellucci, M. A., Bhardwaj, R. M., Bier, I., Bis, J. A., Boese, A. D., Bowskill, D. H., Bramley, J., Brandenburg, J. G., Braun, D. E., Butler, P. W. V., Cadden, J., Carino, S., Chan, E. J., Chang, C., Cheng, B., Clarke, S. M., Coles, S. J., Cooper, R. I., Couch, R., Cuadrado, R., Darden, T., Day, G. M., Dietrich, H., Ding, Y., DiPasquale, A., Dhokale, B., van Eijck, B. P., Elsegood, M. R. J., Firaha, D., Fu, W., Fukuzawa, K., Glover, J., Goto, H., Greenwell, C., Guo, R., Harter, J., Helfferich, J., Hofmann, D. W. M., Hoja, J., Hone, J., Hong, R., Hutchison, G., Ikabata, Y., Isayev, O., Ishaque, O., Jain, V., Jin, Y., Jing, A., Johnson, E. R., Jones, I., Jose, K. V. J., Kabova, E. A., Keates, A., Kelly, P. F., Khakimov, D., Konstantinopoulos, S., Kuleshova, L. N., Li, H., Lin, X., List, A., Liu, C., Liu, Y. M., Liu, Z., Liu, Z.-P., Lubach, J. W., Marom, N., Maryewski, A. A., Matsui, H., Mattei, A., Mayo, R. A., Melkumov, J. W., Mohamed, S., Momenzadeh Abardeh, Z., Muddana, H. S., Nakayama, N., Nayal, K. S., Neumann, M. A., Nikhar, R., Obata, S., O'Connor, D., Oganov, A. R., Okuwaki, K., Otero-de-la-Roza, A., Pantelides, C. C., Parkin, S., Pickard, C. J., Pilia, L., Pivina, T., Podeszwa, R., Price, A. J. A., Price, L. S., Price, S. L., Probert, M. R., Pulido, A., Ramteke, G. R., Rehman, A. U., Reutzel-Edens, S. M., Rogal, J., Ross, M. J., Rumson, A. F., Sadiq, G., Saeed, Z. M., Salimi, A., Salvalaglio, M., Sanders de Almada, L., Sasikumar, K., Sekharan, S., Shang, C., Shankland, K., Shinohara, K., Shi, B., Shi, X., Skillman, A. G., Song, H., Strasser, N., van de Streek, J., Sugden, I. J., Sun, G., Szalewicz, K., Tan, B. I., Tan, L., Tarczynski, F., Taylor, C. R., Tkatchenko, A., Tom, R., Tuckerman, M. E., Utsumi, Y., Vogt-Maranto, L., Weatherston, J., Wilkinson, L. J., Willacy, R. D., Wojtas, L., Woollam, G. R., Yang, Z., Yonemochi, E., Yue, X., Zeng, Q., Zhang, Y., Zhou, T., Zhou, Y., Zubatyuk, R. & Cole, J. C. (2024). Acta Cryst. B80, https://doi.org/10.1107/S2052520624007492.]) focuses on structure generation, while part two (this article) describes the final ranking of crystal structures.

CSP aims to predict the crystal structure of any given compound using computer simulations. CSP techniques have gained much attention due to its potential applications in fields such as pharmaceuticals, materials science, and solid-state chemistry, where a thermodynamically stable material is normally sought. A key challenge in CSP is therefore the accurate ranking of predicted crystal structures by their relative stabilities (free energies). A CSP study is often gauged by the successful prediction of the thermodynamically stable crystal structure. Ranking methods are also often employed as a standalone method for assessing relative stabilities when multiple forms are obtained from experiment.

The accurate determination of the most stable polymorph is crucial in many applications. For example, in the pharmaceutical industry, the solubility and bioavailability of a drug can be significantly affected by its crystal structure (Bauer et al., 2001[Bauer, J., Spanton, S., Henry, R., Quick, J., Dziki, W., Porter, W. & Morris, J. (2001). Pharm. Res. 18, 859-866.]). Predicting the most stable polymorph can guide experimental efforts to optimize drug formulation, manufacturing, and storage. Conversely, in cases where a metastable form is chosen or the stable form has not crystallized, an accurate energy ranking can be used to assess the risk that a late appearing, more stable form poses to the performance (bioavailability, shelf life) of the drug product. In materials science, the properties of a material, such as its electronic, optical, or mechanical behaviour, can be strongly influenced by its crystal structure. Therefore, accurate CSP methods can inform the design of materials with tailored properties for various applications (Tom et al., 2023[Tom, R., Gao, S., Yang, Y., Zhao, K., Bier, I., Buchanan, E. A., Zaykov, A., Havlas, Z., Michl, J. & Marom, N. (2023). Chem. Mater. 35, 1373-1386.]).

Recently, the CCDC conducted the seventh CSP blind test, providing a valuable opportunity to review and benchmark the performance of current crystal structure energy ranking methods. In this article, we present a detailed analysis of the results of the seventh blind test, focusing on understanding the strengths and weaknesses of various stability ranking techniques.

This report includes three distinct supplementary information (SI) sections. SI-A offers more information, tables, and figures on the analysis of the generated sets of structures and the preparation of structure lists. In SI-B, participating groups define their approach and possibly provide additional analysis of their landscape and results. Finally, SI-C contains the theoretically generated structures (and metadata) from each group, and any experimental structures that are not available through the CSD in the Crystallographic Information File (CIF) file format. Detailed experimental reports are provided in the supplementary information attached to phase one of this study (Hunnisett et al., 2024[Hunnisett, L. M., Nyman, J., Francia, N., Abraham, N. S., Adjiman, C. S., Aitipamula, S., Alkhidir, T., Almehairbi, M., Anelli, A., Anstine, D. M., Anthony, J. E., Arnold, J. E., Bahrami, F., Bellucci, M. A., Bhardwaj, R. M., Bier, I., Bis, J. A., Boese, A. D., Bowskill, D. H., Bramley, J., Brandenburg, J. G., Braun, D. E., Butler, P. W. V., Cadden, J., Carino, S., Chan, E. J., Chang, C., Cheng, B., Clarke, S. M., Coles, S. J., Cooper, R. I., Couch, R., Cuadrado, R., Darden, T., Day, G. M., Dietrich, H., Ding, Y., DiPasquale, A., Dhokale, B., van Eijck, B. P., Elsegood, M. R. J., Firaha, D., Fu, W., Fukuzawa, K., Glover, J., Goto, H., Greenwell, C., Guo, R., Harter, J., Helfferich, J., Hofmann, D. W. M., Hoja, J., Hone, J., Hong, R., Hutchison, G., Ikabata, Y., Isayev, O., Ishaque, O., Jain, V., Jin, Y., Jing, A., Johnson, E. R., Jones, I., Jose, K. V. J., Kabova, E. A., Keates, A., Kelly, P. F., Khakimov, D., Konstantinopoulos, S., Kuleshova, L. N., Li, H., Lin, X., List, A., Liu, C., Liu, Y. M., Liu, Z., Liu, Z.-P., Lubach, J. W., Marom, N., Maryewski, A. A., Matsui, H., Mattei, A., Mayo, R. A., Melkumov, J. W., Mohamed, S., Momenzadeh Abardeh, Z., Muddana, H. S., Nakayama, N., Nayal, K. S., Neumann, M. A., Nikhar, R., Obata, S., O'Connor, D., Oganov, A. R., Okuwaki, K., Otero-de-la-Roza, A., Pantelides, C. C., Parkin, S., Pickard, C. J., Pilia, L., Pivina, T., Podeszwa, R., Price, A. J. A., Price, L. S., Price, S. L., Probert, M. R., Pulido, A., Ramteke, G. R., Rehman, A. U., Reutzel-Edens, S. M., Rogal, J., Ross, M. J., Rumson, A. F., Sadiq, G., Saeed, Z. M., Salimi, A., Salvalaglio, M., Sanders de Almada, L., Sasikumar, K., Sekharan, S., Shang, C., Shankland, K., Shinohara, K., Shi, B., Shi, X., Skillman, A. G., Song, H., Strasser, N., van de Streek, J., Sugden, I. J., Sun, G., Szalewicz, K., Tan, B. I., Tan, L., Tarczynski, F., Taylor, C. R., Tkatchenko, A., Tom, R., Tuckerman, M. E., Utsumi, Y., Vogt-Maranto, L., Weatherston, J., Wilkinson, L. J., Willacy, R. D., Wojtas, L., Woollam, G. R., Yang, Z., Yonemochi, E., Yue, X., Zeng, Q., Zhang, Y., Zhou, T., Zhou, Y., Zubatyuk, R. & Cole, J. C. (2024). Acta Cryst. B80, https://doi.org/10.1107/S2052520624007492.]).

1.2. Previous blind tests of CSP

Here, we provide a brief summary of the first six CSP blind tests, focusing on the ranking methods, showing how the methods have evolved over the years and highlighting important developments. Computational methods are often referred to by acronyms, we have therefore included a dictionary of abbreviations at the end of this paper to aid the reader.

The first blind test in 1999 (Lommerse et al., 2000[Lommerse, J. P. M., Motherwell, W. D. S., Ammon, H. L., Dunitz, J. D., Gavezzotti, A., Hofmann, D. W. M., Leusen, F. J. J., Mooij, W. T. M., Price, S. L., Schweizer, B., Schmidt, M. U., van Eijck, B. P., Verwer, P. & Williams, D. E. (2000). Acta Cryst. B56, 697-714.]) featured primarily various empirical force fields), and force fields where the electrostatic model was parameterized to electronic structure calculations on the isolated molecule using Hartree–Fock or second-order Møller–Plesset perturbation theory (MP2) charge densities. Both atomic point charges and multipoles were used for the electrostatics. Besides force fields, statistical fitness functions based on probability distributions derived from the Cambridge Structural Database (CSD) were also used (Apostolakis et al., 2001[Apostolakis, J., Hofmann, D. W. M. & Lengauer, T. (2001). Acta Cryst. A57, 442-450.]), demonstrating that the scoring function used to assess predicted crystal structures is not necessarily a direct estimate of the structures' energy or thermodynamic stability.

The second blind test (Motherwell et al., 2002[Motherwell, W. D. S., Ammon, H. L., Dunitz, J. D., Dzyabchenko, A., Erk, P., Gavezzotti, A., Hofmann, D. W. M., Leusen, F. J. J., Lommerse, J. P. M., Mooij, W. T. M., Price, S. L., Scheraga, H., Schweizer, B., Schmidt, M. U., van Eijck, B. P., Verwer, P. & Williams, D. E. (2002). Acta Cryst. B58, 647-661.]) featured a wide range of force fields that were used to calculate lattice energies, and in one case the lattice vibrational contribution to the free energy, Fvib. A couple of participants used statistical fitness functions. Since the participants were only allowed to submit three crystal structures per target compound in the first five blind tests, more subjective assessments occasionally affected the selection of the final candidates. Besides the lattice energy, the predicted morphology, density, elastic constants and `chemical intuition' were used to influence the selection in some cases.

The third blind test saw participation from 18 groups, most of which used energy-based methods to rank their predicted structures (Day et al., 2005a[Day, G. M., Motherwell, W. D. S., Ammon, H. L., Boerrigter, S. X. M., Della Valle, R. G., Venuti, E., Dzyabchenko, A., Dunitz, J. D., Schweizer, B., van Eijck, B. P., Erk, P., Facelli, J. C., Bazterra, V. E., Ferraro, M. B., Hofmann, D. W. M., Leusen, F. J. J., Liang, C., Pantelides, C. C., Karamertzanis, P. G., Price, S. L., Lewis, T. C., Nowell, H., Torrisi, A., Scheraga, H. A., Arnautova, Y. A., Schmidt, M. U. & Verwer, P. (2005). Acta Cryst. B61, 511-527.]). Several potentials were more sophisticated than generic off-the-shelf force fields, featuring anisotropic repulsion for halogen atoms and distributed multipoles (Stone, 1981[Stone, A. J. (1981). Chem. Phys. Lett. 83, 233-239.]; Stone & Price, 1988[Stone, A. J. & Price, S. L. (1988). J. Phys. Chem. 92, 3325-3335.]; Coombes et al., 1996[Coombes, D. S., Price, S. L., Willock, D. J. & Leslie, M. (1996). J. Phys. Chem. 100, 7352-7360.]; Day & Price, 2003[Day, G. M. & Price, S. L. (2003). J. Am. Chem. Soc. 125, 16434-16443.]; Day et al., 2005b[Day, G. M., Motherwell, W. D. S. & Jones, W. (2005). Cryst. Growth Des. 5, 1023-1033.]). Angelo Gavezzotti used his PIXEL method, which calculates intermolecular interaction energies by direct numerical integration over electron densities (Gavezzotti, 2002[Gavezzotti, A. (2002). J. Phys. Chem. B, 106, 4145-4154.], 2005[Gavezzotti, A. (2005). Z. Kristallogr. Cryst. Mater. 220, 499-510.]). Detlef Hofmann, similarly to the first and second blind tests, used a non-energy based statistical potential trained on experimental structures in the CSD (Hofmann & Apostolakis, 2003[Hofmann, D. W. M. & Apostolakis, J. (2003). J. Mol. Struct. 647, 17-39.]).

The fourth blind test (Day et al., 2009[Day, G. M., Cooper, T. G., Cruz-Cabeza, A. J., Hejczyk, K. E., Ammon, H. L., Boerrigter, S. X. M., Tan, J. S., Della Valle, R. G., Venuti, E., Jose, J., Gadre, S. R., Desiraju, G. R., Thakur, T. S., van Eijck, B. P., Facelli, J. C., Bazterra, V. E., Ferraro, M. B., Hofmann, D. W. M., Neumann, M. A., Leusen, F. J. J., Kendrick, J., Price, S. L., Misquitta, A. J., Karamertzanis, P. G., Welch, G. W. A., Scheraga, H. A., Arnautova, Y. A., Schmidt, M. U., van de Streek, J., Wolf, A. K. & Schweizer, B. (2009). Acta Cryst. B65, 107-125.]) featured the first use of periodic dispersion-corrected density functional theory (DFT-D), which has since become very common. Marcus A. Neumann, Frank J. J. Leusen and John Kendrick used periodic PBE calculations supplemented with an empirically parameterized atom–atom C6 dispersion correction for their final energy minimization to successfully predict all four structures as the global minimum on the potential energy surface (Perdew et al., 1996a[Perdew, J. P., Burke, K. & Ernzerhof, M. (1996a). Phys. Rev. Lett. 77, 3865-3868.]; Neumann & Perrin, 2005[Neumann, M. A. & Perrin, M.-A. (2005). J. Phys. Chem. B, 109, 15531-15541.]; Neumann et al., 2008[Neumann, M. A., Leusen, F. J. J. & Kendrick, J. (2008). Angew. Chem. Int. Ed. 47, 2427-2430.]). The more sophisticated methodology used for the final lattice energy minimization produced far better results than the general purpose force fields with isotropic atom–atom interaction and atomic point charges (Day et al., 2009[Day, G. M., Cooper, T. G., Cruz-Cabeza, A. J., Hejczyk, K. E., Ammon, H. L., Boerrigter, S. X. M., Tan, J. S., Della Valle, R. G., Venuti, E., Jose, J., Gadre, S. R., Desiraju, G. R., Thakur, T. S., van Eijck, B. P., Facelli, J. C., Bazterra, V. E., Ferraro, M. B., Hofmann, D. W. M., Neumann, M. A., Leusen, F. J. J., Kendrick, J., Price, S. L., Misquitta, A. J., Karamertzanis, P. G., Welch, G. W. A., Scheraga, H. A., Arnautova, Y. A., Schmidt, M. U., van de Streek, J., Wolf, A. K. & Schweizer, B. (2009). Acta Cryst. B65, 107-125.]). It was, however, also noted that this method was by far the most computationally demanding of the methods used for the final optimization and it could only be applied to a limited number of crystal structures for each molecule. This DFT-D method was further validated by retrospectively applying it to structures from the first three blind tests, demonstrating that it ranked eight out of ten target crystal structures as the global minimum and, in general, also reproduced the experimentally observed geometry more accurately than other methods (Asmadi et al., 2009[Asmadi, A., Neumann, M. A., Kendrick, J., Girard, P., Perrin, M.-A. & Leusen, F. J. J. (2009). J. Phys. Chem. B, 113, 16303-16313.]).

The fifth blind test saw a wider adoption of quantum chemical methods, periodic DFT-D in particular, following the impressive performance in the previous blind test (Bardwell et al., 2011[Bardwell, D. A., Adjiman, C. S., Arnautova, Y. A., Bartashevich, E., Boerrigter, S. X. M., Braun, D. E., Cruz-Cabeza, A. J., Day, G. M., Della Valle, R. G., Desiraju, G. R., van Eijck, B. P., Facelli, J. C., Ferraro, M. B., Grillo, D., Habgood, M., Hofmann, D. W. M., Hofmann, F., Jose, K. V. J., Karamertzanis, P. G., Kazantsev, A. V., Kendrick, J., Kuleshova, L. N., Leusen, F. J. J., Maleev, A. V., Misquitta, A. J., Mohamed, S., Needs, R. J., Neumann, M. A., Nikylov, D., Orendt, A. M., Pal, R., Pantelides, C. C., Pickard, C. J., Price, L. S., Price, S. L., Scheraga, H. A., van de Streek, J., Thakur, T. S., Tiwari, S., Venuti, E. & Zhitkov, I. K. (2011). Acta Cryst. B67, 535-551.]). It was also the first blind test that featured a large, flexible drug-like molecule, which catalysed the adoption of CSP methods in the pharmaceutical industry (Nyman & Reutzel-Edens, 2018[Nyman, J. & Reutzel-Edens, S. M. (2018). Faraday Discuss. 211, 459-476.]).

The sixth blind test involved five target compounds, including challenging flexible molecules and a multi-component crystal, and participants were allowed to submit two sets of structures for each target compound, ranked with different methods (Reilly et al., 2016[Reilly, A. M., Cooper, R. I., Adjiman, C. S., Bhattacharya, S., Boese, A. D., Brandenburg, J. G., Bygrave, P. J., Bylsma, R., Campbell, J. E., Car, R., Case, D. H., Chadha, R., Cole, J. C., Cosburn, K., Cuppen, H. M., Curtis, F., Day, G. M., DiStasio, R. A. Jr, Dzyabchenko, A., van Eijck, B. P., Elking, D. M., van den Ende, J. A., Facelli, J. C., Ferraro, M. B., Fusti-Molnar, L., Gatsiou, C.-A., Gee, T. S., de Gelder, R., Ghiringhelli, L. M., Goto, H., Grimme, S., Guo, R., Hofmann, D. W. M., Hoja, J., Hylton, R. K., Iuzzolino, L., Jankiewicz, W., de Jong, D. T., Kendrick, J., de Klerk, N. J. J., Ko, H.-Y., Kuleshova, L. N., Li, X., Lohani, S., Leusen, F. J. J., Lund, A. M., Lv, J., Ma, Y., Marom, N., Masunov, A. E., McCabe, P., McMahon, D. P., Meekes, H., Metz, M. P., Misquitta, A. J., Mohamed, S., Monserrat, B., Needs, R. J., Neumann, M. A., Nyman, J., Obata, S., Oberhofer, H., Oganov, A. R., Orendt, A. M., Pagola, G. I., Pantelides, C. C., Pickard, C. J., Podeszwa, R., Price, L. S., Price, S. L., Pulido, A., Read, M. G., Reuter, K., Schneider, E., Schober, C., Shields, G. P., Singh, P., Sugden, I. J., Szalewicz, K., Taylor, C. R., Tkatchenko, A., Tuckerman, M. E., Vacarro, F., Vasileiadis, M., Vazquez-Mayagoitia, A., Vogt, L., Wang, Y., Watson, R. E., de Wijs, G. A., Yang, J., Zhu, Q. & Groom, C. R. (2016). Acta Cryst. B72, 439-459.]). This seems to have encouraged experimentation and most groups submitted structures ranked with methods not used in previous blind tests. Various DFT-D electronic structure calculations were used on the crystals, molecules and multimers. There were pure periodic electronic structure methods (periodic PBE with a variety of dispersion corrections), mixed quantum chemical plus force field methods (Ψmol), and potentials fitted to results of symmetry adapted perturbation theory (SAPT) (Misquitta et al., 2005[Misquitta, A. J., Podeszwa, R., Jeziorski, B. & Szalewicz, K. (2005). J. Chem. Phys. 123, 214103.]). Several groups took the opportunity to innovate corrections to the lattice energy, accounting for lattice vibrations (van Eijck, 2001[van Eijck, B. P. (2001). J. Comput. Chem. 22, 816-826.]; Nyman & Day, 2015[Nyman, J. & Day, G. M. (2015). CrystEngComm, 17, 5154-5165.]), polarization (Welch et al., 2008[Welch, G. W. A., Karamertzanis, P. G., Misquitta, A. J., Stone, A. J. & Price, S. L. (2008). J. Chem. Theory Comput. 4, 522-532.]; Mennucci et al., 2002[Mennucci, B., Tomasi, J., Cammi, R., Cheeseman, J. R., Frisch, M. J., Devlin, F. J., Gabriel, S. & Stephens, P. J. (2002). J. Phys. Chem. A, 106, 6102-6113..]), and even nucleation using kinetic Monte Carlo simulations to determine critical-nucleus sizes (Boerrigter et al., 2004[Boerrigter, S. X. M., Josten, G. P. H., van de Streek, J., Hollander, F. F. A., Los, J., Cuppen, H. M., Bennema, P. & Meekes, H. (2004). J. Phys. Chem. A, 108, 5894-5902.]; Deij et al., 2007[Deij, M. A., ter Horst, J. H., Meekes, H., Jansens, P. & Vlieg, E. (2007). J. Phys. Chem. B, 111, 1523-1530.]). One method based on Monte Carlo parallel tempering for structure generation and final ranking by periodic DFT-D successfully predicted all of the experimental target structures (Reilly et al., 2016[Reilly, A. M., Cooper, R. I., Adjiman, C. S., Bhattacharya, S., Boese, A. D., Brandenburg, J. G., Bygrave, P. J., Bylsma, R., Campbell, J. E., Car, R., Case, D. H., Chadha, R., Cole, J. C., Cosburn, K., Cuppen, H. M., Curtis, F., Day, G. M., DiStasio, R. A. Jr, Dzyabchenko, A., van Eijck, B. P., Elking, D. M., van den Ende, J. A., Facelli, J. C., Ferraro, M. B., Fusti-Molnar, L., Gatsiou, C.-A., Gee, T. S., de Gelder, R., Ghiringhelli, L. M., Goto, H., Grimme, S., Guo, R., Hofmann, D. W. M., Hoja, J., Hylton, R. K., Iuzzolino, L., Jankiewicz, W., de Jong, D. T., Kendrick, J., de Klerk, N. J. J., Ko, H.-Y., Kuleshova, L. N., Li, X., Lohani, S., Leusen, F. J. J., Lund, A. M., Lv, J., Ma, Y., Marom, N., Masunov, A. E., McCabe, P., McMahon, D. P., Meekes, H., Metz, M. P., Misquitta, A. J., Mohamed, S., Monserrat, B., Needs, R. J., Neumann, M. A., Nyman, J., Obata, S., Oberhofer, H., Oganov, A. R., Orendt, A. M., Pagola, G. I., Pantelides, C. C., Pickard, C. J., Podeszwa, R., Price, L. S., Price, S. L., Pulido, A., Read, M. G., Reuter, K., Schneider, E., Schober, C., Shields, G. P., Singh, P., Sugden, I. J., Szalewicz, K., Taylor, C. R., Tkatchenko, A., Tuckerman, M. E., Vacarro, F., Vasileiadis, M., Vazquez-Mayagoitia, A., Vogt, L., Wang, Y., Watson, R. E., de Wijs, G. A., Yang, J., Zhu, Q. & Groom, C. R. (2016). Acta Cryst. B72, 439-459.]; Kendrick et al., 2011[Kendrick, J., Leusen, F. J. J., Neumann, M. A. & van de Streek, J. (2011). Chem. A Eur. J. 17, 10736-10744.]).

1.3. Contributions to energy rankings

1.3.1. The Gibbs free energy

The relative thermodynamic stability between two polymorphs can be calculated as the difference in Gibbs free energy. Effects due to thermal expansion are often neglected, and most CSP practitioners use some variant of these generalized expressions:

[\Delta{}G(T,P) = \Delta{}E_{\rm latt}+\Delta{}F_{\rm vib}(T)-T\Delta{}S_{\rm conf }(T)+P\Delta{}V, \eqno(1)]

where the difference in vibrational energy between structures is

[\Delta{}F_{\rm vib}(T) = \Delta{E}_{\rm ZPE}-T\Delta{}S_{\rm vib}(T)+\int_{0}^{T}\Delta{ }C_{\rm v}(T){\rm d}T. \eqno(2)]

Here, Elatt is the lattice energy, EZPE the vibrational zero point energy, Cv the heat capacity, Svib and Sconf the vibrational and configurational entropies, respectively, and V the specific volume. We discuss the intricacies of calculating some of these contributions in some detail below.

1.3.2. The lattice energy

Calculating the relative stabilities of the many hypothetical structures generated in a CSP investigation is difficult for several reasons. Firstly, experimental evidence and computational investigations have shown that (free) energy differences between alternative crystal structures of the same compound are small, typically 1–2 kJ mol−1, and almost always less than 8 kJ mol−1 (Yu, 1995[Yu, L. (1995). J. Pharm. Sci. 84, 966-974.]; Gavezzotti & Filippini, 1995[Gavezzotti, A. & Filippini, G. (1995). J. Am. Chem. Soc. 117, 12299-12305.]; Yu et al., 2005[Yu, L., Huang, J. & Jones, K. J. (2005). J. Phys. Chem. B, 109, 19915-19922.]; Nyman & Day, 2015[Nyman, J. & Day, G. M. (2015). CrystEngComm, 17, 5154-5165.]; Cruz-Cabeza et al., 2015[Cruz-Cabeza, A. J., Reutzel-Edens, S. M. & Bernstein, J. (2015). Chem. Soc. Rev. 44, 8619-8635.]). Secondly, the thermodynamic stability at realistic temperatures depends on several contributing factors such as intermolecular interactions, conformational energy, lattice vibrational energy, and other subtle effects like the morphology of polar crystals (van Eijck & Kroon, 1997[van Eijck, B. P. & Kroon, J. (1997). J. Phys. Chem. B, 101, 1096-1100.]), thermal expansion and the entropic contribution from crystallographic disorder (Heit & Beran, 2016[Heit, Y. N. & Beran, G. J. O. (2016). Acta Cryst. B72, 514-529.]; Nyman et al., 2016[Nyman, J., Pundyke, O. S. & Day, G. M. (2016). Phys. Chem. Chem. Phys. 18, 15828-15837.]; Woollam et al., 2018[Woollam, G. R., Neumann, M. A., Wagner, T. & Davey, R. J. (2018). Faraday Discuss. 211, 209-234.]; O'Connor et al., 2022[O'Connor, D., Bier, I., Hsieh, Y.-T. & Marom, N. (2022). J. Chem. Theory Comput. 18, 4456-4471.]; Pokorný et al., 2022[Pokorný, V., Touš, P., Štejfa, V., Růžička, K., Rohlíček, J., Czernek, J., Brus, J. & Červinka, C. (2022). Phys. Chem. Chem. Phys. 24, 25904-25917.]; Tous & Červinka, 2023[Touš, P. & Červinka, C. (2023). Cryst. Growth Des. 23, 4082-4097.]).

The static cohesive energy between the crystal's constituents, the `lattice energy', is often the largest and most important contribution. There are a vast number of energy methods that can be used to calculate it. General purpose force fields like GAFF, COMPASS and OPLS are quick and computationally the most affordable (Wang et al., 2004[Wang, J., Wolf, R. M., Caldwell, J. W., Kollman, P. A. & Case, D. A. (2004). J. Comput. Chem. 25, 1157-1174.]; Sun, 1998[Sun, H. (1998). J. Phys. Chem. B, 102, 7338-7364.]; Jorgensen et al., 1996[Jorgensen, W. L. S. M. D., Maxwell, D. S. & Tirado-Rives, J. (1996). J. Am. Chem. Soc. 118, 11225-11236.]). To improve the reliability of a force field, it may be re-parameterized or tailored to the compound(s) of interest (Neumann, 2008[Neumann, M. A. (2008). J. Phys. Chem. B, 112, 9810-9829.]; Metz et al., 2016[Metz, M. P., Piszczatowski, K. & Szalewicz, K. (2016). J. Chem. Theory Comput. 12, 5895-5919.]; Zhang et al., 2018[Zhang, P., Wood, G. P., Ma, J., Yang, M., Liu, Y., Sun, G., Jiang, Y. A., Hancock, B. C. & Wen, S. (2018). Cryst. Growth Des. 18, 6891-6900.]; Mattei et al., 2022[Mattei, A., Hong, R. S., Dietrich, H., Firaha, D., Helfferich, J., Liu, Y. M., Sasikumar, K., Abraham, N. S., Miglani Bhardwaj, R., Neumann, M. A. & Sheikh, A. Y. (2022). J. Chem. Theory Comput. 18, 5725-5738.]; Nikhar & Szalewicz, 2022[Nikhar, R. & Szalewicz, K. (2022). Nat. Commun. 13, 3095.]).

Density functional tight binding (DFTB) approximations may be used as an intermediate energy model during CSP. The computational cost lies between force fields and DFT-D. Empirical corrections to improve the modelling of hydrogen bonding interactions, dispersion, and halogen bonds in DFTB are generally recommended (Brandenburg & Grimme, 2014a[Brandenburg, J. G. & Grimme, S. (2014a). J. Phys. Chem. Lett. 5, 1785-1789.]; Řezáč, 2017[Řezáč, J. (2017). J. Chem. Theory Comput. 13, 4804-4817.]; Iuzzolino et al., 2018[Iuzzolino, L., McCabe, P., Price, S. L. & Brandenburg, J. G. (2018). Faraday Discuss. 211, 275-296.]; Mortazavi et al., 2018[Mortazavi, M., Brandenburg, J. G., Maurer, R. J. & Tkatchenko, A. (2018). J. Phys. Chem. Lett. 9, 399-405.]; Řezáč, 2019[Řezáč, J. (2019). J. Comput. Chem. 40, 1633-1642.]; Bannwarth et al., 2019[Bannwarth, C., Ehlert, S. & Grimme, S. (2019). J. Chem. Theory Comput. 15, 1652-1671.]).

In the more recent blind tests, energy rankings based on dispersion-corrected generalized gradient approximations (GGAs) to the exchange-correlation functional of DFT demonstrated a remarkably consistent high predictive ability (Neumann & Perrin, 2005[Neumann, M. A. & Perrin, M.-A. (2005). J. Phys. Chem. B, 109, 15531-15541.]; Grimme, 2006[Grimme, S. (2006). J. Comput. Chem. 27, 1787-1799.]; Day et al., 2009[Day, G. M., Cooper, T. G., Cruz-Cabeza, A. J., Hejczyk, K. E., Ammon, H. L., Boerrigter, S. X. M., Tan, J. S., Della Valle, R. G., Venuti, E., Jose, J., Gadre, S. R., Desiraju, G. R., Thakur, T. S., van Eijck, B. P., Facelli, J. C., Bazterra, V. E., Ferraro, M. B., Hofmann, D. W. M., Neumann, M. A., Leusen, F. J. J., Kendrick, J., Price, S. L., Misquitta, A. J., Karamertzanis, P. G., Welch, G. W. A., Scheraga, H. A., Arnautova, Y. A., Schmidt, M. U., van de Streek, J., Wolf, A. K. & Schweizer, B. (2009). Acta Cryst. B65, 107-125.]; Bardwell et al., 2011[Bardwell, D. A., Adjiman, C. S., Arnautova, Y. A., Bartashevich, E., Boerrigter, S. X. M., Braun, D. E., Cruz-Cabeza, A. J., Day, G. M., Della Valle, R. G., Desiraju, G. R., van Eijck, B. P., Facelli, J. C., Ferraro, M. B., Grillo, D., Habgood, M., Hofmann, D. W. M., Hofmann, F., Jose, K. V. J., Karamertzanis, P. G., Kazantsev, A. V., Kendrick, J., Kuleshova, L. N., Leusen, F. J. J., Maleev, A. V., Misquitta, A. J., Mohamed, S., Needs, R. J., Neumann, M. A., Nikylov, D., Orendt, A. M., Pal, R., Pantelides, C. C., Pickard, C. J., Price, L. S., Price, S. L., Scheraga, H. A., van de Streek, J., Thakur, T. S., Tiwari, S., Venuti, E. & Zhitkov, I. K. (2011). Acta Cryst. B67, 535-551.]). The PBE (Perdew et al., 1996a[Perdew, J. P., Burke, K. & Ernzerhof, M. (1996a). Phys. Rev. Lett. 77, 3865-3868.]) and B86bPBE (Becke, 1986[Becke, A. (1986). J. Chem. Phys. 85, 7184-7187.]) exchange-correlation functionals are the most commonly used for molecular crystals and are widely considered transferable and reliable, reaching a tolerable error in relative lattice energies of about 3 kJ mol−1 (Moellmann & Grimme, 2014[Moellmann, J. & Grimme, S. (2014). J. Phys. Chem. C, 118, 7615-7621.]; Abramov et al., 2021[Abramov, Y. A., Li, B., Chang, C., Zeng, Q., Sun, G. & Gobbo, G. (2021). Cryst. Growth Des. 21, 5496-5502.]; Firaha et al., 2023[Firaha, D., Liu, Y. M., van de Streek, J., Sasikumar, K., Dietrich, H., Helfferich, J., Aerts, L., Braun, D. E., Broo, A., DiPasquale, A. G., Lee, A. Y., Le Meur, S., Nilsson Lill, S. O., Lunsmann, W. J., Mattei, A., Muglia, P., Putra, O. D., Raoui, M., Reutzel-Edens, S. M., Rome, S., Sheikh, A. Y., Tkatchenko, A., Woollam, G. R. & Neumann, M. A. (2023). Nature, 623, 324-328.]) for most electrically neutral species when care is taken to properly converge the calculation and use a k-point sampling that compensates for the different unit cell sizes of CSP structures. Reliable dispersion corrections include the variations of Grimme's D3 and D4 methods, many body dispersion (MBD) and the exchange hole dipole model (XDM) (Grimme et al., 2010[Grimme, S., Antony, J., Ehrlich, S. & Krieg, H. (2010). J. Chem. Phys. 132, 154104.]; Grimme et al., 2011[Grimme, S., Ehrlich, S. & Goerigk, L. (2011). J. Comput. Chem. 32, 1456-1465.]; Tkatchenko et al., 2012[Tkatchenko, A., DiStasio, R. A., Car, R. & Scheffler, M. (2012). Phys. Rev. Lett. 108, 236402.]; Caldeweyher et al., 2020[Caldeweyher, E., Mewes, J.-M., Ehlert, S. & Grimme, S. (2020). Phys. Chem. Chem. Phys. 22, 8499-8512.]; Otero-de-la-Roza & Johnson, 2013[Otero-de-la-Roza, A. & Johnson, E. R. (2013). J. Chem. Phys. 138, 054103.]; Reilly & Tkatchenko, 2015[Reilly, A. M. & Tkatchenko, A. (2015). Chem. Sci. 6, 3289-3301.]; Whittleton et al., 2016[Whittleton, S. R., Otero-de-la-Roza, A. & Johnson, E. R. (2016). J. Chem. Theory Comput. 13, 441-450.]; Price et al., 2023b[Price, A. J. A., Mayo, R. A., Otero-de-la-Roza, A. & Johnson, E. R. (2023b). CrystEngComm, 25, 953-960.]).

There are also non-local density functionals that include dispersion contributions in the functional itself (Vydrov & van Voorhis, 2010[Vydrov, O. A. & van Voorhis, T. (2010). J. Chem. Phys. 33, 244103.]; Schröder et al., 2017[Schröder, E., Cooper, V. R., Berland, K., Lundqvist, B. I., Hyldgaard, P. & Thonhauser, T. (2017). In Non-Covalent Interactions in Quantum Chemistry and Physics, edited by A. Otero-de-la-Roza & G. A. DiLabio, pp. 241-274. Elsevier.]; Chakraborty et al., 2020[Chakraborty, D., Berland, K. & Thonhauser, T. (2020). J. Chem. Theory Comput. 16, 5893-5911.]), rather than as a correction applied after the convergence of the charge density. Such functionals have not been used in previous blind tests, but one such functional, optPBE-vdW (Klimeš et al., 2009[Klimeš, J., Bowler, D. R. & Michaelides, A. (2009). J. Phys. Condens. Matter, 22, 022201.]), features here for the first time.

Improved accuracy in the electronic density and energy may be achieved with density functionals, known as meta-GGA functionals, that account for the second derivative of the charge density, or the kinetic energy density, in addition to the density and its gradient. Popular variants include TPSS and SCAN, and numerically stable variants thereof (Tao et al., 2003[Tao, J., Perdew, J. P., Staroverov, V. N. & Scuseria, G. E. (2003). Phys. Rev. Lett. 91, 146401.]; Sun et al., 2015[Sun, J., Ruzsinszky, A. & Perdew, J. P. (2015). Phys. Rev. Lett. 115, 036402.]; Bartók & Yates, 2019[Bartók, A. P. & Yates, J. R. (2019). J. Chem. Phys. 150, 161101.]; Mejía-Rodríguez & Trickey, 2019[Mejía-Rodríguez, D. & Trickey, S. B. (2019). J. Chem. Phys. 151, 207101.]; Ehlert et al., 2021[Ehlert, S., Huniar, U., Ning, J., Furness, J. W., Sun, J., Kaplan, A. D., Perdew, J. P. & Brandenburg, J. G. (2021). J. Chem. Phys. 154, 061101.]; Brandenburg et al., 2016[Brandenburg, J., Bates, J., Sun, J. & Perdew, J. (2016). Phys. Rev. B, 94, 115144.]).

Computationally efficient local and semi-local exchange-correlation functionals, including GGA and meta-GGA functionals, suffer from self-interaction errors (SIE), the spurious Coulomb repulsion of an electron from its own density (Perdew & Zunger, 1981[Perdew, J. P. & Zunger, A. (1981). Phys. Rev. B, 23, 5048-5079.]), which can cause large unpredictable problems in certain cases (LeBlanc et al., 2018[LeBlanc, L. M., Dale, S. G., Taylor, C. R., Becke, A. D., Day, G. M. & Johnson, E. R. (2018). Angew. Chem. 130, 15122-15126.]; Nyman et al., 2019[Nyman, J., Yu, L. & Reutzel-Edens, S. M. (2019). CrystEngComm, 21, 2080-2088.]; Greenwell & Beran, 2020[Greenwell, C. & Beran, G. J. (2020). Cryst. Growth Des. 20, 4875-4881.]; Beran et al., 2022[Beran, G. J. O., Wright, S. E., Greenwell, C. & Cruz-Cabeza, A. J. (2022). J. Chem. Phys. 156, 104112.]; O'Connor et al., 2023[O'Connor, D., Bier, I., Tom, R., Hiszpanski, A. M., Steele, B. A. & Marom, N. (2023). Cryst. Growth Des. 23, 6275-6289.]). SIE can be mitigated by hybrid functionals, such as the PBE-based hybrid PBE0, which include a fraction of Hartree–Fock exchange (Becke, 1993[Becke, A. D. (1993). J. Chem. Phys. 98, 5648-5652.]; Perdew et al., 1996b[Perdew, J. P., Ernzerhof, M. & Burke, K. (1996b). J. Chem. Phys. 105, 9982-9985.]; Adamo & Barone, 1999[Adamo, C. & Barone, V. (1999). J. Chem. Phys. 110, 6158-6170.]; Reilly & Tkatchenko, 2013[Reilly, A. M. & Tkatchenko, A. (2013). J. Chem. Phys. 139, 024705.]). This reduces their tendency to exaggerate electron delocalization (Cohen et al., 2012[Cohen, A. J., Mori-Sánchez, P. & Yang, W. (2012). Chem. Rev. 112, 289-320.]). However, owing to the non-locality of the exact exchange, the computational cost of hybrid functionals is higher than that of (semi-)local functionals by an order of magnitude, making them impractical for ranking a large number of putative crystal structures. Other methods for improving the accuracy of GGA DFT-D include the application of a monomer correction, based on, for instance, MP2 (Greenwell et al., 2022[Greenwell, C., Řezáč, J. & Beran, G. J. (2022). Phys. Chem. Chem. Phys. 24, 3695-3712.]), or using density corrected DFT (Rana et al., 2022[Rana, B., Beran, G. J. O. & Herbert, J. M. (2022). Mol. Phys. 121, e2138789.]). In addition to monomer corrections, dimer and multimer corrections have also been used in order to approximate hybrid functionals (Loboda et al., 2018[Loboda, O. A., Dolgonos, G. A. & Boese, A. D. (2018). J. Chem. Phys. 149, 124104.]; Hoja et al., 2023[Hoja, J., List, A. & Boese, A. D. (2024). J. Chem. Theory Comput. 20, 357-367.]).

There are also mixed energy models that combine molecular ab initio calculations with an intermolecular force field, referred to as Ψmol (Price et al., 2010[Price, S. L., Leslie, M., Welch, G. W. A., Habgood, M., Price, L. S., Karamertzanis, P. G. & Day, G. M. (2010). Phys. Chem. Chem. Phys. 12, 8478.]; Kazantsev et al., 2010[Kazantsev, A., Karamertzanis, P., Pantelides, C. & Adjiman, C. (2010). Process Systems Engineering, Vol. 6: Molecular Systems Engineering, pp. 1-42.]; Wen & Beran, 2011[Wen, S. & Beran, G. J. O. (2011). J. Chem. Theory Comput. 7, 3733-3742.]; Williams, 2001b[Williams, D. E. (2001b). J. Comput. Chem. 22, 1154-1166..]; Williams, 2001a[Williams, D. E. (2001a). J. Comput. Chem. 22, 1-20.]; Pyzer-Knapp et al., 2016[Pyzer-Knapp, E. O., Thompson, H. P. G. & Day, G. M. (2016). Acta Cryst. B72, 477-487.]). The molecular wavefunction can be used to obtain distributed multipoles, greatly improving the modelling of intermolecular electrostatic interactions (Stone, 1981[Stone, A. J. (1981). Chem. Phys. Lett. 83, 233-239.]; Coombes et al., 1996[Coombes, D. S., Price, S. L., Willock, D. J. & Leslie, M. (1996). J. Phys. Chem. 100, 7352-7360.]; Mooij & Leusen, 2001[Mooij, W. T. M. & Leusen, F. J. J. (2001). Phys. Chem. Chem. Phys. 3, 5063-5066.]; Day et al., 2005b[Day, G. M., Motherwell, W. D. S. & Jones, W. (2005). Cryst. Growth Des. 5, 1023-1033.]). Such models have featured prominently throughout the blind tests, and have produced several successful predictions (Kazantsev et al., 2011[Kazantsev, A. V., Karamertzanis, P. G., Adjiman, C. S., Pantelides, C. C., Price, S. L., Galek, P. T. A., Day, G. M. & Cruz-Cabeza, A. J. (2011). Int. J. Pharm. 418, 168-178.]).

To refine the lattice energies, high-level ab initio calculations, including CCSD(T), on molecular clusters can also be coupled with lower-level QM methods, such as periodic DFT (Pokorný et al., 2022[Pokorný, V., Touš, P., Štejfa, V., Růžička, K., Rohlíček, J., Czernek, J., Brus, J. & Červinka, C. (2022). Phys. Chem. Chem. Phys. 24, 25904-25917.]) or HF (Červinka & Beran, 2018[Červinka, C. & Beran, G. J. (2018). Chem. Sci. 9, 4622-4629.]), yielding efficient QM:QM fragmentation frameworks for molecular crystals (Herbert, 2019[Herbert, J. M. (2019). J. Chem. Phys. 151, 170901.]).

Machine learned potentials have recently gained considerable attention since a carefully trained model should be able to achieve DFT-D level accuracy and be orders of magnitude faster (Musil et al., 2018[Musil, F., De, S., Yang, J., Campbell, J. E., Day, G. M. & Ceriotti, M. (2018). Chem. Sci. 9, 1289-1300.]). A previously limiting factor for why machine learned potentials have not been suitable for crystal structure prediction was their lack of long-range interaction components. The models included only strictly local physics and chemistry, representing the crystalline environment by SOAP kernels and similar methods (Bartók et al., 2013[Bartók, A. P., Kondor, R. & Csányi, G. (2013). Phys. Rev. B, 87, 184115.]). In recent years, a number of methods have emerged for the treatment of long-range dispersion, polarization, and electrostatic interactions in machine learning (Anstine & Isayev, 2023[Anstine, D. M. & Isayev, O. (2023). J. Phys. Chem. A, 127, 2417-2431.]; Grisafi & Ceriotti, 2019[Grisafi, A. & Ceriotti, M. (2019). J. Chem. Phys. 151, 204105.]; Ko et al., 2021[Ko, T. W., Finkler, J. A., Goedecker, S. & Behler, J. (2021). Nat. Commun. 12, 398.]; Yue et al., 2021[Yue, S., Muniz, M. C., Calegari Andrade, M. F., Zhang, L., Car, R. & Panagiotopoulos, A. Z. (2021). J. Chem. Phys. 154, 034111.]; Zhang et al., 2022[Zhang, L., Wang, H., Muniz, M. C., Panagiotopoulos, A. Z., Car, R. & E, W. (2022). J. Chem. Phys. 156, 124107.]; Phuc Tu et al., 2023[Tu, N. T. P., Rezajooei, N., Johnson, E. R. & Rowley, C. N. (2023). Digit. Discov. 2, 718-727.]).

The 2018 Faraday Discussion on crystal structure prediction featured an insightful session on energy ranking methods, covering many of the aspects touched upon here in greater detail (Addicoat et al., 2018[Addicoat, M., Adjiman, C. S., Arhangelskis, M., Beran, G. J. O., Bowskill, D., Brandenburg, J. G., Braun, D. E., Burger, V., Cole, J., Cruz-Cabeza, A. J., Day, G. M., Deringer, V. L., Guo, R., Hare, A., Helfferich, J., Hoja, J., Iuzzolino, L., Jobbins, S., Marom, N., McKay, D., Mitchell, J. B. O., Mohamed, S., Neumann, M., Nilsson Lill, S., Nyman, J., Oganov, A. R., Piaggi, P., Price, S. L., Reutzel-Edens, S., Rietveld, I., Ruggiero, M., Ryder, M. R., Sastre, G., Schön, J. C., Taylor, C., Tkatchenko, A., Tsuzuki, S., van den Ende, J., Woodley, S. M., Woollam, G. & Zhu, Q. (2018). Faraday Discuss. 211, 325-381.]), as well as several benchmarking studies, often using the X23 benchmark set (Otero-de-la-Roza & Johnson, 2012[Otero-de-la-Roza, A. & Johnson, E. R. (2012). J. Chem. Phys. 137, 054103.]; Reilly & Tkatchenko, 2013[Reilly, A. M. & Tkatchenko, A. (2013). J. Chem. Phys. 139, 024705.]; Reilly & Tkatchenko, 2015[Reilly, A. M. & Tkatchenko, A. (2015). Chem. Sci. 6, 3289-3301.]; Cutini et al., 2016[Cutini, M., Civalleri, B., Corno, M., Orlando, R., Brandenburg, J. G., Maschio, L. & Ugliengo, P. (2016). J. Chem. Theory Comput. 12, 3340-3352.]; Hoja et al., 2017[Hoja, J., Reilly, A. M. & Tkatchenko, A. (2017). Wiley Interdiscip. Rev. Comput. Mol. Sci. 7, e1294.]; Hoja & Tkatchenko, 2018[Hoja, J. & Tkatchenko, A. (2018). Faraday Discuss. 211, 253-274.]; Loboda et al., 2018[Loboda, O. A., Dolgonos, G. A. & Boese, A. D. (2018). J. Chem. Phys. 149, 124104.]; Hoja et al., 2019[Hoja, J., Ko, H.-Y., Neumann, M. A., Car, R., DiStasio, R. A. Jr & Tkatchenko, A. (2019). Sci. Adv. 5, eaau3338.]). An alternative benchmark set focusing on energetic materials has since been proposed (O'Connor et al., 2022[O'Connor, D., Bier, I., Hsieh, Y.-T. & Marom, N. (2022). J. Chem. Theory Comput. 18, 4456-4471.]), which highlighted the need for further method development for applications in this field. The need for further benchmark data, highlighted during the discussions, has since resulted in the ongoing BEST-CSP COST action1. Accuracy of first-principles methods and their potential for reliable polymorph ranking at finite temperatures can be consistently benchmarked against critically assessed sublimation enthalpies or pressures for organic molecular materials (Červinka & Fulem, 2017[Červinka, C. & Fulem, M. (2017). J. Chem. Theory Comput. 13, 2840-2850.]; Červinka & Fulem, 2018[Červinka, C. & Fulem, M. (2018). Cryst. Growth Des. 19, 808-820.]). Experimental state-of-the-art in this field enables one to reach an uncertainty of the sublimation enthalpy around 0.5 kJ mol−1 for volatile organic materials (Fulem et al., 2014[Fulem, M., Růžička, K., Červinka, C., Bazyleva, A. & Della Gatta, G. (2014). Fluid Phase Equilib. 371, 93-105.]) and below 4 kJ mol−1 for extremely low-volatile materials (Červinka et al., 2019[Červinka, C., Klajmon, M. & Štejfa, V. (2019). J. Chem. Theory Comput. 15, 5563-5578.]), allowing the identification of computational methods which provide uncertainties of the predicted enthalpic data well within the chemical accuracy threshold.

1.3.3. Geometry optimization

Geometry optimizations by minimizing the lattice energy are generally performed with one of the Broyden-Fletcher-Goldfarb-Shanno quasi-Newton algorithms (Broyden, 1967[Broyden, C. G. (1967). Math. C, 21, 368-381.]; Head & Zerner, 1985[Head, J. D. & Zerner, M. C. (1985). Chem. Phys. Lett. 122, 264-270.]; Liu & Nocedal, 1989[Liu, D. C. & Nocedal, J. (1989). Math. Program. 45, 503-528.]), but these can converge to saddle points or arbitrarily shallow minima, leading to the prediction of crystal structures that cannot be observed experimentally (Price, 2013[Price, S. L. (2013). Acta Cryst. B69, 313-328.]). The FIRE optimization algorithm, implemented in the Atomic Simulation Environment (ASE), passes over stationary points and shallow minima and often finds deeper energy minima faster than quasi-Newton methods (Bitzek et al., 2006[Bitzek, E., Koskinen, P., Gähler, F., Moseler, M. & Gumbsch, P. (2006). Phys. Rev. Lett. 97, 170201.]; Larsen et al., 2017[Hjorth Larsen, A., Jørgen Mortensen, J., Blomqvist, J., Castelli, I. E., Christensen, R., Dułak, M., Friis, J., Groves, M. N., Hammer, B., Hargus, C., Hermes, E. D., Jennings, P. C., Bjerre Jensen, P., Kermode, J., Kitchin, J. R., Leonhard Kolsbjerg, E., Kubal, J., Kaasbjerg, K., Lysgaard, S., Bergmann Maronsson, J., Maxson, T., Olsen, T., Pastewka, L., Peterson, A., Rostgaard, C., Schiøtz, J., Schütt, O., Strange, M., Thygesen, K. S., Vegge, T., Vilhelmsen, L., Walter, M., Zeng, Z. & Jacobsen, K. W. (2017). J. Phys. Condens. Matter, 29, 273002.]). Since the computational cost of geometry optimizations with quantum chemical methods is substantial, it is important to use efficient algorithms. Techniques such as preconditioners (Packwood et al., 2016[Packwood, D., Kermode, J., Mones, L., Bernstein, N., Woolley, J., Gould, N., Ortner, C. & Csányi, G. (2016). J. Chem. Phys. 144, 164109.]) and using internal coordinates (Bučko et al., 2005[Bučko, T., Hafner, J. & Ángyán, J. G. (2005). J. Chem. Phys. 122, 124508.]) may speed up the calculations. Force fields, machine-learned potentials or semi-empirical methods such as DFTB can be employed to pre-optimize structures to drastically speed up geometry optimizations.

1.3.4. Thermal effects

In efforts to improve upon static lattice energies, many groups include effects due to temperature in their most accurate rankings. By free energy calculations, we mean methods that explicitly calculate a thermodynamic ensemble of some kind. That can be an ensemble of microstates from a Monte Carlo or Molecular Dynamics simulation, an ensemble of configurations in a disordered crystal, or an ensemble of phonons from lattice dynamics.

Lattice vibrational contributions to the stability are commonly calculated in the harmonic approximation, leading to a temperature-dependent Helmholtz vibrational free energy (Fultz, 2010[Fultz, B. (2010). Prog. Mater. Sci. 55, 247-352.]; Day et al., 2003[Day, G. M., Price, S. L. & Leslie, M. (2003). J. Phys. Chem. B, 107, 10919-10933.]). For such calculations, it is important to consider phonon dispersion by sampling several k-points in the Brillouin zone, or modelling the dispersion by some other means (Gilat & Alder, 1976[Gilat, G. & Alder, B. J. (1976). Editors. Methods in Computational Physics: Vibrational Properties of Solids of Advances in Research and Applications. Academic Press.]; Nyman et al., 2016[Nyman, J., Pundyke, O. S. & Day, G. M. (2016). Phys. Chem. Chem. Phys. 18, 15828-15837.]; Kamencek et al., 2020[Kamencek, T., Wieser, S., Kojima, H., Bedoya-Martínez, N., Dürholt, J. P., Schmid, R. & Zojer, E. (2020). J. Chem. Theory Comput. 16, 2716-2735.]). The harmonic approximation neglects thermal expansion, and although it is a small contribution, it can be significant for the accurate ranking of CSP structures, or for high accuracy calculations of temperature-dependent properties of molecular crystals (Heit & Beran, 2016[Heit, Y. N. & Beran, G. J. O. (2016). Acta Cryst. B72, 514-529.]; Heit et al., 2016[Heit, Y. N., Nanda, K. D. & Beran, G. J. O. (2016). Chem. Sci. 7, 246-255.]; O'Connor et al., 2022[O'Connor, D., Bier, I., Hsieh, Y.-T. & Marom, N. (2022). J. Chem. Theory Comput. 18, 4456-4471.]).

Thermal expansion can be split into a finite temperature contribution and a zero-temperature contribution due to the atoms' zero-point motion. The latter has been found to amount to 2% on average for the X23 set of molecular crystals (Dolgonos et al., 2019[Dolgonos, G. A., Hoja, J. & Boese, A. D. (2019). Phys. Chem. Chem. Phys. 21, 24333-24344.]). The quasi-harmonic approximation is a convenient way to model anisotropic thermal expansion (Nyman et al., 2016[Nyman, J., Pundyke, O. S. & Day, G. M. (2016). Phys. Chem. Chem. Phys. 18, 15828-15837.]; O'Connor et al., 2022[O'Connor, D., Bier, I., Hsieh, Y.-T. & Marom, N. (2022). J. Chem. Theory Comput. 18, 4456-4471.]), but still fails to capture the true anharmonicity of the atomic vibrations. The latter may be modelled by molecular dynamics (Gray et al., 2004[Gray, A. E., Day, G. M., Leslie, M. & Price, S. L. (2004). Mol. Phys. 102, 1067-1083.]; Rossi et al., 2016[Rossi, M., Gasparotto, P. & Ceriotti, M. (2016). Phys. Rev. Lett. 117, 115702.]).

An alternative way to calculate polymorph free energy differences is the Einstein crystal method (Frenkel & Ladd, 1984[Frenkel, D. & Ladd, A. J. (1984). J. Chem. Phys. 81, 3188-3193.]; Frenkel & Smit, 2001[Frenkel, D. & Smit, B. (2001). Understanding Molecular Simulation: from Algorithms to Applications. Elsevier.]). The method calculates the relative free energy difference between two crystals by thermodynamic integration over the path to a common ideal reference state, for which the free energy can be calculated analytically. This reference state is an Einstein crystal, which consists of a set of non-interacting atoms tethered to their positions by harmonic restraints (Yang et al., 2020[Yang, M., Dybeck, E., Sun, G., Peng, C., Samas, B., Burger, V. M., Zeng, Q., Jin, Y., Bellucci, M. A., Liu, Y., Zhang, P., Ma, J., Jiang, Y. A., Hancock, B. C., Wen, S. & Wood, G. P. F. (2020). Cryst. Growth Des. 20, 5211-5224.]).

1.3.5. Disorder

With improvements in laboratory diffraction hardware and greater access to high energy synchrotron facilities, it is increasingly common to see disorder in experimentally determined structures, and disorder was an important factor in this blind test challenge, as four of the target crystal structures were disordered (XXVII, XXX, XXXI, XXXII).

It is also common to find clusters of similar crystal structures in CSP landscapes, which have the same overall packing except for some minor conformational change (Braun et al., 2019[Braun, D. E., McMahon, J. A., Bhardwaj, R. M., Nyman, J., Neumann, M. A., van de Streek, J. & Reutzel-Edens, S. M. (2019). Cryst. Growth Des. 19, 2947-2962.]). In many cases, such clusters of lattice energy minima correspond to a single disordered structure. In this blind test, two groups (20, 24) correctly predicted the occurrence of disorder in a target crystal structure for the first time.

Configurational disorder gives rise to a small, but possibly significant entropic contribution to the crystal's free energy, which can be calculated in several ways, but perhaps most efficiently with symmetry-adapted ensemble theory (Grau-Crespo & Hamad, 2015[Grau-Crespo, R. & Hamad, S. (2015). In Proceedings of MOL2NET, International Conference on Multidisciplinary Sciences. MDPI. https://dx.doi.org/10.3390/MOL2NET-1-c002.]; Habgood et al., 2011[Habgood, M., Grau-Crespo, R. & Price, S. L. (2011). Phys. Chem. Chem. Phys. 13, 9590.]; Woollam et al., 2018[Woollam, G. R., Neumann, M. A., Wagner, T. & Davey, R. J. (2018). Faraday Discuss. 211, 209-234.]).

2. Motivation, organization and approach

2.1. Motivation

Over the years, the blind tests of CSP have showcased the evolution of CSP techniques, highlighting the increasing accuracy of energy models, the expanding role of DFT calculations, and the need to consider many subtle effects that contribute to the stability ranking. The lessons learned from these blind tests have informed ongoing research efforts and continue to inspire advancements in the field of crystal structure prediction.

Given the importance of structure ranking to the success of a CSP study and the emergence of different methodologies in recent years, it was decided that structure ranking would be benchmarked separately in a controlled exercise, one which would provide a consistent starting point for all ranking methods. This would hopefully give us valuable insights into the current state of crystal structure energy ranking methods, their limitations, and potential directions for further research.

2.2. Organization

The seventh blind test was a two-phase initiative and was coordinated by Lily M. Hunnisett (CCDC). The first phase focused on structure generation methods and the second on structure ranking methods. The choice of this format was heavily influenced by feedback received by the CCDC following the sixth blind test. Running from October 2020 until June 2022, the challenges presented were intended to test methods considered state-of-the-art and, in doing so, provoke innovation and continued development of CSP methods.

The structure ranking phase took place over December 2021 to June 2022 and involved the CCDC providing participants with prepared sets of structures to rank in order of likelihood of observation. The prepared sets contained either 100 or 500 structures (dependent on the target compound, see Table 1[link]); the former to provide a tractable challenge for those with limited resources, and the latter to pose a more realistic ranking exercise than that of previous blind tests. Whilst this allowed a more informative and controlled analysis, it did not mimic how a real-world CSP calculation is performed, where structure ranking is carried out on a larger scale. To participate, it was not a requirement for participants to have taken part in the first phase of the test.

Table 1
Two-dimensional chemical structures of the target compounds investigated, and the number of structures provided to participants for the ranking exercise

Target Chemical diagram Experimental structures Number of structures provided Experimental investigators
XXVII
[Scheme 1]
Form A (known) 100 J. A. Anthony, S. Parkin (F. Tarczynski)
Form B (unknown)
XXVIII
[Scheme 2]
Form A (known) 500 M. R. J. Elsegood, P. F. Kelly, L. Wilkinson (M. R. Probert, J. Weatherston)
XXXI
[Scheme 3]
Form A (known) 100 J. Hone, A. Keates, I. Jones
Form B (known)
XXXII
[Scheme 4]
Form A (known) 500 A. DiPasquale, J. W. Lubach
Form B (known)
Form C (unknown)
Form D (unknown)
Form E (unknown)
Form F (unknown)
Form G (unknown)
Form H (unknown)
XXXIII
[Scheme 5]
Form A (known) 500 S. Coles, S. Aitipamula, J. Cadden
Form B (known)

2.3. Target compounds

The second phase involved ranking the structures of five target compounds that fit under one of two categories: methods development (XXVII and XXVIII) and pharmaceutical or agrochemical applications (XXXI, XXXII, and XXXIII), see Table 1[link], labelled according to the scheme set by previous blind tests (see SI-A Section 6). The methods development category presented systems with diverse chemistry and applications, while the pharmaceutical/agrochemical category aimed to test computational efficiency. A detailed explanation behind the choice of systems and individual descriptions of systems are provided in the preceding report on the first phase of the test. Since targets XXIX and XXX were presented as bespoke challenges – powder X-ray diffraction (PXRD) structure determination and co-crystal stoichiometry prediction – and involved their own ranking exercise (see phase one report), they were not included in this phase.

From extensive experimental investigations (see supplementary information of the first phase of this blind test), the following crystal structures are known: XXVII; one polymorph (Form A, Z′ = 1, [P{\overline 1}]), XXVIII; one polymorph (Form A, Z′ = 0.5, [P{\overline 1}]), XXXI; two polymorphs (Form A, Z′ = 1, P21/c, and Form B, Z′ = 1, P21/c) in addition to a solvate which was not a target for this exercise, XXXII; two polymorphs (Form A, Z′ = 1, [P{\overline 1}], and Form B, Z′ = 2, [P{\overline 1}]), XXXIII; two polymorphs (Form A, Z′ = 1, C2/c, and Form B, Z′ = 1, Pna21).

It was emphasized to all participating groups that not all target compounds needed to be attempted.

2.4. Format of phase two: structure ranking

Similar to the first phase of the test, we invited all those interested in taking part to provide details of their intended ranking method beforehand to, (a) avoid duplication of ranking methods, and (b) ensure ranking methods were novel and/or benchmarked as demonstrated by reports or published research. In December 2021, participants were provided with sets of structures prepared by the CCDC organizers as described below. Participants were required to rank structures in order of likelihood of observation using their own ranking method and to return results within six months. Organizers analysed which structure(s) matched the experimental forms of each target system and the associated rank. For targets XXXI–XXXIII, where relevant experimental data was available (available in the supplementary information of the first phase of this blind test), the accuracy of predicted thermodynamic relationships was also assessed. An in-person meeting was held in September 2022 in Cambridge, UK, to present and discuss the results, challenges, and outlooks of the test.

2.5. Structure set preparation

The structures that were provided to the participants for the ranking exercise were sampled from datasets obtained in the structure generation phase of the seventh blind test. To compare crystal structures, organizers employed the molecular overlay method commonly known as COMPACK (Chisholm & Motherwell, 2005[Chisholm, J. A. & Motherwell, S. (2005). J. Appl. Cryst. 38, 228-231.]), since implemented as Crystal Packing Similarity, available through Mercury 2022.2.1 and the CSD Python API 3.0.15 (Macrae et al., 2020[Macrae, C. F., Sovago, I., Cottrell, S. J., Galek, P. T. A., McCabe, P., Pidcock, E., Platings, M., Shields, G. P., Stevens, J. S., Towler, M. & Wood, P. A. (2020). J. Appl. Cryst. 53, 226-235.]; Groom et al., 2016[Groom, C. R., Bruno, I. J., Lightfoot, M. P. & Ward, S. C. (2016). Acta Cryst. B72, 171-179.]). This method, hereafter referred to as `COMPACK', overlays within given distance and angle tolerances, clusters of molecules taken from each crystal and minimizes the root mean-square distance (RMSD) between atoms, typically omitting hydrogen atoms. The method thus returns the number of molecules that could be overlaid and the RMSD.

The structure sets were first populated with structures of interest: experimental representatives and potential matches to undetermined forms showed by PXRD data. For each target system, the closest matches (lowest RMSD upon COMPACK comparison) predicted from CSP were selected as representatives for the experimentally known form(s). For targets XXVII and XXXII where additional polymorphs were showed by PXRD only and crystal structures had not been conclusively determined, a search by PXRD similarity was carried out across all CSP structures using the PXRD similarity measure by de Gelder et al. (2001[Gelder, R. de, Wehrens, R. & Hageman, J. A. (2001). J. Comput. Chem. 22, 273-289.]), as implemented in the CSD Python API. The structures with the largest similarity score were included in the set (15 structures for XXVII, and 53 structures for XXXII), see SI-A Tables 20 and 21. Each of the selected structures was then optimized under constraints using a CSD knowledge-based force field, see item (4) in the protocol below. A structural comparison using COMPACK was carried out to ensure each experimental representative structure matched the original experimentally determined crystal structure.

Experimental forms of targets XXVII, XXXI and XXXII contained disorder. For this ranking exercise, the major and minor components of disorder were included as separate structures for XXXI, which exhibited disorder of the fluorinated ring combining two components of 0.6:0.4 occupancy. The structures included in the list were generated by CSP in the first phase of this blind test. However, separate components were not included for XXVII and XXXII; the disorder of XXVII was not known until after the test, see Section 5.1[link]. It was decided by the organizers not to include the minor component of XXXII Form A since it was not generated by CSP in the first phase, so would require manual input to recreate the disorder, a rotation of a terminal difluoromethyl group, which could unintentionally have provided a clue to the experimental crystal structure.

The sampling process for the remainder of the structure sets followed the below protocol for each target molecule:

(1) All predicted structures from each group were combined into a single global dataset.

(2) The global dataset was clustered with COMPACK to form groups of similar structures using a leaders clustering approach (Spath, 1980[Spath, H. (1980). Cluster Analysis Algorithms for Data Reduction and Classification of Objects. Chichester: Ellis Horwood.]) and sorted in order of cluster size.

(3) An initial sample of 2000 − n structures (where n is the number of structures of interest already selected as described above) were selected to include in the set: clusters with the largest number of common structures identified in the previous step of which a single structure was selected at random. These represented the most frequently predicted structures across all groups.

(4) The sampled structures were optimized under constraints using a CSD knowledge-based force field (Cole et al., 2016[Cole, J. C., Groom, C. R., Korb, O., McCabe, P. & Shields, G. P. (2016). J. Chem. Inf. Model. 56, 652-661.]), and fitness score and density were calculated. Unit-cell parameters, global molecular rotation and translation, and internal atomic torsional rotations were optimized in this step, constraining each parameter from changing by more than 3% of its start value. This allowed slight perturbation of the original structural geometry to avoid easy identification by a group of a structure generated by their own method, while preventing a significant structural change that could push the structure out of the local potential energy basin.

(5) The final sample (containing 100 or 500 structures depending on the target compound) were selected according to the calculated fitness score of the CSD-based force field. Of the overall energy range (indicated by fitness score), 50% of structures represented the lowest third, and 25% of structures in each of the remaining thirds. The structures were iterated through in order of frequency of observation across the CSP landscapes when populating the sample until the desired energy ranges were sufficiently populated.

(6) A full optimization (no constraints) using the CSD force field was carried out on each sampled structure and compared against its starting structure to ensure the structure had not deviated from the corresponding energy well.

(7) To ensure anonymity of participants, atom labels and the order of the atoms in each CIF file were standardized using the CSD Python API. Each structure was also named and numbered in a consistent way.

3. Computational methods used in this blind test

3.1. Categorization of computational methods

In total, 28 groups participated in at least one part of this blind test. Of those, 22 took part in the structure ranking phase of the test. An overview of the methods applied by the various groups is given in Table 2[link].

Table 2
Summary of the structure ranking methods utilized by each participating group

* indicates the principal investigator of each group.

Class Subclass Group Group members Ranking method Free energy
A. Periodic DFT-D methods A1. GGA density functionals 4 Červinka*, Kostková, Ludík, Touš PBE-D3/PAW electronic energy Harmonic DFTB3-D phonons
5 Day*, Arnold, Bramley, Butler, Taylor PBE+GD3BJ
22 Oganov*, Maryewski, Momenzadeh Abardeh, Bahrami, Salimi PBE-D3
A2. Beyond GGA functionals 9 Hušák* rSCAN+MBD
10 Jin*, Yang, L. Tan, Chang, Sun, X. Shi, C. Liu, Yue, Fu, Lin, Y. Zhou, Z. Liu, Zeng, Li, B. Shi, T. Zhou, Greenwell, Bellucci, Sekharan XXVII: optPBE-vdW, XXVIII: r2SCAN-D4, XXXI-XXXIII: PBE0-MBD Einstein crystals
11 Johnson*, Otero-de-la-Roza*, Clarke, Rumson, Mayo, A. J. A. Price B86bPBE-XDM/NAO optimization; 25% and 50% hybrid single points  
14 Klimeš* RPA(SCAN) on optPBE-vdW structures
A3. DFT-D with monomer or multimer corrections 2 Beran*, Cook, Unzueta B86bPBE-XDM + monomer energy corrections
3 Boese*, List, Strasser, Hoja, Braun PBE0+MBD multimers embedded into periodic PBE+MBD Harmonic PBE-MBD or PBE0-MBD:PBE-MBD
20 Neumann*, Anelli, Woollam, Abraham, Dietrich, Firaha, Helfferich, Y. M. Liu, Mattei, Sasikumar, Tkatchenko, van de Streek Cascade of DFT methods of increasing accuracy PBE(0)+MBD+MP2D+Fvib
 
B. Intra (ΔEintra) and inter (Uinter) molecular contributions to lattice energy B1. Electronic structure calculations on multimers 19 Muddana*, Jain, Darden, Skillman Atomic multipole force field, IEFF / HF-3c + DFT
  21 Obata*, Goto*, Utsumi, Ikabata, Okuwaki, Fukuzawa, Nakayama, Yonemochi XXVII: PBE-D3; XXXI, XXXII: FMO-MP2/6-31G†; XXXIII: FMO-MP2/6-31G(d)
B2. Force fields fitted to SAPT calculations 26 Szalewicz*, Ishaque, Nikhar, Podeszwa, Rogal, Vogt-Maranto XXXI: SAPT(DFT) fitted potentials (intermolecular), modified GAFF (intramolecular), flexible-monomer minimizations and simulations, only GAFF monomer energies used XXXI: MD simulations in NPT ensemble
27 Szalewicz*, Tuckerman*, Bhardwaj, Chan, Hong, Ishaque, Jing, Melkumov, Nikhar, Podeszwa, Rehman, Rogal, Song, Vogt-Maranto SAPT(DFT) or PBE0-D3 fitted potentials (intermolecular), PBE0-D3 monomer deformation energy penalties, modified GAFF (intramolecular) in flexible-monomer simulations for XXXIII XXXIII: MD simulations in NPT ensemble
B3. Electronic structure calculations on individual molecules 6 van Eijck* Price-Williams exp-6 potential; RHF/6-31G(d,p) point charges and intramolecular energies
18 Mohamed*, Dhokale, Saeed, Alkhidir, Almehairbi Atomic multipoles and exp-6
24 S. L. Price*, L. S. Price, Guo Molecular Ψmol model, atomic multipoles + empirical exp-6 for intermolecular, Ψ for intramolecular Rigid-body harmonic phonons
B4. General purpose force field models 17 Matsui*, Shinohara Dreiding force field Quasi-harmonic
 
C. Alternative approaches C1. Machine learned models 12 Jose*, Ramteke Cardinality and Gaussian process regression potential
  15 Lončarić*, Bianco, Mladineo, Parunov ANI-2x retrained on r2SCAN single point energies and forces on CCDC structures Anharmonic by SSCHA
  16 Marom*, Isayev*, Anstine, Deng, Nayal, O'Connor, Tang, Yang, Zubatyuk System-specific AIMNet machine learned potentials trained on PBE-D4/def2-TZVPP calculations on N-mers, up to trimers Quasi-harmonic
C2. Ranking not based on thermodynamics 7 Tuckerman*, Galanakis Topological scoring function

The seventh blind test ranking exercise introduces several new energy-based approaches, particularly dimer or multimer electronic structure calculations and machine learned potentials, alongside developments of the methods used in previous blind tests and preliminary results with a novel method not based on thermodynamics. The computational methods applied in this exercise can be coarsely divided into three main categories based on the level of theory primarily applied. Category A: periodic DFT-D methods (Groups 2, 3, 4, 5, 9, 10, 11, 12, 14, 20, 22); B: methods based on dividing the crystal into molecules (6, 18, 19, 21, 24, 26 and 27); C: Any other method (Groups 7, 12, 15, 16 and 17). These can be further sorted into nine subcategories, which will now be described.

3.2. A. Periodic DFT-D methods

3.2.1. GGA density functionals

Generalized gradient approximation functionals were employed by Groups 4, 5 and 22.

Group 4 applied the D3BJ (Grimme et al., 2011[Grimme, S., Ehrlich, S. & Goerigk, L. (2011). J. Comput. Chem. 32, 1456-1465.]) dispersion-corrected PBE functional with the projector augmented wave (PAW) method (Kresse & Joubert, 1999[Kresse, G. & Joubert, D. (1999). Phys. Rev. B, 59, 1758-1775.]), with a 500 eV plane wave kinetic energy cutoff and sampling only the Γ-point of the Brillouin zone. Group 4 additionally calculated a lattice-vibrational contribution to the free energy for the lowest energy structures with harmonic phonons by D4 dispersion-corrected third-order self-consistent DFTB theory with 3ob-3-1 parameterization (Červinka et al., 2016[Červinka, C., Fulem, M., Stoffel, R. P. & Dronskowski, R. (2016). J. Phys. Chem. A, 120, 2022-2034.]; Gaus et al., 2013[Gaus, M., Goez, A. & Elstner, M. (2013). J. Chem. Theory Comput. 9, 338-354.]).

Group 5 applied a three-stage approach, optimizing structures with plane wave PBE-D3BJ while (i) fixing the unit cell (500 eV cutoff), (ii) relaxing the cell (500 eV cutoff), (3) relaxing the cell with tighter convergence criteria and a larger 600 eV cutoff.

Group 22 first assessed the performance of a few different DFT-D methods (PBE-D3, PBE-MBD, PBE0-MBD) against results from a synthon approach (Sarma & Desiraju, 2002[Sarma, J. & Desiraju, G. R. (2002). Cryst. Growth Des. 2, 93-100.]; Abardeh et al., 2022[Momenzadeh Abardeh, Z., Salimi, A. & Oganov, A. R. (2022). CrystEngComm, 24, 6066-6075.]), before choosing to optimize all structures at the PBE-D3 level of theory (Grimme et al., 2010[Grimme, S., Antony, J., Ehrlich, S. & Krieg, H. (2010). J. Chem. Phys. 132, 154104.]). The PBE-D3 ranking was considered the more reliable because its low-energy structures more often contained synthons detected by a CSD search in the experimental crystal structures of similar compounds.

3.2.2. Beyond GGA functionals

Ranking methods from a further four groups (9, 10, 11 and 14) primarily applied functionals considered to be above GGA functionals in the `Jacob's ladder' of density functional approximations (Perdew & Schmidt, 2001[Perdew, J. P. & Schmidt, K. (2001). AIP Conf. Proc. 577, 1-20.]).

Group 9 fully optimized all structures using the rSCAN metaGGA functional and MBD dispersion correction and on-the-fly generated ultrasoft pseudopotentials, with kinetic energy cutoff values dependent on the system.

Group 10 performed a hierarchical ranking process where lattice energies for all structures were first calculated with the non-local dispersion-inclusive optPBE-vdW density functional (Klimeš et al., 2009[Klimeš, J., Bowler, D. R. & Michaelides, A. (2009). J. Phys. Condens. Matter, 22, 022201.]), then re-evaluated using a level of theory chosen by a decision tree to best match MP2 energies (Abramov et al., 2021[Abramov, Y. A., Li, B., Chang, C., Zeng, Q., Sun, G. & Gobbo, G. (2021). Cryst. Growth Des. 21, 5496-5502.]). For molecule XXVII the final lattice energies were calculated with optPBE-vdW, for molecule XXVIII, the r2SCAN-D4 metaGGA functional was used, and for molecules XXXI–XXXIII PBE0-MBD was used.

Free energies for all targets except XXVIII were calculated by applying a modified version of the Einstein crystal method with a pseudo-supercritical path approach (Frenkel & Ladd, 1984[Frenkel, D. & Ladd, A. J. (1984). J. Chem. Phys. 81, 3188-3193.]; Eike et al., 2005[Eike, D. M., Brennecke, J. F. & Maginn, E. J. (2005). J. Chem. Phys. 122, 014115.]). For molecule XXVIII, harmonic phonon frequencies were calculated with third-order self-consistent charge DFTB in the DFTB+ program (Hourahine et al., 2020[Hourahine, B., Aradi, B., Blum, V., Bonafé, F., Buccheri, A., Camacho, C., Cevallos, C., Deshaye, M. Y., Dumitrică, T., Dominguez, A., Ehlert, S., Elstner, M., van der Heide, T., Hermann, J., Irle, S., Kranz, J. J., Köhler, C., Kowalczyk, T., Kubař, T., Lee, I. S., Lutsker, V., Maurer, R. J., Min, S. K., Mitchell, I., Negre, C., Niehaus, T. A., Niklasson, A. M. N., Page, A. J., Pecchia, A., Penazzi, G., Persson, M. P., Řezáč, J., Sánchez, C. G., Sternberg, M., Stöhr, M., Stuckenberg, F., Tkatchenko, A., Yu, V. W.-Z. & Frauenheim, T. (2020). J. Chem. Phys. 152, 124101.]), using custom Slater–Koster parameters for Cu together with the 3ob-3-1 set.

Group 11 first carried out geometry optimizations with the GGA functional B86bPBE and XDM dispersion in two steps of increasing strictness of relaxation convergence. Final rankings were based on subsequent single point energies calculated with a hybrid functional that combines B86bPBE-XDM with either 25% or 50% Hartree–Fock exchange (Otero-de-la-Roza et al., 2019[Otero-de-la-Roza, A., LeBlanc, L. M. & Johnson, E. R. (2019). J. Chem. Theory Comput. 15, 4933-4944.]; Price et al., 2023a[Price, A. J., Otero-de-la-Roza, A. & Johnson, E. R. (2023a). Chem. Sci. 14, 1252-1262.]).

Group 14 initially optimized structures using the optPBE-vdW level of theory, first fully optimizing all structures setting a criterion for the largest force on atoms to 0.02 eV Å−1, then optimizing atomic positions only with a force cut-off of 0.001 eV Å−1. Final energies were calculated using the Random Phase Approximation (RPA) and SCAN functional, applying PAW potentials.

3.2.3. Periodic DFT-D with monomer or multimer corrections

In addition to applying DFT-D methods, Groups 2, 3 and 20 applied energy corrections based on monomer or multimer energies.

Group 2 primarily used the B86bPBE-XDM GGA functional with PAW potentials. Final energies incorporated a conformational energy correction: the energy difference between DFT and a higher level of theory, either domain-based local pair-natural orbital (DLPNO) coupled cluster theory (CCSD) or spin-component-scaled dispersion-corrected second-order Møller–Plesset perturbation theory (SCS-MP2D) (Greenwell et al., 2022[Greenwell, C., Řezáč, J. & Beran, G. J. (2022). Phys. Chem. Chem. Phys. 24, 3695-3712.]), calculated using gas phase calculations on each monomer in the unit cell. Specifically for molecule XXXII, pre-optimizations were also carried out using dispersion- and basis set superposition error-corrected Hartree–Fock calculations in a minimal basis set, the so-called HF-3c method (Brandenburg & Grimme, 2014b[Brandenburg, J. G. & Grimme, S. (2014b). In Prediction and Calculation of Crystal Structures, edited by S. Atahan-Evrenk & A. Aspuru-Guzik, vol. 345 of Topics in Current Chemistry, pp. 1-23. Springer International Publishing.]).

Group 3 applied a multi-step approach, applying geometry optimizations at the PBE-MBD level of theory followed by optimizations that embedded multimers with PBE0-MBD into PBE-MBD (PBE0-MBD:PBE-MBD) (Loboda et al., 2018[Loboda, O. A., Dolgonos, G. A. & Boese, A. D. (2018). J. Chem. Phys. 149, 124104.]; Hoja et al., 2023[Hoja, J., List, A. & Boese, A. D. (2024). J. Chem. Theory Comput. 20, 357-367.]); a subtractive multimer embedding scheme where PBE-MBD periodic calculations are first carried out, then monomer and dimer energies are replaced with values from PBE0-MBD calculations. Final rankings were based on energies calculated at this level of theory with tight convergence criteria. Additionally, free energies were calculated from harmonic phonons at either the PBE-MBD (XXVII, XXVIII, XXXII) or PBE0-MBD:PBE-MBD (XXXI, XXXIII) level of theory.

Group 20 applied a multi-step approach of increasing level of theory. Tailor-made force fields and machine learned algorithms were applied to filter out high-energy structures prior to the most computationally demanding calculations. The final energies are free energies at room temperature as calculated with the TRHu(ST) method (Firaha et al., 2023[Firaha, D., Liu, Y. M., van de Streek, J., Sasikumar, K., Dietrich, H., Helfferich, J., Aerts, L., Braun, D. E., Broo, A., DiPasquale, A. G., Lee, A. Y., Le Meur, S., Nilsson Lill, S. O., Lunsmann, W. J., Mattei, A., Muglia, P., Putra, O. D., Raoui, M., Reutzel-Edens, S. M., Rome, S., Sheikh, A. Y., Tkatchenko, A., Woollam, G. R. & Neumann, M. A. (2023). Nature, 623, 324-328.]), with the exceptions that for compounds XXVII and XXVIII no monomer MP2D correction was added and for XXVIII, ab initio minimizations and phonon calculations were done with periodic PBE+MBD.

3.3. B. Mixed intra- and intermolecular models

Periodic DFT methods are fairly accurate in predicting relative energies, but they come with significant computational costs. On the other hand, force fields can be useful due to their speed but they may not be optimal for CSP applications where relative energies need to be calculated very accurately, with errors on the order of 1 kJ mol−1. In between these two methods, there are a series of approaches that limit ab initio quantum mechanical calculations to a certain subgroup of the crystal, such as dimers, single molecules or fragments. The resulting lattice energy is made up of two main components. The dominant contribution to the lattice energy, the intermolecular energy (Uinter) is modelled by summing up the interactions within the crystals, as obtained from electronic structure calculations on the multimers, or by atomistic calculations using analytical anisotropic force fields, which are parameterized from electronic structure calculations on molecules or dimers, or by empirical fitting.

3.3.1. Electronic structure calculations on multimers

After an initial step of conformer and crystal geometry optimization, using atoms in the asymmetric unit as a reference, a molecular cluster of finite size defined by a distance cutoff is created. The approaches adopted by Groups 19 and 21, the dimer expansion and the Fragment Molecular Orbital (FMO) method (Kitaura et al., 1999[Kitaura, K., Ikeo, E., Asada, T., Nakano, T. & Uebayasi, M. (1999). Chem. Phys. Lett. 313, 701-706.]), calculate the intermolecular term of the cluster as a sum of the energies of dimers and inter-fragments, respectively. Single-point calculations are performed for pairs of the reference and any other molecule or fragment within the cluster. In the case of FMO method, the calculations are performed in the presence of environmental electrostatic potential to take into account contributions from other fragments (Nakano et al., 2002[Nakano, T., Kaminuma, T., Sato, T., Fukuzawa, K., Akiyama, Y., Uebayasi, M. & Kitaura, K. (2002). Chem. Phys. Lett. 351, 475-480.]). The dimer energy and FMO calculations can be performed in parallel, taking advantage of modern high-performance computers.

Group 19 calculates dimer energies with B3LYP-D3BJ for dimers at less than 6 Å from each other and HF-3c for those up to 12 Å distance.

Group 21 calculates inter-fragment interaction energies with FMO at MP2 level of theory (FMO-MP2) (Mochizuki et al., 2004a[Mochizuki, Y., Koikegami, S., Nakano, T., Amari, S. & Kitaura, K. (2004a). Chem. Phys. Lett. 396, 473-479.], Mochizuki et al., 2004b[Mochizuki, Y., Nakano, T., Koikegami, S., Tanimori, S., Abe, Y., Nagashima, U. & Kitaura, K. (2004b). Theor. Chem. Acc. 112, 442-452.]) using molecular clusters within a radius of 12 Å.

3.3.2. Force fields fitted to quantum chemical calculations

Improvements in structures' energy evaluations are reached when a force field is parameterized specifically for the target compound instead of adopting common transferable force fields. In this blind test, Groups 26 and 27 used either symmetry-adapted perturbation theory based on DFT description of monomers, SAPT(DFT) (Misquitta et al., 2005[Misquitta, A. J., Podeszwa, R., Jeziorski, B. & Szalewicz, K. (2005). J. Chem. Phys. 123, 214103.]), or supermolecular DFT-D in parameterizing ab initio-based intermolecular force fields (Nikhar & Szalewicz, 2022[Nikhar, R. & Szalewicz, K. (2022). Nat. Commun. 13, 3095.]). Thousands of dimer configurations were generated to evaluate intermolecular interaction energies and fit a system-specific intermolecular potential using the autoPES codes (Metz et al., 2016[Metz, M. P., Piszczatowski, K. & Szalewicz, K. (2016). J. Chem. Theory Comput. 12, 5895-5919.]). In most cases, the intramolecular term of the lattice energy was determined by the DFT-D energy difference relative to the most stable conformer. For target XXXI, GAFF monomer deformation penalties were used. Monomers in CCDC-provided lists of polymorphs were constraint-optimized, with the soft dihedral angles determining the shape of molecule fixed at their original values. All structures were optimized using either rigid monomers or flexible monomers with modified GAFF intramonomer energies.

The parameterization of intermolecular force fields was based on supermolecular PBE0-D3BJ for compounds XXVII, XXVIII, and XXXII and on SAPT(DFT) for compounds XXXI and XXXIII. An additional step of molecular dynamics simulations was performed for targets XXXI and XXXIII to assess structures' stability at finite temperatures. For these, the intramolecular term was represented by reparameterized GAFF force field. For XXXI, the equilibrium bond lengths and angles were replaced by the ab initio values from the equilibrium conformer. For XXXIII, in addition the force constants and torsional parameters were fitted to ab initio calculations on a grid of 2000[Momenzadeh Abardeh, Z., Salimi, A. & Oganov, A. R. (2022). CrystEngComm, 24, 6066-6075.] monomer conformations.

3.3.3. Electronic structure calculation on individual molecules

In the Ψmol method (Price et al., 2010[Price, S. L., Leslie, M., Welch, G. W. A., Habgood, M., Price, L. S., Karamertzanis, P. G. & Day, G. M. (2010). Phys. Chem. Chem. Phys. 12, 8478.]) used by Groups 6, 18 and 24, ab initio calculations at the molecular level are used to both estimate the energy penalty of the different conformers, ΔEintra, and model the electrostatic term of Uinter. The charge density of each conformer is used to calculate either atomic point charges from the RHF/6-31 G(d,p) charge density (Group 6) or more sophisticated distributed multipoles from B97/6-31G(d,p) or PBE0/6-31G(d,p) charge densities (Groups 18 and 24, respectively). Uinter is completed with an empirical repulsion-dispersion potential, often based on the FIT parameterization (Coombes et al., 1996[Coombes, D. S., Price, S. L., Willock, D. J. & Leslie, M. (1996). J. Phys. Chem. 100, 7352-7360.]).

Group 6 used RHF/6-31G(d,p) calculations to fit atomic point charges while Groups 18 and 24 generated multipoles starting from B97D/6-31G(d,p) and PBE0/6-31G(d,p), respectively.

3.3.4. General purpose force field models

Group 17 was alone in using a general purpose atom-atom potential, namely the Dreiding force field (Mayo et al., 1990[Mayo, S. L., Olafson, B. D. & Goddard, W. A. (1990). J. Phys. Chem. 94, 8897-8909.]) and atomic point charges derived by electrostatic fitting to B3LYP/6-311G(d,p) charge densities. With this energy model, they performed quasi-harmonic approximation lattice dynamics with GULP (Gale & Rohl, 2003[Gale, J. D. & Rohl, A. L. (2003). Mol. Simul. 29, 291-341.]) to obtain the free energies used for ranking the structures of molecule XXVII.

3.4. C. Alternative approaches

3.4.1. Machine learned models

Machine learning methods were used by several groups and this constitutes a major development in the field of CSP. While Group 12 used a relatively simple machine learning method, a Gaussian process regression (GPR) (Deringer et al., 2021[Deringer, V. L., Bartók, A. P., Bernstein, N., Wilkins, D. M., Ceriotti, M. & Csányi, G. (2021). Chem. Rev. 121, 10073-10141.]) trained on DFTB results, more advanced methods for ranking the structures with machine learning were used by Groups 15 and 16. Group 15 used transfer learning to enhance the ANI-2x model. The training data consisted of single point r2SCAN calculations on the sets of crystal structures provided by the CCDC. The ANI-2x neural network potential was retrained with torchANI on the new data, using both energies and atomic forces (Devereux et al., 2020[Devereux, C., Smith, J. S., Huddleston, K. K., Barros, K., Zubatyuk, R., Isayev, O. & Roitberg, A. E. (2020). J. Chem. Theory Comput. 16, 4192-4202.]; Gao et al., 2020[Gao, X., Ramezanghorbani, F., Isayev, O., Smith, J. S. & Roitberg, A. E. (2020). J. Chem. Inf. Model. 60, 3408-3415.]). Structure optimization and fully anharmonic vibrational free energy calculations were then performed with the stochastic self-consistent harmonic approximation (SSCHA) using the SSCHA software (Monacelli et al., 2021[Monacelli, L., Bianco, R., Cherubini, M., Calandra, M., Errea, I. & Mauri, F. (2021). J. Phys. Condens. Matter, 33, 363001.]).

Group 16 performed unit cell relaxations and calculated lattice energies and quasi-harmonic lattice vibrational free energies with system-specific AIMNet neural network potentials trained to each blind test target compound (Zubatyuk et al., 2021[Zubatyuk, R., Smith, J. S., Nebgen, B. T., Tretiak, S. & Isayev, O. (2021). Nat. Commun. 12, 4870.]; Anstine et al., 2023[Anstine, D., Zubatyuk, R. & Isayev, O. (2023). ChemRxiv, 10.26434/chemrxiv-2023-296ch.]). Training data for the target specific AIMNet models were based on molecular clusters extracted from the crystal structures, which contained the reference molecule and up to ten of its neighbours. Additional sampling of out-of-equilibrium configurations was performed by running short MD trajectories on the molecular clusters. To accelerate convergence, the models were pre-trained using GFN-xTB (Bannwarth et al., 2019[Bannwarth, C., Ehlert, S. & Grimme, S. (2019). J. Chem. Theory Comput. 15, 1652-1671.]). Subsequently, transfer learning was performed to DFT data for smaller clusters containing up to three molecules, calculated using PBE-D4/def2-TZVPP. When applied to crystal structures, the AIMNet model accounts for long-range interaction with Ewald sum approximation (Ewald, 1921[Ewald, P. P. (1921). Ann. Phys. 369, 253-287.]) to the Coulomb energy of the crystal, and pairwise C6 and C8 dispersion energy terms. The many-body dispersion terms were calculated using the Axilrod–Teller–Muto formula with DFT-D4 software (Caldeweyher et al., 2017[Caldeweyher, E., Bannwarth, C. & Grimme, S. (2017). J. Chem. Phys. 147, 034112.]). Additional details and analyses are available in SI-B Section 1[link]3.

3.4.2. Non-energy methods

A newly developed non-energy based method was applied by Group 7, ranking crystal structures of molecule XXXI based on a topological analysis approach. For each structure a number of vectors are calculated, including the molecular inertial eigenvectors, ring plane normal vectors, vectors based on the positions of atoms with substantial Gasteiger partial charges (Gasteiger & Marsili, 1978[Gasteiger, J. & Marsili, M. (1978). Tetrahedron Lett. 19, 3181-3184.]) (|q| > 0.1 e), and between atoms forming close contacts. The scoring function is based on observed correlations in the angles between these vectors and the crystal's Miller planes (Tuckerman & Galanakis, 2023[Tuckerman, M. & Galanakis, N. (2023).Topological Crystal Structure Prediction, https://doi.org/10.21203/rs.3.rs-3361974/v1.]).

4. Assessment of results

Geometry-optimized crystal structures submitted by the participants were compared against experimental data using COMPACK. A 30-molecule cluster was applied with distance and angle tolerances of 25% and 25°, respectively, to compare the predictions against the experimental structures. In comparison with the structure generation phase of the blind test (which applied tolerances of 35% and 35°), stricter tolerances were set since the more accurate methods used in this phase can be assumed to yield structures that closely match the experimental reference structures. Tolerances were, however, looser than those used in previous blind tests since it has been shown that matching structures may exhibit a large degree of structural difference (Sacchi et al., 2020[Sacchi, P., Lusi, M., Cruz-Cabeza, A. J., Nauha, E. & Bernstein, J. (2020). CrystEngComm, 22, 7170-7185.]; Mayo & Johnson, 2021[Mayo, R. A. & Johnson, E. R. (2021). CrystEngComm, 23, 7118-7131.]; Mayo et al., 2022[Mayo, R. A., Otero-de-la-Roza, A. & Johnson, E. R. (2022). CrystEngComm, 24, 8326-8338.]). Predicted structures demonstrating a 30 out of 30 molecule match and RMSD < 1 Å were visualized using Mercury to confirm each match.

5. Results and discussion

Here, we discuss and compare several energy ranking methods, including force fields, density functional theory (DFT), tight binding approximations, and machine learning techniques. The performance of these methods in the context of the seventh blind test are assessed, highlighting factors that contribute to their ability to predict experimentally observed polymorphs and facilitate the rational design of materials with desired properties and functions.

Solid-form screening has been carried out for all target systems to increase the likelihood of the thermodynamically stable forms being present among the target structures for this blind test. It is important to note that such a screening does not constitute a guarantee for the observation of the stable forms. If it was otherwise, late-appearing forms and disappearing polymorphs would not be a substantial risk pharmaceutical companies are confronted with in late development. Indeed, it has been reported that for 15–45% of the pharmaceutical compounds in late development, i.e. after a significant amount of experimental screening, the stable form has not been discovered yet (Neumann & van de Streek, 2018[Neumann, M. & van de Streek, J. (2018). Faraday Discuss. 211, 441-458.]). For target systems with multiple experimentally observed forms (XXXI, XXXII and XXXIII have two structurally characterized polymorphs each), we have analysed whether the predicted relative stabilities qualitatively reflected those observed experimentally.

Overall, methods employing periodic dispersion-corrected DFT – PBE0 with MBD or D3; B86bPBE-XDM; rSCAN-MBD – were found to most consistently rank the experimental forms amongst the lowest in energy. Mixed results were found for machine learning methods; System-specific AIMNet machine learned potentials (Zubatyuk et al., 2021[Zubatyuk, R., Smith, J. S., Nebgen, B. T., Tretiak, S. & Isayev, O. (2021). Nat. Commun. 12, 4870.]) performed consistently well, while other ML methods, including a Gaussian process regression and free energies calculated with ML interatomic potentials performed inconsistently, as would be expected from the extent to which the specific molecules' intermolecular interactions can be approximated as molecularly pairwise additive and the extent to which the different approaches accounted for the non-pairwise additivity. Methods that consisted of a combination of an intermolecular force field and intramolecular quantum-chemical components also performed inconsistently. In agreement with previous blind tests, general purpose force fields and a non-energy-based method did not perform well.

Comparisons of the submitted structures versus the CCDC-provided set were carried out by the organizers using the pointwise distance distribution approach (Widdowson et al., 2021[Widdowson, D., Mosca, M. M., Pulido, A., Cooper, A. I. & Kurlin, V. (2022). Match, 87, 529-559.]), see Table 10 in SI-A. Structural differences indicated that all groups except 7 and 12 optimized the structures using their own methods. A few groups directly optimized the structures provided by CCDC with the model used for the final energy evaluation, see Table 2[link]. Step 4 of the structure set preparation (Section 2.5[link]) produced some molecules with higher conformational energies than would have been sampled in many CSP workflows. This meant that many methods that divided the crystal into molecules had to first adapt the conformations as a novel step. This resulted in multiple cases where geometry optimizations led away from the experimentally observed structure. The use of periodic DFT-D always maintained the structure.

In predicting the thermodynamic stability relationship of polymorphs, the overall accuracy of CSP structure ranking methods was mixed. However, in many cases where the incorrect stability order was predicted, the experimental energy difference between polymorphs is likely to be small as was strongly indicated from calculated relative lattice energies.

The greatest consistency was observed for target compound XXXIII, with the majority of methods correctly predicting the stability relationship between the two experimental forms. Additionally, periodic DFT-D methods proved to be the most accurate in this case, with the majority of groups predicting the most stable form as the global minimum in the crystal energy landscape.

Results are reported below per target system. The raw data for each submission is available in SI-C.

5.1. XXVII

There exist two experimentally determined crystal structures for one form of XXVII [Form A, CSD: XIFZOF (290 K), XIFXOF01 (100 K)]. During structure set preparation for molecule XXVII, a large amount of void space was found in structures generated by some groups. After analysis of overall void space, see SI-A Table 8, it was decided to exclude structures from Groups 12 and 17 in the sampling process due to many structures having unreasonably low densities.

The compilation of 100 structures provided to all participant groups was seeded with a CSP structure representative of the experimentally determined crystal structure of Form A at 90 K. Due to a limit on the number of comparisons due to topological symmetry in the CCDC implementation of COMPACK, the closest match to Form A (90 K) was originally incorrectly identified (analysis from Oct 2021), meaning the selected representative structure of the experimental form was a structural variant in terms of the isopropyl group conformation. The CCDC implementation has since been updated.

Based on discussions between organizers and participants, it was agreed for CCDC organizers to investigate the nature of disorder further using molecular dynamics and metadynamics simulations to determine whether an ensemble of varying triisopropylsilane (TIPS) group conformations needed to be considered in the final analysis. The subsequent work suggested that there is dynamic disorder related to the rotation of the isopropyl groups with respect to the pentacene and a possible static disorder related to the change in conformation of the two TIPS groups (see the supplementary information of phase one of this blind test). The results were therefore analysed based on the molecular `core' only (excluding the triisopropyl groups).

Analysis of the set of 100 structures prepared and provided by the CCDC indicates that the structure of the experimental Form A (90 K) was not present in the list, but four structures (28, 38, 59 and 61 matching with 0.53, 0.80, 0.83 and 0.57 Å RMSD30, respectively) exhibit the same crystal packing with varied isopropyl conformations (matches identified upon comparisons excluding isopropyl groups). Molecular overlays were carried out (allowing inversion) comparing all four of the `core'-matching structures. This demonstrated that structures 28 and 61 represent the same structure (also confirmed with COMPACK), whilst structures 38 and 59 represent different structural variants due to differing isopropyl conformations, see Fig. 1[link].

[Figure 1]
Figure 1
Structure overlay of structures 28, 38, and 59 from the provided structure set for target XXVII, demonstrating the variation in TIPS conformation.

Analysis of the ranks and calculated relative energies of these structural variants submitted by each group demonstrates that, despite exhibiting common core crystal packing, the difference in isopropyl conformation translates to a large variation in energy, see Fig. 2[link]. Of the 15 participating methods, 11 (Groups 3, 5, 6, 10, 11, 16, 20, 21, 22, 24, 27) ranked structure 28 as lowest in energy amongst the four conformational variants, while all (except Group 6) ranked structure 38 as the highest in energy, see SI-A Table 11. The lowest in energy of the four `core'-matching structures was ranked as the global minimum by methods from four groups (5, 9, 16, 24) at 0 K, and three groups (3, 16, 24) at room temperature, see Table 3[link] and SI-A Table 11. Since the experimental structure can be regarded as a dynamic ensemble rather than a single point, the previous statement does not discredit methods that have not ranked any of the `core'-matching structures as the global minimum. Group 20 demonstrated from post hoc calculations (see SI-B Section 17) that an alternative isopropyl conformation ranked at the global minimum if taken into account (corresponding to one of the experimental structure determinations), as could be the case for other groups if similar post-analysis were carried out.

Table 3
Summary of results for target systems XXVII, XXVIII, XXXI, XXXII, and XXXIII, where numbers are the predicted rank at 0 K and those in brackets at ambient temperature

`–' indicates no structure was found to match the experimental form.

    XXVII XXVIII XXXI XXXII XXXIII
Class Group Form A Form A Form Amaj Form Amin Form B Form A Form B Form A Form B
A1 4     6 [6] 3 [2] 9 [7] 64 [30] 76 [147] 50 [60] 28 [30]
  5 1   2 3 6 9 24 4 1
A2 22 2 3 2 5 10 21 30 3 1
  9 1   5 7 8     9 1
  10 4 [3] 1 [1] 8 [7] 11 [8] 1 [3] 13 [5] 30 [51] 7 [4] 1 [1]
  11 2 1 8 12 13 31 37 5 1
  14     6 10 11        
A3 2     7 9 17 27 30    
  3 7 [1] 1 [1] 3 [3] 10 [5] 6 [1] 18 [24] 22 [25] 6 [4] 1 [1]
  20 [2] [1] [10] [11] [6] [11] [35] [2] [1]
B1 19     2 3 337 82 214 14
  21 4   15 22 4 62 33 302
B2 26     22 34        
  27 3 1       23 487 349 132
B3 6 2 14 205 3
  18     4 5     29 22
  24 1 [1] 6 [6] 5 [10] 2 [2] 47 [43] 209 [195] 41 [42] 4 [6]
B4 17 23                
C1 12 25 63 33 38 12 490 129 90 470
  15 [56] [85§] [18] [13] [12] [18] [288]
  16 1   2 [3] 4 [18] 10 [10] 41 [38] 3 [15] 60 [56] 20 [20]
C2 7     36 26        
†Ranks reported correspond to the lowest ranked predicted structure matching the crystal packing of XXVII excluding the isopropyl groups.
‡The experimental structure of XXVIII was available to all groups due to a coincidental publication, so the exercise was not a true blind test.
§An alternative originating structure was identified to match the experimental form.
[Figure 2]
Figure 2
(Top) Lattice and free energy difference between structure 28 of molecule XXVII and structures 38, 61 and 59 which share the same core packing of the experimental form. The global minimum (black filled circle) of each group has been included to show if other packings were found to be more stable within an energy model. (Bottom) Lattice and free energy difference with respect to Form A of molecule XXVIII. The energy range between the global minimum (black filled circle) and the 100th ranked structure (open circle) is shown to highlight the position of the experimental structure within the CSP set. If a subset of less than 100 structures was used in the energy calculation, filled circle is used instead of an open circle. As the initial set of structures includes 500 structures, the experimental one can lie outside of the 1st–100th range. In both plots, groups are organized as in Table 2[link], with the methodology class shown at the top. Groups that did not participate in the ranking of these two compounds are shown with a grey cross, while those that did not reproduce the geometry of the most stable polymorph are displayed with a red cross. If any of the structures' energies lie outside of the energy range considered, this is shown with an arrow.

Where both lattice and free energies were calculated (Groups 3, 10, 16, 17, 24), free energies brought the experimental crystal packing closer in energy to the global minimum – becoming the global minimum for Group 3 – while Groups 16 and 24 ranked it as the global minimum with both lattice and free energies.

5.2. XXVIII

There exists one experimentally determined crystal structure of XXVIII (Form A, CSD: OJIGOG01). The experimental crystal structure of XXVIII was coincidentally published by an external group during the first phase of this blind test. Despite this, XXVIII was still included in the structure ranking exercise with disclosure that all ten participating groups had access to the experimentally determined form.

It was reported that most of the polymorph screen experiments resulted in oxidative dimerization of the ligand with no observed crystallization of the desired complex. This could indicate kinetic factors may be involved and so there is a degree of uncertainty whether Form A is the thermodynamically most stable form.

The majority (71%) of the provided structure set represent the experimentally observed trans square planar geometry, while 25% of the structures are cis square planar, and 4% exhibit the see-saw conformation, see Fig. 2[link].

The experimental structure was ranked relatively low in energy in the majority of cases with five of the ten participating groups (3, 10, 11, 20, and 27) ranking the observed form as the most stable, see Fig. 3[link]. Methods that ranked the experimental form as the most stable applied dispersion-corrected DFT; PBE0-MBD was applied in some form by three groups, and PBE0-D3 and B86bPBE-XDM by the remainder.

[Figure 3]
Figure 3
Examples of trans-square planar (left), cis-square planar (centre), and see-saw geometries of target XXVIII (right).

No differences were observed in the ranking of Form A at low versus ambient temperatures for those groups that calculated both lattice and free energies, with Groups 3 and 10 predicting Form A to be the most stable structure while Group 24 calculated a relative energy of around +4.8 kJ mol−1 from the global minimum in both cases.

5.3. XXXI

There exist three experimentally determined polymorphs of XXXI: Forms A (CSD: ZEHFUR02), B (CSD: ZEHFUR) and C (CSD: ZEHFUR01). Form C is a channel-type solvate containing unresolved solvent (see supplementary information from phase one) and therefore falls outside the scope of this ranking exercise. Form A contained disorder of the fluorinated ring (see SI-A Fig. 1) resulting in two disorder components, both of which were represented by structures in the lists provided to participants. Competitive slurry experiments have demonstrated an enantiotropic relationship between Form A (more stable above 55°C) and Form B (more stable below 55°C). Since no group calculated the properties of the crystals at temperatures higher than 55°C, results are discussed with respect to the stability relationship at lower temperatures, with Form B being the most stable form.

Many methods rank Forms A and B within the lowest 5 kJ mol−1 of structures: eight out of 20 methods at 0 K (the periodic DFT methods of Groups 2, 3, 5, 9, 10, 14 and 22, and the machine learned model of Group 16) and three out of seven methods at ambient temperature (Groups 3, 10, 20).

All periodic DFT-D methods calculated Forms A and B to be within 5.7 kJ mol−1 from the global minimum. Of those methods, that of Group 10 (at 0 K) and Group 3 (at ambient temperature) ranked Form B as the global minimum, while Groups 2, 5, 9, 20 and 22 ranked both forms within the lowest 3 kJ mol−1 region. The machine learned model of Group 16 also predicted the observed structures within the same region at 0 K, although energies were higher at ambient temperature.

Of the 20 groups that submitted results for XXXI, five groups ranked Form B as the most stable of the observed forms; two groups ranked Form B as the most stable of all theoretical structures (Group 10 at 0 K and Group 3 at 300 K), while Groups 12 and 21 ranked the structure at 12th (+13.6 kJ mol−1) and 4th (+5.9 kJ mol−1) respectively, and Group 7 at 26th (where a geometric-based scoring function was used), see Fig. 4[link].

[Figure 4]
Figure 4
Lattice and free energy difference of the experimental structures with respect to the most stable polymorph of molecule XXXI (top), XXXII (middle) and XXXIII (bottom). The energy range between the global minimum (black filled circle) and the 100th-ranked structure (open circle) is shown to highlight the position of the experimental structures within the CSP set. If a subset of less than 100 structures was used in the energy calculation, the filled circle is used instead of an open circle. As the initial set of structures of compounds XXXII and XXXIII includes 500 structures, the experimental one can lie outside of the 1st–100th range. Groups are organized as in Table 2[link], with the methodology class shown at the top of each plot. Groups that did not participate in the ranking of these compounds are shown with a grey cross, while those that did not reproduce the geometry of the most stable polymorph are displayed with a red cross. If any of the structures' energies lie outside of the energy range considered, this is shown with an arrow. For molecule XXXI, Group 7 used a ranking method not based on thermodynamics but on topological probabilities (highlighted in red). In this case the higher the score, the more probable it is to observe a structure.

Periodic DFT-D methods predicted Forms A and B to be close in energy (within 3 kJ mol−1 in nearly all cases, falling within the error bars of accuracy of most DFT methods (Abramov et al., 2021[Abramov, Y. A., Li, B., Chang, C., Zeng, Q., Sun, G. & Gobbo, G. (2021). Cryst. Growth Des. 21, 5496-5502.]; Firaha et al., 2023[Firaha, D., Liu, Y. M., van de Streek, J., Sasikumar, K., Dietrich, H., Helfferich, J., Aerts, L., Braun, D. E., Broo, A., DiPasquale, A. G., Lee, A. Y., Le Meur, S., Nilsson Lill, S. O., Lunsmann, W. J., Mattei, A., Muglia, P., Putra, O. D., Raoui, M., Reutzel-Edens, S. M., Rome, S., Sheikh, A. Y., Tkatchenko, A., Woollam, G. R. & Neumann, M. A. (2023). Nature, 623, 324-328.]), an indicator that the two are likely of such similar stability that a correct prediction of the relationship may be beyond the accuracy capable of periodic DFT-D in this case.

Periodic DFT-D methods of Groups 3, 4 and 10, the machine learned model of Group 16, and the ΔEintra/Uinter method of Group 24 provided both lattice and free energies. The incorporation of temperature effects was important for this system, resulting in stabilized relative energies for the experimental Forms by Groups 3, 4 and 24. Significantly, for Group 3, thermal contributions resulted in a different (correct) prediction of the relative stability relationship.

It is noted for Groups 6, 18, 19, and 26 (ΔEintra/Uinter methods) geometry optimization of the experimental representative of Form B resulted in a different structure. Notably, this did not occur for any periodic DFT-D methods. Interestingly, of those groups, Group 26 ranked the deviated structure as the lowest in energy on the solid-form landscape.

5.4. XXXII

There exist two known crystal structures of XXXII determined from single crystal X-ray diffraction at 90 K; Form A (CSD: JEKVII) and Form B (CSD: JEKVII01). Experimental slurry experiments have demonstrated that Form B is more stable than Form A at room temperature (RT) and above (see the experimental report in the supplementary information of phase one of this blind test). Form B was also determined to be the most stable of all known anhydrous forms (including those showed by PXRD alone, though not included in this study due to no structural determination).

An additional structure of Form B at RT was determined from PXRD. The prepared set of structures supplied to participants contained this structural determination. However, upon analysing the results, it was found in all cases that the structure no longer resembled the starting structure after geometry optimization. A redetermination of the crystal structure of Form B at RT was later provided by Group 20 based on a predicted structure and the original PXRD data. This showed greater agreement with the PXRD data and was subsequently corroborated by solid-state nuclear magnetic resonance (NMR) data using 13C CPMAS and 1H–13C CP-HETCOR experiments (see the experimental report in the supplementary information of phase one of this blind test). Structural comparisons by the organizers concluded that, despite a small difference in symmetry (from space group [P{\overline 1}] at 90 K to P21/c at RT) between the two structures due to a minor conformational change, the structural difference was not large enough for any structural comparison tools to differentiate unambiguously when assigning structural matches to either one. Visual crystal packing and molecule overlays of the two structures are provided in SI-A Figs. 2–4. This relates to a wider issue on the definition of isostructurality which is raised in the report of the first phase of this blind test. The analysis therefore only involved one of the two structures (Form B at 90 K).

For this exercise, there were two structures present in the list to analyse: the major disorder component of Form A (referred to as Form A), and the low temperature structure of Form B (referred to as Form B). There were two cases where the geometry optimized structure no longer matched the experimental form (the mixed method of Group 6 for Form A, and the machine learned model of Group 15 for Form B). Similarly to what was observed for molecule XXXI, this did not occur for periodic DFT methods.

No method predicted any of the experimental forms to reside within 3 kJ mol−1 of the global minimum. Furthermore, only four groups (machine-learned models of Groups 12 and 16, and ΔEintra/Uinter methods of Groups 19 and 24) predicted Form B to be more stable than Form A in line with reported experimental data, see Fig. 4[link]. For results where both lattice and free energies are reported, temperature corrections seemingly offer no clear improvement on either ranking experimental structures close to the global minimum or the correct ranking of the stability relationship between Forms A and B with only Groups 3, 16 and 24 showing a smaller energy difference between these two observed forms.

Attempts by the experimental providers of this system to reproduce and determine the structures of the unresolved forms showed previously by low-quality PXRD patterns were unsuccessful. The several additional unknown forms of XXXII raise uncertainty on whether the true global minimum structure has been observed experimentally, with the overall predictions – particularly the consensus of the periodic DFT-D results – further fuelling this uncertainty. Indeed, it has been predicted previously (Neumann & van de Streek, 2018[Neumann, M. & van de Streek, J. (2018). Faraday Discuss. 211, 441-458.]) that for between 15–45% of all chemical compounds, the thermodynamically most stable form has not yet been found because it is kinetically hindered. It is possible that XXXII is one of those cases.

The CCDC-prepared structure sets contained many structures (see SI-A Table 21) which showed high similarity to PXRD patterns of unresolved polymorphs of XXXII. These patterns (labelled H, K, L, N, P, and R) are available in the supplementary information experimental report of phase one of this blind test. Analysis of results submitted for this structure ranking exercise (see SI-A Tables 22–24) found that structures with large PXRD similarity to pattern H were ranked within 3 kJ mol−1 of Form B. Additionally, structures with substantial PXRD similarity to N were ranked lower in energy than Form B in the majority of cases. Comparisons of structures with high PXRD similarity to patterns H and N show nearly identical packings but molecules with either or both the difluoromethyl and the oxazine groups in different conformations, see SI-A Figs. 5 and 6, suggesting disorder could be present in these two forms. Further analyses, although desired, are beyond the scope of this study, but the initial observations outlined here serve as further evidence for the possibility of a more stable structure of XXXII yet to be observed experimentally.

5.5. XXXIII

Two crystal structures of XXXIII are known to exist; Form A (CSD: ZEGWAN) and Form B (CSD: ZEGWAN01). Experimental studies show Form A is a disappearing polymorph (see the experimental report in the supplementary information of phase one of this blind test).

In comparison with other target systems, the most consistency observed across all methods was seen for target XXXIII, with 14 out of 17 groups correctly ranking Form B as more stable than Form A, see Fig. 4[link].

All periodic DFT methods ranked Form B as the lowest energy structure on the landscape via both 0 K lattice energies (Groups 3, 5, 9, 10, 11, and 22), and free energies at room temperature (Groups 3, 10, and 20). Notably, Group 20 predicted the stable Form B as rank 1 and the metastable Form A as rank 2. None of the remaining methods (machine learning-based, force field, or mixed methods) predicted either experimental form as the global minimum: Form B, the most stable form, was ranked at 20th (+5.6 kJ mol−1) by Group 16 (machine learning-based), 22nd (+8.1 kJ mol−1) and 14th (+53.4 kJ mol−1) by Groups 18 and 19 (mixed methods), and third (+4.0 kJ mol−1) by Group 6 (force field-based). This possibly reflects that charge distributions of ions in crystals can differ significantly from those of the isolated ions, making the molecular pairwise additive approximation less appropriate.

There was one case of the experimental representative no longer resembling the known structure following geometry optimization (Form B by Group 24, attributed to human error – see SI-B Section 20). All methods, except for the machine learned model of Group 12 and the ΔEintra/Uinter method of Group 21, predicted Form B to be more stable than Form A at both 0 K and room temperature, in agreement with the experimental observation that Form A became difficult to crystallize once Form B had been isolated.

5.6. Free energy results

In cases where both lattice energies and free energies were reported, overall little advantage was gained for targets XXXII and XXXIII in terms of improvement in predicted rank or relative energies of experimental forms. Cases have been reported previously in which state-of-the-art dispersion-inclusive DFT methods, even with finite temperature corrections, failed to reproduce the experimentally observed order of stability, for example for the α and β forms of the energetic material HMX (O'Connor et al., 2023[O'Connor, D., Bier, I., Tom, R., Hiszpanski, A. M., Steele, B. A. & Marom, N. (2023). Cryst. Growth Des. 23, 6275-6289.]) The influence on predictions can be observed for target XXXI where the majority of free energy methods from classes A and B (Groups 3, 4, 24) predicted experimental forms to be closer to the global minimum. In one case (Group 3), free energies provided the correct relative stability relationship of polymorphs in contrast to energies without temperature corrections. Additionally, improvements were observed in relative energies for structures matching the experimental packing for XXVII. However, in the case of the machine-learned method from Group 16, calculated free energies worsened the relative energies and ranks of experimental forms for molecules XXXI and XXXII, whilst little impact was observed for XXXIII.

The results here demonstrate that, while free energies may not offer a clear improvement in predictions for some systems, the application of methods in classes A and B for target XXXI has demonstrated the benefit that temperature corrections can have on predicted relative stabilities. Free energies are essential for predicting whether the relative stability of polymorphs changes with temperature.

5.7. Resource utilization

All participants were required to report an estimate of the number of central processing unit (CPU) core hours utilized for ranking each list of structures, shown in Table 4[link]. While the reported numbers provide an idea of method efficiency, they should be handled in the context of used hardware.

Table 4
Summary of CPU core hours reported per target molecule for each group where predictions were attempted

Class Group XXVII XXVIII XXXI XXXII XXXIII Total Hardware details
A1 4     30,000 600,000 180,000 810,000 AMD Zen 2 EPYC 7H12
  5 492,544   26,777 448,829 959,728 1,927,878 Intel Skylake 2.0 GHz
  22 48,120 46,080 18,432 34,560 69,120 216,312 Intel Xeon Gold 6230
A2 9 350,000   200,000   1,500,000 2,050,000 AMD Zen 2 EPYC 7H12
  10 786,380 1,689,100 240,000 1,700,000 1,824,016 6,239,496 Intel Xeon Platinum 8124M
  11 442,488 1,444,036 40,124 580,001 349,766 2,856,415 Intel Xeon E5-2683 v4
  14     1,400,000     1,400,000 AMD EPYC 7351
A3 2     ? ?   0 AMD 64 GPUs / bespoke cluster
  3 3,000,000 1,100,000 400,000 4,000,000 1,500,000 10,000,000 Intel Xeon X5650, E5-2650 v3, Silver 4214R, Platinum 8174
  20 385,229 379,699 64,512 776,909 77,414 1,683,763 Intel Xeon E5-2650 v4
B1 19     80,000 2,000,000 656,000 2,736,000 Intel Xeon Hasswell E5-2666 v3
  21 163,278   640,875 4,361,740 2,155,256 7,321,149 Intel Xeon Gold 6154, 6258R / FUJITSU A64FX
B2 26     11,458     11,458 Intel Xeon Gold
  27 30,880 54,769   132,606 12,586 230,841 Intel Xeon Platinum, Gold-6132, Xeon E5-2695 v3
B3 6 455 10 55 1,200 195 1,915 Various computers, all CPU times standardized to one 2.66 GHz processor Intel Quad 9400
  18     715   2,666 3,381 Intel Xeon Gold 6230R
  24 1,963 50,266 1,023 86,668 70,193 210,113 Intel Xeon E5-2650v3, L5630 / E5-2660v4 mixed clusters
B4 17 2,732         2,732 Intel Xeon Gold 6154
C1 12 10,000 10,000 10,000 10,000 10,000 50,000 Intel Xeon Gold 6132
  15 36,277 248,763 7,449 148,942 67,289 508,720 AMD EPYC 7401
  16 755,000   80,000 47,000 530,000 1,412,000 AMD EPYC 7742 / Intel Platinum 8280 / Nvidia RTX 3090, GTX 1080, GTX 1080ti / Tesla V100S
C2 7     10     10 Intel Core i7-10750H

It is evident from the resources reported for XXVII (where only 100 structures were ranked compared with 500 for XXVIII, XXXII, and XXXIII) that calculation intensity heavily depends on the chemistry and complexity of the system. Furthermore, there is a large variation in the reported computational costs, even amongst those within the same category of theory. This is a reflection of whether multiple rounds of optimizations or energy calculations were applied. The calculation of vibrational frequencies for obtaining free energies, especially for large molecules, is for instance extremely time-demanding when performed with periodic DFT methods. While some groups applied multiple rounds of calculations on all structures, others have made efforts towards limiting the number of structures undergoing calculations at the highest levels of theory. Free energies were calculated in addition to lattice energies by Groups 3, 4, 10, 16, and 24, creating further variation in resources utilized.

Another consideration for variation in resources utilized is the likely effect of structure anonymization by the organizers in structure list preparation (see step four in the protocol outlined in Section 2.5[link]) on methods based on dividing the crystal into molecules (categories B1–B3, Table 2[link]). Groups 24, 26 and 27 reported high intramolecular energies in many of the supplied structures (see SI-B Sections 20 and 21), requiring corrections at the molecular level and leading to the use of further resources. The effects of this extended beyond resource utilization as raised by Groups 26 and 27 at the blind test meeting (see Section 6[link]).

Positive observations to note are the machine learning-based approach from Group 16 (AIMNet) which offers an efficient alternative to DFT methods. Additionally, the topological analysis method (Group 7) represents an interesting new approach to performing extremely fast ranking by completely avoiding energy ranking. However, this method cannot be used for geometry optimization and its current performance leaves much to be desired. These demonstrate that genuinely new ideas are still being developed as efficient alternatives to periodic DFT methods.

It is difficult to judge the progress made in efficiency of ranking methods alone as we have not previously analysed the separate components of CSP methods in such a way. Considering this point, we acknowledge that the workflows applied here may differ vastly due to the difficulty in drawing a line between the structure generation and ranking parts of any one CSP method, and due to the different interpretations of the ranking exercise by different groups as a result of the synthetic nature of the structure lists provided by organizers. Nevertheless, we hope the numbers reported here serve as a useful indication and motivation for future research into efficient structure ranking approaches.

6. Blind Test meeting

A two-day in-person meeting was held in Cambridge, UK in September 2022, following the final results submissions. This provided an opportunity for participants to present their results to fellow investigators, blind test organizers, and active researchers in the CSP community from both industry and academia. A session was also held between participants and organizers to discuss any issues that arose during the test and to reflect on the current and possible future blind test initiatives. We include here some topics from the discussions which were important with respect to either the test results or future initiatives and research.

The structure ranking exercise constructed by the CCDC organizers, whilst providing a valuable opportunity to benchmark and compare ranking methods alone, did not reflect how CSP is carried out in reality. This point was highlighted during discussions, and it was emphasized that the relatively small numbers of structures provided were better suited to more intensive methods and did not provide an incentive to use recently developed more efficient methods (such as those based on machine learning). This was acknowledged by the organizers and serves as valuable feedback to guide future such initiatives.

In the case of molecule XXVII where the experimental form was not provided in the ranking list due to problems encountered with COMPACK (described in Section 5.1[link]), many groups expressed interest in being provided with the experimental structure to calculate relative energy and add to their results. The likelihood of dynamic disorder in the system was discussed which motivated subsequent MD and metadynamics work outlined in Section 5.1[link], culminating in the decision to analyse the results based on the molecular `core' only (excluding the TIPS groups).

During the meeting, Groups 26 and 27 (a collaboration of individuals across the two groups) presented an issue with the guidance of the exercise due to the assumption that the organizers deformed the crystal structures during structure list preparation. This was assumed due to gas phase calculations of monomers revealing a large majority to exhibit high energies relative to the global minimum. This deformation is suspected to be due to step 4 of preparation, see Section 2.5[link]. For transparency and understanding, comparisons of the structures before and after this preparation step were carried out by the organizers and results are reported in Tables 3–7 in SI-A. At the same time, the experimental representatives were very close to experimental crystals: the monomers' RMSD was between 0.0 Å and 0.28 Å, while the crystal RMSD30 values were between 0.180 Å and 0.332 Å, except for target XXVII. All representatives were within the acceptance criteria. The analysis of monomer energies led the group to carry out a workflow which significantly changed the provided crystal structures and consequently worsened their outcomes. It was agreed amongst participants and organizers that this was attributed to an unfortunate assumption by the participating group rather than ambiguous instructions for the exercise from the organizers.

7. Conclusions of the ranking exercise

In this paper, we have presented the computational methods for crystal structure energy ranking employed in the seventh blind test of crystal structure prediction organized by the CCDC. Allowing for a more effective and fair analysis of ranking methods via two separate assessments, this second phase of the test involved 22 of the total 28 participant groups. The results of this study offer valuable insights into the performance of various current approaches, including force field based methods, density functional theory (DFT) calculations, and machine learning techniques.

Assessing the accuracy of CSP ranking methods for this constructed exercise can be simplified into two questions: (i) Are the experimentally observed structures ranked at or close to the global minimum of the structure set (with consideration of the expected error bars in keeping with the limitations of the method applied (Abramov et al., 2021[Abramov, Y. A., Li, B., Chang, C., Zeng, Q., Sun, G. & Gobbo, G. (2021). Cryst. Growth Des. 21, 5496-5502.]; Firaha et al., 2023[Firaha, D., Liu, Y. M., van de Streek, J., Sasikumar, K., Dietrich, H., Helfferich, J., Aerts, L., Braun, D. E., Broo, A., DiPasquale, A. G., Lee, A. Y., Le Meur, S., Nilsson Lill, S. O., Lunsmann, W. J., Mattei, A., Muglia, P., Putra, O. D., Raoui, M., Reutzel-Edens, S. M., Rome, S., Sheikh, A. Y., Tkatchenko, A., Woollam, G. R. & Neumann, M. A. (2023). Nature, 623, 324-328.])? (ii) Did the method correctly reproduce the relative stability relationship observed experimentally in the case of multiple observed polymorphs? The periodic dispersion-corrected GGA density functional-based methods of Groups 3, 10, and 20 produced results in excellent agreement with experimental data satisfying both assessments outlined above for all other targets, with the exception of target XXXII where confidence is low in the completeness of the experimental solid-form observations (discussed in Section 5.4[link]). All three methods employed the calculation of free energies. The results overall showed that temperature corrections, where calculated, provided an improvement of calculated energies relative to global minima for targets XXVII and XXXI, but offered no clear improvement for the remainder of target compounds. Taking error into account for the prediction of stability relationships, Groups 2, 5, 9 and 22 also produced results in line with experimental data. A highly challenging exercise for ranking methods was predicting the relative stability of Forms A and B of XXXI (where Form B is more stable than Form A) which were consistently calculated to have 1–2 kJ mol−1 difference in many cases.

When periodic GGA DFT-D is used, it is probably often worthwhile to add corrections such as monomer or dimer calculations with MP2D or (doubly) hybrid functionals, or single point periodic calculations at a higher level of theory, such as a metaGGA or hybrid functional. Such approaches make the predictions less sensitive to spurious self-interaction errors. However, a comparison of the GGA and hybrid functional rankings suggests that none of the compounds considered in the current blind test exhibited significant self-interaction errors.

Machine learning techniques show promise as an emerging approach for crystal structure energy ranking. By training on existing data, these methods can provide rapid and sometimes accurate predictions. However, their performance is contingent upon the quality of the training data and the choice of machine learning algorithm. Machine learned potentials were utilized by three groups, one of which (the system-specific AIMNet potentials used by Group 16) was applied for the calculation of final ranking free energies yielding reasonable results in agreement with the experiment in many cases, thus demonstrating a viable alternative to DFT that, once trained, is orders of magnitude faster.

A new fast-ranking method based solely on crystal structure geometry was proposed by Group 7 and applied in this test, and similar statistical methods have been used in several previous tests. These methods still face the challenge of accurate predictions. Despite overall poor predictive ability, it is still valuable to explore alternative routes to ranking, particularly now that the question is being raised on whether we are reaching the limits of DFT capabilities.

The use of the Dreiding force field as the sole ranking method showed poor performance. Although a valuable tool when dealing with large numbers of structures in the structure generation stages of CSP workflows, general purpose force fields are not suitable for the accurate calculation of relative polymorph stability.

The resources utilized by CSP methods have continued to increase as methods have developed, as showed by the reported CPU hours here and in previous blind tests. This exercise encouraged development for methods focusing on high accuracy given the relatively small numbers of structures provided to rank. Since environmental impact and method efficiency are important considerations for future developments, such future initiatives should aim to provide a suitable platform to test and benchmark methods aiming to optimize speed and resource utilization.

Based on these findings, we propose several avenues for future research and development in the field of crystal structure energy ranking:

(1) Exploration of more advanced DFT approaches, including hybrid functionals and monomer corrections to enhance the reliability of DFT-based calculations.

(2) Increase the accuracy of low-cost alternatives to periodic DFT methods. Machine-learning techniques or gas-phase ab initio calculations can help develop a robust workflow to parameterize more accurate force fields tailored to the molecule or design new potentials. Larger and more diverse data sets can facilitate the training of machine learning approaches and speed up CSP calculations.

(3) Integration of multiple energy ranking methods, leveraging their respective strengths, to develop more robust and accurate hybrid approaches for crystal structure prediction.

(4) The computation of free energies at ambient temperature appears to be beneficial for obtaining the right answer for the right reason; i.e. modelling real physical effects. Whether computed free energies add value, though, is also heavily dependent on the system as showed in this exercise. However, free energy corrections improve the predictive ability only when calculated with already accurate lattice energy methods. The development of efficient free energy methods is desirable.

(5) Increasing the efficiency of CSP methods with serious considerations for resources utilized and environmental impact. New benchmarking tests with a more focused assessment, on geometric optimization algorithms for example, would be beneficial.

A number of groups have provided post-analysis of their own results, involving further calculations on experimental forms, benchmarking against alternative methods, and providing explanations for any unexpected or unreasonable results. We urge the reader to see SI-B for further reading. Additionally, all participant groups were encouraged to provide additional explanations and assessments via their own peer-reviewed reports which we also refer the reader to.

8. Overall conclusions of the blind test and outlook

The seventh blind test fulfilled its objective of allowing the participants to benchmark and improve their methodologies. It can be seen in the supplementary information (SI-B) from many participating groups that they did have to adjust their approaches to meet the challenges posed by the different target compounds. Many groups have already further developed their methods in response to the results. For example, improvements in the non-thermodynamic topological prediction results for compound XXXI tackled by Group 7 are given explicitly in SI-B Section 6.

The seventh blind test aimed at being more realistic, going beyond asking whether it was possible to predict a carefully selected structure of defined Z′ with no disorder. The introduction of polymorph screening was intended to support the likelihood that the most thermodynamically stable form was among the targets. The extensive experimental work during this blind test led to a valuable interplay between experiment and theory, revealing the complexity of the organic solid state. The computational investigations revealed that the target structure XXVII was dynamically disordered, which considerably complicated the analysis of the results. One group proposed a better crystal structure for XXXII Form B than originally provided. This is a good example of how computational methods can lead to a reinterpretation of experimental data.

It is impressive that the Z′ = 3 crystal structure of XXIX could be solved by comparing simulated laboratory PXRD data. It highlights the now widely used and successful cross-correlation-based PXRD pattern similarity measure, and the necessity of optimizing the lattice parameters in order to maximize the pattern similarity.

The detection of disorder in XXVII, XXXI Form A, the XXX 2:1 cocrystal and XXXII Form A, raises important problems in the distinction between static and dynamic disorder, whether CSP can predict such disorder, and how the thermodynamic effects of disorder should be calculated.

The splitting of the blind test into two phases, structure generation and ranking, had the great advantage of enabling a wider range of methods to be applied and more effective comparisons to be made. However, while it is clear that improving the computational efficiency of CSP is a worthwhile aim, the disruption of established CSP workflows prevented much meaningful analysis of success in terms of CPU efficiency. The selection and anonymization process of the 100 or 500 structures also resulted in jumping to another minimum on some potential surfaces. Also implicit in the huge range of structures generated in part one of this blind test is the need to tackle the over-prediction problem, so that structures which are effectively duplicates at experimental temperatures are eliminated.

The overall conclusion of the first phase is that there are search methods that can successfully locate the experimental structures, provided the search is exhaustive enough. For example, the Z′ = 3 structure of XXIX would not have been found in a standard CSP search. These show the value of using experimental information to tailor the CSP search to the system.

The results of the energy ranking submissions are extremely encouraging as they suggest some very different approaches to balancing the changes in intramolecular conformation with the various types of intermolecular forces in a system are converging. Predicting the observed structures to reside within the likely energy range of solution-grown polymorphs proved challenging for some systems. Highly sophisticated methods are clearly needed to be confident that a specific structure is the most thermodynamically stable, but the importance of different contributions will depend on the molecule and the types of crystal packing that are thermodynamically competitive. The choice of systems for this blind test has pushed many participants beyond their comfort zone. However, many of the expensive corrections, such as going beyond the PBE GGA functional in periodic DFT-D, using a highly converged monomer energy for correction, or evaluating the free energy with or without anharmonicity, appear less important in these than other polymorphic systems. To put recent developments in the field of energy calculations of organic polymorphs to the test, future blind tests should probe the ability to predict enthalpy differences between polymorphs and the transition temperatures of enantiotropic systems. Building on the co-crystal challenge of system XXX, future blind tests should also assess the ability to compare different compositions, generalizing to other multi-component systems such as hydrates. It will be interesting to see whether alternative ranking methods can reduce the number of structures where state-of-the-art thermodynamic calculation methods need to be used.

Thus the seventh blind test has significantly increased the diversity and size of the target systems and the nature of the challenges. By including polymorph screening results, it is clear that the solid state landscape of some of these molecules is more complex than as described by a Z′ = 1 structure without disorder. The best methods have been very successful, and the progress since the sixth blind test is remarkable. However, the computational cost is so large that right-sizing and balancing the computational and experimental effort in studying solid form landscapes will be very dependent on the aim of the study.

9. Glossary

Ψmol A method that combines a quantum chemical model of individual molecules and an atomistic force field model for the intermolecular interactions in the crystalline environment.

ANI-2x A neural network type of machine learning atomic potential

API Application programming interface

ASE Atomic Simulation Environment, a Python library

B86bPBE A GGA density functional consisting of the exchange functional proposed by Becke in 1986 and the PBE correlation functional

B97D A variation of Becke's GGA functional introduced in 1997[Momenzadeh Abardeh, Z., Salimi, A. & Oganov, A. R. (2022). CrystEngComm, 24, 6066-6075.], including Grimme's dispersion correction

B3LYP A variation of Becke's three-parameter hybrid functional with the LYP (Lee–Yang–Parr) correlation term

CCSD(T) Coupled cluster theory with full single and double excitations and noniterated triple excitations

CIF Crystallographic Information File, a standardized file format for crystallographic data

COMPACK An algorithm for calculating crystal structure similarity based on atomic distances

COMPASS Condensed-phase Optimized Molecular Potentials for Atomistic Simulation Studies, a force field

D3 Grimme's dispersion correction, version 3.

D3BJ The D3 dispersion correction with Becke–Johnson damping

D4 Grimme's dispersion correction, version 4

DFT-D Dispersion-corrected density functional theory

DFTB Density functional tight binding

Fvib The lattice vibrational contribution to the free energy

FIRE Fast Inertial Relaxation Engine, an optimization algorithm

GAFF General Amber Force Feld

GFN-xTB A self-consistent and dispersion-corrected DFTB method

GGA Generalized gradients approximation

GPR Gaussian process regression, a machine learning method

HF The Hartree–Fock method

MBD Many body dispersion, a dispersion correction

MD Molecular dynamics, a simulation method

ML Machine learning

MP2 Second-order Møller–Plesset perturbation theory

MP2D Dispersion-corrected second-order Møller–Plesset perturbation theory

OPLS Optimized potentials for liquid simulations, a force field

PBE The GGA exchange correlation functional by Perdew, Burke and Ernzerhof

PBE0 A hybrid exchange-correlation functional, PBE with 25% Hartree–Fock exchange

PIXEL A method for calculating intermolecular interaction energies by direct numerical integration over electron densities

RHF The Restricted Hartree–Fock method

SAPT Symmetry-adapted perturbation theory

SCAN The strongly constrained and appropriately normed meta-GGA density functional

SOAP Smooth overlap of atomic positions, a descriptor that encodes regions of atomic geometries

TIPS Triisopropylsilane, a functional group

X23 A benchmark dataset consisting of 23 crystal structures of small organic molecules

XDM The exchange-hole dipole moment dispersion correction

Supporting information


Footnotes

1www.cost.eu/actions/CA22107

Acknowledgements

The CCDC Blind Test Team. The CCDC organizers (L. M. Hunnisett, J. Nyman, N. Francia, I. Sugden, G. Sadiq, and J. C. Cole) gratefully acknowledge numerous CCDC colleagues for their helpful feedback and suggestions on the manuscript (P. McCabe, E. Pidcock, P. Martinez-Bulit, C. Kingsbury), providing useful python knowledge (A. Moldovan), providing and maintaining internal compute resources (K. Taylor, M. Burling, J. Swift, L. Wallis), monitoring and depositing structures in the CSD (S. Ward, K. Orzechowska, V. Menon), support in organization of the blind test meeting (E. Clarke), and improvements to the Crystal Packing Similarity tool (M. Read). Data analysis was performed using resources provided by the Cambridge Service for Data Driven Discovery operated by the University of Cambridge Research Computing Service (www.csd3.cam.ac.uk), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/T022159/1), and DiRAC funding from the Science and Technology Facilities Council (www.dirac.ac.uk).

Group 2. GJOB gratefully acknowledges funding from the National Science Foundation (CHE-1955554) and supercomputer time from ACCESS (CHE110064). Additional support for this work to PAU came from the U.S. Department of Energy, Office of Science, Office of Workforce Development for Teachers and Scientists, Office of Science Graduate Student Research (SCGSR) program. The SCGSR program is administered by the Oak Ridge Institute for Science and Education (ORISE) for the DOE. ORISE is managed by ORAU under contract number DE-SC0014664. All opinions expressed in this paper are the author's and do not necessarily reflect the policies and views of DOE, ORAU, or ORISE.

Group 3. The computational results presented have been achieved using the Vienna Scientific Cluster as well as the HPC facilities at the University of Graz. This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 890300.

Group 4. Computational resources for this work were provided by the Ministry of Education, Youth and Sports of the Czech Republic through the project e-INFRA CZ (ID:90254). The work on this paper was supported by the Czech Science Foundation grant No. 23-05476M, and by grants of specific university research No. A1_FCHI_2023_001, A2_FCHI_2023_004, and and A2_FCHI_2023_015.

Group 5. We thank the University of Southampton for a University of Southampton Presidential Scholarship (Patrick W. V. Butler), Johnson Matthey for funding (James Bramley), the Air Force Office of Scientific Research for funding under award No. FA8655-20-1-7000 (Joseph E. Arnold) and the European Research Council under the European Union's Horizon 2020 research and innovation program (grant agreement No. 856405) (Christopher Taylor, Graeme M. Day). We acknowledge the use of the IRIDIS High-Performance Computing Facility and associated support services at the University of Southampton. Via our membership of the UK's HEC Materials Chemistry Consortium, which is funded by the EPSRC (EP/R029431), this work used the UK Materials and Molecular Modelling Hub for computational resources, MMM Hub, which is partially funded by EPSRC (EP/T022213 and EP/W032260), and ARCHER2 UK National Supercomputing Service (https://www.archer2.ac.uk).

Group 6. Toine Schreurs and Martin Lutz provided computer facilities and assistance.

Group 9. This work was supported by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254).

Group 10. Competing interests: Many authors work at XtalPi Inc., a company that provide crystal structure prediction services. We would also like to thank other platform builders in our group. Although they did not directly participate in this blind test, some of them contributed to the construction of our early platform, and some of them contributed to the stable operation of our computing system. They are: Peiyu Zhang, Minjun Yang, Yang Liu, Dong Fang, Bochen Li, Jiuchuang Yuan, Ziqi Jiang, Xiaoqi Kang, Fei Li, Yanpeng Ma, Wenpeng Mei, Liang Tan, Huobin Wang, Hesheng Zhu.

Group 11. Group 11 thanks the Natural Sciences and Engineering Research Council of Canada and the Atlantic Computing Excellence Network (ACENET) for providing computational resources. AOR thanks: the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación, projects PGC2021-125518NB-I00 and RED2022-134388-T cofinanced by EU FEDER funds; the Principality of Asturias (FICYT), project AYUD/2021/51036 cofinanced by EU FEDER; and the Spanish MCIN/AEI/10.13039/501100011033 and European Union NextGenerationEU/PRTR for grant TED2021-129457B-I00.

Group 14 This work was supported by the European Union's Horizon 2020 research and innovation programme via the ERC grant APES (No 759721) and by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90140).

Group 15. This work has been supported in part by Croatian Science Foundation under the project UIP-2020-02-5675.

Group 16. OI acknowledges support from NSF CHE-2154447. We also acknowledge the Extreme Science and Engineering Discovery Environment award CHE200122, which is supported by National Science Foundation grant number ACI-1053575. This research is part of the Frontera computing project at the Texas Advanced Computing Center. Frontera is made possible by the National Science Foundation award OAC-1818253. This research in part was done using resources provided by the Open Science Grid, which is supported by the award 1148698, and the US DOE Office of Science. NM acknowledges support from National Science Foundation through grant DMR-2131944. This research used resources of Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357. We also acknowledge the Extreme Science and Engineering Discovery Environment award MAT210006, which supported 3M CPU core hours.

Group 18. Financial support for this work was made possible by Khalifa University (KU) under the Research and Innovation Grant (Award No. RIG-2023-054). This work was performed with the support of the Center for Catalysis and Separations (RC2-2018-024). All the computational calculations were performed using the High-Performance Computing (HPC) clusters of KU and the authors acknowledge the support of the Research Computing Department. Finally, we thank Professor Costas Pantelides and Professor Claire S. Adjiman for providing access to the CrystalPredictor code. SM would also like to express sincere gratitude to Dr Isaac J. Sugden for providing technical support on the use of CrystalPredictor.

Group 20. Competing interests: MAN is the founder, owner, and director, and DF, YML, JvdS, KS, and HD are employees of Avant-garde Materials Simulation Deutschland GmbH (AMS), a software company specializing in organic crystal structure prediction, and have no additional conflict of interest to disclose.

Group 21. In this work, we used the computer resources by Research Institute for Information Technology, Kyushu University, ACCMS, Kyoto University, and Information and Media Center, Toyohashi University of Technology. Part of this work used computational resources of Fugaku supercomputer through the HPCI System Research Project (Project ID: hp220143). The FMO calculations were performed in the activities of the FMO drug design consortium (FMODD). This work was supported by JSPS KAKENHI Grant Numbers 17H06373 (HG), 21K05002 (YI), and 21K05105 (NN).

Group 22. The group acknowledges support from the Russian Science Foundation (grant 19-72-30043). Competing interests: the USPEX code is free for academic researchers, but is distributed at a fee to companies.

Group 24. The work was supported by a Digital Design project funded by Eli Lilly and Company.

Groups 26 and 27. The work at the University of Delaware was supported by the US Army Research Laboratory and Army Research Office under grant W911NF-19-0117 and by National Science Foundation under grants CHE-1900551, CHE-2154908, and CHE-2313826. The use of the DARWIN computing system funded by NSF grant 1919839 is also acknowledged. JR acknowledges financial support from the Deutsche Forschungsgemeinschaft (DFG) through the Heisenberg Programme project 428315600. JR and MET acknowledge funding from the National Science Foundation grant DMR-2118890. MET acknowledges support from the National Science Foundation, grant No. CHE-1955381.

University of Kentucky group Support for the synthesis of diiodo TIPS pentacene was provided by the US National Science Foundation under grant DMR-1627428. Diffractometers were purchased using funds from the MRI program of the US National Science Foundation, grant CHE-1625732.

University of Reading group We thank the University of Reading's Chemical Analysis Facility for the instrumentation used in the collection of diffraction data from crystals of structure XXIX, and the UK Materials and Molecular Modelling Hub, which is partially funded by EPSRC (EP/T022213/1, EP/W032260/1 and EP/P020194/1), for computational resources.

References

First citationAbramov, Y. A., Li, B., Chang, C., Zeng, Q., Sun, G. & Gobbo, G. (2021). Cryst. Growth Des. 21, 5496–5502.  CrossRef CAS Google Scholar
First citationAdamo, C. & Barone, V. (1999). J. Chem. Phys. 110, 6158–6170.  Web of Science CrossRef CAS Google Scholar
First citationAddicoat, M., Adjiman, C. S., Arhangelskis, M., Beran, G. J. O., Bowskill, D., Brandenburg, J. G., Braun, D. E., Burger, V., Cole, J., Cruz-Cabeza, A. J., Day, G. M., Deringer, V. L., Guo, R., Hare, A., Helfferich, J., Hoja, J., Iuzzolino, L., Jobbins, S., Marom, N., McKay, D., Mitchell, J. B. O., Mohamed, S., Neumann, M., Nilsson Lill, S., Nyman, J., Oganov, A. R., Piaggi, P., Price, S. L., Reutzel-Edens, S., Rietveld, I., Ruggiero, M., Ryder, M. R., Sastre, G., Schön, J. C., Taylor, C., Tkatchenko, A., Tsuzuki, S., van den Ende, J., Woodley, S. M., Woollam, G. & Zhu, Q. (2018). Faraday Discuss. 211, 325–381.  CrossRef CAS PubMed Google Scholar
First citationAnstine, D., Zubatyuk, R. & Isayev, O. (2023). ChemRxiv, 10.26434/chemrxiv-2023-296ch.  Google Scholar
First citationAnstine, D. M. & Isayev, O. (2023). J. Phys. Chem. A, 127, 2417–2431.  CrossRef CAS PubMed Google Scholar
First citationApostolakis, J., Hofmann, D. W. M. & Lengauer, T. (2001). Acta Cryst. A57, 442–450.  Web of Science CrossRef CAS IUCr Journals Google Scholar
First citationAsmadi, A., Neumann, M. A., Kendrick, J., Girard, P., Perrin, M.-A. & Leusen, F. J. J. (2009). J. Phys. Chem. B, 113, 16303–16313.  CrossRef PubMed CAS Google Scholar
First citationBannwarth, C., Ehlert, S. & Grimme, S. (2019). J. Chem. Theory Comput. 15, 1652–1671.  Web of Science CrossRef CAS PubMed Google Scholar
First citationBardwell, D. A., Adjiman, C. S., Arnautova, Y. A., Bartashevich, E., Boerrigter, S. X. M., Braun, D. E., Cruz-Cabeza, A. J., Day, G. M., Della Valle, R. G., Desiraju, G. R., van Eijck, B. P., Facelli, J. C., Ferraro, M. B., Grillo, D., Habgood, M., Hofmann, D. W. M., Hofmann, F., Jose, K. V. J., Karamertzanis, P. G., Kazantsev, A. V., Kendrick, J., Kuleshova, L. N., Leusen, F. J. J., Maleev, A. V., Misquitta, A. J., Mohamed, S., Needs, R. J., Neumann, M. A., Nikylov, D., Orendt, A. M., Pal, R., Pantelides, C. C., Pickard, C. J., Price, L. S., Price, S. L., Scheraga, H. A., van de Streek, J., Thakur, T. S., Tiwari, S., Venuti, E. & Zhitkov, I. K. (2011). Acta Cryst. B67, 535–551.  Web of Science CrossRef IUCr Journals Google Scholar
First citationBartók, A. P., Kondor, R. & Csányi, G. (2013). Phys. Rev. B, 87, 184115.  Google Scholar
First citationBartók, A. P. & Yates, J. R. (2019). J. Chem. Phys. 150, 161101.  Web of Science PubMed Google Scholar
First citationBauer, J., Spanton, S., Henry, R., Quick, J., Dziki, W., Porter, W. & Morris, J. (2001). Pharm. Res. 18, 859–866.  Web of Science CSD CrossRef PubMed CAS Google Scholar
First citationBecke, A. (1986). J. Chem. Phys. 85, 7184–7187.  CrossRef CAS Google Scholar
First citationBecke, A. D. (1993). J. Chem. Phys. 98, 5648–5652.  CrossRef CAS Web of Science Google Scholar
First citationBeran, G. J. O., Wright, S. E., Greenwell, C. & Cruz-Cabeza, A. J. (2022). J. Chem. Phys. 156, 104112.  CrossRef PubMed Google Scholar
First citationBitzek, E., Koskinen, P., Gähler, F., Moseler, M. & Gumbsch, P. (2006). Phys. Rev. Lett. 97, 170201.  CrossRef PubMed Google Scholar
First citationBoerrigter, S. X. M., Josten, G. P. H., van de Streek, J., Hollander, F. F. A., Los, J., Cuppen, H. M., Bennema, P. & Meekes, H. (2004). J. Phys. Chem. A, 108, 5894–5902.  Web of Science CrossRef CAS Google Scholar
First citationBrandenburg, J., Bates, J., Sun, J. & Perdew, J. (2016). Phys. Rev. B, 94, 115144.  CrossRef Google Scholar
First citationBrandenburg, J. G. & Grimme, S. (2014a). J. Phys. Chem. Lett. 5, 1785–1789.  Web of Science CrossRef CAS PubMed Google Scholar
First citationBrandenburg, J. G. & Grimme, S. (2014b). In Prediction and Calculation of Crystal Structures, edited by S. Atahan-Evrenk & A. Aspuru-Guzik, vol. 345 of Topics in Current Chemistry, pp. 1–23. Springer International Publishing.  Google Scholar
First citationBraun, D. E., McMahon, J. A., Bhardwaj, R. M., Nyman, J., Neumann, M. A., van de Streek, J. & Reutzel-Edens, S. M. (2019). Cryst. Growth Des. 19, 2947–2962.  Web of Science CSD CrossRef CAS Google Scholar
First citationBroyden, C. G. (1967). Math. C, 21, 368–381.  CrossRef Google Scholar
First citationBučko, T., Hafner, J. & Ángyán, J. G. (2005). J. Chem. Phys. 122, 124508.  PubMed Google Scholar
First citationCaldeweyher, E., Bannwarth, C. & Grimme, S. (2017). J. Chem. Phys. 147, 034112.  CrossRef PubMed Google Scholar
First citationCaldeweyher, E., Mewes, J.-M., Ehlert, S. & Grimme, S. (2020). Phys. Chem. Chem. Phys. 22, 8499–8512.  CrossRef CAS PubMed Google Scholar
First citationČervinka, C. & Beran, G. J. (2018). Chem. Sci. 9, 4622–4629.  PubMed Google Scholar
First citationČervinka, C. & Fulem, M. (2017). J. Chem. Theory Comput. 13, 2840–2850.  PubMed Google Scholar
First citationČervinka, C. & Fulem, M. (2018). Cryst. Growth Des. 19, 808–820.  Google Scholar
First citationČervinka, C., Fulem, M., Stoffel, R. P. & Dronskowski, R. (2016). J. Phys. Chem. A, 120, 2022–2034.  Web of Science PubMed Google Scholar
First citationČervinka, C., Klajmon, M. & Štejfa, V. (2019). J. Chem. Theory Comput. 15, 5563–5578.  PubMed Google Scholar
First citationChakraborty, D., Berland, K. & Thonhauser, T. (2020). J. Chem. Theory Comput. 16, 5893–5911.  CrossRef CAS PubMed Google Scholar
First citationChisholm, J. A. & Motherwell, S. (2005). J. Appl. Cryst. 38, 228–231.  Web of Science CrossRef IUCr Journals Google Scholar
First citationCohen, A. J., Mori-Sánchez, P. & Yang, W. (2012). Chem. Rev. 112, 289–320.  Web of Science CrossRef CAS PubMed Google Scholar
First citationCole, J. C., Groom, C. R., Korb, O., McCabe, P. & Shields, G. P. (2016). J. Chem. Inf. Model. 56, 652–661.  Web of Science CrossRef CAS PubMed Google Scholar
First citationCoombes, D. S., Price, S. L., Willock, D. J. & Leslie, M. (1996). J. Phys. Chem. 100, 7352–7360.  CrossRef CAS Web of Science Google Scholar
First citationCruz-Cabeza, A. J., Reutzel-Edens, S. M. & Bernstein, J. (2015). Chem. Soc. Rev. 44, 8619–8635.  Web of Science CAS PubMed Google Scholar
First citationCutini, M., Civalleri, B., Corno, M., Orlando, R., Brandenburg, J. G., Maschio, L. & Ugliengo, P. (2016). J. Chem. Theory Comput. 12, 3340–3352.  Web of Science CrossRef CAS PubMed Google Scholar
First citationDay, G. M., Cooper, T. G., Cruz-Cabeza, A. J., Hejczyk, K. E., Ammon, H. L., Boerrigter, S. X. M., Tan, J. S., Della Valle, R. G., Venuti, E., Jose, J., Gadre, S. R., Desiraju, G. R., Thakur, T. S., van Eijck, B. P., Facelli, J. C., Bazterra, V. E., Ferraro, M. B., Hofmann, D. W. M., Neumann, M. A., Leusen, F. J. J., Kendrick, J., Price, S. L., Misquitta, A. J., Karamertzanis, P. G., Welch, G. W. A., Scheraga, H. A., Arnautova, Y. A., Schmidt, M. U., van de Streek, J., Wolf, A. K. & Schweizer, B. (2009). Acta Cryst. B65, 107–125.  Web of Science CSD CrossRef IUCr Journals Google Scholar
First citationDay, G. M., Motherwell, W. D. S., Ammon, H. L., Boerrigter, S. X. M., Della Valle, R. G., Venuti, E., Dzyabchenko, A., Dunitz, J. D., Schweizer, B., van Eijck, B. P., Erk, P., Facelli, J. C., Bazterra, V. E., Ferraro, M. B., Hofmann, D. W. M., Leusen, F. J. J., Liang, C., Pantelides, C. C., Karamertzanis, P. G., Price, S. L., Lewis, T. C., Nowell, H., Torrisi, A., Scheraga, H. A., Arnautova, Y. A., Schmidt, M. U. & Verwer, P. (2005). Acta Cryst. B61, 511–527.  Web of Science CSD CrossRef CAS IUCr Journals Google Scholar
First citationDay, G. M., Motherwell, W. D. S. & Jones, W. (2005). Cryst. Growth Des. 5, 1023–1033.  Web of Science CrossRef CAS Google Scholar
First citationDay, G. M. & Price, S. L. (2003). J. Am. Chem. Soc. 125, 16434–16443.  Web of Science CrossRef PubMed CAS Google Scholar
First citationDay, G. M., Price, S. L. & Leslie, M. (2003). J. Phys. Chem. B, 107, 10919–10933.  Web of Science CrossRef CAS Google Scholar
First citationDeij, M. A., ter Horst, J. H., Meekes, H., Jansens, P. & Vlieg, E. (2007). J. Phys. Chem. B, 111, 1523–1530.  Web of Science CrossRef PubMed CAS Google Scholar
First citationDeringer, V. L., Bartók, A. P., Bernstein, N., Wilkins, D. M., Ceriotti, M. & Csányi, G. (2021). Chem. Rev. 121, 10073–10141.  CrossRef CAS PubMed Google Scholar
First citationDevereux, C., Smith, J. S., Huddleston, K. K., Barros, K., Zubatyuk, R., Isayev, O. & Roitberg, A. E. (2020). J. Chem. Theory Comput. 16, 4192–4202.  CrossRef CAS PubMed Google Scholar
First citationDolgonos, G. A., Hoja, J. & Boese, A. D. (2019). Phys. Chem. Chem. Phys. 21, 24333–24344.  Web of Science CrossRef CAS PubMed Google Scholar
First citationEhlert, S., Huniar, U., Ning, J., Furness, J. W., Sun, J., Kaplan, A. D., Perdew, J. P. & Brandenburg, J. G. (2021). J. Chem. Phys. 154, 061101.  CrossRef PubMed Google Scholar
First citationvan Eijck, B. P. (2001). J. Comput. Chem. 22, 816–826.  CrossRef CAS Google Scholar
First citationvan Eijck, B. P. & Kroon, J. (1997). J. Phys. Chem. B, 101, 1096–1100.  CrossRef CAS Google Scholar
First citationEike, D. M., Brennecke, J. F. & Maginn, E. J. (2005). J. Chem. Phys. 122, 014115.  CrossRef Google Scholar
First citationEwald, P. P. (1921). Ann. Phys. 369, 253–287.  CrossRef Google Scholar
First citationFiraha, D., Liu, Y. M., van de Streek, J., Sasikumar, K., Dietrich, H., Helfferich, J., Aerts, L., Braun, D. E., Broo, A., DiPasquale, A. G., Lee, A. Y., Le Meur, S., Nilsson Lill, S. O., Lunsmann, W. J., Mattei, A., Muglia, P., Putra, O. D., Raoui, M., Reutzel-Edens, S. M., Rome, S., Sheikh, A. Y., Tkatchenko, A., Woollam, G. R. & Neumann, M. A. (2023). Nature, 623, 324–328.  Web of Science CrossRef CAS PubMed Google Scholar
First citationFrenkel, D. & Ladd, A. J. (1984). J. Chem. Phys. 81, 3188–3193.  CrossRef CAS Google Scholar
First citationFrenkel, D. & Smit, B. (2001). Understanding Molecular Simulation: from Algorithms to Applications. Elsevier.  Google Scholar
First citationFulem, M., Růžička, K., Červinka, C., Bazyleva, A. & Della Gatta, G. (2014). Fluid Phase Equilib. 371, 93–105.  CrossRef CAS Google Scholar
First citationFultz, B. (2010). Prog. Mater. Sci. 55, 247–352.  Web of Science CrossRef CAS Google Scholar
First citationGale, J. D. & Rohl, A. L. (2003). Mol. Simul. 29, 291–341.  Web of Science CrossRef CAS Google Scholar
First citationGao, X., Ramezanghorbani, F., Isayev, O., Smith, J. S. & Roitberg, A. E. (2020). J. Chem. Inf. Model. 60, 3408–3415.  CrossRef CAS PubMed Google Scholar
First citationGasteiger, J. & Marsili, M. (1978). Tetrahedron Lett. 19, 3181–3184.  CrossRef Google Scholar
First citationGaus, M., Goez, A. & Elstner, M. (2013). J. Chem. Theory Comput. 9, 338–354.  Web of Science CrossRef CAS PubMed Google Scholar
First citationGavezzotti, A. (2002). J. Phys. Chem. B, 106, 4145–4154.  Web of Science CrossRef CAS Google Scholar
First citationGavezzotti, A. (2005). Z. Kristallogr. Cryst. Mater. 220, 499–510.  Web of Science CrossRef CAS Google Scholar
First citationGavezzotti, A. & Filippini, G. (1995). J. Am. Chem. Soc. 117, 12299–12305.  CrossRef CAS Web of Science Google Scholar
First citationGelder, R. de, Wehrens, R. & Hageman, J. A. (2001). J. Comput. Chem. 22, 273–289.  Web of Science CrossRef Google Scholar
First citationGilat, G. & Alder, B. J. (1976). Editors. Methods in Computational Physics: Vibrational Properties of Solids of Advances in Research and Applications. Academic Press.  Google Scholar
First citationGrau-Crespo, R. & Hamad, S. (2015). In Proceedings of MOL2NET, International Conference on Multidisciplinary Sciences. MDPI. https://dx.doi.org/10.3390/MOL2NET-1-c002Google Scholar
First citationGray, A. E., Day, G. M., Leslie, M. & Price, S. L. (2004). Mol. Phys. 102, 1067–1083.  Web of Science CrossRef CAS Google Scholar
First citationGreenwell, C. & Beran, G. J. (2020). Cryst. Growth Des. 20, 4875–4881.  Web of Science CrossRef CAS Google Scholar
First citationGreenwell, C., Řezáč, J. & Beran, G. J. (2022). Phys. Chem. Chem. Phys. 24, 3695–3712.  CrossRef CAS PubMed Google Scholar
First citationGrimme, S. (2006). J. Comput. Chem. 27, 1787–1799.  Web of Science CrossRef PubMed CAS Google Scholar
First citationGrimme, S., Antony, J., Ehrlich, S. & Krieg, H. (2010). J. Chem. Phys. 132, 154104.  Web of Science CrossRef PubMed Google Scholar
First citationGrimme, S., Ehrlich, S. & Goerigk, L. (2011). J. Comput. Chem. 32, 1456–1465.  Web of Science CrossRef CAS PubMed Google Scholar
First citationGrisafi, A. & Ceriotti, M. (2019). J. Chem. Phys. 151, 204105.  CrossRef PubMed Google Scholar
First citationGroom, C. R., Bruno, I. J., Lightfoot, M. P. & Ward, S. C. (2016). Acta Cryst. B72, 171–179.  Web of Science CrossRef IUCr Journals Google Scholar
First citationHabgood, M., Grau-Crespo, R. & Price, S. L. (2011). Phys. Chem. Chem. Phys. 13, 9590.  CrossRef PubMed Google Scholar
First citationHead, J. D. & Zerner, M. C. (1985). Chem. Phys. Lett. 122, 264–270.  CrossRef CAS Google Scholar
First citationHeit, Y. N. & Beran, G. J. O. (2016). Acta Cryst. B72, 514–529.  Web of Science CrossRef IUCr Journals Google Scholar
First citationHeit, Y. N., Nanda, K. D. & Beran, G. J. O. (2016). Chem. Sci. 7, 246–255.  Web of Science CrossRef CAS PubMed Google Scholar
First citationHerbert, J. M. (2019). J. Chem. Phys. 151, 170901.  Web of Science CrossRef PubMed Google Scholar
First citationHjorth Larsen, A., Jørgen Mortensen, J., Blomqvist, J., Castelli, I. E., Christensen, R., Dułak, M., Friis, J., Groves, M. N., Hammer, B., Hargus, C., Hermes, E. D., Jennings, P. C., Bjerre Jensen, P., Kermode, J., Kitchin, J. R., Leonhard Kolsbjerg, E., Kubal, J., Kaasbjerg, K., Lysgaard, S., Bergmann Maronsson, J., Maxson, T., Olsen, T., Pastewka, L., Peterson, A., Rostgaard, C., Schiøtz, J., Schütt, O., Strange, M., Thygesen, K. S., Vegge, T., Vilhelmsen, L., Walter, M., Zeng, Z. & Jacobsen, K. W. (2017). J. Phys. Condens. Matter, 29, 273002.  Web of Science CrossRef PubMed Google Scholar
First citationHofmann, D. W. M. & Apostolakis, J. (2003). J. Mol. Struct. 647, 17–39.  Web of Science CrossRef CAS Google Scholar
First citationHoja, J., Ko, H.-Y., Neumann, M. A., Car, R., DiStasio, R. A. Jr & Tkatchenko, A. (2019). Sci. Adv. 5, eaau3338.  Web of Science CrossRef PubMed Google Scholar
First citationHoja, J., List, A. & Boese, A. D. (2024). J. Chem. Theory Comput. 20, 357–367.  CrossRef CAS PubMed Google Scholar
First citationHoja, J., Reilly, A. M. & Tkatchenko, A. (2017). Wiley Interdiscip. Rev. Comput. Mol. Sci. 7, e1294.  CrossRef Google Scholar
First citationHoja, J. & Tkatchenko, A. (2018). Faraday Discuss. 211, 253–274.  CrossRef CAS PubMed Google Scholar
First citationHourahine, B., Aradi, B., Blum, V., Bonafé, F., Buccheri, A., Camacho, C., Cevallos, C., Deshaye, M. Y., Dumitrică, T., Dominguez, A., Ehlert, S., Elstner, M., van der Heide, T., Hermann, J., Irle, S., Kranz, J. J., Köhler, C., Kowalczyk, T., Kubař, T., Lee, I. S., Lutsker, V., Maurer, R. J., Min, S. K., Mitchell, I., Negre, C., Niehaus, T. A., Niklasson, A. M. N., Page, A. J., Pecchia, A., Penazzi, G., Persson, M. P., Řezáč, J., Sánchez, C. G., Sternberg, M., Stöhr, M., Stuckenberg, F., Tkatchenko, A., Yu, V. W.-Z. & Frauenheim, T. (2020). J. Chem. Phys. 152, 124101.  CrossRef PubMed Google Scholar
First citationHunnisett, L. M., Nyman, J., Francia, N., Abraham, N. S., Adjiman, C. S., Aitipamula, S., Alkhidir, T., Almehairbi, M., Anelli, A., Anstine, D. M., Anthony, J. E., Arnold, J. E., Bahrami, F., Bellucci, M. A., Bhardwaj, R. M., Bier, I., Bis, J. A., Boese, A. D., Bowskill, D. H., Bramley, J., Brandenburg, J. G., Braun, D. E., Butler, P. W. V., Cadden, J., Carino, S., Chan, E. J., Chang, C., Cheng, B., Clarke, S. M., Coles, S. J., Cooper, R. I., Couch, R., Cuadrado, R., Darden, T., Day, G. M., Dietrich, H., Ding, Y., DiPasquale, A., Dhokale, B., van Eijck, B. P., Elsegood, M. R. J., Firaha, D., Fu, W., Fukuzawa, K., Glover, J., Goto, H., Greenwell, C., Guo, R., Harter, J., Helfferich, J., Hofmann, D. W. M., Hoja, J., Hone, J., Hong, R., Hutchison, G., Ikabata, Y., Isayev, O., Ishaque, O., Jain, V., Jin, Y., Jing, A., Johnson, E. R., Jones, I., Jose, K. V. J., Kabova, E. A., Keates, A., Kelly, P. F., Khakimov, D., Konstantinopoulos, S., Kuleshova, L. N., Li, H., Lin, X., List, A., Liu, C., Liu, Y. M., Liu, Z., Liu, Z.-P., Lubach, J. W., Marom, N., Maryewski, A. A., Matsui, H., Mattei, A., Mayo, R. A., Melkumov, J. W., Mohamed, S., Momenzadeh Abardeh, Z., Muddana, H. S., Nakayama, N., Nayal, K. S., Neumann, M. A., Nikhar, R., Obata, S., O'Connor, D., Oganov, A. R., Okuwaki, K., Otero-de-la-Roza, A., Pantelides, C. C., Parkin, S., Pickard, C. J., Pilia, L., Pivina, T., Podeszwa, R., Price, A. J. A., Price, L. S., Price, S. L., Probert, M. R., Pulido, A., Ramteke, G. R., Rehman, A. U., Reutzel-Edens, S. M., Rogal, J., Ross, M. J., Rumson, A. F., Sadiq, G., Saeed, Z. M., Salimi, A., Salvalaglio, M., Sanders de Almada, L., Sasikumar, K., Sekharan, S., Shang, C., Shankland, K., Shinohara, K., Shi, B., Shi, X., Skillman, A. G., Song, H., Strasser, N., van de Streek, J., Sugden, I. J., Sun, G., Szalewicz, K., Tan, B. I., Tan, L., Tarczynski, F., Taylor, C. R., Tkatchenko, A., Tom, R., Tuckerman, M. E., Utsumi, Y., Vogt-Maranto, L., Weatherston, J., Wilkinson, L. J., Willacy, R. D., Wojtas, L., Woollam, G. R., Yang, Z., Yonemochi, E., Yue, X., Zeng, Q., Zhang, Y., Zhou, T., Zhou, Y., Zubatyuk, R. & Cole, J. C. (2024). Acta Cryst. B80, https://doi.org/10.1107/S2052520624007492.  Google Scholar
First citationIuzzolino, L., McCabe, P., Price, S. L. & Brandenburg, J. G. (2018). Faraday Discuss. 211, 275–296.  CrossRef CAS PubMed Google Scholar
First citationJorgensen, W. L. S. M. D., Maxwell, D. S. & Tirado-Rives, J. (1996). J. Am. Chem. Soc. 118, 11225–11236.  CrossRef CAS Google Scholar
First citationKamencek, T., Wieser, S., Kojima, H., Bedoya-Martínez, N., Dürholt, J. P., Schmid, R. & Zojer, E. (2020). J. Chem. Theory Comput. 16, 2716–2735.  CrossRef CAS PubMed Google Scholar
First citationKazantsev, A., Karamertzanis, P., Pantelides, C. & Adjiman, C. (2010). Process Systems Engineering, Vol. 6: Molecular Systems Engineering, pp. 1–42.  Google Scholar
First citationKazantsev, A. V., Karamertzanis, P. G., Adjiman, C. S., Pantelides, C. C., Price, S. L., Galek, P. T. A., Day, G. M. & Cruz-Cabeza, A. J. (2011). Int. J. Pharm. 418, 168–178.  Web of Science CrossRef CAS PubMed Google Scholar
First citationKendrick, J., Leusen, F. J. J., Neumann, M. A. & van de Streek, J. (2011). Chem. A Eur. J. 17, 10736–10744.  Web of Science CrossRef CAS Google Scholar
First citationKitaura, K., Ikeo, E., Asada, T., Nakano, T. & Uebayasi, M. (1999). Chem. Phys. Lett. 313, 701–706.  Web of Science CrossRef CAS Google Scholar
First citationKlimeš, J., Bowler, D. R. & Michaelides, A. (2009). J. Phys. Condens. Matter, 22, 022201.  PubMed Google Scholar
First citationKo, T. W., Finkler, J. A., Goedecker, S. & Behler, J. (2021). Nat. Commun. 12, 398.  CrossRef PubMed Google Scholar
First citationKresse, G. & Joubert, D. (1999). Phys. Rev. B, 59, 1758–1775.  Web of Science CrossRef CAS Google Scholar
First citationLeBlanc, L. M., Dale, S. G., Taylor, C. R., Becke, A. D., Day, G. M. & Johnson, E. R. (2018). Angew. Chem. 130, 15122–15126.  CrossRef Google Scholar
First citationLiu, D. C. & Nocedal, J. (1989). Math. Program. 45, 503–528.  CrossRef Web of Science Google Scholar
First citationLoboda, O. A., Dolgonos, G. A. & Boese, A. D. (2018). J. Chem. Phys. 149, 124104.  CrossRef PubMed Google Scholar
First citationLommerse, J. P. M., Motherwell, W. D. S., Ammon, H. L., Dunitz, J. D., Gavezzotti, A., Hofmann, D. W. M., Leusen, F. J. J., Mooij, W. T. M., Price, S. L., Schweizer, B., Schmidt, M. U., van Eijck, B. P., Verwer, P. & Williams, D. E. (2000). Acta Cryst. B56, 697–714.  Web of Science CSD CrossRef CAS IUCr Journals Google Scholar
First citationMacrae, C. F., Sovago, I., Cottrell, S. J., Galek, P. T. A., McCabe, P., Pidcock, E., Platings, M., Shields, G. P., Stevens, J. S., Towler, M. & Wood, P. A. (2020). J. Appl. Cryst. 53, 226–235.  Web of Science CrossRef CAS IUCr Journals Google Scholar
First citationMattei, A., Hong, R. S., Dietrich, H., Firaha, D., Helfferich, J., Liu, Y. M., Sasikumar, K., Abraham, N. S., Miglani Bhardwaj, R., Neumann, M. A. & Sheikh, A. Y. (2022). J. Chem. Theory Comput. 18, 5725–5738.  CrossRef CAS PubMed Google Scholar
First citationMayo, R. A. & Johnson, E. R. (2021). CrystEngComm, 23, 7118–7131.  Web of Science CrossRef CAS Google Scholar
First citationMayo, R. A., Otero-de-la-Roza, A. & Johnson, E. R. (2022). CrystEngComm, 24, 8326–8338.  Web of Science CrossRef CAS Google Scholar
First citationMayo, S. L., Olafson, B. D. & Goddard, W. A. (1990). J. Phys. Chem. 94, 8897–8909.  CrossRef CAS Web of Science Google Scholar
First citationMejía-Rodríguez, D. & Trickey, S. B. (2019). J. Chem. Phys. 151, 207101.  PubMed Google Scholar
First citationMennucci, B., Tomasi, J., Cammi, R., Cheeseman, J. R., Frisch, M. J., Devlin, F. J., Gabriel, S. & Stephens, P. J. (2002). J. Phys. Chem. A, 106, 6102–6113..  Web of Science CrossRef CAS Google Scholar
First citationMetz, M. P., Piszczatowski, K. & Szalewicz, K. (2016). J. Chem. Theory Comput. 12, 5895–5919.  CrossRef CAS PubMed Google Scholar
First citationMisquitta, A. J., Podeszwa, R., Jeziorski, B. & Szalewicz, K. (2005). J. Chem. Phys. 123, 214103.  Web of Science CrossRef PubMed Google Scholar
First citationMochizuki, Y., Koikegami, S., Nakano, T., Amari, S. & Kitaura, K. (2004a). Chem. Phys. Lett. 396, 473–479.  CrossRef CAS Google Scholar
First citationMochizuki, Y., Nakano, T., Koikegami, S., Tanimori, S., Abe, Y., Nagashima, U. & Kitaura, K. (2004b). Theor. Chem. Acc. 112, 442–452.  CrossRef CAS Google Scholar
First citationMoellmann, J. & Grimme, S. (2014). J. Phys. Chem. C, 118, 7615–7621.  Web of Science CrossRef CAS Google Scholar
First citationMomenzadeh Abardeh, Z., Salimi, A. & Oganov, A. R. (2022). CrystEngComm, 24, 6066–6075.  CSD CrossRef CAS Google Scholar
First citationMonacelli, L., Bianco, R., Cherubini, M., Calandra, M., Errea, I. & Mauri, F. (2021). J. Phys. Condens. Matter, 33, 363001.  CrossRef Google Scholar
First citationMooij, W. T. M. & Leusen, F. J. J. (2001). Phys. Chem. Chem. Phys. 3, 5063–5066.  CrossRef CAS Google Scholar
First citationMortazavi, M., Brandenburg, J. G., Maurer, R. J. & Tkatchenko, A. (2018). J. Phys. Chem. Lett. 9, 399–405.  CrossRef CAS PubMed Google Scholar
First citationMotherwell, W. D. S., Ammon, H. L., Dunitz, J. D., Dzyabchenko, A., Erk, P., Gavezzotti, A., Hofmann, D. W. M., Leusen, F. J. J., Lommerse, J. P. M., Mooij, W. T. M., Price, S. L., Scheraga, H., Schweizer, B., Schmidt, M. U., van Eijck, B. P., Verwer, P. & Williams, D. E. (2002). Acta Cryst. B58, 647–661.  Web of Science CrossRef CAS IUCr Journals Google Scholar
First citationMusil, F., De, S., Yang, J., Campbell, J. E., Day, G. M. & Ceriotti, M. (2018). Chem. Sci. 9, 1289–1300.  Web of Science CrossRef CAS PubMed Google Scholar
First citationNakano, T., Kaminuma, T., Sato, T., Fukuzawa, K., Akiyama, Y., Uebayasi, M. & Kitaura, K. (2002). Chem. Phys. Lett. 351, 475–480.  Web of Science CrossRef CAS Google Scholar
First citationNeumann, M. A. (2008). J. Phys. Chem. B, 112, 9810–9829.  Web of Science CrossRef PubMed CAS Google Scholar
First citationNeumann, M. A., Leusen, F. J. J. & Kendrick, J. (2008). Angew. Chem. Int. Ed. 47, 2427–2430.  Web of Science CrossRef CAS Google Scholar
First citationNeumann, M. A. & Perrin, M.-A. (2005). J. Phys. Chem. B, 109, 15531–15541.  Web of Science CrossRef PubMed CAS Google Scholar
First citationNeumann, M. & van de Streek, J. (2018). Faraday Discuss. 211, 441–458.  Web of Science CrossRef CAS PubMed Google Scholar
First citationNikhar, R. & Szalewicz, K. (2022). Nat. Commun. 13, 3095.  Web of Science CrossRef PubMed Google Scholar
First citationNyman, J. & Day, G. M. (2015). CrystEngComm, 17, 5154–5165.  Web of Science CrossRef CAS Google Scholar
First citationNyman, J., Pundyke, O. S. & Day, G. M. (2016). Phys. Chem. Chem. Phys. 18, 15828–15837.  Web of Science CrossRef CAS PubMed Google Scholar
First citationNyman, J. & Reutzel-Edens, S. M. (2018). Faraday Discuss. 211, 459–476.  Web of Science CrossRef CAS PubMed Google Scholar
First citationNyman, J., Yu, L. & Reutzel-Edens, S. M. (2019). CrystEngComm, 21, 2080–2088.  Web of Science CSD CrossRef CAS Google Scholar
First citationO'Connor, D., Bier, I., Hsieh, Y.-T. & Marom, N. (2022). J. Chem. Theory Comput. 18, 4456–4471.  CAS PubMed Google Scholar
First citationO'Connor, D., Bier, I., Tom, R., Hiszpanski, A. M., Steele, B. A. & Marom, N. (2023). Cryst. Growth Des. 23, 6275–6289.  Web of Science CAS PubMed Google Scholar
First citationOtero-de-la-Roza, A. & Johnson, E. R. (2012). J. Chem. Phys. 137, 054103.  Web of Science PubMed Google Scholar
First citationOtero-de-la-Roza, A. & Johnson, E. R. (2013). J. Chem. Phys. 138, 054103.  PubMed Google Scholar
First citationOtero-de-la-Roza, A., LeBlanc, L. M. & Johnson, E. R. (2019). J. Chem. Theory Comput. 15, 4933–4944.  CAS PubMed Google Scholar
First citationPackwood, D., Kermode, J., Mones, L., Bernstein, N., Woolley, J., Gould, N., Ortner, C. & Csányi, G. (2016). J. Chem. Phys. 144, 164109.  CrossRef PubMed Google Scholar
First citationPerdew, J. P. & Schmidt, K. (2001). AIP Conf. Proc. 577, 1–20.  CrossRef CAS Google Scholar
First citationPerdew, J. P., Burke, K. & Ernzerhof, M. (1996a). Phys. Rev. Lett. 77, 3865–3868.  CrossRef PubMed CAS Google Scholar
First citationPerdew, J. P., Ernzerhof, M. & Burke, K. (1996b). J. Chem. Phys. 105, 9982–9985.  CrossRef CAS Google Scholar
First citationPerdew, J. P. & Zunger, A. (1981). Phys. Rev. B, 23, 5048–5079.  CrossRef CAS Web of Science Google Scholar
First citationPokorný, V., Touš, P., Štejfa, V., Růžička, K., Rohlíček, J., Czernek, J., Brus, J. & Červinka, C. (2022). Phys. Chem. Chem. Phys. 24, 25904–25917.  PubMed Google Scholar
First citationPrice, A. J. A., Mayo, R. A., Otero-de-la-Roza, A. & Johnson, E. R. (2023b). CrystEngComm, 25, 953–960.  CrossRef CAS Google Scholar
First citationPrice, A. J., Otero-de-la-Roza, A. & Johnson, E. R. (2023a). Chem. Sci. 14, 1252–1262.  CrossRef CAS PubMed Google Scholar
First citationPrice, S. L. (2013). Acta Cryst. B69, 313–328.  Web of Science CrossRef CAS IUCr Journals Google Scholar
First citationPrice, S. L., Leslie, M., Welch, G. W. A., Habgood, M., Price, L. S., Karamertzanis, P. G. & Day, G. M. (2010). Phys. Chem. Chem. Phys. 12, 8478.  Web of Science CrossRef PubMed Google Scholar
First citationPyzer-Knapp, E. O., Thompson, H. P. G. & Day, G. M. (2016). Acta Cryst. B72, 477–487.  Web of Science CrossRef IUCr Journals Google Scholar
First citationRana, B., Beran, G. J. O. & Herbert, J. M. (2022). Mol. Phys. 121, e2138789.  CrossRef Google Scholar
First citationReilly, A. M. & Tkatchenko, A. (2013). J. Chem. Phys. 139, 024705.  Web of Science CrossRef PubMed Google Scholar
First citationReilly, A. M. & Tkatchenko, A. (2015). Chem. Sci. 6, 3289–3301.  Web of Science CrossRef CAS PubMed Google Scholar
First citationReilly, A. M., Cooper, R. I., Adjiman, C. S., Bhattacharya, S., Boese, A. D., Brandenburg, J. G., Bygrave, P. J., Bylsma, R., Campbell, J. E., Car, R., Case, D. H., Chadha, R., Cole, J. C., Cosburn, K., Cuppen, H. M., Curtis, F., Day, G. M., DiStasio, R. A. Jr, Dzyabchenko, A., van Eijck, B. P., Elking, D. M., van den Ende, J. A., Facelli, J. C., Ferraro, M. B., Fusti-Molnar, L., Gatsiou, C.-A., Gee, T. S., de Gelder, R., Ghiringhelli, L. M., Goto, H., Grimme, S., Guo, R., Hofmann, D. W. M., Hoja, J., Hylton, R. K., Iuzzolino, L., Jankiewicz, W., de Jong, D. T., Kendrick, J., de Klerk, N. J. J., Ko, H.-Y., Kuleshova, L. N., Li, X., Lohani, S., Leusen, F. J. J., Lund, A. M., Lv, J., Ma, Y., Marom, N., Masunov, A. E., McCabe, P., McMahon, D. P., Meekes, H., Metz, M. P., Misquitta, A. J., Mohamed, S., Monserrat, B., Needs, R. J., Neumann, M. A., Nyman, J., Obata, S., Oberhofer, H., Oganov, A. R., Orendt, A. M., Pagola, G. I., Pantelides, C. C., Pickard, C. J., Podeszwa, R., Price, L. S., Price, S. L., Pulido, A., Read, M. G., Reuter, K., Schneider, E., Schober, C., Shields, G. P., Singh, P., Sugden, I. J., Szalewicz, K., Taylor, C. R., Tkatchenko, A., Tuckerman, M. E., Vacarro, F., Vasileiadis, M., Vazquez-Mayagoitia, A., Vogt, L., Wang, Y., Watson, R. E., de Wijs, G. A., Yang, J., Zhu, Q. & Groom, C. R. (2016). Acta Cryst. B72, 439–459.  Web of Science CrossRef IUCr Journals Google Scholar
First citationŘezáč, J. (2017). J. Chem. Theory Comput. 13, 4804–4817.  PubMed Google Scholar
First citationŘezáč, J. (2019). J. Comput. Chem. 40, 1633–1642.  PubMed Google Scholar
First citationRossi, M., Gasparotto, P. & Ceriotti, M. (2016). Phys. Rev. Lett. 117, 115702.  CrossRef PubMed Google Scholar
First citationSacchi, P., Lusi, M., Cruz-Cabeza, A. J., Nauha, E. & Bernstein, J. (2020). CrystEngComm, 22, 7170–7185.  Web of Science CrossRef CAS Google Scholar
First citationSarma, J. & Desiraju, G. R. (2002). Cryst. Growth Des. 2, 93–100.  Web of Science CrossRef CAS Google Scholar
First citationSchröder, E., Cooper, V. R., Berland, K., Lundqvist, B. I., Hyldgaard, P. & Thonhauser, T. (2017). In Non-Covalent Interactions in Quantum Chemistry and Physics, edited by A. Otero-de-la-Roza & G. A. DiLabio, pp. 241–274. Elsevier.  Google Scholar
First citationSpath, H. (1980). Cluster Analysis Algorithms for Data Reduction and Classification of Objects. Chichester: Ellis Horwood.  Google Scholar
First citationStone, A. J. (1981). Chem. Phys. Lett. 83, 233–239.  CrossRef CAS Web of Science Google Scholar
First citationStone, A. J. & Price, S. L. (1988). J. Phys. Chem. 92, 3325–3335.  CrossRef CAS Google Scholar
First citationSun, H. (1998). J. Phys. Chem. B, 102, 7338–7364.  Web of Science CrossRef CAS Google Scholar
First citationSun, J., Ruzsinszky, A. & Perdew, J. P. (2015). Phys. Rev. Lett. 115, 036402.  Web of Science CrossRef PubMed Google Scholar
First citationTao, J., Perdew, J. P., Staroverov, V. N. & Scuseria, G. E. (2003). Phys. Rev. Lett. 91, 146401.  Web of Science CrossRef PubMed Google Scholar
First citationTkatchenko, A., DiStasio, R. A., Car, R. & Scheffler, M. (2012). Phys. Rev. Lett. 108, 236402.  Web of Science CrossRef PubMed Google Scholar
First citationTom, R., Gao, S., Yang, Y., Zhao, K., Bier, I., Buchanan, E. A., Zaykov, A., Havlas, Z., Michl, J. & Marom, N. (2023). Chem. Mater. 35, 1373–1386.  Web of Science CrossRef CAS PubMed Google Scholar
First citationTouš, P. & Červinka, C. (2023). Cryst. Growth Des. 23, 4082–4097.  Google Scholar
First citationTu, N. T. P., Rezajooei, N., Johnson, E. R. & Rowley, C. N. (2023). Digit. Discov. 2, 718–727.  CrossRef CAS Google Scholar
First citationTuckerman, M. & Galanakis, N. (2023).Topological Crystal Structure Prediction, https://doi.org/10.21203/rs.3.rs-3361974/v1Google Scholar
First citationVydrov, O. A. & van Voorhis, T. (2010). J. Chem. Phys. 33, 244103.  CrossRef Google Scholar
First citationWang, J., Wolf, R. M., Caldwell, J. W., Kollman, P. A. & Case, D. A. (2004). J. Comput. Chem. 25, 1157–1174.  Web of Science CrossRef PubMed CAS Google Scholar
First citationWelch, G. W. A., Karamertzanis, P. G., Misquitta, A. J., Stone, A. J. & Price, S. L. (2008). J. Chem. Theory Comput. 4, 522–532.  Web of Science CrossRef CAS PubMed Google Scholar
First citationWen, S. & Beran, G. J. O. (2011). J. Chem. Theory Comput. 7, 3733–3742.  Web of Science CrossRef CAS PubMed Google Scholar
First citationWhittleton, S. R., Otero-de-la-Roza, A. & Johnson, E. R. (2016). J. Chem. Theory Comput. 13, 441–450.  CrossRef Google Scholar
First citationWiddowson, D., Mosca, M. M., Pulido, A., Cooper, A. I. & Kurlin, V. (2022). Match, 87, 529–559.  Web of Science CrossRef Google Scholar
First citationWilliams, D. E. (2001a). J. Comput. Chem. 22, 1–20.  CrossRef CAS Google Scholar
First citationWilliams, D. E. (2001b). J. Comput. Chem. 22, 1154–1166..  CrossRef CAS Google Scholar
First citationWoollam, G. R., Neumann, M. A., Wagner, T. & Davey, R. J. (2018). Faraday Discuss. 211, 209–234.  Web of Science CSD CrossRef CAS PubMed Google Scholar
First citationYang, M., Dybeck, E., Sun, G., Peng, C., Samas, B., Burger, V. M., Zeng, Q., Jin, Y., Bellucci, M. A., Liu, Y., Zhang, P., Ma, J., Jiang, Y. A., Hancock, B. C., Wen, S. & Wood, G. P. F. (2020). Cryst. Growth Des. 20, 5211–5224.  Web of Science CrossRef CAS Google Scholar
First citationYu, L. (1995). J. Pharm. Sci. 84, 966–974.  CrossRef CAS PubMed Web of Science Google Scholar
First citationYu, L., Huang, J. & Jones, K. J. (2005). J. Phys. Chem. B, 109, 19915–19922.  Web of Science CrossRef PubMed CAS Google Scholar
First citationYue, S., Muniz, M. C., Calegari Andrade, M. F., Zhang, L., Car, R. & Panagiotopoulos, A. Z. (2021). J. Chem. Phys. 154, 034111.  CrossRef PubMed Google Scholar
First citationZhang, L., Wang, H., Muniz, M. C., Panagiotopoulos, A. Z., Car, R. & E, W. (2022). J. Chem. Phys. 156, 124107.  CrossRef PubMed Google Scholar
First citationZhang, P., Wood, G. P., Ma, J., Yang, M., Liu, Y., Sun, G., Jiang, Y. A., Hancock, B. C. & Wen, S. (2018). Cryst. Growth Des. 18, 6891–6900.  Web of Science CrossRef CAS Google Scholar
First citationZubatyuk, R., Smith, J. S., Nebgen, B. T., Tretiak, S. & Isayev, O. (2021). Nat. Commun. 12, 4870.  CrossRef PubMed Google Scholar

This is an open-access article distributed under the terms of the Creative Commons Attribution (CC-BY) Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are cited.

Journal logoSTRUCTURAL SCIENCE
CRYSTAL ENGINEERING
MATERIALS
ISSN: 2052-5206
Follow Acta Cryst. B
Sign up for e-alerts
Follow Acta Cryst. on Twitter
Follow us on facebook
Sign up for RSS feeds