LLM-Prop: predicting the properties of crystalline materials using large language models
Source abstract
Abstract
Abstract The prediction of crystal properties plays a crucial role in materials science and applications. Current methods for predicting crystal properties focus on modeling crystal structures using graph neural networks (GNNs). However, accurately modeling the complex interactions between atoms and molecules within a crystal remains a challenge. Surprisingly, predicting crystal properties from crystal text descriptions is understudied, despite the rich information and expressiveness that text data offer. In this paper, we develop and make public a benchmark dataset (TextEdge) that contains crystal text descriptions with their properties. We then propose LLM-Prop, a method that leverages the general-purpose learning capabilities of large language models (LLMs) to predict properties of crystals from their text descriptions. LLM-Prop outperforms the current state-of-the-art GNN-based methods by approximately 8% on predicting band gap, 3% on classifying whether the band gap is direct or indirect, and 65% on predicting unit cell volume, and yields comparable performance on predicting formation energy per atom, energy per atom, and energy above hull. LLM-Prop also outperforms the fine-tuned MatBERT, a domain-specific pre-trained BERT model, despite having 3 times fewer parameters. We further fine-tune the LLM-Prop model directly on CIF files and condensed structure information generated by Robocrystallographer and found that LLM-Prop fine-tuned on text descriptions provides a better performance on average. Our empirical results highlight the importance of having a natural language input to LLMs to accurately predict crystal properties and the current inability of GNNs to capture information pertaining to space group symmetry and Wyckoff sites for accurate crystal property prediction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Computational Drug Discovery Methods · X-ray Diffraction in Crystallography
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (47)
Show source-provided references
1. Tao, Q., Xu, P., Li, M. & Lu, W. Machine learning for perovskite materials design and discovery. npj Comput. Mater. 7, 1–18 (2021).10.1038/s41524-021-00495-8 ↗
2. Parikh, N. et al. Is machine learning redefining the perovskite solar cells? J. Energy Chem. 66, 74–90 (2022).10.1016/j.jechem.2021.07.020 ↗
3. Meredig, B. et al. Combinatorial screening for new materials in unconstrained composition space with machine learning. Phys. Rev. B 89, 094104 (2014).10.1103/PhysRevB.89.094104 ↗
4. Oliynyk, A. O. et al. High-throughput machine-learning-driven synthesis of full-heusler compounds. Chem. Mater. 28, 7324–7331 (2016).10.1021/acs.chemmater.6b02724 ↗
5. Raccuglia, P. et al. Machine-learning-assisted materials discovery using failed experiments. Nature 533, 73–76 (2016).10.1038/nature17439 ↗
6. Ward, L., Agrawal, A., Choudhary, A. & Wolverton, C. A general-purpose machine learning framework for predicting properties of inorganic materials. npj Comput. Mater. 2, 1–7 (2016).10.1038/npjcompumats.2016.28 ↗
7. Chen, C., Ye, W., Zuo, Y., Zheng, C. & Ong, S. P. Graph networks as a universal machine learning framework for molecules and crystals. Chem. Mater. 31, 3564–3572 (2019).10.1021/acs.chemmater.9b01294 ↗
8. Choudhary, K. & DeCost, B. Atomistic line graph neural network for improved materials property predictions. npj Comput. Mater. 7, 1–8 (2021).10.1038/s41524-021-00650-1 ↗
9. Chen, C. & Ong, S. P. A universal graph deep learning interatomic potential for the periodic table. Nat. Comput. Sci. 2, 718–728 (2022).10.1038/s43588-022-00349-3 ↗
10. Ioannidis, V. N., Marques, A. G. & Giannakis, G. B. Graph neural networks for predicting protein functions. In: 2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP) 221–225 (IEEE, 2019).10.1109/CAMSAP45676.2019.9022646 ↗
11. Strokach, A., Becerra, D., Corbi-Verge, C., Perez-Riba, A. & Kim, P. M. Fast and flexible protein design using deep graph neural networks. Cell Syst. 11, 402–411 (2020).10.1016/j.cels.2020.08.016 ↗
12. Jiang, M. et al. Drug–target affinity prediction using graph neural network and contact maps. RSC Adv. 10, 20701–20712 (2020).10.1039/D0RA02297G ↗
13. Shen, Z.-A., Luo, T., Zhou, Y.-K., Yu, H. & Du, P.-F. Npi-gnn: predicting ncrna–protein interactions with deep graph neural networks. Brief. Bioinform. 22, bbab051 (2021).10.1093/bib/bbab051 ↗
14. Wang, Z. et al. Advanced graph and sequence neural networks for molecular property prediction and drug discovery. Bioinformatics 38, 2579–2586 (2022).10.1093/bioinformatics/btac112 ↗
15. Jha, K., Saha, S. & Singh, H. Prediction of protein–protein interaction using graph neural networks. Sci. Rep. 12, 8360 (2022).10.1038/s41598-022-12201-9 ↗
16. R´eau, M., Renaud, N., Xue, L. C. & Bonvin, A. M. Deeprank-gnn: a graph neural network framework to learn patterns in protein–protein interfaces. Bioinformatics 39, btac759 (2023).10.1093/bioinformatics/btac759 ↗
17. Schu¨tt, K. et al. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Adv. Neural Inform. Process. Syst. 30 (2017).
18. Xie, T. & Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 145301 (2018).10.1103/PhysRevLett.120.145301 ↗
19. Jain, A. et al. A high-throughput infrastructure for density functional theory calculations. Comput. Mater. Sci. 50, 2295–2310 (2011).10.1016/j.commatsci.2011.02.023 ↗
20. De Jong, M. et al. Charting the complete elastic properties of inorganic crystalline compounds. Sci. Data 2, 1–13 (2015).
21. Kirklin, S. et al. The open quantum materials database (oqmd): assessing the accuracy of dft formation energies. npj Comput. Mater. 1, 1–15 (2015).10.1038/npjcompumats.2015.10 ↗
22. Yan, K., Liu, Y., Lin, Y. & Ji, S. Periodic graph transformers for crystal material property prediction. Adv. Neural Inf. Process. Syst. 35, 15066–15080 (2022).
23. Kaba, O. & Ravanbakhsh, S. Equivariant networks for crystal structures. Adv. Neural Inf. Process. Syst. 35, 4150–4164 (2022).
24. Beltagy, I., Lo, K. & Cohan, A. Scibert: A pretrained language model for scientific text. In: Proc. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 3615–3620 (2019).10.18653/v1/D19-1371 ↗
25. Walker, N. et al. The impact of domain-specific pre-training on named entity recognition tasks in materials science. Available at SSRN 3950755 (2021).10.2139/ssrn.3950755 ↗
26. Huang, S. & Cole, J. M. Batterybert: a pretrained language model for battery database enhancement. J. Chem. Inf. Modeling 62, 6365–6377 (2022).10.1021/acs.jcim.2c00035 ↗
27. Gupta, T., Zaki, M. & Krishnan, N. A. Matscibert: a materials domain language model for text mining and information extraction. npj Comput. Mater. 8, 102 (2022).10.1038/s41524-022-00784-w ↗
28. Korolev, V. & Protsenko, P. Accurate, interpretable predictions of materials properties within transformer language models. Patterns 4 (2023).10.1016/j.patter.2023.100803 ↗
29. Qu, J., Xie, Y.R., Ciesielski, K.M. et al. Leveraging language representation for materials exploration and discovery. npj Comput. Mater. 10, 58 (2024).10.1038/s41524-024-01231-8 ↗
30. Song, Y., Miret, S. & Liu, B. Matsci-nlp: Evaluating scientific language models on materials science language tasks using text-to-schema modeling. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (eds Rogers, A., Boyd-Graber, J. & Okazaki, N) (Volume 1: Long Papers) 3621–3639 (Association for Computational Linguistics, 2023).10.18653/v1/2023.acl-long.201 ↗
31. Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 5485–5551 (2020).
32. Devlin, J., Chang, M., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics (2019).
33. Joshi, M. et al. Spanbert: Improving pre-training by representing and predicting spans. Trans. Assoc. computational Linguist. 8, 64–77 (2020).10.1162/tacl_a_00300 ↗
34. Dieng, A. B., Ruiz, F. J. & Blei, D. M. Topic modeling in embedding spaces. Trans. Assoc. Computational Linguist. 8, 439–453 (2020).10.1162/tacl_a_00325 ↗
35. Niyongabo, R. A., Hong, Q., Kreutzer, J. & Huang, L. Kinnews and kirnews: Benchmarking cross-lingual text classification for kinyarwanda and kirundi. In: Proceedings of the 28th International Conference on Computational Linguistics (eds Scott, D., Bel, N. & Zong, C) 5507–5521 (International Committee on Computational Linguistics, 2020).10.18653/v1/2020.coling-main.480 ↗
36. Wallace, E., Wang, Y., Li, S., Singh, S. & Gardner, M. Do nlp models know numbers? probing numeracy in embeddings. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (eds Inui, K., Jiang, J., Ng, V. & Xiaojun Wan) (EMNLP-IJCNLP) 5307–5315 (Association for Computational Linguistics, 2019).10.18653/v1/D19-1534 ↗
37. Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y. & Roth, D. Do language embeddings capture scales? 4889–4896 (2020).10.18653/v1/2020.findings-emnlp.439 ↗
38. Geva, M., Gupta, A. & Berant, J. Injecting Numerical Reasoning Skills into Language Models. In Proc. 58th Annual Meeting of the Association for Computational Linguistics, 946–958, Online. (Association for Computational Linguistics, 2020).10.18653/v1/2020.acl-main.89 ↗
39. Thawani, A., Pujara, J., Ilievski, F. & Szekely, P. Representing Numbers in NLP: a Survey and a Vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–656, Online. Association for Computational Linguistics (2021).10.18653/v1/2021.naacl-main.53 ↗
40. Jain, A. et al. Commentary: The materials project: a materials genome approach to accelerating materials innovation. APL Mater. 1 (2013).10.1063/1.4812323 ↗
41. Ganose, A. M. & Jain, A. Robocrystallographer: automated crystal structure text descriptions and analysis. MRS Commun. 9, 874–881 (2019).10.1557/mrc.2019.94 ↗
42. Golkar, S. et al. xval: a continuous number encoding for large language models. In: NeurIPS 2023 AI for Science Workshop (2023).
43. Chen, T. & Guestrin, C. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). Association for Computing Machinery, New York, NY, USA, 785–794. https://doi.org/10.1145/2939672.2939785 (2016).10.1145/2939672.2939785 ↗
44. Breiman, L. Random forests. Mach. Learn. 45, 5–32 (2001).10.1023/A:1010933404324 ↗
45. Omee, S. S. et al. Scalable deeper graph neural networks for high-performance materials property prediction. Patterns 3 (2022).10.1016/j.patter.2022.100491 ↗
46. Smith, Leslie N., & Topin, N. "Super-convergence: Very fast training of neural networks using large learning rates." Artificial intelligence and machine learning for multi-domain operations applications. Vol. 11006. (SPIE, 2019).10.1117/12.2520589 ↗
47. Vaswani, A. et al. Attention is all you need. Adv. Neural Inform. Process. Syst. 30 (2017).