journal article·2024-08-08Open access

MagBERT: Magnetics Knowledge Aware Language Model Coupled with a Question Answering Pipeline for Curie Temperature Extraction Task

Aigerim Zhumabayeva, Nikhil Ranjan, Martin Takáč, Stefano Sanvito, Huseyin Ucar

The Journal of Physical Chemistry C· Volume 128· Issue 31

Abstract

High Resolution Image Download MS PowerPoint Slide In this study, we develop and release two Bidirectional Encoder Representations (BERT) models that are trained primarily with roughly ≈144 K peer-reviewed publications within the magnetic materials domain, which we refer to as MagBERT and MagMatBERT, respectively. These transformer models are then used in chemical named entity recognition (CNER) and question answering (QA) tasks in a data extraction workflow. We demonstrate this approach by developing a magnetics data set of well-known magnetic property, i.e., Curie temperature T C . We evaluate the efficacy of these models along with the ones that were not trained with magnetics corpus in CNER and QA tasks. Our results indicate that an initial training using the magnetics corpus brings about an enhancement in these tasks. Additionally, the quality of each data set is assessed by comparing them with a manually developed ground truth T C data set as well as by employing a random forest (RF) model in its predictive ability of T C as the target quantity. Our analyses demonstrate that models pretrained with magnetic corpus, i.e., MagBERT and MagMatBERT are more efficient than the ones without pretraining.
Source: openalex_inverted_index · Quality: baseline

Research context

OpenAlex topics: Machine Learning in Materials Science · Topic Modeling · Nuclear Physics and Applications

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Machine Learning Department, Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Masdar City, Abu Dhabi, 00000, UAE

School of Physics, AMBER and CRANN Institute, Trinity College Dublin, Dublin 2, D02 W085, Ireland

Chemical and Materials Engineering Department, California State Polytechnic University, Pomona, California 91768, United States

Funding

Irish Research Council · Fulbright U.S. Scholar Program

References (40)

Show source-provided references
1. 10.1016/j.eng.2018.11.03410.1016/j.eng.2018.11.034 ↗
2. 10.1007/s11664-013-2725-610.1007/s11664-013-2725-6 ↗
3. 10.1063/1.283097510.1063/1.2830975 ↗
4. 10.1007/s11837-018-3291-410.1007/s11837-018-3291-4 ↗
5. 10.1038/npjcompumats.2015.1010.1038/npjcompumats.2015.10 ↗
6. 10.1016/j.commatsci.2012.02.00210.1016/j.commatsci.2012.02.002 ↗
7. 10.1063/1.481232310.1063/1.4812323 ↗
8. 10.48550/arXiv.1711.0509910.48550/arXiv.1711.05099 ↗
9. 10.1038/s41524-019-0199-710.1038/s41524-019-0199-7 ↗
10. 10.1016/j.jallcom.2020.15526310.1016/j.jallcom.2020.155263 ↗
11. 10.1021/acs.chemmater.3c0089210.1021/acs.chemmater.3c00892 ↗
12. 10.1103/PhysRevMaterials.4.06441410.1103/PhysRevMaterials.4.064414 ↗
13. 10.1126/sciadv.160224110.1126/sciadv.1602241 ↗
14. 10.1073/pnas.220448511910.1073/pnas.2204485119 ↗
15. 10.1038/sdata.2018.11110.1038/sdata.2018.111 ↗
16. 10.1021/acs.chemmater.1c0136810.1021/acs.chemmater.1c01368 ↗
17. 10.48550/arXiv.1903.1067610.48550/arXiv.1903.10676 ↗
18. 10.1093/bioinformatics/btz68210.1093/bioinformatics/btz682 ↗
19. 10.48550/arXiv.1904.0332310.48550/arXiv.1904.03323 ↗
20. 10.48550/arXiv.1908.1006310.48550/arXiv.1908.10063 ↗
21. 10.1038/s41524-022-00784-w10.1038/s41524-022-00784-w ↗
22. 10.1016/j.patter.2022.10048810.1016/j.patter.2022.100488 ↗
23. 10.1038/s41524-023-01003-w10.1038/s41524-023-01003-w ↗
24. 10.1021/acs.jcim.2c0003510.1021/acs.jcim.2c00035 ↗
25. 10.1021/acs.jcim.2c0125910.1021/acs.jcim.2c01259 ↗
26. 10.1038/s41524-023-01171-910.1038/s41524-023-01171-9 ↗
27. 10.1088/1367-2630/15/12/12502310.1088/1367-2630/15/12/125023 ↗
28. 10.1016/j.cocom.2016.10.00110.1016/j.cocom.2016.10.001 ↗
29. 10.1103/PhysRevB.73.21441210.1103/PhysRevB.73.214412 ↗
30. 10.1109/ICCV.2015.1110.1109/ICCV.2015.11 ↗
31. 10.48550/arXiv.cs/000900810.48550/arXiv.cs/0009008 ↗
32. 10.1186/1758-2946-7-S1-S210.1186/1758-2946-7-S1-S2 ↗
33. 10.1021/acs.jcim.9b0047010.1021/acs.jcim.9b00470 ↗
34. 10.48550/arXiv.2006.0303910.48550/arXiv.2006.03039 ↗
35. 10.1186/s12859-017-1776-810.1186/s12859-017-1776-8 ↗
36. 10.48550/arXiv.1606.0525010.48550/arXiv.1606.05250 ↗
37. 10.1016/j.commatsci.2012.10.02810.1016/j.commatsci.2012.10.028 ↗
38. 10.1103/PhysRevMaterials.3.10440510.1103/PhysRevMaterials.3.104405 ↗
39. 10.1016/j.commatsci.2022.11141410.1016/j.commatsci.2022.111414 ↗
40. 10.18653/v1/2020.emnlp-demos.610.18653/v1/2020.emnlp-demos.6 ↗