MagBERT: Magnetics Knowledge Aware Language Model Coupled with a Question Answering Pipeline for Curie Temperature Extraction Task
Source abstract
Abstract
High Resolution Image Download MS PowerPoint Slide In this study, we develop and release two Bidirectional Encoder Representations (BERT) models that are trained primarily with roughly ≈144 K peer-reviewed publications within the magnetic materials domain, which we refer to as MagBERT and MagMatBERT, respectively. These transformer models are then used in chemical named entity recognition (CNER) and question answering (QA) tasks in a data extraction workflow. We demonstrate this approach by developing a magnetics data set of well-known magnetic property, i.e., Curie temperature T C . We evaluate the efficacy of these models along with the ones that were not trained with magnetics corpus in CNER and QA tasks. Our results indicate that an initial training using the magnetics corpus brings about an enhancement in these tasks. Additionally, the quality of each data set is assessed by comparing them with a manually developed ground truth T C data set as well as by employing a random forest (RF) model in its predictive ability of T C as the target quantity. Our analyses demonstrate that models pretrained with magnetic corpus, i.e., MagBERT and MagMatBERT are more efficient than the ones without pretraining.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Topic Modeling · Nuclear Physics and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Machine Learning Department, Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Masdar City, Abu Dhabi, 00000, UAE
School of Physics, AMBER and CRANN Institute, Trinity College Dublin, Dublin 2, D02 W085, Ireland
Chemical and Materials Engineering Department, California State Polytechnic University, Pomona, California 91768, United States
Funding
Irish Research Council · Fulbright U.S. Scholar Program
References (40)
Show source-provided references
1. 10.1016/j.eng.2018.11.03410.1016/j.eng.2018.11.034 ↗
2. 10.1007/s11664-013-2725-610.1007/s11664-013-2725-6 ↗
3. 10.1063/1.283097510.1063/1.2830975 ↗
4. 10.1007/s11837-018-3291-410.1007/s11837-018-3291-4 ↗
5. 10.1038/npjcompumats.2015.1010.1038/npjcompumats.2015.10 ↗
6. 10.1016/j.commatsci.2012.02.00210.1016/j.commatsci.2012.02.002 ↗
7. 10.1063/1.481232310.1063/1.4812323 ↗
8. 10.48550/arXiv.1711.0509910.48550/arXiv.1711.05099 ↗
9. 10.1038/s41524-019-0199-710.1038/s41524-019-0199-7 ↗
10. 10.1016/j.jallcom.2020.15526310.1016/j.jallcom.2020.155263 ↗
11. 10.1021/acs.chemmater.3c0089210.1021/acs.chemmater.3c00892 ↗
12. 10.1103/PhysRevMaterials.4.06441410.1103/PhysRevMaterials.4.064414 ↗
13. 10.1126/sciadv.160224110.1126/sciadv.1602241 ↗
14. 10.1073/pnas.220448511910.1073/pnas.2204485119 ↗
15. 10.1038/sdata.2018.11110.1038/sdata.2018.111 ↗
16. 10.1021/acs.chemmater.1c0136810.1021/acs.chemmater.1c01368 ↗
17. 10.48550/arXiv.1903.1067610.48550/arXiv.1903.10676 ↗
18. 10.1093/bioinformatics/btz68210.1093/bioinformatics/btz682 ↗
19. 10.48550/arXiv.1904.0332310.48550/arXiv.1904.03323 ↗
20. 10.48550/arXiv.1908.1006310.48550/arXiv.1908.10063 ↗
21. 10.1038/s41524-022-00784-w10.1038/s41524-022-00784-w ↗
22. 10.1016/j.patter.2022.10048810.1016/j.patter.2022.100488 ↗
23. 10.1038/s41524-023-01003-w10.1038/s41524-023-01003-w ↗
24. 10.1021/acs.jcim.2c0003510.1021/acs.jcim.2c00035 ↗
25. 10.1021/acs.jcim.2c0125910.1021/acs.jcim.2c01259 ↗
26. 10.1038/s41524-023-01171-910.1038/s41524-023-01171-9 ↗
27. 10.1088/1367-2630/15/12/12502310.1088/1367-2630/15/12/125023 ↗
28. 10.1016/j.cocom.2016.10.00110.1016/j.cocom.2016.10.001 ↗
29. 10.1103/PhysRevB.73.21441210.1103/PhysRevB.73.214412 ↗
30. 10.1109/ICCV.2015.1110.1109/ICCV.2015.11 ↗
31. 10.48550/arXiv.cs/000900810.48550/arXiv.cs/0009008 ↗
32. 10.1186/1758-2946-7-S1-S210.1186/1758-2946-7-S1-S2 ↗
33. 10.1021/acs.jcim.9b0047010.1021/acs.jcim.9b00470 ↗
34. 10.48550/arXiv.2006.0303910.48550/arXiv.2006.03039 ↗
35. 10.1186/s12859-017-1776-810.1186/s12859-017-1776-8 ↗
36. 10.48550/arXiv.1606.0525010.48550/arXiv.1606.05250 ↗
37. 10.1016/j.commatsci.2012.10.02810.1016/j.commatsci.2012.10.028 ↗
38. 10.1103/PhysRevMaterials.3.10440510.1103/PhysRevMaterials.3.104405 ↗
39. 10.1016/j.commatsci.2022.11141410.1016/j.commatsci.2022.111414 ↗
40. 10.18653/v1/2020.emnlp-demos.610.18653/v1/2020.emnlp-demos.6 ↗