journal article·2024-09-06Open access

Active Learning for Biomedical Article Classification with Bag of Words and FastText Embeddings

Paweł Cichosz

Applied Sciences· Volume 14· Issue 17

Abstract

In several applications of text classification, training document labels are provided by human evaluators, and therefore, gathering sufficient data for model creation is time consuming and costly. The labeling time and effort may be reduced by active learning, in which classification models are created based on relatively small training sets, which are obtained by collecting class labels provided in response to labeling requests or queries. This is an iterative process with a sequence of models being fitted, and each of them is used to select query articles to be added to the training set for the next one. Such a learning scenario may pose different challenges for machine learning algorithms and text representation methods used for text classification than ordinary passive learning, since they have to deal with very small, often imbalanced data, and the computational expense of both model creation and prediction has to remain low. This work examines how classification algorithms and text representation methods that have been found particularly useful by prior work handle these challenges. The random forest and support vector machines algorithms are coupled with the bag of words and FastText word embedding representations and applied to datasets consisting of scientific article abstracts from systematic literature review studies in the biomedical domain. Several strategies are used to select articles for active learning queries, including uncertainty sampling, diversity sampling, and strategies favoring the minority class. Confidence-based and stability-based early stopping criteria are used to generate active learning termination signals. The results confirm that active learning is a useful approach to creating text classification models with limited access to labeled data, making it possible to save at least half of the human effort needed to assign relevant or irrelevant class labels to training articles. Two of the four examined combinations of classification algorithms and text representation methods were the most successful: the SVM algorithm with the FastText representation and the random forest algorithm with the bag of words representation. Uncertainty sampling turned out to be the most useful query selection strategy, and confidence-based stopping was found more universal and easier to configure than stability-based stopping.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Machine Learning and Algorithms · Topic Modeling · Text and Document Classification Technologies

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Institute of Computer Science, Warsaw University of Technology, 00-665 Warsaw, Poland

References (66)

Show source-provided references
1. McCallum, A., and Nigam, K. (1998, January 26–27). A Comparison of Event Models for Naive Bayes Text Classification. Proceedings of the AAAI/ICML-98 Workshop on Learning for Text Categorization, Menlo Park, CA, USA.
2. Joachims, T. (1998, January 21–23). Text Categorization with Support Vector Machines: Learning with Many Relevant Features. Proceedings of the Tenth European Conference on Machine Learning (ECML-1998), Berlin, Germany.10.1007/BFb0026683 ↗
3. Text Mining: Approaches and Applications
4. Rousseau, F., Kiagias, E., and Vazirgiannis, M. (2015, January 26–31). Text Categorization as a Graph Classification Problem. Proceedings of the Fifty-Third Annual Meeting of the Association for Computational Linguistics and the Sixth International Joint Conference on Natural Language Processing (ACL-IJCNLP-2015), Beijing, China.10.3115/v1/P15-1164 ↗
5. Ensembles of Classifiers for Parallel Categorization of Large Number of Text Documents Expressing Opinions
6. Zymkowski, T., Szymański, J., Sobecki, A., Drozda, P., Szałapak, K., Komar-Komarowski, K., and Scherer, R. (2022). Short Texts Representations for Legal Domain Classification. Proceedings of the Twenty-First International Conference on Artificial Intelligence and Soft Computing (ICAISC-2022), Springer.10.1007/978-3-031-23492-7_10 ↗
7. Automatic Text Classification to Support Systematic Reviews in Medicine10.1016/j.eswa.2013.08.047 ↗
8. Automatic Classification of Literature in Systematic Reviews on Food Safety Using Machine Learning10.1016/j.crfs.2021.12.010 ↗
9. Improving Generalization with Active Learning10.1007/BF00993277 ↗
10. Tharwat, A., and Schenck, W. (2023). A Survey on Active Learning: State-of-the-Art, Practical Challenges and Research Directions. Mathematics, 11.10.3390/math11040820 ↗
11. Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., and Kasneci, G. (2022). Deep Neural Networks and Tabular Data: A Survey. arXiv.
12. A Survey of Deep Active Learning
13. Bag of Words and Embedding Text Representation Methods for Medical Article Classification10.34768/amcs-2023-0043 ↗
14. Support Vector Machine Active Learning with Applications to Text Classification
15. Zhu, J., Wang, H., Yao, T., and Tsou, B.K. (2008, January 18–22). Active Learning with Sampling by Uncertainty and Density for Word Sense Disambiguation and Text Classification. Proceedings of the Twenty-Second International Conference on Computational Linguistics (COLING-2008), Stroudsburg, PA, USA.10.3115/1599081.1599224 ↗
16. Reducing Systematic Review Workload through Certainty-Based Screening10.1016/j.jbi.2014.06.005 ↗
17. A Novel Active Learning Method Using SVM for Text Classification10.1007/s11633-015-0912-z ↗
18. Active Learning for Biomedical Text Classification Based on Automatically Generated Regular Expressions10.1109/ACCESS.2021.3064000 ↗
19. An Open Source Machine Learning Framework for Efficient and Transparent Systematic Reviews10.1038/s42256-020-00287-7 ↗
20. Active Learning Approaches for Labeling Text: Review and Assessment of the Performance of Active Learning Approaches10.1017/pan.2020.4 ↗
21. Jacobs, P.F., Maillette de Buy Wenniger, G., Wiering, M., and Schomaker, L. (2021, January 10–12). Active Learning for Reducing Labeling Effort in Text Classification Tasks. Proceedings of the Artificial Intelligence and Machine Learning: Proceedings of the Thirty-Third Benelux on Artificial Intelligence (BNAIC/Benelearn-2021), Cham, Switzerland.10.1007/978-3-030-93842-0_1 ↗
22. Performance of Active Learning Models for Screening Prioritization in Systematic Reviews: A Simulation Study into the Average Time to Discover Relevant Record10.1186/s13643-023-02257-7 ↗
23. Teijema, J.J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., and van de Schoot, R. (2023). Active Learning-Based Systematic Reviewing Using Switching Classification Models: The Case of the Onset, Maintenance, and Relapse of Depressive Disorders. Front. Res. Metrics Anal., 8.10.3389/frma.2023.1178181 ↗
24. Reducing Workload in Systematic Review Preparation Using Automated Citation Classification10.1197/jamia.M1929 ↗
25. Random Forests10.1023/A:1010933404324 ↗
26. Support-Vector Networks10.1007/BF00994018 ↗
27. Aggarwal, C.C., and Zhai, C.X. (2012). Mining Text Data, Springer.10.1007/978-1-4614-3223-4 ↗
28. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2016). Enriching Word Vectors with Subword Information. arXiv.10.1162/tacl_a_00051 ↗
29. Dumais, S.T., Platt, J.C., Heckerman, D., and Sahami, M. (1998, January 2–7). Inductive Learning Algorithms and Representations for Text Categorization. Proceedings of the Seventh International Conference on Information and Knowledge Management (CIKM-98), Bethesda, MD, USA.10.1145/288627.288651 ↗
30. Comparative Analysis of Text Representation Methods Using Classification10.1080/01969722.2014.874828 ↗
31. Term Weighting Approaches in Automatic Text Retrieval10.1016/0306-4573(88)90021-0 ↗
32. Mikolov, T., Chen, K., Corrado, G.S., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv.
33. Schölkopf, B., Burges, C.J.C., and Smola, A.J. (1998). Fast Training of Support Vector Machines using Sequential Minimal Optimization. Advances in Kernel Methods: Support Vector Learning, MIT Press.
34. Hamel, L.H. (2009). Knowledge Discovery with Support Vector Machines, Wiley.10.1002/9780470503065 ↗
35. Smola, A.J., Barlett, P., Schölkopf, B., and Schuurmans, D. (2000). Probabilistic Outputs for Support Vector Machines and Comparison to Regularized Likelihood Methods. Advances in Large Margin Classifiers, MIT Press.10.7551/mitpress/1113.001.0001 ↗
36. Rios, G., and Zha, H. (2004, January 30–31). Exploring Support Vector Machines and Random Forests for Spam Detection. Proceedings of the First International Conference on Email and Anti Spam (CEAS-2004), Mountain View, CA, USA.
37. Bagging Predictors10.1007/BF00058655 ↗
38. Learning to Classify E-Mail
39. Xue, D., and Li, F. (2015, January 13–14). Research of Text Categorization Model Based on Random Forests. Proceedings of the 2015 IEEE International Conference on Computational Intelligence and Communication Technology (CICT-2015), Los Alamitos, CA, USA.10.1109/CICT.2015.101 ↗
40. A Case Study in Text Mining of Discussion Forum Posts: Classification with Bag of Words and Global Vectors10.2478/amcs-2018-0060 ↗
41. Yuan, W., Han, Y., Guan, D., Lee, S., and Lee, Y.K. (2011, January 21–23). Initial Training Data Selection for Active Learning. Proceedings of the Fifth International Conference on Ubiquitous Information Management and Communication (ICUIMC-2011), Seoul, Republic of Korea.10.1145/1968613.1968619 ↗
42. Bezdek, J.C. (1981). Pattern Recognition with Fuzzy Objective Function Algorithms, Springer.10.1007/978-1-4757-0450-1 ↗
43. Jolliffe, I.T. (2002). Pricipal Component Analysis, Springer.
44. A Survey on Instance Selection for Active Learning10.1007/s10115-012-0507-8 ↗
45. How to Measure Uncertainty in Uncertainty Sampling for Active Learning10.1007/s10994-021-06003-9 ↗
46. Kurlandski, L., and Bloodgood, M. (2022, January 26–28). Impact of Stop Sets on Stopping Active Learning for Text Classification. Proceedings of the Sixteenth International Conference on Semantic Computing (ICSC-2022), Los Alamitos, CA, USA.10.1109/ICSC52841.2022.00012 ↗
47. Confidence-Based Stopping Criteria for Active Learning for Data Annotation10.1145/1753783.1753784 ↗
48. Bloodgood, M., and Vijay-Shanker, K. (2009, January 4–5). A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping. Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009). Association for Computational Linguistics, Boulder, CO, USA.10.3115/1596374.1596384 ↗
49. (2024, July 01). Systematic Drug Class Review Gold Standard Data. Available online: https://dmice.ohsu.edu/cohenaa/systematic-drug-class-review-data.html.
50. Cohen, A.M. (2008, January 8–12). Optimizing Feature Representation for Automated Systematic Review Work Prioritization. Proceedings of the AMIA Annual Symposium Proceedings, AMIA, Washington, DC, USA.
51. A New Algorithm for Reducing the Workload of Experts in Performing Systematic Reviews10.1136/jamia.2010.004325 ↗
52. A New Iterative Method to Reduce Workload in Systematic Review Process10.1504/IJCBDD.2013.052198 ↗
53. Learning to Identify Relevant Studies for Systematic Reviews Using Random Forest and External Information10.1007/s10994-015-5535-7 ↗
54. Using Ontology-Based Semantic Similarity to Facilitate the Article Screening Process for Systematic Reviews10.1016/j.jbi.2017.03.007 ↗
55. Scikit-learn: Machine Learning in Python
56. Honnibal, M., Montani, I., Van Landeghem, S., and Boyd, A. (2024, July 01). spaCy: Industrial-Strength Natural Language Processing in Python. Library version 3.3.1. 2022. Available online: https://github.com/explosion/spaCy/blob/master/CITATION.cff.
57. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2024, July 01). fastText: Library for Efficient Text Classification and Representation Learning. Library version 0.9.2. 2020. Available online: https://fasttext.cc/.
58. Egan, J.P. (1975). Signal Detection Theory and ROC Analysis, Academic Press.
59. An Introduction to ROC Analysis10.1016/j.patrec.2005.10.010 ↗
60. A Survey of Cross-Validation Procedures for Model Selection10.1214/09-SS054 ↗
61. Wallace, B.C., Trikalinos, T.A., Lau, J., Brodley, C., and Schmid, C.H. (2010). Semi-Automated Screening of Biomedical Citations for Systematic Reviews. BMC Bioinform., 11.10.1186/1471-2105-11-55 ↗
62. Performance of Support-Vector-Machine-Based Classification on 15 Systematic Review Topics Evaluated with the WSS@95 Measure10.1136/jamia.2010.008177 ↗
63. SMOTE: Synthetic Minority Over-sampling Technique10.1613/jair.953 ↗
64. Training and Assessing Classification Rules with Imbalanced Data10.1007/s10618-012-0295-5 ↗
65. Zhu, X., and Goldberg, A. (2009). Introduction to Semi-Supervised Learning, Morgan & Claypool.10.1007/978-3-031-01548-9 ↗
66. (2024, July 01). Text Classification Data from 15 Drug Class Review SLR Studies. Available online: https://figshare.com/articles/dataset/Text_Classification_Data_from_15_Drug_Class_Review_SLR_Studies/23626656/1?file=41457282.