journal article·2020-02-21Open access

An Efficient Approach to Consolidating Job Schedulers in Traditional Independent Scientific Workflows

Byungyun Kong, Geonmo Ryu, Sangwook Bae, Seo-Young Noh, Heejun Yoon

Applied Sciences· Volume 10· Issue 4

Abstract

The current research paradigm is one of data-driven research. Researchers are beginning to deploy computer facilities to produce and analyze large amounts of data. As requirements for computing power grow, data processing in traditional workstations is always under pressure for efficient resource management. In such an environment, a tremendous amount of data is being processed using parallel computing for efficient and effective research results. HTCondor, as an example, provides computing power for data analysis for researchers. Although such a system works well in a traditional computing cluster environment, we need an efficient methodology to meet the ever-increasing demands of computing using limited resources. In this paper, we propose an approach to integrating clusters that can share their computing power on the basis of a priority policy. Our approach makes it possible to share worker nodes while maintaining the resources allocated to each group. In addition, we have utilized the historical data of user usage in order to analyze problems that have occurred during job execution due to resource sharing and the actual operating results. Our findings can provide a reasonable guideline for limited computing powers shared by multiple scientific groups.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Distributed and Parallel Computing Systems · Scientific Computing and Data Management · Cloud Computing and Resource Management

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Global Science Experimental Data Hub Center, Korea Institute of Science and Technology Information, 245 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea

Department of Computer Science, Chungbuk National University, Chungdae-ro 1, Seowon-Gu, Cheongju, Chungbuk 28644, Korea

Funding

National Research Foundation of Korea

References (27)

Show source-provided references
1. Hey, A., Tansley, S., and Tolle, K. (2009). The Fourth Paradigm: Data-Intensive Scientific Discovery, Microsoft Research.
2. Perspective: Materials informatics and big data: Realization of the ‘fourth paradigm’ of science in materials science10.1063/1.4946894 ↗
3. Trajanov, D., Zdraveski, V., Stojanov, R., and Kocarev, L. (2018, January 12–14). Dark Data in Internet of Things (IoT): Challenges and Opportunities. Proceedings of the 7th Small Systems Simulation Symposium, Niš, Serbia.
4. Oliveira, S.F., Fürlinger, K., and Kranzlmüller, D. (2012, January 25–27). Trends in computation, communication and storage and the consequences for data-intensive science. Proceedings of the 2012 IEEE 14th International Conference on High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems, Liverpool, UK.10.1109/HPCC.2012.83 ↗
5. The rise of big data
6. Data Analytics Implications10.1109/MPOT.2018.2824361 ↗
7. Mechanisms for High Throughput Computing
8. Distributed computing in practice: The condor experience10.1002/cpe.938 ↗
9. Sterling, T. (2002). Condor—A Distributed Job Scheduler. Beowulf Cluster Computing with Linux, The MIT Press.10.7551/mitpress/1556.001.0001 ↗
10. (2019, December 29). HTCondor, High Throughput Computing. Available online: https://research.cs.wisc.edu/htcondor.
11. Henderson, R., and Tweten, D. (1996). Portable Batch System: External Reference Specification, Ames Research Center. Technical Report.
12. (2019, December 29). PBS. Available online: http://pbspro.org.
13. Jette, M.A., Yoo, A.B., and Grondona, M. (2003, January 24). SLURM: Simple Linux Utility for Resource Management. Proceedings of the Job Scheduling Strategies for Parallel Processing (JSSPP), Seattle, WA, USA. Lecture Notes in Computer Science.
14. (2019, December 29). Slurm. Available online: https://slurm.schedmd.com.
15. The extension of TORQUE scheduler allowing the use of planning and optimization algorithms in Grids10.7494/csci.2012.13.2.5 ↗
16. (2019, December 29). Torque. Available online: https://www.adaptivecomputing.com/products/open-source/torque.
17. Kim, B., Ahn, S., Khan, T., and Jnag, H. (2011, January 19–25). Introduction of GSDC project and activities. Proceedings of the International Symposium on Grids and Clouds and the Open Grid Forum, Taipei, Taiwan.10.22323/1.133.0013 ↗
18. Cho, K. Personal communication.
19. Cho, K. (2019). 8th GSDC Resources Review Boards, KISTI.
20. A Roadmap for HEP Software and Computing R&D for the 2020s10.1007/s41781-018-0018-8 ↗
21. Multi-experimental Support Through HTCondor Scheduling Policy Integrated Pool Configurationr
22. Experience on HTCondor batch system for HEP and other research fields at KISTI-GSDC10.1088/1742-6596/898/8/082013 ↗
23. (2019, December 29). HTCondor Manual. Available online: https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html.
24. Raman, R., Livny, M., and Solomon, M. (1998, January 28–31). Matchmaking: Distributed Resource Management for High Throughput Computing. Proceedings of the Seventh IEEE International Symposium on High Performance Distributed Computing, Chicago, IL, USA.
25. A study on performance degradation when using Singularity container of HTCondor scheduler
26. Kurtzer, G.M., Sochat, V., and Bauer, M.W. (2017). Singularity: Scientific containers for mobility of compute. PLoS ONE, 12.10.1371/journal.pone.0177459 ↗
27. (2019, December 29). Singularity, Available online: https://singularity.lbl.gov.