Porazdeljeno procesiranje z orodji Dask in Singularity na superračunalniških gručah

UJČIČ, MARCEL (author), Lotrič, Uroš (mentor), Sluga, Davor (co-mentor)

Za mnoge uporabnike predstavlja preskok iz rabe osebnega računalniku na računanje na superračunalnikih velik izziv, saj se morajo spoznati z novo strojno in programsko opremo. V diplomskem delu smo želeli uporabnikom ta preskok poenostaviti tako, da smo za superračunalniško okolje prilagodili prijazno in poznano delovno okolje. Pripravili smo prenosljivi vsebnik Singularity, ki vključuje zvezke Jupyter, programski jezik Python in knjižnico Dask. Uporabnik lahko v vsebniku izvaja svojo obstoječo programsko kodo v jeziku Python. Z nekaj popravki programske kode, ki vključujejo klice funkcij knjižnice Dask, pa lahko svojo programsko kodo pohitri s porazdeljenim računanjem na več procesorskih jedrih. Pripravljeno okolje smo preizkusili z dvema testnima aplikacijama: izrisom Mandelbrotove množice in analizo sentimenta, ter predstavili dobljene rezultate.

• Language: Slovenian
• Keywords: porazdeljeno procesiranje, knjižnica Dask, vsebniki Singularity, superračunalniška gruča, programski jezik Python.
• Work type: Bachelor thesis/paper
• Typology: 2.11 - Undergraduate Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2023
• PID: 20.500.12556/RUL-144579
• COBISS.SI-ID: 144030211
• Publication date in RUL: 02.03.2023

Secondary language

• Language: English
• Title: Distributed processing using Dask and Singularity on HPC clusters

Abstract

For many users, migrating from a personal computer to a supercomputer can be challenging, as they must familiarize themselves with new hardware and software. In the thesis, we wanted to simplify the process for users by creating a friendly and familiar work environment deployed on a supercomputing cluster. Therefore, we have prepared a portable Singularity container incorporating Jupyter notebooks, the Python programming language, and the Dask library. The user can run their existing Python programming code in the container. With a few tweaks to the code and Dask library function calls, the user can speed up the code with distributed computing on multiple processor cores. We tested the environment with two applications: plotting the Mandelbrot set and sentiment analysis, and presented the results.

• Keywords: distributed processing, library Dask, containers Singularity, HPC cluster, programming language Python.