Aplikacija za poganjanje jezikovnih modelov na mobilnih napravah

Hribar, Samo (author), Pesek, Matevž (mentor)

Predstavljen je postopek izdelave aplikacije za mobilni operacijski sistem Android, ki omogoča poganjanje modelov nevronskih mrež z arhitekturo transformer. Vse procesiranje se odvija lokalno, kar omogoča večjo varnost naših podatkov in neodvisnost od internetne povezave. Aplikacija je testirana z modeloma Whisper za transkripcijo govora in Phi-2 za obdelavo naravnega jezika. Praktični del obsega izgradnjo aplikacije za Android z ogrodjem Jetpack Compose in vključevanje obstoječih C++ knjižnic s spletišča GitHub. Te omogočajo poganjanje umetnointeligenčnih modelov, ki sem jih natreniral v okviru te diplomske naloge. Gre za trening STT modela Whisper-Small in LLM Phi-2 s končnim ciljem razumeti slovenski jezik. Obsega tudi pripravo učne množice ARTUR1.0 s transkripcijami zvočnih posnetkov, prevajanje učne množice OASST1 in uporabo učnih podatkov za učenje modelov. Omenjena modela sta v aplikaciji pognana s pomočjo knjižnic whisper.cpp in llama.cpp . V končni verziji aplikacije lahko izrečemo vprašanje in posneti zvok se preda v obdelavo modelu Whisper, ki iz njega ustvari besedilo. Alternativno lahko svoje vprašanje tudi natipkamo v za to vnaprej določeno polje. Ob kliku na gumb "pošlji" aplikacija vnešeno besedilo preda jezikovnemu modelu. Ta ga obdela in zatem začne ustvarjati odgovor, besedo po besedo, podobno kot vidimo v ostalih spletnih storitvah. Generirano besedilo se pojavi kot odgovor sistema pod našim vprašanjem. Modele sem evalviral s testnimi množicami in jih primerjal z obstoječimi spletnimi rešitvami. Proces transkripcije govora sem natančenje primerjal z aplikacijo Govori.si. Vnos besedila s pomočjo govora predstavlja varno alternativo tipkanju v situacijah, ki od nas zahtevajo pozornost na okolico. Vnešeno besedilo obdela LLM Phi-2, ki sem ga primerjal z verjetno najbolj znanim LLM-om ChatGPT, pa tudi z odprtokodnim Mistralom. Odgovarjanje sem meril na svojem lastnem telefonu, Poco F3. Transkripcija govora je relativno počasna glede na spletno-povezljive rešitve, prav tako čas do generiranja prvega zloga v odgovoru. Nadaljnji zlogi so generirani hitreje, s hitrostjo okoli 2,6 zloga/s.

• Language: Slovenian
• Keywords: Android, transformer, jezikovni model, prepoznava govora, aplikacija
• Work type: Bachelor thesis/paper
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2024
• PID: 20.500.12556/RUL-164816
• Publication date in RUL: 13.11.2024

Secondary language

• Language: English
• Title: Application for running language models on mobile devices

Abstract

This thesis presents the process of developing an application for the Android mobile operating system that can run neural networks with a transformer architecture. The application has been tested with the Whisper model for speech transcription and the Phi-2 model for natural language processing. The practical part includes construction of the Android application using the Jetpack Compose framework and integration of existing C++ libraries from GitHub. Additionally, it covers the training of the Whisper-Small STT model and the Phi-2 LLM. It also involves preparing the ARTUR1.0 dataset with audio transcription and translating the OASST1 dataset, utilizing these datasets for model training. The mentioned models are executed in the application using the whisper.cpp and llama.cpp libraries. In the final version of the application, users can ask a question by speaking, with the recorded audio being processed by the Whisper model to convert it into text. Alternatively, users can type their questions into a predefined field. Once the text is submitted, it is processed by the language model, which generates a response. The application then displays this response below the user's question. I evaluated the models using test datasets and compared them with existing online solutions. I compared the speech transcription process more precisely with the Govori.si application, developed in my mentor's laboratory. Voice input represents a safe alternative to typing in situations that require our attention to the surroundings. The entered text is processed by the Phi-2 LLM, which I compared with probably the most well-known LLM, ChatGPT, as well as the open-source Mistral. I measured the response time on my own phone, a Poco F3. Speech transcription is quite slow, as is the time to generate the first token in the response. Subsequent tokens are generated faster, at a rate of approximately 2.6 tokens per second.

• Keywords: Android, transformer, language model, speech recognition, application