Uspešnost učenja več zaporednih nalog v globokih nevronskih mrežah

Pevcin, Andraž (author), Faganeli Pucer, Jana (mentor)

Inkrementalno učenje je ključno za učinkovito učenje prilagodljivih nevronskih mrež, ki se soočajo s konstantnim tokom novih podatkov. Pri njem se pojavlja velik problem---katastrofalno pozabljanje. V delu obravnavamo problem katastrofalnega pozabljanja pri inkrementalnem učenju in sistematično raziskujemo, kako na pozabljanje vplivata širina in globina nevronskih mrež. Eksperimentalni del je sestavljen iz učenja večslojnih perceptronov (MLP) in konvolucijskih nevronskih mrež (CNN) na prosto dostopnih podatkovnih množicah CIFAR-100 in pMNIST. Spremljamo spremembe klasifikacijske točnosti čez naloge in povprečno stopnjo pozabljanja po koncu učnega procesa. Rezultate pojasnimo z analizo ortogonalnosti gradientov posameznih nalog in njihove gostote. Poleg tega prikažemo spreminjanje odmika parametrov modela od optimuma prve naloge. V delu uspešno prikažemo, da v večini primerov inkrementalnega učenja širina opazno zmanjša pozabljanje, čeprav v nekaterih scenarijih manj kot v drugih. Pokažemo tudi, da globina, odvisno od scenarija, ne vpliva ali pa celo poslabša pozabljanje.

• Language: Slovenian
• Keywords: inkrementalno učenje, globoke nevronske mreže, klasifikacija slik, katastrofalno pozabljanje
• Work type: Bachelor thesis/paper
• Typology: 2.11 - Undergraduate Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2025
• PID: 20.500.12556/RUL-173260
• COBISS.SI-ID: 250521859
• Publication date in RUL: 15.09.2025

Secondary language

• Language: English
• Title: Effectiveness of learning multiple sequential tasks in deep neural networks

Abstract

Incremental learning is key to effectively training adaptive neural networks that face a constant stream of new data. It suffers from a major issue---catastrophic forgetting. In this work, we address catastrophic forgetting in incremental learning and systematically investigate how the width and depth of neural networks affect forgetting. The experimental part consists of training multilayer perceptrons (MLP) and convolutional neural networks (CNN) on the publicly available datasets CIFAR-100 and pMNIST. We track changes in classification accuracy across tasks and the average forgetting rate after the training process concludes. We explain the results by analyzing the orthogonality of task gradients and their density. In addition, we show how far the model parameters drift from the optimum of the first task. We demonstrate that, in most incremental learning scenarios, width noticeably reduces forgetting, although in some cases less than in others. We also show that, depending on the scenario, depth may have no effect or can even worsen forgetting.

• Keywords: incremental learning, deep neural networks, image classification, catastrophic forgetting