Superposition and compression of deep neutral networks

Zeman, Marko (author), Bosnić, Zoran (mentor), Osipov, Evgeny (comentor)

In this work we investigate a combination of the two recently proposed techniques: superposition of multiple neural networks into one and neural network compression. We show that these two techniques can be successfully combined to deliver a great potential for trimming down deep (convolutional) neural networks. We study the trade-offs between the model compression rate and the accuracy of the superimposed tasks and present a new approach, where the fully connected layers are isolated from the convolutional layers and serve as a general purpose processing unit for several CNN models. We evaluate our techniques on adapted MNIST and CIFAR-100 dataset, calculating classification accuracy and comparing baseline to the superposition method. Our experiments confirm the usability of superposition in terms of avoiding the catastrophic forgetting effect. The work has a significant importance in the context of implementing deep learning on low-end computing devices as it enables neural networks to fit edge devices with constrained computational resources (e.g. sensors, mobile devices, controllers).

• Language: English
• Keywords: Artificial Intelligence, Machine Learning, Deep Learning, Convolutional Neural Networks, Model Compression, Superposition of Models
• Work type: Master's thesis/paper
• Typology: 2.09 - Master's Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2020
• PID: 20.500.12556/RUL-119135
• COBISS.SI-ID: 27690499
• Publication date in RUL: 03.09.2020

ZEMAN, Marko, 2020, Superposition and compression of deep neutral networks [online]. Master’s thesis. [Accessed 26 March 2025]. Retrieved from: https://repozitorij.uni-lj.si/IzpisGradiva.php?lang=eng&id=119135

Secondary language

• Language: Slovenian
• Title: Superpozicija in kompresija globokih nevronskih mrež

Abstract

V našem delu preučujemo kombinacijo dveh nedavno predlaganih tehnik, in sicer superpozicijo več nevronskih mrež v eni in kompresijo nevronskih mrež. Pokazali smo, da je mogoče ti dve tehniki uspešno kombinirati, kar kaže na velik potencial zmanjševanja velikosti globokih (konvolucijskih) nevronskih mrež. Preučujemo kompromis med stopnjo kompresije modela in natančnostjo naučenih nalog ter predstavljamo nov pristop, pri katerem so polno povezani nivoji mreže izolirani od konvolucijskih nivojev in služijo kot splošno namenska procesna enota za več modelov konvolucijskih nevronskih mrež. Uspešnost naših tehnik ocenjujemo na prilagojenih MNIST in CIFAR-100 podatkih, izračunamo točnost klasifikacije in primerjamo izhodiščno metodo z metodo superpozicije. Naši poskusi potrjujejo uporabnost superpozicije v smislu izogibanja učinku katastrofalnega pozabljanja pri učenju več zaporednih nalog. Namen dela je pomemben v smislu izvajanja globokega učenja na napravah z omejenimi računskimi viri (npr. senzorji, mobilne naprave, krmilniki).

• Keywords: umetna inteligenca, strojno učenje, globoko učenje, konvolucijske nevronske mreže, kompresija modelov, supepozicija modelov