Superposition and compression of deep neutral networks
Zeman, Marko (Author), Bosnić, Zoran (Mentor) More about this mentor... This link opens in a new window, Osipov, Evgeny (Co-mentor)

.pdfPDF - Presentation file, Download (2,79 MB)

In this work we investigate a combination of the two recently proposed techniques: superposition of multiple neural networks into one and neural network compression. We show that these two techniques can be successfully combined to deliver a great potential for trimming down deep (convolutional) neural networks. We study the trade-offs between the model compression rate and the accuracy of the superimposed tasks and present a new approach, where the fully connected layers are isolated from the convolutional layers and serve as a general purpose processing unit for several CNN models. We evaluate our techniques on adapted MNIST and CIFAR-100 dataset, calculating classification accuracy and comparing baseline to the superposition method. Our experiments confirm the usability of superposition in terms of avoiding the catastrophic forgetting effect. The work has a significant importance in the context of implementing deep learning on low-end computing devices as it enables neural networks to fit edge devices with constrained computational resources (e.g. sensors, mobile devices, controllers).

Keywords:Artificial Intelligence, Machine Learning, Deep Learning, Convolutional Neural Networks, Model Compression, Superposition of Models
Work type:Master's thesis/paper (mb22)
Organization:FRI - Faculty of computer and information science
COBISS.SI-ID:27690499 This link opens in a new window
Average score:(0 votes)
Your score:Voting is allowed only to logged in users.
AddThis uses cookies that require your consent. Edit consent...

Secondary language

Title:Superpozicija in kompresija globokih nevronskih mrež
V našem delu preučujemo kombinacijo dveh nedavno predlaganih tehnik, in sicer superpozicijo več nevronskih mrež v eni in kompresijo nevronskih mrež. Pokazali smo, da je mogoče ti dve tehniki uspešno kombinirati, kar kaže na velik potencial zmanjševanja velikosti globokih (konvolucijskih) nevronskih mrež. Preučujemo kompromis med stopnjo kompresije modela in natančnostjo naučenih nalog ter predstavljamo nov pristop, pri katerem so polno povezani nivoji mreže izolirani od konvolucijskih nivojev in služijo kot splošno namenska procesna enota za več modelov konvolucijskih nevronskih mrež. Uspešnost naših tehnik ocenjujemo na prilagojenih MNIST in CIFAR-100 podatkih, izračunamo točnost klasifikacije in primerjamo izhodiščno metodo z metodo superpozicije. Naši poskusi potrjujejo uporabnost superpozicije v smislu izogibanja učinku katastrofalnega pozabljanja pri učenju več zaporednih nalog. Namen dela je pomemben v smislu izvajanja globokega učenja na napravah z omejenimi računskimi viri (npr. senzorji, mobilne naprave, krmilniki).

Keywords:umetna inteligenca, strojno učenje, globoko učenje, konvolucijske nevronske mreže, kompresija modelov, supepozicija modelov

Similar documents

Similar works from RUL:
Similar works from other Slovenian collections:


Leave comment

You have to log in to leave a comment.

Comments (0)
0 - 0 / 0
There are no comments!