SUPERFORMER : continual learning superposition method for text classification

Zeman, Marko (avtor), Faganeli Pucer, Jana (avtor), Kononenko, Igor (avtor), Bosnić, Zoran (avtor)

One of the biggest challenges in continual learning domains is the tendency of machine learning models to forget previously learned information over time. While overcoming this issue, the existing approaches often exploit large amounts of additional memory and apply model forgetting mitigation mechanisms which substantially prolong the training process. Therefore, we propose a novel SUPERFORMER method that alleviates model forgetting, while spending negligible additional memory and time. We tackle the continual learning challenges in a learning scenario, where we learn different tasks in a sequential order. We compare our method against several prominent continual learning methods, i.e., EWC, SI, MAS, GEM, PSP, etc. on a set of text classification tasks. We achieve the best average performance in terms of AUROC and AUPRC (0.7% and 0.9% gain on average, respectively) and the lowest training time among all the methods of comparison. On average, our method reduces the total training time by a factor of 5.4-8.5 in comparison to similarly performing methods. In terms of the additional memory, our method is on par with the most memory-efficient approaches.

• Jezik: Angleški jezik
• Ključne besede: deep learning, continual learning, superposition, transformers
• Vrsta gradiva: Članek v reviji
• Tipologija: 1.01 - Izvirni znanstveni članek
• Organizacija: FRI - Fakulteta za računalništvo in informatiko
• Status publikacije: Objavljeno
• Različica publikacije: Objavljena publikacija
• Leto izida: 2023
• Št. strani: Str. 418-436
• Številčenje: Vol. 161
• PID: 20.500.12556/RUL-144861
• UDK: 004.8
• ISSN pri članku: 0893-6080
• DOI: 10.1016/j.neunet.2023.01.040
• COBISS.SI-ID: 141099267
• Datum objave v RUL: 17.03.2023

Gradivo je del revije

• Naslov: Neural networks
• Skrajšan naslov: Neural netw.
• Založnik: Elsevier
• ISSN: 0893-6080
• COBISS.SI-ID: 26011904

Licence

• Licenca: CC BY 4.0, Creative Commons Priznanje avtorstva 4.0 Mednarodna
• Povezava: http://creativecommons.org/licenses/by/4.0/deed.sl
• Opis: To je standardna licenca Creative Commons, ki daje uporabnikom največ možnosti za nadaljnjo uporabo dela, pri čemer morajo navesti avtorja.

Sekundarni jezik

• Jezik: Slovenski jezik
• Ključne besede: globoko učenje, nenehno učenje, superpozicija, transformerji

Projekti

• Financer: ARRS - Agencija za raziskovalno dejavnost Republike Slovenije
• Številka projekta: P2-0209
• Naslov: Umetna inteligenca in inteligentni sistemi

• Financer: ARRS - Agencija za raziskovalno dejavnost Republike Slovenije
• Program financ.: Young researchers