20.500.12556/RUL-144861
SUPERFORMER
continual learning superposition method for text classification
One of the biggest challenges in continual learning domains is the tendency of machine learning models to forget previously learned information over time. While overcoming this issue, the existing approaches often exploit large amounts of additional memory and apply model forgetting mitigation mechanisms which substantially prolong the training process. Therefore, we propose a novel SUPERFORMER method that alleviates model forgetting, while spending negligible additional memory and time. We tackle the continual learning challenges in a learning scenario, where we learn different tasks in a sequential order. We compare our method against several prominent continual learning methods, i.e., EWC, SI, MAS, GEM, PSP, etc. on a set of text classification tasks. We achieve the best average performance in terms of AUROC and AUPRC (0.7% and 0.9% gain on average, respectively) and the lowest training time among all the methods of comparison. On average, our method reduces the total training time by a factor of 5.4-8.5 in comparison to similarly performing methods. In terms of the additional memory, our method is on par with the most memory-efficient approaches.
deep learning
continual learning
superposition
transformers
globoko učenje
nenehno učenje
superpozicija
transformerji
true
false
true
Angleški jezik
Slovenski jezik
Članek v reviji
2023-03-17 09:20:50
2023-03-17 09:20:51
2023-09-01 04:00:53
0000-00-00 00:00:00
2023
0
0
Str. 418-436
Vol. 161
2023
0000-00-00
Zaloznikova
Objavljeno
NiDoloceno
0000-00-00
0000-00-00
0000-00-00
004.8
0893-6080
10.1016/j.neunet.2023.01.040
141099267
1-s2.0-S0893608023000527-main.pdf
1-s2.0-S0893608023000527-main.pdf
1
DFB7BF84CED0CBD6F965252450E4D560
6e9148d15b9ec8c6a9fabd995c8a89a0bc2d296b31c87d71775f99158e931d10
ee3cdf33-c49d-11ed-a314-00155dcfd717
https://repozitorij.uni-lj.si/Dokument.php?lang=slv&id=167414
https://www.sciencedirect.com/science/article/pii/S0893608023000527
1
909aebde-c49c-11ed-a314-00155dcfd717
https://repozitorij.uni-lj.si/Dokument.php?lang=slv&id=167413
Fakulteta za računalništvo in informatiko
0
0
0