This work touches upon the problem of where to put adapters in large lan-
guage models to achieve the best performance possible. Large language -
models are nowadays a key tool for natural language processing. We use
them for an array of tasks, but we must first fine-tune them for these tasks.
Training the whole model, which is often very large, is computationally ex-
pensive and time consuming, so we use adapters instead, but we must place
them in the right places in the large language model architecture and pick the
right hyperparameters for them to be successful. We approach this problem
by making use of genetic algorithms, as checking every possible configura-
tion would take way too long. We tested this on the BERT model with the
LoRA adapter, on emotion classficiation. That way we get not only con-
crete solutions but also insight into the patterns that appear in favorable
solutions. We find that increasing the rank of the adapters is more effective
than changing their placement, but a higher rank means more parameters, so
care must be taken. Our results also show that the biggest sensible amount
of parameters is up to 5% the amount of parameters of the original model,
as more parameters don’t contribute to higher accuracy, and that the best
possible accuracy for a certain adapter size has an upper bound.
|
