Structured pruning of large language models (LLMs) combined with knowledge distillation and quantization represents an effective approach for obtaining smaller and more computationally efficient language models. Pruning
reduces the number of parameters and thus the size of the LLM, knowledge
distillation helps preserve its performance, while quantization further reduces
memory consumption and computational requirements by using lower numerical precision, particularly during LLM inference. We tested ten different
combinations of width and depth pruning, as well as two types of quantization, FP8 and NVFP4. We evaluated all variants using Slovenian LLM
Eval, while data on speed and efficiency were obtained using the vLLM library. In the experiment, we show that reducing LLM width is the most
effective among the evaluated structural compression approaches when combined with short knowledge distillation, while the results also confirm the
high efficiency of quantization to the FP8 format. Based on the findings,
we propose three efficient LLM compression approaches and develop an 8Bparameter LLM that is 62.3 % smaller, 84.2 % faster, and uses 53.1 % less
energy than GaMS3 12B, with a 12.8 % lower quality score after distillation
on only 2B tokens.
|
