Statistical comparison of machine learning algorithms with respect to multiple performance measuresDULAR, LARA (Avtor) Todorovski, Ljupčo (Mentor) Ankerst, Donna (Komentor) comparison of machine learning algorithmspairwise comparisoncomparative studiesmultiple performance measuresWilcoxon signed-rank testFriedman testPareto frontIn the theory and practice of machine learning, we often face the task of comparing the performance of learning algorithms on multiple data sets. On the one hand, theoretical studies that propose new algorithms or improvements of the existing ones, compare the newly proposed algorithms to the existing ones. On the other hand, empirical studies on the application of machine learning methods often compare the performance of learning algorithms on various instances of a practical real-world problem. In both cases, an appropriate statistical analysis, which is the subject of this master's thesis, is crucial to determine the significance of the comparison's results. This thesis has two main goals. The first is a thorough presentation of the most commonly used nonparametric statistical tests used for comparing machine learning algorithms with respect to a single performance measure, namely, the Wilcoxon signed-rank test and the Friedman test. The second goal of the master's thesis is to overcome the limitations of existing approaches for comparison of algorithms with respect to a single, pre-selected performance measure. We present a new approach for the comparison of machine learning algorithms with respect to multiple performance measures simultaneously. To this end, the concept of Pareto fronts, used in the field of multi-objective optimization, will be utilized to rank the algorithms according to multiple performance measures. Thus, the above-mentioned nonparametric statistical tests may also be used in the context of the new approach. We illustrate the use of the newly developed approach on an example of comparing the performance of four learning algorithms for classification on ten publicly available data sets. We compare the algorithms with respect to two performance measures that assess two aspects of the accuracy of the trained classification models. The results of the comparison show that in most cases, the new approach rejects the null hypothesis for comparison of algorithms with respect to both performance measures simultaneously, if the existing approach rejects at least one of the two null hypotheses for a single performance measure.20182018-08-30 07:45:01Magistrsko delo/naloga102425sl