Ocenjevanje zanesljivosti posameznih napovedi inkrementalnih modelov

Javornik, Anže (author), Kononenko, Igor (mentor)

Diplomsko delo obravnava ocenjevanje zanesljivosti posameznih napovedi inkrementalnih modelov. Namen ocene zanesljivosti je obogatiti napoved modela z dodatno informacijo. Ta dodatna informacija ima lahko kritični pomen, zlasti če imajo napačne napovedi lahko hude posledice. Klasični modeli napovedovanja so zgrajeni na primerih problematike, za katero podajajo svojo napoved. Klasične metode ocenjevanja zanesljivosti se gradijo na podoben način na primerih, ki predstavljajo učno množico modelu napovedovanja. Na tak način zgrajeni modeli in ocene zanesljivosti so na kratek rok uspešni. V daljšem obdobju uporabe pa najverjetneje pride do sprememb pravil v problematiki, ki negativno vplivajo na uspešnost klasičnih napovednih modelov in ocen zanesljivosti. V realnosti je večina problematik takih, da se pravila v njih spreminjajo. Grajenje novih klasičnih modelov za prilagajanje problematiki je zamudno ali sploh ne pride v poštev, če je podatkovni tok tak, da lahko primer vidimo samo enkrat – npr. mrežna komunikacija. Za analizo takih problemskih domen se zato odločimo za uporabo inkrementalnih modelov napovedovanja, ki se znajo prilagajati spremembam pravil problematike. V diplomskem delu predstavimo tri znane metode ocenjevanja zanesljivosti in predlagamo njihove različice z inkrementalnim značajem. Znane metode in razvite različice smo preizkusili na dvaindvajsetih fiksnih množicah in osmih podatkovnih tokovih (generatorjih) brez zamika in z zamikom, ki nam predstavlja spremembo pravil v problemski domeni. Dobljene ocene smo statistično obdelali. Pri statistični analizi nas je zanimala statistična značilnost korelacije dejanske napake z oceno. V tem delu pridobljeni rezultati kažejo, da predlagane inkrementalne metode ocenjevanja zanesljivosti ponujajo skoraj enake rezultate kot klasične metode na fiksnih problemskih domenah in mnogo boljše rezultate pri podatkovnih tokovih, še posebej pri tokovih z zamikom. Najboljše rezultate nudi ocena iCNK. Dobre rezultate nudi tudi ocena iBAGV. Inkrementalna različica ocene LCV, iLCV, pa ne odstopa od svoje klasične različice.

• Language: Slovenian
• Keywords: inkrementalno strojno učenje, klasifikacija, inkrementalno ocenjevanje zanesljivosti, klasifikacijska točnost
• Work type: Undergraduate thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2016
• PID: 20.500.12556/RUL-84437
• Publication date in RUL: 23.08.2016

Secondary language

• Language: English
• Title: Reliability estimation of individual predictions for incremental models

Abstract

The thesis deals with the reliability estimates of individual predictions of incremental models. The purpose of reliability estimates is to enrich the models predictions with additional information. This information might have a critical meaning, especially if wrong predictions can have serious consequences. Classical prediction models are built on examples from a problem domain for which they provide predictions. Classical methods of the reliability estimation are constructed in a similar way, on examples representing the learning set for the prediction model. Models and reliability estimate scores built in this way in short term provide good results. In the long term use there is a high probability that a change happens in the problem domain that adversely affects the performance of conventional predictive models and estimates of reliability. Building a new predictive model to adapt to the changes can be time consuming or even not an option, if the data stream is such that we can observe each example only once – e.g. network communication. Therefore, for the analysis of such problem domains we choose to use the incremental prediction models, which are able to adapt to changes. In this thesis we present three known methods of the reliability estimation and propose their versions with an incremental character. These methods were tested on twenty-two fixed problem domains and eight data streams (generators) with and without drift, which represents a change in the problem domain. The resulting estimates were statistically analysed. In the statistical analysis, we were interested in statistically significant correlations between factual error and the reliability assessment. Obtained results show that the proposed incremental methods of reliability estimation offer almost the same results in fixed problem domains and much better results in data streams, especially if the drift is applied. Best results are offered by the reliability estimate iCNK. Good results are also offered by the reliability estimate iBAGV. An incremental version of the reliability estimate LCV, iLCV, does not deviate from its classical version.

• Keywords: incremental machine learning, classification, incremental reliability estimates, classification accuracy