Razpodvajanje podatkov v podatkovni shrambi

KOVAČ, DEJAN (author), Brodnik, Andrej (mentor)

V diplomski nalogi smo podrobneje predstavili področje razpodvajanja podatkov in implementirali algoritem shranjevanja objektov s privzetim odstranjevanjem podvojenih kosov. V prvem delu naloge smo predstavili shranjevalni sistem kot drevesno urejeno strukturo imenikov in datotek. Opisali smo funkcije shranjevalnega sistema ter preproste načine shranjevanja podatkov na medij. Podrobneje smo pregledali lastnosti porazdeljenega shranjevalnega sistema Ceph, njegove sestavne dele in delovanje. V drugem delu predstavimo metodo razpodvajanja podatkov kot pomembno sestavino sodobnih shranjevalnih sistemov. Pregledali smo tehnike razpodvajanja na centraliziranih in porazdeljenih sistemih. V zadnjem delu naloge smo implementirali primer razpodvajanja skupaj s preprostim sistemom za shranjevanje objektov. S pomočjo opisanih tehnik smo izvedli zaznavanje spremenljivo dolgih podvojenih kosov znotraj objektov in dodali CLI orodja za manipulacijo objektov v shrambi.

• Language: Slovenian
• Keywords: razpodvajanje, datotečni sistem, shranjevanje podatkov
• Work type: Bachelor thesis/paper
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2014
• PID: 20.500.12556/RUL-29997
• Publication date in RUL: 30.10.2014

Secondary language

• Language: English
• Title: Data deduplication in a data store

Abstract

In our thesis we presented the area of data deduplication and implemented an algorithm for object storage with support for elimination of duplicate chunks within those objects. In the first part we presented storage system as a tree-likestructure of directories and files. We described the features of storage system and simple ways of storing data on a medium. We examined in detail the properties of distributed storage system Ceph, it's components and operation. In the second part we presented deduplicatin as an important feature of modern storage systems. We surveyed deduplication techniques for centralized as well as distributed systems. In the last part we implemented an example of deduplication technique along with a simple object storage system. Using the described techniques we implemented detection of variable-length duplicated chunks within objects and added CLI tools for manipulating objects in the store.

• Keywords: deduplication, filesystem, data store