Vzpostavitev sistema Hadoop Map/Reduce v šolske namene

PLUT, JAKA (author), Kukar, Matjaž (mentor)

Namestitev in konfiguracija porazdeljenega sistema Hadoop MapReduce je časovno precej zamudna in terja temeljito ter dosledno upoštevanje navodil. Kot takšna lahko povzroča precej nevšečnosti novim uporabnikom, ki bi se želeli spoznati s programskim modelom MapReduce. Cilj diplomskega dela je raziskati možnosti za čim enostavnejšo namestitev in konfiguracijo porazdeljenega sistema Hadoop. Diplomsko delo se osredotoča na vzpostavitev porazdeljenega sistema Hadoop s pomočjo programske rešitve ameriškega podjetja Cloudera Inc., imenovane CDH, in temelji na platformi Apache Hadoop. Podjetje je izdalo več različic programske opreme CDH – trenutno je najnovejša CDH 5.5, ki jo je moč poganjati na različnih distribucijah operacijskega sistema Linux. V diplomskem delu so zapisani postopki in nasveti, ki so potrebni za uspešno vzpostavitev tovrstnega porazdeljenega sistema. V tekstu preučujemo različne možnosti namestitve in vzpostavitve sistema predvsem s pomočjo virtualizacije. Poleg omenjenega podrobno opisujemo in s primeri ilustriramo izvajanje poslov MapReduce. Na koncu izvedemo še kratko analizo, ki primerja učinkovitost (skalabilnost) izvedbe poslov MapReduce na enem, dveh in več vozliščih.

• Language: Slovenian
• Keywords: Hadoop, namestitev, Linux, Cloudera
• Work type: Bachelor thesis/paper
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2016
• PID: 20.500.12556/RUL-81062
• Publication date in RUL: 25.03.2016

Secondary language

• Language: English
• Title: Establishing Hadoop Map/Reduce cluster for teaching purposes

Abstract

Installation and configuration of Hadoop MapReduce distributed system is quite time-consuming and requires thorough compliance with instructions. As such, it can cause considerable inconveniences to new users who would like to get familiar with the MapReduce programming model. The aim of this thesis is to research the possibilities for straight forward installation and configuration of Hadoop distributed system. The thesis focuses on creating a distributed system using Hadoop software with the help of a solution called CDH, developed by an American company Cloudera Inc. The solution is based on Apache Hadoop platform. The company has released several versions of CDH software. The latest available CDH 5.5 can be run on different distributions of Linux operating system. The bachelor thesis is comprised out of instructions and tips which are necessary for the successful setup of such distributed system. The text researches various setup options and the creation of a system using predominantly virtualization. Furthermore, we are describing in detail, with examples, the process of running MapReduce jobs. In the end, we have a brief analysis which compares performance (scalability) of MapReduce jobs run on one, two and more nodes.

• Keywords: Hadoop, setup, Linux, Cloudera