Poravnava RGBD in multispektralnih slik v aplikaciji nadzora rasti paradižnikov

Gorkič, David (author), Perš, Janez (mentor)

Poravnava slik predstavlja enega izmed pomembnih korakov v procesu analize slik na področju računalniškega vida. To se pokaže tudi v primeru te naloge, katere cilj je poravnava slik in predstavlja del širšega projekta za nadzorovanje rasti paradižnikov. Na voljo imamo RGBD in multispektralne slike, ki so zajete z dvema različnima senzorjema. To rezultira v nepopolni medsebojni poravnanosti ter paralaksi med slikami istega prizora, obenem pa se le-te razlikujejo tudi v sami velikosti. Poravnava je bila izvedena v treh korakih. V prvem smo izvajali poravnavo po tradicionalnem postopku (z nekaterimi elementi metod globokega učenja), ki temelji na iskanju in ujemanju značilk/značilnih točk (angl. feature based methods), pri čemer smo uporabili algoritem SuperGlue za pridobitev ujemajočih točk ter knjižnico openCV za oceno transformacije. Drugi korak predstavlja poravnava 2D multispektralnih slik ter 3D oblaka točk (angl. pointcloud) pridobljenega iz globinske in RGB slike poravnane v prvem koraku. V ta namen smo uporabili program, ki ga bomo v nadaljevanju naslavljali s kratico MCPCIA (Multimodal Colored Point Cloud to Image Alignment). Rezultat poravnave je RGB slika z več temnimi področji, ki so nastali kot posledica 3D transformacije. Ta področja smo odstranili v zadnjem, tretjem koraku našega postopka, z vzvratno projekcijo 2D slike v 3D prostor in interpolacijo po principu najbližjega soseda v oblaku točk. Analiza rezultatov je pokazala zadovoljivo dobro delovanje algoritma za poravnavo, v kolikor vhodni podatki zadoščajo določenim merilom (navedeni v nadaljevanju). Ugotovljeno je bilo, da lahko rezultate še izboljšamo z uporabo kvalitetnejših globinskih slik v procesu generiranja oblaka točk.

• Language: Slovenian
• Keywords: poravnava slik, računalniški vid, nadzor rasti paradižnikov, multispektralne slike, RGBD, oblak točk
• Work type: Master's thesis/paper
• Organization: FE - Faculty of Electrical Engineering
• Year: 2024
• PID: 20.500.12556/RUL-162503
• Publication date in RUL: 24.09.2024

Secondary language

• Language: English
• Title: Registration of RGBD and Multispectral Images in Tomato Growth Monitoring Applications

Abstract

Image alignment is one of the crucial steps in the process of image analysis in the field of computer vision. This is evident in the task at hand, which aims to align images as part of a broader project to monitor tomato growth. We have RGBD and multispectral images captured by two different sensors, resulting in incomplete mutual alignment and parallax between images of the same scene, as well as differences in their sizes. The alignment was carried out in three steps. In the first step, traditional feature-based alignment methods (with some elements of deep learning) were employed, using the SuperGlue algorithm to obtain matching points and the OpenCV library to estimate the transformation. The second step involves the alignment of the 2D multispectral images with the 3D point cloud derived from the depth and RGB images aligned in the first step. For this purpose, we used a program referred to as MCPCIA (Multimodal Colored Point Cloud to Image Alignment). The result of the alignment is an RGB image with several dark areas, which are a consequence of the 3D transformation. These areas were removed in the final, third step of our process by back-projecting the 2D image into 3D space and interpolating using the nearest neighbor principle in the point cloud. The analysis of the results showed satisfactory performance of the alignment algorithm, provided that the input data meet certain criteria (specified later). It was found that the results could be further improved by using higher quality depth images in the point cloud generation process.

• Keywords: image alignment, computer vision, tomato growth monitoring, multispectral images, RGBD, point cloud