Obstacle detection based on semantic segmentation for autonomous surface vehiclesBovcon, Borja (Avtor) Kristan, Matej (Mentor) obstacle detectionsemantic segmentationrobotic boatsautonomous unmanned surface vehiclesscene perceptionRecent research in marine robotics has led to the establishment of a new class of small-size unmanned surface vehicles (USVs) i.e., robotic boats. These vessels present an affordable and highly portable tool for navigating in difficult-to-reach areas (e.g. near dams) that might pose a danger to a human operator. For efficient and uninterrupted navigation in such environments a high level of autonomy is required, which above all depends on timely detection and avoidance of nearby obstacles and floating debris. Various sensors can be used to detect obstacles, however, in this doctoral thesis, we focus on an approach that combines visual cameras with semantic segmentation algorithms. Visual cameras have proven as a lightweight and information-rich sensor, while semantic segmentation algorithms enable indirect detection of arbitrary artefacts (i.e. obstacles) and provide us with a broader scene understanding. In the first part of the thesis, we extend a state-of-the-art hand-crafted graphical model for semantic segmentation to incorporate boat pitch and roll measurements from the on-board inertial measurement unit (IMU). The IMU readings are used to estimate the location of the horizon line in the image, which automatically adjusts the priors in the probabilistic semantic segmentation model. This improves the overall segmentation accuracy, especially in the presence of visual ambiguities, and consequently leads to a better detection rate. To consolidate tentative detections we propose further extending the graphical model with multiple views by adding a constraint that prefers consistent class labels assignment to pixels in each camera image corresponding to the same parts of a 3D scene. This unifies the class-label posterior map and reduces the amount of false alarms. In the second part of the thesis, we propose a novel deep encoder-decoder architecture that is specifically designed for segmentation of the maritime environment. Its deep encoder, coupled with atrous convolutions, is capable of extracting rich visual features, which ensures generalization and accurate segmentation even in the presence of sun glitter and reflections. On the other hand, a novel decoder gradually fuses visual features with inertial information which greatly improves the segmentation accuracy of the water component in the presence of visual ambiguities, e.g. at haze. In addition, a novel loss function is proposed to enforce the separation of different semantic components to further increase the robustness. In the third, and final part of this thesis, we present a new diverse maritime obstacle detection benchmark which contains approximately 81k stereo images synchronized with an on-board IMU, with over 60k objects annotated. In addition to the dataset, we propose a new obstacle segmentation performance evaluation protocol that reflects the detection accuracy in a way meaningful for practical USV navigation. On this benchmark we evaluate our proposed methods and compare their performance to fourteen state-of-the-art segmentation methods. Results show, that our deep-learning-based approach outperforms other methods by a large margin, especially in the danger zone i.e., in the near proximity of the USV.20222023-01-27 10:00:00Doktorsko delo/naloga144041sl