Proposed pipeline Sample Clauses

Proposed pipeline. As part of the research work carried out for Harmony, we implemented a pipeline to easily train and deploy a state-of-the-art pose estimator on novel objects, such as the boxes from the Harmony use cases. One of the main challenges in 6D object pose estimation is that state-of-the-art approaches assume the availability of textured CAD models of the object of interest and require the user to manually set up physically based rendering (PBR) pipelines for synthetic data generation. In many practical use cases, a 3D model of the object is not available beforehand. Additionally, deploying the pose estimation system for a new system should require as little manual intervention on the part of the user as possible. To address these points, we propose a pipeline, which we name NeuSurfEmb, that does not require CAD models and allows training a state-of-the-art pose estimator with only a small number of real reference images as inputs. Our method employs an object representation based on a neural-implicit surface reconstruction method, NeuS2 [1], that we learn through a semi-automated procedure based on Structure-from-Motion (SfM) and object-agnostic segmentation. In particular, we first run the SfM tool COLMAP [2] on the set of reference images to estimate the camera poses associated with the reference images. A user-provided bounding box indicates the object of interest in the first frame, we then automatically obtain object masks for each of the reference images, combining the object-agnostic segmentation system Segment Anything (XXX) [3] and the state-of-the-art object tracker MixFormer [4] to propagate object masks across the reference frames. The reference images with the associated object masks and camera poses are then used to train an object-level NeuS2, which compactly and accurately reconstructs the object, effectively replacing the need for a CAD model. After training the NeuS2 model, we exploit its novel-view synthesis capabilities together with a simple cut-and-paste [5] augmentation approach to automatically generate photorealistic object renderings, which we use to train the state-of-the-art dense correspondence-based SurfEmb [6] pose estimator. A convolutional neural network (CNN) and a multi-layer perceptron with sinusoidal activations (SIREN) [7] are jointly trained to produce dense features for each pixel in the input training images (CNN) and for a set of 3D points (SIREN) sampled on the object surface. A contrastive learning objective is employe...