3D object reconstruction

Contact person: Anton S. Konushin (ktosh@graphics.cs.msu.ru)

Introduction

This project is devoted to research into combination of sketch-based modeling and computer vision techniques for reconstructing complex real-world objects from videos and image sequences. Initially this project was started in 2004 in cooperation with Samsung Advanced Institute of Technology and continued as internal project of the laboratory.

Despite serious progress in the field of shape-from-X reconstruction techniques, fully automated reconstruction of arbitrary real-world objects still fail to yield acceptable result over a broad class of object shapes, image types and scene complexity. Hence, research into efficient and easy-to-use combination of automated and manual 3D modeling from images is very promising.

  Images from input sequences of 35 frames  

 Project goals

he project key idea is to support computer vision techniques with user interaction to contribute to overall scheme robustness. The project goals include:

  1. User-assisted creation of 3D models of objects from the raw 3D data (dense depth maps, quasi-dense & sparse 3D points);
  2. Development of the UI mechanisms for interactive 3D object shape reconstruction support. This includes the general paradigm of user intervention into reconstruction process and efficient ways for its implementation;
  3. Enhancement and adaptation of automatic camera calibrations and 3D reconstruction methods to the needs of interactive reconstruction and augmented reality;

Reconstruction overview

The overall reconstruction process comprises of the following steps:

  1. Images acquisition:
    •     Acquiring object images with a hand-held camera;
  2. Structure and motion estimation:
    •     Feature tracking;
    •     Sequential or hierarchical SAM estimation
    •     Pose refinement of all cameras by bundle adjustment
  3. User-guided basic geometric shapes fitting to sparse 3D points:
    •     Segmenting object 3D points from background;
    •     Guided fitting of geometric shape (cylinder, plane, polyhedron, surface of revolution);
  4. Sketch-based editing of reconstructed object
    •     Adding missed detail;
    •     Correcting reconstruction errors;
  5. Texturing
    •     Texture generation from object images;

Automatic camera calibration from image sequences

amera calibration is vital information for all 3d reconstruction methods that are essentially based on triangulation of points and lines position in 3d space from their projections extracted from images. Common interactive 3d reconstruction software such as “Canoma” or “ImageModeler” relies on tedious manual matching of large set of points and lines between several input images for camera calibration. The amount of boring user interaction limits the applicability and ease-of-use of such systems. We have developed an automatic camera calibration system based on renowned sparse structure and motion (SAM) estimation approach, which is also a basis of film-industry camera calibration software like “Boujou” by 2d3.

Sparse SAM consists of two steps: feature tracking and camera calibration. We have developed a feature tracking framework that provides several enhancements compared to existing methods. Guided tracking and matching methods combined reduce the number of erroneously “lost” tracks.Uniform feature detection in images increase the robustness of camera tracking in presense of outlier-dense natural textures like trees.
Our current implementation of camera calibration relies on supplied intrinsic camera parameters (e.g. focal length) that can be obtained directly from camera or using the pattern-based off-line camera calibration toolbox. Extrinsic calibration (camera trajectory) is then calculated automatically using sequential or hierarchical scheme. Camera poses are then refined by bundle-adjustment.

  Example of feature tracking and SAM estimation

 User-guided reconstruction

Camera calibration step provides number of 3d points on scene object surfaces. This information is exploited for shape estimation by guided model fitting. User selects in several images object of interest by rectangular marque. Parametric model of selected type (box,cylinder,plane) is robustly fitted to 3d point cloud.

A lot of real-world objects can be modeled as a combination of simple and complex shape (e.g. cup). We have developed a sketch-based modeling tool for reconstruction such complex shapes as cup handle, which are untedious for the user.

  Cup handle reconstruction from 2 silhouettes  

Additional reconstruction approaches

We are currently investigating additional approaches to shape reconstruction that can boost the user-guided geometric fitting. One of them is quasi-dense reconstruction, which is a kind of intermidiate step between sparse and dense reconstruction. Sparse SAM suffers from low number of 3d points. Dense SAM is very unreliable and noise. Quasi-dense reconstruction benifits from precision and reliability of sparse matches, which are used as input, and greatly increases the number of matches and corresponding 3d points.

Reprojection and 3D view of quasi-dense 3D points

 Results

Originally our image-based modeling system were designed for small-sized objects. Such image sequences are taken in lab environment. Several examples of reconstructed objects are shown below. Pay attention to complex cup handle shape captured precisely.

Later feature tracking and camera calibration algorithms were adapted to video sequences of city streets objects, captured by hand-held generic photocamera like Canon IXUS 500. This allows reconstruction of large-sized objects like car garages and buildings:

Projects

Photogrammetric and 3D reconstruction C++ library
2006-2006
Manual distance measurement is real situations is often complicated, for example, when distance should be measured on the ceiling. Photogrammetric approach can greatly simplify the procedure by replacing actual measurement by taking a number of photos and tagging several points of scene in images.
Robust Estimation
2004-2005
One of the key tasks in the field of computer vision is to establish a relation between information extracted from images and some mathematical parametric model. The problem is complicated due to the presence of noise and ouliers, which are the part of the data that has not been generated by estimated model. The estimation method is called robust if it can give accurate results in presence of noise and outliers.
GML RANSAC Matlab Toolbox
2005-2005
The problem of parametric model estimation is very important in Computer Vision and many other fields of science. But often it became very complicated, due to presence of noise and a big percentage of outliers. In that case the RANSAC algorithm family can be useful and give a good solution.

Team

  • Anton S. Konushin, Lomonosov Moscow State University
  • Victor Gaganov, Lomonosov Moscow State University
  • Anton Yakubenko, Lomonosov Moscow State University
  • Vladimir Vezhnevets
  • Evgeny Lisitsin
  • Evgeny Seredenko
  • Andrey Khropov
  • Keun Ho Kim (Multimedia Lab, SAIT, 2004 year only)

Publications