3 PiNoculars – Custom 3D Camera – Casimir "Caz" Sowinski

Pinoculars

Stereo Vision with Cheap Sensors

Abstract

There are many 3D cameras on the market today employing a host of techniques to extract a three-dimensional image from a set of sensors. Although open source drivers exist for some of them, most parameters are fixed, and many are presented as black boxes with limited interfaces to test novel capturing, recording, and sensor fusion ideas. In this project, I attempted to build a platform to use inexpensive cameras and embedded devices to produce a two-camera array for depth extraction with the goal of eventually scaling the system to include an arbitrarily large sensor array for super HD imaging, high fidelity depth extraction, and conversion to point cloud arrays. I used a platform called Robot Operating System (ROS), which makes heavy use of OpenCV for much of the image processing pipeline. Using this framework sped up development as the first step, depth processing with two cameras, is well understood and standardized packages exist for these purposes. ROS is completely open source, and expanding packages to fit particular purposes is allowed and actively encouraged. I achieved moderate success in producing depth maps from the two sensors.

Background and Related Work

There are many approaches to extracting depth information from a scene. The popular consumer 3D cameras, Microsoft’s Kinect and the Intel Realsense, use structured light and a single imager to produce a depth map. This technique illuminates a scene with a known pattern of IR laser dots and uses the apparent size of their reflections to infer depth. Another technique uses a sheet of light that sweeps a scene. Using the known positions and angles of the camera and light source, the depth of a scene can be inferred one plane at a time. LiDAR operates similarly to radar, where it scans pulses of laser light onto a scene and uses the time-of-flight to calculate depth values. A similar technique called ToF uses a single light pulse and high speed CCD imager to take a rapid succession of images to find the depths of all portions of a scene. The technique used in this project is stereo imaging. It relies on multiple cameras getting images from different perspectives and using the difference of position of features in the images to infer depth. The closer a feature is to the cameras, the greater the disparity of position it will have on the two cameras. This technique has the advantage of being completely passive, but suffers from ambiguity of depth for textureless surfaces.

To accurately map a point from an imaging sensor to a point in 3D space, a number of techniques need to be used to account for common failings and distortions from imaging systems, especially cheap systems. The main causes can be mathematically modeled and compensated for to produce a more accurate image. There are two groups of parameters that OpenCV seeks to find to describe the characteristics of an imaging system: intrinsic and extrinsic parameters. Intrinsic parameters describe the geometric layout of the system, which includes misalignment of the sensor from the optical axis and the distance between the sensor and the aperture. Extrinsic parameters describe the distortions introduced by the lens.

Figure 1 – Modeling of a camera

The image above shows an adaptation of a simple pinhole camera. The center of projection (the pinhole) and the image plane (the sensor) are swapped to simplify the math without loss of accuracy. The sensor may be off axis in both the x and y axes, and these are described by the variables c_x and c_y. The distance between the sensor and the center of projection is described by the variables f_x and f_y. Two variables are needed to account for the likely condition of individual pixels being rectangular instead of square.

Using projective geometry, this can be arranged into:

Extrinsic parameters describe radial distortion and tangential distortion. Radial distortion results from lenses that are not parabolic and bend light more strongly near the edges, while tangential distortion describes the lens not being fully parallel to the imaging sensor.

Figure 2 – Sources of distortion, radial distortion on the left, and tangential distortion on the right

Figure 3 – Effects of different types of distortion on an image

Radial distortions are usually 0 in the center of the image and increase as the radius increases. A few terms of a Taylor series expansion of an equation describing the lens distortion is usually good enough to compensate for this effect. Using the first three terms, we get:

Tangential distortion can be characterized by:

To find these parameters, an object of known properties may be used. Using this technique also introduces 6 more parameters: the 3 spatial coordinates and 3 rotational values. By taking a series of images at different locations, distances, and rotations, the whole system of equations may be solved for. In addition to those parameter,s there are more that describe the relative positions of the two sensors and the divergence they have from being on the same plane and at the same angle as each other. After this process, the compensated images need to be cropped to a smaller size that represents the largest common rectangle shared between the two images.

Figure 4 – Camera calibration

Technical Approach

To begin the project, I set up Ubuntu Mate 16.04, OpenCV 3.2, and ROS Kinetic on my desktop computer and on two Raspberry Pi 2’s. I gave all three devices a static IP and modified the hosts tables to facilitate routing via hostnames instead of IP addresses. I modified the Bash files that the terminal uses when it creates a new instance to set itself up and set environment variables (.bashrc) to set a few ROS environment variables to allow for seamless network communications over multiple hosts. Specifically, I added the lines:

export ROS_ID=192.168.1.31

export ROS_MASTER_URI=http://caz-lin:11311

source /opt/ros/kinetic/setup.bash

source /home/caz/catkin_ws/devel/setup.bash

After network communications were dealt with, I wrote C++ programs that would broadcast a stream of 640×480 images from the cameras at 30 frames per second. This uses OpenCV’s VideoCapture object and a ROS package called cv_bridge to convert Mats into ROS image messages. To broadcast the images, I used a ROS package calledimage_transport. Each Raspberry Pi would run a slightly different version of this program to broadcast their images with changes for the messages to be traced back to a particular camera. I wrote subscriber nodes that would gather the messages on the desktop host for further processing.

Once image streaming was taken care of, I moved to calibration. I used a ROS package for calibration. It uses OpenCVs calibration including functions: cv::findChessboardCorners() and cv::drawChessboardCorners(). The ROS implementation uses a ROS node called camera_calibration and uses a Python script that can handle monocular and stereo setups. After around 50-60 images, it had found the matrices for K (intrinsic), D (extrinsic), R (rectification), and P (projection).

I began to write my own stereo correspondence program that subscribed to the processed image streams. This code is based on the Stereo calibration, rectification, and correspondence code given in Learning OpenCV. I experimented with this and a ROS package that uses a very similar structure and the same techniques and OpenCV function calls, but with the ROS wrappers already in place.

Experimental Results

The system produces disparity maps at between 10 and 15 frames per second. It can be recalibrated using different distances between the cameras and with different values of vergence between the cameras. One reason that I chose to work mainly with the ROS package stereo_image_proc was to be able to change the large number of parameters for the Block Matching and Semi-Global Block Matching algorithms at run time. I felt that my time would be better spent getting a sense of the parameter space than writing a HighGUI trackbar interface to achieve the same results. I used a package called dynamic_reconfigure that offers this service for packages that have reconfigurable parameters. The parameters cover three different aspects: pre-filtering, correspondence search, and post-filtering.

I had modest success with the final results of this project. I only calibrated the cameras once and should probably revisit that step. I tried many pre-filtering techniques before streaming the images to the block matchers, including changing contrast and using various convolutional filters on the streams. The intelligability of the disparity maps seems to be highly dependent on the parameters of the filtering and correspondence matching. To find a better set of parameters, my next steps are to use disparity maps from a Realsense or Kinect as the ground truth and have an automation similarity matching program perform a brute force parameter space exploration. Using visual inspection and trying this manually would be far too time-consuming.

Below are some samples of the disparity mapping results:

Figure 5 – RGB image of an office chair

Figure 6 – Extracted depth image of an office chair

Figure 7 – RGB image of a calibration board

Figure 8 – Extracted depth image of a calibration board

Conclusions

Depth sensing is quite a difficult task, and there are many steps between getting a few cameras and having meaningful depth data. Getting the network communications setup took longer than I had anticipated, but it works fairly well. For the purposes of disparity mapping, I only use 640×480 at 30 frames a second, but the setup can easily send HD at that frame rate. I wish to expand this to use larger arrays of networked cameras and experiment with mapping more than one side of an object to a 3D point cloud at a single time. Having used ROS for some of the work will translate to using this system on a robotic platform and will integrate nicely with all other systems sharing the devices running the system. Point cloud streams are useful for navigation, mapping, recognition, and a host of other activities pertinent to robotics.