The two different sensors of the Kinect module, RGB and depth, are not calibrated and produce images whose coordinates will very much differ. Here we provide information and software allowing to calculate the coordinates in the RGB image (thus, the grayscale image in our dataset) for each pixel of the depth image. Note that this correspondence requires a correct depth information, therefore it cannot be calculated for pixels where the depth sensor gives back an error value.
We calibrated the Kinect module with a checkerboard, using Nicolas Burrus's RGBDemo-v0.4.0 tool and procedure. The technique is described here.
Intrinsic camera parameters for both cameras, camera distortion parameters for both cameras, as well as rotation and translation between rgb and depth cameras, are given as below (see the following opencv page for documentation on these values).
// Intrinsic parameters
// | fx 0 cx |
// | 0 fy cy |
// | 0 0 1 |
//
// stored as float[9] = { fx, 0, cx, 0, fy, cy, 0, 0, 1 }
// Distortion coeff float[5] = { k_1, k_2, p_1, p_2, k_3 }
const float rgbIntrinsics[9] =
{
5.2161910696979987e+02, 0., 3.1755491910920682e+02,
0., 5.2132946256749767e+02, 2.5921654718027673e+02,
0., 0., 1.
};
const float rgbDistortion[5] =
{
2.5673002693536984e-01, -9.3976085633794137e-01, -1.8605549188751580e-03,
-2.2232238578189420e-03, 1.2453643444153988e+00
};
const float depthIntrinsics[9] =
{
5.8818670481438744e+02, 0., 3.1076280589210484e+02,
0., 5.8724220649505514e+02, 2.2887144980135292e+02,
0., 0., 1.
};
const float depthDistortion[5] =
{
-1.8932947734719333e-01, 1.1358015104098631e+00,
-4.4260345347128536e-03, -5.4869578635708153e-03,
-2.2460143607712921e+00
};
// 3x3 rotation matrix
// | r11 r12 r13 |
// | r21 r22 r23 |
// | r31 r32 r33 |
//
// stored as float[] = { r11, r12, r13, r21, r22, r23, r31, r32, r33 }
//
const float RDepthAxisToRgbAxis[9] =
{
9.9996518012567637e-01, 2.6765126468950343e-03, -7.9041012313000904e-03,
-2.7409311281316700e-03, 9.9996302803027592e-01, -8.1504520778013286e-03,
7.8819942130445332e-03, 8.1718328771890631e-03, 9.9993554558014031e-01
};
const float TDepthAxisToRgbAxis[3] =
{
-2.5558943178152542e-02, 1.0109636268061706e-04, 2.0318321729487039e-03
};
The software package is available on our download page.
It is called from the command line and requires three arguments: a depth image, a corresponding rgb ou grayscale image, and a look up table which allows to map the depth values to pseudo-color. An example look-up table is provided with the package (file 'depth.lut'):
ATTENTION!! Due to a glitch in the sampling and streaming software, there is a temporal offset in the gray and depth videos. Frame X in the gray video corresponds to frame X-1 in the depth video! We apologize for this.
The software shows 3 windows: two windows displaying depth information (one in grayscale, the other in pseudo color), and one window displaying color/grayscale information. A click into one of the depth windows will also display the corresponding position in the color/grayscale window.
The most usefull functions and calibration datas are packed in depthToColorMapping.h file. An example of use is given in match-depth-rgb.cpp file.
The function depthPointToColorPoint() in file depthToColorMapping.h calculates the x and y coordinates in the RGB/grayscale image of a pixel with coordinates dx and dy in the depth images.
It is important to note that both images, depth and RGB/grayscale, must be undistorded before calling depthPointToColorPoint() function. For this purpose, an undistort() function is provided in file depthToColorMapping.h.
Depth values can be converted into meters using the following equation (from Nicolas Burrus' Kinect page)
float raw_depth_to_meters(int raw_depth)
{
if (raw_depth < 2047)
return 1.0 / (raw_depth * -0.0030711016 + 3.3309495161);
return 0;
}
Please note that only raw_depth values in range [ 0, 1084 ] give coherent distance.
The software has been written by Eric Lombardi.
Send questions to eric.lombardi (at) liris.cnrs.fr