Getting Images From The Kinect

Once all the libfreenect Kinect drivers were successfully installed, we had to figure out how to adapt the available methods to fit our needs. There is not much  documentation available about libfreenect, so the easiest way to go about it was to base our program off the exisiting sample programs that comes with the libfreenect library. While the glview program was a good start to help introduce us to Kinect's features, the record.c program provided more insight into how to get the RGB and Depth streams.

Before you can start getting video and depth information from the Kinect there are a few steps you need to do to initialize the device. First include the libfreenect header file. Then you need to get a context, select the subdevices and finally open the device.

freenect_context *f_ctx;
freenect_device *f_dev;
freenect_init(&ctx, 0)
freenect_select_subdevices(ctx, (freenect_device_flags)(FREENECT_DEVICE_MOTOR | FREENECT_DEVICE_CAMERA));
freenect_open_device(ctx, &dev, 0)

If all this works correctly, you can start the video and depth.

freenect_set_video_mode(dev, freenect_find_video_mode(FREENECT_RESOLUTION_MEDIUM, FREENECT_VIDEO_RGB));
freenect_start_video(dev);
freenect_set_depth_mode(dev, freenect_find_depth_mode(FREENECT_RESOLUTION_MEDIUM, FREENECT_DEPTH_11BIT));
freenect_start_depth(dev);

At this point the Kinect RGB and IR cameras start doing their job. But to be able to do something with the data that the Kinect creates, you need to identify callback functions that get called everytime RGB or depth data is received.

freenect_set_video_callback(dev, rgb_cb_ffmpeg);
freenect_set_depth_callback(dev, depth_cb_ffmpeg);

Now whenever the program receives video and depth information from the Kinect, it calls the rgb_cb_ffmpeg and the depth_cb_ffmpeg functions respectively. Since we decided to send frames over a TCP socket rather than use a streaming protocol like RTSP, we use ffmpeg to generate jpeg images of the frames. The following commands are written in the callback functions to create a jpeg image on the file system whenever it receives data.

snprintf(cmd, 1024, "ffmpeg -pix_fmt rgb24 -s %dx%d -r 5 -f rawvideo -vframes 1 "
"-i /dev/stdin -f image2 -r 5 Pics/sample%d.jpg",
FREENECT_FRAME_W, FREENECT_FRAME_H, rgb_count);
proc = popen(cmd, "w");
fwrite(rgb, freenect_get_current_video_mode(dev).bytes, 1, proc);

Once you've written all this code*, you want to see your Kinect in action. The easiest way to compile your program is to place it in the include subdirectory of your libfreenect directory. To compile, run the following:

gcc -lfreenect -o yourProgram yourProgram.c

If that doesn't work, try:

cc -c yourProgram.c
cc  yourProgram.o -o  yourProgram -lfreenect

To run the program:

sudo LD_PRELOAD="/usr/local/lib/libfreenect.so" ./yourProgram

Remember that you need superuser privileges to run the program!



*NOTE: The code in this blog post is not complete. It is only intended to get you started with your Kinect.