Pushing Images to a Server

Once the images are generated by the Kinect and ffmpeg, we need to push them to a server before they can be read by any other application. For each image that ffmpeg generates, we initially calculate the size the image (in is the File pointer).

fseek (in , 0 , SEEK_END);
int lSize = ftell (in);
rewind (in);

We then send this to the server so that it knows how many bytes it needs to read before it writes it to the file system.

sprintf(Buffer, "%d", lSize);
send(s, Buffer, sizeof(Buffer), 0);

Now we need to send the actual image to the server. We read MAX_BUF bytes from the currently opened file into the Buffer and send it over the socket to the server. If the remaining bytes in the file are less than MAX_BUF, fread fails, so we need to keep track of how many bytes were read and how many more need to be read. If the remaining bytes are less than MAX_BUF, we read and send only the remaining number of bytes to the server. 

while (1) {
bzero(Buffer, sizeof(Buffer));
len = fread(Buffer,sizeof(Buffer),1, in);
if(len <= 0) { 
break; 
}
send(s, Buffer, sizeof(Buffer), 0);
sz+= len*MAX_BUF;
int rem = lSize - sz;
if (rem > 0 && rem < MAX_BUF) {
len = fread(Buffer, rem, 1, in);
send(s, Buffer, rem, 0);
sz+= len*MAX_BUF;
break; 
}
}

Getting Images From The Kinect

Once all the libfreenect Kinect drivers were successfully installed, we had to figure out how to adapt the available methods to fit our needs. There is not much  documentation available about libfreenect, so the easiest way to go about it was to base our program off the exisiting sample programs that comes with the libfreenect library. While the glview program was a good start to help introduce us to Kinect's features, the record.c program provided more insight into how to get the RGB and Depth streams.

Before you can start getting video and depth information from the Kinect there are a few steps you need to do to initialize the device. First include the libfreenect header file. Then you need to get a context, select the subdevices and finally open the device.

freenect_context *f_ctx;
freenect_device *f_dev;
freenect_init(&ctx, 0)
freenect_select_subdevices(ctx, (freenect_device_flags)(FREENECT_DEVICE_MOTOR | FREENECT_DEVICE_CAMERA));
freenect_open_device(ctx, &dev, 0)

If all this works correctly, you can start the video and depth.

freenect_set_video_mode(dev, freenect_find_video_mode(FREENECT_RESOLUTION_MEDIUM, FREENECT_VIDEO_RGB));
freenect_start_video(dev);
freenect_set_depth_mode(dev, freenect_find_depth_mode(FREENECT_RESOLUTION_MEDIUM, FREENECT_DEPTH_11BIT));
freenect_start_depth(dev);

At this point the Kinect RGB and IR cameras start doing their job. But to be able to do something with the data that the Kinect creates, you need to identify callback functions that get called everytime RGB or depth data is received.

freenect_set_video_callback(dev, rgb_cb_ffmpeg);
freenect_set_depth_callback(dev, depth_cb_ffmpeg);

Now whenever the program receives video and depth information from the Kinect, it calls the rgb_cb_ffmpeg and the depth_cb_ffmpeg functions respectively. Since we decided to send frames over a TCP socket rather than use a streaming protocol like RTSP, we use ffmpeg to generate jpeg images of the frames. The following commands are written in the callback functions to create a jpeg image on the file system whenever it receives data.

snprintf(cmd, 1024, "ffmpeg -pix_fmt rgb24 -s %dx%d -r 5 -f rawvideo -vframes 1 "
"-i /dev/stdin -f image2 -r 5 Pics/sample%d.jpg",
FREENECT_FRAME_W, FREENECT_FRAME_H, rgb_count);
proc = popen(cmd, "w");
fwrite(rgb, freenect_get_current_video_mode(dev).bytes, 1, proc);

Once you've written all this code*, you want to see your Kinect in action. The easiest way to compile your program is to place it in the include subdirectory of your libfreenect directory. To compile, run the following:

gcc -lfreenect -o yourProgram yourProgram.c

If that doesn't work, try:

cc -c yourProgram.c
cc  yourProgram.o -o  yourProgram -lfreenect

To run the program:

sudo LD_PRELOAD="/usr/local/lib/libfreenect.so" ./yourProgram

Remember that you need superuser privileges to run the program!



*NOTE: The code in this blog post is not complete. It is only intended to get you started with your Kinect.