Now we get to the sensing part that maybe you were thinking about initially. This is sensing of the world outside of the turtlebot (not using proprioceptive sensors). The default version comes with a kinect sensor, so it has an on-board RGB camera and also a depth image. To access it you would have to do one of the following two things (1) run the minimal launch file plus a sensor launch file separately (can you figure out what this second launch file should be), or (2) run a launch file with more services specified to be launched within in (can you find this richer launch file). Of course, there is a third more appealing option from a long term understanding perspective. That would be to create your own launch file by figuring out what the above launch files do and then custom writing one that does only what is needed by the code.

What are the ROS topics that must be subscribed to in order to get these sensor measurements? Identify the main two that provide the proper raw data associated to the image stream and to the depth stream for the kinect camera. Name a few of the different imagine sensor topics and explain how they differ from the raw versions.

1. Properly launch the necessary services to have kinect sensor data be published (beyond the linescan). Show that this worked by doing a rostopic list which should then output the kinect sensor messages.
2. Python and ROS fortunately have harnessed the power of OpenCV. Using the OpenCV bridge code for python, demonstrate the ability to subscribe to the sensor data and display it. The libraries to import are ``cv2`` and ``cv_bridge``. The first one gives access to OpenCV code. The second one lets ROS and OpenCV work together (usually through conversion routines that translate data forms between the two system snad their standards). When run, this should just display to separate windows the color image data and the depth data. The ``cv2`` function is called ``imshow`` which expects the data to be in a specific format (either from 0 to 1, or from o to 255, I forget). The depth data may have to be normalized for proper display.
3. Now that we can get images, let's try to do something with them. The classic first pass is to perform color-based detection. Identify a color that you think is unique within the world, and most likely for which you have an object of that color (fist sized to head sized). Identify the color range and perform color-based detection on the Turtlebot image data. Instead of plotting the color image, plot the detection image. It should show white (or true) values when the target is in the field of view, and nothing when it isn't (assuming that your color is pretty unique relative to the Turtlebot's environment).
4. Next, perhaps it would be useful to know how far the object is from the image. So long as the object is sufficiently far away, the color values and the depth values will align. Take the binary image from the previous step and apply it to the depth image. Extract only the depth values that correspond to true binary image values. Get the approximate depth of the object pixels by computing the median or the average of the extracted depth values (you may want to go through the depth data and remove the NaN values as these are bad measurements). While you are at it, extract the coordinates of the binary mask and compute the mean of the x and y values (this is called taking the centroid of the detected object blob). Print out these values to the screen.
5. Let's make our system more capable. Rather than print the depth and target centroid to the screen, implement some kind of feedback on the data.

As usual, here are some Hints.

Adventure Contents