CS4733 - Homework 4
PPP: Personal Pet PER
Originally Written by Matei Ciocarlie
Description
The goal of this project is to create a Personal Pet PER that will
obediently follow you around. The sensing method will be
vision, using the PER's webcam and color finding algorithms to track a
particular color that you are wearing. To make your task easier we
suggest wearing a very bright and distinctive color or carry a distinctly
colored object.
More specifically, your code must accomplish the following tasks:
- bring up a GUI where the PER can be asked to take a picture of
the
person it is supposed to follow
- inside this picture, the user must be able to tell the PER what
color
it should track by clicking on one or more pixels
- once the color has been selected, the person will start walking
around.
The PER has to follow her / him.
- seek mode: at some point,
the PER might lose track of the person it's
following. Your code has to detect this (see notes below), and the PER
must enter a seek
mode where it is constantly moving the camera and looking for the
person. If the person re-enters the PER's field of view, the PER must
resume tracking.
- robustness: try to make
your code as robust as possible. Some clues
follow, but you can also experiment and come up with your own ideas for
making the code more reliable. Attach a text file to your submissions
explaining the methods that you have tried (even if they don't fully
work as expected), and how they have
influenced tracking robustness. Creativity will be appreciated.
- safety: the PER should
not hit the person that it is following. Other then that, assume a
friendly environment with no obstacles.
Color updating
One of the biggest problems you will have to address in order to make
your code more robust is color constancy. Even if the PER starts out
with good tracking parameters, as it is following the person around
conditions change. This can mean changes in lighting, the orientation
of the person relative to the PER's webcam etc. All these will change
the tracked color as it is perceived by the PER, and can ultimately
make
the PER lose track of its target. One solution is to keep updating the
tracking paramters as the PER drives along. At some short intervals,
you can check how well the colored blob being tracked falls within the
tracking thresholds. If you find the color of the blob is changing, you
need to update the tracking parameters to match, before the color
changes so much
that you lose it completely.
To find representative values for the color of the blob currently being
tracked, you can use the color of its centroid. However, making such
decision based on only one pixel is not recommended, as it can be
heavily influenced by noise. Instead, average the values of a number of
pixels around the centroid, or suggest your own methods for a live
update of the tracking paramteres.
Getting started - the PER API Vision Class
You will find that the Vision Class
of the PER API already contains a
good percent of the features you need to implement. Use it as your
starting point and feel free to build on it: it is OK if your code is
added on top of this class, as long as you clearly mark all your
additions and contributions.
You might also want to look at how the Vision Class algorithms decide if
the tracking information is sufficiently reliable to act on. It uses
the number of pixels that are currently being tracked, as well as the
extent of the rectangular box that encloses these pixels.
Note: be careful when using
functionality built into the Vision
Class.
There might be subtle differences between that code and what you need
to do. For example, the Vision Class
contains a line following
algorithm. This algorithm assumes that the object being tracked is on
the ground. What kind of information does the algorithm extract from
this? Would this work in your case?
Computation: on the PER vs.
on the laptop
Transferring images from the PER to the laptop where your code is
running is slow. The PER API mentions that the completion time for TakePicture(...) is about 300 ms.
In practice, you can expect that streaming images from the PER will be
refreshed on the laptop at about 1 frame per second. Taking pictures
very often can slow your tracking code down, and prevent you from
obtaining the smooth continous motion that helps robust tracking.
In order to alleviate this problem, a lot of the tracking code that the
Vision Class uses runs on the
PER, and images are not transferred to the laptop. By studying the Vision Class code, you will notice
that the tracking information it uses comes from the PER's ReceiveThread which is used to
receive information from the tracking code that is running on the PER.
The ReceiveThread will
contain a lot of the information you need, such as the centroid of the
currently tracked blob, the number of pixels in the blob etc. Take a
look at the ReceiveThread
documentation to see exactly what it can provide. This information is
updated at high frequency by the code running on the PER, but does not
transfer any images to the laptop.
You can download and study the code running on the PER from the PER
Website (the exhibit22
download). However, at this moment we can not recompile this code, so
you can not make any changes to it. If you need more information
than the ReceiveThread is
providing, you will need to take pictures explicitly (with TakePicture(...) or a similar
method) and process them on the laptop. Keep in mind however that doing
this too often will slow your code down.
Submission
- Deadline: April 19th
We will demo your programs in class on April 19th
- What to submit: all your
source code. Please do not
submit compiled files or executables
- Please include a detailed README
file with compilation instructions, as well as descriptions of the
methods you have implemented and their effect on tracking.
- How to submit: via email
( dal2103@columbia.edu )