Object tracking project

Required functionalities

Each group should implement a system with the following functionality:

1. Tracking of objects in a simple sequence
2. Management of object identity (solve the assignment problem for detections over time)
3. Evaluation of results, relative to ground truth. See details here.

In addition to the above, implement functionality that can deal with two of the following:

1. occlusions, using one (or several) of the following:
   • foreground modelling (i.e. region descriptors)
   • prediction (e.g. using a Kalman filter)
   • ground plane modeling (camera hand over)
   • KLT tracking or optical flow
2. shadows / specular reflections(bright spots)
3. spurious motions using a multi-modal background model

For groups with only three members, one of the above is sufficient. For groups with five members all three are required.

Data sets

There are several data sets for motion analysis available on the internet, but we recommend that you use the following

• CAVIAR Low-res (384x288), with ground truth bounding boxes. Small, and easy to start with.
• PETS 2009 E.g. S2.xx sequences are RGB with ground truth bounding boxes. If they are too high-res for your code, downsample to e.g. 640x480.
• MOT 2015 Multiple object tracking challenge. Only some of these sequences have stationary cameras.
• Edinburgh office monitoring video dataset

These are 720p and 1fps videos of people sitting at desks. Illumination changes slowly, but foreground motion can be very fast due to the low framerate.

• changedetection.net Has ground truth for background model, but not for object identity.
• IVSS sequences See description below.

The IVSS dataset was produced in the IVSS project at CVL. It is available only in our local file system, e.g., if you are in the student computer halls at ISY. These sequences contain signficant amount of shadows, both in terms of the moving objects that cast shadows on the background and features of the background that cast shadows onto the moving objects. The path to the sequences is:

• /courses/TSBB15/sequences/ivss/ This disk is mounted on demand, so you first need to run module add courses/TSBB15 (as in the labs), or cd into the folder.

The sequences in the ivss/ folder are:

• harder1, images 1 to 2800. Has changing lighting conditions, from overcast to moving scattered clouds.

Shadows appear and disappear. People fetching parked trucks.

• harder2, images 1 to 941. An interesting challenge, don’t start with this one!
• Renova_20070907_081432_Cam10_0005, images 1 to 1901. Shadows.
• Renova_20080420_083025_Cam10_0000, images 1 to 1101. Shadows.
• Renova_20080618_083045_Cam10_0002, images 1 to 2101. Almost no shadows, no reflections, interesting occlusion starting at frame 1255. Suitable to start with.

See the computer lesson material for alternative ways to read these images into python. Filename strings may be built using string formatting, e.g. file_name = ‘dataRenova_20080420_083025_Cam10_0000_{:05d}.jpg’.format(frame_number)

Code

For functionalities that you are meant to implement yourseleves, the use of software packages that solve them is explicitly forbidden. If you are unsure about a particlular piece of software, ask your guide.

Deliverables

In order to pass, each project group should do the following:

1. Make a design plan, and get it approved by the guide.

   The design plan should contain the following:

   • A list of the tasks/functionalities that will be implemented
   • A group member list, with responsibilities for each person (e.g. what task)
   • A flow-chart of the system components
   • A brief description of the components
2. Deliver a good presentation at the seminar
3. Hand in a written report to the project guide.

The report should contain the following:

• A group member list, with responsibilities for each person
• A description of the problem that is solved
• How the problem is solved
• What the result is (i.e. performance relative to ground truth)
• Why the result is what it is
• References to used methods

We recommend that you use the CVPR LaTeX template when writing the report.

References

• John Wood, Statistical Background Models with Shadow Detection for Video Based Tracking, Master Thesis 2007. Linköping University Note the typo in Algorithm 7 on p 44. Line 11 should read: “if B=0 then”.
• Håkan Ardö, Multi-target Tracking Using on-line Viterbi Optimisation and Stochastic Modelling, PhD thesis, 2009, Lund University
• O. Barnich, M. van Droogenbroeck, ViBe: A Universal Background Subtraction Algorithm for Video Sequences, IEEE Transactions on Image Processing(TIP) 2011

Other pointers to related stuff:

• The supplementary course TSBB19 Machine Learning for Computer Vision (covers detection, tracking and tracking by detection).
• The PhD course Target Tracking at Automatic Control (covered detection filtering only)
• Martin Danelljan, Learning Convolution Operators for Visual Tracking, PhD thesis, 2018, Linköping University (covers learning for visual tracking)