Adaptive on-line performance evaluation of video trackers
[Description][Related publication][Dataset][Sample results][Software][Additional references]
This page presents sample results of our framework to evaluate the performance of tracking algorithm without using ground-truth data (named ARTE, Adaptive Reverse Tracking Evaluation). The framework is divided into two main stages, namely the estimation of the tracker condition to identify intervals during which a target is lost and the measurement of the quality of the estimated track when the tracker is successful. A key novelty of the proposed framework is the capability of evaluating video trackers with multiple failures and recoveries over long sequences. Successful tracking is identified by analyzing the uncertainty of the tracker, whereas track recovery from errors is determined based on the time-reversibility constraint.
Original test sequences with ground truth data and a MATLAB implementation are also provided.
Adaptive on-line performance evaluation of video
J. SanMiguel, A. Cavallaro and J. Martínez
IEEE Transactions on Image Processing , 21(5): 2812-2823, May 2012
Juan C. SanMiguel - show email
CAVIAR dataset: targets P1 (Browse_WhileWaiting1), P2 (OneLeaveShopReenter1front), P3 (OneLeaveShopReenter2front), P4 (ThreePastShop2cor) are available here
PETS2001 dataset: targets P5-P10 (Camera1_testing) are available here
PETS2010 dataset: targets P11-P14 (S2_L1_view001), P15-P16 (S2_L2_view0001) and P17-P18 (S2_L3_view001) are available here
Targets F1 (seq_bb), F2 (seq_mb), F3 (seq_sb) and F4 (seq_villains2) are available here
VISOR dataset: targets F5 (visor_video_1) and F6 (visor_video_2) are available here
The ground-truth for all the targets in the dataset is available here
Download the videos1 and images from the links below to look at the sample experimental results of our proposed framework. A color-based particle filter was used to generate the tracking results . Additionally, a comparison with representative state-of-the-art approaches for empirical standalone quality evaluation (Observation Likelihood (OL) , covariance of the target state (SU) , frame-by-frame reverse-tracking evaluation using template inverse matching (TIM)  and full-length reverse-tracking evaluation using the same applied tracking algorithm (FBF) ) is included.
Sample images for target P6
1The XviD ISO MPEG-4 codec is needed to watch the video files (download it here)
The software listed in this section can be freely used for research purposes.
Two separate tools are provided for the main stages described in the paper (also available in GitHub):
Additionally, the implementation of the particle filter tracker and data needed for the above tools are provided:
*If you use this software, please cite the related reference.
 K. Nummiaro, E. Koller-Meier, and E. Van Gool,
“An adaptive colour-based particle filter,” in Image
and Vision Computing, 21(1):99–110, 2003.
 N. Vaswani, “Additive change detection in nonlinear systems with unknown change parameters,” in IEEE Trans. on Signal Processing, 55(3):859–872, 2007.
 E. Maggio, F. Smerladi, and A. Cavallaro, “Adaptive multifeature tracking in a particle filtering framework„” in IEEE Trans. on Circuits and Systems for Video Technology, 17(10):1348–1359, 2007.
 R. Liu, S. Li, X. Yuan, and R. He, “Online determination of track loss using template inverse matching,” in Proc. of the Int. Workshop on Visual Surveillance, Marseille (France), 17 October 2008.
H. Wu, A. Sankaranarayanan, and R. Chellappa, “Online empirical evaluation of tracking algorithms,” in IEEE Trans. on Pattern Analysis and Machine Intelligence, 32(8):1443–1458, 2010.