In order to reduce the computational cost of sequence similarity detection algorithm (SSDA), a fast SSDA with double thresholds (DTSSDA) was proposed in this paper. DTSSDA introduces another threshold related with error-increasing-speed (EIS), when EIS of a reference region exceeds the threshold, operations cease for the region. Threshold of EIS can be determined adaptively according to correlation of inter-frame. Integral image was introduced in and search strategy of rough to fine is adopted in the algorithm. Experimental results show DTSSDA can reduce the computational cost in large scale and maintain almost the same accuracy as SSDA, so it is more suitable for real-time target tracking.