A novel layered object tracking algorithm in FLIR imagery was proposed based on mean shift algorithm and feature matching. First, infrared object was modeled by kernel histogram. Bhattacharyya coefficient was used to measure the similarity between object model and candidate model. The object was then localized by mean shift algorithm rapidly and efficiently. Because of the low contrast between infrared object and background, low dynamic range of gray level, however, the mean shift tracking results may bring some errors. So, feature matching was employed to eliminate the tracking errors. Feature points were extracted in template object and candidate area by Harris detector. Finally, the accurate localization of infrared object was realized by matching the feature points with the measurement of improved Hausdorff distance. Experiment results verify the effectiveness and robustness of this extraction algorithm which can improve the tracking performance efficiently.