The drone industry not only brings convenience to the society, but also poses a serious threat to the social and military security. Therefore, it is very important to quickly and accurately locate and identify unknown UAVs. In this paper, a detection and recognition method of low-altitude UAV based on infrared image was proposed by building a deep residual network and prediction network. First, the depth features of infrared images were extracted by residual network, and then the prediction network uses multi-scale model structure to predict the location and category of the extracted features. Finally, the repeated results are eliminated by means of non-maximum suppression. Compared with other methods, the experimental results show that the mAP value reaches 78.21%, the detection speed is about 28 images/second, and the detection and recognition performance is better than other methods.