Visible and Thermal Infrared Light Target Tracking via Multi-Adapter Network with Feature-Wise-Transformations

Abstract Existing deep learning-based RGBT target tracking algorithms focus on extracting the features of each modal image separately by using the same CNN model, and then stitching the extracted features to achieve the fusion of multimodal data. However, the reliability of single-modal visible light images and infrared light images will degrades with time shift, and when one of the modal data is unreliable, the related operations of feature extraction and fusion of the two modal data tend to introduce discriminative information that impairs tracking robustness, and makes it easy to lose the initial target in long-time tracking. To this end, we design a Feature-wise transformations network for feature modulation for RGBT target tracking to ensure that the information fusion between the two modalities is as complementary as possible to the strengths of the multimodal data. Most of the existing RGBT tracking algorithms focus on the fusion of multimodal information, and the selection of the target frame mainly depends on the classification confidence of classifying the object as the target or background. Due to the classification confidence and the localization confidence may not match, it is easy to cause inaccurate target localization. In order to improve the localization accuracy of the tracking process, we optimize the instance adapter in the MANet al.gorithm with a new method of updating the classification confidence based on the sample box with the highest localization confidence, and design a new method of updating the localization confidence based on the sample box with the highest classification confidence to obtain the final candidate sample box for bounding box regression. The three improvements are applied to the MANet al.gorithm to construct a new RGBT target tracking network, and the test results on three benchmark datasets of RGBT target tracking show that our improvements are effective in improving the accuracy and success rate of the tracker.