In the field of gimbal targeting systems, image error tracking plays a crucial role in various applications, including object detection, enemy surveillance, and aircraft inspection. Enhancing image tracking performance presents a significant challenge due to singularity issues at the azimuth and elevation joints. To tackle this problem, this paper proposes a rotation-matrix-based tracking error compensation method centered on real-time object tracking. Specifically, our approach involves creating a virtual rotation frame that aligns the visual tracking frame with the gimbal base frame. Using our method, a gimbal with two degrees of freedom (DOF) can achieve superior tracking performance near the ±90° joint positions compared to conventional gimbal tracking methods. We compare the proposed method with existing approaches in the literature by assessing initial pose RMS error and singular pose RMS error through MATLAB Simscape simulations. The experimental results demonstrate that our method can reduce the line of sight RMS error by 89% in the azimuth position and by 99% in the elevation position, respectively.