Magnetometer nuisance parameter is changed when high-speed spinning projectile is launched, and orientation measuring accuracy can be affected by poor measurement output. A novel calibration algorithm was introduced for high-speed spinning projectile applications. Magnetometer error model was used for the system dynamic equations, and the measurements were sensor outputs. Adaptive recursive least squares estimation method was used to estimate the magnetometer nuisance parameter online. The 152 mm diameter projectile was simulated, and results show that the adaptive filter algorithm has strong feasibility, which can effectively weaken the impact of various noise sources of the attitude detection system.