In view of the problems of numerical instability and large amount of computation in recursive process, the modified square root cubature Kalman filter was put forward. The modified filter not only enhances the numerical stability, guarantees positive definiteness of the state covariance, but also increases accuracy. The simulation shows that the accuracy of the filter is higher than that of cubature Kalman filter, and the accuracy of cubature Kalman filter is higher than that of the extended Kalman filter; with the increase of dimension of nonlinear system, the estimation performance of the modified filter is better than that of the extended Kalman filter. The modified filter is an effective nonlinear filtering algorithm for high dimension of nonlinear system.