Abstract: Measurement error is the main factor that affects the accuracy of the autonomous navigation system for deep space exploration. In this paper,aiming at restraining the measurement error of autonomous navigation system,an autonomous celestial navigation method of deep space probe based on coplanar constrained auxiliary measurement is proposed. Based on the model of the system nonlinear inequality geometric plane constraints, Sequential Quadratic Programming (SQP) is used to deal with the nonlinear inequality constraints of deep space probe autonomous celestial navigation system,which directly aids in reducing the measurement error of the deep space probe autonomous navigation system. CKF-SQP,nonlinear constrained filtering method with measurement optimization, estimates the states of deep space probe autonomous navigation system to further reduce the system random error. Simulation results show that the proposed method can effectively suppress the measurement error and achieve highprecision autonomous navigation of deep space probes. The method can provide a feasible high-precision autonomous navigation method for deep space probe.