Abstract: To address optical parameters are affected by the complex environment in space will produce a certain bias in spacecraft on-orbit operation, it will seriously affect the accuracy of spacecraft attitude estimation. The traditional on-orbit calibration method generally base on star angular distance invariance, but it has high computational complexity, the computational and storage resources of deep space probe are very limited, it is difficult to be realized. This paper introduces a calibration method based on singular value decomposition invariance, the size of the singular value is used to measure the observability of the calibration system, on this basis based on the observability to optimize the selection of star distribution and combination, combined with extended Kalman filtering to calibrate the optical parameters of the star tracker. The simulation results show that compared with the traditional star optimal selected model, the optimal selected model proposed in this paper has an advantage in calibrated accuracy and can better suppress the star observational error.