Abstract: A novel algorithm for attitude estimation of deep spacecraft based on star sensor is proposed. The Lie group is used to describe the attitude, which avoids the non-uniqueness and complex calculation of the quaternion conversion into the attitude matrix. A description of the motion model based on star sensor and the kinematics model of space rigid body in Lie group is given. A method based on Lie group is proposed to determine the dynamic attitude of deep spacecraft. The linearization model solves the errors generated by the traditional nonlinear model in the filtering process, and eliminates the steps of converting the quaternion into the attitude matrix and reduces the computational complexity. In the simulation, the proposed method is compared with the traditional quaternion attitude determination algorithm. The results prove that the algorithm has better stability and accuracy.