Abstract: Obstacle detection is crucial for the autonomous obstacle avoidance of extraterrestrial rovers, traditional image-based obstacle detection methods can only locate obstacles in 2D image plane, requiring additional measurement methods such as stereo matching to obtain depth information and then determine the 3D positions of obstacles. However, stereo matching faces challenges of high computational cost and decreased accuracy when dealing with complex environments. Therefore, we propose an implicit 3D representation learning method for extraterrestrial obstacle detection. It encodes the potential three-dimensional coordinates of each point into image features, and the generated features can effectively establish an implicit conversion from 2D images to 3D space, thereby enabling direct prediction of the 3D positions of obstacles. Experiments conducted on Mars surface images collected by the Spirit rover demonstrate that the proposed method can effectively identify locations and sizes of obstacles, achieving 85.5% average precision. The proposed method in this study presents an innovative framework for planetary surface obstacle detection, with substantial potential to advance autonomous navigation capabilities in lunar/Martian exploration rovers.