Perceiving and understanding the surgical environment play crucial roles in achieving autonomous robotic surgery, as they provide essential information for planning, navigation, and executing surgical tasks. Surgical perception is very challenging due to the unstructured and patient-specific nature of the surgical environments, which involve dynamic, deformable objects such as surgical instruments and soft tissue, as well as flowing fluids like blood. 3D Scene Reconstruction and Deformation Tracking Capturing the 3D geometry of surgical scenes and tracking the changes and deformations of target organs or tissue are core tasks in surgical perception. This information can provide surgeons with an updated ‘map’ of the surgical site and potentially an intuitive display of the current surgical progress. Additionally, robots will require this 3D data in the future to autonomously plan and perform surgical procedures. We have developed a series of works, referred to as the Surgical Perception (SuPer) framework, which accurately reconstructs tissue surfaces and tracks their deformation by leveraging emerging deep learning technologies. Semantic Segmentation Identifying different surgical instruments and types of tissues or lesions is crucial for understanding surgical scenes. However, in surgical applications, semantic segmentation performance is often limited by the availability and quality of annotated datasets, which are costly to produce due to the need for domain expertise. As part of our work on semantic segmentation, in one project aimed at automating hemostasis management, we developed HemoSet, the first dataset to segment blood in surgical settings. We further benchmark the performance of existing segmentation algorithms on our dataset and provide recommendations for future research in this area. This work received the Best Student Paper Award at ISMR 2024.