Surgical Task Automation

Surgical robots, such as Intuitive Surgical’s da Vinci Surgical System, have brought about more efficient surgeries by improving dexterity and reducing surgeon fatigue through teleoperational control. While these systems are already providing great care to patients, they have also opened the door to a variety of research, including surgical task automation. Surgical task automation has furthermore been an increasing area of research in an effort to improve patient throughput, reduce quality-of-care variance among surgeries, and potentially deliver automated surgery in the future. We are developing algorithms and control policies that automate surgical tasks to work towards this future.

Learning to Automate Surgery

Reinforcement Learning (RL) is a machine learning framework for AI systems to solve complex problems. In recent years, success in solving challenging games and robotic manipulation tasks has increased, partly due to collaborative efforts on open-sourced environment simulators like OpenAI's Gym. We present the first open-sourced reinforcement learning environment for surgical robotics, called dVRL, which is functionally equivalent to Gym. dVRL enables prototyping and implementing state-of-art RL algorithms on surgical robotics problems, aiming to introduce autonomous robotic precision and accuracy during surgery. Combining dVRL with the da Vinci Surgical Research Kits network, we enable the surgical robotics community to leverage the newest RL strategies and RL scientists to develop algorithms for autonomous surgery challenges.

Robust Surgical Automation

Robustness is a critical factor in surgical automation since the nature of surgical procedures demands unwavering reliability and resilience. Robust surgical automation should be capable of seamlessly adapting to dynamic surgical scenarios, handling variability in patient anatomy, and quickly recovering from disruptions, minimizing the potential for adverse impacts on the patient's well-being. Achieving robustness in surgical automation is crucial for instilling confidence in both surgeons and patients, as it guarantees consistent and reliable performance, even in challenging or unpredictable circumstances.

Recent efforts in our group such as SURESTEP (Surgical Uncertainty-aware Robust ESTimation TrajEctory Protocol) present frameworks for uncertainty-aware trajectory optimization that enables robust automation by enhancing visual tool tracking accuracy. It optimizes trajectories from any policies to be robust to motion and observation uncertainties commonly encountered in surgical settings.



  • Fei Liu
  • Jingpei Liu
  • Nikhil Shinde
  • Zih-Yun Chiu
  • Neelay Joglekar
  • Yun-Jie Ho
  • *Florian Richter



  • *Ryan Orosco
  • *Emily Funk



AutoPeel: Adhesion-aware Safe Peeling Trajectory Optimization for Robotic Wound Care


arXiv preprint arXiv:2409.14282 (2024)
Xiao Liang, Youcheng Zhang, Fei Liu, Florian Richter, Michael Yip

Autonomous Image-to-Grasp Robotic Suturing Using Reliability-Driven Suture Thread Reconstruction


arXiv preprint arXiv:2408.16938 (2024)
Neelay Joglekar, Fei Liu, Florian Richter, Michael C Yip

MEDiC: Autonomous Surgical Robotic Assistance to Maximizing Exposure for Dissection and Cautery


arXiv preprint arXiv:2409.14287 (2024)
Xiao Liang, Chung-Pang Wang, Nikhil Uday Shinde, Fei Liu, Florian Richter, Michael Yip

JIGGLE: An Active Sensing Framework for Boundary Parameters Estimation in Deformable Surgical Environments


Robotics: Science and Systems (2024)
Nikhil Uday Shinde, Xiao Liang, Fei Liu, Yutong Zhang, Florian Richter, Sylvia Herbert, Michael C Yip

SURESTEP: An Uncertainty-Aware Trajectory Optimization Framework to Enhance Visual Tool Tracking for Robust Surgical Automation


IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2024)
Nikhil U Shinde, Zih-Yun Chiu, Florian Richter, Jason Lim, Yuheng Zhi, Sylvia Herbert, Michael C Yip

SuPerPM: A Large Deformation-Robust Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Data


IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2024)
Shan Lin, Albert J Miao, Ali Alabiad, Fei Liu, Kaiyuan Wang, Jingpei Lu, Florian Richter, Michael C Yip

Bridging Real-to-Sim Gaps through Online Stiffness Optimization with Perception-Enabled Residual Mapping


Workshop on Integrated Perception, Planning, and Control for Physically and Contextually-Aware Robot Autonomy, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2024)
Xiao Liang, Fei Liu, Yutong Zhang, Michael Yip

HemoSet: The First Blood Segmentation Dataset for Automation of Hemostasis Management


IEEE International Symposium on Medical Robotics (ISMR) (2024)
Albert J Miao, Shan Lin, Jingpei Lu, Florian Richter, Benjamin Ostrander, Emily K Funk, Ryan K Orosco, Michael C Yip
(BEST PAPER AWARD)

The current state of autonomous suturing: a systematic review


Surgical Endoscopy (2024)
Benjamin T Ostrander, Daniel Massillon, Leo Meller, Zih-Yun Chiu, Michael Yip, Ryan K Orosco

Real-to-Sim Deformable Object Manipulation: Optimizing Physics Models with Residual Mappings for Robotic Surgery


IEEE International Conference on Robotics and Automation (ICRA) (2024)
Xiao Liang, Fei Liu, Yutong Zhang, Yuelei Li, Shan Lin, Michael Yip

Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives


IEEE International Conference on Robotics and Automation (ICRA) (2024)
Christopher D'Ambrosia, Florian Richter, Zih-Yun Chiu, Nikhil Shinde, Fei Liu, Henrik I Christensen, Michael C Yip

Suture thread spline reconstruction from endoscopic images for robotic surgery with reliability-driven keypoint detection


IEEE International Conference on Robotics and Automation (ICRA) (2023)
Neelay Joglekar, Fei Liu, Ryan Orosco, Michael Yip

Real-time constrained 6d object-pose tracking of an in-hand suture needle for minimally invasive robotic surgery


IEEE International Conference on Robotics and Automation (ICRA) (2023)
Zih-Yun Chiu, Florian Richter, Michael C Yip
(BEST PAPER AWARD)

Artificial intelligence meets medical robotics


Science (2023)
Michael Yip, Septimiu Salcudean, Ken Goldberg, Kaspar Althoefer, Arianna Menciassi, Justin D Opfermann, Axel Krieger, Krithika Swaminathan, Conor J Walsh, He Huang, I-Chieh Lee

Porcine Model for Open Robotic Thyroidectomy


Research Square (2022)
Emily Funk, Won Seo Park, Florian Richter, Benjamin T Ostrander, Michael Yip, Philip A Weissbrod, Ryan K Orosco

Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking


IEEE International Conference on Robotics and Automation (ICRA) (2022)
Shan Lin, Albert J Miao, Jingpei Lu, Shunkai Yu, Zih-Yun Chiu, Florian Richter, Michael C Yip

Markerless suture needle 6d pose tracking with robust uncertainty estimation for autonomous minimally invasive robotic surgery


IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2022)
Zih-Yun Chiu, Albert Z Liao, Florian Richter, Bjorn Johnson, Michael C Yip

Real-to-sim registration of deformable soft tissue with position-based dynamics for surgical robot autonomy


IEEE International Conference on Robotics and Automation (ICRA) (2021)
Fei Liu, Zihan Li, Yunhai Han, Jingpei Lu, Florian Richter, Michael C Yip

Super deep: A surgical perception framework for robotic tissue manipulation using deep learning for feature extraction


IEEE International Conference on Robotics and Automation (ICRA) (2021)
Jingpei Lu, Ambareesh Jayakumari, Florian Richter, Yang Li, Michael C Yip

Model-predictive control of blood suction for surgical hemostasis using differentiable fluid simulations


IEEE International Conference on Robotics and Automation (ICRA) (2021)
Jingbin Huang, Fei Liu, Florian Richter, Michael C Yip
(BEST PAPER AWARD NOMINEE)

Optimal multi-manipulator arm placement for maximal dexterity during robotics surgery


IEEE International Conference on Robotics and Automation (ICRA) (2021)
Mingwei Xu, James Di, Nikhil Das, Michael C Yip

From bench to bedside: The first live robotic surgery on the dVRK to enable remote telesurgery with motion scaling


IEEE International Symposium on Medical Robotics (ISMR) (2021)
Florian Richter, Emily K Funk, Won Seo Park, Ryan K Orosco, Michael C Yip

Autonomous robotic suction to clear the surgical field for hemostasis using image-based blood flow detection


IEEE Robotics and Automation Letters (2021)
Florian Richter, Shihao Shen, Fei Liu, Jingbin Huang, Emily K Funk, Ryan K Orosco, Michael C Yip

Bimanual regrasping for suture needles using reinforcement learning for rapid motion planning


IEEE International Conference on Robotics and Automation (ICRA) (2021)
Zih-Yun Chiu, Florian Richter, Emily K Funk, Ryan K Orosco, Michael C Yip

SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics


IEEE Robotics and Automation Letters (2020)
Yang Li, Florian Richter, Jingpei Lu, Emily Funk, Ryan Orosco, Jianke Zhu, Michael C Yip

Vibration-based multi-axis force sensing: Design, characterization, and modeling


IEEE Robotics and Automation Letters (2020)
Winnie Kuang, Michael Yip, Jun Zhang

Open-sourced reinforcement learning environments for surgical robotics


arXiv preprint arXiv:1903.02090 (2019)
Florian Richter, Ryan K Orosco, Michael C Yip

A 2D surgical simulation framework for tool-tissue interaction


arXiv preprint arXiv:2010.13936 (2020)
Yunhai Han, Fei Liu, Michael C Yip

Robot Autonomy for Surgery


The Encyclopedia of Medical Robotics (2017)
Michael Yip, Nikhil Das

Autonomous control of continuum robot manipulators for complex cardiac ablation tasks


Journal of Medical Robotics Research (2017)
Michael C Yip, Jake A Sganga, David B Camarillo

Model-less control of a flexible robotic catheter


Workshop on Advances in Flexible Robots for Surgical Interventions, Proc. IEEE International Conference on Robotics and Automation (ICRA) (2014)
Michael C Yip, Paul J. Wang, David B. Camarillo (BEST PAPER AWARD)

Intra-Operative Laryngoscopic Instrument for Characterizing Vocal Fold Viscoelasticity


Frontiers in Biomedical Devices (2007)
Mark P Ottensmeyer, Michael Yip, Conor J Walsh, James B Kobler, James T Heaton, Steven M Zeitels

Robotic force stabilization for beating heart intracardiac surgery


Medical Image Computing and Computer-Assisted Intervention (MICCAI) (2009)
Shelten G Yuen, Michael C Yip, Nikolay V Vasilyev, Douglas P Perrin, Pedro J Del Nido, Robert D Howe

A robust uniaxial force sensor for minimally invasive surgery


IEEE Transactions on Biomedical Engineering (2010)
Michael C Yip, Shelten G Yuen, Robert D Howe

Indirect low-intensity ultrasonic stimulation for tissue engineering


Journal of Tissue Engineering (2010)
Hyoungshin Park, Michael C Yip, Beata Chertok, Joseph Kost, James B Kobler, Robert Langer, Steven M Zeitels

SURESTEP: An Uncertainty-Aware Trajectory Optimization Framework to Enhance Visual Tool Tracking for Robust Surgical Automation

N.U. Shinde*, Z.Y. Chiu*, F. Richter, J. Lim, Y. Zhi, S. Herbert, M.C. Yip

arXiv preprint arXiv:2404.00123 [arxiv]

Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning

Z.Y. Chiu, F. Richter, E.K. Funk, R.K. Orosco, M.C. Yip

IEEE Conference on Robotics and Automation (Accepted). Xi'an, China (2021). [arxiv][video]

Real-to-Sim Registration of Deformable Soft-Tissue with Position-Based Dynamics for Surgical Robot Autonomy

F. Liu, Z. Li, Y. Han, J. Lu, F. Richter, M.C. Yip

IEEE Conference on Robotics and Automation (Accepted). Xi'an, China (2021). [arxiv][video]

Model-Predictive Control of Blood Suction for Surgical Hemostasis using Differentiable Fluid Simulations

J. Huang*, F. Liu*, F. Richter, M.C. Yip

IEEE Conference on Robotics and Automation (Accepted). Xi'an, China (2021).  [arxiv][video]

SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction

J. Lu, A. Jayakumari, F. Richter, Y. Li, M.C. Yip

IEEE Conference on Robotics and Automation (Accepted). Xi'an, China (2021). [website][arxiv][video]

Optimal Multi-Manipulator Arm Placement for Maximal Dexterity during Robotics Surgery

J. Di, M. Xu, N. Das, M.C. Yip

IEEE Conference on Robotics and Automation (Accepted). Xi'an, China (2021). [arxiv][video]

Autonomous Robotic Suction to Clear the Surgical Field for Hemostasis using Image-based Blood Flow Detection

F. Richter, S. Shen, F. Liu, J. Huang, E.K. Funk, R.K. Orosco, M.C. Yip

IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1383-1390.  [arxiv][video]

SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics

Y Li, F Richter, J Lu, EK Funk, RK Orosco, J Zhu, MC Yip

arXiv preprint arXiv:1909.05405, 2019. [pdf][website]

Open-Sourced Reinforcement Learning Environments for Surgical Robotics 

F. Richter, R. K. Orosco, M.C. Yip

arXiv preprint arXiv:1903.02090, 2019. [arxiv][vid][git]