AI for Transcontinental Robotic Telesurgery

Robotic teleoperation brings great potential for advances within the field of surgery. The ability of a surgeon to reach patient remotely opens exciting opportunities. Early experience with telerobotic surgery has been interesting, but the clinical feasibility remains out of reach, largely due to the deleterious effects of communication delays. Teleoperation tasks are significantly impacted by unavoidable signal latency, which directly results in slower operations, less precision in movements, and increased human errors. Introducing significant changes to the surgical workflow, for example by introducing semi-automation or self-correction, present too significant a technological and ethical burden for commercial surgical robotic systems to adopt. We develop innovative solutions that reduce the risk of long distance operations while still giving the surgeon full control of the operation.

One solution is to give the operator immediate visual feedback of the surgeons motions by overlaying an Augmented Reality (AR) rendering a prediction of the operators move on the delayed video feed. This is also known as a predictive display. To realize this solution active tracking of the surgical robotic tools is required because relying on kinematic calibrations to localize the surgical robotic tools fails due to the millimeter scale of a surgical operation and the often utilized cable driven actuation. Also an adaptive transparency filter is utilized to prevents confusion arising from overlapping virtual and real visuals of the robot arms. The results from a user study with both experts and novices showed with statistical significance that using the predictive display decreased time to complete task while having no effect on error rates when operating under delay.

Another solution developed simply changes the scaling relationship between the surgeons motions in the master console and the patient side manipulators which hold the surgical robotic tools. We call this the motion scaling solution. Motion scaling has been shown to be an effective method of improving accuracy for surgeons when operating under normal conditions. We extend this idea to the remote telerobotic surgery and show that the motion scaling solutions reduces errors while having a minimal effect on time to complete tasks.


Students & Collaborators

  • Florian Richter
  • Zihyun Chiu
  • Fei Liu
  • Ryan Orosco
  • Emily Funk

Publications

Artificial intelligence meets medical robotics


Science (2023)
Michael Yip, Septimiu Salcudean, Ken Goldberg, Kaspar Althoefer, Arianna Menciassi, Justin D Opfermann, Axel Krieger, Krithika Swaminathan, Conor J Walsh, He Huang, I-Chieh Lee

Mixed reality surgical mentoring of combat casualty care related procedures in a perfused cadaver model: Initial results of a randomized feasibility study


Surgery (2022)
Matthew D Tadlock, Erik J Olson, Danilo Gasques, Roland Champagne, Michael J Krzyzaniak, Shawn A Belverud, Vijay Ravindra, Jakob Kerns, Pamela M Choi, Jennifer Deveraux, Janet Johnson, Thomas Sharkey, Michael Yip, Nadir Weibel, Konrad Davis

Remote telesurgery in humans: a systematic review


Surgical Endoscopy (2022)
Patrick Barba, Joshua Stramiello, Emily K Funk, Florian Richter, Michael C Yip, Ryan K Orosco

From bench to bedside: The first live robotic surgery on the dVRK to enable remote telesurgery with motion scaling


IEEE International Symposium on Medical Robotics (ISMR) (2021)
Florian Richter, Emily K Funk, Won Seo Park, Ryan K Orosco, Michael C Yip

Compensatory motion scaling for time-delayed robotic surgery


Surgical Endoscopy (2021)
Ryan K Orosco, Benjamin Lurie, Tokio Matsuzaki, Emily K Funk, Vasu Divi, F Christopher Holsinger, Steven Hong, Florian Richter, Nikhil Das, Michael Yip

Artemis: A collaborative mixed-reality system for immersive surgical telementoring


Conference on Human Factors in Computing Systems (CHI) (2021)
Danilo Gasques, Janet G Johnson, Tommy Sharkey, Yuanyuan Feng, Ru Wang, Zhuoqun Robin Xu, Enrique Zavala, Yifei Zhang, Wanze Xie, Xinming Zhang, Konrad Davis, Michael Yip, Nadir Weibel

Artemis: Mixed-reality environment for immersive surgical telementoring


Conference on Human Factors in Computing Systems (CHI) (2020)
Nadir Weibel, Danilo Gasques, Janet Johnson, Thomas Sharkey, Zhuoqun Robin Xu, Xinming Zhang, Enrique Zavala, Michael Yip, Konrad Davis

Motion scaling solutions for improved performance in high delay surgical teleoperation


IEEE International Conference on Robotics and Automation (ICRA) (2019)
Florian Richter, Ryan K Orosco, Michael C Yip

Augmented reality predictive displays to help mitigate the effects of delayed telesurgery


IEEE International Conference on Robotics and Automation (ICRA) (2019)
Florian Richter, Yifei Zhang, Yuheng Zhi, Ryan K Orosco, Michael C Yip

Performance analysis of a manipulation task in time-delayed teleoperation


IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2010)
Michael C Yip, Mahdi Tavakoli, Robert D Howe