Rendering of Cystoscopy Video Footage: A Novel Method Utilizing Neural Radiance Field Processing

The Journal of Urology, 2024

Jonathan E Katz, Jamie Finnegan, Jingpei Lu, Shan Lin, Michael Yip, Roger Sur

Abstract:

Surgical Technology & Simulation: Artificial Intelligence II (PD27): Podium 27: Saturday, May 4, 2024: 9: 30 AM-11: 30 AM
PD27-06 3D RENDERING OF CYSTOSCOPY VIDEO FOOTAGE: A NOVEL METHOD UTILIZING NEURAL RADIANCE FIELD PROCESSING

Katz, Jonathan E.; Finnegan, Jamie; Lu, Jingpei; Lin, Shan; Yip, Michael; Sur, Roger
The Journal of Urology 211(5S):p e552, May 2024. | DOI: 10.1097/01.JU.0001008580.58088.27.06

INTRODUCTION AND OBJECTIVE:

Recent developments in Neural Radiance Field Processing (NERF) have leveraged the power of neural networks to quickly reconstruct 3D spaces from 2D images. Our objective was to utilize this technology to 3D render video recordings of diagnostic cystoscopies and test their fidelity.
METHODS:

With IRB approval we recorded two diagnostic cystoscopies (one with an Ambu disposable flexible cystoscope and the other with a Wolf digital cystoscope). We converted the videos to images and then curated the images to choose approximately 100 images, which minimized blur and spanned a large segment of the bladder. We then utilized the NVIDIA Instant Neural Graphics Primitives (iNGP), a NeRF algorithm that uses multiresolution hash encoding with a compact neural network for significantly faster convergence, to reconstruct the bladder and render novel, unseen views. We computed the structural index similarity (SSIM) and Peak signal to noise ratio (PSNR) to assess the quality and fidelity of the 3D rendering.
RESULTS:

Both videos were able to be utilized for 3D rendering using iNGP. The rendering derived from the Wolf cystoscopy had a PSNR=29.8 [min=27.2, max=32.6] and SSIM=0.89. Similarly, the rendering derived from the Ambu cystoscopy had a PSNR=31.3 [min=27.1, max=35.1] and SSIM=0.90, static sample images below (Image 1).
CONCLUSIONS:

Independent of cystoscopy equipment, both 3D renderings achieved reasonable fidelity. Major limitations to widespread adoption of this technology include the need for a curator to select representative and high-quality images from the initial cystoscopy video recording and the relatively small segments of bladder successfully rendered. Nonetheless, we feel that with further refinement this technology can be scaled to create 3D renderings of cystoscopies that will enable evaluation of both completeness and quality of the cystoscopy. Furthermore, this technology would be able to facilitate the comparison of cystoscopies performed in the same patient over time.

Katz et al. (2024) Rendering of Cystoscopy Video Footage: A Novel Method Utilizing Neural Radiance Field Processing, The Journal of Urology, vol. 211, no. 5S, pp. e552.

Pub Link: http://journals.lww.com/auajuro/fulltext/2024/05001/pd27_06_3d_rendering_of_cystoscopy_video_footage_.1140.aspx
arXiv: