Visualisation and localisation of a kidney tumour, and a generated descriptive guide for teaching tumour surgery

Visualisation and localisation of a kidney tumour and a generated descriptive guide for teaching tumour surgery

The technology used in kidney surgery requires the surgeon to correlate pre-operative CT scans with images from a robotic camera. However, the tumour is often not visible on the surface of the kidney, and doctors must visualise its exact location, depth and relationship to blood vessels from 2D cross-sections. Furthermore, it is difficult for junior doctors to learn the procedure: whilst surgical recordings exist, they lack explanations of what is happening at any given moment.

Our solution combines pre-operative anatomy and intra-operative events into a single tool. The first part uses the patient’s CT scan to reconstruct a 3D model of the kidney, including the tumour and surrounding tissue. We use four reconstruction models, combining their outputs. Each model has its own limitations, and combining them provides a high-quality representation of the CT scans in the form of a 3D model, thereby increasing the chances of correctly detecting the tumour. The resulting model shows the tumour’s position, size and depth, as well as other potentially relevant parameters. It can be used for planning and teaching, and can also be overlaid onto the camera feed to help specialists better visualise where the tumour is located beneath the surface.

The second part automatically generates a description of the individual steps of the operation from the video. The vision-language model (VLM) itself often ‘hallucinates’ when analysing surgical footage, as it is unfamiliar with the specific instruments and anatomy. We therefore provide it with structured context. The detection pipeline recognises instruments and anatomical structures in the images, and we use these detections to form triplets in the format ‘instrument – action – target’, for example, ‘scissors – cutting – parenchyma’. The VLM uses the images and triplets to generate a comprehensible description of what is happening and determines the stage of the procedure. The triplets serve as an anchor: the description is based on verified detections, is more accurate, and the source of each sentence can be traced back to its origin.

The main application is for teaching. The trainee reviews the recording of the procedure with automatic commentary and a 3D model, and the trainer does not need to annotate the videos manually. Each operation thus generates educational material. This mode operates on recordings, does not interfere with the course of the operation, and is therefore realistically deployable even from a regulatory perspective. Processing can run locally within the hospital, so patient data does not leave the hospital’s network. However, expert judgement will still be required for a higher-quality description. The inputs are standard (DICOM and video from the robotic system). The solution can be gradually extended to intraoperative support, where the surgeon would see the tumour’s position and the current stage of the procedure in real time.

We see innovation in the integration of a 3D CT model with, potentially, real-time imagery from the surgical instrument camera. Furthermore, we see potential for a generative model, which, as a result, provides more reliable and interpretable output.