From Pokemon to saving lives: using augmented reality in the operating room could usher in a new age of surgery.

The surgeon’s vision — elements of the patient’s foot were digitized and then fed into a 3D model. Image credits: Philip Pratt et al. Eur Radiol Exp, 2018 / Microsoft HoloLens (c) Microsoft.

Augmented Reality, the technique popularized last year by Pokemon Go, overlays real-life elements with “augmented” bits — most often, computer-generated information. It’s a world where real life as we know it interacts with holograms.

Now, for the first time, doctors have used augmented reality as an aid for surgery. Specifically, they’ve used Microsoft HoloLens headsets to overlay CT scans, indicating the position of bones and key blood vessels, over each of the patient’s legs. Basically, they were able to ‘see’ through the patient’s skin.

The technology helped with a very delicate procedure: the reconnecting of blood vessels, an essential part of reconstructive surgery.

“We are one of the first groups in the world to use the HoloLens successfully in the operating theatre,” said Dr. Philip Pratt, a Research Fellow in the Department of Surgery & Cancer and lead author of the study, published in European Radiology Experimental.

“Through this initial series of patient cases we have shown that the technology is practical, and that it can provide a benefit to the surgical team. With the HoloLens, you look at the leg and essentially see inside of it. You see the bones, the course of the blood vessels, and can identify exactly where the targets are located.”

So far, the technology has only been used in reconstructive limb surgery, but there’s no reason why it couldn’t be adapted to other types of surgery. Image credits: Philip Pratt et al. Eur Radiol Exp, 2018 / Microsoft HoloLens (c) Microsoft.

Doctors carried out five surgeries using the technology. Prior to the surgery, CT scans mapped the structure of the limb. The elements revealed by the CT scan were then split into bone, muscle, fatty tissue and blood vessels by Dr. Dimitri Amiras, a consultant radiologist at Imperial College Healthcare NHS Trust. Amiras used the data to develop a 3D model of the patients’ legs. The models were fed into the HoloLens, allowing surgeons to see them as they were carrying out the procedure. The surgeons also fine-tuned the model — with a simple hand gesture, they made sure that the model lined up with real life.

The procedure is time-consuming but in the future, algorithms could greatly simplify and reduce the work volume.

“The application of AR technology in the operating theatre has some really exciting possibilities,” said Jon Simmons, a plastic and reconstructive surgeon who led the team. “It could help to simplify and improve the accuracy of some elements of reconstructive procedures.

“While the technology can’t replace the skill and experience of the clinical team, it could potentially help to reduce the time a patient spends under anaesthetic and reduce the margin for error. We hope that it will allow us to provide more tailored surgical solutions for individual patients.”

Right now, the technique has only been used for lower limb reconstructive surgery, but the proof of concept is there. This study shows that the technology is practical, accurate, and safe to use. There’s no reason why a similar approach couldn’t be used in different types of surgery.

Augmented reality does nothing to replace the skill and experience of the operating team, but it does complement and amplify it, significantly reducing the margin for error.

The paper ‘Through the HoloLens looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels’ by Philip Pratt et al. is published in the journal European Radiology Experimental.