Performing surgery is the most complex job on earth; therefore, having fellow surgeons assist you with your surgery makes sense. However, what if the fellow surgeon is not in town or is present in another country? Well, metaverse surgery has made it possible to gather all the surgeons together in the operation theater, no matter where they might be located, and perform the surgery together.
Let’s Explore how Metaverse Surgery Setup Can be Done in a Hospital Setting?
To start with, you shall need the following: –
- Microsoft HoloLens, preferably, the latest version for all the surgeons participating in the surgery.
- DICOM Player connected with Electronic Medical Record
- Spectator View (Mixed Reality Visualization Tool)
- Main Server
This setup enables the surgeon to view the patient, the patient’s digital 3D images, and thereby discuss the surgery in a meaningful way.
1 Microsoft HoloLens
HoloLens is a device by Microsoft that can create a mixed-reality environment. One of the used cases of HoloLens is in healthcare, especially during the pre-operative, operative, and post-operative stages of surgery.
During the various stages of surgery, HoloLens helps in bridging the real and the virtual worlds to form a new environment (metaverse surgery) where the users can interact with fellow surgeons in the presence of digital anatomical and diagnostic images.
The surgeons now do not need to move around the operation theatre to see the diagnostic images, patient vitals, etc. HoloLens enables surgeons to pull up patients’ medical data in the metaverse space, with simple hand gestures.
Specifications of HoloLens
a) Electronic Unity
Electronic unity helps in creating a virtual reality metaverse for the surgeons using the HoloLens device. It combines the concept of virtual reality and mixed reality and helps create a spatial metaverse during the surgery.
b) Inertial Measurement Unit (Integration of Gyroscope, Accelerometer, and Magnetometer)
It helps in mapping the room in which the master HoloLens is present. Further for fellow surgeons, this feature helps map their head and body movement in three-dimensional space. This makes it possible for all the participating surgeons to view each other in the metaverse surgery space.
c) Camera
HoloLens consists of a wide-angle camera with high resolution to capture and showcase a larger portion of the OT room to fellow surgeons connected in the same space. It also helps in capturing hand gestures and thereby prepare the command for the HoloLens to execute various actions. Below is an illustration of various gesture movements you can do with your HoloLens.
d) Infrared Illuminator
Infrared Illuminator helps in determining the depth of nearby and far-placed objects and portrays them accordingly in the metaverse. Near sensing capability helps in determining the hand gestures and makes it possible to sync them with the functionalities of HoloLens. Far sensing capability of the HoloLens helps in making the far-kept objects, appear nearby.
e) Microphones and Speakers
Microphone in HoloLens helps record the sounds and transmit them to fellow surgeons with the help of speakers, in their respective HoloLens.
f) Light Sensors
Light sensors help capture the surroundings and display it in the metaverse (by exactly displaying the colors and shapes it visualizes through master HoloLens).
g) See Through Lenses
See Through Lens is the most magical part of HoloLens. It helps display the stored information (like patient’s clinical data, diagnostic images saved in DICOM Server) in front of the eye. This capability of HoloLens is also quite helpful for the surgeons, sitting remotely, who want to annotate on the patient’s body or the diagnostic images so that it is visible to the performing surgeon (in front of his eyes).
Find out more about annotation with HoloLens
h) Microsoft App Store (Mesh and Remote Assist Application)
There are two important applications (present in HoloLens device), that we will cover in this article that help in creating Metaverse Surgery a success.
MESH APPLICATION
HoloLens consists of a Microsoft application store (just like google play in android or app store in iPhone) from where, various applications can be installed. One of the applications available in the HoloLens Microsoft app store is ‘Mesh’, which we shall be discussing today, to set up metaverse surgery in the operation theatre.
Mesh application is built with the intent to create a metaverse space where organizations can collaborate and work together. The virtual ecosystem created with the help of the Microsoft Mesh application is so natural and productive, it feels as if your colleague is present just next to you, even though they might be present in a different country.
Prerequisites for Installing Mesh Application
Prior to installing Mesh Application, make sure you have:-
- HoloLens 2 or its later version
- Windows 10 or higher installed in HoloLens
- Microsoft Azure Account with the organization’s email id
Once the prerequisites are met, the next step is to install the Microsoft Mesh application.
How to Install Microsoft Mesh Application and Use It?
I) Wear HoloLens
To install the Mesh application, wear your HoloLens.
2) Open Start Menu
Thereafter, open the start menu. To do so, use the start gesture – bloom, in case you are using HoloLens Generation 1st however if you are using HoloLens2, the start menu can be opened either by voice command, ‘Go to Start’, or by twisting tapping on your wrist.
3) Open Microsoft App Store
Once the start menu opens, select the Microsoft App store and search for the Mesh Application.
4) Create Yourself in Mesh
Once the Mesh application is installed in the HoloLens, the next step is to log in using a Microsoft Azure account. Thereafter, either upload your own photo or create an avatar of yours. This shall be visible to your colleagues while you are connected in space.
5) Creating, Joining, and Removing from Space
Once your avatar is ready, creating and joining a space is next. Spaces are mini metaverse worlds where you can invite your colleagues to join and collaborate.
Mesh application provides the flexibility to share the invite for the space with fellow workers or add them directly by entering their Microsoft Azure ID and also remove a fellow worker, if not required.
The Co-Worker can also be added by adding his/her email id.
The Co-Workers can be removed from the space by the organizers at any point of time during the meeting.
A Summary of How to Use Mesh Application in HoloLens with a Demo
6) Content Management while in Microsoft Mesh Metaverse Surgery Space
The content for the display can be saved in:-
a) Microsoft OneDrive
b) Azure Blob Storage Connected to Azure Remote Rendering Service
Microsoft OneDrive
The most exciting feature about HoloLens is the ability to import content, for discussion, in the space where you and your colleagues are connected in. The real essence of metaverse surgery is in its ability to have the patient’s diagnostic images available as 3D images (Holograms) in front of the team. These diagnostic images can be viewed during pre-op, during operation, and also during post-op stages to discuss the patient’s case in detail.
To add the diagnostic images from the DICOM server/Electronic Medical Record, the images first have be transferred to the Microsoft OneDrive. The images can be either transferred on need basis or automatically with interoperable engine like InterSystems HealthShare.
Once the images are moved to Microsoft OneDrive, from the HoloLens start menu, choose Local File Picker to import the images saved in One Drive to the metaverse surgery space.
b) Azure Blob Storage Connected to Azure Remote Rendering Server
While importing the 3-D images (Holograms) to the metaverse surgery space, it is quite important to feel swiftness while rendering the images as well as while interacting with the image/hologram. To avoid any lag, Microsoft has come up with Azure Remote Rendering Service. It is a cloud-based service where 3D images stored in Azure Blob storage can be pulled in the metaverse space. These images are displayed with utmost clarity as compared to the image saved in Microsoft OneDrive.
Azure Remote Rendering Service makes use of high-quality GPUs which eliminates the lag while interacting or displaying the image and therefore portrays a 100% accurate detailed image in front of you.
7. Annotation and Managing Size of the Image in the Space
Once the images are imported in the metaverse surgery space by either using the Local File Picker or Azure Remote Rendering service, the next step is to be able to manipulate it for discussion. This can be either done with the help of annotation tool or by changing the size, shape, and position of the image in the metaverse.
With Annotation tool, the surgeons can highlight, mark a certain part of the diagnostic image to make the conversation meaningful.
Further the images can be moved and resized in the metaverse surgery space for enhancing the medical discussion and looking at the affected part closely by zooming in it.
REMOTE ASSIST APPLICATION
Just like you were able to download the Microsoft Mesh Application from the start menu (with Bloom Action in HoloLens 1 and tap on the wrist in HoloLens 2), download Remote Assist application too.
Remote Assist application enables the Surgeon sitting remotely to guide the surgeon performing the surgery vocally and by creating annotations to execute a specific action. For example, a remote surgeon can guide the surgeon in the Operation Theatre with the right place of incision by creating an annotation. The see through lens in the HoloLens enables the surgeon performing the surgery to see the annotation drawn by the remote surgeon.
With this, the setup for the HoloLens device is complete. Now let’s proceed to see, how to setup the DICOM Player
2) DICOM Player
Digital Imaging and Communication in Medicine (DICOM) is a standard for storing and retrieving diagnostic medical image across the healthcare industry. HoloLens makes use of an open-source software, ‘DICOM Viewer’, to retrieve and display diagnostic images to the surgeons wearing the HoloLens device.
The diagnostic images are saved in the DICOM server, which is further connected to the main server of the hospital. This main server could either be a cloud server or an in house data server maintained in hospital premises. While requesting a diagnostic image via Azure Remote Rendering Service or Local File Picker, the DICOM format image gets converted into STL Format (3D Image Format). During the transition of the DICOM format image to STL, a bridge software called as InVesalius is used, to execute the conversion.
The final converted 3D image can then be pulled in Metaverse Surgery by the surgeons using hand gestures, as discussed earlier. Once these 3D images start appearing in the metaverse space, the surgeons can manipulate the image by changing its size, slicing it, annotating it, moving it across the Operation Theatre etc.
DICOM Database should have a bidirectional flow of information with the main hospital server. This means that not only image retrieving, but this setup also enables capturing new images using HoloLens and thereby storing it back in DICOM Database.
3) Spectator View (Mixed Reality Visualization Tool)
Spectator View makes it possible to share the images, videos, space coordinates and all other information created by HoloLens from the metaverse surgery to external devices like smartphone, tablet, laptop etc. In short, a person using external device can view what the main surgeon is seeing using his/her HoloLens.
A unidirectional flow of information should take place from the main hospital server to the external devices.
This can be used for training medical students by showing them a live a surgery as purchasing HoloLens for all the students may not be feasible. Also, in some cases, the access to view the surgery from external device enabled by Spectator view is granted to the relatives of the patient, undergoing the surgery.
4) Main Server
The main server of the hospital can be either be deployed on cloud or can be deployed on an inhouse secured network.
Generally, hospitals want to start with a prototype setup wherein they want to train their surgeons on using HoloLens to create metaverse surgery setup. In such a case, having a cloud service along with public internet shall be a preferred way to start with.
However, once the surgeons become comfortable with the HoloLens a large-scale setup shall be required and in such a case, it makes sense to connect the devices to the organization’s network with corporate Wi-Fi connection.
Metaverse Surgery Adoption
Metaverse now is not the thing of the past. Technology is moving quite fast and the adoption in healthcare is matching the pace. Following is an example of Metaverse Surgery adoption across the globe:-
References
- Evaluation of the Microsoft hololens for the mapping of Indoor Building … (n.d.). Retrieved March 9, 2023, from https://www.researchgate.net/publication/331311412_Evaluation_of_the_Microsoft_HoloLens_for_the_Mapping_of_Indoor_Building_Environments
- Import content. Microsoft Learn. Retrieved March 9, 2023, from https://learn.microsoft.com/en-us/mesh/mesh-app/use-mesh/import-content
- Galati, R., Simone, M., Barile, G., De Luca, R., Cartanese, C., & Grassi, G. (2020, August 7). Experimental setup employed in the operating room based on virtual and mixed reality: Analysis of pros and cons in open abdomen surgery. Journal of Healthcare Engineering. Retrieved March 9, 2023, from https://www.hindawi.com/journals/jhe/2020/8851964/
- Getting around hololens 2. Microsoft Learn. Retrieved March 9, 2023, from https://learn.microsoft.com/en-us/hololens/hololens2-basic-usage
