Currently 4.71/5
1
1
2
2
3
3
4
4
5
5

Rating: 4.7/5 (112 votes cast)

Send Email

Title

Integration of 3D Virtual Reality (VR) in Diagnostic and Therapeutic Imaging

Theme

Diagnostic & Interventional Radiology

Authors

Saadia Talay
Huda Mubarak
Amjad Aldarwish
Fatimah Alhamoud
Noor Aljabr
Kamran Hameed
Mahbubunnabi Tamal

Institutions

Imam Abdulrahman bin Faisal University

Background

Virtual reality (VR) is an artificial, three-dimensional computer-generated environment. It places the user inside a virtual environment where one may interact with the surroundings.
In this era of endoscopic surgeries where it is vital to delineate the extent of an abnormality and pinpoint its exact localization, integration of VR in medical imaging would provide an interactive method to view 3-dimensional medical scans.
Medical specialists may view the 3D image though the headset in a virtual environment where one may interact with it and manipulate it to view finer details.
With this technology, the specialist will be able to gain an intuitive feeling of the scanned organ, its dimensions, the arrangement of various tissue components within the organ, and natural variations between patients.
This would significantly reduce the stress on the surgeon, curtail the duration of invasive procedures, and permit the surgeon to decide for a safer approach.

Aim

To design an integrated platform that reads DICOM images from the scanner as an input and generates an interactive 3D virtual model of the scanned organ.

Summary of Work

Scanner

Medical image data sets from imaging modalities such as CT and MRI are loaded on to 3D Slicer as DICOM (Digital Imaging and Communications in Medicine) files.

3D Slicer

Using the image registration toolkit (ITK) in 3D Slicer, threshold based segmentation is performed. The threshold is chosen based on the range which best covers the bone tissue. This was 155.90 - 1150.77 for the CTA scan chosen in the trial. After each slice in the dataset has been segmented, the 3-dimensional volume model is rendered. This model is then exported in an .stl format as a surface rendered model.

Blender

As Unity does not provide direct support for stl data format, the surface render is converted from .stl to .fbx using Blender, which has support for both formats. Next, the .fbx file is imported to Unity where it is interfaced with the Vive.

Unity

A steamVR plugin is downloaded from the Unity asset store. This allows for placement of camera, and controller rigs that allows the scene to be viewed and interacted using the Vive. Each object is placed such that there is an appropriate view.

Head-mounted Display

The scene is played and the view transfers to the VR head-mounted display (HMD). The user can view the rendered model in virtual reality and interact with it using the controllers.

more detail

Summary of Results

We were able to obtain a surface render model of the CTA 3D image dataset. The surface render can be virtually viewed by the HTC Vive HMD. At present, the user is only able to view the surface rendered model.


3D Slicer (Segmentation)	Surface Render product from 3D Slicer

Blender to convert image from .stl to .fbx	Unity to interface with the Head-mounted VR headset

Conclusion

Technology is advancing at a rapid rate. With VR integrated into medical imaging, efficient and accurate diagnosis could be made leading to better healthcare. Finer details may be viewed and an intuitive feeling of the scanned organ will be gained by the specialist which will make it easier to plan patient prognosis.

FUTURE WORK

The project is still ongoing and we are working to achieve the following:

Ability to interact with the model
Obtaining a volume render instead of surface render to obtain information from each voxel of the model
Focusing on the coronary artery for quantitative and qualitative assessment of plaque formation

Take-home Messages

Improving the efficacy of medical imaging through Virtual Reality

Detailed 3D realistic and interactive visualization will allow accurate diagnosis
It will assist in the planning of complicated surgical procedures
It has the potential to enhance medical education

References

[1] F. King, "An immersive virtual reality environment for diagnostic imaging," Dissertation/Thesis, ProQuest Dissertations Publishing, 2015.

[2] J. Egger, M. Gall, J. Wallner, P. Boechat, A. Hann, X. Li, X. J. Chen, and D. Schmalstieg, "HTC Vive MeVisLab integration via OpenVR for medical applications," PLOS ONE, vol. 12, p. e0173972, 2017.

[3] Virtual Reality in Medicine: New Opportunities for Diagnostics and Surgical Planning. Available: https://www.unibas.ch/en/News-Events/News/Uni-Research/Virtual-Reality-in-Medicine.html

Summary of Work

3D Slicer

3D slicer is a software used for analysing, processing, and scientific three-dimensional visualization of medical images. It supports common data formats and features such as image segmentation and registration toolkits, and rendering methods. Renders can be exported in a vtk or stl format, with vtk being the volume render and stl being the surface render.

Blender

Blender is a 3D creation software that has multiple uses. It can be used to create 3D games and simulation. Commonly used with Unity, it offers import and export of multiple data formats including obj, fbx, 3ds, ply, and stl.

Unity

Unity is a game engine used to develop 2D and 3D video games for consoles, mobile devices and PCs. It features rendering of 2D and 3D graphics, as well as interaction with those objects using physics engines. It supports the import of fbx and obj formats. Unity offers VR support with the steamvr plugin.

HTC Vive

HTC Vive is a virtual reality system developed in partnership by HTC and Valve corporations. It is an excellent candidate for this project due to its ability to imitate real world actions inside the virtual world with high levels of accuracy and low system latency. The basic system consists of a virtual reality headset for viewing, two hand-held motion controllers for interaction, and two base station lighthouses for tracking purposes.