Beating hearts pump up virtual reality for pre-surgery planning
The most immediate impact isn’t going to be the 4D heart. The impact at a grander scale will be the scalability of how any program will be able to create models for pre-surgical planning; 3D printed or for VR with this technology.
In a rapidly evolving technological landscape, machine learning has emerged as a transformative force; it is reshaping countless industries, health care included. Machine learning and artificial intelligenceincreases productivity, reduces human errors and frees up time for other tasks that can have greater impact. In the case of converting medical scans into 3D or 4D images for a deeper view of organs, tissues and tumors, AI has the possibility of changing how surgeons prepare to operate on patients with complex issues.
Matthew Bramlet, MD, is a pediatric cardiologist at the University of Illinois College of Medicine at Peoria who specializes in congenital cardiac MRI for the Children’s Hospital of Illinois.His Advanced Imaging and Modeling (AIM) lab atJump Trading Simulation & Education Centeris focused on translation of medical images into 3D and 4D interactive models (for 3D printing for virtual reality) to assist in pre-surgical planning of complex cardiac and cancer cases.
According to Dr. Bramlet, the heart is a unique organ because it changes shape as it expands and contracts to pump blood through the body. This change in shape becomes a problem when the muscle of the heart grows abnormally and obstructs its own ability to pump blood out of the heart. This is a problem, because no current technology exists that allows a surgeon to view this 4D, dynamic obstruction.
Rather than relying on years of trial and error of surgical expertise to determine the correct way to cut out muscle to relieve the obstruction, this project seeks to provide the surgeon with a VR view of their next patient’s beating heart in 4D.
The Innovation for Health (IFH) grant program with Bradley University in Peoria, Illinois is designed to inject funding ($50,000) to combine clinical problems with computer science expertise.Assistant Professor Sam Hawkins, PhD, and graduate students at Bradley University planto utilize machine learning to convert standard medical images into a sequence of 3D models, that when played sequentially create a 3D beating heart; the “4D heart”.
This concepthas been previously demonstratedby Dr. Bramlet’s AIM lab, but it took an entire summer to convert images of a 4D heart, showing both the expansion and contraction of the heart that could be viewed through a virtual reality headset. Dr. Bramlet says that’s not a sustainable approach.
“How do we put the CT data into a computer and slice by slice, say this is the myocardial tissue so we get an exact replica of the heart. But I don't need just one (3D replica of a heart); I need 20 to make each phase of that heart (he simulates the sound of repetitive beats) into a 4D heart.”
Assistant professor Hawkins believes machine learning can be leveraged to perform the labor-intensive process of converting images. Hawkins explains Computer Science graduate students at Bradley are part of the research effort.
“First, we need to figure out what part of the image is the heart and what part is not. And then we need to combine these tiny images into a 3D image. And then we need to do that many times to get the 4D.”
Improving pre-surgery planning
The solution should allow surgeons to virtually preview the beating heart of their patient, showing the obstruction in 4D, therefore providing a completely new pre-surgical analysis tool. The multidimensional view allows surgeons to zero in on structures of the heart, significantly enlarge elements and get a much better view of the anatomy. This machine learned 4D generation of a beating heart will be a first of its kind technology.
Researchers believe creating an automated solution can reduce the manual process of converting images from months, to hours, and eventually perhaps minutes. That option could eventually be applied to images for other types of complex medical cases in adults, children and the smallest infants.
Dr. Bramlet says new AI-supported software could provide a scaled solution for surgeons everywhere.
“The most immediate impact isn’t going to be the 4D heart. The impact at a grander scale will be the scalability of how any program will be able to create models for pre-surgical planning; 3D printed or for VR with this technology.”
Hawkins believes the technology will lead to other hospitals adopting the approach for pre-surgery planning.
“It has the potential to really remove the barrier of entry for institutions that don’t have the expertise or the time but do have images they wanted to view (more intricately).”
This entire machine learning project has been made possible by Jump Simulation’s long-term investment in sharing 3D insights. Since, 2014, Jump has contributed 3D models of congenital heart disease to Dr. Fauci’s 3D database at theNIH. This library of annotated 3D models provides the rich dataset of raw material needed for machine learned insights into automated segmentation of patient specific 3D models.
Within a year, this project aims to automate, through machine learned algorithms, the conversion of cardiac CTs into virtual reality. The 4D view will enable surgeons to see the beating hearts specific to individual patients.
Imagine being a surgeon who can plan a surgery in VR by seeing the beating heart of your patient before you walk into the OR. This project seeks to transition this vision from science fiction to science fact.