Merging virtual and physical experiences: extended realities in cardiovascular medicine

Abstract

Technological advancement and the COVID-19 pandemic have brought virtual learning and working into our daily lives. Extended realities (XR), an umbrella term for all the immersive technologies that merge virtual and physical experiences, will undoubtedly be an indispensable part of future clinical practice. The intuitive and three-dimensional nature of XR has great potential to benefit healthcare providers and empower patients and physicians. In the past decade, the implementation of XR into cardiovascular medicine has flourished such that it is now integrated into medical training, patient education, pre-procedural planning, intra-procedural visualization, and post-procedural care. This review article discussed how XR could provide innovative care and complement traditional practice, as well as addressing its limitations and considering its future perspectives.

Graphical Abstract

Extended reality (XR) is the umbrella term for all immersive technologies that merge virtual and physical experiences. XR has the potential to be integrated into all stages of cardiovascular medicine, including education, telemedicine, procedure planning, intraprocedural visualization, post-procedure care, and for rehabilitation. XR can be applied to all participants in cardiovascular medicine, from patients to trainees to clinicians. However, XR technologies need to overcome the barriers of operation cost, lack of evidence, insufficient patient/physician input in development, and system reliability to achieve its full potential in cardiovascular medicine.

Open in new tabDownload slide

Introduction

Extended realities (XR) is an umbrella term for all immersive technologies that merge virtual and physical experiences. For decades, the expectation has been that XR will take cardiovascular medicine to the next level, but this has never fully materialized; however, we now see the infiltration of virtual experiences into every aspect of our daily lives, from augmented reality (AR) applications in our phones to the inception of the ‘metaverse.’ With the rapid improvement of high-resolution, miniaturized wearable devices, exponential growth in computational power, and extensive use of artificial intelligence (AI) for image processing, XR is poised to revolutionize the landscape of cardiovascular medicine, with the potential to benefit everyone involved from patients to trainees to clinicians. If successful, XR will be integrated into the workflow of all stages of cardiovascular care, including education, pre-procedural planning, intra-procedural visualization, post-procedural follow-up, and rehabilitation (Graphical Abstract).^1,2 In this narrative review, we will discuss the current utilization of XR, the challenges toward its implementation, and its future perspectives in cardiovascular medicine.

Spectrum of XR

The spectrum of XR ranges from fully immersive virtual reality (VR) to AR, which provides an unobstructed view of the surroundings while providing critical information ‘at a glance’ outside the regular visual field³ (Figure 1). In the middle lies merged reality (MeR) and mixed reality (MxR) technologies. MeR utilizes a high-definition camera and onboard computer to project a rendered vision of the surroundings with added three-dimensional (3D) objects for interaction.⁴ MxR uses a similar approach, but instead of projecting a fully rendered vision, it only projects the digital 3D objects into its semi-transparent screen, allowing the user to have situational awareness.⁵ The ideal XR system should provide high-quality images while still allowing in-person interaction, which may be achieved by a seamless transition between different types of XR displays.

Each XR technology has its own advantages and disadvantages for clinical practice. Whilst the immersive experience with VR is indispensable for medical education, patient education, rehabilitation, image analysis, and telemedicine,³ VR devices block the user’s visual field, thus rendering the user stationary (Figure 2). AR goggles, such as Google Glass (Google LLC, Mountain View, CA, USA), can provide critical reference information on the fly at the cost of poor display quality, with its lightweight, unobstructive profile best suited for clinical rounds and emergency medical settings where eye contact and situational awareness are prioritized. AR applications, which are now ubiquitous on smart devices, are highly accessible at a meager cost, with the most famous example probably Pokémon GO (Niantic Inc., San Francisco, CA, USA) (Figure 3). Unfortunately, although AR on smart devices are perfect for education and visualization, they are limited by the lack of true 3D vision and do not support direct interaction with 3D objects. MxR and MeR technologies have the advantage over VR and AR as they allow interpersonal interaction while displaying high-definition images such that they are perfect for group education, pre-procedure planning, and intraprocedural visualization.¹ The technology of XR has benefited not just from visual hardware improvements such as better, lighter head-mount displays (HMDs) but also advances in sensory technology such as gaze tracking, hand gesture, and voice recognition which frees the users’ hands from a stylus or a controller, allowing intuitive manipulation.

Clinical implementation of XR

Education

XR was first used for medical education in pioneering institutions, with the Stanford Virtual Heart project (Stanford University, Stanford, CA) and HoloAnatomy (Case Western Reserve University, Cleveland, OH) developed to teach medical students/surgeons/paramedical staff complex cardiac anatomy and congenital heart disease.^6,7 These anatomical models (Virtual heart, Anima Res, HoloAnatomy…) are now commercially available,⁸ and can be accessed with commercial XR devices without the limitation of time and location and at a much lower cost than cadavers, which was especially useful for remote teaching situations such as during the COVID-19 pandemic.^9,10 Studies have shown that using XR simulation to teach anatomy is comparable to traditional methods and provides additional value.^11,12 VR can assist clinical skill training with virtual models and simulations which provide realistic responses to interactions. The highly realistic images of XR can complement traditional plastic models, which provides tactile feedback to manipulations. This is especially useful for high-risk procedures that may cause catastrophic complications if done incorrectly. In addition, VR training may improve the learning curve in actual practice. James et al. have demonstrated that VR-assisted trans-septal puncture was more accurate and intuitive than trans-septal puncture guided by fluoroscopy, especially for inexperienced operators.¹³ VR-based skills simulators have also been developed for vascular access, angiography, electrophysiology (EP), percutaneous valve implantation procedures, and surgeries.^10,14,15 VR can provide valuable mock training for emergent procedures that are generally unsafe to be performed by inexperienced operators so that trainees can become well-customed before the need to perform in an emergency; endovascular repair of abdominal aortic aneurysms and other similar endovascular procedures are excellent examples.¹⁶ VR-based resuscitation training has also been shown as immersive and effective.¹⁷ With the new MxR/MeR HMDs, these educational applications can be performed in a group session to facilitate teamwork and discussions. Furthermore, with current HMD devices, multiple users can effectively share the operator’s view during the operation in real time using MeR/MxR or retrospectively with VR,¹⁸ all of which may prove very useful for the education of minimally invasive and endoscopic operations.

VR may one day be instrumental in patient education. Cardiovascular diseases are frequently complex and challenging to comprehend for patients without background knowledge. XR can provide patients with an intuitive view of their condition while the physicians fill in the clinical details. In a meta-analysis by van der Kruk et al., multiple studies have shown that using VR in patient education can improve satisfaction, understanding, knowledge, and a sense of engagement or empowerment.¹⁹ More importantly, XR can help relieve the anxiety and terror innate to many cardiovascular interventions. The VIRTUAL CATH trial, which randomized 64 patients to either VR or traditional care, found that patients with VR training have significantly lower periprocedural anxiety levels, better procedural understanding, and overall satisfaction.²⁰ The VR approach was also implemented in vulnerable populations; for example, Project Brave Heart at Lucile Packard Children’s Hospital Stanford utilized VR to give pediatric patients several virtual catheterization experiences and mindfulness training before the actual procedure.²¹ Recently, VR-assisted conscious sedation during transfemoral transcatheter aortic valve implantation (TAVI) has been shown to be safe, feasible, and satisfactory in the elderly population.²²

Telemedicine

The COVID-19 pandemic brought significant challenges to existing medical systems. Mandatory social distancing, isolation, and the risk of exposure made traditional clinical visits and meetings impossible during the pandemic. VR can provide immersive rehabilitation training for patients while the risk of exposure stays minimal.²³ The combination of VR and AR can profoundly transform telemedicine from traditional video calls into an immersive, personalized experience with security provided by non-fungible tokens.²⁴ Telemedicine may become the norm, where patients can give their medical data remotely and discuss it with physicians/surgeons. In complex cases, virtual consultation with experts can be arranged seamlessly without the limitation of time and space.²⁵ Sadeghi et al. demonstrated that VR-based remote multidisciplinary heart team meetings are achievable and highly rated among VR naïve users to overcome social distancing rules during COVID-19 lockdowns.²⁶ The immersive nature of VR even made the heart team meetings more engaging than traditional ones (Figure 4);²⁷ hence, the need for telemedicine will continue long after the pandemic.

Pre-procedural planning

Modern cardiologists and surgeons have multi-sliced computed tomography (CT), magnetic resonance imaging (MRI), 3D echocardiography, positron emission tomography, and angiography at their disposal for procedural planning.²⁸ However, most still rely on traditional two-dimensional displays to evaluate and measure these innately 3D structures, and in this regard, XR is the ideal tool for accurate 3D visualization and manipulation. Specifically, XR can make these 3D images easier to interpret through intuitive 3D visualization, more manipulatable through voice/gesture commands, and more accessible through the mobility of XR headsets. The goal of XR is to complement the ever-improving cardiovascular imaging technologies by enabling surgeons and interventional cardiologists to comfortably examine the anatomy and plan the procedure by themselves, freeing their gaze from the screen to the patient and heart team. In addition, with contemporary computers, combining all useful clinical information, such as hemodynamic data and anatomy, is feasible in one cohesive patient avatar, which can then be displayed on XR systems for in-depth pre-procedural discussion, planning, and practice. These XR 3D models have many advantages over their 3D printed counterparts; they are dynamic, more maneuverable, and potentially faster to produce. In Figure 5 and Supplementary data online, Videos S1 and S2, we demonstrate an example of XR-integrated clinical flow in our heart team. Contemporary medical XR systems such as The Body VR (Meta, Menlo Park, CA, USA) accept images in the universal digital imaging and communications in medicine format (DICOM), enabling patient-specific evaluation.³⁰ VR-based procedure planning was pioneered at Stanford. In their project, 3D models were created using the True3D technology of EchoPixel (Mountain View, CA, USA), allowing surgeons or interventionists to perform virtual run-throughs of the procedure prior to actually performing it.³ EchoPixel’s software can be used with its desktop 3D display or VR HMD to allow an operator to have a 3D visualization of the cardiac anatomy. Using a stylus, the operator can rotate, cut into, and measure specific anatomical parts. EchoPixel is the first 3D display to gain approval from the U.S. Food and Drug Administration (FDA) and recently gained device registration in Japan.³¹ Since its FDA approval, EchoPixel has been used in various adult and pediatric cases, including left atrial appendage occlusion, trans-septal puncture, percutaneous mitral valve repair, TAVI, atrial septal defect, ventricular septal defect, etc. An increasing number of centers have adopted this kind of technology for pre-procedural planning.³² In Supplementary data online, Video S3, we demonstrate an example of our XR approach to TAVR implantation. XR implementation in congenital heart disease is rapidly maturing.³³ For multi-modality imaging, Narang et al. demonstrated that merging dynamic echocardiography and CT imaging could be performed quickly (<3 min) while retaining high fidelity. In their study, measurement variabilities and the time needed for measurement were reduced using VR.³⁴ Kamiya et al. demonstrated the feasibility of reconstructing a 3D model of the aortic root in exquisite detail, which in the VR environment was highly accessible and allowed precise measurements of the geometrically complex curves of the aortic root, whilst also showing the dynamic changes in the aortic valve with the accuracy of measurements against a known prosthesis within 1 mm of error (<0.5%).³⁵ Wang et al. created a CT-derived 3D model of a case with a residual aortic dissection following surgery for a type A dissection. This model provided not only a detailed anatomical overview, including the location of entrances, exits, and recirculation zones of the false lumen, but also, through the use of computational fluid dynamics, it provided a wealth of hemodynamic information, including flow, time-averaged wall shear stress, and pressure distribution.³⁶ This non-invasive patient-specific model can help guide the timing and strategy of intervention (Figure 6).

For difficult vascular access, interventionists in Japan have now developed a 3D model using free online software, CT scans, and surface scan imaging for VR-based planning.³⁷ There are over a dozen XR studies in surgical planning, with most in cases of congenital heart disease and complex anatomy.³⁸ However, multicenter, prospective, and observational studies are still absent, and therefore data regarding the safety and efficacy of these techniques and assessment of patient outcomes are lacking.

Intra-procedural visualization

Most previous investigations of XR are limited to the pre-procedural phase; however, physicians have much to gain from the intra-procedural use of XR. For example, physiological data such as heart rate and blood pressure can be displayed on AR goggles during interventions, thereby reducing the operator’s distraction from the procedural field. The real potential of XR, however, lies in displaying 3D models constructed pre-operatively in the procedural field, which is now possible with MxR/MeR HMDs such as the HoloLens (Microsoft, Redmond, WA, USA), which is equipped with voice and gesture commands for navigation in a sterile environment. Intra-procedural XR aims to provide detailed anatomical (and physiological) information and intra-operative guidance to support a procedure and make procedures more accurate, safe, and efficient (Figure 7, Supplementary data online, Video S3). The intra-procedure use of XR was pioneered by the Cardiology group in Warsaw, Poland, in 2015, who used the Google Glass AR system (Google LLC, Mountain View, CA, USA) in a case of coronary artery disease where static images of the pre-procedural CT angiogram showing a perfusion defect were sent to the goggles worn by the interventionist with suggested angiographic angulations to assist coronary intervention.³⁹ In 2016, Israeli investigators demonstrated the feasibility of creating real-time interactive 3D digital holograms with RealView holographic display (Realview Imaging Inc., Yokneam, Israel) using intraprocedural data from 3D rotational angiography and live 3D trans-esophageal echocardiography directly in the catheterization room.⁴⁰ The RealView display is a stand-alone platform that eliminates discomfort from prolonged HMD use, with the RealView HOLOSCOPE™-i display recently receiving FDA approval.⁴¹ Ballocca et al. introduced the EchoPixel system to measure complex mitral valve anatomy during mitral valve repair procedures.⁴² Kasprzak and coworkers described using MxR guidance for performing balloon mitral commissurotomy with real-time 3D trans-esophageal echocardiography images transferred to the CarnaLife Holo station (MedApp, Krakow, Poland), which then projected the reconstructed 3D images to a HoloLens HMD (Figure 8). This MxR tool allows the operator to control the hologram by voice commands and hand gestures in the sterile procedural field,⁴³ giving this wearable platform the advantage of being able to keep image references in the procedural field, which is impossible for traditional screens, and enabling the operator to actively interact with the 3D images without calling assistance or interrupting the procedure. XR has also demonstrated its benefit during transcatheter heart valve implantation/repair and atrial septal defect closure; in these cases, the control of 3D transesophageal echography was shared with the operator.⁴⁴ MxR has been used for complex coronary interventions such as chronic total occlusion, where visualization of the distal vessel can facilitate successful revascularization.³⁹ EP is another field that may benefit tremendously from using XR. Modern EP procedures rely heavily on electro-anatomical mapping systems, which record the location and signals of electrodes to generate complex 3D mapping information. Visualization of these 3D mappings can prove challenging on 2D displays, even for experienced operators. In addition, the current EP workflow relies on seamless cooperation between technicians at multiple workstations and the physician, who has no direct control of the mapping system.⁴⁵ The Enhanced ELectrophysiology and Interaction System (ELVIS) project, developed by Silva et al., is an essential step to overcoming these hurdles, and recently, the first-in-man MxR-assisted ablation was done showing low system latency (<100 ms), high display quality, and decreased navigation errors.^45–47 One of the most significant benefits of the ELVIS system is to return the control of the 3D mapping to the hands of electrophysiologists. This has great potential to improve the speed and quality of the mapping/ablation process. In a follow-up study, the group further demonstrated, in silico, the superior navigation accuracy of ELVIS-guided ventricular tachycardia ablation in two MRI-based virtual hearts.⁴⁸ Although the evidence of XR in the EP field is still scarce, given the heavy reliance on 3D mapping, the unique capacity to localize instruments in 3D space and the trend toward more complex procedures make EP the optimal target for XR implementation.

Post-procedural care

The role of XR technology can extend into post-procedural pain management and rehabilitation.⁴⁹ The MindMotionPRO and MindMotionGo are AR devices widely used in patient and outpatient settings.⁵⁰ These platforms encourage users to perform intensive physical rehabilitation by controlling their 3D avatar in a virtual 3D environment. In their 2017 pilot study in 10 patients with chronic upper-limb paralysis, Perez-Marcos and colleagues demonstrated that using XR led to significant improvements in motor function (5.3%) and the range of shoulder movements.⁵¹ The use of XR is well studied in the rehabilitation of stroke patients, and a recent meta-analysis of 19 subacute stroke rehabilitation studies showed significant improvement in motor function in the VR group when compared with pre-intervention values.⁵² VR has also been shown to reduce pain levels and periprocedural anxiety and contribute to faster post-procedural functional recovery in patients undergoing cardiovascular interventions.⁴⁹ These kinds of ‘VR hypnosis’ has been proven to be non-inferior in randomized trials to traditional painkillers.^53,54 Furthermore, small studies have shown that short VR sessions can significantly reduce healthcare providers’ fatigue and stress levels.⁵⁵ With these encouraging results, XR will likely have a more significant role in other aspects of post-procedural care, such as wound care education, heart failure rehabilitation, or increasing adherence to post-procedural recommendations.

XR for patients

In the discussions above, we have reviewed how XR may transform all stages of cardiovascular intervention from a clinician’s perspective; however, from a patient’s perspective, they also have tremendous potential to improve patients’ comfort, autonomy, and quality of life. The increasing reliance on medical imaging and electronic health records has significantly eroded the time clinicians spend interacting with patients,^56,57 which may reduce the quality of communication and patient-clinician trust. However, XR technologies have the potential to reverse this trend by engaging patients in interactive discussions⁵⁸ whilst also granting clinicians mobility to leave their screens and visit the bedside. As discussed above, they have great potential to increase patients’ knowledge and aid informed decision-making.⁵⁹ The immersive nature of VR can also help patients escape the horror of invasive procedures, reducing anxiety and discomfort,¹⁹ whilst XR will likely make rehabilitation and telemedicine sessions more effective and engaging.⁶⁰ However, it should be left to patients to give the final judgment, and thus future XR trials should be designed with the patient-reported outcome measures and the patient-reported experience measures in mind.

Limitations

The term ‘VR’ was first introduced in 1987,⁶¹ and ever since there have been predictions that it will revolutionize future health care. In the 1990s, the expectation was that the marriage between 3D echocardiography and VR would change our perspective of cardiac structures.⁶² In the 2000s, the family of VR branched out into AR and MxR, and the vision was that these XR technologies would find their way into every classroom and surgical theater.⁶³ Five years ago, Silva et al. described in detail the overwhelming progress of XR technologies and their many applications,¹ however, five years on, as we close in on the mid-2020s, XR continues to be the ‘next big thing’ in medicine.

The remaining barriers to overcome before XR can be a part of our daily practice include:

1. Cost: Current VR, MxR, and MeR headsets can easily cost over 1000 US dollars, and with the accompanying 3D processing software, logistics, and dedicated personnel, costs are even higher, rendering an XR project financially out of reach for most. However, more affordable options are emerging; for example, to overcome the economic barriers of expensive HMDs, the Google cardboard headset requires only simple materials and a smartphone to build VR experiences.⁶⁴ Although the image quality is suboptimal (frame rate usually <15 frames per second), and the device is not comfortable for long-term use, this does not prevent the Google cardboard headset from being used for education and visualization.^65,66 This kind of simple but easily obtainable XR display may prove helpful in resource-limited environments. Indeed, some centers have also devised a creative way to produce VR at a lower cost using free, open-source software like the NIH-funded Slicer 3D (https://www.slicer.org/), which has enabled the creation of patient-specific four-dimensional cardiac models of congenital heart disease for <250 US dollars.^37,67

2. Limited evidence: Most XR studies are performed in pioneering centers, and these ‘first-in-man,’ single-center studies offer little evidence toward changes in practice, whilst their lack of methodological standardization and significant sample sizes make comparisons between studies impossible. Furthermore, meta-analyses of the use of XR in education, pre-procedure planning, and rehabilitation have shown modest results.^19,52,68 Currently, only a few platforms have been approved by the FDA as medical devices, and therefore without robust evidence from randomized, multicenter trials with a consistent methodology, it is challenging to convince hospital administrators or governments to fund further XR projects.

3. Lack of physician/patient input in development: Although medical applications were always on the mind of developers, the enthusiasm for XR has primarily been driven by the entertainment industry. Apart from a few selected dedicated medical platforms (RealView, EchoPixel, etc.), most of the XR systems mentioned above were developed for gaming. Interventionists need precise, high-definition images for measurement and navigation, with an intuitive and straightforward control system for physicians to manipulate while wearing gloves or holding equipment. On the other hand, cardiovascular patients frequently require visual/hearing aids and are elderly, who may be more prone to motion sickness and need systems with a simple interface to navigate the XR space.⁶⁹

4. Another critical hurdle for XR technologies is the lack of reliability: Medical care usually involves extreme environments such as big temperature swings, high humidity, vibration, sterilization, and ionizing radiation that may damage electronic devices.⁷⁰ In addition, users may also pose a threat to XR devices; for example, they may be accidentally dropped during rehabilitation or procedures. The software interface must also withstand being operated by inexperienced users without crashing. For XR systems to be practical, they must remain stable without any unplanned or unexpected pauses, glitches, or shutdowns during heart team discussions, education, rehabilitation, or during a procedure. Importantly, the current evidence on XR implementation must be taken with a grain of salt, as these studies were performed by experienced users with excellent technical support. Future studies will not only have to prove the benefit of XR-integrated healthcare but also its reliability in everyday clinical practice.

Future perspective

The ideal future model of XR is to construct a patient-specific avatar using the combination of pre-procedural CT or MRI images, which are simultaneously being updated with real-time 3D echocardiography or device navigation systems. The XR system should have high fidelity and precise measurement in regions of interest. It should be able to switch between displayed information, such as hemodynamic data or electrophysiological mapping, in a sterile environment. Furthermore, the system must be efficient, user-friendly, and cost-effective for daily clinical use. Most importantly, the clinical utility of the system must be tested by rigorous randomized controlled trials. Although the ideal XR system is still far from reality, many investigators, entrepreneurs, and institutions have invested in the field.⁷¹ In the current trend, XR development will be decentralized from large corporations such as Microsoft or Meta to small businesses, reducing the threshold for XR application. Undoubtedly, the future of XR will benefit significantly from AI and communication technologies, such as the fifth-generation communication network, which can free the user from bulky workstations, and blockchain, which enables secure and decentralized communication.

Conclusion

Rapid hardware and software development has ushered us into a new era of XR in cardiovascular medicine. XR have the potential to improve all aspects of cardiovascular care, from patient education to pre-procedural planning, procedural visualization, and post-procedure care. The evidence for each of these implementations is quickly accumulating, and the results are promising. However, future studies are needed to cement the role of XR in cardiovascular medicine. Time will tell whether XR technologies will successfully reshape the landscape of cardiology, enabling physicians to provide more precise, safer, and advanced care to their patients.

Supplementary data

Supplementary data are available at European Heart Journal online.

Declarations

Disclosure of Interest

Patrick W. Serruys reports consultancy fees from Philips/Volcano, Sahajanand Medical Technologies (SMT), Novartis, Xeltis, and Merillife, outside the submitted work.

Data Availability

No new data were generated or analysed in support of this manuscript.

Funding

This work was supported by CÚRAM, SFI Research Centre for Medical Devices, Galway, Ireland.

References

Abbreviations

 
• 3D –

  three-dimensional

 
• AI –

  artificial intelligence

 
• AR –

  augmented reality

 
• CT –

  computed tomography

 
• DICOM –

  digital imaging and communications in medicine

 
• FDA –

  Food and Drug Administration

 
• HMD –

  head-mount-display

 
• MxR –

  mixed reality

 
• MeR –

  merged reality

 
• MRI –

  magnetic resonance imaging

 
• NFT –

  non-fungible token

 
• PET –

  positron emission tomography

 
• TAVI –

  transcatheter aortic valve implantation

 
• VR –

  virtual reality

 
• XR –

  extended realities