Cardiology AR

The cognitive process, i.e. the process of acquiring knowledge and understanding, in human beings occurs through thought, senses and experience.

Up to 60% of this process depends on the sense of sight and hearing, the remaining 40% is attributed to the practical or abstract application of what is stored.

A study carried out by the National Laboratory for Education in the US states that rates of mnestic retention (i.e. the ability to remember what you have experienced) increase to 75% when referring to the use of augmented reality within the educational process.

If, in addition to the above, we add that specialised training in the medical field, it involves added risks for the safety of students and teachers. When they find organic material and places to study it represents a considerable cost as well. So we can immediately see the potential of augmented reality in promoting safe and sustainable learning.

With the collaboration of Prof. Vladimiro Vida, Director of the UOC of Paediatric Cardiac Surgery and Congenital Heart Diseases and with the student of Medicine Francesco Bertelli, we analysed the usual learning path and the tools commonly used. The solution adopted so far has been the 3D printing of organs of the human body made by a software that uses scans obtained by means of CT, ECO and magnetic resonance imaging. These 3D models allow students to touch, see and dissect organs in the same way as real ones.

The need was therefore to obtain models as close as possible to real organs that could be fully analysed and dissected as if they were in the hands of a student. It was therefore vital not only to have the 3D model but also to conceive ways of interacting with it, all within a complex range of hearts with different shapes and different pathologies.

What can augmented reality offer more than a 3D printed model or even a real organ?

The positive impact, as explained above, has reduced risks and costs, but this technology also allows you to have a huge amount of material, models and information within a simple smartphone. It is not necessary to make, transport and store physical objects, moreover during the analysis more information can be requested of the system that, through a careful study of the UI, makes them appear on virtual models.

An accurate analysis of the most suitable technology has been carried out to reach, first of all, a mobile app able to host 3D models with related information and, subsequently, the hearts in augmented reality.

An initial MVP allowed us to submit the results of our study to the University of Padua. The findings on interfaces, navigation flow and digital asset yield were then endorsed and formed the basis for the development of a complete and structured app.

Therefore, all the screens were designed to allow users to access the multiple functionalities through a menu able to manage a large amount of information in a very intuitive way.

At the same time, 3D models of organ cavities have been optimised for use in real-time. Starting from assets already in possession of the University of Padua and giving them a photorealistic appearance as the result of applying high quality textures from the photogrammetry technique.

Augmented reality hearts can then be dissected and enriched with information to provide a complete experience for students and without compromising the quality of teaching. In addition, to maintain the flow of normal teaching, the virtual dissection of the models follows the process carried out during echocardiographic analysis: Long axis parasternal (PLAX), Short axis parasternal (PSSAX) and 4 chamber apical (A4C).

These models allow a precise morphological and functional analysis of the organ and of each of its parts.

To highlight areas of the heart with defects using colour, we had to create the UV Maps, which were not present in the scans. To do this, we painted directly in 3D, making it intuitive and easy to perform. The UV Maps also made it possible to effectively and correctly distribute the textures on the 3D models.

The development team had to manage many models in a short time, consequently the latest generation UV unwrapper was used, which reduces manual work and shortens development time.

In addition, we decided to select only one set of textures for all hearts, so that the mobile application would remain light and efficient (less than 60MB) while maintaining a 4K resolution.

A further challenge was to enrich the 3D models, both used through a 3D viewer and displayed in augmented reality, with the information in written form. This text had to appear in the least invasive way possible but, at the same time, it had to be easily readable. 

The result of all this is a non-linear user experience. Each student can interact in a different way. They can use different models with related sections, without a set path for each user but rather fostering different experiences adapted to the needs of all users. A training that is in some ways personalised and, therefore, more effective while managing and not increasing costs.

The output is therefore a mobile app for iOS and Android operating systems that uses ARKit and ARCore for real-time 3D visualization of the environment around us. Simple 3D visualization is also provided for older devices that do not support augmented reality.

The next developments include the possibility to use WebAR technology to enjoy the models without the need to download an app, minimising additional effort.

Augmented reality, like other immersive technologies, has a strong evocative power that allows users to engage fully in their experiences, as in the case of Nice Spa e Phasys VR.

In the training sector this kind of content allows a greater learning experience thanks to the process being comparable to the real one and lived through the normal didactic tools.

This effectiveness then meets efficiency. Virtual assets that can be analysed and manipulated anywhere and at any time drastically reduces the cost of finding and disposing of real organs, of using a specific space and equipment prepared for each student.

We are therefore facing an important evolution in the world of specialist training, which comes from the potential of new technology like augmented reality, already widely used and easy to interact with. A simple smartphone is all that is required to place virtual objects in the environment created around us, and to interact with them simply takes taps on the screen.

The University of Padua, close to 800 years of history, has embraced this new technology to improve the teaching of its PhD students in medicine, testifying to how the technology has been really useful; it offers a real benefit and fits seamlessly into processes developed over years and years of work.