Institute of Fundamental Technological Research

The dynamic COVID-19 pandemic has destabilized education and forced academic centers to explore non-traditional teaching modalities. A key challenge this creates is in reconciling the fact that hands-on time in lab settings has been shown to increase student understanding and peak their interests. Traditional visualization methods are already limited and topics such as 3D molecular structures remain difficult to understand. This is where advances in Information and Communication Technologies (ICT), including remote meetings, Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Extended Reality (XR, so-called Metaverse) offer vast potential to revolutionize the education landscape. Specifically, how MR merges real and virtual life in a uniquely promising way and offers opportunities for entirely new educational applications. In this paper, we briefly overview and report our initial experience using MR to teach medical and pharmacy students. We also explore the future usefulness of MR in pharmacy education. MR mimics real-world experiences both in distance education and traditional laboratory classes. We also propose ICT-based systems designed to run on the Microsoft HoloLens2 MR goggles and can be successfully applied in medical and pharmacy coursework. The models were developed and implemented in Autodesk Maya and exported to Unity. Our findings demonstrate that MR-based solutions can be an excellent alternative to traditional classes, notably in medicine, anatomy, organic chemistry, and biochemistry (especially 3D molecular structures), in both remote and traditional in-person teaching modalities. MR therefore has the potential to become an integral part of medical education in both remote learning and in-person study

information and communication technologies, immersive technologies, information and communication technologies in education, immersive technologies in education, Mixed Reality, 3D human–computer interaction, advanced medical education, pharmacy, Metaverse

Background: Immersive technologies like Mixed Reality (MR), Virtual Reality (VR) and Augmented Reality (AR) are becoming increasingly popular and gain user trust across various fields, particularly in medicine. In this paper we will use the general term Mixed Reality (MR) to refer to the various virtual reality methods, namely VR and AR. These new immersive technologies require varying degrees of instruction, both in their practice use, as well as in how to adjust to interacting with 3D virtual spaces. This study assesses the pedagogical value of these immersive technologies in medical education. Method: We surveyed a group of 211 students and 47 academic faculty at a medical college regarding potential applications of MR in the medical curriculum by using a questionnaire comprised of eight questions. Results were analyzed accounting for user age and professional position, i.e., student vs faculty. Results: 70% of students and 60% of the academic faculty think that MR-supplemented education is advantageous over a classical instruction. Most highly valued were the 3D visualization capabilities of MR, especially in anatomy classes. There was no significant statistical difference between students and faculty responders. Moreover, screensharing between faculty and students contributed to better, longer lasting absorption of knowledge. Surprisingly, the main issue was related to availability, i.e., only 5% of students had access to MR, while 17% of faculty use MR regularly, and 36% occasionally. Conclusions: MR technology can be a valuable resource that supports traditional medical education, especially via 3D anatomy classes, however MR availability needs to be increased. Moreover, MR expands the capabilities and effectiveness of remote learning, which was normalized during the COVID-19 pandemic, to ensure effective student and patient education. MR-based lessons, or even select modules, provide a unique opportunity to ex-change experiences inside and outside the medical community.

mixed reality, e-learning, remote learning, real-time rendering, 3D visualization, medical education

Early pancreatic cancer diagnosis and therapy drastically increase the chances of survival. Tumor visualization using CT scan images is an important part of these processes. In this paper, we apply Mixed Reality (MR) and Artificial Intelligence, in particular, Machine Learning (ML) to prepare image-guided 3D models of pancreatic cancer in a population of oncology patients. Object detection was based on the convolution neural network, i.e. the You Only Look Once (YOLO) version 7 algorithm, while the semantic segmentation has been done with the 3D-UNET algorithm. Next, the 3D holographic visualization of this model as an interactive, MR object was performed using the Microsoft HoloLens2. The results indicated that the proposed MR and ML-based approach can precisely segment the pancreas along with suspected lesions, thus providing a reliable tool for diagnostics and surgical planning, especially when considering organ reconstruction during pancreaticoduodenectomy.

Extended Reality, Mixed Reality, Augmented Reality, Head-Mounted Displays, Artificial Intelligence, Image-guided surgery