Abstract : With the development of the metaverse, integrating the virtual world into real life has become a trend for creating interactive applications. In recent years, the technologies of mixed reality (MR) systems have made significant advancements and their applications can be found in various fields. In this paper, we propose a prototype system that can assist doctors to perform their surgeries in real-time by using MR glasses. The surgery MR system primarily operates through the proposed medical object detection network and the MR glasses module. In the medical object detection network, we introduce an object aware network to optimize the detection flow to a stacked light-weight hourglass network in conjunction with the half cross adaptive fusion modules to improve the detection of small objects, i.e., the scalpel-tip. In the MR glasses module, the detection results of the network can be processed and precisely exhibited on glasses. The experimental results show that the proposed detection network can detect scalpel-tip better in real-time and also can be easily combined with MR glasses to achieve medical surgery applications. The system demonstration shows that the detected results overlaying augment virtual information onto the liver in the physical world on the MR glasses where the system can provide the alerts for detection of scalpel-tip away from the aorta on the MR glasses.

Keyword : Convolutional neural network, medical application, MR glasses, object aware network, object detection.