Augmented reality based real-time subcutaneous vein imaging system.

The visualization of medical images with advanced techniques, such as augmented reality and virtual reality, represent a breakthrough for medical professionals. In contrast to more traditional visualization tools lacking 3D capabilities, these systems use the three available dimensions. To visualize medical images in 3D, the anatomical areas of interest must be segmented. Currently, manual segmentation, which is the most commonly used technique, and semi-automatic approaches can be time consuming because a doctor is required, making segmentation for each individual case unfeasible. Using new technologies, such as computer vision and artificial intelligence for segmentation algorithms and augmented and virtual reality for visualization techniques implementation, we designed a complete platform to solve this problem and allow medical professionals to work more frequently with anatomical 3D models obtained from medical imaging. As a result, the Nextmed project, due to the different implemented software applications, permits the importation of digital imaging and communication on medicine (dicom) images on a secure cloud platform and the automatic segmentation of certain anatomical structures with new algorithms that improve upon the current research results. A 3D mesh of the segmented structure is then automatically generated that can be printed in 3D or visualized using both augmented and virtual reality, with the designed software systems. The Nextmed project is unique, as it covers the whole process from uploading dicom images to automatic segmentation, 3D reconstruction, 3D visualization, and manipulation using augmented and virtual reality. There are many researches about application of augmented and virtual reality for medical image 3D visualization; however, they are not automated platforms. Although some other anatomical structures can be studied, we focused on one case: a lung study. Analyzing the application of the platform to more than 1000 dicom images and studying the results with medical specialists, we concluded that the installation of this system in hospitals would provide a considerable improvement as a tool for medical image visualization.

Novel devices based on the emission of near-infrared electromagnetic radiation (NIR) have been developed to minimize venous puncture failures. These instruments produce an "augmented reality" image, in which subcutaneous veins are depicted on a LCD display. We compared the new technique with standard venipuncture in a population of elderly patients. Patients admitted in Intensive Care Unit were randomized to standard or to NIR assisted procedure. In the 103 enrolled patients (age 74±12years; standard venipuncture-N=56; NIR-N=47), no differences were found in procedure length, number of attempts, and referred pain. With NIR there was a lower incidence of hematomas and fewer anxiety and depressive symptoms. The use of the novel NIR-based device is safer and more psychologically tolerable, and it is not associated to an increase of procedure length or number of attempts.

Three-dimensional (3D) reconstruction models are useful for many situations in maintenance and decommissioning work at nuclear power plants (NPPs). For construction of 3D reconstruction models, it is desirable that they be constructed from images obtained by scanning work sites with a camera. Nevertheless, it is difficult for users to scan with being conscious of the appropriate scanning techniques and unscanned areas. Users who have no knowledge about scanning with a camera therefore have difficulty scanning work sites and capturing enough images to construct accurate 3D reconstruction models of them. In this study, we aim to develop a scanning support system using augmented reality (AR). This system detects and visualizes the current scanning status to remind and encourage users to scan as few unscanned areas as possible. For this study, we evaluated the scanning support system effectiveness by comparing the performance of developed system and non-support system. Results demonstrated that the developed system could scan larger areas and that it could capture more useful images. We also conducted an evaluation experiment at an actual NPP work site to ascertain the effectiveness, even at an actual NPP work site. Results showed that the developed system can scan larger areas, even at an actual work site.

The objective of this research was to develop a three-dimensional (3D) reconstruction system based on a time-domain optical coherence tomography (OCT) microscope. One of the critical drawbacks of OCT microscopes is that their axial measurement ranges are typically limited by their depths of field (DOFs), which are determined by the numerical apertures of their objective lenses and the central wavelengths of their light sources. If a low-coherence interference fringe is far outside the DOF, the measurement accuracy inevitably decreases, regardless of how well-adjusted the reference mirror is. To address this issue and improve the axial measurement range of the OCT microscope in this study, an object-scanning measurement scheme involving a Linnik interferometer was developed. To calibrate the system in the proposed technique, image post-processing is performed for a well-conditioned state to ensure that a low-coherence interference fringe is generated within the DOF, enabling 3D objects with high-aspect-ratio structures to be scanned along the axial direction. During object-scanning, this state is always monitored and is corrected by adjusting the reference mirror. By using this method, the axial measurement range can be improved up to the working distance (WD) of the objective lens without compromising the measurement accuracy. The WD is typically longer than 10 mm, while the DOF of the microscope is around 0.01 mm in general, although it varies depending on the imaging system. In this report, the experimental setup of a 3D reconstruction system is presented, a series of experimental verifications is described, and the results are discussed. The axial measurement range was improved to at least 35 times that of a typical OCT microscope with identical imaging optics.

The technique of subcutaneous veins imaging by using NIR (Near Infrared Radiation) is widely used in medical applications, such as the intravenous injection and the blood sampling. In the previous study, an automatic 3D blood vessel search and automatic blood sampling system was newly developed. In order to validate this NIR imaging system, we adopted the subcutaneous vein in the human arm and its artificial phantom, which imitate the human fat and blood vessel. The human skin and subcutaneous vein is characterized as the uncertainty object, which has the individual specificity, non-accurate depth information, non-steady state and hardly to be fixed in the examination apparatus. On the other hand, the conventional phantom was quite distinct from the human’s characteristics, such as the non-multilayer structure, disagreement of optical property. In this study, we develop a multilayer phantom, which is quite similar with human skin, for improvement of NIR detection system evaluation. The phantom consists of three layers, such as the epidermis layer, the dermis layer and the subcutaneous fat layer. In subcutaneous fat layer, we built a blood vessel. We use the intralipid to imitate the optical scattering characteristics of human skin, and the hemoglobin and melanin for the optical absorption characteristics. In this study, we did two subjects. First, we decide the fabrication process of the phantom. Second, we compared newly developed phantoms with human skin by using our NIR detecting system, and confirm the availability of these phantoms.

Augmented Reality (AR) is one of the latest developments in human-computer interaction technology. It aims to generate illusions from seamless fusion of virtual objects and real world. Typical AR system requires two basic parts: three-dimensional registration and real-virtual fusion. Occlusion handling is crucial for visual realism. To optimize visual realism, we generated a real-time systematic architecture to operate occlusion handling. The architecture is based on RGBD images, and it consists of three parts: real-time camera tracking system, 3D reconstruction system and AR fusion system. Specifically, we used a two-pass scheme strategy to execute the AR system. The first pass tracks camera poses timely at video rate, which allows the reconstruction results be updated and visualized correspondingly during the scanning. The second pass takes place simultaneously to handle occlusion between virtual objects and real scene according to camera pose. Finally, the render results of virtual objects and the color images are fused to generate AR contents. Our results indicate that this method is stable and precise for occlusion handling, and can effectively improve realism in AR system.

Three-dimensional (3D) reconstruction technology has found widespread applications across various domains, including intelligent driving and underwater exploration. But the existing imaging systems and methods still have deficiencies in terms of reconstruction accuracy, detection distance and system volume. Herein, this paper presents a three-dimensional detection and reconstruction method based on a compact Risley-prism 3D imaging system that achieves multi-viewpoint imaging by rotating the Risley prism to adjust the camera's optical axis. A refractive camera model that integrates the pinhole camera model with the vector form of Snell's law is established to precisely describe beam trajectory. A forward projection method suitable for refractive interfaces is developed based on Fermat's principle, and the influence of systematic errors on the reconstruction is analyzed in detail through simulation. Furthermore, a new 3D reconstruction method combining error calibration based on the optimization iteration is introduced to avoid the influence of error and improve reconstruction quality. Experimental results demonstrate that the proposed approach markedly enhances 3D reconstruction accuracy, reducing the Normalized Root Mean Square Error (NRMSE) from 0.9076 to 0.0207.

This paper studies the application of augmented reality real-time depth image technology to 3D human motion recognition technology. The accuracy and real-time performance of sensor-based 3D human reconstruction are affected by visual characteristics and illumination changes. Features are not easily extracted and cannot be tracked, leading to failures in the 3D reconstruction of the human body. Based on this system, the sensor-based visual inertial initialization algorithm is studied, which is integrated in the two-frame image time interval to provide accurate initial values for vision-based motion estimation, improve the accuracy of the calculated posture, and finally improve the accuracy of the 3D reconstruction system. Based on the relationship between the depth image and the distance and reflectivity, a model for correcting the distance error and reflectivity error of the depth image is established to improve the accuracy of the depth image, and finally the accuracy of the three-dimensional reconstruction of the human body.

Three‐dimensional (3D) reconstruction based on structured light (SL) has wide applications in fields such as artificial intelligence, surveying and mapping, and precision manufacturing. An ideal 3D reconstruction system features both wide field‐of‐views (FOV), high‐accuracy and high‐speed reconstruction and ultracompact volume. However, these criteria have been achieved only partially with conventional SL techniques like optical projector and differentiate optical element, which will substantially limit their practical applications. In this study, an ultracompact SL and its 3D reconstruction system is designed and experimentally demonstrated, which exhibits super‐large FOV angles up to 210° × 210°, only 3.8‰ reconstruction errors, real‐time processing speeds up to 30 fps and SL dimensions as small as 0.3 mm × 0.3 mm. Specifically, the SL is realized with a resonant geometric metasurface, which can produce tens of thousands of random dots and project them into the whole 4π free space. The metasurface enabled SL is collaborated with a binocular stereoscopic camera and advanced 3D reconstruction algorism to form a practical 3D reconstruction system, which can provide robust hardware and software support for high‐accuracy and real‐time 3D scene reconstruction.

Visualization of subcutaneous veins is very difficult with the naked eye, but important for diagnosis of medical conditions and different medical procedures such as catheter insertion and blood withdrawal. Moreover, recent studies showed that the images of subcutaneous veins could be used for biometric identification. The majority of methods used for enhancing the contrast between the subcutaneous veins and surrounding tissue are based on simple imaging systems utilizing CMOS or CCD cameras with LED illumination capable of acquiring images from the near infrared spectral region, usually near 900 nm. However, such simplified imaging methods cannot exploit the full potential of the spectral information. In this paper, a new highly versatile method for enhancing the contrast of subcutaneous veins based on state-of-the-art high-resolution hyper-spectral imaging system utilizing the spectral region from 550 to 1700 nm is presented. First, a detailed analysis of the contrast between the subcutaneous veins and the surrounding tissue as a function of wavelength, for several different positions on the human arm, was performed in order to extract the spectral regions with the highest contrast. The highest contrast images were acquired at 1100 nm, however, combining the individual images from the extracted spectral regions by the proposed contrast enhancement method resulted in a single image with up to ten-fold better contrast. Therefore, the proposed method has proved to be a useful tool for visualization of subcutaneous veins.

The status of axillary lymph node metastases determines the treatment and overall survival of breast cancer (BC) patients. Three-dimensional (3D) assessment methods have advantages for spatial localization and are more responsive to morphological changes in lymph nodes than two-dimensional (2D) assessment methods, and we speculate that methods developed using 3D reconstruction systems have high diagnostic efficacy. This exploratory study included 43 patients with histologically confirmed BC diagnosed at Second Xiangya Hospital of Central South University between July 2017 and August 2020, all of whom underwent preoperative CT scans. Patients were divided into a training cohort to train the model and a validation cohort to validate the model. A 3D axillary lymph node atlas was constructed on a 3D reconstruction system to create various methods of assessing lymph node metastases for a comparison of diagnostic efficacy. Receiver operating characteristic (ROC) curve analysis was performed to assess the diagnostic values of these methods. A total of 43 patients (mean [SD] age, 47 [10] years) met the eligibility criteria and completed 3D reconstruction. An axillary lymph node atlas was established, and a correlation between lymph node sphericity and lymph node metastasis was revealed. By continuously fitting the size and characteristics of axillary lymph nodes on the 3D reconstruction system, formulas and models were established to determine the presence or absence of lymph node metastasis, and the 3D method had better sensitivity for axillary lymph node assessment than the 2D method, with a statistically significant difference in the correct classification rate. The combined diagnostic method was superior to a single diagnostic method, with a 92.3% correct classification rate for the 3D method combined with ultrasound. In addition, in patients who received neoadjuvant chemotherapy (NAC), the correct classification rate of the 3D method (72.7%) was significantly higher than that of ultrasound (45.5%) and CT (54.5%). By establishing an axillary lymph node atlas, the sphericity formula and model developed with the 3D reconstruction system achieve a high correct classification rate when combined with ultrasound or CT and can also be applied to patients receiving NAC.

The combination of augmented reality (AR) systems and stereoscopic display devices has created a powerful tool with which to "supplement" our view. One application for such systems is for the augmented teleoperation of unmanned vehicles, deployed in remote or hazardous environments. Research in this area has highlighted the need for accurate 3D measurement techniques - such as through the use of virtual cursors. This paper describes our work in the development of an AR interface designed to assist the accurate selection of position in 3D space. We describe some preliminary experimental work using virtual cursors before discussing how we believe depth cues can be utilized to allow a user to make a more informed judgment of depth in unprepared environments. It is expected that the guidelines outlined in this report will be used as a benchmark for the development of further 3D ARA cursors.

A high-resolution, real-time range imaging system based on a LED array phase-shift range finder was developed to measure three-dimensional surface profiles. The range image is captured by a time-division scanning LED array heterodyne phase-shift range finder. The phase measurement used for obtaining depth is implemented by using the fast "four-bucket" algorithm. The design of a fast automatic gain control circuit obviously reduces the feedback response time and improves system measurement speed. Experimental tests for evaluation of the imaging system performance are described. Based on the theoretical analysis, a simplified signal-to-noise ratio model is proposed for guiding the design of this imaging system. Last, some issues requiring attention on constructing a large-pixel version of the system are also discussed. It was found that the range images can be acquired at a rate of 10 frames per second with a depth resolution better than ±5 mm in the range of 50 to 1000 mm.

PurposeIn soft tissue reconstructive surgery, perforator localization and flap harvesting have always been critical challenges, but augmented reality (AR) has become a dominant technology to help map perforators.MethodsThe lateral circumflex femoral artery (LCFA) and its perforators were reconstructed by CTA in consecutive patients (N = 14). Then, the anterolateral thigh perforators and the points from which the perforators emerged from the deep fascia were marked and projected onto the skin surface. As the virtual images were projected onto patients according to bony markers, the courses of the LCFA and its perforators were depicted on the skin surface for intraoperative guidance. Finally, the locations of the emergence points were verified by intraoperative findings and compared to those determined by handheld Doppler ultrasound.ResultsThe sources, locations, and numbers of perforators were determined by CTA. The perforators and their emergence points were accurately mapped on the skin surface by a portable projector to harvest the anterolateral thigh perforator flap. During the operation, the accuracy of the CTA & AR method was 90.2% (37/41), and the sensitivity reached 97.4% (37/38), which were much higher than the corresponding values of Doppler ultrasound. Additionally, the differences between the AR-marked points and the intraoperative findings were much smaller than those seen with Doppler ultrasound (P < 0.001). Consequently, all of the flaps were well designed and survived, and only one complication occurred.ConclusionAugmented reality, namely, CTA combined with projection in this study, plays a vital and reliable role in locating the perforator emergence points and guiding the procedure to harvest flaps and has fewer potential risks.

In today’s world, people increasingly rely on mobile apps to do their day-today activities, like checking their Instagram feed and online shopping from websites like Amazon and Flipkart. People are depending on WhatsApp and Instagram stories to communicate with local businesses and to leverage the said platforms for online advertising. Using Google Maps to find their way when they travel and finding out the immediate road and traffic conditions with digital banners around the road, has obviously led to a boom in advertising and marketing. Recently, internet users have increasingly desired to immerse themselves in a Metaverse-like platform where they can interact and socialize. Meta’s Metaverse is a tightly connected network of 3D digital spaces that allow users to escape into a virtual world. It is designed to change the way you socialize, work, shop, and connect with the real and virtual world around you. These platforms are not fully submerged in the real world; they are inclined toward virtual spaces only, making it obvious to fill this gap. Thus, the proposed framework in this chapter would be a new kind of system that may develop a socio-meta platform, powered by augmented reality and other technologies like photogrammetry and LiDAR. Augmented reality provides an interactive way of experiencing the real world, where the objects of the natural world are enhanced by computer-generated perceptual vision. One of the significant problems with augmented reality is the process of building virtual 3D objects that can be augmented into real spaces, which could be solved with photogrammetry and LiDAR. Photogrammetry is the technique of producing 3D objects using 2D images of a physical object taken from different angles and orientations. LiDAR, on the other hand, is another 3D reconstruction technology used widely by Apple’s eco-system. The functioning of LIDAR is very similar to sonar and radar, and the detection and ranging part are where it stands out from the others. The idea behind this platform is to open tons of virtual dimensions in the real world using the principles of mixed reality and geographic mapping tools such as Google Maps, MapBox, and GeoJSON; it would consequently transition the way people spend their time on social media by opening a portal for generating 3D objects that can be augmented to the real-world location using photogrammetry and cloud anchors by just a few 2D digital photographs taken from their camera.