Volumetric image interpretation in radiology: scroll behavior and cognitive processes

This article presents an overview of the field of knowledge-based approaches in the processing and interpretation of medical images. It describes the major developments in artificial intelligence and image processing that have led up to the current potential to further the increasing interest in automated processing (knowledge-based systems) and interpretation (expert systems) of medical images. After a brief discussion of the principal scientific and engineering issues in the field of knowledge-based image processing, the process of building knowledge-based systems, and the frontiers of research and development, a number of reported systems for processing and interpretation of medical images are briefly reviewed. The development of an expert system for the analysis of scintigraphic images is then presented, emphasizing knowledge acquisition, implementation and evaluation. Finally, a perspective view on reasoning under uncertainty is presented.

Although widely used as a pointing device on personal computers (PCs), the mouse was originally designed for control of two-dimensional (2D) cursor movement and is not suited to complex three-dimensional (3D) image manipulation. Augmented reality (AR) is a field of computer science that involves combining the physical world and an interactive 3D virtual world; it represents a new 3D user interface (UI) paradigm. A system for 3D and four-dimensional (4D) image manipulation has been developed that uses optical tracking AR integrated with a smartphone remote control. The smartphone is placed in a hard case (jacket) with a 2D printed fiducial marker for AR on the back. It is connected to a conventional PC with an embedded Web camera by means of WiFi. The touch screen UI of the smartphone is then used as a remote control for 3D and 4D image manipulation. Using this system, the radiologist can easily manipulate 3D and 4D images from computed tomography and magnetic resonance imaging in an AR environment with high-quality image resolution. Pilot assessment of this system suggests that radiologists will be able to manipulate 3D and 4D images in the reading room in the near future. Supplemental material available at http://radiographics.rsna.org/lookup/suppl/doi:10.1148/rg.324115086/-/DC1.

Alterations within central nervous system (CNS) autonomic nuclei contribute to a wide array of diseases. In this context, significant advancements have been made into disease‐related molecular mechanisms in autonomic brain regions. While helpful, it is important to consider that information processing within the CNS occurs through dynamic micro‐architectural structural interactions, such astrocyte‐neuron synaptic connections. However the traditional method for evaluating CNS morphology and synaptic contacts between cells is performed with immunohistochemistry on thin brain sections. An inherent limitation of this approach is not only the narrow 2‐dimensional field of view, but also the lack of optical transparency that minimizes high‐resolution imaging of anatomical structures and cell‐cell interactions. To overcome these limitations, here we sought to optimize tissue clearing approaches combined with 3D volumetric imaging for autonomic CNS nuclei. We focused our initial efforts on the forebrain subfornical organ (SFO), a region lacking a blood‐brain‐barrier, given: 1 Alterations in the SFO have been described to contribute to numerous disease conditions; and 2 The SFO represents a unique technical challenge for 3D imaging due to the complexity of cellular processes and surrounding lateral ventricle space. As a first step, we used a “passive” CLARITY tissue approach in 8‐week‐old C57Bl/6J male mice. Mice were transcardially perfused with an acrylamide‐based hydrogel in standard fixative. Subsequently, a brain section encompassing the entire SFO (~1mm) was obtained, and a series of thermal diffusion steps was performed to place existing protein structures in a “scaffold” and remove the lipid contents ‐ rendering the tissue transparent. Whole‐mount immunohistochemistry for astrocytes was then conducted using GFAP and DAPI as a nuclei marker, followed by 3D volumetric multiphoton imaging to acquire high‐resolution visualization of the entire SFO in three planes (Figure top). In brief, this approach proved to be a significant advancement beyond standard 2D histology, and 3D visualization of intricate astrocyte processes throughout the whole SFO was possible. However, a major limitation was the extensive amount of time required to process a single sample; 2 weeks for tissue clearing and 2 weeks for primary and secondary antibody incubation. Based on this, we adapted our protocol to an “active” CLARITY approach and employed electrophoretic‐based clearing of large brain sections (n=5), which resulted in a reduction in clearing time from 2 weeks to 4 hours. It was also readily apparent that this approach rendered the tissue more transparent than passive clearing, allowing for greater visualization of astrocyte processes (Figure middle). Preliminary analysis with IMARIS software to analyze astrocyte surface area and dendritic processing (Figure bottom) revealed highly complex SFO astrocytic structures with some dendrites extending from soma at the lateral ventricle throughout and surrounding nearly the entire SFO; these processes are not visualized with standard histology. Collectively, these approaches highlight the utility of tissue‐based clearing for 3D visualization of anatomical structures within autonomic nuclei and provide a framework to understand structure‐function relationships within these brain regions in the regulation of physiological and pathophysiological conditions.

Since the discovery of X-rays in 1895, medical imaging systems have played a crucial role in medicine by permitting the visualization of internal structures and understanding the function of organ systems. Traditional imaging modalities including Computed Tomography (CT), Magnetic Resonance Imaging (MRI) and Ultrasound (US) present fixed two-dimensional (2D) images which are difficult to conceptualize complex anatomy. Advanced volumetric medical imaging allows for three-dimensional (3D) image post-processing and image segmentation to be performed, enabling the creation of 3D volume renderings and enhanced visualization of pertinent anatomic structures in 3D. Furthermore, 3D imaging is used to generate 3D printed models and extended reality (augmented reality and virtual reality) models. A 3D image translates medical imaging information into a visual story rendering complex data and abstract ideas into an easily understood and tangible concept. Clinicians use 3D models to comprehend complex anatomical structures and to plan and guide surgical interventions more precisely. This chapter will review the volumetric radiological techniques that are commonly utilized for advanced 3D visualization. It will also provide examples of 3D printing and extended reality technology applications in radiology and describe the positive impact of advanced radiological image visualization on patient care.

A common interaction with medical 2D and 3D images is to apply windows, icons, menus and pointers (WIMP) paradigm; however, WIMP might not be convenient in some situation, such as in the operating room where sterilization becomes an issue. Thus, in this project, the controller-free natural interaction on 3D medical image viewer software through a Kinect camera will be developed to replace WIMP such that surgeons can view medical images in the operating room without contacting mouse and keyboard. The interaction will include both body and figure gestures. The 3D medical image viewer software will be based on the Visualization ToolKit (VTK) library to visualize 3D image data in the DICOM (Digital Imaging and Communications in Medicine) format. Moreover, suitable controller patterns will be designed according to practical use in the operating room.

Computer-vision techniques based on wavelets are presented with applications to computer-aided diagnosis (CAD) in medical images. CAD is a method for assisting radiologists’ interpretation of medical images. CAD schemes act as a second reader and alert radiologists to suspicious lesions. Development of CAD schemes is an active area of research, motivated by the fact that radiologists tend to misdiagnose cancerous lesions in medical images. In the past several years, we have been extensively developing wavelet-based techniques for CAD schemes. Development of these techniques has led to a new CAD scheme and has yielded significant improvements in the performance of the existing CAD schemes. This chapter describes two wavelet-based, low-level computer vision techniques: simultaneous segmentation and registration, and matching pursuit with optimally weighted wavelet packets. We present their application to the computer-aided detection of lung nodules in chest radiographs and microcalcifications in mammograms.

All of the research publications describe, emphasise, and classify one of the constituent aspects of deep learning models (DL) employed in medical image interpretation, but they do not provide a unified picture of the importance and impact of each constituent on DL model performance. Deep learning (DL) has experienced an exponential development of medicine, but applications in interpretations of medical imaging are in continuous development.  Our paper is unique in that it takes a unitary strategy to the constituent elements of DL models, such as data, tools used by DL architectures, or specifically constructed DL architecture combinations, and highlights their "key" features for completing tasks in current applications in medical image interpretation. Future study could focus on the utilisation of "key" properties particular to each ingredient of DL models, as well as the correct determination of their correlations, with the goal of improving the performance of DL models in the interpretation of medical pictures.

The CAS3D image processing method intuitively applies a combination of Fourier space and real space 3D analysis algorithms to volumetric images of single skeletal muscle fiber Myosin II Second Harmonic Generation (SHG) XYZ image data. Our developed tool automatically quantifies the myofibrillar orientation in muscle samples by determining the cosine angle sum of intensity gradients in 3D (CAS3D) while determining the mean sarcomere length (SL) and sample orientation. The expected CAS3D values could be reproduced from ideal artificial data sets. Applied random noise in artificial images lowers the detected CAS3D value, and for noise levels below 20%, the correlation can be approximated by a linear function with a slope of −0.006 CAS3D/noise%. The deviations in SL and orientation detection were determined on ideal and noisy artificial data sets and were statistically indistinguishable from 0 (null hypothesis t-test P > 0.1). The software was applied to a previously published data set of single skeletal muscle fiber volumetric SHG image data from a rat intensive care unit (ICU) model of ventilator-induced diaphragm dysfunction (VIDD) with treatment regimens involving the small anti-inflammatory molecules BGP-15, vamorolone, or prednisolone. Our method reliably reproduced the results of the previous work and improved the standard deviation of the cosine angle sum detection in all sample groups from a mean of 0.03 to 0.008. This improvement is achieved by applying analysis algorithms to the whole volumetric images in 3D in contrast to the previously common method of slice-wise XY analysis.

Introduction: Medical images (e.g. radiographs) are the most commonly ordered tests in emergency medicine. As such, emergency medicine physicians are faced with the task of learning the skill of interpreting these images to an expert performance level by the time they provide opinions that guide patient management decisions. However, discordant interpretations of these images between emergency physicians and expert counterparts (e.g. radiologists) is a common cause of medical error. In pediatrics, this problem is even greater due to the changing physiology with age. Methods: ImageSim (https://imagesim.com/) is an evidence-based on-line learning platform derived and validated over an 11 year period (https://imagesim.com/research-and-efficacy/). This learning system incorporates the concepts of cognitive simulation, gamification, deliberate practice, and performance-based competency in the presentation and interpretation of medical images. Specifically, ImageSim presents images as they are experienced in clinical practice and incorporates a normal to abnormal ratio is representative of that seen in emergency medicine. Further, it forces the participant to commit to the case being normal or abnormal and if abnormal, the participant has to visually locate the specific area of pathology on the image. The participant submits a response and gets text and visual feedback with every case. After each case, the participant gets to play again until they reach a desired competency threshold (80% is bronze resident; 90% silver staff emergency medicine physician; 97% gold radiologist). Importantly, the learning experience also emphasizes deliberate practice such that the learning system provides hundreds of case examples and therefore each participants performance has the opportunity to improve along their individual learning curve. Results: Course selection was made based on known medical image interpretation knowledge gaps for practicing emergency physicians. Currently, ImageSim live courses include pediatric musculoskeletal radiographs (2,100 cases, 7 modules) and pediatric chest radiographs (434 cases). In 2018, we will also release a pediatric point-of-care ultrasound course (400 cases, 4 modules) and the pre-pubertal female genital examination (150 cases). For a demo, go to https://imagesim.com/demo/. Using ImageSim, the deliberate practice of about 120 cases (1 hour time commitment) increases accuracy on average by 15%. Currently integrated into 10 emergency medicine training programs and there are about 300 continuing medical education world-wide participants. Conclusion: While acquiring mastery for these images may take years to acquire via clinical practice alone, this learning system can potentially help achieve this in just a few hours.

Medical images represent a core portion of the information clinicians utilize to render diagnostic and treatment decisions. Fundamentally, viewing a medical image involves two basic processes - visually inspecting the image (visual perception) and rendering an interpretation (cognition). The interpretation is often followed by a recommendation. The likelihood of error in the interpretation of medical images is unfortunately not negligible. Errors occur and patients' lives are impacted. Thus we need to understand how clinicians interact with the information in an image during the interpretation process. We also need to understand how clinicians develop expertise throughout their careers and why some people are better at interpreting medical images than others. If we can better understand how expertise develops, perhaps we can develop better training programs, incorporate more effective ways of teaching image interpretation into the medical school and residency curriculums, and create new tools that would enhance and perhaps speed up the learning process. With improved understanding we can also develop ways to further improve decision-making in general and at every level of the medical imaging profession, thus improving patient care. The science of medical image perception is dedicated to understanding and improving the clinical interpretation process.

Medicube: A 3D medical imaging architecture

Modern medicine is reliant on various medical imaging technologies for non-invasively observing patients’ anatomy. However, the interpretation of medical images can be highly subjective and dependent on the expertise of clinicians. Moreover, some potentially useful quantitative information in medical images, especially that which is not visible to the naked eye, is often ignored during clinical practice. In contrast, radiomics performs high-throughput feature extraction from medical images, which enables quantitative analysis of medical images and prediction of various clinical endpoints. Studies have reported that radiomics exhibits promising performance in diagnosis and predicting treatment responses and prognosis, demonstrating its potential to be a non-invasive auxiliary tool for personalized medicine. However, radiomics remains in a developmental phase as numerous technical challenges have yet to be solved, especially in feature engineering and statistical modeling. In this review, we introduce the current utility of radiomics by summarizing research on its application in the diagnosis, prognosis, and prediction of treatment responses in patients with cancer. We focus on machine learning approaches, for feature extraction and selection during feature engineering and for imbalanced datasets and multi-modality fusion during statistical modeling. Furthermore, we introduce the stability, reproducibility, and interpretability of features, and the generalizability and interpretability of models. Finally, we offer possible solutions to current challenges in radiomics research.

Interpretation of medical images is a socially critical application of visual search. The prevalence of targets in medical image search tasks varies widely. While a screening mammographer will see one cancer for every 300 cases, a pathologist interpreting biopsies will see mostly positive cases. How does prevalence affect visual search? Do low and high prevalence exert similar effects? Can findings from the basic vision science literature be generalized to the medical image perception context? I have previously reported a meta-analysis results show that basic cognitive psychology and applied medical image perception studies converge on an account of the low prevalence effect: sensitivity (d') is constant with prevalence, but criterion (c) shifts to a more conservative level at low prevalence (Horowitz, 2017). Here I show that these results generalize to the high prevalence effect: sensitivity does not change at high prevalence, but criterion shifts to a more liberal level. I synthesize findings from both vision science and medical image perception, I will identify practical suggestions for medical image professionals and theoretical insights for vision scientists.

Computer Aided Diagnosis (CADx) systems are designed to assist doctors in medical image interpretation. However, a CADx is often thought of as a black box whose diagnostic decision is not intelligible to a radiologist. Therefore, a system that uses semantic image interpretation, and mimics human image analysis, has clear benefits. We propose a new method for automatic textual description of medical image findings, such as lesions in medical images. The method performs joint estimation of semantic features of lesions from image measurements. We formalize this problem as learning to map a set of diverse medical image measurements to a set of semantic descriptor values. We use a structured learning framework to model individual semantic descriptors and their relationships. The parameters of the model are efficiently learned using the Structured Support Vector Machine (SSVM). The proposed approach generates radiological lexicon descriptors used to make a diagnosis of various diseases. This can help radiologists easily understand a diagnosis recommendation made by an automatic system, such as CADx. We apply the proposed method to publicly available and to proprietary breast and brain imaging datasets, and show that our method generates more accurate descriptions, as compared to other alternative approaches.

Investigations on nanomedicine involve conventional two dimensional (2D) imaging techniques for observing the nanoparticle internalization at a single time point where various phases of internalization can be overlooked. In contrast, three dimensional (3D) imaging of fluorescent nanoparticles with anticancer potential can be used for obtaining the time course of cellular retention of particles, and cells can be followed for days. This article demonstrates the application of laser scanning confocal microscopy to quantify poly‐l‐lysine coated fluorescent ZnO nanoparticle retention and reactive oxygen species (ROS) generation using volumetric imaging. Synthesis of these particles allows monitoring of ROS formation, internalization, and cytotoxicity using the same imaging platform that offers an advantage over measurement using various instruments. PLL‐coated ZnO particles' ability to induce a significant reduction in cell‐viability suggests its potential as a therapeutic agent. The proposed framework opens up a new avenue for investigating mechanistic aspects of ZnO adsorption and the evaluation of therapeutic efficiency.