A fast key frame extraction algorithm and an accurate feature matching method for 3D reconstruction from aerial video

Key frame selection, which aims to choose a small number of frames that well represent the entire video contents, is one of the important research issues in video content analysis. Many existing methods developed so far use some prior information about the contents of video frames based on high-level semantic features, and thus they are applicable only in particular problems. Without any prior information, approaches based on low-level index features such as clusters, correlation, and consecutive frame difference are useful. Key frames found by these approaches are expected to be as much different from each other as possible. Therefore, accurately evaluating the difference among candidate key frames is a key challenge. However, a standard similarity metric such as the squared error and the cross correlation tends to provide redundant key frames. To cope with this problem, we propose to use quadratic mutual information (QMI) as an alternative similarity measure, which allows us to capture higher-order correlation for more than two variables simultaneously. Through experiments, we demonstrate the usefulness of the proposed QMI-based key frame selection method that uses an accurate and computationally efficient least-squares QMI estimator.

Finding enough accurate matching points is key for image stitching. However, the existing state-of-the-art algorithms fail to find enough accurate matching points when facing the challenge where detectable features are not obvious. In this paper, a novel algorithm called CNN-MP is proposed to directly obtain Matching Points between two images using the feature maps extracted by Convolution Neural Network (CNN) and CNN-MP skips the step of detecting keypoints. There are mainly five contributions in CNN-MP: 1) break the conventional image stitching steps without detecting keypoints; 2) a feature map calculation model is built to obtain matching points between the feature maps of two images; 3) establish a position model to map the obtained matching points to the original images; 4) the process of obtaining matching points is accelerated by dividing it into pre-locate and fine-locate; 5) establish the dataset to evaluate CNN-MP in the case where detectable features are not obvious. The experimental results show that the number of accurate matching points obtained by the proposed CNN-MP is at least 1.7 times that of the state-of-the-art algorithms: ORB, SIFT, LIFT and SuperPoint when facing the challenge where detectable features are not obvious. Moreover, CNN-MP also achieves good performance when the input images own significant detectable features.

Accurate 3D shape measurement is crucial for surgical support and alignment in robotic surgery systems. Stereo cameras in laparoscopes offer a potential solution; however, their accuracy in stereo image matching diminishes when the target image has few textures. Although stereo matching with deep learning has gained significant attention, supervised learning requires a large dataset of images with depth annotations, which are scarce for laparoscopes. Thus, there is a strong demand to explore alternative methods for depth reconstruction or annotation for laparoscopes. Active stereo techniques are a promising approach for achieving 3D reconstruction without textures. In this study, a 3D shape reconstruction method is proposed using an ultra-small patterned projector attached to a laparoscopic arm to address these issues. The pattern projector emits a structured light with a grid-like pattern that features node-wise modulation for positional encoding. To scan the target object, multiple images are taken while the projector is in motion, and the relative poses of the projector and a camera are auto-calibrated using a differential rendering technique. In the experiment, the proposed method is evaluated by performing 3D reconstruction using images obtained from a surgical robot and comparing the results with a ground-truth shape obtained from X-rayCT.

Focused ion beam scanning electron microscopy is well suited for volumetric extractions and 3D reconstructions of plant cells and its organelles. The three-dimensional (3D) reconstruction of individual plant cells is an important tool to extract volumetric data of organelles and is necessary to fully understand ultrastructural changes and adaptations of plants to their environment. Methods such as the 3D reconstruction of cells based on light microscopical images often lack the resolution necessary to clearly reconstruct all cell compartments within a cell. The 3D reconstruction of cells through serial sectioning transmission electron microscopy (ssTEM) and focused ion beam scanning electron microscopy (FIB-SEM) are powerful alternatives but not widely used in plant sciences. Here, we present a method for the 3D reconstruction and volumetric extraction of plant cells based on FIB milling and compare the results with 3D reconstructions obtained with ssTEM. When compared to 3D reconstruction based on ssTEM, FIB-SEM delivered similar results. The data extracted in this study demonstrated that tobacco cells were larger (31410 µm3) than pumpkin cells (20697 µm3) and contained more chloroplasts (175 vs. 124), mitochondria (1317 vs. 291) and peroxisomes (745 vs. 79). While individual chloroplasts, mitochondria, peroxisomes were larger in pumpkin plants (25, 53, and 50%, respectively) they covered more total volume in tobacco plants (5390, 395, 374 µm3, respectively) due to their higher number per cell when compared to pumpkin plants (4762, 134, 59 µm3, respectively). While image acquisition with FIB-SEM was automated, software controlled, and less difficult than ssTEM, FIB milling was slower and sections couldnot be revised or re-imaged as they were destroyed by the ion beam. Nevertheless, the results in this study demonstrated that both, FIB-SEM and ssTEM, are powerful tools for the 3D reconstruction of and volumetric extraction from plant cells and that there were large differences in size, number, and organelle composition between pumpkin and tobacco cells.

In modern medicine, 3D reconstruction plays an important role in improving the accuracy of disease detection and treatment especially in the process of robot-assisted surgery. Accordingly, the choice of 3D reconstruction method is of great importance, which provides real-time visualization. This study analyses and compares two types of methods in 3D medical image reconstruction and visualization: Surface Rendering and Volume Rendering. To be more specific, four algorithms including Contour Filter (CF), Marching Cubes (MC), Composite Volume Rendering (CVR) and Texture Mapping Hardware (TMH) were implemented and compared from the perspective of the rendering effect, speed and interactivity, in which the former two algorithms belong to the concept of Surface Rendering and the latter two algorithms are in the range of Volume Rendering. These algorithms were implemented by Microsoft Visual Studio with Visualization Toolkit (VTK) to reconstruct the 3D images of patients CT image data. The mechanism and realization of these algorithms were described, and the merits and shortcomings of these different rendering methods were compared. Based on the comparison results, the 3D reconstruction and rendering method can be determined according to the specific requirements of robot-assisted surgery.

The main objective of this study framework is to help the user to management the interesting video images from the huge storage device. The study can be divided into three main steps is listing as follows: shot boundary detection, key frame selection and evaluation method are designed by analyzing the video content. In this paper we proposed a novel algorithm for shot boundary detection. First, transform the RGB color model into LUV color model according our experimental results. The useful features of each frame are extracted from sequence images by the automatically computation of their color moment-preserving and matching 64-predefined bit planes in order to detect the shot boundary. The excellent shot boundary detection will result in the good key frames selection. Second, the key frame selection is proceed by applying the geometric invariant moments which successful be used in target tracing, pattern recognition, and facial recognition. In our experiments we used five test sequences that belong to five different program categories: movies, advertising, sport game, news, and commercial documentary. The experimental results showed that our method is feasible, effective, and is better than the former approaches.

Based on compressed video segment detection, the decompression step can be omitted, features can be directly extracted from the original video data stream, and the detection rate can be accelerated. This paper analyzes the characteristics of video data, video segmentation and key frame selection. Based on the analysis of the key technologies of video retrieval technology, the compressed video stream extraction method based on video segmentation and key frame is introduced. A method of extracting key frames from MPEG compressed video based on DC coefficients and motion vectors is proposed. Experiments show that the proposed method can reduce the computational burden, and can better represent the video content.

Surface hardness monitoring of laser shock Peening: Acoustic emission and key frame selection

ObjectivesThe primary objective was to determine which of the following angular insertion depth (AID) estimation methods showed the strongest correlation with the postoperative AID (AIDpostop) of a lateral wall (LW) electrode: the Escudé formula, based on the largest distance from the round window (RW) to the LW (distance A); the elliptic-circular approximation (ECA) method, based on both distance A and the perpendicular distance (distance B); and a three-dimensional (3D) reconstruction method. The secondary objective was to assess the impact of using the actual electrode insertion length on the evaluation of AID estimation methods.MethodsThis study included 45 cochleae implanted with the Advanced Bionics SlimJ electrode, with all 16 active electrode contacts positioned intracochlear. Distance A, distance B, and a 3D reconstruction along the LW were evaluated on preoperative magnetic resonance imaging using 3D Slicer. The AIDpostop and insertion length were assessed using cone-beam computed tomography. Two sets of estimated AID values were obtained: assuming a perfect 23-mm insertion length with the reference electrode at the RW (AID23mm); and using the actual insertion length (AIDinsertionlength). Spearman’s rank correlation coefficient was used to assess the association between the estimated AID values and AIDpostop.ResultsA moderately positive correlation was observed between AID23mm and AIDpostop for all three approaches, with no statistically significant differences among them (rs=0.410–0.579). When the analysis was repeated using AIDinsertionlength, the strength of the correlation with AIDpostop increased significantly for all three methods, with the 3D reconstruction method showing a stronger correlation (rs=0.952) than the Escudé formula (rs=0.734) or the ECA method (rs=0.787) (P<0.001).ConclusionThe 3D reconstruction approach has the potential to improve prediction of the AID for LW electrode arrays compared to spiral formulae based on 2D cochlear dimensions. Accurate assessment of the actual insertion length is important for the proper evaluation of AID estimation methods.

Due to video data exponential growth supported by quick advances in multimedia technology, it became difficult for the user to retrieve information from a large videos collection. Key frame extraction consists in providing an abstract of the entire video, containing the most informative frames. In this paper, we present an efficient key frame extraction method. This method is based on local interest points description and repeatability graph clustering via approaching modularity value. To minimize the data to be treated, the process will be applied only on a set candidate frames selected with a windowing rule. Indeed, the first step consists in extracting interest points on a set of candidate frames. After that, we compute repeatability values between each two frames from the candidate set. These values are represented by a repeatability direct graph. The selection of key frames is performed using graph clustering by approaching modularity principle. The experiments performed showed that the proposed method succeeds in extracting key frames that preserve the relevant video content without redundancy.

In this paper we carry out a survey on key frame extraction methods for Video Summary. We also discuss the summary evaluation criteria and compare the approaches based on the method, data set and the results. Video Summary is a process of presenting an abstract of entire video within a short period of time. It aims to provide a compact video representation, while preserving the essential activities of the original video. It is an essential task in video analysis and indexing applications. Most of the video summaries are based on selection of key frames within the shots of a video. Many of them use motion features and few use visual features for extracting the key frames. The video summary quality assessment methods are based more on subjective and less on objective measures. Tong wei Ren et al has provided a framework to assess the quality of the video against a given reference summary using both subjective and objective measures. Ciocca et al used the objective measures for evaluation of summary and most of them evaluate by taking the subjective opinion of experts. A framework for automatic evalution is needed based on both subjective and objective measures without the reference summary.

The high quality three-dimensional (3D) presentation of computed tomography (CT) data has become a common technique since fast imaging systems and 3D software became widely available at the beginning of the 1980s. Unfortunately, the quality of the 3D images produced by standard 3D packages depends not only on several basic scanning parameters like voltage, amperage, slice, and imcrernent, but also on the primary reconstruction algorithm, image separation, and 3D reconstruction and shading method. Therefore, no standard 3D scanning method providing a high quality 3D display combined with an acceptable acquisition and reconstruction time has been established. New volume scanning methods such as the spiral CT offer additional methods to shorten the examination time. There are some problems for 3D imaging in the use of spiral CT. The aim of our study was to compare different scanning — and 3D — reconstruction methods in order to develop standard scanning sequences for routine 3D CT imaging.

Real-time three-dimensional (3D) reconstruction of real-world environments has many significant applications in various fields, including telepresence technology. When depth sensors, such as those from Microsoft's Kinect series, are introduced simultaneously and become widely available, a new generation of telepresence systems can be developed by combining a real-time 3D reconstruction method with these new technologies. This combination enables users to engage with a remote person while remaining in their local area, as well as control remote devices while viewing their 3D virtual representation. There are numerous applications in which having a telepresence experience could be beneficial, including remote collaboration and entertainment, as well as education, advertising, and rehabilitation. The purpose of this systematic literature review is to analyze the recent advances in 3D reconstruction methods for telepresence systems and the significant related work in this field. Next, we determine the input data and the technological device employed to acquire the input data, which will be utilized in the 3D reconstruction process. The methods of 3D reconstruction implemented in the telepresence system as well as the evaluation of the system, have been extracted and assessed from the included studies. Through the analysis and summarization of many dimensions, we discussed the input data used for the 3D reconstruction method, the real-time 3D reconstruction methods implemented in the telepresence system, and how to evaluate the system. We conclude that real-time 3D reconstruction methods for telepresence systems have progressively improved over the years in conjunction with the advancement of machines and devices such as Red Green Blue-Depth (RGB-D) cameras and Graphics Processing Unit (GPU).

In cases of severe cardiac stenosis, the selection of treatment options including angioplasty, stent insertion, and bypass surgery are based on cardiac diagnosis provided by electrocardiograms, stress tests, and angiograms. Although these diagnostic tools are all vital, it is still difficult to determine the true extent to which normal blood flow has been compromised. To date, a clear guideline for aggressive intervention does not exist, as stenosis characteristics can vary greatly between patients. Plaque composition, compressibility, and shape are all factors that can modify a patient's risk of incurring a life‐threatening cardiac event in the near future. Our work aims to establish a streamlined computational model that can provide accurate estimation of dynamic shear stress and tensile strain on a plaque based on a patient's specific geometry. Together with flow information provided by 4D cardiac MRI, information from the computational model will allow a cardiologist to make better informed decisions regarding therapeutic options for patient cardiac health.The goal of this undergraduate research project was to determine the most effective method for 3D reconstruction and segmentation of the heart and coronary arteries from patient‐specific CTA (computed tomography angiography) data. The resulting 3D geometry will be imported into COMSOL, a computational fluid dynamics software, for hemodynamics and stress‐strain analysis.Patients' CTA data was provided by the Cardiac Imaging Department of the St. Francis Hospital (with IRB approval). A set of 2097 images were imported into a 3D reconstruction software, 3D Slicer (http://www.slicer.org), and visualized in sagittal, axial, and coronal views. By scanning through images in each plane, major components of the heart (i.e., atria, ventricles, aorta, coronary arteries) were identified manually. 3D reconstruction and segmentation were accomplished using a seeding procedure in which each part of the heart was carefully selected and identified as an individual component. 3D Slicer then generated a 3D‐reconstructed geometry of the heart from seeds. Proper seeding allowed different components of the heart to be visualized individually in the 3D viewer, enabling isolation of the coronary arteries from the whole heart model.Preliminary results showed a 3D model of the whole heart from CTA data with the left main coronary isolated. Manual segmentation was found to be challenging because coronary artery branches are embedded within the cardiac muscle. It was demonstrated to be efficient to refine seeds for coronary artery identification and segmentation by carefully examining the 3D model and performing several iterations of correction. The coronary artery geometry generated in 3D Slicer is then processed by a Vascular Modeling Tool Kit (VMTK, an open source software) to generate the proper geometries (in .stl format) needed for COMSOL, for hemodynamics modeling and analysis.As methods for reconstruction and segmentation are further refined, computational modeling will be used as a high throughput tool to generate useful information on stress/strain conditions within the coronary arteries, moving toward our goal of streamlined personalized cardiac treatment.This abstract is from the Experimental Biology 2019 Meeting. There is no full text article associated with this abstract published in The FASEB Journal.

Shot reconstruction degree: a novel criterion for key frame selection