Experiments with a spatiotemporal correlator

The explosive growth of online short videos has brought great challenges to the efficient management of video content classification, retrieval, and recommendation. Video features for video management can be extracted from video image frames by various algorithms, and they have been proven to be effective in the video classification of sensor systems. However, frame-by-frame processing of video image frames not only requires huge computing power, but also classification algorithms based on a single modality of video features cannot meet the accuracy requirements in specific scenarios. In response to these concerns, we introduce a short video categorization architecture centered around cross-modal fusion in visual sensor systems which jointly utilizes video features and text features to classify short videos, avoiding processing a large number of image frames during classification. Firstly, the image space is extended to three-dimensional space–time by a self-attention mechanism, and a series of patches are extracted from a single image frame. Each patch is linearly mapped into the embedding layer of the Timesformer network and augmented with positional information to extract video features. Second, the text features of subtitles are extracted through the bidirectional encoder representation from the Transformers (BERT) pre-training model. Finally, cross-modal fusion is performed based on the extracted video and text features, resulting in improved accuracy for short video classification tasks. The outcomes of our experiments showcase a substantial superiority of our introduced classification framework compared to alternative baseline video classification methodologies. This framework can be applied in sensor systems for potential video classification.

Angular velocity is a critical parameter for satellites during on-orbit operation. Current gyro-free angular velocity estimation methods based on star sensors predominantly rely on multiple frames of star images and are effective under relatively small maneuvers of the star sensor. This paper proposes an angular velocity estimation approach utilizing single-frame star image data captured by a high-dynamic star sensor operating under rolling shutter exposure mode. By analyzing the motion patterns of star points in single-frame images, an angular velocity estimation optimization model was established with inter-star angular distances as the objective function. Accurate angular velocity estimation was achieved through the simulated annealing algorithm. To evaluate the star sensor's performance under high-dynamic conditions, three representative operational working conditions were simulated numerically. Experimental results demonstrate: under composite motion with dual-axis angular velocity of 15°/s and angular acceleration of 0.8°/s², the root mean square errors (RMSE) for triaxial angular velocity estimation are 0.0378°/s (X-axis), 0.0365°/s (Y-axis), and 0.1947°/s (Z-axis). Notably, when subjected to sinusoidal/cosinusoidal dynamic angular velocity excitation with amplitude of 15°/s, the maximum comprehensive error of triaxial angular velocity estimation does not exceed 0.103°/s, validating the algorithm's robustness in complex motion environments.

Traditional trajectory analysis methods are based on time-series multi-frame image data, target shape and other information. But they do not work well for single frame or sparse time-series multi-target image data. In this paper, a simple and effective sparse trajectory analysis method, based on extended perception domain and skeleton extraction, is proposed for multitarget single frame image data. First of all, the convex hull region feature of sparse points is extracted. The second step, these points are connected into different lines. Then, the skeleton of each trajectory and the coordinate points that make up the trajectory is extracted and recorded respectively. Finally, all kinds of curves are classified and connected by the extending perception domain. The experimental results show that the method can extract multitarget trajectories well under time series data and target features free.

We present a new approach to image analysis in temporal sequence of images (data cube). Our method is based on high-order statistical moments (skewness and kurtosis) giving interesting information about a dynamic event in the temporal sequence. The moments enable precise determination of the ”turning points” in the temporal sequence of images. The moment’s curves are analyzed by continuous complex Morlet wavelet that leads to the description of quasi-periodic processes in the investigated event as a time sequence of local spectra. These local spectra are compared with Fourier spectrum. We experimentally illustrate the performance on the real data from astronomical observations.KeywordsStatistical momentsFrequency analysisFourier and wavelet transformationsDynamic processes

Interphase fluorescence in situ hybridization (FISH) technology is a potential and promising molecular imaging tool, which can be applied to screen and detect cervical cancer. However, manual FISH detection method is a subjective, tedious, and time-consuming process that results in a large inter-reader variability and possible detection error (in particular for heterogeneous cases). Automatic FISH image analysis aims to potentially improve detection efficiency and also produce more accurate and consistent results. In this preliminary study, a new computerized scheme is developed to automatically segment analyzable interaphase cells and detect FISH signals using digital fluorescence microscopic images acquired from Pap-smear specimens. First, due to the large intensity variations of the acquired interphase cells and overlapping cells, an iterative (multiple) threshold method and a feature-based classifier are applied to detect and segment all potentially analyzable interphase nuclei depicted on a single image frame. Second, a region labeling algorithm followed up a knowledge-based classifier is implemented to identify splitting and diffused FISH signals. Finally, each detected analyzable cell is classified as normal or abnormal based on the automatically counted number of FISH signals. To test the performance of this scheme, an image dataset involving 250 Pap-smear FISH image frames was collected and used in this study. The overall accuracy rate for segmenting analyzable interphase nuclei is 86.6% (360/424). The sensitivity and specificity for classifying abnormal and normal cells are 88.5% and 86.6%, respectively. The overall cell classification agreement rate between our scheme and a cytogeneticist is 86.6%. The testing results demonstrate the feasibility of applying this automated scheme in FISH image analysis.

This paper aims to answer the questions whether human height measurement from video is affected by human motion and how accurately human static height can be measured from a video or from a single image frame that captures a walking subject. In this paper, we present a new approach for measuring human height based on the cross ratio. This approach is performed in parallel with the vanishing point based approach for comparison. The accuracy of each approach is examined by comparing the estimated value to the actual static height value which is measured directly. For the height measurement from video, our analyses show that human height varies significantly during human motion and the highest point in a human stride is the most accurate measurement of the static height. We also model the height variation as a sinusoidal pattern to maximize the accuracy of the estimated height. For the height measurement from a single frame, a correction method is developed in which the length of the human torso with human head and the length of the human leg are separately estimated. We conclude that static human height can be measured accurately, even though the variation of human height is significant during human motion.

Improved Rapid Corner Detection Algorithm in Medical Image

In the infrared small target images with complex backgrounds, there exist various interferences that share similar characteristics with the target (such as building edges). The accurate detection of small targets is crucial in applications involving infrared search and tracking. However, traditional detection methods based on small target feature detection in a single frame image may result in higher error rates due to insufficient features. Therefore, in this paper, we propose an infrared moving object detection method that integrates spatio-temporal information. To address the limitations of single-frame detection, we introduce a temporal sequence of images to suppress false alarms caused by single-frame detection through analyzing motion features within the sequence. Firstly, based on spatial feature detection, we propose a multi-scale layered contrast feature (MLCF) filtering for preliminary target extraction. Secondly, we utilize the spatio-temporal context (STC) as a feature to track the image sequence point by point, obtaining global motion features. Statistical characteristics are calculated to obtain motion vector data that correspond to abnormal motion, enabling the accurate localization of moving targets. Finally, by combining spatial and temporal features, we determine the precise positions of the targets. The effectiveness of our method is evaluated using a real infrared dataset. Through analysis of the experimental results, our approach demonstrates stronger background suppression capabilities and lower false alarm rates compared to other existing methods. Moreover, our detection rate is similar or even superior to these algorithms, providing further evidence of the efficacy of our algorithm.

A system for simultaneously measuring surface normal vectors and BTF (bidirectional texture function) profiles of an object is proposed. This system consists of a correlation camera and a spherical scanning light source. The system can compress BTF measurement in one dimension of incidence direction to a single image frame, as well as measure a profile of the BTF under the scanning light source just from a single image frame. Experimental results on an implemented system demonstrate the feasibility of the proposed system.

A comparative study of two generation partial light intensity imager based on liquid crystal

Endoscope size is a major design constraint that must be managed with the clinical demand for high-quality illumination and imaging. Existing commercial endoscopes most often use an arc lamp to produce bright, incoherent white light, requiring large-area fiber bundles to deliver sufficient illumination power to the sample. Moreover, the power instability of these light sources creates challenges for computer vision applications. We demonstrate an alternative illumination technique using red-green-blue laser light and a data-driven approach to combat the speckle noise that is a byproduct of coherent illumination. We frame the speckle artifact problem as an image-to-image translation task solved using conditional Generative Adversarial Networks (cGANs). To train the network, we acquire images illuminated with a coherent laser diode, with a laser diode source made partially- coherent using a laser speckle reducer, and with an incoherent LED light source as the target domain. We train networks using laser-illuminated endoscopic images of ex-vivo, porcine gastrointestinal tissues, augmented by images of laser-illuminated household and laboratory objects. The network is then benchmarked against state of-the-art optical and image processing speckle reduction methods, achieving an increased peak signal-to-noise ratio (PSNR) of 4.1 db, compared to 0.7 dB using optical speckle reduction, 0.6 dB using median filtering, and 0.5 dB using non-local means. This approach not only allows for endoscopes with smaller, more efficient light sources with extremely short triggering times, but it also enables imaging modalities that require both coherent and incoherent sources, such as combined widefield and speckle ow contrast imaging in a single image frame.

Conventional (Doppler-based) blood flow velocity measurement methods using ultrasound are capable of resolving the axial component (i.e., that aligned with the ultrasound propagation direction) of the blood flow velocity vector. However, these methods are incapable of detecting blood flow in the direction normal to the ultrasound beam. In addition, these methods require repeated pulse-echo interrogation at the same spatial location. A new method has been introduced which estimates the lateral component of blood flow within a single image frame using the observation that the speckle pattern corresponding to blood reflectors (typically red blood cells) stretches (i.e., is smeared) if the blood is moving in the same direction as the electronically-controlled transducer line selection in a 2-D image. The situation is analogous to the observed distortion of a subject photographed with a moving camera. The results of previous research showed a linear relationship between the stretch factor (increase in lateral speckle size) and blood flow velocity. However, errors exist in the estimation when used to measure blood flow velocity. In this paper, the relationship between speckle size and blood flow velocity is investigated further with both simulated flow data and measurements from a blood flow phantom. It can be seen that: 1) when the blood flow velocity is much greater than the scan velocity (spatial rate of A-line acquisition), the velocity will be significantly underestimated because of speckle decorrelation caused by quick blood movement out of the ultrasound beam; 2) modeled flow gradients increase the average estimation error from a range between 1.4% and 4.4%, to a range between 4.4% and 6.8%; and 3) estimation performance in a blood flow phantom with both flow gradients and random motion of scatterers increases the average estimation error to between 6.1% and 7.8%. Initial attempts at a multiple-scan strategy for estimating flow by a least-squares model suggest the possibility of increased accuracy using multiple scan velocities.

Medical images taken from camera based devices (e.g. laparoscope, colposcope, retinoscope, etc) are greatly affected by numerous bright reflection spots (called glare or specular reflections). This may affect the visibility of the abnormal features (if present in the glare locations). We have developed a novel solution to overcome this problem by incorporating a multi-LED lighting solution. This will intelligently and rapidly switch on and off the LED's in a pattern that dynamically and geometrically shifts/shuffles these glare spots back and forth in the image such that every glare-affected area of a single image frame can be reconstructed from a few adjacent time-frame images. We have built the prototype that successfully demonstrates how the glare problem in the medical video/image can be satisfactorily solved, significantly enhancing the accuracy of this vital procedure in the diagnosis of diseases. We achieve 65-95% reduction in specularity on phantom model using the proposed approach.

Abstract. Close-range acquisition of multispectral images is becoming widespread in many applications, such as forestry and agriculture monitoring. However, some challenges still need to be addressed, such as the co-registration of the spectral bands generated with multi-lens cameras. Depth variations can introduce differential parallax effects depending on the lens position, which makes the coregistration more challenging. Therefore, a rigorous registration of this multispectral imagery requires pixel-wise parallax compensation, based on a depth model. This work presents a detailed evaluation of the accuracy of depth estimation with images acquired from a single-frame camera with a very short baseline. The methodology involves accurate camera calibration and three-dimensional reconstruction using Agisoft Metashape software. Field validation was conducted using a high-resolution laser scanner to produce reference data. Results show that even a single image frame, composed of six sub-images from different lenses, enables 3D reconstruction with reasonable (centimetre) accuracy, while a full image set achieves sub-centimetre accuracy. Comparison with a laser scanner reference point cloud showed the spatial consistency of the reconstructed surface. The results demonstrate the viability of reconstruction with short baseline images, aiming for pixel-wise registration.

Image colorization is a process where each pixel in images or videos is given a suitable amount of red, green, and blue (RGB) main color components in order to make it appear more vivid. The process of colorization has significant values while preserving historical documentaries or investigating cosmological photography. However, the colorization of historically significant documentaries requires real witnesses, and a significant amount of human labor is needed to colorize even a single image frame. Recently, deep learning-based approaches have become one of the most reliable techniques to automate challenging computer vision tasks. Various deep learning methods are applied to automate the colorization process. However, we have found that most of the colorization models show poor performance while colorizing Bangladeshi historical documentaries. In our work, we developed ColorNet: a deep learning-based model using an encoder and a pre-trained feature extractor named InceptionV3, also, a fusion layer and a decoder. We have built a new dataset from Bangladeshi historical documentaries. After training our colorization model, it shows a significant result while colorizing Bangladeshi historical documentaries.KeywordsColorizationEncoder networkDecoder networkFusion network