Affective Video Content Analyses by Using Cross-Modal Embedding Learning Features

We investigated the application of multiresolution global and local texture features to reduce false-positive detection in a computerized mass detection program. One hundred and sixty-eight digitized mammograms were randomly and equally divided into training and test groups. From these mammograms, two datasets were formed. The first dataset (manual) contained four regions of interest (ROIs) selected manually from each of the mammograms. One of the four ROIs contained a biopsy-proven mass and the other three contained normal parenchyma, including dense, mixed dense/fatty, and fatty tissues. The second dataset (hybrid) contained the manually extracted mass ROIs, along with normal tissue ROIs extracted by an automated Density-Weighted Contrast Enhancement (DWCE) algorithm as false-positive detections. A wavelet transform was used to decompose an ROI into several scales. Global texture features were derived from the low-pass coefficients in the wavelet transformed images. Local texture features were calculated from the suspicious object and the peripheral subregions. Linear discriminant models using effective features selected from the global, local, or combined feature spaces were established to maximize the separation between masses and normal tissue. Receiver Operating Characteristic (ROC) analysis was conducted to evaluate the classifier performance. The classification accuracy using global features were comparable to that using local features. With both global and local features, the average area, Az, under the test ROC curve, reached 0.92 for the manual dataset and 0.96 for the hybrid dataset, demonstrating statistically significant improvement over those obtained with global or local features alone. The results indicated the effectiveness of the combined global and local features in the classification of masses and normal tissue for false-positive reduction.

In this study, we investigated how shapes are classified based on local and global features by four representative convolutional neural networks (CNNs), i.e., AlexNet, VGG, ResNet and Inception. While the local features are based on simple components, such as orientation of line segment, the global features are based on the whole object, such as whether an object has a hole. For example, solid triangles and solid squares are differentiated by local features, solid circles and rings are differentiated by a global feature. Two sets of experiments were performed in this research. In the first experiment, we examined how the four CNNs pre-trained on ImageNet (with transfer learning) learned to differentiate the regular shapes (equilateral-triangles, squares, circles and rings). Our results showed that the pre-trained CNNs exhibited faster learning rates in the tasks discriminating the local features than in the tasks discriminating the global feature. However, the transfer learning of discriminating the global feature in regular shapes were better generalized to irregular shapes than the transfer learning of discriminating local features. In the second experiment, the CNNs were trained from scratch (with random weights initialization) to discriminate local and global features in regular and irregular shapes. Different from the transfer learning, the CNNs exhibited faster learning rates in discriminating the global feature than the local features. Similar to transfer learning, the CNNs exhibited excellent learning generalization to discriminating the global feature of irregular shapes, but poor learning generalization to discriminating the local features in the irregular shapes. The overarching goal of this research is to create a paradigm and benchmark to directly compare how the CNNs and primate visual systems process geometrical invariants. In contrast to the ImageNet approach which employs natural images to train CNNs, we employed the “ShapeNet” approach which features geometrical shapes with well-defined properties. The ShapeNet approach will not only help elucidate the strengths and limitations of CNN computation, but also provide insights into visual information processing in the primates.

This study analyzes the effectiveness of the global (the whole face) and local (regions of eyes, nose, and mouth) features for face recognition. Features describing human faces are encoded in local ternary patterns. The two-class support vector machine is used as the supervised learning algorithm for training recognition models. In the recognition process, recognition modes based on the global features and local features are cascaded. For identifying a face image, the local features are used iteratively for filtering out candidates that can not be clearly identified by the global features, until the one with highest possibility is concluded. The experimental results show that cascading the recognition models of global and local features obtains better classification accuracy than the single classification process.

Sentence similarity is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. However, a variety of linguistic expressions and ambiguities of words in sentences make it difficult to measure sentence similarity. Many studies show that using local features or global features of a sentence will produce satisfactory sentence representations that can be utilized to measure sentence similarity. Local features reflect the relationships of adjacent words for each sentence and the sequence information of a sentence are usually expressed by global features. However, local features lack abilities to capture sequence information while a small amount of extracted global features is not enough to produce sentence representations with good qualities. In this paper, we propose A Hybrid Model combining Local and Global features into a Siamese Network (HM-LGSN) for sentence similarity calculation. We first propose a new convolution neural network architecture called group convolution neural network to extract the most representative local features (or word semantic features). Then we combine these new features with pre-trained embeddings of words as input to the Bidirectional Gated Recurrent Units to extract global features of sentences. Finally, we select the global features to form sentence representations and calculate sentence similarity through Manhattan distance. The experimental results on SICK, MSRVID, STS-B datasets show that the accuracy of our proposed model is significantly improved by combining local features and global features.

In the literature of psychophysics and neurophysiology, many studies have shown that both global and local features are crucial for face representation and recognition. This paper proposes a novel face recognition method which combines both global and local discriminative features. In this method, global features are extracted from whole face images by Fourier transform and local features are extracted from some spatially partitioned image patches by Gabor wavelet transform. After this, multiple classifiers are obtained by applying Fisher Discriminant Analysis on global Fourier features and local patches of Gabor features. All these classifiers are combined to form a hierarchical ensemble by sum rule. We evaluated the proposed method using Face Recognition Grand Challenge (FRGC) experimental protocols and database known as the largest data sets available. Experimental results on FRGC version 2.0 data set have shown that the proposed method achieves a verification rate of 86%, while the best reported was 76%.

This mixed methods study sought to extend what is known about preservice teachers’ [PSTs’] enactment of the core practice of reading and responding to students’ writing by examining their reading of elementary students’ writing at the start and end of writing-focused methods coursework. PSTs’ reading of students’ writing involves analysis of children’s writing strengths and needs and undergirds their teaching of writing, influencing their feedback and instructional decision-making. Participants included 115 elementary PSTs. Their identifications of local and global text features on a beginning- and end-of-semester assessment task were examined. Overall, PSTs showcased differences in their reading of children’s writing at the two time points, displaying a lessened focus on local features (e.g., spelling, punctuation) and a greater focus on global features (e.g., content, structure) at the end of the semester. This trend was especially apparent in their identification of children’s instructional needs. Moreover, PSTs’ reading of children’s informative and persuasive writing tended to be similar to their reading of children’s narrative writing at both time points. The results offer insight into the text features PSTs make available for use in their subsequent responding and suggest the importance of highlighting global text features among children’s instructional needs.

During the continual transitions from lab research to real-world applications of vision-based algorithms, there are significant challenges, e.g. the robustness to adapt to complex environments and the high demands of multi-task, multi-modal learning models. More specifically on visual simultaneous localization and mapping (visual SLAM) for mobile robots, two notable limitations are: (i) the drift issue during pose estimation, especially in dynamic environments, makes the positioning system unstable; (ii) the additional loop detection takes many computational resources for image retrieval and global feature geometry checking. This thesis explores the approaches of visual localization on unmanned ground vehicles by solving those limitations. The conventional feature extraction and matching pipeline for vision-based tasks involve three steps: (i) local feature detection and description, (ii) feature matching, and (iii) outlier rejection. The first contribution of the thesis is adding a feature selection and anticipation stage to reduce tracking drift. We explore the noise model of image features to select a subset of all the observed image features with the best ”contribution” during data association and pose estimation across multiple frames. Conventional SLAM algorithms take a strong assumption of scene rigidity, which limits the application under challenging environments. The second part of the thesis addresses the tough issue in dynamic environments with moving objects. We presented GMC, namely the motion clustering approach, a lightweight dynamic object filtering method. It can distinguish moving objects from static landmarks. Based on the theory of motion coherence within a particular image area, GMC could segment dynamic objects in 3D space. We can provide an efficient and robust correspondence algorithm that can extract dynamic objects from a static background with the method. In this way, we propose a dynamic SLAM system that is real-time and free from expensive GPU processors. In contrast to GMC, the thesis’s third part turns to an object-aware learning-based model for more general dynamic scenarios. We use object detection and tracking as points, lines, planes, etc. We utilize semantic information and extract sparse image features simultaneously to keep track of dynamic objects. The static background and different dynamic objects are jointly optimized in a newly developed bundle adjustment sliding window. The estimated 3D bounding boxes can provide more robust camera tracking and better scene understanding, and better map merging. The fourth part of the thesis leverages the emerging feature learning framework. It proposes a unified self-supervised model called LGDNet to generate both global and local image feature descriptors end-to-end. Global feature descriptors embed the whole image into a compact representation, leading to easier scene comparison. On the other hand, local features focus more on the local region similarities of some salient parts for structure from motion. Our proposed method can directly extract features together with descriptors that encode both local maximum responses and global context information, avoiding duplicate calculations based on different feature extraction criteria.

Automatic target recognition (ATR) of synthetic aperture radar (SAR) images is performed on either global or local features. The global features can be extracted and classified with high efficiency. However, they lack the reasoning capability thus can hardly work well under the extended operation conditions (EOCs). The local features are often more difficult to extract and classify but they provide reasoning capability for EOC target recognition. To combine the efficiency and robustness in an ATR system, a hierarchical fusion of the global and local features is proposed for SAR ATR in this paper. As the global features, the random projection features can be efficiently extracted and effectively classified by sparse representation-based classification (SRC). The physically relevant local descriptors, i.e., attributed scattering centers (ASCs), are employed for local reasoning to handle various EOCs like noise corruption, resolution variance, and partial occlusion. A one-to-one correspondence between the test and template ASC sets is built by the Hungarian algorithm. Then, the local reasoning is performed by evaluating individual matched pairs as well as the false alarms and missing alarms. For the test image to be recognized, it is first classified by the global classifier, i.e., SRC. Once a reliable decision is made, the whole recognition process terminates. When the decision is not reliable enough, it is passed to the local classifier, where a further classification by ASC matching is carried out. Therefore, by the hierarchical fusion strategy, the efficiency of global features and the robustness of local descriptors to various EOCs can be maintained jointly in the ATR system. Extensive experiments on the moving and stationary target acquisition and recognition data set demonstrate that the proposed method achieves superior effectiveness and robustness under both SOC and typical EOCs, i.e., noise corruption, resolution variance, and partial occlusion, compared with some other SAR ATR methods.

Decision letter: Visual and motor signatures of locomotion dynamically shape a population code for feature detection in Drosophila

Content based image retrieval systems (CBIR) are used to search images on the basis of their visual content in a huge image database. This approach uses a multi-clustering technique with a multi-searching process. In addition it integrate, Mean SIFT (Scale Invariant Feature Transform) descriptor as local feature and HSV(hue, saturation, value) histogram as global feature. Our proposition aims to a maximum separation of the execution to gain time and keep the performance of each descriptor. Local and global features are combined for more relevant results. Getting several views to relevant results to cover the subjectivity in displaying results. This article, detailed our proposed method with a comparison with the FIRE (Flexible Image Retrieval) Engine and LIRE (Lucene Image Retrieval). The results demonstrate the feasibility and relevance of our proposition.

Surveillance cameras can be deployed in various environments where lighting conditions are constantly changing. However, due to the limited dynamic range of current image sensors, the captured images are only low dynamic range images that usually suffer from over-exposure and under-exposure situations where important details are lost. Therefore, it is critical to recover the lost details of such images in order to improve visual experience for observers and performance for possible computer vision processing. In this paper, we propose a reformulated Laplacian pyramid and a convolutional neural network (CNN) model to enhance and recover the lost detail of a degraded image. The reformulated Laplacian first decomposes the image into two sub-images that contain global and local image features, respectively. The global features and local features are processed by the proposed CNN model to manipulate the global luminance terrain and enhance local details. The final image is obtained by reconstructing the CNN generated local and global features. Various experiments have been conducted. The results demonstrate that the proposed model outperforms the state-of-the-art methods.

Human perception of the face involves the observation of both coarse (global) and detailed (local) features of the face to identify and categorize a person. Face categorization involves finding common visual cues, such as gender, race and age, which could be used as a precursor to a face recognition system to improve recognition rates. In this paper, we investigate the fusion of both global and local features for gender classification. Global features are obtained using the principal component analysis (PCA) and discrete cosine transformation (DCT) approaches. A spatial local binary pattern (LBP) approach augmented with a two-dimensional DCT approach has been used to find the local features. The performance of the proposed approach has been investigated through extensive experiments performed on FERET database. The proposed approach gives a recognition accuracy of 98.16% on FERET database. Comparisons with some of the existing techniques have shown a marked reduction in number of features used per image to produce results more efficiently and without loss of accuracy for gender classification.

In the harsh underground environment, the transmission gear in the rocker arm of the shearer is susceptible to impact load, frequently leading to failures that result in economic losses or casualties. Therefore, real-time and accurate gear fault diagnosis is crucial. Existing diagnostic models struggle with insufficient feature extraction and low diagnostic accuracy when addressing nonlinear, nonsmooth signals with strong noise in this environment. Therefore, an innovative parallel integration model of Swin Transformer and multiscale convolution is proposed to extract both global and local features under different fault modes and noise conditions. A spatial interaction block (SIB) is introduced into the Swin Transformer branch to enhance global information acquisition. In multiscale convolutional neural network (CNN) branches, depthwise separable convolutions with different expansion coefficients are utilized for multiscale extraction of local features to reduce computational complexity. Considering the differences between global and local features, a feature interaction module (FIM) is added between the two branches to realize the interaction between local and global features, considerably improving the model diagnosis performance. Furthermore, to meet the high real-time diagnosis requirements, a cloud-edge collaboration (CEC) framework is designed to realize the timely alarm of gear faults through the efficient collaboration between edge nodes and the industrial cloud platform. Using the vibration data collected from the rocker arm loading test bed of Taizhong Coal Machinery and the gear data provided by Southeast University for validation, the results show that the diagnostic precision of the proposed method can reach up to 99.55%. Even when the signal-to-noise ratio (SNR) is −2 db, the accuracy is still 93.67%, which is superior to other advanced comparison models, and significantly reduces the completion time of fault diagnosis tasks, and has strong anti-noise and real-time diagnosis capabilities.

One of the main issues of face recognition is to extract features from face images, which include both local and global features. We present a novel method to perform feature fusion at the feature level. First, global features are extracted by principal component analysis (PCA), while local features are obtained by active appearance model (AAM) and Gabor wavelet transform (GWT). Second, two types of features are fused by weighted concatenation. Finally, Euclidean and feature distances of fused features are applied to carry out a nearest neighbor classifier. The method is evaluated by the recognition rates and computation cost over two face image databases [AR (created by A. Martinez and R. Benavente) and SJTU-IPPR (Shanghai JiaoTong University-Institute of Image Processing and Pattern Recognition)]. Compared with PCA and elastic bunch graph matching (EBGM), the presented method is more effective. Though the recognition rate of the presented method is not as good as nonlinear feature combination (NFC), low computation cost is its superiority. In addition, experimental results show that the novel method is robust to variations over time, expression, illumination, and pose to a certain extent.

B-cell epitopes (BCEs) are a set of specific sites on the surface of an antigen that binds to an antibody produced by B-cell. The recognition of BCEs is a major challenge for drug design and vaccines development. Compared with experimental methods, computational approaches have strong potential for BCEs prediction at much lower cost. Moreover, most of the currently methods focus on using local information around target residue without taking the global information of the whole antigen sequence into consideration. We propose a novel deep leaning method through combing local features and global features for BCEs prediction. In our model, two parallel modules are built to extract local and global features from the antigen separately. For local features, we use Graph Convolutional Networks (GCNs) to capture information of spatial neighbors of a target residue. For global features, Attention-Based Bidirectional Long Short-Term Memory (Att-BLSTM) networks are applied to extract information from the whole antigen sequence. Then the local and global features are combined to predict BCEs. The experiments show that the proposed method achieves superior performance over the state-of-the-art BCEs prediction methods on benchmark datasets. Also, we compare the performance differences between data with or without global features. The experimental results show that global features play an important role in BCEs prediction. Our detailed case study on the BCEs prediction for SARS-Cov-2 receptor binding domain confirms that our method is effective for predicting and clustering true BCEs.