Multi-level feature representations for video semantic concept detection

The influence of architectural design on well-being is increasingly recognised, underscoring the need to understand how humans evaluate their surroundings. This study examines the impact of low-level visual features and high-level architectural elements in images of house facades generated by a generative adversarial network (GAN). We introduce a novel method to create controlled image datasets using the style mixing component of StyleGAN2-ADA, enabling the combination of specific architectural features with styles representing low-level image properties, such as brightness or colour. Our dataset consists of 900 images encompassing five high-level features (house size, garage door visibility, number of windows, entrance door visibility, and roof shape) and 16 low-level visual features. These GAN-generated images were rated in an online experiment by 303 participants on six dimensions: facelikeness, hominess, relaxation, invitingness, safety, and price. Linear mixed-effects models identified significant predictors, including the number of floors as a high-level feature and green pixel percentage, saturation (SD), brightness (SD), and contrast as low-level features. This indicated that houses with two floors and images with an overall higher amount of green pixel, as well as diversity of saturation and brightness and a lower contrast were rated as, e.g., more inviting and safer. The analysis showed that high- and low-level features explained up to 54% of the variance in house price ratings, with high-level features accounting for the larger share and highlighting their relevance for house facade evaluations. This work advances the use of GANs and offers a method to modify visual elements that shape architectural evaluations.

To investigate form-related activity in motion-sensitive cortical areas, we recorded cell responses to animate implied motion in macaque middle temporal (MT) and medial superior temporal (MST) cortex and investigated these areas using fMRI in humans. In the single-cell studies, we compared responses with static images of human or monkey figures walking or running left or right with responses to the same human and monkey figures standing or sitting still. We also investigated whether the view of the animate figure (facing left or right) that elicited the highest response was correlated with the preferred direction for moving random dot patterns. First, figures were presented inside the cell's receptive field. Subsequently, figures were presented at the fovea while a dynamic noise pattern was presented at the cell's receptive field location. The results show that MT neurons did not discriminate between figures on the basis of the implied motion content. Instead, response preferences for implied motion correlated with preferences for low-level visual features such as orientation and size. No correlation was found between the preferred view of figures implying motion and the preferred direction for moving random dot patterns. Similar findings were obtained in a smaller population of MST cortical neurons. Testing human MT+ responses with fMRI further corroborated the notion that low-level stimulus features might explain implied motion activation in human MT+. Together, these results suggest that prior human imaging studies demonstrating animate implied motion processing in area MT+ can be best explained by sensitivity for low-level features rather than sensitivity for the motion implied by animate figures.

We propose a novel unsupervised learning framework for activity perception. To understand activities in complicated scenes from visual data, we propose a hierarchical Bayesian model to connect three elements: low-level visual features, simple "atomic" activities, and multi-agent interactions. Atomic activities are modeled as distributions over low-level visual features, and interactions are modeled as distributions over atomic activities. Our models improve existing language models such as Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) by modeling interactions without supervision. Our data sets are challenging video sequences from crowded traffic scenes with many kinds of activities co-occurring. Our approach provides a summary of typical atomic activities and interactions in the scene. Unusual activities and interactions are found, with natural probabilistic explanations. Our method supports flexible high-level queries on activities and interactions using atomic activities as components.

Scene recognition aims to find a semantic explanation of a scene, i.e., it helps intelligent machines to know where they are. It can be widely applied into various tasks in computer vision and robotics. Most of pioneer methods extracted a set of low-level features and put them into classifier directly to identify scene category. But it has been proved that low-level features do not work well. Currently researchers aim to overcome the semantic gap between the low-level vision features and high-level semantic categories to improve the recognition performance. Therefore, much attention has been put on transforming low-level descriptors into richer intermediate representations. This paper proposed a novel method based on intermediate feature representation to solve the problem of recognizing the semantic category of scene image. This proposed method uses sparse coding on SIFT features and presents a spatial layout sensitive pooling method. The space layout for pooling is based on three rectangles with size of 1∗1,1∗4 and 4∗1 in each image. They are derived from inherent characteristics of the scene images by regularly dividing the image in horizontal and vertical direction. This spatial pooling strategy is easier and it can get optimal representation of scene images. Extreme learning machine (ELM) is used as a classifier. ELM has shown great ability to fit nonlinear classification boundaries. Experimental results have shown that this proposed method not only extracts lower dimension image feature but also outperforms other similar state-of-the-art methods in terms of recognition performance.

Image retrieval is one of the most interesting and fastest growing research areas in all fields. It is an effective and efficient tool for managing large image databases. In most Content-Based Image Retrieval (CBIR) systems, an image is represented by a set of low-level visual features; hence a direct correlation with high-level semantic information will be absent. Therefore, a gap exists between high-level information and low-level features, which is the main reason that hinders the improvement of the image retrieval accuracy. In this work, main focus is on the semantic based image retrieval system using Gray Level Co-occurrence Matrix (GLCM) for texture feature extraction. Based on the texture features, semantic interpretations are given to the extracted textures. The images are retrieved according to user satisfaction and thereby reduce the semantic gap between low level features and high level features.

In content-based image retrieval (CBIR), a set of low-level features are extracted from an image to represent its visual content. Retrieval is performed by image example where a query image is given as input by the user and an appropriate similarity measure is used to find the best matches in the corresponding feature space. This approach suffers from the fact that there is a large discrepancy between the low-level visual features that one can extract from an image and the semantic interpretation of the image’s content that a particular user may have in a given situation. That is, users seek semantic similarity, but we can only provide similarity based on low-level visual features extracted from the raw pixel data, a situation known as the semantic gap. The selection of an appropriate similarity measure is thus an important problem. Since visual content can be represented by different attributes, the combination and importance of each set of features varies according to the user’s semantic intent. Thus, the retrieval strategy should be adaptive so that it can accommodate the preferences of different users. Relevance feedback (RF) learning has been proposed as a technique aimed at reducing the semantic gap. It works by gathering semantic information from user interaction. Based on the user’s feedback on the retrieval results, the retrieval scheme is adjusted. By providing an image similarity measure under human perception, RF learning can be seen as a form of supervised learning that finds relations between high-level semantic interpretations and low-level visual properties. That is, the feedback obtained within a single query session is used to personalize the retrieval strategy and thus enhance retrieval performance. In this chapter we present an overview of CBIR and related work on RF learning. We also present our own previous work on a RF learning-based probabilistic region relevance learning algorithm for automatically estimating the importance of each region in an image based on the user’s semantic intent.

The analytic hierarchy process (AHP) is a method to help people make a complex decision by analyzing and synthesizing multiple criteria for the decision objective in a hierarchy. We adopt the AHP as a new strategy for the semantic concept detection (SCD) in video so that multiple factors involved in SCD, including multiple modalities and relating concepts, can be hierarchically analyzed and synthesized. In this paper, we first explain, by an example, why and how the SCD problem can be analyzed by the AHP. Then, following the idea of the AHP, we develop a new rank aggregation (RA) method, called AHP-RA. Experimental results of RA for SCD in video show the effectiveness of this method.

Infrared and visible image fusion scheme based on NSCT and low-level visual features

Semantic guidance of eye movements in real-world scenes

In content based image retrieval (CIBR), low-level visual features are usually extracted from the highest quality and resolutions of visual contents. When content is scalable coded, their bit streams can be adapted at various nodes in multimedia usage chains to cater the variations in network bandwidths, display device resolutions and resources and usage preferences by just discarding insignificant resolution-quality layers. This can result in the existence of different version of the same content with dissimilar low-level visual features. Therefore, mapping of low level visual descriptors into content resolution- quality spaces is important in order to obtain low-level visual features that are resilient to content adaptations. A new scalable domain feature extraction using the compression modes and decisions is presented and its CBIR performance is evaluated. The proposed scheme outperforms MPEG-7 visual descriptors in both the original image and scaled resolution-quality space domains.

Video semantic concept detection is considered as an important research problem by the multimedia industry in recent years. Classification is the most accepted method used for concept detection, where, the output of the classification system is interpreted as semantic concepts. These concepts can be employed for automatic indexing, searching and retrieval of video objects. However, employed features have high dimensions and thus, concept detection with the existing classifiers experiences high computation complexity. In this paper, a new approach is proposed to reduce the classification complexity and the required time for learning and classification by choosing the most important features. For this purpose genetic algorithms are employed as a feature selector. Simulation results illustrate improvements in the behavior of the classifier.

In order to improve the performance of content-based image retrieval (CBIR) systems, the 'semantic gap' between the low-level visual features and the high-level semantic features attracts more and more research interest. We propose an approach to describe and to extract the fuzzy color semantics. According to human color perception model, we utilize the linguistic variable to describe the image color semantics, so it becomes possible to depict the image in linguistic expression such as mostly red. Furthermore, we apply the feedforward neural network to model the vagueness of human color perception and to extract the fuzzy semantic feature vector. Our experiments show that the color semantic features have good accordance with the human perception, and also have good retrieval performance. In some extent, our approach shows the potential to reduce the semantic gap in CBIR.

Action recognition using spatial-optical data organization and sequential learning framework

Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, we propose a novel BIQA model for in-the-wild images by addressing two critical problems in this field: how to learn better quality-aware feature representation, and how to solve the problem of insufficient training samples in terms of their content and distortion diversity. Considering that perceptual visual quality is affected by both low-level visual features (e.g. distortions) and high-level semantic information (e.g. content), we first propose a staircase structure to hierarchically integrate the features from intermediate layers into the final feature representation, which enables the model to make full use of visual information from low-level to high-level. Then an iterative mixed database training (IMDT) strategy is proposed to train the BIQA model on multiple databases simultaneously, so the model can benefit from the increase in both training samples and image content and distortion diversity and can learn a more general feature representation. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model shows an excellent performance in the cross-database evaluation experiments, which further demonstrates that the learned feature representation is robust to images with diverse distortions and content. The code is available at https://github.com/sunwei925/StairIQA.

The accurate classification of remote sensing (RS) images can quickly identify various geographical features, which is important for planning, utilizing, and protecting natural resources. Recently, the visual Mamba model, as an extension of the vision transformer (ViT), is attracting widespread attention due to its global receptive field and linear complexity. However, the self-attention mechanism of visual transformers can lead to feature collapse in the deep layers, resulting in the disappearance of low-level visual features. In RS images, low-level features, and especially luminance gradient features, can help discern object boundaries and contour information. This is beneficial for the accurate classification of images but has not been fully leveraged. To make full use of contour information and explore the impact of using handcrafted low-level features on the deep layers of the model, in this study, a contour-enhanced Mamba model based on Vision Mamba (VMamba), is proposed, named G-VMamba. The core novelty of G-VMamba lies in its contour enhancement module (ConEM). First, two separate paths are used to extract adaptive luminance gradients and multidimensional convolutional features at each network layer. Subsequently, the features are combined to impose the constraints of low-level features onto the deeper networks. RS image classification experiments were conducted to evaluate the model’s performance, and the results demonstrate the superior performance of G-VMamba in classification tasks. An analysis of class activation maps (CAMs) across different categories shows that G-VMamba focuses more on color (or luminance) change significantly regions in images than models like VMamba, highlighting the efficacy of contour enhancement. The code will be available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/yanliyue/Contour-enhanced-Visual-State-Space-Model</uri>.