Exploring Rich Semantics for Open-Set Action Recognition

In this work, we focus on the problem of news citation recommendation. The task aims to recommend news citations for both authors and readers to create and search news references. Due to the sparsity issue of news citations and the engineering difficulty in obtaining information on authors, we focus on content similarity-based methods instead of collaborative filtering-based approaches. In this paper, we explore word embedding (i.e., implicit semantics) and grounded entities (i.e., explicit semantics) to address the variety and ambiguity issues of language. We formulate the problem as a reranking task and integrate different similarity measures under the learning to rank framework. We evaluate our approach on a real-world dataset. The experimental results show the efficacy of our method.

Zhekun Luo, Shalini Ghosh, Devin Guillory, Keizo Kato, Trevor Darrell, Huijuan Xu. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Abstract. Classical and more recently deep computer vision methods are optimized for visible spectrum images, commonly encoded in grayscale or RGB colorspaces acquired from smartphones or cameras. A more uncommon source of images exploited in the remote sensing field are satellite and aerial images. However the development of pattern recognition approaches for these data is relatively recent, mainly due to the limited availability of this type of images, as until recently they were used exclusively for military purposes. Access to aerial imagery, including spectral information, has been increasing mainly due to the low cost of drones, cheapening of imaging satellite launch costs, and novel public datasets. Usually remote sensing applications employ computer vision techniques strictly modeled for classification tasks in closed set scenarios. However, real-world tasks rarely fit into closed set contexts, frequently presenting previously unknown classes, characterizing them as open set scenarios. Focusing on this problem, this is the first paper to study and develop semantic segmentation techniques for open set scenarios applied to remote sensing images. The main contributions of this paper are: 1) a discussion of related works in open set semantic segmentation, showing evidence that these techniques can be adapted for open set remote sensing tasks; 2) the development and evaluation of a novel approach for open set semantic segmentation. Our method yielded competitive results when compared to closed set methods for the same dataset.

In recent years, the synergy between artificial intelligence and turbulence big data has given rise to a new data-driven paradigm in turbulence research. Data-driven turbulence modeling has emerged as one of the forefront directions in fluid mechanics. Most existing studies focus on feature construction, selection, and the development of modeling frameworks, often overlooking the practical deployment and application of trained models. This paper examines the entire process from model construction to real-world deployment, using data-driven turbulence modeling for high Reynolds number flows over complex three-dimensional configurations as a case study. Key stages include data generation, input-output feature construction, model training, model compilation and optimization, deployment, and validation. We successfully implemented the entire workflow in a heterogeneous supercomputing environment and, through mixed programming techniques, integrated the resulting turbulence model into the Platform for Hybrid Engineering Simulation of Flows (PHengLEI) open-source software framework. This allowed for mixed-precision simulations, with the main equations solved in double precision and the turbulence model in half precision. The new computational framework was validated through large-scale parallel numerical simulations on grids with tens of millions of elements for three-dimensional complex configurations. The results highlight the efficiency of our model deployment, with overall computational efficiency improving by 13.35% and the turbulence model’s solution speed increasing by approximately 3.9 times. The accuracy of the computations was also confirmed, with the average relative error in the lift and drag coefficients calculated by the data-driven turbulence model within 3%. Across various computing nodes, the relative error in the computed aerodynamic coefficients remained within 1%, demonstrating the framework’s scalability. Notably, our contributions have been incorporated as a case study in the latest PHengLEI open-source project5 5 https://forge.osredm.com/PHengLEI/PHengLEI-TestCases/tree/master/Y02_ThreeD_M6_Unstruct_Branch_Ascend. .

Integrating commonsense knowledge with GPT embeddings for emotion classification☆

Activity modelling plays a critical role in activity recognition and assistance in smart home based assisted living. Ontology-based activity modelling is able to leverage domain knowledge and heuristics to create Activities of Daily Living (ADL) models with rich semantics. However, they suffer from incompleteness, inflexibility, and lack of adaptation. In this paper, we propose a novel approach for learning and evolving activity models. The approach uses predefined ADL ontologies to identify activities from sensor activation streams. We develop algorithms that analyze logs of activity data to discover new activities as well as the conditions for evolving the seed ADL ontologies. We illustrate our approach through a scenario that shows how ADL models can be evolved to accommodate new ADL activities and preferences of individual smart home's inhabitants.

All software systems execute within an environment or context. Reasoning about the correct behavior of such systems is a ternary relation linking the requirements, system and context models. Formal methods are concerned with providing tool (automated) support for the synthesis and analysis of such models. These methods have quite successfully focused on binary relationships, for example: validation of a formal model against an informal one, verification of one formal model against another formal model, generation of code from a design, and generation of tests from requirements. The contexts of the systems in these cases are treated as second-class citizens: in general, the modelling is implicit and usually distributed between the requirements model and the system model. This paper is concerned with the explicit modelling of contexts as first-class citizens and illustrates concepts related to implicit and explicit semantics on an example using the Event B language.KeywordsVerificationmodellingContextsDomains

Visual representation learning plays a key role in solving medical computer vision tasks. Recent advances in the literature often rely on vision-language models aiming to learn the representation of medical images from the supervision of paired captions in a label-free manner. The training of such models is however very data/time intensive and the alignment strategies involved in the contrastive loss functions may not capture the full richness of information carried by inter-data relationships. We assume here that considering expert knowledge from the medical domain can provide solutions to these problems during model optimization. To this end, we propose a novel knowledge-augmented vision-language contrastive representation learning framework consisting of the following steps: (1) Modeling the hierarchical relationships between various medical concepts using expert knowledge and medical images in a dataset through a knowledge graph, followed by translating each node into a knowledge embedding; And (2) integrating knowledge embeddings into a vision-language contrastive learning framework, either by introducing an additional alignment loss between visual and knowledge embeddings or by relaxing binary constraints of vision-language alignment using knowledge embeddings. Our results demonstrate that the proposed solution achieves competitive performances against state-of-the-art approaches for downstream tasks while requiring significantly less training data. Our code is available at https://github.com/Wxy-24/KL-CVR . • Rich semantics can be developed in representation learning by leveraging. • The inter-data relationships can be modelled using external knowledge. • Additional cross-modal alignment leads to better visual understanding. • Fine-grained inter-data similarity can serve as soft targets for cross-modal alignment.

Recognition of Hand Activities for Automatic Generation of Assembly Instructions

The technique of human action recognition can be applied in a number of fields, such as medical rehabilitation, posture in the domain of sports, and emotion perception in counselling. Deep learning action recognition models that consider the continuous changes in the human skeleton can efficiently identify action states, such as rehabilitation processes, posture, and changes in emotion. However, the keypoints of the human skeleton are susceptible to occlusion due to body movement, resulting in a lower confidence level in their positions, which in turn affects the training and execution of human action recognition models. This paper proposes a keypoint correction algorithm based on continuous-time images of temporal variation that can predict and correct the skeleton keypoints with low confidence, based on the previous and subsequent multiple images. The proposed temporal-variation skeleton keypoint correction algorithm can provide accurate feature association training for the locations of the skeleton keypoints. The proposed method is shown to improve the accuracy of human action recognition by at least 30% over three alternative models in the literature: STV-GCN, GCN-NAS, and 2S-AGCN. The proposed algorithm is also improved by at least 30% compared with the above-mentioned methods in the literature, based on an action recognition model with a Gaussian filter.

Human Action Recognition (HAR) has grown significantly because of its many uses, including real-time surveillance and human-computer interaction. Various variations in routine human actions make the recognition process of action more difficult. In this paper, we proposed a novel deep learning architecture known as Inverted Bottleneck Residual with Self-Attention (InBRwSA). The proposed architecture is based on two different modules. In the first module, 6-parallel inverted bottleneck residual blocks are designed, and each block is connected with a skip connection. These blocks aim to learn complex human actions in many convolutional layers. After that, the second module is designed based on the self-attention mechanism. The learned weights of the first module are passed to self-attention, extract the most essential features, and can easily discriminate complex human actions. The proposed architecture is trained on the selected datasets, whereas the hyperparameters are chosen using the particle swarm optimization (PSO) algorithm. The trained model is employed in the testing phase for the feature extraction from the self-attention layer and passed to the shallow wide neural network classifier for the final classification. The HMDB51 and UCF 101 are frequently used as action recognition standard datasets. These datasets are chosen to allow for meaningful comparison with earlier research. UCF101 dataset has a wide range of activity classes, and HMDB51 has varied real-world behaviors. These features test the generalizability and flexibility of the presented model. Moreover, these datasets define the evaluation scope within a particular domain and guarantee relevance to real-world circumstances. The proposed technique is tested on both datasets, and accuracies of 78.80% and 91.80% were achieved, respectively. The ablation study demonstrated that a margin of error value of 70.1338 ± 3.053 (±4.35%) and 82.7813 ± 2.852 (±3.45%) for the confidence level 95%,1.960σx̄ is obtained for HMDB51 and UCF datasets respectively. The training time for the highest accuracy for HDMB51 and UCF101 is 134.09 and 252.10 seconds, respectively. The proposed architecture is compared with several pre-trained deep models and state-of-the-art (SOTA) existing techniques. Based on the results, the proposed architecture outperformed existing techniques.

In this paper, we address an open-set action recognition problem. While the closed-set action recognition classifies test samples into the same classes of actions used for model training, the problem of the open-set action recognition is more challenging because there is a possibility that the trained model has to recognize actions which do not appear in the training set. To address this issue, we propose an action matching network (AMN) that can identify and classify both actions in the training dataset and the actions not included in the set. AMN extracts spatio-temporal representations from the given video clips and constructs an action dictionary using the given samples. Then, AMN classifies an action by computing the similarity based on Euclidean distance or generates a new action class in the constructed dictionary if it is necessary. Experimental results on UCF101 dataset and a large human motion dataset (a.k.a., HMDB dataset) demonstrate the benefits of AMN over the state-of-the-art approaches to open-set action recognition problems.

Human activity recognition based on channel state information (CSI) has received widespread attention in recent years due to its low cost and privacy protection. However, its accuracy can be significantly reduced when most of the current recognition approaches are applied to new users who have not participated in model training. To address this issue, a new passive human activity recognition method based on CSI is proposed in this work. A cycle-consistent generative adversarial network (CycleGAN) is used to map old user CSI activity data to new user CSI activity data. Multiple dense blocks from a dense convolutional network (DenseNet) and a large margin cosine (margin) module are then combined to automatically extract features with representative differences in different activities to identify multiple human activities. According to the results of the model on SignFi data and self-collected data on the eight daily activities of ten volunteers, the activity recognition accuracy of new users is found to reach 95.63% and 96.67%, respectively. The experimental results demonstrate that the proposed method can achieve better performance than the benchmark approaches.

Open set action recognition (OSAR) is a rising research domain that simultaneously identifies all videos from known classes and rejects videos from unknown classes. Existing methods rarely consider the open set data distribution and the spatial-temporal relations of video subsequence. Recently proposed Capsule Network (CapsNet) has shown robust performance in many fields, especially image recognition. However, the current CapsNet has not been directly applied to the OSAR task since it cannot explicitly consider the data distribution of known and unknown classes along with the spatial-temporal relations for videos. This paper proposes the Spatial-Temporal Exclusive Capsule Network (STE-CapsNet) to solve the problems in the OSAR task. The STE-CapsNet designs the temporal-spatial routing mechanism to jointly capture the spatial-temporal information of the videos. Furthermore, the exclusive capsules are learned with dot product routing mechanism to limit the data distribution of closed set and open set and reduce the open set risk for OSAR. Extensive experimental results demonstrate that our proposed approach performs favorably compared with state-of-the-art methods on three standard datasets, which verifies its effectiveness and generalization ability.

The amount of data in our world has been exploding. Hence, integrating, managing and analyzing large amounts of data – i.e. big data - will become a key issue for businesses for better operating and competing in today’s markets. We strongly believe that big data are really useful if used in a smart way. So we introduce the concept of Smart Data that is web and enterprise structured and unstructured big data with explicit and implicit semantics that leverages context to understand intent for better driving business processes and for better and more informed decisions making. This paper proposes a language able to give a representation of big data based on ontologies and a system that implements an approach capable to satisfy the increasing need for efficiency and scalability in semantic data management. The proposed MANTRA Language allows for: (i) representing the semantics of data by knowledge representation constructs; (ii) acquiring big data from disparate heterogeneous sources (e.g. data bases, documents), integrating and managing them; (iii) reasoning and querying with big data. The syntax of the proposed language is partially derived from logic programming, but the semantic is completely revised. The novelty of the language we propose is that a class can be thought of as a flexible collection of structurally heterogeneous individuals that have different properties (schema-less). The language also allows executing efficient querying and reasoning for revealing implicit knowledge. These have been achieved by using a triple-based data persistency model and a scalable No-SQL storage system.