ATTENTION-BASED LSTM NETWORK FOR ACTION RECOGNITION IN SPORTS

Model-based analysis of production systems is one of the main areas in manufacturing research. The foundation of the successful application of these theoretical studies is the availability of valid and high-fidelity mathematical models that are capable of capturing the behavior of job flow in production systems. The modeling process of a production system, however, may require a significant amount of nonstandardized work that can only be done properly by someone with solid training in the area and extensive experience through real case studies. This poses a critical challenge in the effective implementation of these valuable theoretical results in the Industry 4.0 era. To overcome this, we propose a new production systems modeling paradigm inspired by system identification: calculate production system model parameters that best match the standard system performance metrics measured on the factory floor. Specifically, in this article, we consider production lines characterized by the Bernoulli serial line model and develop algorithms that identify model parameters to fit the system throughput and work-in-process. Analytical algorithms are derived to solve this problem in a two-machine line case and then extended to multi-machine lines. The accuracy and computational efficiency of the algorithms are demonstrated through extensive numerical experiments. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —A high-fidelity mathematical model is of critical importance to the implementation of any model-based production system analysis method. Currently, the construction of such models is carried out in an ad hoc manner. The quality of the resulting models may heavily depend on the training, experience, intuition, and personal preference of the modeler. The proposed model parameter identification method focuses on standard key performance indices commonly measured on the factory floor. The advantage is twofold. First, these standard performance metrics are consistently defined regardless of industry, thus avoiding any data-ambiguity issue that may occur when using complex machine/equipment status data. Second, measuring these performance metrics in real time is typically convenient and cost effective, even for manufacturing plants without high-end IT infrastructure, thus making the technology accessible to not only large but also small- and mid-sized manufacturers. Using the algorithms developed in this article, a practitioner can quickly construct a serial production line model and then utilize it to access the rich library of production analysis, design, and control methods available in the literature.

Heart rate variability (HRV) analysis is considered as a preliminary diagnosis method to check the cardiac health of the human heart. The reliability of the HRV analysis system solely depends on the accuracy of the QRS complex detector. Hence, in this paper, an optimally designed digital differentiator (DD) for precise detection of QRS complex is proposed. The proposed DD is designed by using an efficient evolutionary optimization technique called gases Brownian motion optimization (GBMO) algorithm and is used in the preprocessing stage of the QRS detector. In GBMO algorithm, a balanced trade-off is maintained between both the exploration and the exploitation phases to find the global optimum solution. The electrocardiogram signal is preprocessed by using the proposed DD to generate the feature signals corresponding to the R-peaks only. The detection technique utilizes the principle of Hilbert transform and zeroes crossing detection. The proposed approach is verified against all the first channel records of MIT/BIH arrhythmia database by considering the standard QRS detection performance metrics and produces a sensitivity (Se) of 99.92%, positive predictivity (+P) of 99.92%, detection error rate (DER) of 0.1562%, QRS detection rate of 99.92%, accuracy (Acc) of 99.84%, and F score of 0.9992%. With respect to the standard performance metrics, the proposed QRS detector outperforms all the recently reported QRS detection techniques.

Schneider, C, Rothschild, J, and Uthoff, A. Change-of-direction speed assessments and testing procedures in tennis: a systematic review. J Strength Cond Res 37(9): 1888-1895, 2023-Change-of-direction speed (CODS) plays an essential role in tennis match play, and CODS performance is, therefore, commonly assessed and monitored in tennis players. Thus, the aim of this systematic review was to describe test characteristics, performance metrics, test-retest reliability, construct validity, and test outcomes of tests that are used to assess CODS in tennis players. A literature search conducted on PubMed and SPORTDiscus yielded 563 results. After applying the eligibility criteria, a total of 27 studies were included in the present review. Ten unique CODS tests were identified. 505 test variations were most frequently used across all studies, and total time required to complete the test was the predominant performance metric investigated. Intrasession test-retest reliability ranged from "moderate" to "excellent." Intersession test-retest reliability as well as the effects of tennis performance, sex, and age on CODS performance were unclear given the subject demographics and the limited number of studies that investigated these aspects. In conclusion, most studies included CODS tests that exhibit longer COD entry and total distances but similar COD angles to those seen during tennis match play. All CODS tests have at least "moderate" intrasession test-retest reliability. However, to improve CODS assessment methods and to increase our current understanding of CODS performance in tennis players, there is a need to conduct more research on the intersession test-retest reliability, construct validity, and the effects of sex, age, and tennis performance and to investigate other performance metrics that might provide additional insights into CODS (e.g., phase-specific performance variables).

Focus on software

This work aims to temporally localize events that are both audible and visible in video. Previous methods mainly focused on temporal modeling of events with simple fusion of audio and visual features. In natural scenes, a video records not only the events of interest but also ambient acoustic noise and visual background, resulting in redundant information in the raw audio and visual features. Thus, direct fusion of the two features often causes false localization of the events. In this paper, we propose a co-attention model to exploit the spatial and semantic correlations between the audio and visual features, which helps guide the extraction of discriminative features for better event localization. Our assumption is that in an audio-visual event, shared semantic information between audio and visual features exists and can be extracted by attention learning. Specifically, the proposed co-attention model is composed of a co-spatial attention module and a co-semantic attention module that are used to model the spatial and semantic correlations, respectively. The proposed co-attention model can be applied to various event localization tasks, such as cross-modality localization and multimodal event localization. Experiments on the public audio-visual event (AVE) dataset demonstrate that the proposed method achieves state-of-the-art performance by learning spatial and semantic co-attention.

Air Quality Index (AQI) is an indicator of the rank of air pollution that is very vital for the environmental impacts to the public health. In this paper, we propose an association model between visual feature and AQI rank of lifelog data. Visual data (i.e., environmental pictures) and numerical data (i.e., environmental AQI measurements) of lifelog are utilized for the data training stage. The features of the visual data are extracted using a CNN-based method, where the latter are calculated using the standard AQI ranking. The extracted visual features and ranked AQI are combined as the input data for a deep neural network MLP (Multi-layer Perception) to study the association relationship between visual feature and AQI rank. The experimental results show that the proposed method can provide accurate predictions of good or unhealthy AQI ranks from lifelog visual data.

With the development of the Shanghai International Shipping Center, the diversity of vehicle types on the highways and arterial roads near Yangshan port is continually increasing. Within such a container port corridor, large container trucks are primarily utilized for mainline transportation. Their larger size and significant inertia would increase psychological pressure on sedan drivers, and elevate their behavior risk. To investigate the effects of container trucks on drivers’ visual characteristics and driving behavior as well as to predict driving risk, firstly, this research conducted field tests in four scenarios surrounding the port. Visual characteristics and behavior data of sedan drivers were collected. Secondly, a “Visual‐behavior” chain model was established. The relationship between drivers’ visual characteristics, driving behavior, and driving risk was illustrated from the perspective of time‐series behavior patterns. Thirdly, three driving risk prediction models were built with Autoregressive Integrated Moving Average (ARIMA), Long Short‐Term Memory (LSTM), and ARIMA‐LSTM. The results indicate that the ARIMA‐LSTM model shows the most effective prediction performance. This research provides a field‐data comparative analysis of the driving risks influenced by a high proportion of container trucks. The findings contribute to understanding the unique mixed traffic visual environment around large‐scale container ports.

Visual Place Recognition (VPR) is the task of matching current visual imagery from a camera to images stored in a reference map of the environment. While initial VPR systems used simple direct image methods or hand-crafted visual features, recent work has focused on learning more powerful visual features and further improving performance through either some form of sequential matcher / filter or a hierarchical matching process. In both cases the performance of the initial single-image based system is still far from perfect, putting significant pressure on the sequence matching or (in the case of hierarchical systems) pose refinement stages. In this paper we present a novel hybrid system that creates a high performance initial match hypothesis generator using short learnt sequential descriptors, which enable selective control sequential score aggregation using single image learnt descriptors. Sequential descriptors are generated using a temporal convolutional network dubbed SeqNet, encoding short image sequences using 1-D convolutions, which are then matched against the corresponding temporal descriptors from the reference dataset to provide an ordered list of place match hypotheses. We then perform selective sequential score aggregation using shortlisted single image learnt descriptors from a separate pipeline to produce an overall place match hypothesis. Comprehensive experiments on challenging benchmark datasets demonstrate the proposed method outperforming recent state-of-the-art methods using the same amount of sequential information. Source code and supplementary material can be found online. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref> <fn id="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><label>¹</label> [Online]. Available: <uri>https://github.com/oravus/seqNet</uri>. </fn>

The diagnosis and classification of oral diseases are vital to overall health, yet traditional methods depend mainly on visual inspection, often ignoring the valuable context from textual symptom descriptions. While deep learning has advanced medical image analysis, most models are unimodal, processing only visual data. To address this gap, we propose VIT-MEDBERT, a multimodal transformer-based framework designed to classify oral diseases using both visual and textual data. Our approach uses a Vision Transformer (ViT) for processing oral cavity images and MEDBERT, a pre-trained language model, for interpreting symptom descriptions. These two modalities are fused to form a comprehensive representation for classification. A key contribution of this work is the introduction of OralSDv1, a custom-curated, expert-verified multi-modal dataset comprising 1163 paired samples of oral images and their corresponding symptom descriptions across six common oral conditions. Our experiments demonstrate that the VIT-MEDBERT model achieves superior performance compared to state-of-the-art single-modality models, attaining an accuracy of 90.0%, precision of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$9 0. 0 \%$</tex>, recall of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$8 9. 0 \%$</tex>, and an F1-score of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$8 9. 0 \%$</tex>. The results highlight the value of integrating textual symptom data with visual features, which offers enhanced robustness and interpretability in the classification of oral diseases.

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.

A novel spatio-temporal convolutional neural framework for multimodal emotion recognition

Stacked auto-encoders based visual features for speech/music classification

The target of generalized zero-shot learning (GZSL) is to train a model that can classify data samples from both seen categories and unseen categories under the circumstances that only the labeled samples from seen categories are available. In this paper, we propose a GZSL approach based on conditional generative models that adopts a contrastive disentanglement learning framework to disentangle visual information in the latent space. Specifically, our model encodes original and generated visual features into a latent space in which these visual features are disentangled into semantic-related and semantic-unrelated representations. The proposed contrastive learning framework leverages class-level and instance-level supervision, where it not only formulates contrastive loss based on semantic-related information at the instance level, but also exploits semantic-unrelated representations and the corresponding semantic information to form negative sample pairs at the class level to further facilitate disentanglement. Then, GZSL classification is performed by training a supervised model (e.g, softmax classifier) based only on semantic-related representations. The experimental results show that our model achieves state-of-the-art performance on several benchmark datasets, especially for unseen categories. The source code of the proposed model is available at: https://github.com/fwt-team/GZSL.

Face recognition is a biometric technology that involves identifying and verifying individuals based on their facial features. It finds applications in security, surveillance, and user authentication systems. The extraction of facial image features and classifier selection are more challenging to identify with conventional facial recognition technologies, and the recognition rate is lower. The paper present proposed model combined between deep wavelet scattering transform network regarding the extraction of features and machine learning for classification purposes. The proposed model consists four stage: obtaining images, performing pre-processing, extracting features, and then applying classification techniques. using both SoftMax classifier (part of deep learning model) and Support Vector Machine classifier (SVM). We used property collected dataset called MULB dataset. The experimental result shows that SVM classifier provide better results than SoftMax classifier. The results from the experiments conducted on the MULB face database showcased the efficacy of the suggested face recognition approach. The proposed method achieved an outstanding recognition accuracy of 98.29% with SVM classifier and 97.87% with SoftMax classifier.

This paper addresses the problem of geotagging images, i.e., assigning GPS coordinates (i.e., latitude, longitude) to images using image contents. Due to the huge appearance variability of visual features across the world, the images’ contents and their GPS coordinates may be inconsistent. This means images captured from geographically close areas may appear visually distinct; and images with visually similar contents may be taken from geographically distant areas. In this paper, we propose a deep Geo-constrained Auto-encoder (DGAE) to solve these inconsistency problems. Using clustered GPS data and visual data, our approach identifies inconsistent data pairs (i.e., image, GPS). We then propose a novel deep learning framework that can learn similar feature representations for geographically close images and distinct feature representations for geographically distant images. We introduce two new constraints: the same-area constraint and the easy-confusing constraint to our feature learning networks. The former one penalizes images from the same area but with very distinct visual features, and the latter one penalizes images from distant areas but with very similar visual features. A deep architecture is developed to further improve learning discriminative features, which can disambiguate different geometric locations. Our approach is extensively evaluated on a newly-compiled large image geotagging dataset from large-scale community-contributed images with 664,720 images and outperforms comparison approaches.