Stacked Memory Network for Video Summarization

Human action recognition in videos is an important task with a broad range of applications. In this study, we improve the performance of recurrent attention convolutional neural network (RACNN) by proposing a novel model, “attention-again”. We consider the nature of video frames as sequences, which will cause the change of regions of interest in the frame, thus we cannot use an attention mechanism similar to that in images. “Attention-again” model is a variant from traditional attention model for recognizing human activities and is embedded in two long short-term memory (LSTM) layers. Different from hierarchal LSTM which change the LSTM structure to combine the hidden states from two LSTM layers, our proposals introduce “attention-again” model to avoid the change of LSTM structure. Furthermore, this model not only learns the relations in each frame, but also obtains the relations among all frames, and these relations instruct the next learning stage. Therefore, our proposed model outperform the baseline and is superior to methods with the same experimental conditions on three benchmark datasets: UCF-11, HMDB-51 and UCF-101. To understand how the model works, we also visualize the region of interest in the frame.

Many human activities consist of physical gestures that tend to be performed in certain sequences. Wearable inertial sensor data have as a consequence been employed to automatically detect human activities, lately predominantly with deep learning methods. This article focuses on the necessity of recurrent layers—more specifically Long Short-Term Memory (LSTM) layers—in common Deep Learning architectures for Human Activity Recognition (HAR). Our experimental pipeline investigates the effects of employing none, one, or two LSTM layers, as well as different layers' sizes, within the popular DeepConvLSTM architecture. We evaluate the architecture's performance on five well-known activity recognition datasets and provide an in-depth analysis of the per-class results, showing trends which type of activities or datasets profit the most from the removal of LSTM layers. For 4 out of 5 datasets, an altered architecture with one LSTM layer produces the best prediction results. In our previous work we already investigated the impact of a 2-layered LSTM when dealing with sequential activity data. Extending upon this, we now propose a metric, rGP, which aims to measure the effectiveness of learned temporal patterns for a dataset and can be used as a decision metric whether to include recurrent layers into a network at all. Even for datasets including activities without explicit temporal processes, the rGP can be high, suggesting that temporal patterns were learned, and consequently convolutional networks are being outperformed by networks including recurrent layers. We conclude this article by putting forward the question to what degree popular HAR datasets contain unwanted temporal dependencies, which if not taken care of, can benefit networks in achieving high benchmark scores and give a false sense of overall generability to a real-world setting.

Over the years, well test analysis or pressure transient analysis (PTA) methods have progressed from straight lines via type curve analysis to pressure derivatives and deconvolution methods. Today, analysis of the log-log (pressure and its derivative) response is the most used method for PTA. Although these methods are widely available through commercial software, they are not fully automated, and human interaction is needed for their application. Furthermore, PTA is described as an inverse problem, whose solution in general is non-unique, and several models (well, reservoir and boundary) can be found applicable to similar pressure-derivative response. This tends to always bring about confusion in choosing the correct model using the conventional approach. This results in multiple iterations that are time consuming and requires constant human interaction. Our approach automates the process of PTA using a Siamese neural network (SNN) architecture comprised of Convolutional neural network (CNN) and Long Short-Term Memory (LSTM) layers. The SNN model is trained on simulated experimental data created using a design of experiments (DOE) approach involving most common 14 interpretation scenarios across well, reservoir, and boundary model types. Across each model type, parameters such as permeability, horizontal well length, skin factor, and distance to the boundary were sampled to compute 560 different pressure derivative responses. SNN is trained using a self-supervised training strategy where the positive and negative pairs are generated from the training data. We use transformations such as compression and expansion to generate positive pairs and negative pairs for the well test model responses. For a given well test model response, similarity scores are computed against the candidates in each model class, and the best match from each class is identified. These matches are then ranked according to the similarity scores to identify optimal candidates. Experimental analysis indicated that the true model class frequently appeared among the top ranked classes. The model achieves an accuracy of 93% for the top one model recommendations when tested on 70 samples from the 14 interpretation scenarios. Prior information on the top ranked probable well test models, significantly reduces the manual effort involved in the analysis. This machine learning (ML) approach can be integrated with any PTA software or function as a standalone application in the interpreter's system. Current work using SNN with LSTM layers can be used to speed up the process of detecting the pressure derivative response explained by a certain combination of well, reservoir and boundary models and produce models with less user interaction. This methodology will facilitate the interpretation engineer in making the model recognition faster for detailed integration with additional information from sources such as geophysics, geology, petrophysics, drilling, and production logging.

Packet loss may affect a wide range of applications that use voice over IP (VoIP), e.g. video conferencing. In this paper, we investigate a time-domain convolutional recurrent network (CRN) for online packet loss concealment. The CRN comprises a convolutional encoder-decoder structure and long short-term memory (LSTM) layers, which have been shown to be suitable for real-time speech enhancement applications. Moreover, we propose lookahead and masked training to further improve the performance of the CRN framework. Experimental results show that the proposed system outperforms a baseline system using only LSTM layers in terms of two objective metrics – perceptual evaluation of speech quality (PESQ) and short-term objective intelligibility (STOI); it also reduces the word error rate (WER) more than the baseline when used as a frontend for speech recognition. The advantage of the proposed system is also verified in a subjective evaluation by the mean opinion score (MOS).

Autism Spectrum Disorder (ASD) is a heterogeneous disorder in children, and the current clinical diagnosis is accomplished using behavioral, cognitive, developmental, and language metrics. These clinical metrics can be imperfect measures as they are subject to high test-retest variability, and are influenced by assessment factors such as environment, social structure, or comorbid disorders. Advances in neuroimaging coupled with machine-learning provides an opportunity to develop methods that are more quantifiable, and reliable than existing clinical techniques. In this paper, we design and develop a deep-learning model that operates on functional magnetic resonance imaging (fMRI) data, and can classify between ASD and neurotypical brains. We introduce a novel strategy to transform time-series data extracted from fMRI signals into Gramian Angular Field (GAF) while locking in the temporal and spatial patterns in the data. Our motivation is to design and develop a novel framework that could encode the time-series, acquired from fMRI data, into images that can be used by deep-learning architectures that have been successful in computer vision. In our proposed framework called ASD-GResTM, we used a Convolutional Neural Network (CNN) to extract useful features from GAF images. We then used a Long Short-Term Memory (LSTM) layer to learn the activities between the regions. Finally, the output representations of the last LSTM layer are applied to a single-layer perceptron (SPL) to get the final classification. Our extensive experimentation demonstrates high accuracy across 4 centers, and outperforms state-of-the-art models on two centers with an increase in the accuracy of 17.58% and 6.7%, respectively as compared to the state of the art. Our model achieved the maximum accuracy of 81.78% with high degree of sensitivity and specificity. All training, validation, and testing was accomplished using openly available ABIDE-I benchmarking dataset.

The analysis and prediction of industrial production plants are of great significance for reducing energy consumption, improving economic efficiency. Therefore, a production prediction method based on bidirectional long short-term memory (Bi-LSTM) is proposed to accurately analyze and evaluate the energy efficiency status of ethylene production plants in industrial processes. Bi-LSTM is a Indirection ally connected network with two layers of long short-term memory (LSTM), it gives full consideration to the relationship between the current data and the data before and after it. Bi-LSTM solves the gradient disappearance or gradient explosion problem in recurrent neural network (RNN), and overcomes the drawback that LSTM only consider the relationship between the current data and its previous data. The comparison results show that the prediction effect of the Bi-LSTM model is superior to that of the back propagation (BP) neural network model, and the average relative error is reduced by 70%, which proves that the Bi-LSTM can effectively raise the accuracy and stability of the ethylene production prediction.

This paper presents a waveform modeling and generation method using hierarchical recurrent neural networks (HRNN) for speech bandwidth extension (BWE). Different from conventional BWE methods that predict spectral parameters for reconstructing wideband speech waveforms, this BWE method models and predicts waveform samples directly without using vocoders. Inspired by SampleRNN, which is an unconditional neural audio generator, the HRNN model represents the distribution of each wideband or high-frequency waveform sample conditioned on the input narrowband waveform samples using a neural network composed of long short-term memory (LSTM) layers and feed-forward layers. The LSTM layers form a hierarchical structure and each layer operates at a specific temporal resolution to efficiently capture long-span dependencies between temporal sequences. Furthermore, additional conditions, such as the bottleneck features derived from narrowband speech using a deep neural network based state classifier, are employed as auxiliary input to further improve the quality of generated wideband speech. The experimental results of comparing several waveform modeling methods show that the HRNN-based method can achieve better speech quality and run-time efficiency than the dilated convolutional neural network based method and the plain sample-level recurrent neural network based method. Our proposed method also outperforms the conventional vocoder-based BWE method using LSTM-RNNs in terms of the subjective quality of the reconstructed wideband speech.

Hybrid blockchain-based spectrum sharing algorithm for dynamic channel selection in cognitive radio

Ocean surface currents (OSC), abiotic features of the environment, are continuous and directed movements of ocean water. Prediction to OSC is of significant interests in physical oceanography. Recently, deep learning technology has shown feasibility in mining the intrinsic change pattern of marine numerical values, such as SSS, SST and SSHA, but not been applied to OSC prediction yet. In this work, a deep learning method, named skipped dual path network (SDPNet), is proposed for OSC prediction. Specifically, SDPNet has a convolutional neural network (CNN) module with a one-dimensional convolution layer, and a recurrent neural network (RNN) module with a dual-path structure. Each path consists of a stack of three layers of long short-term memory (LSTM) and gated recurrent unit (GRU), respectively. As well, a skipped-connection structure is added in the both paths. It aims to gradually mine the intrinsic change pattern contained in OSC time series data itself. Experiments are conducted on the South China Sea OSC data set in REDOS. SDPNet achieves accuracy 80.83%, 75.9%, 74.9%, 73.9%, 72.58%, 70.35%, 69.93% in predicting the coming 7 days OSC values and directions. It performs better than state-of-the-art machine learning methods, include Artificial Neural Network, Simple RNN, LSTM and GRU.

Recently, Recurrent Neural Network (RNN) based approach presents good performance in skeleton based action recognition. Utilizing deep layers of RNN or Long-Short Term Memory (LSTM) has its on weakness when handling long-sequence data because of vanishing or exploding gradients through time. Batch Normalized LSTM (BN-LSTM) is able to give a solution for the problem, with the merit of converging faster in training. In contrast, when deeply layered, BN-LSTM structure shows slow convergence and worse accuracy. In this work, we analyze deep-layered BN-LSTM shows slower convergence in early training phase in training scheme and, finally we propose a deep BN-LSTM structure with auxiliary classifier that is able to converge faster and gives better results at skeleton based action recognition problem. Some experiment are conducted with Penn Action dataset and our own Computer Assembling Video dataset, we verified our proposal shows better results in skeleton based action recognition.

In this paper, we propose a generative recurrent model for human-character interaction. Our model is an encoder-recurrent-decoder network. The recurrent network is composed by multiple layers of long short-term memory (LSTM) and is incorporated with an encoder network and a decoder network before and after the recurrent network. With the proposed model, the virtual character's animation is generated on the fly while it interacts with the human player. The coming animation of the character is automatically generated based on the history motion data of both itself and its opponent. We evaluated our model based on both public motion capture databases and our own recorded motion data. Experimental results demonstrate that the LSTM layers can help the character learn a long history of human dynamics to animate itself. In addition, the encoder---decoder networks can significantly improve the stability of the generated animation. This method can automatically animate a virtual character responding to a human player.

Our objective in this study was to determine if machine learning (ML) can automatically recognize neonatal manipulations, along with associated changes in physiological parameters. A retrospective observational study was carried out in two Neonatal Intensive Care Units (NICUs) between December 2019 to April 2020. Both the video and physiological data (heart rate (HR) and oxygen saturation (SpO2)) were captured during NICU hospitalization. The proposed classification of neonatal manipulations was achieved by a deep learning system consisting of an Inception-v3 convolutional neural network (CNN), followed by transfer learning layers of Long Short-Term Memory (LSTM). Physiological signals prior to manipulations (baseline) were compared to during and after manipulations. The validation of the system was done using the leave-one-out strategy with input of 8 s of video exhibiting manipulation activity. Ten neonates were video recorded during an average length of stay of 24.5 days. Each neonate had an average of 528 manipulations during their NICU hospitalization, with the average duration of performing these manipulations varying from 28.9 s for patting, 45.5 s for a diaper change, and 108.9 s for tube feeding. The accuracy of the system was 95% for training and 85% for the validation dataset. In neonates <32 weeks’ gestation, diaper changes were associated with significant changes in HR and SpO2, and, for neonates ≥32 weeks’ gestation, patting and tube feeding were associated with significant changes in HR. The presented system can classify and document the manipulations with high accuracy. Moreover, the study suggests that manipulations impact physiological parameters.

This paper develops a general framework for learning interpretable data representation via Long Short-Term Memory (LSTM) recurrent neural networks over hierarchal graph structures. Instead of learning LSTM models over the pre-fixed structures, we propose to further learn the intermediate interpretable multi-level graph structures in a progressive and stochastic way from data during the LSTM network optimization. We thus call this model the structure-evolving LSTM. In particular, starting with an initial element-level graph representation where each node is a small data element, the structure-evolving LSTM gradually evolves the multi-level graph representations by stochastically merging the graph nodes with high compatibilities along the stacked LSTM layers. In each LSTM layer, we estimate the compatibility of two connected nodes from their corresponding LSTM gate outputs, which is used to generate a merging probability. The candidate graph structures are accordingly generated where the nodes are grouped into cliques with their merging probabilities. We then produce the new graph structure with a Metropolis-Hasting algorithm, which alleviates the risk of getting stuck in local optimums by stochastic sampling with an acceptance probability. Once a graph structure is accepted, a higher-level graph is then constructed by taking the partitioned cliques as its nodes. During the evolving process, representation becomes more abstracted in higher-levels where redundant information is filtered out, allowing more efficient propagation of long-range data dependencies. We evaluate the effectiveness of structure-evolving LSTM in the application of semantic object parsing and demonstrate its advantage over state-of-the-art LSTM models on standard benchmarks.

Temperature is the main index that affects human comfort, and it is easily affected by air flow, rainfall, ocean currents and other factors to produce large fluctuations. In this paper, LSTM (Long Short-Term Memory) neural network algorithm is used to predict the long-term temperature of coastal cities. The training and testing data included eight measurements (every three hours) a day, spanning 10 years (2009–2019). Shanghai was chosen as the coastal city to be predicted. The structure of the neural network is composed of two kinds of neural networks, two LSTM layers and one full connection layer. The function of LSTM layer is to deal with the correlation of temperature time series, while the full connection layer maps the output layer of LSTM to the final prediction result. By comparing Back Propagation (BP), Recurrent Neural Network (RNN) and LSTM, the results show that LSTM is more suitable for temperature time series prediction.KeywordsTemperature forecastingMachine learningArtificial intelligenceDeep learningNeural network

This paper presents a computer simulation model for predicting solar irradiance in a three-dimensional (3D) environment. Solar irradiance prediction is critical for solar energy systems and related fields. Machine learning techniques, such as recurrent neural networks (RNNs), are employed for more accurate predictions. Integrating a 3D environmental simulation with the RNN models achieves accurate predictions with reasonable resolutions. The training model uses selected astronomical and atmospheric factors to train the RNN models. The proposed method allows the user to obtain the corresponding solar irradiance prediction values for arbitrary periods. Astronomical and atmospheric factors affect solar irradiance; hence, data from the Korea Meteorological Administration are used for training. The RNNs, including the long short-term memory (LSTM) and gated recurrent unit methods, are employed for the prediction. The LSTM layers outperformed other configurations, accurately predicting zero irradiation values. A set of solar irradiance models is presented using RNNs by configuring their layers, and the layout consisting of four LSTM layers performed best. This layout achieved reasonable error bounds, with relatively good root mean squared error and mean absolute error values. A computer graphics-based solar irradiance prediction model is proposed based on this prediction model, incorporating simulations of the surrounding environment. A case study is presented with surrounding buildings to analyze the solar irradiance over the year with a one-hour forecasting horizon to demonstrate its feasibility. Moreover, we plan to improve the results with other neural network models, such as the fuzzyembedded RNN.OPEN ACCESS Received: 19/09/2024 Accepted: 13/12/2024