Computational Mechanisms of the Visual Processing of Action Stimuli

Computational modeling helps neuroscientists to integrate and explain experimental data obtained through neurophysiological and anatomical studies, thus providing a mechanism by which we can better understand and predict the principles of neural computation. Computational modeling of the neuronal pathways of the visual cortex has been successful in developing theories of biological motion processing. This review describes a range of computational models that have been inspired by neurophysiological experiments. Theories of local motion integration and pattern motion processing are presented, together with suggested neurophysiological experiments designed to test those hypotheses.

With the rapid development of the computer and sensor field, inertial sensor data have been widely used in human activity recognition. At present, most relevant studies divide human activities into basic actions and transitional actions, in which basic actions are classified by unified features, while transitional actions usually use context information to determine the category. For the existing single method that cannot well realize human activity recognition, this paper proposes a human activity classification and recognition model based on smartphone inertial sensor data. The model fully considers the feature differences of different properties of actions, uses a fixed sliding window to segment the human activity data of inertial sensors with different attributes and, finally, extracts the features and recognizes them on different classifiers. The experimental results show that dynamic and transitional actions could obtain the best recognition performance on support vector machines, while static actions could obtain better classification effects on ensemble classifiers; as for feature selection, the frequency-domain feature used in dynamic action had a high recognition rate, up to 99.35%. When time-domain features were used for static and transitional actions, higher recognition rates were obtained, 98.40% and 91.98%, respectively.

Recently, activity recognition and classification (ARC) of human activity opens new research area in the field health care, security, and privacy of human society. Specifically, the promise of device-free activity recognition platform attracts researchers to develop platform to ensure the correct detection of activity recognition. The technologies, such as Wi-Fi, GSM, and radars, do not require installing cameras or wearable sensors for activity monitoring and recognition. Therefore, this device-free technology has gain popularity in health care and safety measurement systems. Traditional ARC systems depend on wearable sensors such as magic rings and vision technology such as a Microsoft Kinect. In the future, researchers are striving to reduce such devices and targeting a promising device-free sensing system. In this paper, a software-defined radio platform was designed for the detection of human activity. The extensive experiments were performed in the laboratory environment by using two Universal Software Radio Peripheral (USRP) to extract the wireless channel state information (WCSI). The 64-Fast Fourier Transform (FFT) point’s Orthogonal frequency division multiplexing (OFDM) signal was used to determine the WCSI. The design of the proposed system can be used for multiple applications due to scalability and flexibility of the software-defined hardware.

Typical patterns of hand-joint covariation arising in the context of grasping actions enable one to provide simplified descriptions of these actions in terms of small sets of hand-joint parameters. The computational model of mirror mechanisms introduced here hypothesizes that mirror neurons are crucially involved in coding and making this simplified motor information available for both action recognition and control processes. In particular, grasping action recognition processes are modeled in terms of a visuo-motor loop enabling one to make iterated use of mirror-coded motor information. In simulation experiments concerning the classification of reach-to-grasp actions, mirror-coded information was found to simplify the processing of visual inputs and to improve action recognition results with respect to recognition procedures that are solely based on visual processing. The visuo-motor loop involved in action recognition is a distinctive feature of this model which is coherent with the direct matching hypothesis. Moreover, the visuo-motor loop sets the model introduced here apart from those computational models that identify mirror neuron activity in action observation with the final outcome of computational processes unidirectionally flowing from sensory (and usually visual) to motor systems.

We propose a unified method for recognizing human action and human related events in a realistic video. We use an efficient pipeline of (a) a 3D representation of the Improved Dense Trajectory Feature (DTF) and (b) Fisher Vector (FV). Further, a novel descriptor is proposed, capable of representing human actions and human related events based on the FV representation of the input video. The proposed unified descriptor is a 168-dimensional vector obtained from each video sequence by statistically analyzing the motion patterns of the 3D joint locations of the human body. The proposed descriptor is trained using binary Support Vector Machine (SVM) for recognizing human actions or human related events. We evaluate the proposed approach on two challenging action recognition datasets: UCF sports and CMU Mocap datasets. In addition to the two action recognition dataset, the proposed approach is tested on the Hollywood2 event recognition dataset. On all the benchmark datasets for both action and event recognition, the proposed approach has shown its efficacy compared to the state-of-the-art techniques.

Human activity recognition is one of the most challenging research areas in human-machine interaction. Activity recognition by wearable sensors produces significant information about a person's functional ability and lifestyle. In this paper, we present an inertial sensors-based approach for activity and postural transition recognition. Efficient features are extracted from the sensor data. Then, the features are normalized using Z-score followed by Generalized Discriminant Analysis (GDA) to make them more robust. Finally, the features are trained with a Recurrent Neural Network (RNN) for activity and postural transition recognition. The proposed method recognizes 12 activities with an overall accuracy of 97%. In the activity recognition system, the multimodal input data is collected from only one accelerometer and one gyroscope sensor embedded in the user's smartphone. The proposed system can contribute in research fields such as human-robot interaction for practical applications such as eldercare in smart homes.

One of the long term objectives of robotics and artificial cognitive systems is that robots will increasingly be capable of interacting in a cooperative and adaptive manner with their human counterparts in open-ended tasks that can change in real-time. In such situations, an important aspect of the robot behavior will be the ability to acquire new knowledge of the cooperative tasks by observing humans. At least two significant challenges can be identified in this context. The first challenge concerns development of methods to allow the characterization of human actions such that robotic systems can observe and learn new actions, and more complex behaviors made up of those actions. The second challenge is associated with the immense heterogeneity and diversity of robots and their perceptual and motor systems. The associated question is whether the identified methods for action perception can be generalized across the different perceptual systems inherent to distinct robot platforms. The current research addresses these two challenges. We present results from a cooperative human-robot interaction system that has been specifically developed for portability between different humanoid platforms. Within this architecture, the physical details of the perceptual system (e.g. video camera vs IR video with reflecting markers) are encapsulated at the lowest level. Actions are then automatically characterized in terms of perceptual primitives related to motion, contact and visibility. The resulting system is demonstrated to perform robust object and action learning and recognition on two distinct robotic platforms. Perhaps most interestingly, we demonstrate that knowledge acquired about action recognition with one robot can be directly imported and successfully used on a second distinct robot platform for action recognition. This will have interesting implications for the accumulation of shared knowledge between distinct heterogeneous robotic systems.

The function of smooth pursuit is to keep the fovea pointed at a small visual target that moves smoothly across a patterned background. Chemical lesions, single cell recordings, and behavioral measures have shown that the cortical motion processing pathways form the afferent limb for pursuit. Important areas include at least the striate cortex and the middle temporal visual area, and probably the medial superior temporal visual area and the posterior parietal cortex. We argue that the visual inputs are transmitted through a simple sensory motor interface in the pons, to the efferent limb in the brain stem and cerebellum. The efferent limb uses neural velocity memory to maintain pursuit automatically. We present evidence that the velocity memory is provided, at least in part, by eye velocity positive feedback between the flocculus of the cerebellum and the brain stem. Finally, we use a computer model to show how the maintenance of pursuit can be simulated on a millisecond time scale. The structure and internal elements of the model are based on the biological experiments reviewed in our paper. In the past five years, progress on the neural basis of pursuit eye movements has been rapid. Several areas of research have made substantial contributions, by using combinations of new and conventional methods. Many of the pathways that contribute to pursuit have been identified, and their physiological activity and functions are becoming understood. Continuing progress promises to yield an understanding of one specific form of visually guided movement, at the level of neuronal circuits and behavior, in the primate.

Due to the huge number of online videos uploaded and viewed every day, there is an emerging need nowadays for the action recognition techniques. Applying these techniques in uncontrolled and realistic videos is still a challenging task, considering the large variations in camera motion, viewpoint, cluttered background etc. Moreover, they need to be automated to be able to handle such an amount of different actions. The goal of this study is to introduce a new technique for mining mid‐level discriminative patches from videos. These patches are the most representative parts that can describe an action. To achieve this goal, the authors generalise a technique borrowed from 2D images to generate bounding boxes with a high motion and appearance saliencies. Then, a clustering‐classification iterative procedure is applied on the generated boxes. Then, they calculate a discriminative score for each box. Finally, they select top ranked boxes to train exemplar‐SVM on low‐level features which are extracted from the selected boxes. The proposed approach has been evaluated using two challenging datasets YouTube and JHMDB. The experimental results demonstrated the effectiveness of their approach to achieve a better average recognition accuracy than the state‐of‐the‐art techniques.

People drive on the road and eat in the kitchen. Can the road imply driving or the kitchen imply eating? This paper addresses such a problem by studying the relations between actions and scenes. To get effective scene representation, we use a deep convolutional neural networks (CNN) model trained from a scene-centric database to predict scene responses for videos. We employ two encoding schemes based on frame features to represent the scene and its changes, respectively. We conduct experiments on two challenging datasets, HMDB51 and Hollywood2, and compare action recognition results of different encodings based on different scene features. Our results demonstrate that scene features, when combined with motion features, improve the state-of-the-art results for action recognition. Finally, we explore the relationship between actions and scenes by analyzing scene preferences to a particular action qualitatively and quantitatively.

Local fusion networks with chained residual pooling for video action recognition

A recent paper in the journal by Jones and Johnson1 describes the use of designed experiments with computer models for product design and development activities. In recent years we have seen a dramatic increase in the use of computer models for engineering design activities. Some typical applications include computational fluid dynamics models to study turbulent flow, finite element analysis models to investigate the distribution of stresses in systems, and computer design tools for electrical circuits and components. Computer models are also widely used in weather forecasting and the study of hurricanes. These applications are continuing to grow and the applications environments are expanding. Researchers can use these models as substitutes for the physical systems that they represent because if they are properly calibrated and validated, experiments on the model can provide basically the same information as experiments on the real physical system at a fraction of the time and cost. Many of these models are deterministic; that is, they are systems of complex equations without any random components; hence, many of the traditional principles of designing experiments such as randomization, replication, and blocking do not directly apply. In many cases the traditional low-order polynomial models that we typically associate with the designed experiments are not appropriate either, because the underlying system is very complex. The Gaussian process model, a spatial correlation model, is often used to fit the data from a computer experiment. Space-filling designs such as the Latin hypercube are used instead of the familiar factorial and response surface designs. This is a dynamic area of research and development in the experimental design field; interaction between engineers, scientists, and experimental design experts will produce new methods and techniques. There is an another class of computer models not discussed in the Jones and Johnson paper. These are stochastic simulations, that is, computer simulations driven by streams of random numbers producing output responses that can be viewed as random variables. Discrete-event simulations, typical of these models, have been widely used since the 1960s to study factory scheduling problems, hospital operating room activities, and supply chain operations, just to name a few applications. They are also being used increasingly to study many types of transactional and service operations. These computer models also present challenges to the experimental designer. Often they are very high-fidelity models with many input factors; hence, traditional experimental approaches to factor screening and optimization may not be directly applicable. The output may be a time series or some type of functional response; hence, traditional low-order polynomials may not be appropriate. Sometimes these models include both controllable factors and uncontrollable or ‘noise’ factors, hence, some type of robust design or process robustness analysis may be required. Designing experiments for both types of computer model present significant challenges. Quality and Reliability Engineering International welcomes papers, including applications and case studies, on all aspects of these problems, illustrating how computer model and designed experiments can be effectively deployed to solve a significant practical problem, as well as methodological papers on aspects of design.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Dance-like actions are complex visual stimuli involving multiple changes in body posture across time and space. Visual perception research has demonstrated a difference between the processing of dynamic body movement and the processing of static body posture. Yet, it is unclear whether this processing dissociation continues during the retention of body movement and body form in visual working memory (VWM). When observing a dance-like action, it is likely that static snapshot images of body posture will be retained alongside dynamic images of the complete motion. Therefore, we hypothesized that, as in perception, posture and movement would differ in VWM. Additionally, if body posture and body movement are separable in VWM, as form- and motion-based items, respectively, then differential interference from intervening form and motion tasks should occur during recognition. In two experiments, we examined these hypotheses. In Experiment 1, the recognition of postures and movements was tested in conditions in which the formats of the study and test stimuli matched (movement-study to movement-test, posture-study to posture-test) or mismatched (movement-study to posture-test, posture-study to movement-test). In Experiment 2, the recognition of postures and movements was compared after intervening form and motion tasks. These results indicated that (1) the recognition of body movement based only on posture is possible, but it is significantly poorer than recognition based on the entire movement stimulus, and (2) form-based interference does not impair memory for movements, although motion-based interference does. We concluded that, whereas static posture information is encoded during the observation of dance-like actions, body movement and body posture differ in VWM.

Computational modelling has proven to be a valuable approach in developing theories of spoken-word processing. In this paper, we focus on a particular class of theories in which it is assumed that the spoken-word recognition process consists of two consecutive stages, with an ‘abstract’ discrete symbolic representation at the interface between the stages. In evaluating computational models, it is important to bring in independent arguments for the cognitive plausibility of the algorithms that are selected to compute the processes in a theory. This paper discusses the relation between behavioural studies, theories, and computational models of spoken-word recognition. We explain how computational models can be assessed in terms of the goodness of fit with the behavioural data and the cognitive plausibility of the algorithms. An in-depth analysis of several models provides insights into how computational modelling has led to improved theories and to a better understanding of the human spoken-word recognition process.