Events and Objects Interaction Recognition using Aerial Videos

We present a general framework for complex event recognition that is well-suited for integrating information that varies widely in detail and granularity. Consider the scenario of an agent in an instrumented space performing a complex task while describing what he is doing in a natural manner. The system takes in a variety of information, including objects and gestures recognized by RGB-D and descriptions of events extracted from recognized and parsed speech. The system outputs a complete reconstruction of the agent's plan, explaining actions in terms of more complex activities and filling in unobserved but necessary events. We show how to use Markov Logic (a probabilistic extension of first-order logic) to create a model in which observations can be partial, noisy, and refer to future or temporally ambiguous events; complex events are composed from simpler events in a manner that exposes their structure for inference and learning; and uncertainty is handled in a sound probabilistic manner. We demonstrate the effectiveness of the approach for tracking kitchen activities in the presence of noisy and incomplete observations.

Symbolic event recognition systems have been successfully applied to a variety of application domains, extracting useful information in the form of events, allowing experts or other systems to monitor and respond when significant events are recognised. In a typical event recognition application, however, these systems often have to deal with a significant amount of uncertainty. In this article, we address the issue of uncertainty in logic-based event recognition by extending the Event Calculus with probabilistic reasoning. Markov logic networks are a natural candidate for our logic-based formalism. However, the temporal semantics of the Event Calculus introduce a number of challenges for the proposed model. We show how and under what assumptions we can overcome these problems. Additionally, we study how probabilistic modelling changes the behaviour of the formalism, affecting its key property—the inertia of fluents. Furthermore, we demonstrate the advantages of the probabilistic Event Calculus through examples and experiments in the domain of activity recognition, using a publicly available dataset for video surveillance.

Event and activity modelling and recognition is at the centre of situational awareness (SA). Being able to detect and extract useful semantic information from diverse data sources in the form of events and activities, an SA system may assist human operators with cognitive analysis, situation monitoring and to inform decision making. Recognition of realworld activities with complex spatial and temporal interactions in the presence of uncertainty has been a major line of research. In this respect, Statistical Relational Learning methods such as Markov Logic Networks (MLNs) have been shown to provide a powerful and promising framework, and efficiently leveraging their representational power for event and activity recognition are active research and development activities. In this paper we propose a complementary method to existing approaches, FQSTAG-MLN, that combines Fuzzy MLNs with Fuzzy Qualitative Spatio-Temporal Activity Graphs (FQSTAGs) toward achieving efficient and explainable activity modelling and recognition. With FQSTAG-MLNs, the complexity of real-world activities is addressed by combining the representational richness of high-level activity specification with compact local spatio-temporal narratives for low-level events; while the uncertainty of activity and world models is calibrated using probabilistic methods and the vagueness of observations is calibrated using fuzzy set theory. We describe its current implementation within RUSH, which aims to be a flexible reasoning and learning platform for situational awareness. We present a use case illustration and early results.

Situational awareness requires continual learning from observations and adaptive reasoning from domain and contextual knowledge. The integration of reasoning and learning has been a standing goal of machine learning and AI in general, and a pressing need for real-world situational awareness in particular. Representative techniques among the numerous methods proposed include integrating logics with learning formalisms, whether probabilistic graphical models or neural methods. These techniques are motivated by the need to model and exploit the symmetry, regularities and complex relations between entities exhibited in real world scenarios (in the form of relational or graph data) for effective reasoning and learning. In this work, we investigate the benefits of integrating two prominent methods for reasoning and learning with relational/graph data, Markov Logic Networks (or simply Markov Logic) and Graph Neural Networks. The former is well-recognised for its powerful representation and uncertainty handling, while the latter have gained much attention due to their efficiency in handling large-scale graph datasets. This paper reports on the potential benefits of combining their respective strengths and applying them to a use case illustration in the maritime domain, together with empirical results.

Recognizing the actions, plans, and goals of a person in an unconstrained environment is a key feature that future robotic systems will need in order to achieve a natural human-machine interaction. Indeed, we humans are constantly understanding and predicting the actions and goals of others, which allows us to interact in intuitive and safe ways. While action and plan recognition are tasks that humans perform naturally and with little effort, they are still an unresolved problem from the point of view of artificial intelligence. The immense variety of possible actions and plans that may be encountered in an unconstrained environment makes current approaches be far from human-like performance. In addition, while very different types of algorithms have been proposed to tackle the problem of activity, plan, and goal (intention) recognition, these tend to focus in only one part of the problem (e.g., action recognition), and techniques that address the problem as a whole have been not so thoroughly explored. This review is meant to provide a general view of the problem of activity, plan, and goal recognition as a whole. It presents a description of the problem, both from the human perspective and from the computational perspective, and proposes a classification of the main types of approaches that have been proposed to address it (logic-based, classical machine learning, deep learning, and brain-inspired), together with a description and comparison of the classes. This general view of the problem can help on the identification of research gaps, and may also provide inspiration for the development of new approaches that address the problem in a unified way.

While approaches based on bags of features excel at low-level action classification, they are ill-suited for recognizing complex events in video, where concept-based temporal representations currently dominate. This paper proposes a novel representation that captures the temporal dynamics of windowed mid-level concept detectors in order to improve complex event recognition. We first express each video as an ordered vector time series, where each time step consists of the vector formed from the concatenated confidences of the pre-trained concept detectors. We hypothesize that the dynamics of time series for different instances from the same event class, as captured by simple linear dynamical system (LDS) models, are likely to be similar even if the instances differ in terms of low-level visual features. We propose a two-part representation composed of fusing: (1) a singular value decomposition of block Hankel matrices (SSID-S) and (2) a harmonic signature (HS) computed from the corresponding eigen-dynamics matrix. The proposed method offers several benefits over alternate approaches: our approach is straightforward to implement, directly employs existing concept detectors and can be plugged into linear classification frameworks. Results on standard datasets such as NIST's TRECVID Multimedia Event Detection task demonstrate the improved accuracy of the proposed method.

Representation and recognition of human activities or other events generated by vehicles in restricted settings such as airports, railway station and parking lots is of great importance in automated surveillance systems. The problem is difficult because: 1) at the lower level, detection of primitive actions is inaccurate due to changes in illumination, noise and resolution of video camera. 2) at the higher level, activities or events in such settings are usually originated by multi-object. It is hard to represent and parse the complex activities and events. In this paper, we describe a new approach for representing and recognizing the events in video sequences, and the approach is able to handle complex multi-agent interactions. Our experimental results on a dataset that consisting of videos took by normal camera and videos took by surveillance camera in laboratory.

In recent years, the task of event recognition from videos has attracted increasing interest in multimedia area. While most of the existing research was mainly focused on exploring visual cues to handle relatively small-granular events, it is difficult to directly analyze video content without any prior knowledge. Therefore, synthesizing both the visual and semantic analysis is a natural way for video event understanding. In this paper, we study the problem of Web video event recognition, where Web videos often describe large-granular events and carry limited textual information. Key challenges include how to accurately represent event semantics from incomplete textual information and how to effectively explore the correlation between visual and textual cues for video event understanding. We propose a novel framework to perform complex event recognition from Web videos. In order to compensate the insufficient expressive power of visual cues, we construct an event knowledge base by deeply mining semantic information from ubiquitous Web documents. This event knowledge base is capable of describing each event with comprehensive semantics. By utilizing this base, the textual cues for a video can be significantly enriched. Furthermore, we introduce a two-view adaptive regression model, which explores the intrinsic correlation between the visual and textual cues of the videos to learn reliable classifiers. Extensive experiments on two real-world video data sets show the effectiveness of our proposed framework and prove that the event knowledge base indeed helps improve the performance of Web video event recognition.

Social and economic conflicts bring along irregular migrations, which are risky and dangerous and lead to humanitarian movements on a global scale. Irregular migration movements increasingly have become a problem of identity, belonging and recognition, and bring with new areas of discussion. Irregular migration, which goes beyond being a phenomenon that nation states can control alone, is perceived as a security threat, especially by European countries, and restrictions are increased with revised migration policies. The blockage of legal routes increases the demand for illegal methods and makes migrant smuggling more apparent. On the other hand, the Mediterranean, which is used as a migration route to reach Europe, has almost become identified with the bodies washed ashore. The number of recorded deaths of migrants in the Mediterranean reveals that the phenomenon of migration is a matter of right to life beyond the political, economic, and social interests of the nation states and accelerated the discussion of the problem on different platforms. Based on this, the film Terraferma, which is about irregular migration over the Mediterranean, was examined as a research subject and analyzed using the dramaturgical analysis technique. The film has been thematically interpreted on the basis of the plot, main idea, assumption, conflicts and characters. Policies such as the fact that global inequalities put pressure on the developed countries with irregular migration on one hand, and the situation of ignoring the phenomenon of irregular migration with strict migration on the other have been evaluated together.

Vehicles' safety surveillances have been paid more attentions on by the governments with increasingly rampant terrorist activities like car bombing attacks and illegal immigrations. Lots of terrorists may hide the bombs under the vehicles to avoid the searches by the workers. Illegal immigrants usually adopt the same tricks. Hence, vehicles' bottom will be one of the most important searching places in the safety inspection. However, traditional bottom inspections are accomplished manually with hand-held security mirrors. This method has poor efficiency, limited searching areas, unbearable reliability and large consumption of human resources. So computer vision and intelligent transportation technologies are led into the safety inspections and surveillances according to these critical defects. Line scanned and area scanned CCD cameras are trigged by the vehicle detection apparatus to grab images of the vehicles' bottom and license plates respectively. These images will be processed by the industrial computers real-timely. Sounded alarms can remind the workers there are existing doubtable foreign goods under the bottom. The experimental results indicate preliminarily that the proposed method not only have good performances on detection rate and low false alarm rate, also can improve the reliability effectively.

Finding an empty parking space in a crowded area is very stressful due to congestion. The absence of empty space leads to fatigue before the main activity, increased fuel consumption, which leads to increased pollution, and increased traffic due to the search for empty space. Therefore, it was necessary to have a system that would help drivers know the condition of each parking space. Empty or occupied with a vehicle. Where empty and occupied spaces are calculated and determined. The system displays on the screen at a convenient location in the parking lot a map to guide the driver to vacant positions and a warning if all positions are occupied. All this is done through images taken from the surveillance cameras in the parking lot. We find that the advantages of using vision-based systems over other existing systems are threefold. First There is no need to update the infrastructure of the parking lot, provided that the place is equipped with CCTV cameras that monitor the parking spaces covering the entire place. Secondly, camera-based systems give the exact location i.e. a detailed map of the vacant car parks which are good and necessary vacant lots. Third, camera-based methods are highly applicable in street parking spaces and residential areas.

In recent days, pedestrian surveillance paths are utilized by many people, and the safety issue has become a point of concern in crowded areas. In general, most surveillance systems rely on people watching the cameras, which is inefficient. Although watching in general, it is difficult to spot abnormal events when many events are happening at the same time. Because of this problem, the idea of automating the process of spotting abnormal events was investigated in this project. This project involves the detection of abnormal events by using deep learning methodologies. A CNN model with VGG16 is utilized to analyze images captured from surveillance cameras. Transfer learning is applied to make the model learn faster and better with fewer images. The system is developed using the Django framework, where people can upload images and get predictions. From the predictions, it is clear that the system is more efficient than watching in general and saves human

The accurate detection and recognition of human actions play a pivotal role in aerial surveillance, enabling the identification of potential threats and suspicious behavior. Several approaches have been presented to address this problem, but the limitation still remains in devising an accurate and robust solution. To this end, this paper presents an effective action recognition framework for aerial surveillance, employing the YOLOv8-Pose keypoints extraction algorithm and a customized sequential ConvLSTM (Convolutional Long Short-Term Memory) model for classifying the action. We performed a detailed experimental evaluation and comparison on the publicly available Drone Action dataset. The evaluation and comparison of the proposed framework with several existing approaches on the publicly available Drone Action dataset demonstrate its effectiveness, achieving a very encouraging performance. The overall accuracy of the framework on three provided dataset splits is 74%, 80%, and 70%, with a mean accuracy of 74.67%. Indeed, the proposed system effectively captures the spatial and temporal dynamics of human actions, providing a robust solution for aerial action recognition.

In this paper, we consider the problem of event recognition on single images. In contrast to conventional fine-tuning of convolutional neural networks (CNN), we proposed to use image captioning, i.e., a generative model that converts images to textual descriptions. The motivation here is the possibility to combine conventional CNNs with a completely different approach in an ensemble with high diversity. As event recognition task has nothing serial or temporal, obtained captions are one-hot encoded and summarized into a sparse feature vector suitable for the learning of an arbitrary classifier. We provide the experimental study of several feature extractors for Photo Event Collection, Web Image Dataset for Event Recognition and Multi-Label Curation of Flickr Events Dataset. It is shown that the image captions trained on the Conceptual Captions dataset can be classified more accurately than the features from an object detector, though they both are obviously not as rich as the CNN-based features. However, an ensemble of CNN and our approach provides state-of-the-art results for several event datasets.

Computational models for goal recognition hold great promise for enhancing the capabilities of drama managers and director agents for interactive narratives. The problem of goal recognition, and its more general form, plan recognition, have been the subjects of extensive investigation in the AI community. However, relatively little effort has been undertaken to examine goal recognition in interactive narrative. In this paper, we propose a research agenda to improve the accuracy of goal recognition models for interactive narratives using explicit representations of narrative structure inspired by the natural language processing community. We describe a particular category of narrative representations, narrative schemas, that we anticipate will effectively capture patterns of player behavior in interactive narratives and improve the accuracy of goal recognition models.