Exploiting Multimedia Content: A Machine Learning Based Aproach

In this paper, we propose an effective fusion scheme of visual and auditory modalities to detect events in sports video. The proposed scheme is built upon semantic shot classification, where we classify video shots into several major or interesting classes, each of which has clear semantic meanings. Among major shot classes we perform classification of the different auditory signal segments (i.e. silence, hitting ball, applause, commentator speech) with the goal of detecting events with strong semantic meaning. For instance, for tennis video, we have identified five interesting events: serve, reserve, ace, return, and score. Since we have developed a unified framework for semantic shot classification in sports videos and a set of audio mid-level representation with supervised learning methods, the proposed fusion scheme can be easily adapted to a new sports game. We are extending this fusion scheme to three additional typical sports videos: basketball, volleyball and soccer. Correctly detected sports video events will greatly facilitate further structural and temporal analysis, such as sports video skimming, table of content, etc.

We summarize our recent work on “highlight” events detection and recognition in sports video. We have developed two different joint audio-visual fusion frameworks for this task, namely “audio-visual coupled hidden Markov model” and “audio classification then visual hidden Markov model verification”. Our comparative study of these two frameworks shows that the second approach outperforms the first approach by a large margin. Our study also suggests the importance of modeling the so-called middle-level features such as audience reactions and camera patterns in sports video.

Anomaly detection in video is a challenging computer vision problem. Due to the lack of anomalous events at training time, anomaly detection requires the design of learning methods without full supervision. In this paper, we approach anomalous event detection in video through self-supervised and multi-task learning at the object level. We first utilize a pre-trained detector to detect objects. Then, we train a 3D convolutional neural network to produce discriminative anomaly-specific information by jointly learning multiple proxy tasks: three self-supervised and one based on knowledge distillation. The self-supervised tasks are: (i) discrimination of forward/backward moving objects (arrow of time), (ii) discrimination of objects in consecutive/intermittent frames (motion irregularity) and (iii) reconstruction of object-specific appearance information. The knowledge distillation task takes into account both classification and detection information, generating large prediction discrepancies between teacher and student models when anomalies occur. To the best of our knowledge, we are the first to approach anomalous event detection in video as a multi-task learning problem, integrating multiple self-supervised and knowledge distillation proxy tasks in a single architecture. Our lightweight architecture outperforms the state-of-the-art methods on three benchmarks: Avenue, ShanghaiTech and UCSD Ped2. Additionally, we perform an ablation study demonstrating the importance of integrating self-supervised learning and normality-specific distillation in a multi-task learning setting.

Unsupervised anomaly detection defines an abnormal event as an event that does not conform to expected behavior. In the field of unsupervised anomaly detection, it is a pioneering work that leverages the difference between a future frame predicted by a generative adversarial network and its ground truth to detect an abnormal event. Based on the work, we improve the ability of video prediction framework to detect abnormal events by enhancing the difference between prediction results for normal and abnormal events. We incorporate super-resolution and self-attention mechanism to design a generative adversarial network. We propose an auto-encoder as a generator, which incorporates dense residual networks and self-attention. Moreover, we propose a new discriminator, which introduces self-attention on the basis of a relativistic discriminator. To predict a future frame with higher quality for normal events, we impose a constraint on the motion in video prediction by fusing optical flow and gradient difference between frames. We also introduce a perception constraint in video prediction to enrich the texture details of a frame. The AUC of our method on CUHK Avenue and Shanghai Tech datasets reaches 89.2% and 75.7% respectively, which is better than most existing methods. In addition, we propose a processing flow that can realize real-time anomaly detection in videos. The average running time of our video prediction framework is 37 frames per second. Among all real-time methods for abnormal event detection in videos, our method is competitive with the state-of-the-art methods.

Video motion target tracking has been widely used in the field of sports. Content-based video search has a broad application prospect and attracts more and more researchers’ attention. In view of the shortcomings of moving object detection in motion video at present, a human motion recognition method based on correlation vector machine is proposed. Combined with the improved Gaussian mixture model of moving object detection and tracking method in motion video, moving object detection and tracking are carried out. The design of the system is realized. Firstly, it collects sports videos, considers all kinds of sports objects in sports videos, uses improved background update difference method to detect sports objects, then uses correlation vector machine to recognize human motion, and finally uses AdaBoost classifier to use it in multiple sports videos and sports objects. Carry out follow-up experiments. Results in this way, we can track the sports target in sports video accurately and quickly, and the real-time performance of sports target tracking is better than other tracking methods, which provides a new research tool for tracking sports target in sports video.

The paper presents a novel approach for event detection in sports videos by topic based graphical model learning. The characteristics features defining various sport events are extracted by contextual grouping of low-level video and audio features using topic modeling. Event detection is performed by learning the structure of context based distribution of characteristic features by CRF based graphical model. Experimental evaluation of the proposed concept is presented on recorded video of Handball and Soccer game.

This paper presents an efficient and accurate automated system based on the hybrid XGBoost with Random forest (XGBRF) ensemble model in order to classify the Glioma (type of mostly diagnosed brain tumor) into low grade and high grade Glioma. In this approach initially global thresholding is employed on various MRI sequence and their fusion combinations in order to perform the accurate segmentation. Then uses a proposed Enhanced wavelet binary pattern run length matrix method (EWBPRL) for textural features extraction from the region of interest or segmented Glioma tumor region. This proposed feature extraction method is based on the Discrete wavelet transform (DWT), Local Binary pattern (LBP) and Gray level run length Matrix (GLRLM) methods to extract texture features from the segmented region. Some morphological features are also computed from the segmented region along the textural features. Finally both these extracted features are employed in order to train a hybrid XGBoost with Random forest ensemble model for the first time. The proposed automated system apart from accurately detecting and segmenting the tumors region from the fused MRI sequences, also tends to determine the grading of Glioma in terms of severity. The proposed system is evaluated on the large size balance local dataset and as well as on the popular global datasets like BRATS 2013 and BRATS 2015. This approach offers an encouraging accuracy of 99.25% on the local dataset with the fusion of T1C+T2+Flair MRI sequence as compare to 96.75% accuracy, which is achieved utilizing the fusion of T1+T1C+T2+Flair MRI sequence.

We describe a framework for human action retrieval in still web images by verb queries, for instance “phoning”. Firstly, we build a group of visual discriminative instances for each action class, called “Exemplarlets”. Thereafter we employ Multiple Kernel Learning (MKL) to learn an optimal combination of histogram intersection kernels, each of which captures a state-of-the-art feature channel. Our features include the distribution of edges, dense visual words and feature descriptors at different levels of spatial pyramid. For a new image we can detect the hot-region using a sliding-window detector learnt via MKL. The hot-region can imply latent actions in the image. After the hot-region has been detected, we build a inverted index in the visual search path, which we called Visual Inverted Index (VII). Finally, fusing the visual search path and the text search path, we can get the accurate results either relevant to text or to visual information. We show both the detection and retrieval results on our newly collected dataset of six actions as well as demonstrate improved performance over existing methods.

With rapid growth of egocentric videos from wearable devices, the need for instant video event detection is emerging. Different from conventional video event detection, it requires more considerations on real-time event detection and immediate video recording due to the computational cost on wearable devices (e.g., Google Glass). Conventional work of video event detection analyzed video content in an offline process and it is time-consuming for visual analysis. Observing that wearable devices are usually along with sensors, we propose a novel approach for instant event detection in egocentric videos by leveraging sensor-based motion context. We compute statistics of sensor data as features. Next, we predict the user's current motion context by a hierarchical model, and then choose the corresponding ranking model to rate the importance score of the timestamp. With importance score provided in real-time, camera on the wearable device can dynamically record micro-videos without wasting power and storage. In addition, we collected a challenging daily-life dataset called EDS (Egocentric Daily-life Videos with Sensor Data), which contains both egocentric videos and sensor data recorded by Google Glass of different subjects. We evaluate the performance of our system on the EDS dataset, and the result shows that our method outperforms other baselines.

Athlete identification is important for sp ort video content analysis since users often care about the video clips with their preferred athletes. In this paper, we propose a method for athlete id entification by combing th e segmentation, tracking and recognition procedures in to a coarse-to-fine scheme for jersey number (digital characters on sport shirt) detection. Firstly, image segmentation is employed to separate the jersey number regions with its background. And size/pipe-like attributes of digital characters are used to filter out candidates. Then, a K -NN ( K nearest neighbor) classifier is employed to classify a candidate into a digit in “0-9” or negative. In the recognition procedure, we use the Zernike moment features, which are invariant to ro tation and scale for digital shape recognition. Synth etic training samples with different fonts are used to represent the pattern of digital characters with non-rigid deformation. Once a character candidate is detected, a SSD (smallest square distance)-based tracking procedure is started. The recognition pro cedure is performed every several frames in the tracking process. After tracking tens of frames, the overall recognition results are co mbined to determine if a candidate is a true jersey number or not by a voting procedur e. Experiments on several types of sports video shows encouraging result. Keywords: Jersey number, sports vi deo, character recognition.

Surveillance systems are mainly used to monitor activity and behaviour for the purpose of managing small human group. In particularly, the depicted approach focused on the small human groups stayed in the same place for a while and characterizing the behaviour of the group. The described approach has wide functions in several areas such as surveillance, group interface and behaviour classification. The video surveillance mainly deals with the recognition and classification of group behaviour with respect to several activities such as Normal speech, kicking, hugging and punching. The processing steps include frame generation, segmentation using Fuzzy C-Mean (FCM) clustering; features are extracted using Local Binary Pattern (LBP) and Hierarchical Centroid (HC). Convolution Neural Network (CNN) is used to classify the features. The proposed model is better than existing methods by achieving 80% accuracy.

Segmentation, where pixels are categorized by tissue types, is essential in medical image processing. This paper proposes a multi-level Fuzzy C-Means method to extract an intracranial from its background and skull. Then, a two-level Otsu multi-thresholding method is applied to segment the intracranial structure into cerebrospinal fluid, brain matters and other homogenous regions. Based on symmetrical properties in the intracranial structures, the left-half and right-half segmented intracranial regions are quantitatively compared with respect to the intracranial midline. The segmented regions are found to be very useful in providing information regarding normal and abnormal structures in the intracranial because any asymmetry that is detected would indicate a high probability of abnormalities. Additionally, pixel intensity information such as standard deviation and the maximum value of the pixels of the segmented regions are used to distinguish abnormalities such as bleeding and calcification from normal cases. This experimental work uses a medical image database consisting of 519 normal and 201 abnormal serial computed tomography (CT) brain images from 31 patients. The proposed multi-level segmentation approach proved to effectively isolate important homogenous regions in CT brain images. The extracted features of the regions would provide a strong basis for the application of content-based medical image retrieval (CMBIR).

This paper introduces a theory for max-product systems by analyzing them as discrete-time nonlinear dynamical systems that obey a superposition of a weighted maximum type and evolve on nonlinear spaces which we call complete weighted lattices. Special cases of such systems have found applications in speech recognition as weighted finite-state transducers and in belief propagation on graphical models. Our theoretical approach establishes their representation in state and input-output spaces using monotone lattice operators, finds analytically their state and output responses using nonlinear convolutions, studies their stability, and provides optimal solutions to solving max-product matrix equations. Further, we apply these systems to extend the Viterbi algorithm in HMMs by adding control inputs and model cognitive processes such as detecting audio and visual salient events in multimodal video streams, which shows good performance as compared to human attention.

In this paper we address the problem of online video abnormal event detection. A vast number of methods to automatically detect abnormal events in videos have been recently proposed. However, the majority of these recently proposed methods cannot attain online performance; in other words, they cannot detect events as soon as they occur. Thus there is a lack of methods specifically aimed to detect events in online fashion. In this paper, we propose to incorporate binary features to detect abnormal events in an online manner. This is based on the fact that binary features are well known to require short processing times, compared to double-precision features. The main contribution of this work is then at the feature extraction step. Our experiment results of our binary-based framework show that our proposed binary features help to reduce processing times for anomaly detection, while outperforming other online methods, in terms of detection accuracy.

Complex events essentially include human, scenes, objects and actions that can be summarized by visual attributes, so leveraging relevant attributes properly could be helpful for event detection. Many works have exploited attributes at image level for various applications. However, attributes at image level are possibly insufficient for complex event detection in videos due to their limited capability in characterizing the dynamic properties of video data. Hence, we propose to leverage attributes at video level (named as video attributes in this work), i.e., the semantic labels of external videos are used as attributes. Compared to complex event videos, these external videos contain simple contents such as objects, scenes and actions which are the basic elements of complex events. Specifically, building upon a correlation vector which correlates the attributes and the complex event, we incorporate video attributes latently as extra informative cues into the event detector learnt from complex event videos. Extensive experiments on a real-world large-scale dataset validate the efficacy of the proposed approach.