A Model of Structural Semantics of Image

Identifying different audio segments in videos is the first step for many important tasks such as event detection and speech transcription. Approaches using Mel-Frequency Cepstral coefficients (MFCCs) with Gaussian mixture models (GMMs) and hidden Markov models (HMMs) perform reasonably well in stationary conditions but do not scale to a broad range of environmental conditions. This paper focuses on the audio segmentation in broadcast soccer videos into audio classes such as Silence, Speech Only, Speech Over Crowd, Crowd Only and Excited, with an alternative feature set which is simplistic as well as robust to changes in the environment conditions. Support Vector Machines (SVMs), Neural Networks and Random Forest are used for the classification. The accuracy achieved with SVMs, Neural Networks and Random Forest are 83.80%, 86.07%, and 88.35% respectively. The proposed features and Random Forest classifier are found to achieve better accuracy compared to the other classifiers.

In this work, we address the problem of improving an automatic speech recognition (ASR) system. We want to efficiently model long-term semantic relations between words and introduce this information through a semantic model. We propose neural network (NN) semantic models for rescoring the N-best hypothesis list. These models use two types of representations as part of DNN input features: static word embeddings (from word2vec) and dynamic contextual embeddings (from BERT). Semantic information is computed thanks to these representations and used in the hypothesis pair comparison mode. We perform experiments on the publicly available dataset TED-LIUM. Clean speech and speech mixed with real noise are experimented, according to our industrial project context. The proposed BERT-based rescoring approach gives a significant improvement of the word error rate (WER) over the ASR system without rescoring semantic models under all experimented conditions and with n-gram and recurrent NN language model (Long Short-Term model, LSTM).

There has been remarkable progress in the field of Semantic segmentation in recent years. Yet, it remains a challenging problem to apply segmentation to the video-based applications. Videos usually involve significantly larger volume of data compared to images. Particularly, a video contains around 30 frames per second. Segmentation of the similar frames unnecessarily adds to the time required for segmentation of complete video. In this paper, we propose a contour detection-based approach for detection of salient frames for faster semantic segmentation of videos. We propose to detect the salient frames of the video and pass only the salient frames through the segmentation block. Then, the segmented labels of the salient frames are mapped to the non-salient frames. The salient frame is defined by the variation in the pixel values of the background subtracted frames. The background subtraction is done using MOG2 background subtractor algorithm for background subtraction in various lighting conditions. We demonstrate the results using the Pytorch model for semantic segmentation of images. We propose to concatenate the semantic segmentation model to our proposed framework. We evaluate our result by comparing the time taken and the mean Intersection over Union (mIoU) for segmentation of the video with and without passing the video input through our proposed framework. We evaluate the results of Saliency Detection Block using Retention and Condensation ratio as the quality metrics.

Concept classification for consumer videos is a challenging task considering the co-occurrence of a variety objects and arbitrary motions in video segments. In this paper, we present a novel video concept classification framework with random-sampling-based spatialtemporal features. Short-term random-sampled point tracks are obtained within video segments. The spatial-temporal features are extracted from these tracks. Concept codebooks are constructed using Multiple Instance Learning upon the spatial-temporal features. The SVM classifiers are trained over codebook-based histograms for an online concept detection. We performed experiments on a video database taken from YouTube. The experimental results demonstrate that the consumer videos can be efficiently assigned concept labels by our approach.

Caption graphically superimposed in video frames can provide important indexing information. The automatic detection and recognition of video captions can be of great help in querying topics of interest in digital news library. To detect the caption from video sequence, we present algorithms based on fuzzy clustering neural networks. Since neural networks have the capabilities of learning and self-organizing and parallel computing mechanism, with the great increasing of digital images and video databases, neural networks based techniques become more efficient and popular tools for multimedia processing. Experimental results show that our caption detection scheme is effective and robust.

In this work, we present a new semantic segmentation model for historical city maps that surpasses the state of the art in terms of flexibility and performance. Research in automatic map processing is largely focused on homogeneous corpora or even individual maps, leading to inflexible algorithms. Recently, convolutional neural networks have opened new perspectives for the development of more generic tools. Based on two new maps corpora, the first one centered on Paris and the second one gathering cities from all over the world, we propose a method for operationalizing the figuration based on traditional computer vision algorithms that allows large-scale quantitative analysis. In a second step, we propose a semantic segmentation model based on neural networks and implement several improvements. Finally, we analyze the impact of map figuration on segmentation performance and evaluate future ways to improve the representational flexibility of neural networks. To conclude, we show that these networks are able to semantically segment map data of a very large figurative diversity with efficiency.

Machine learning is the study of algorithms and models for computing systems to do tasks based on pattern identification and inference. When it is difficult or infeasible to develop an algorithm to do a particular task, machine learning algorithms can provide an output based on previous training data. A well-known machine learning model is deep learning. The most recent deep learning models are based on artificial neural networks (ANN). There exist several types of artificial neural networks including the feedforward neural network, the Kohonen self-organizing neural network, the recurrent neural network, the convolutional neural network, the modular neural network, among others. This article focuses on convolutional neural networks with a description of the model, the training and inference processes and its applicability. It will also give an overview of the most used CNN models and what to expect from the next generation of CNN models.

With the advancement of multimedia technology and the internet, numerous applications have arisen which require the storage and retrieval of large image and video databases. A novel method (Eigenwavelet) was developed to retrieve images from a large heterogeneous image database upon a user-specified query. The queries are in the form of an image(s) that the user seeks to find similar matches to in the database. Using the queries, an efficient algorithm was developed which decomposed each image in the database using wavelet packet analysis. Along each node of the packet tree, Principal Component Analysis was applied to the database images after wavelet packet decomposition, and a set of eigenvectors were generated for each node of the packet tree. To search the image database, the query images are projected onto these eigenvectors (Eigenwavelet coefficients). A distance metric is computed between the projections of the queries and the projections of the images in the database onto the eigenwavelets. Those images with minimal distance (L1) are retrieved in response to a unique query set. Simulations with a heterogeneous image database suggest the Eigenwavelet method of image retrieval is a robust and computationally tractable method of retrieving images with a probability of detection >.8.

Ultrasound video segmentation gains increasing attention in clinical practice due to the redundant dynamic references in video frames. However, traditional convolutional neural networks have a limited receptive field and transformer-based networks are unsatisfactory in constructing long-term dependency from the perspective of computational complexity. This bottleneck poses a significant challenge when processing longer sequences in medical video analysis tasks using available devices with limited memory. Recently, state space models (SSMs), famous by Mamba, have exhibited linear complexity and impressive achievements in efficient long sequence modeling, which have developed deep neural networks by expanding the receptive field on many vision tasks significantly. Unfortunately, vanilla SSMs failed to simultaneously capture causal temporal cues and preserve non-casual spatial information. To this end, this paper presents a Video Vision Mamba-based framework, dubbed as Vivim, for ultrasound video segmentation tasks. Our Vivim can effectively compress the long-term spatiotemporal representation into sequences at varying scales with our designed Temporal Mamba Block. We also introduce an improved boundary-aware affine constraint across frames to enhance the discriminative ability of Vivim on ambiguous lesions. Extensive experiments on thyroid segmentation in ultrasound videos, breast lesion segmentation in ultrasound videos, and polyp segmentation in colonoscopy videos demonstrate the effectiveness and efficiency of our Vivim, superior to existing methods. The code and dataset are available at: https://github.com/scott-yjyang/Vivim.

Machine Learning (ML) and in particular Neural Networks (NN) are currently being used for different image/video processing, speech recognition and other tasks. The goal of supervised NN is to classify raw input data according to the patterns learned from an input training set. Training and validation of NN is very computationally intensive. In this paper we present an NN infrastructure to accelerate model training, specifically tuning of hyper-parameters, and model inference or prediction using distributed systems techniques. By accelerating model training, we give ability to researchers to obtain a large set of potential models to use and compare in a shorter amount of time. Automating this process not only reduces development time but will provide an easy means for comparing results for different classifiers and/or different hyper-parameters. With a single set of training data, our application will run different classifiers on different servers each running models with tweaked hyper-parameters. To give more control over the automation process the degree by which these hyper-parameters will be tweaked can be set by the user prior to running. The prediction step in most ML algorithms can also be very slow, especially on video prediction where current systems calculate their inference predictions on an entire input video, and then evaluate accuracy based on human annotations of objects of interest within the video. To reduce this bottleneck, we also accelerate and distribute this important part of the ML algorithm development. This process involves sending to each server the data; the model weights; and human annotations within the video segmentation. Our efficient distribution of input frames among each node greatly reduced the amount taken for testing and to generate accuracy metrics. To make our implementation robust to common distributed system failures (servers going down, lost of communication among some nodes, and others) we use heartbeat/gossip style protocol for failure detection and recovery. We tested our infrastructure for fast testing and inference of ML on video with data generated by a group of marine biologists researching the behavior of different marine species on the deep sea. Results show that by using our infrastructure times improved by a factor of 15.KeywordsMachine learningNeural networkDistributed computingHPC

Rapid increase in video databases has forced the industry to have efficient and effective frameworks for video retrieval and indexing. Video segmentation into scenes is widely used for video summarization, partitioning, indexing and retrieval. In this paper, we propose a framework for scene detection mainly based on entropy and Speeded Up Robust Features (SURF) features. First, we detect the fade and abrupt boundaries based on frame entropy analysis and SURF features matching. Fade boundaries are smart indication of scenes beginning or ending in many videos and dramas, and are detected by frame entropy analysis. Before abrupt boundary detection, unnecessary frames which are obviously not abrupt boundaries, such as blank screens, high intensity influenced images, sliding credits, are removed. Candidate boundaries are detected to make SURF features efficient for abrupt boundary detection, and SURF features between candidate boundaries and their adjacent frames are used to detect the abrupt boundaries. Second, key frames are extracted from abrupt shots. We evaluate our key frame extraction with other famous algorithms and show the effectiveness of the key frames. Finally, scene boundaries are detected using sliding window of size K over the key frames in temporal order. In experimental evaluation on the TRECVID-2007 shot boundary test set, the algorithm for shot boundary achieves substantial improvements over state-of-the-art methods with the precision of 99% and the recall of 97.8%. Experimental results for video segmentation into scenes are also promising, compared to famous state-of-the-art techniques.

Video caption detection and extraction is an important step for information retrieval in video databases. In this paper, we extract text information in video by fully utilizing the temporal information contained in the video. First we create a binary abstract sequence from a video segment. By analyzing the statistical pixel changes in the sequence, we can effectively locate the (dis)appealing frames of captions. Finally we extract the captions to create a summary of the video segment.

The article suggests an algorithm of graphical image segmentation. The suggested algorithm uses a neural network to identify one particular pixel as belonged to the certain segment of an image. As a segmentation method, is used a method of growing areas based on comparing the nearest neighbors of one particular pixel. To make a decision about the similarity of two pixels, a three-layer perceptron is used. Three RGB color components are compared during processing. Thus, there are 6 neurons in the input layer of the neural network, namely 3 for the RGB component of the first pixel and 3 for the RGB component of the second one. In a specific implementation there are 50 neurons in the middle layer of the neural network. In the output layer of the neural network there are 2 neurons that represent similarity or difference of the comparing pixels. A training set of the neural network is formed using a specially generated impulse noise. There is a linear congruent generator of pseudo-random number used for noise generation. This generator is used to generate both the color and the coordinates of the noisy pixel. To form a training set, the certain number of noisy pixels is generated. In the article, this number is 10% of all pixels in the image. Then, for each damaged pixel, the training sets are formed so that all the nearest neighbors are considered to be in different clusters with a damaged pixel. A computer experiment was carried out both in automatic mode and in interactive one. The results of the experiment have shown that the algorithm provides fairly good training neural network for image segmentation without involving additional images.

We present a novel approach towards web video classification and recounting that uses video segments to model an event. This approach overcomes the limitations faced by the classical video-level models such as modeling semantics, identifying informative segments in a video and background segment suppression. We posit that segment-based models are able to identify both the frequently-occurring and rarer patterns in an event effectively, despite being trained on only a fraction of the training data. Our framework employs a discriminative approach to optimize our models in distributed and data-driven fashion while maintaining semantic interpretability. We evaluate the effectiveness of our approach on the challenging TRECVID MEDTest 2014 dataset. We demonstrate improvements in recounting and classification, particularly in events characterized by inherent intra-class variations.

Context.The characterization of the interior of an exoplanet is an inverse problem. The solution requires statistical methods such as Bayesian inference. Current methods employ Markov chain Monte Carlo (MCMC) sampling to infer the posterior probability of the planetary structure parameters for a given exoplanet. These methods are time-consuming because they require the evaluation of a planetary structure model ~105times.Aims.To speed up the inference process when characterizing an exoplanet, we propose to use conditional invertible neural networks to calculate the posterior probability of the planetary structure parameters.Methods.Conditional invertible neural networks (cINNs) are a special type of neural network that excels at solving inverse problems. We constructed a cINN following the framework for easily invertible architectures (FreIA). This neural network was then trained on a database of 5.6 × 106internal structure models to recover the inverse mapping between internal structure parameters and observable features (i.e., planetary mass, planetary radius, and elemental composition of the host star). We also show how observational uncertainties can be accounted for.Results.The cINN method was compared to a commonly used Metropolis-Hastings MCMC. To do this, we repeated the characterization of the exoplanet K2-111 b, using both the MCMC method and the trained cINN. We show that the inferred posterior probability distributions of the internal structure parameters from both methods are very similar; the largest differences are seen in the exoplanet water content. Thus, cINNs are a possible alternative to the standard time-consuming sampling methods. cINNs allow infering the composition of an exoplanet that is orders of magnitude faster than what is possible using an MCMC method. The computation of a large database of internal structures to train the neural network is still required, however. Because this database is only computed once, we found that using an invertible neural network is more efficient than an MCMC when more than ten exoplanets are characterized using the same neural network.