Speech synthesis integrated circuit device having variable frame rate capability

Joint variable frame rate and length analysis for speech recognition under adverse conditions

In this paper we investigate the use of variable frame rate (VFR) analysis in automatic speech recognition (ASR). First, we review VFR technique and analyze its behavior. It is experimentally shown that VFR improves ASR performance for signals with low signal-to-noise ratios since it generates improved acoustic models and substantially reduces insertion and substitution errors although it may increase deletion errors. It is also underlined that the match between the average frame rate and the number of hidden Markov model states is critical in implementing VFR. Secondly, we analyze an effective VFR method that uses a cumulative, weighted cepstral-distance criterion for frame selection and present a revision for it. Lastly, the revised VFR method is combined with spectral- and cepstral-domain enhancement methods including the minimum statistics noise estimation (MSNE) based spectral subtraction and the cepstral mean subtraction, variance normalization and ARMA filtering (MVA) process. Experiments on the Aurora 2 database justify that VFR is highly complementary to the enhancement methods. Enhancement of speech both facilitates the frame selection in VFR and provides de-noised speech for recognition.

We consider the problem of capturing distortions arising from changes in frame rate as part of Video Quality Assessment (VQA). Variable frame rate (VFR) videos have become much more common, and streamed videos commonly range from 30 frames per second (fps) up to 120 fps. VFR-VQA offers unique challenges in terms of distortion types as well as in making non-uniform comparisons of reference and distorted videos having different frame rates. The majority of current VQA models require compared videos to be of the same frame rate, but are unable to adequately account for frame rate artifacts. The recently proposed Generalized Entropic Difference (GREED) VQA model succeeds at this task, using natural video statistics models of entropic differences of temporal band-pass coefficients, delivering superior performance on predicting video quality changes arising from frame rate distortions. Here we propose a simple fusion framework, whereby temporal features from GREED are combined with existing VQA models, towards improving model sensitivity towards frame rate distortions. We find through extensive experiments that this feature fusion significantly boosts model performance on both HFR/VFR datasets as well as fixed frame rate (FFR) VQA databases. Our results suggest that employing efficient temporal representations can result much more robust and accurate VQA models when frame rate variations can occur.

In the first part of this paper, we derived a source model describing the relationship between bits, distortion, and quantization step size for transform coders. Based on this source model, a variable frame rate coding algorithm is developed. The basic idea is to select a proper picture frame rate to ensure a minimum picture quality for every frame. Because our source model can predict approximately the number of coded bits when a certain quantization step size is used, we could predict the quality and bits of coded images without going through the entire real-coding process. Therefore, we could skip the right number of picture frames to accomplish the goal of constant image quality. Our proposed variable frame rate coding schemes are simple but quite effective as demonstrated by simulation results. The results of using another variable frame rate scheme, Test Model for H.263 (TMN-5), and the results of using a fixed frame rate coding scheme, Reference Model 8 for H.261 (RM8), are also provided for comparison.

This paper addresses the issue of data compression in distributed speech recognition on the basis of a variable frame rate and length analysis method. The method first conducts frame selection by using a posteriori signal-to-noise ratio weighted energy distance to find the right time resolution at the signal level, and then increases the length of the selected frame according to the number of non-selected preceding frames to find the right time-frequency resolution at the frame level. It produces high frame rate and small frame length in rapidly changing regions and low frame rate and large frame length for steady regions. The method is applied to scalable source coding in distributed speech recognition where the target bitrate is met by adjusting the frame rate. Speech recognition results show that the proposed approach outperforms other compression methods in terms of recognition accuracy for noisy speech while achieving higher compression rates.

The consonant and transition regions around the vowel onset point hear important contextual cues for distinguishing between different classes of consonant vowel units. It is important to model these regions for improved speech recognition rates. Conventional fixed frame rate analysis does not distinguish between the different regions of the speech signal. In this paper we detail an approach to emphasize the consonant and transition regions in speech sounds using a variable frame rate, anchored around the vowel onset point. The usefulness of this technique is demonstrated on an isolated consonant-vowel recognition task for Indian languages.

For gamers, high frame rates are important for a smooth visual display and good quality of experience (QoE). However, high frame rates alone are not enough as variations in the frame display times can degrade QoE even as the average frame rate remains high. While the impact of steady frame rates on player QoE is fairly well-studied, the effects of frame rate variation is not. This paper presents a 33-person user study that evaluates the impact of frame rate variation on users playing three different computer games. Analysis of the results shows average frame rate alone is a poor predictor of QoE, and frame rate variation has a significant impact on player QoE. While the standard deviation of frame times is promising as a general predictor for QoE, frame time standard deviation may not be accurate for all individual games. However, 95% frame rate floor -– the bottom 5% of frame rates the player experiences –- appears to be an effective predictor of both QoE overall and for the individual games tested.

We present a variable frame rate LPC (Linear Predictive Coding) vocoder based on a normalized exact-least squares ladder-form analysis algorithm by Lee and Morf. This algorithm works on a sample by sample basis, hence no a priori frame rate has to be preselected. As a byproduct of this algorithm a log-likelihood variable is computed that can be used to detect discontinuities in the speech signal such as plosive sounds, pitch pulses, glottal stops and other underlying transitions. Based on this algorithm a complete design of a vocoder is presented, that allows variable frame rates, pitch synchronous speech analysis/synthesis. Such vocoders are of interest in digital communication systems, especially packet oriented systems, or digitally stored speech based systems. Computer simulations of this vocoder using a digitized speech data base are presented, and several alternative implementations are discussed.

This paper presents a modulation scheme in the time domain based on On-Off-Keying and proposes various compatible supports for different types of image sensors. The content of this article is a sub-proposal to the IEEE 802.15.7r1 Task Group (TG7r1) aimed at Optical Wireless Communication (OWC) using an image sensor as the receiver. The compatibility support is indispensable for Image Sensor Communications (ISC) because the rolling shutter image sensors currently available have different frame rates, shutter speeds, sampling rates, and resolutions. However, focusing on unidirectional communications (i.e., data broadcasting, beacons), an asynchronous communication prototype is also discussed in the paper. Due to the physical limitations associated with typical image sensors (including low and varying frame rates, long exposures, and low shutter speeds), the link speed performance is critically considered. Based on the practical measurement of camera response to modulated light, an operating frequency range is suggested along with the similar system architecture, decoding procedure, and algorithms. A significant feature of our novel data frame structure is that it can support both typical frame rate cameras (in the oversampling mode) as well as very low frame rate cameras (in the error detection mode for a camera whose frame rate is lower than the transmission packet rate). A high frame rate camera, i.e., no less than 20 fps, is supported in an oversampling mode in which a majority voting scheme for decoding data is applied. A low frame rate camera, i.e., when the frame rate drops to less than 20 fps at some certain time, is supported by an error detection mode in which any missing data sub-packet is detected in decoding and later corrected by external code. Numerical results and valuable analysis are also included to indicate the capability of the proposed schemes.

Extensive research has been devoted to robustness in the presence of various types and degrees of environmental noise over the past several years, however this remains one of the main problems facing automatic speech recognition systems. This paper describes a new variable frame rate analysis technique, based upon searching a predefined lookahead interval for the next frame position that maximizes the firstorder difference of the log energy (ΔE) between the consecutive frames. The application of this novel technique to noise-robust ASR front-end processing is also reported. In comparison with existing variable frame rate methods in the literature, the proposed energy search approach is simpler and achieves similar recognition accuracy improvements at lower complexity. Experimental work on the Aurora II connected digits database reveals that the proposed front-end, together with cumulative distribution mapping, achieves average digit recognition accuracies of 78.32% for a model set trained from clean data and 89.95% for a model set trained from data with multiple noise conditions, representing 6.1% and 2.3% reductions in word error rates respectively over a cumulative distribution mapping baseline.

In previous work, a speech enhancement algorithm based on phase opponency and a periodicity measure (MPO-APP) was developed for speech recognition. Axiomatic thresholds were used in the MPO-APP regardless of the signal-to-noise ratio (SNR) of the corrupted speech or any characterization of the noise. The current work developed an algorithm for adjusting the threshold in the MPO-APP based on the SNR and whether the speech signal is clean, corrupted by aperiodic noise or corrupted with noise with periodic components. In addition, variable frame rate (VFR) analysis has been incorporated so that dynamic regions in the speech signal are more heavily sampled than steady-state regions. The result is a 2-stage algorithm that gives superior performance to the previous MPO-APP, and to several other state-of-the-art speech enhancement algorithms. Index Terms: Speech enhancement, robust speech recognition, SNR estimation, variable frame rate analysis, phase opponency.

Memory bandwidth and on-chip memory requirements are critical issues in motion estimation (ME) implementations for video compression. The H.264/AVC scalable extension (SVC) provides variable frame rate and resolution video in a compressed digital sequence with interlayer prediction, which complicates the problems of limited memory bandwidth and on-chip memory size. In this paper, an ME algorithm is proposed for the hardware encoder design of SVC that meets memory bandwidth and on-chip memory requirements. Clustered motion estimation and coding sequence reordering at macroblock and frame level processing are proposed. Compared with existing algorithms, the proposed algorithm has a 49.20% lower external memory bandwidth and reduces the on-chip memory requirement by 80.45% with video quality enhancements of up to 0.087, 0.090, 0.078, and 0.070dB for four-layer (FullHD-HD-D1-CIF) spatial scalability, respectively.

Variable frame rate (VFR) analysis is a technique used in speech processing and recognition for discarding frames that are too much alike. The article introduces a new method for VFR. Instead of calculating the distance between frames, the norm of the derivative parameters is used in deciding to retain or to discard a frame, informal inspection of speech spectrograms shows that this new method puts more emphasis on the transient regions of the speech signal. Experimental results with a hidden Markov model (HMM) based system show that the new method outperforms the classical method. >

Present methods for quantitative measurement of cerebral perfusion during neurosurgical operations require additional technology for measurement, data acquisition, and processing. This study used conventional fluorescence video angiography--as an established method to visualize blood flow in brain vessels--enhanced by a quantifying perfusion software tool. For these purposes, the fluorescence dye indocyanine green is given intravenously, and after activation by a near-infrared light source the fluorescence signal is recorded. Video data are analyzed by software algorithms to allow quantification of the blood flow. Additionally, perfusion is measured intraoperatively by a reference system. Furthermore, comparing reference measurements using a flow phantom were performed to verify the quantitative blood flow results of the software and to validate the software algorithm. Analysis of intraoperative video data provides characteristic biological parameters. These parameters were implemented in the special flow phantom for experimental validation of the developed software algorithms. Furthermore, various factors that influence the determination of perfusion parameters were analyzed by means of mathematical simulation. Comparing patient measurement, phantom experiment, and computer simulation under certain conditions (variable frame rate, vessel diameter, etc.), the results of the software algorithms are within the range of parameter accuracy of the reference methods. Therefore, the software algorithm for calculating cortical perfusion parameters from video data presents a helpful intraoperative tool without complex additional measurement technology.

Tree-structured probabilistic models admit simple, fast inference. However they are not well suited to phenonena such as occlusion, where multiple components of an object may disappear simultaneously. We address this problem with mixtures of trees, and demonstrate an efficient and compact representation of this mixture, which admits simple learning and inference algorithms. We use this method to build an automated tracker for Muybridge sequences of a variety of human activities. Tracking is difficult, because the temporal dependencies rule out simple inference methods. We show how to use our model for efficient inference, using a method that employs alternate spatial and temporal inference. The result is a cracker that (a) uses a very loose motion model, and so can track many different activities at a variable frame rate and (b) is entirely, automatic.