Variable frame rate and length analysis for data compression in distributed speech recognition

Joint variable frame rate and length analysis for speech recognition under adverse conditions

In this paper we investigate the use of variable frame rate (VFR) analysis in automatic speech recognition (ASR). First, we review VFR technique and analyze its behavior. It is experimentally shown that VFR improves ASR performance for signals with low signal-to-noise ratios since it generates improved acoustic models and substantially reduces insertion and substitution errors although it may increase deletion errors. It is also underlined that the match between the average frame rate and the number of hidden Markov model states is critical in implementing VFR. Secondly, we analyze an effective VFR method that uses a cumulative, weighted cepstral-distance criterion for frame selection and present a revision for it. Lastly, the revised VFR method is combined with spectral- and cepstral-domain enhancement methods including the minimum statistics noise estimation (MSNE) based spectral subtraction and the cepstral mean subtraction, variance normalization and ARMA filtering (MVA) process. Experiments on the Aurora 2 database justify that VFR is highly complementary to the enhancement methods. Enhancement of speech both facilitates the frame selection in VFR and provides de-noised speech for recognition.

The consonant and transition regions around the vowel onset point hear important contextual cues for distinguishing between different classes of consonant vowel units. It is important to model these regions for improved speech recognition rates. Conventional fixed frame rate analysis does not distinguish between the different regions of the speech signal. In this paper we detail an approach to emphasize the consonant and transition regions in speech sounds using a variable frame rate, anchored around the vowel onset point. The usefulness of this technique is demonstrated on an isolated consonant-vowel recognition task for Indian languages.

In distributed speech recognition, vector quantization is used to reduce the number of bits for coding speech features at the user end in order to save energy for transmitting speech feature streams to remote recognizers and reduce data traffic congestion. We notice that the overall bit rate of the transmitted feature streams could be further reduced by not sending redundant frames that can be interpolated at the remote server from received frames. Interpolation introduces errors and may degrade speech recognition. This paper investigates the methods of selecting frames for transmission and the effect of interpolation on recognition. Experiments on a large vocabulary recognizer show that with spline interpolation, the overall frame rate for transmission can be reduced by about 50% with a relative increase in word error rate less than 5.2% for clean and noisy speech.

An integrated circuit device or chip digitally synthesizes human speech employing a linear predictive filter and a variable frame rate. The variable frame rate provides a more natural speech by slowing or speeding the frame rate for a particular application used in a system which constructs the speech data to be synthesized from allophone codes. The speech synthesis system incorporates a controller, such as a microprocessor, therein. The controller furnishes to the speech synthesizer a control signal (CTL 1, CTL 2) via an input (73), the control signal being used within the speech synthesizer to alter the timing signals, thereby altering the frame rate. The control signals are in binary code and are decoded by a decode and counter preset circuit (72). The decoded outputs load a counter (71) to the determined value, and the outputs of the counter and a programmable logic array are then decoded by a timing output decoder (74) to produce one of a plurality of possible signals indicating the frame speed for the frame just loaded.

In previous work, a speech enhancement algorithm based on phase opponency and a periodicity measure (MPO-APP) was developed for speech recognition. Axiomatic thresholds were used in the MPO-APP regardless of the signal-to-noise ratio (SNR) of the corrupted speech or any characterization of the noise. The current work developed an algorithm for adjusting the threshold in the MPO-APP based on the SNR and whether the speech signal is clean, corrupted by aperiodic noise or corrupted with noise with periodic components. In addition, variable frame rate (VFR) analysis has been incorporated so that dynamic regions in the speech signal are more heavily sampled than steady-state regions. The result is a 2-stage algorithm that gives superior performance to the previous MPO-APP, and to several other state-of-the-art speech enhancement algorithms. Index Terms: Speech enhancement, robust speech recognition, SNR estimation, variable frame rate analysis, phase opponency.

We consider the problem of capturing distortions arising from changes in frame rate as part of Video Quality Assessment (VQA). Variable frame rate (VFR) videos have become much more common, and streamed videos commonly range from 30 frames per second (fps) up to 120 fps. VFR-VQA offers unique challenges in terms of distortion types as well as in making non-uniform comparisons of reference and distorted videos having different frame rates. The majority of current VQA models require compared videos to be of the same frame rate, but are unable to adequately account for frame rate artifacts. The recently proposed Generalized Entropic Difference (GREED) VQA model succeeds at this task, using natural video statistics models of entropic differences of temporal band-pass coefficients, delivering superior performance on predicting video quality changes arising from frame rate distortions. Here we propose a simple fusion framework, whereby temporal features from GREED are combined with existing VQA models, towards improving model sensitivity towards frame rate distortions. We find through extensive experiments that this feature fusion significantly boosts model performance on both HFR/VFR datasets as well as fixed frame rate (FFR) VQA databases. Our results suggest that employing efficient temporal representations can result much more robust and accurate VQA models when frame rate variations can occur.

User performance in virtual environments with degraded visual conditions due to low frame rates is an interesting area of inquiry. Visual content shown in a low frame rate simulation has the quality of the original image, but persists for an extended period until the next frame is displayed (so-called high persistence-HP). An alternative, called low persistence (LP), involves displaying the rendered frame for a single display frame and blanking the screen while waiting for the next frame to be generated. Previous research has evaluated the usefulness of the LP technique in low frame rate simulations during a static target acquisition task. To gain greater knowledge about the LP technique, we have conducted a user study to evaluate user performance and learning during a dynamic target acquisition task. The acquisition task was evaluated under a high frame rate, (60 fps) condition, a traditional low frame rate HP condition (10 fps), and the experimental low frame rate LP technique. The task involved the acquisition of targets moving along several different trajectories, modeled after a shotgun trap shooting task. The results of our study indicate the LP condition approaches high frame rate performance within certain classes of target trajectories. Interestingly we also see that learning is consistent across conditions, indicating that it may not always be necessary to train under a visually high frame rate system to learn a particular task. We discuss implications of using the LP technique to mitigate low frame rate issues as well as its potential usefulness for training in low frame rate virtual environments.

In the first part of this paper, we derived a source model describing the relationship between bits, distortion, and quantization step size for transform coders. Based on this source model, a variable frame rate coding algorithm is developed. The basic idea is to select a proper picture frame rate to ensure a minimum picture quality for every frame. Because our source model can predict approximately the number of coded bits when a certain quantization step size is used, we could predict the quality and bits of coded images without going through the entire real-coding process. Therefore, we could skip the right number of picture frames to accomplish the goal of constant image quality. Our proposed variable frame rate coding schemes are simple but quite effective as demonstrated by simulation results. The results of using another variable frame rate scheme, Test Model for H.263 (TMN-5), and the results of using a fixed frame rate coding scheme, Reference Model 8 for H.261 (RM8), are also provided for comparison.

This paper presents a modulation scheme in the time domain based on On-Off-Keying and proposes various compatible supports for different types of image sensors. The content of this article is a sub-proposal to the IEEE 802.15.7r1 Task Group (TG7r1) aimed at Optical Wireless Communication (OWC) using an image sensor as the receiver. The compatibility support is indispensable for Image Sensor Communications (ISC) because the rolling shutter image sensors currently available have different frame rates, shutter speeds, sampling rates, and resolutions. However, focusing on unidirectional communications (i.e., data broadcasting, beacons), an asynchronous communication prototype is also discussed in the paper. Due to the physical limitations associated with typical image sensors (including low and varying frame rates, long exposures, and low shutter speeds), the link speed performance is critically considered. Based on the practical measurement of camera response to modulated light, an operating frequency range is suggested along with the similar system architecture, decoding procedure, and algorithms. A significant feature of our novel data frame structure is that it can support both typical frame rate cameras (in the oversampling mode) as well as very low frame rate cameras (in the error detection mode for a camera whose frame rate is lower than the transmission packet rate). A high frame rate camera, i.e., no less than 20 fps, is supported in an oversampling mode in which a majority voting scheme for decoding data is applied. A low frame rate camera, i.e., when the frame rate drops to less than 20 fps at some certain time, is supported by an error detection mode in which any missing data sub-packet is detected in decoding and later corrected by external code. Numerical results and valuable analysis are also included to indicate the capability of the proposed schemes.

The analysis of knee kinematics and kinematics during high-risk movements associated with anterior cruciate ligament (ACL) injury is essential to studying the mechanism of ACL injury. Motion capture at low frame rates may not always detect the actual peak values during high-speed movements. However, the knee kinetic differences between various frame rate measurements during high-risk movements have not been reported. The purpose of this study was to investigate whether 3D knee kinetics would be measured differently between frame rates such as a high frame rate (1200 Hz), 400 Hz, 240 Hz and the popular low frame rate (120 Hz). Knee kinetics during a single-leg drop landing and side-step cutting under different frame rates were repeatedly measured and statistically compared. Peak knee valgus and tibial internal rotation moments measured at a popular low frame rate were significantly lower than those measured at the other frame rates of 400 Hz and 1200 Hz. The peak anterior and superior forces measured at the highest (1200 Hz) frame rate were significantly higher than those measured at any other frame rates. In addition, the variations in the peak kinetic values were significantly larger at the lowest frame rate capture and trended toward being smaller at the higher frame rates. In conclusion, significant differences in the knee kinetics between the frame rates suggest that the high frequency capturing increases the accuracy of the knee kinetics measurement for high risk maneuvers for the study of ACL injury.

For gamers, high frame rates are important for a smooth visual display and good quality of experience (QoE). However, high frame rates alone are not enough as variations in the frame display times can degrade QoE even as the average frame rate remains high. While the impact of steady frame rates on player QoE is fairly well-studied, the effects of frame rate variation is not. This paper presents a 33-person user study that evaluates the impact of frame rate variation on users playing three different computer games. Analysis of the results shows average frame rate alone is a poor predictor of QoE, and frame rate variation has a significant impact on player QoE. While the standard deviation of frame times is promising as a general predictor for QoE, frame time standard deviation may not be accurate for all individual games. However, 95% frame rate floor -– the bottom 5% of frame rates the player experiences –- appears to be an effective predictor of both QoE overall and for the individual games tested.

Wireless sensor networks are data centric networks. Clustering of nodes is an energy efficient approach that improves the data rate of networks. E-LEACH is an energy efficient approach based on clustering. E-LEACH does not guarantee the even distribution of nodes in the clusters. Uneven size of clusters affects the load balancing of networks because TDMA schedule of nodes of smaller cluster is smaller than others that results more number of data frames per node. In this paper, a variable frame slot scheme is proposed with E-LEACH. Variable frame slots for clusters make network more load balanced. Simulation is carried out in NS-2 to analyze the node death rate and data units gathered with node death rate for E-LEACH with variable frame length. Variable frame slot scheme for clusters is also measured with the varying distance of base station from the field. Simulation results show that E_LEACH with variable frame length has an improvement of 7% in node death rate over E-LEACH and an improvement of 5% in data gathering with node death rate. Results suggest that variable frame length scheme improves the performance of E_LEACH scheme for Wireless Sensor Networks and have most significant result at base station located at 75m from the field.

Variable frame rate (VFR) analysis is a technique used in speech processing and recognition for discarding frames that are too much alike. The article introduces a new method for VFR. Instead of calculating the distance between frames, the norm of the derivative parameters is used in deciding to retain or to discard a frame, informal inspection of speech spectrograms shows that this new method puts more emphasis on the transient regions of the speech signal. Experimental results with a hidden Markov model (HMM) based system show that the new method outperforms the classical method. >

The frame rate (FR) of a video plays an important role in affecting the perceptual video quality. Most studies about the effect of FR on the video quality mainly focused on low frame rate, e.g. less than 30 frames per second (fps), at low resolutions like CIF or QCIF. As the video frame rate and resolution advance, we reconsider this issue and investigate the relationship between frame rate and the perceptual video quality under high frame rate and high resolution. In this paper, we discuss the impact of frame rate on the perceptual quality of High Definition (HD) video with high frame rates (up to 120 fps) considered. Firstly, we design and conduct subjective experiment to construct the video dataset, which includes video sequences at different frame rates and the corresponding mean opinion scores (MOS) which represent the perceptual video quality. Based on the MOS results, we analyze how perceptual video quality changes as frame rate varies among different video sequences and propose some meaningful findings. The video dataset will be made publicly available. We deem that this study will enrich video quality assessment and benefit the development of high frame rate and high definition video business.KeywordsVideo quality assessmentSubjective experienceHigh frame rateHigh definition