Transparent bitrate estimation for perceptual audio coding

We propose a frame loss concealment technique for decoders compatible with MPEG advanced audio coding (AAC). The spectral information of the lost frame is estimated in the modified discrete cosine transform (MDCT) domain via efficient techniques that are tailored to individual source signal components: in noise-like spectral bins the MDCT coefficients are obtained by shaped-noise insertion, while coefficients in tone-dominant bins are estimated by frame interpolation followed by a refinement procedure so as to optimize the fit of the concealed frames with neighboring frames. Experimental results demonstrate that the proposed technique offers performance superior to techniques adopted in commercial AAC decoders.

This paper presents a novel parametric vector quantization (PVQ) scheme as the secondary encoding to code perceptually salient percussive sounds such as drums in the time domain. It is deployed to improve the quality of service in the case of packet losses during percussive events. As a generalization and an improvement of our earlier system, the new scheme can achieve a better balance between bandwidth efficiency and error robustness. The proposed coding technique has been implemented with the MPEG-2 AAC (advanced audio coding) frame structure. Experimental results with music samples have shown the effectiveness of the proposed scheme.

In this paper, we develop a 3D audio reproduction scheme for the purpose of delivering audio over IP networks. In this scheme, audio streams constructed at a server are composed of the TCP/IP header followed by multi-channel audio data compressed by MPEG advanced audio coding (AAC), and the decoded audio signals are played out on a stereo loudspeaker system at the client. Since the audio source is recorded by a multi-channel microphone but the playout is dedicated to stereo speakers, the quality mismatch between the multi-channel and the stereo system should be overcome. As a potential solution, we first investigate the effect of 3D audio processing on the audio quality at the client by applying a head-related transfer function (HRTF). Next, a crosstalk cancellation process is applied to the audio with 3D effects in order to improve the immersion of the processed 3D effects on a stereo loudspeaker system. Finally, we evaluate the performance of the 3D audio reproduction system in terms of the identification of an audio source and quality comparison before and after applying the crosstalk cancellation technique.

Audio compression systems exploit periodicity in signals to remove inter-frame redundancies via the long term prediction (LTP) tool. This simple tool capitalizes on the periodic component of the waveform by selecting a past segment as the basis for prediction of the current frame. However, most audio signals are polyphonic in nature, containing a mixture of several periodic components. While such polyphonic signals may themselves be periodic with overall period equaling the least common multiple of the individual component periods, the signal rarely remains sufficiently stationary over the extended period, rendering the LTP tool suboptimal. Instead of seeking a past segment that represents a “compromise” for incompatible component periods, we propose a more complex filter that predicts every periodic component of the signal from its immediate history, and this is achieved by cascading LTP filters, each corresponding to individual periodic component. We also propose a recursive “divide and conquer” technique to estimate parameters of all the LTP filters. We then demonstrate the effectiveness of cascaded LTP in two distinct settings of the ultra low delay Bluetooth Subband Codec and the MPEG Advanced Audio Coding (AAC) standard. In MPEG AAC, we specifically adapt the cascaded LTP parameter estimation to take into account the perceptual distortion criteria, and also propose a low decoder complexity variant. Objective and subjective results for all the settings validate the effectiveness of the proposal on a variety of polyphonic signals.

We describe an efficient implementation for converting MPEG-2/MPEG-4 Advanced Audio Coding (AAC) encoded data to Dolby Digital AC-3. We describe many techniques to exploit the information in the AAC bitstream to simplify the AC-3 encoder. These techniques can be straightforwardly used in other transcoding schemes between different multimedia standards.

The notion of perceptual features is introduced for describing general music properties based on human perception. This is an attempt at rethinking the concept of features, aiming to approach the underlying human perception mechanisms. Instead of using concepts from music theory such as tones, pitches, and chords, a set of nine features describing overall properties of the music was selected. They were chosen from qualitative measures used in psychology studies and motivated from an ecological approach. The perceptual features were rated in two listening experiments using two different data sets. They were modeled both from symbolic and audio data using different sets of computational features. Ratings of emotional expression were predicted using the perceptual features. The results indicate that (1) at least some of the perceptual features are reliable estimates; (2) emotion ratings could be predicted by a small combination of perceptual features with an explained variance from 75% to 93% for the emotional dimensions activity and valence; (3) the perceptual features could only to a limited extent be modeled using existing audio features. Results clearly indicated that a small number of dedicated features were superior to a "brute force" model using a large number of general audio features.

In this paper, we describe the implementation of the multi-channel AAC encoder system for digital audio broadcasting. The encoder system is based on MPEG-2/4 advanced audio coding (AAC) and is capable of real-time encoding up to 5.1 channel audio. To give a flexible functionality, it consists of multiple DSPs, IEC61937 and TCP/IP interface and 6 channel audio input facilities. The reference AAC decoder was implemented for verification test of the encoder. The encoder system is also integrated with the AAC streaming system for interoperation test. Through these tests, the encoder system is verified to be a good solution for high quality audio broadcasting.

MPEG-4 scalable lossless (SLS) coding is a unified solution for demands in high compression perceptual audio and high quality lossless audio. It provides a fine-grain scalable extension to the MPEG-4 advanced audio coding (AAC) perceptual audio coder up to fully lossless reconstruction. It is observed that the quality of SLS coded audio is still far from optimal when the core bitrate is low. With this observation, an adaptive bit-plane scanning (ABPS) method is proposed in this paper. In ABPS, the bit-plane scanning order is adaptive to the energy distribution of the signal to be coded. The results show that with ABPS, the perceptual quality of the audio under low core bitrate scenario is significantly improved without any extra payload.

Several end-to-end deep learning approaches have been recently presented which extract either audio or visual features from the input images or audio signals and perform speech recognition. However, research on end-to-end audiovisual models is very limited. In this work, we present an end-to-end audiovisual model based on residual networks and Bidirectional Gated Recurrent Units (BGRUs). To the best of our knowledge, this is the first audiovisual fusion model which simultaneously learns to extract features directly from the image pixels and audio waveforms and performs within-context word recognition on a large publicly available dataset (LRW). The model consists of two streams, one for each modality, which extract features directly from mouth regions and raw waveforms. The temporal dynamics in each stream/modality are modeled by a 2-layer BGRU and the fusion of multiple streams/modalities takes place via another 2-layer BGRU. A slight improvement in the classification rate over an end-to-end audio-only and MFCC-based model is reported in clean audio conditions and low levels of noise. In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.

Analysis of communication effectiveness is an important task for understanding business outcomes. Prior research has shown that voice data can be used to predict communication effectiveness. However, to our knowledge, no existing studies have used both vocal and verbal cues to predict conversation outcomes in naturally occurring, dyadic business interactions. We use recorded audio calls collected from a partnering Fortune 500 firm that captures conversations between inside salespeople and business customers. Analysis of communication effectiveness is accomplished by transcribing these audio files and subsequently segmenting each conversation into customer and salesperson speaker turns to enable extraction of audio features and text embeddings for each speaker turn. All the speaker turns from the same conversation can be treated as a time series data, which can be modeled by temporal models, like LSTM or transformers. In this paper we propose that a multimodal transformer network (MTN) can capture the importance of different speaker turns and can be used to effectively predict the outcome of the call using both audio and text features. Results from the proposed model outperform current state-of-the-art results and reveal that text features offer superior outcome prediction compared to audio features.

Enhanced aacPlus is an audio codec which is composed of advanced audio coding (AAC), spectral band replication (SBR), and parametric stereo (PS) for efficient audio coding at low bit rates. We propose a new coding scheme for lossless bit rate reduction of PS in enhanced aacPlus. We first determine the optimal contexts for context-based coding of quantized stereo parameter indexes in PS. Then we propose a coding scheme which is based on context-based coding using the determined optimal contexts in conjunction with arithmetic coding. The proposed scheme has normal and memory-reduced versions, both of which are shown to guarantee significant and consistent lossless bit rate reduction of PS. The saved bits by the proposed scheme can be used for the other modules of enhanced aacPlus, AAC and SBR.

AV-FDTI: Audio-visual fusion for drone threat identification

In this paper, we propose a novel Siamese Adversarial Network (SAN) architecture for automatic speech recognition, which aims at solving the difficulty of fuzzy audio recognition. Specifically, SAN constructs two sub-networks to differentiate the audio feature input and then introduces a loss to unify the output distribution of these sub-networks. Adversarial learning enables the network to capture more essential acoustic features and helps the models achieve better performance when encountering fuzzy audio input. We conduct numerical experiments with the SAN model on several datasets for the automatic speech recognition task. All experimental results show that the siamese adversarial nets significantly reduce the character error rate (CER). Specifically, we achieve a competitive 4.37 CER result on the AISHELL-1 dataset without any language models. To reveal the generality of the siamese adversarial net, we also conduct experiments on the phoneme recognition task, which also shows the superiority of the siamese adversarial network.

Most existing audio steganographic methods embed secret messages according to a pseudorandom number generator, thus some auditory sensitive parts in cover audio, such as mute or near-mute segments, will be contaminated, which would lead to poor perceptual quality and may introduce some detectable artifacts for steganalysis. In this paper, we propose a novel adaptive audio steganography in the time domain based on the advanced audio coding (AAC) and the Syndrome-Trellis coding (STC). The proposed method firstly compresses a given wave signal into AAC compressed file with a high bitrate, and then obtains a residual signal by comparing the signal before and after AAC compression. According to the quantity and sign of the residual signal, \(\pm 1\) embedding costs are assigned to the audio samples. Finally, the STC is used to create the stego audio. The extensive results evaluated on 10,000 music and 10,000 speech audio clips have shown that our method can significantly outperform the conventional \(\pm 1\) LSB based steganography in terms of security and audio quality.

Given the popularity of audio and video applications, compressed audio has become an important carrier of covert communication on the Internet. Many novel compressed audio steganography schemes have emerged that offer good hiding capability and aural concealment. In this paper, a universal steganalysis scheme called MultiSpecNet is proposed to detect steganography based on multiple embedding domains (advanced audio coding (AAC) and MPEG-1 Audio Layer III (MP3)), which are currently the two most popular compressed audio standards. The basic idea is that modification of either domain by a steganography scheme will change the time-frequency relationship of the audio signal after decoding. The proposed approach adopts the spectrogram as the input feature to extract richer information. DeepResNet is used to learn the distinguishing feature representations, and multiscale spectrograms are used to enrich the feature diversity. The experimental results show that the proposed scheme is effective at detecting different steganography schemes based on the AAC and MP3 embedding domains. The detection accuracy of the proposed scheme is higher than that achieved by other state-of-the-art schemes. Using spectrograms as the input, DeepResNet achieves better performance than schemes using quantized modified discrete cosine transform (MDCT) coefficients and mel-spectrogram, although the quantized MDCT coefficient is the parameter modified by the steganography schemes directly and mel-spectrogram is very popular and effective for general audio signal analysis. To the best of our knowledge, this work is the first audio steganalysis scheme that can detect multiple steganography schemes in both the MP3 and AAC embedding domains. The method proposed in this paper can be extended to audio steganalysis for other codecs or for audio forensics purposes.