Music Deep Learning: A Survey on Deep Learning Methods for Music Processing

The following report presents some of the ongoing projects that are taking place in the group’s laboratory. One of the noteable characteristics of this group is the extensive research spectrum, the plurality of research areas that are being studied by it’s members, such as Music Information Retrieval, Signal Processing and New Interfaces for Musical Expression.

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years.

This paper describes the construction of a system called i-Ring that can generate a polyphonic ringtone based on a user's humming input. Algorithms used in the system for music transcription and chord generation include various techniques in speech and music signal processing, such as pitch tracking and dynamic programming, which are explained in the paper. Experimental results demonstrate the feasibility of i-Ring as a handy tool for computer assisted music composition for polyphonic ringtones

The digitization, analysis, and processing technology of music signals are the core of digital music technology. There is generally a preprocessing process before the music signal processing. The preprocessing process usually includes antialiasing filtering, digitization, preemphasis, windowing, and framing. Songs in the popular wav format and MP3 format on the Internet are all songs that have been processed by digital technology and do not need to be digitalized. Preprocessing can affect the effectiveness and reliability of the feature parameter extraction of music signals. Since the music signal is a kind of voice signal, the processing of the voice is also applicable to the music signal. In the study of adaptive wave equation inversion, the traditional full-wave equation inversion uses the minimum mean square error between real data and simulated data as the objective function. The gradient direction is determined by the cross-correlation of the back propagation residual wave field and the forward simulation wave field with respect to the second derivative of time. When there is a big gap between the initial model and the formal model, the phenomenon of cycle jumping will inevitably appear. In this paper, adaptive wave equation inversion is used. This method adopts the idea of penalty function and introduces the Wiener filter to establish a dual objective function for the phase difference that appears in the inversion. This article discusses the calculation formulas of the accompanying source, gradient, and iteration step length and uses the conjugate gradient method to iteratively reduce the phase difference. In the test function group and the recorded music signal library, a large number of simulation experiments and comparative analysis of the music signal recognition experiment were performed on the extracted features, which verified the time-frequency analysis performance of the wave equation inversion and the improvement of the decomposition algorithm. The features extracted by the wave equation inversion have a higher recognition rate than the features extracted based on the standard decomposition algorithm, which verifies that the wave equation inversion has a better decomposition ability.

Artificial intelligence-empowered music processing is a domain that involves the use of artificial intelligence technologies to enhance music analysis, understanding, and generation. This field encompasses a variety of tasks from music generation to music comprehension. In practical applications, the complexity of interwoven tasks, differences in data representation, scattered distribution of tool resources, and the threshold of professional music knowledge often become barriers that hinder developers from smoothly carrying out generative tasks. Therefore, it is essential to establish a system that can automatically analyze their needs and invoke appropriate tools to simplify the music processing workflow. Inspired by the recent success of Large Language Models (LLMs) in task automation, we have developed a system named LongMergent, which integrates numerous music-related tools and autonomous workflows to address user requirements. By granting users the freedom to effortlessly combine tools, this system provides a seamless and rich musical experience.

With the rapid development of information technology, deep learning shows great potential in the field of audio signal processing, especially in personalized music education, which can provide students with customized, real-time feedback learning experience. The purpose of this paper is to study the audio signal processing technology based on deep learning and explore its application and implementation in personalized music education. The article analyzes the main challenges and development trends of audio signal processing in current music education, focusing on the key technologies of deep learning in audio feature extraction, classification, enhancement, and timing data processing. This paper proposes an audio processing framework based on the combination of deep neural network (DNN), convolutional neural network (CNN) and recurrent neural network (RNN) for the implementation of audio signal analysis and processing in personalized music education system. The effectiveness of the proposed technique is verified through experiments, and the impact of different audio processing techniques on students' learning outcomes is analyzed. The application of deep learning in audio signal processing not only improves the degree of personalization of music learning, but also provides new ideas and technical support for the future development of music education.

Stacked auto-encoders based visual features for speech/music classification

In Music Information Retrieval (MIR), a general goal is to recognize times of novelty within music recordings. This includes estimating structural boundaries through the detection of changes in harmony, tempo, or instrumentation and identifying onsets of note and sound events by capturing changes in the music signal’s energy or spectral content. These tasks leverage novelty functions, which are one‑dimensional, time‑dependent functions characterized by sharp local maxima that indicate significant musical and acoustical changes. From a given music recording, novelty functions can be derived using a variety of methods, ranging from traditional signal‑processing techniques to modern data‑driven approaches, where they are often termed “activation functions.” In this tutorial, we explore the concept of novelty functions and some of their essential properties. We discuss methods to enhance these functions and improve their distinctive peak‑like structures. These improvements are crucial for simplifying the identification of specific musical events using post‑processing methods, from basic peak picking to more sophisticated approaches like periodicity analysis. We also assess novelty functions through commonly used metrics such as precision, recall, and F‑measure but with an emphasis on error tolerance. Aimed at Bachelor’s degree and beginning Master’s degree students with basic knowledge of signal processing and mathematics, this tutorial uses illustrative figures to clarify key concepts, thereby broadening its accessibility to a wider MIR audience and enriching their comprehension of this significant subject. Furthermore, Jupyter notebooks, including Python source code for the core algorithms and audio examples that allow for reproducing the tutorial’s figures, are provided at https://github.com/groupmm/edu_novfct.

Chroma features are a popular tool in musical signal processing and information retrieval tasks, providing a compact representation of the tonal content of a piece of music. A variety of approaches to chroma estimation have been proposed, most of which rely on the summation of related frequency partials. However, frequency partials may be incorrectly assigned due to the log/linear relationship of frequency and pitch. Variations of chroma employing overtone suppression strategies are found in the literature. We propose a compositional model of chroma, which considers a coarse modelling of the effects of overtones in the expected chroma vectors of single notes. Synthetic chord recognition experiments indicate the usefulness of the proposed approach.

In the last few years various musicological studies on the Byzantine Chant (BC) have been made via computer analysis. Due to the BC vocal nature and the pitch variability of the BC notes during the performance, the value of a music interval between two such notes in a certain BC mode - “echos”, is ambiguous. Because of that pitch variability a new method of BC intervals measurement is proposed, based on statistical processing of the pitch track that can be done after a proper pitch analysis of the BC recordings. The music scale of a particular BC echos is computed from the histogram of the pitch values by applying an algorithm that calculates a Gaussian Mixture Model approximating the music scale of that BC echos. This new method is applied, as an example, to ten BC recordings - performances of the Cherubic Hymn composed in the fourth echos (aghia) by Petros Lampadarios. The recordings come from the DAMASKINOS Corpus of University of Athens and the analysis results are in agreement with both the theoretical values of the Patriarchal Committee 1881 (approximation of up to 1 moria) and the findings in other similar studies. This method of analysis can be a base for both musicological analysis and music signal processing.

Detecting multiple pitches (F0s) and segregating musical instrument lines from monaural recordings of contrapuntal polyphonic music into separate tracks is a difficult problem in music signal processing. Applications include audio-to-MIDI conversion, automatic music transcription, and audio enhancement and transformation. Past attempts at separation have been limited to separating two harmonic signals in a contrapuntal duet (Maher, 1990) or several harmonic signals in a single chord (Virtanen and Klapuri, 2001, 2002). Several researchers have attempted polyphonic pitch detection (Klapuri, 2001; Eggink and Brown, 2004a), predominant melody extraction (Goto, 2001; Marolt, 2004; Eggink and Brown, 2004b), and instrument recognition (Eggink and Brown, 2003). Our solution assumes that each instrument is represented as a time-varying harmonic series and that errors can be corrected using prior knowledge of instrument spectra. Fundamental frequencies (F0s) for each time frame are estimated from input spectral data using an Expectation-Maximization (EM) based algorithm with Gaussian distributions used to represent the harmonic series. Collisions (i.e., overlaps) between instrument harmonics, which frequently occur, are predicted from the estimated F0s. The uncollided harmonics are matched to ones contained in a pre-stored spectrum library in order that each F0‘s harmonic series is assigned to the appropriate instrument. Corrupted harmonics are restored using data taken from the library. Finally, each voice is additively resynthesized to a separate track. This algorithm is demonstrated for a monaural signal containing three contrapuntal musical instrument voices with distinct timbres.KeywordsHarmonic SpectrumHarmonic SeriesMusic Information RetrievalPitch DetectionSeparate TrackThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The traditional computer-aided design (CAD) software is often not realistic in rendering scenes, such as light, materials, shadows, and other details, which leads designers to spend a lot of energy on manual adjustment.The rendering algorithm based on deep learning (DL) can automatically learn the complex output, and the effect is usually better than traditional methods.In this article, a convolutional neural network (GA-CNN) based on genetic algorithm (GA) optimization is proposed, and it is combined with CAD data and applied to the rendering of environmental art design scenes.This method uses GA to optimize the parameters of CNN to improve its learning effect and generalization ability.From the overall trend, the response speed of GA-CNN algorithm and support vector machine (SVM).The response speed of the traditional CNN method is relatively slow, and the fluctuation range is large.This is because traditional CNN needs a lot of convolution operations and feature extraction when dealing with artistic scenes in complex environments, which leads to a long calculation time.In practical application, it is necessary to select appropriate algorithms and parameter settings according to specific task requirements to achieve the best results.

We develop a theoretical framework for representation and automated generation of South Indian classical music. The foundational part of the latter is based on symbolic dynamics and is implemented by translating the lexicographic rules for a rāga to constraints on a Markov chain whose state space is a layered graph. We analyze the statistical properties of this Markov chain from the point of view of information theory. We also develop several tools in music signal processing, such as, (a) a procedure for automated generation of gamakas or ornamental notes, unique to South Indian classical music, (b) rhythm synchronization, and (c) an algorithm for perceptual scale shifts. The Online Supplement has computer synthesized music from Śankarābharaṇa rāga, rhythm synchronized music for a Mōhana rāga composition, original and synthesized gamakas over madhyama for Begaḍa and Nīlāmbari rāgas, and perceptually scale shifted rāgas Hindoḷa, Madhyamāvati, Śuddha sāveri, and Udaya ravi chandrikā rāgas from the base Mōhana rāga.

Music technologies will open the future up to new ways of enjoying music both in terms of music appreciation and music creation. In this keynote speech, I introduce the frontiers of music technologies by showing some practical examples to demonstrate how end users can benefit from music signal processing, music understanding technologies, singing synthesis technologies, and music interfaces. For example, a web service for active music listening, "Songle" (http://songle.jp), has analyzed more than 830,000 songs on music- or video-sharing services and facilitates deeper understanding of music and music-synchronized control of robot dancers. A web service for large-scale music browsing, "Songrium" (http://songrium.jp), allows users to explore music while seeing and utilizing various relations among more than 680,000 music video clips on video-sharing services. Singing synthesis technologies open up new possibilities in music creation. I conclude by discussing grand challenges.

Music signal processing is one of the active research area now a days. Identifying the musical instrument form the solo recordings is one of the applications under signal processing. Different types of algorithms have been proposed till now to identify the instrument. For this objective first of all we have to extract the features from sound samples and then we can use them for instrument identification. In this paper we are presenting one of efficient method to extract the features from sound sample that is Mel Frequency Cepsrtal Coefficient (MFCC). These MFCCs are closely related to behavior of human auditory system. While calculating these MFCCs we actually deal with spectral envelop. And hence it gives very distinctive features of the sound samples. So using these features for identification of musical instrument we can increase the accuracy of identification. In our work we have used five instruments and 30 samples of each instrument. In the training database total 90 samples (60% of total samples) and for testing database 60 (40%)samples are used. In the classification phase we have used K-Nearest Neighbor (K-NN) classifier. Proposed system gives the accuracy of 91.66% for Cello, Piano and Trumpet and accuracy of 83.33% for Flute and Violin.