Download FoleySet: A Multi-Level Human-Annotated Foley Sound Dataset
We introduce FoleySet, a human-annotated Foley sound dataset designed to support research on sound-event understanding and Foley sound generation. The dataset provides annotations at multiple levels of granularity, capturing both broad event categories and more detailed semantic or production-related attributes. This multi-level structure supports tasks such as classification, retrieval, captioning, and controllable generation. FoleySet is intended to address the limited availability of systematically annotated Foley material and to provide a common resource for evaluating models across different levels of semantic detail.
Download PAEDB: A Synthetic Primary-Ambient Dataset Generation Pipeline for Automatic Upmixing Using Deep Neural Networks
Automatic blind upmixing aims to convert audio from a smaller channel format (e.g. mono or stereo) into a multichannel format using estimates of direct and diffuse spatial statistics within the signal. Current approaches rely on primary-ambient extraction (PAE) algorithms, which lack real-world context through limited processing windows. Deep learning music source separation (MSS) models have been applied in voice-primary-ambient extraction (VPA) upmixing systems for handling direct components, but still rely on DSP methods of surround channel generation. This work further investigates utilizing source separation within VPA upmixing, focusing specifically on the task of stereo decorrelation and ambience extraction for 5.1 surround. We also release PAEDB (Primary–Ambient Extraction Dataset), a high-quality music dataset derived from MUSDB18-HQ and MoisesDB, comprising 1,809 primary–ambient stem pairs totaling over 550 hours of audio. The performance of selected DNNs trained on PAEDB is then evaluated using signal metrics and a listening study. Our findings indicate that DNNs can effectively model the behavior of PAE algorithms, establishing PAEDB as a strong foundation for ML upmixing systems and underscoring the need for higher-quality multichannel data to advance beyond conventional methods.
Download Modal Structure of Plate Boundaries and Klein Bottle Reverberation
Physical modeling sound synthesis has achieved remarkable success in terms of its fidelity to reality. In many cases, since modeling of the physical system is performed on the sounding objects that already exist in the real world, observation precedes the model itself. Departing from this convention, this paper aims to physically model the acoustic characteristics of objects that do not necessarily exist in reality. Specifically, we study wave propagation on compact two-dimensional (2D) manifolds that are non-orientable surfaces, such as the Klein bottle that cannot be embedded in three-dimensional Euclidean space without self-intersection. We derive closed-form expressions for the eigenfrequencies and mode shapes of non-orientable 2D topologies and study their acoustic characteristics. The modal structures are verified through comparison with finite-difference time-domain simulations. The results demonstrate how the topological character formed by the boundaries influences the acoustic resonances, and how the quotient-space framework provides a practical route to reverb synthesis on geometries with no physical counterpart.
Download Beat-Marker Location using a Probabilistic Framework and Linear Discriminant Analysis
This paper deals with the problem of beat-tracking in an audiofile. Considering time-variable tempo and meter estimation as input, we study two beat-tracking approaches. The first one is based on an adaptation of a method used in speech processing for locating the Glottal Closure Instants. The results obtained with this first approach allow us to derive a set of requirements for a robust approach. This second approach is based on a probabilistic framework. In this approach the beat-tracking problem is formulated as an “inverse” Viterbi decoding problem in which we decode times over beat-numbers according to observation and transition probabilities. A beat-template is used to derive the observation probabilities from the signal. For this task, we propose the use of a machine-learning method, the Linear Discriminant Analysis, to estimate the most discriminative beat-template. We finally propose a set of measures to evaluate the performances of a beattracking algorithm and perform a large-scale evaluation of the two approaches on four different test-sets.
Download Searching for Music Mixing Graphs: A Pruning Approach
Music mixing is compositional — experts combine multiple audio processors to achieve a cohesive mix from dry source tracks. We propose a method to reverse engineer this process from the input and output audio. First, we create a mixing console that applies all available processors to every chain. Then, after the initial console parameter optimization, we alternate between removing redundant processors and fine-tuning. We achieve this through differentiable implementation of both processors and pruning. Consequently, we find a sparse mixing graph that achieves nearly identical matching quality of the full mixing console. We apply this procedure to drymix pairs from various datasets and collect graphs that also can be used to train neural networks for music mixing applications.
Download Quadratic Spline Approximation of the Contact Potential for Real-Time Simulation of Lumped Collisions in Musical Instruments
Collisions are an integral part of the sound production mechanism in a wide variety of musical instruments. In physics-based realtime simulation of such nonlinear phenomena, challenges centred around efficient and accurate root-finding arise. Nonlinearly implicit schemes are normally ill-suited for real-time simulation as they rely on iterative solvers for root-solving. Explicit schemes overcome this issue at the cost of a slightly larger error for a given sample rate. In this paper, for the case of lumped collisions, an alternate approach is proposed by approximating the contact potential curve. The approximation is described, and is shown to lead to a non-iterative update for an energy-stable nonlinearly implicit scheme. The method is first tested on single mass-barrier collision simulations, and then employed in conjunction with a modal string model to simulate hammer-string and slide-string interaction. Results are discussed in comparison with existing approaches, and real-time feasibility is demonstrated.
Download Differentiable All-Pole Filters for Time-Varying Audio Systems
Infinite impulse response filters are an essential building block of many time-varying audio systems, such as audio effects and synthesisers. However, their recursive structure impedes end-toend training of these systems using automatic differentiation. Although non-recursive filter approximations like frequency sampling and frame-based processing have been proposed and widely used in previous works, they cannot accurately reflect the gradient of the original system. We alleviate this difficulty by reexpressing a time-varying all-pole filter to backpropagate the gradients through itself, so the filter implementation is not bound to the technical limitations of automatic differentiation frameworks. This implementation can be employed within audio systems containing filters with poles for efficient gradient evaluation. We demonstrate its training efficiency and expressive capabilities for modelling real-world dynamic audio systems on a phaser, time-varying subtractive synthesiser, and feed-forward compressor. We make our code and audio samples available and provide the trained audio effect and synth models in a VST plugin1 .
Download Towards Neural Emulation of Voltage-Controlled Oscillators
Machine learning models have become ubiquitous in modeling analog audio devices. Expanding on this line of research, our study focuses on Voltage-Controlled Oscillators of analog synthesizers. We employ black box autoregressive artificial neural networks to model the typical analog waveshapes, including triangle, square, and sawtooth. The models can be conditioned on wave frequency and type, enabling the generation of pitch envelopes and morphing across waveshapes. We conduct evaluations on both synthetic and analog datasets to assess the accuracy of various architectural variants. The LSTM variant performed better, although lower frequency ranges present particular challenges.
Download Using the Distribution Derivative Method to Model Acoustic Musical Instrument Sounds with Polynomial AM-FM Sinusoids
The oscillatory modes of musical instrument sounds are commonly modeled with time-varying sinusoids. Several estimation methods model quasi-stationary oscillations accurately, yielding a high-quality representation. However, nonstationary oscillations such as attack transients are still very challenging to model accurately. In this work, we propose to model musical instrument sounds with polynomial modulation sinusoids (PMS) estimated with the distribution derivative method (DDM). DDM gives accurate parameter estimations for PMS with arbitrary order, allowing great flexibility in modeling temporal changes inside analysis frames as amplitude and frequency modulations. We used 39 musical instrument sounds to compare DDM objectively against the standard (SM+) and an adaptive sinusoidal model (eaQHM) using time and frequency error measures. We showed that DDM captures more oscillatory energy than SM+ or eaQHM by modeling PMS more accurately. A MUSHRA listening test with 18 selected sounds confirmed that DDM has higher perceptual quality than both SM+ and eaQHM and that DDM is almost perceptually indistinguishable from the original sounds.
Download Sparse Atomic Modeling of Audio: a Review
Research into sparse atomic models has recently intensified in the image and audio processing communities. While other reviews exist, we believe this paper provides a good starting point for the uninitiated reader as it concisely summarizes the state-of-the-art, and presents most of the major topics in an accessible manner. We discuss several approaches to the sparse approximation problem including various greedy algorithms, iteratively re-weighted least squares, iterative shrinkage, and Bayesian methods. We provide pseudo-code for several of the algorithms, and have released software which includes fast dictionaries and reference implementations for many of the algorithms. We discuss the relevance of the different approaches for audio applications, and include numerical comparisons. We also illustrate several audio applications of sparse atomic modeling.