Paper Archive

This paper investigates the synthesis of audio textures and the structure of generative latent spaces using Variational Autoencoders (VAEs) within two paradigms of neural audio synthesis: DSP-inspired and data-driven approaches. For each paradigm, we propose VAE-based frameworks that allow fine-grained temporal control. We introduce datasets across three categories of environmental sounds to support our investigations. We evaluate and compare the models’ reconstruction performance using objective metrics, and investigate their generative capabilities and latent space structure through latent space interpolations.

Download

In this work, we introduce TexStat, a novel loss function specifically designed for the analysis and synthesis of texture sounds characterized by stochastic structure and perceptual stationarity. Drawing inspiration from the statistical and perceptual framework of McDermott and Simoncelli, TexStat identifies similarities between signals belonging to the same texture category without relying on temporal structure. We also propose using TexStat as a validation metric alongside Frechet Audio Distances (FAD) to evaluate texture sound synthesis models. In addition to TexStat, we present TexEnv, an efficient, lightweight and differentiable texture sound synthesizer that generates audio by imposing amplitude envelopes on filtered noise. We further integrate these components into TexDSP, a DDSP-inspired generative model tailored for texture sounds. Through extensive experiments across various texture sound types, we demonstrate that TexStat is perceptually meaningful, time-invariant, and robust to noise, features that make it effective both as a loss function for generative tasks and as a validation metric. All tools and code are provided as open-source contributions and our PyTorch implementations are efficient, differentiable, and highly configurable, enabling its use in both generative tasks and as a perceptually grounded evaluation metric.

Download

Wave Digital Filters were developed to discretize linear time invariant lumped systems, particularly electronic circuits. The timeinvariant assumption is baked into the underlying theory and becomes problematic when simulating audio circuits that are by nature time-varying. We present extensions to WDF theory that incorporate proper numerical schemes, allowing for the accurate simulation of time-varying systems. We present generalized continuous-time models of reactive components that encapsulate the time-varying lossless models presented by Fettweis, the circuit-theoretic time-varying models, as well as traditional LTI models as special cases. Models of timevarying reactive components are valuable tools to have when modeling circuits containing variable capacitors or inductors or electrical devices such as condenser microphones. A power metric is derived and the model is discretized using the alpha-transform numerical scheme and parametric wave definition. Case studies of circuits containing time-varying resistance and capacitance are presented and help to validate the proposed generalized continuous-time model and discretization.

Download

Modern graphics processing units (GPUs) are massively parallel computing environments. They make it possible to run certain tasks orders of magnitude faster than what is possible with a central processing unit (CPU). One such case is simulation of room acoustics with wave-based modeling techniques. In this paper we show that it is possible to run room acoustic simulations with a finite-difference time-domain model in real-time for a modest-size geometry up to 7kHz sampling rate. For a 10% maximum dispersion error limit this means that our system can be used for realtime auralization up to 1.5kHz. In addition, the system is able to handle several simultaneous sound sources and a moving listener with no additional cost. The results of this study include performance comparison of different schemes showing that the interpolated wideband scheme is able to handle in real-time 1.4 times the bandwidth of the standard rectilinear scheme with the same maximum dispersion error.

Download

This article discusses harmonic sinusoid modeling. Unlike standard sinusoid analyzers, the harmonic sinusoid analyzer keeps close watch on partial harmony from an early stage of modeling, therefore guarantees the harmonic relationship among the sinusoids. The key element in harmonic sinusoid modeling is the harmonic sinusoid particle, which can be found by grouping short-time sinusoids. Instead of tracking short-time sinusoids, the harmonic tracker operates on harmonic particles directly. To express harmonic partial frequencies in a compact and robust form, we have developed an inequality-based representation with adjustable tolerance on frequency errors and inharmonicity, which is used in both the grouping and tracking stages. Frequency and amplitude continuity criteria are considered for tracking purpose. Numerical simulations are performed on simple synthesized signals.

Download

This paper proposes reservoir computing as a general framework for nonlinear audio processing. Reservoir computing is a novel approach to recurrent neural network training with the advantage of a very simple and linear learning algorithm. It can in theory approximate arbitrary nonlinear dynamical systems with arbitrary precision, has an inherent temporal processing capability and is therefore well suited for many nonlinear audio processing problems. Always when nonlinear relationships are present in the data and time information is crucial, reservoir computing can be applied. Examples from three application areas are presented: nonlinear system identification of a tube amplifier emulator algorithm, nonlinear audio prediction, as necessary in a wireless transmission of audio where dropouts may occur, and automatic melody transcription out of a polyphonic audio stream, as one example from the big field of music information retrieval. Reservoir computing was able to outperform state-of-the-art alternative models in all studied tasks.

Download

Automatic blind upmixing aims to convert audio from a smaller channel format (e.g. mono or stereo) into a multichannel format using estimates of direct and diffuse spatial statistics within the signal. Current approaches rely on primary-ambient extraction (PAE) algorithms, which lack real-world context through limited processing windows. Deep learning music source separation (MSS) models have been applied in voice-primary-ambient extraction (VPA) upmixing systems for handling direct components, but still rely on DSP methods of surround channel generation. This work further investigates utilizing source separation within VPA upmixing, focusing specifically on the task of stereo decorrelation and ambience extraction for 5.1 surround. We also release PAEDB (Primary–Ambient Extraction Dataset), a high-quality music dataset derived from MUSDB18-HQ and MoisesDB, comprising 1,809 primary–ambient stem pairs totaling over 550 hours of audio. The performance of selected DNNs trained on PAEDB is then evaluated using signal metrics and a listening study. Our findings indicate that DNNs can effectively model the behavior of PAE algorithms, establishing PAEDB as a strong foundation for ML upmixing systems and underscoring the need for higher-quality multichannel data to advance beyond conventional methods.

Download

We propose a new analog filter discretization method that is useful for discretizing systems with features near or above the Nyquist limit. A conformal mapping approach is taken, and we introduce the peaking conformal map and shelving conformal map. The proposed method provides a close match to the original analog frequency response below half the sampling rate and is parameterizable, order preserving, and agnostic to the original filter’s order or type. The proposed method should have applications to discretizing filters that have time-varying parameters or need to be implemented across many different sampling rates.

Download

The article performs a generic comparison of integrator- and differentiator based continuous-time systems as well as their discretetime models, aiming to answer the reoccurring question in the music DSP community of whether there are any benefits in using differentiators instead of conventionally employed integrators. It is found that both kinds of models are practically equivalent, but there are certain reservations about differentiator based models.

Download

Filter banks, both uniform and non-uniform, are widely used for signal analysis and processing. However, the application of a timefrequency localized filter inevitably causes some amount of spectral and temporal leakage that, simultaneously, cannot be arbitrarily reduced. Reassignment is a classical procedure to eliminate this leakage in short-time Fourier spectrograms, thereby providing a sharper, more exact time-frequency domain signal representation. The reassignment technique was recently generalized to general filter banks, opening new possibilities for its application in signal analysis and processing. We present here the very first implementation of filter bank reassignment in a real-time analysis setting, more specifically as visualization in a basic audio player application. The visualization provides a low delay moving spectrogram with respect to virtually any time-frequency filter bank by interfacing the C backend of the LTFAT open-source toolbox for time-frequency processing. Low delay is achieved by blockwise processing, implemented with the JUCE C++ Library.

Download