Continuous Descriptor-Based Control for Deep Audio Synthesis

Despite significant advances in deep models for music generation, the use of these techniques remains restricted to expert users. Before being democratized among musicians, generative models must first provide expressive control over the generation, as this conditions the integration of deep generat...

Full description

Saved in:
Bibliographic Details
Main Authors: Devis, Ninon, Demerle, Nils, Nabi, Sarah, Genova, David, Esling, Philippe
Format: Conference Proceeding
Language:English
Subjects:
adversarial training
continuous control
Controllability
Deep audio synthesis
Hardware
Instruments
Real-time systems
Recording
representation learning
Synthesizers
timbre transfer
Training
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Despite significant advances in deep models for music generation, the use of these techniques remains restricted to expert users. Before being democratized among musicians, generative models must first provide expressive control over the generation, as this conditions the integration of deep generative models in creative workflows. In this paper, we tackle this issue by introducing a deep generative audio model providing expressive and continuous descriptor-based control, while remaining lightweight enough to be embedded in a hardware synthesizer. We enforce the controllability of real-time generation by explicitly removing salient musical features in the latent space using an adversarial confusion criterion. User-specified features are then reintroduced as additional conditioning information, allowing for continuous control of the generation, akin to a synthesizer knob. We assess the performance of our method on a wide variety of sounds including instrumental, percussive and speech recordings while providing both timbre and attributes transfer, allowing new ways of generating sounds.
ISSN:2379-190X
DOI:10.1109/ICASSP49357.2023.10096670