DDSP - Differentiable Digital Signal Processing - Google AI

DSP, Plug-in and Host development discussion.
Winstontaneous
KVRAF
1738 posts since 15 Feb, 2006 from Berkeley, CA

Post Sun Jan 19, 2020 3:06 pm

I came across this on Hacker News a few days ago and checked if anyone here posted it yet. I'm at best an enthusiastic novice experimenting with DSP (Max4Live, PureData, Reaktor), and am starting to explore deep/machine learning techniques for my data-centric dayjob.

Thoughts? Seems more focused on the control of DSP processes than novel approaches to raw DSP. Google-funded hype, an interesting avenue for exploration (or both)?

https://magenta.tensorflow.org/ddsp
https://github.com/magenta/ddsp
Today, we’re pleased to introduce the Differentiable Digital Signal Processing (DDSP) library. DDSP lets you combine the interpretable structure of classical DSP elements (such as filters, oscillators, reverberation, etc.) with the expressivity of deep learning.

Neural networks (such as WaveNet or GANSynth) are often black boxes. They can adapt to different datasets but often overfit details of the dataset and are difficult to interpret. Interpretable models (such as musical grammars) use known structure, so they are easier to understand, but have trouble adapting to diverse datasets.

DSP (Digital Signal Processing, without the extra “differentiable” D) is one of the backbones of modern society, integral to telecommunications, transportation, audio, and many medical technologies. You could fill many books with DSP knowledge, but here are some fun introductions to audio signals, oscillators, and filters, if this is new to you.

The key idea is to use simple interpretable DSP elements to create complex realistic signals by precisely controlling their many parameters. For example, a collection of linear filters and sinusoidal oscillators (DSP elements) can create the sound of a realistic violin if the frequencies and responses are tuned in just the right way. However, it is difficult to dynamically control all of these parameters by hand, which is why synthesizers with simple controls often sound unnatural and “synthetic”.

With DDSP, we use a neural network to convert a user’s input into complex DSP controls that can produce more realistic signals. This input could be any form of control signal, including features extracted from audio itself. Since the DDSP units are differentiable (thus the extra D), we can then train the neural network to adapt to a dataset through standard backpropagation.
OLOKUN - Survival Kit - now on iTunes, CDBaby, and Spotify

Videos: The Beat | Crossing to Safety

puffin
KVRer
15 posts since 17 Jan, 2020

Re: DDSP - Differentiable Digital Signal Processing - Google AI

Post Sun Jan 19, 2020 3:43 pm

So the usability factor is just that any new modelling synthesizer has no more than a certain number of controls? Hopefully they don't copyright what their neural network discovers, that has been a recent topic, too. The idea itself sounds absolutely doable to me.

cron
KVRAF
3287 posts since 27 Dec, 2002 from North East England

Re: DDSP - Differentiable Digital Signal Processing - Google AI

Post Sun Jan 19, 2020 4:20 pm

:tu: This is very, very interesting. Operating directly on audio is IIRC very difficult due to the number of samples (as in ‘sample rate’) involved, so doing it via parametric resynthesis is a great idea.

The problem with additive synthesis, capable of (re)producing any sound imaginable in theory, has always been managing the vast numbers of parameters involved. Advances in high level macro control that go way beyond, say, Harmor’s ‘fake filters’ are very welcome here. Sticking with the Harmor fake filter example, we can view that as a crude means of parameterising brightness, while this thing could presumably see, ‘understand’ and parameterise brightness in a much more complex way than simple high frequency rolloff while still being a one-knob control.

I don’t think the distinction between controlling simple DSP processes and ‘raw’ DSP is particularly relevant from a musical standpoint. When I timestretch a piece of audio for instance, I don’t really care about the ‘procedural purity’ of how it’s achieved assuming I’m not making a ‘concept’ track. The oscillator bank resynthesis approach (which this thing appears to be using an augmented version of) often sounds better than ye olde ‘cut the audio into tiny pieces and overlap’ method, even though there’s arguably much less of the ‘spirit’ of the original audio in the result.

The sound examples are enormously impressive.

jackmazzotti
KVRist
329 posts since 18 Dec, 2006

Re: DDSP - Differentiable Digital Signal Processing - Google AI

Post Sun Jan 19, 2020 4:36 pm

The next levels of synthesis and deep fakes are going to completely change media.
Image

synthpark
KVRer
27 posts since 7 Sep, 2018

Re: DDSP - Differentiable Digital Signal Processing - Google AI

Post Mon Jan 20, 2020 3:35 am

There was a presentation at Native Instruments, I even joined the audience. Yes, quite impressive to sing into the box and get an authentic violin sound out.

Winstontaneous
KVRAF
1738 posts since 15 Feb, 2006 from Berkeley, CA

Re: DDSP - Differentiable Digital Signal Processing - Google AI

Post Mon Jan 20, 2020 2:16 pm

jackmazzotti wrote:
Sun Jan 19, 2020 4:36 pm
The next levels of synthesis and deep fakes are going to completely change media.
Interestingly, as I get into the source texts of modern cybernetics/information theory (Wiener, Shannon) - some of the fundamental texts (von Neumann's in particular) assume that beyond system noise, there is an opponent continually working to distort/jam the intended message. Also intriguing - in no uncertain terms, Wiener sees no theoretical/conceptual hindrance to the teleportation of matter. :party:
OLOKUN - Survival Kit - now on iTunes, CDBaby, and Spotify

Videos: The Beat | Crossing to Safety

Return to “DSP and Plug-in Development”