BrainDamage, a neural network synthesizer - feedback welcome

[QuadrupleA]

Wanted to share a little synth plugin I've been working on, BrainDamage - it lets you train simple neural networks to mimic single-cycle waveforms, then tweak individual neurons and synapses (apply brain damage) to distort the waveform and timbre in interesting and unpredictable ways:



Kind of a weird idea, but it's been surprisingly fruitful and interesting so far - a bit like a wavetable but with more unique and unpredictable transitions. I've had a lot of fun developing and doing sound design for it (there's about 25 presets bundled in). Each neuron/synapse is like a unique knob that distorts the sound in its own way, so it's fun to explore and find happy accidents.

Website with a more detailed description and the download link:

Info: https://quadrupleasound.com/braindamage
VST2 Download (1.21MB): https://quadrupleasound.com/static/Brai ... 1644344314

Tech-wise it's written from scratch in C++ and OpenGL - I reused a lot of code from the Weapon Hacker (https://weaponhacker.com/) game engine, to allow nice high-framerate low-latency UI interactions - wanted to give instant feedback on the waveform display as you tweak neurons and settings.

I know there are a lot of smart and experienced devs on here, so interested to hear any honest feedback and constructive criticism about the idea, the execution, any compatibility issues, etc.

If the prototype here has potential I'd love to develop it into a commercial quality synth (mod matrix / more modulation options, multiple waveform oscillators, built-in effects, integrate the training into the interface, etc.).

Curious what people think!

-Luke

[ghettosynth]

Cool, that's what I think anyway. I'll try it out.

[CinningBao]

Ah, Win only? Would love a mac version. Looks like a wicked load of fun!

[imrae]

Interesting idea! I take it the input to the neural net is a pitch-controlled phasor ramp?

I wonder if it would be possible to use a structure with an additional pitch input to learn the anti-aliasing as well. (i.e. for a given target waveform, generate a set of filtered waveforms for each pitch and then train on pitch/phase combinations.)

Of course, the resulting AA won't be perfect but maybe that's more fun...

I bet some cool stuff also happens when morphing between two sets of trained coefficients.

[QuadrupleA]

@ghettosynth: Thanks!

@CinningBao: Yeah, Win-only for the current prototype. It's built to be easy to port though (OpenGL etc.) so if there's enough interest I'll definitely do some cross-platform builds.

@imrae: Yup, exactly - net input is phase (0..1) and output is the volume level at that part of the waveform. Pitch input would be interesting, where the net could maybe learn to blur / oversample itself and reduce artifacts. Aliasing isn't too bad on high notes, judging by ear - I did add a manual 1x-8x oversampling knob that uses proportionally more CPU, but definitley wondering what other clever optimizations are possible. Could cache the net output over time and generate "mipmaps" on the fly maybe - would save a lot of CPU for some presets with fairly static neuron settings, but might not help much if the neuron tweaks are very dynamic (which is where this method shines I think).

[Architeuthis]

OK DOWNLOADING BRAINDAMAGE... ooh I like the preset called... well I can't see the name as it's been replaced by "(last edit)".

FR: monophony. Monophony is super important. Throw in portamento as well.

FR: stereo. Find a way to make it stereoized, maybe simple multiple unison voices or two voices with some different settings.

FR: Envelope for filter cutoff

Edit: FR: Prettier oscilloscope (use PrettyScope )

Here's a quick audio demo. It was difficult because it's a new synthesizer and I have to get more experience with its sound character and control scheme.

- - - - - - - -
BrainDamage demo by Argitoth - Can't Think [mp3]
- - - - - - - -

hey QuadrupleA, you know OpenGL?

I've been trying to find a programmer to help with PrettyScope and I noticed you have an oscilloscope in your plugin. I wouldn't mind if every audio plugin used this scope if it meant PrettyScope could get a little elbow grease to fix some of the issues, maybe I'd open source it. If you're interested in PrettyScope, contact me (see product page for contact info). If the first issue was to be fixed, I would probably shed a tear at some point.

PrettyScope Employment Page
Prettyscope Issues in detail

Edit: I just randomly made this with PrettyScope, just to give an idea of how pretty it can look. This isn't realtime though, it's more like I'm painting an image slowly.

(click to enlarge)

[DRMR]

Looks neat, but will you make this cross-platform at all?

[mystran]

On the topic of AA: we know that there are methods such as ADAA that can reduce aliasing without concrete oversampling by evaluating integrals over approximate continuous-time reconstruction from a couple of surrounding samples and we know that neural networks (of sufficient size) are universal function approximators.

So provided that an NN has access to multiple samples over time (in one form or another; no idea what the ideal approach would be), I don't see an obvious reason why it couldn't (at least in principle) learn an ADAA-like scheme for anti-aliasing. Whether this would actually work when "damage" is applied to the network is another question though.

[Unaspected]

Love the concept!

Will have to check out at a later date.

[Music Engineer]

This is very interesting! I've pondered for a long time how to use neural networks for audio synthesis but never implemented anything so far. I'm familiar with the (older) work of Axel Röbel where he used neural networks as a sort of nonlinear time series predictor to model the dynamics of musical instruments (after modeling Lorenz and Rössler attractors as proof of concept) - with quite good results. Here is an overview:

https://hal.archives-ouvertes.fr/hal-02911718/document

Are you doing something similar? Would you mind to explain in a bit more detail what your network is doing? Does it also try to predict the sample value at a particular sample instant as output of the network when a bunch of delayed samples is used as input vector? ...but maybe with some special provisions for "wrap-around" behavior to get the single-cycles? And the "damage" parameter just fades out certain synaptic weights (going into and/or out of a specific neuron)?

During my computer science classes, I learned about the "Optimal Brain Surgeon" algorithm which, as I found out, is a sort of refined version of the "Optimal Brain Damage" algorithm - which is hands down my favorite name for any algorithm ever. The idea is to identify those synaptic weights, whose deletion would have the least effect on the loss function - and then delete them (i.e. set them to zero) ...and retrain. The goal is to prune/simplify the network in a meaningful way to achieve potentially better generalization and save computations. Edit: here it is:

https://proceedings.neurips.cc/paper/19 ... -Paper.pdf

The "Surgeon" (as opposed to the "Damage") variant would then also rejiggle the remaining weights to further reduce the impact of the deletion on the error (which, I guess, is taken care of in the retraining step in the "Damage" variant anyway...and you'd probably also retrain in OBS as well...so maybe the additional complexity of the rejiggling in OBS isn't justified?...dunno...).

https://proceedings.neurips.cc/paper/19 ... -Paper.pdf

If I'm not mistaken, there are also variants of both algorithms, which apply the same idea to whole neurons as opposed to individual synapses - which makes actually a lot more sense computationally, because individual zeros in dense matrices do not really give you a computational advantage anyway. But when you can delete a whole row or column (corresponding to deletions of whole neurons), you really do save computations. I guess, you are doing something similar except letting the user manually pick the neurons to be "damaged" and allow for a more gradual "damage" by not setting them brutally to zero but scaling their synaptic weights by a factor <= 1?

[QuadrupleA]

Here's a quick audio demo.

Awesome - you're one the first besides me to make music with it.

FR: monophony. Monophony is super important. Throw in portamento as well.

FR: stereo. Find a way to make it stereoized, maybe simple multiple unison voices or two voices with some different settings.

FR: Envelope for filter cutoff

Thanks - good suggestions, agreed on all. Right now I'm sorta prototyping the idea / synthesis method - if it seems interesting enough to people I hope to do a commercial quality version with fully flexible modulation (modulate any knob with any combination of MIDI inputs, envelope generators, LFOs, etc.) and all the basic synth staples like unison, onboard FX, more filters, FM / RM options using the neural waveforms, etc.

hey QuadrupleA, you know OpenGL?

Looks like you've got some very cool visualizations on the website there - I would try reaching out to some folks on shadertoy.com if you haven't already, there are some total OpenGL geniuses on there (though probably gainfully employed ). I'm LukeRissacher on there (shameless plug). Feel free to steal my code.

[QuadrupleA]

Looks neat, but will you make this cross-platform at all?

CinningBao asked the same thing so I'll copypasta: just Win-only for the current prototype. It's built to be easy to port though (OpenGL etc.) so if there's enough interest I'll definitely do some cross-platform builds.

...we know that there are methods such as ADAA that can reduce aliasing without concrete oversampling by evaluating integrals over approximate continuous-time reconstruction from a couple of surrounding samples...

Thanks mystran - hadn't heard of ADAA. Found a post/paper on it but looks like it would take me an evening or two to grok it, my calculus is pretty rusty

Whether this would actually work when "damage" is applied to the network is another question though.

Yeah it's a good question - the waveform changes while damaging can be pretty extreme:



But there is gradient/derivative info from the neural net, so wonder if there's a way to combine the two?

[Architeuthis]

I would try reaching out to some folks on shadertoy.com if you haven't already.

I see a facebook page. Is that what you'd recommend for contacting them?

[QuadrupleA]

I'm familiar with the (older) work of Axel Röbel where he used neural networks as a sort of nonlinear time series predictor to model the dynamics of musical instruments (after modeling Lorenz and Rössler attractors as proof of concept) - with quite good results. Here is an overview:

Hey Robin - very cool - sounds like you probably know more about this stuff than me. If I'm understanding the Röbel paper right, it's a recurrent neural network model where the inputs are the samples so far and the outputs are the predictions of the next ones - was thinking of exploring something like that actually, would be a good topology for non-harmonic sounds like noise, metallic timbres etc.

BrainDamage's NNs are a bit simpler - the network diagrams in UI are the actual neural net that's running (green = positive bias, blue = negative). Sorry if the animations are distracting, browsers can't pause GIFs for some reason



The single input is phase (0..1) and the single output is amplitude at that part of the waveform (0..1 mapped to -1..1). There's a little bundled utility actually in the data folder to train your own nets. So basically the whole network is just a scalar function that samples the waveform at a given phase and returns amplitude - but can lead to pretty interesting distortions when neurons are damaged (i.e. a weights or biases are added or subtracted to while playing). This is damage from just one neuron:

I learned about the "Optimal Brain Surgeon" algorithm which, as I found out, is a sort of refined version of the "Optimal Brain Damage" algorithm - which is hands down my favorite name for any algorithm ever.

Ha Sounds like I inadvertently stole their name. Yeah that pruning idea during training is really interesting. You can see it playing with the small nets in the synth, that some neurons are pulling a lot of weight and some are not doing much (damaging them has little effect). Might be partly my training regimen isn't as efficient as it could be - I think it's fairly common practice to re-randomize / re-initialize dead neurons/weights etc. so they end up contributing more. But yeah, like you say, removing them in random places doesn't necessarily mean an optimization benefit because multiplying sparse matrices isn't always faster given how vector computation / branch prediction etc. works.

[QuadrupleA]

I see a facebook page. Is that what you'd recommend for contacting them?

Not sure - some of the artists have websites / profiles in their bio.