BrainDamage, a neural network synthesizer - feedback welcome

[mystran]

But there is gradient/derivative info from the neural net, so wonder if there's a way to combine the two?

ADAA needs antiderivatives though (derivatives won't help) ... but I was mostly wondering if you could teach the NN itself to learn it.

[QuadrupleA]

ADAA needs antiderivatives though (derivatives won't help)

Oops, of course Hence the name.

but I was mostly wondering if you could teach the NN itself to learn it.

I suspect what would happen is, if you trained these NNs with a 2nd input for blurring/frequency/phase-delta or some smoothing parameter like that - it would learn a decent approximation of it for the normal case. E.g. if you trained it on a sawtooth wave, it would degrade gracefully to a sine-ish wave as the frequency parameter goes up.

But it probably wouldn't generalize to smoothing the weird waveforms that result from tweaking/damaging individual neurons - you'd be directly messing with some of the neurons that are providing your smoothing, so it'll introduce new possibly high-frequency harmonics etc. and you'll get aliasing there.

Curious to experiment a bit with that though, it might not behave like I'm expecting.

[DRMR]

CinningBao asked the same thing so I'll copypasta: just Win-only for the current prototype. It's built to be easy to port though (OpenGL etc.) so if there's enough interest I'll definitely do some cross-platform builds.

Well in my case I'm only interested in Linux, but please let us know when you get around to other OSes

[QuadrupleA]

So thanks to those who gave some feedback - I can see there's some intellectual interest in the concept and how it works but haven't seen much interest in the actual synth, doing sound design with it, etc. Had a similar response from the friends I've showed it to, concept sounds interesting but they don't seem to want to dive in and play with it.

So think I'm going to shelve the project.

I love the actual work of plugin dev and DSP, this is my first non-toy project (I've got about 150 hours into it) - my brain gets obsessively fascinated and I find myself working long hours almost involuntarily. I'm experienced with native code/graphics/programming but still just beginner/intermediate with DSP.

Would love to dive deeper into it, but I'm wary of investing so much time into this kind of thing again. I spent years and thousands of hours on my game Weapon Hacker, which, though I'm very proud of it and it has its fans, didn't get much traction in the market. So, a little scared to repeat that experience again. We just bought a house too so I'm not feeling in a financial position to put thousands of unpaid hours into a fairly saturated market full of smarter people than me who have been doing it for years and already have the marketing machines in place.

So yeah, trying to learn my lessons of not falling for the sunk-cost fallacy, testing ideas with people before investing too much time, etc.!

[Music Engineer]

I can see there's some intellectual interest in the concept and how it works but haven't seen much interest in the actual synth, doing sound design with it, etc.

Yes indeed. I guess, that's kind of the expected response in this development subforum - people here tend to be more interested in what's going on under the hood. At least, that's certainly true for me. Maybe in the "Instruments" subforum, people are more likely to be interested in actually using the synth for sound design. Thanks for your explanations, btw. I think, I have now a reasonable (high level) idea about what it does. Indeed, maybe I should have realized that my first guess couldn't be right due to the fact that your network has only a single input neuron. Interesting idea - you are using the network to estimate the waveform as function of (normalized) time, i.e. a 1D -> 1D mapping f(t). From a usability perspective, I would perhaps order the neurons in each column from top to bottom by their "saliency" - however that saliency is defined. You could perhaps just brute-force evaluate the error-function with and without a given neuron and take the difference as saliency of the neuron. That would actually give the ideal saliency measure that all those complicated formulas and algorithms in the OBD paper want to approximate ("extend backprop to estimate the Hessian blablabla") because they consider such a brute-force evaluation prohibitively costly (if I'm not mistaken) - which may not apply here in this case. After all, your networks are quite small by todays standards and you may not need a saliency measure for each synapse (as in OBD) but only for each neuron.

[nanostream]

Sorry to hear that! I was and still am very much interested in the sound (and the concept of course), but couldn’t try it because I’m on a Mac.

But like ME suggested above, maybe try another subforum before shelving the project for good? I would expect that developers have different things on their minds than normal users.

[mystran]

So yeah, trying to learn my lessons of not falling for the sunk-cost fallacy, testing ideas with people before investing too much time, etc.!

It's absolutely not a sunk-cost even if you choose to shelve a project. Every project is a learning experience and the first time you do something it's always going to take a lot more time than the next time around. Toy projects are all good, but they typically don't teach you anything about all those little things that you don't bother with because it's just a toy project.

Anyway, the hardest part of making creative software (whether it's an audio plugin or a game) is not the technical side of things (even though there might be very difficult technical challenges too). The hardest part is coming up with something that's useful and/or fun. The design of "useful and/or fun" is a creative process and creative processes are unpredictable, because you often need to try first. Some ideas work out great, some ideas turn out to be garbage as soon as you try them and then there's the annoying category of stuff that seems promising if you polish it, but then it turns out that it actually doesn't work out reasonably no matter what. Other times it appears that your original idea doesn't work out, but while working it you get another idea that seems much better. That's just part of the process. On the bright side, over time you tend to get better at judging what might or might not work out in practice.

edit: Today I tried to make some maps for my toy game project. I had several concepts, I spent a couple of hours on each one blocking them out and none of them worked out to be "fun" so I'll have to think of something else. That's just how creative things go.

[QuadrupleA]

Maybe in the "Instruments" subforum, people are more likely to be interested in actually using the synth for sound design.

Good point on dev forum interests - maybe I'll give Instruments a try too, see if a groundswell of presets and enthusiasm ensues . If not it's probably a workable freeware synth as-is for some extra colors & textures.

From a usability perspective, I would perhaps order the neurons in each column from top to bottom by their "saliency"

I like that idea - spotlight the most influential neurons & synapses. One future feature would be picking parameters by direct clicks, so maybe big circles & small dots to represent influence. Could probably gather some stats on that during backprop, neuron bias / weight derivative as a ratio to output error or something, averaged across the different phases in the training set. Although that's kinda what backprop is training/adjusting on so not sure. But a brute-force damage sampling could also work as a final step after training, and save the saliency metrics with the net.

Sorry to hear that! I was and still am very much interested in the sound (and the concept of course), but couldn’t try it because I’m on a Mac.

Thanks, and yeah sorry about that. At the moment I'm coding directly on the low-level VST2 api (aeffextx.h) since it seemed simple enough for prototyping and I like to at least understand what's underneath big frameworks and abstraction layers like JUCE. And I tend to be a little wary of frameworks, especially ones with lingering legal/financial tendrils. But it might be the right way to go for sanity & productivity - haven't wrestled yet with the intricacies of AU, VST3, etc.

We'll see if this plugin hobby of mine gets that far

then there's the annoying category of stuff that seems promising if you polish it, but then it turns out that it actually doesn't work out reasonably no matter what.

That stuff can be maddening! I had similar dilemmas developing the game, especially since I saved final artwork until last (art plays a huge role in perception of quality, ala plugin GUIs). You often can't know ahead of time whether something would work with more polish, or whether you're throwing good money/time after bad. And in a competitive marketplace there's usually a high quality bar to even get considered so the gamble becomes bigger.

[RunBeerRun]

I'm very interested in it, but I can't get it to show up with Reaper 64.

Some dependency must be missing.

[QuadrupleA]

I'm very interested in it, but I can't get it to show up with Reaper 64.

Some dependency must be missing.

Thanks, strange - seems to load OK here:



Only dependencies are Windows builtins: ADVAPI32.dll, GDI32.dll, KERNEL32.dll, OPENGL32.dll, SHELL32.dII, USER32.dll, WINMM.dll. Also BrainDamageData needs to be alongside the VST dll to load the net models.

[RunBeerRun]

Maybe I don't have Open GL. This is just some old laptop with win10 on it.

[QuadrupleA]

Maybe I don't have Open GL. This is just some old laptop with win10 on it.

Yeah that could be it - might try updating the video driver if you feel like messing with it further.

[gnu23]

Very interesting idea. Downloaded..waiting for Ableton to scan it in.

[kippertoffee]

I think this is a really interesting idea and can produce some really nice timbres. For me, after a quick play, what becomes apparent is that sound design is totally trial and error. Seems like there's very little chance of actually learning the thing and being able to produce a sound I imagine. Not that this is necessarily a bad thing, but it might limit it's appeal.

One thought I had would be to "curate" the nodes that can be modified to a select few that produce the most dramatic and interesting timbres for each source, and just giving a control for them rather than letting the user select.

[bitwise]

... - it lets you train simple neural networks to mimic single-cycle waveforms, ...

Is it an alias-free oscillator ?