Well this is a kick in the nuts: VST2 plug-ins

[S0lo]

Would these problems still occur, had the plugins followed a correct and well defined protocol/standard of how to access DSP and GUI through the none separated single model?

What we've noticed over the years of developing Ardour & Mixbus is that plugins that come out of companies/organizations/processes/developers who have really "systematized" their development all tend to Just Work (TM). Once you can load one Waves plugin, you can load and run them all (for example). U-he's fabulous instruments ... they just work, all the time. Etc. etc.

The problems tend to show up in two places:

First, relatively new developers or developers that only ever create one or three plugins. This has become less of an issue as more and more people use JUCE, iPlug, DPF etc. to take care of the boilerplate. But people who insist on hand-rolling everything tend to take a while to get things right in all the details.

It is important to note that this is also caused by the lack of clear instructions/SDK's/APIs for host implementation. How many times will we read "well, the plugin works in host X" when dealing with a bug report from users? Logic has taken some fairly loose interpretations of the AU "spec" that are certainly within the range of the reasonable, but you won't find them written down anywhere. I don't blame plugin developers creating an AU from saying "it works in Logic" but I really think we should able to "do better" than that.

Second, sometimes there can be fundamental technology that is solid that gets combined with something new. I won't mention any names, but for example when one excellent plugin development company allowed an effort to port its plugins to Linux, the GUI implementation was (and to some extent still is) pretty wacky. The fundamental DSP side is totally solid, and the GUI itself is similar, but the graft of some new technology took some time to get worked out (some might argue it still needs work, but that's not relevant.

In summary, I would say that in general, you're probably right. But we frequently have read things even in this thread about how wonderful it is that "anyone can write a plugin", and it is often (though not exclusively) this sort of development pathway that ends up giving is headaches inside the host.

I see, What I get from this is that one of the main reasons to go for DSP/UI separation is to give more control for the host over "offending plugins". Due to novice dev, lack of documentation, other technicalities, etc. But basically you'd like to lessen the chance of plugins crashing. If thats it, I understand.

But here is the thing. The host is practically helpless. really no matter what you do. The plugin lives inside the address space of the host (unless your sand boxing). It could at any moment touch any part of memory, it could wreak havoc if it wants to. It can do an endless loop when it gets the GUI thread and hang the display, etc.. The host is really at the mercy of the running plugins.

So I think it shouldn't try or even be expected to try to control the plugins, because it simply can't. Now don't get me wrong here. Sure there are some sanity things to do, like if a plugin requests a million parameters, or a thousand outputs, etc. Sure the host should deny that. But really beyond those obvious things I don't think the host can do too much. In otherwords, the host is not an OS, so it shouldn't try to be one. Because if it did, it will fail big time.

The relation between the host and the plugins is collaborative, not governmental. Sandboxing is the only real power the host has.

I think the priority in a standard should go more towards performance. lessening latency, CPU, memory usage etc. Control and host protection should be a lesser priority. Otherwise, the standard may shoot itself in the foot by creating more problems due to complexity, just as Urs described above.

Let the offending plugins crash, and some how find a way to report them to the user in the next session. Eventually the user will learn that the problem is in the plugin. Now if it runs on another host because of some hidden details of a standard that are not known to you, then thats a problem with the standard documentation. Not with your host. (Hence we need a well documented open standard )

Still, I've never coded a host so what do I know

[camsr]

I do not think spreading components over devices is an overly optimistic goal. A client/server type model should be able to handle it. A "server" here is just meaning a central point of reference for any data that clients need to do stuff. The implementation details are where it gets interesting, because there would be different instruction paths based on the component location on the system(s). This is not to say something done this way should use the network stack for communication even on the local system, since there is the "normal" way of doing it like a normal program. However, having different implementations that are implemented through different requests to a single server opens the door to, A) bugs and, B) extendibility.

I do not like the idea of plugins running direct calls. Plugins should be sending events/requests to a server/host, which then the server/host replys by executing the plugin code in a decided context.

[lpr]

I'll never buy anything Steinberg couldn't even get LE to work on multiple computers. Steinberg sucks.

[liquidsonics]

We can talk about crash robustness, but I don’t think the point was to push a crash into a more easily manageable place because we can just run entire plugins out of process if we want to. Logic is doing this with AUs on M1 (and Apple built scheduling support into the OS to help with the real time problems that would bring with it, as discussed in 2020 WWDC slides).

I always thought the main reason for the separation was keeping the options open for running the UI and DSP on separate systems in future. From the customer perspective, like a Waves SoundGrid or AudioGridder setup (but with the benefit of being a less closed system or without the lag of mirroring the UI over). The problem is Avid and Steinberg built it into the spec but never gave anybody the option to use it in their products from day one, had no way to enforce it, and so no plugin devs ever got a penalty for breaking the model by sharing a pointer to the DSP with the UI. Sadly it’s basically now too late as it would break so many plugins to try and split them out if either one of them came up with a DSP server for AAX Native or VST3 that ran the UI locally and only transmitted data through the specified plumbing. Missed opportunity, but in Avid’s case it’s easy to understand why as it would gut the market for external DSP. I bet Steinberg just didn’t have the resource to implement it relative to other priorities.

[Blue Cat Audio]

No plugin devs ever got a penalty for breaking the model by sharing a pointer to the DSP with the UI.

The problem is that this separation (and the ability to distribute plug-ins) should be made an optional feature for the plug-ins, as some plug-ins cannot live without it (for example, our plug-ins that can host third party plug-ins in mutiple formats HAVE to share the pointer to the plug-in between GUI and DSP).

Also, no plug-in format ever left the ability to optimize when both are loaded in the same process and when shared memory is available (you just don't know it inside the plug-in): when plug-ins have to share (very) large amounts of data between GUI and DSP that gets updated quite often, you definitely do not want to serialize the data and duplicate it when it can be avoided. It would just consume way too much memory and slow down the whole process.

The idea to distribute plug-ins on separate systems and providing the right plumbing in the host for it is definitely a great idea. Unfortunately it seems that (for VST3 at least) it was just a "what if" bullet point in the spec, without real thinking about it. The lack of an implementation just makes the whole thing pointless, and probably explains why it was underspecified.

[Urs]

Thoughts about out-of-process/machine/platform processing.

I'm thinking, out-of-process plug-in hosting has been done successfully using any known format, burden on the host side. Examples: Reaper, Bitwig. For this, there is no real need for asynchronous or serialised messaging. Maybe an open standard should supply example code for sandboxed hosting.

I furthermore became accidental part of the dev process for the separation in AU when Logic distributed plug-ins through Logic Nodes. It worked. People could have built AU render farms, but they didn't. Distributing plug-ins over several machines is very niche, and I believe it becomes less relevant with the path forward for CPUs to evolve. Consequently, Apple dropped the concept. Instead, protocols like Ableton Link pose a more practical way to sync different hardware devices. Sure, it was nice, but did it require asynchronous or serialised messaging? - Nope, not part of AU specs.

Lastly I think outboard DSP boxes are slowly but surely becoming an anachronism that merely serve as a dongle these days. Some even moved to general purpose CPUs, such that the code could simply run natively. Occasionally someone will do something outstanding within the limits of GPU power or such. But again, companies have harnessed these options just fine without the need of a plug-in format explicitly adding the burden of asynchronous or serialised messaging. They had to do that themselves, it's what they do, so why put that on us?

#----

About the ability to show alternative UIs.

Many hosts have the ability to show generic views assembled from parameter information, usually using system controls. This is very cool because it can be used to make the plug-in accessible to visually impaired people. The parameter system *is* already a means for the environment to display alternative views. It does not require the burden of asynchronous or serialised messaging between UI and DSP part of a plug-in.

UI displays on hardware devices such a the solutions by NI, Nektar or Ableton are extremely diverse. Some plug-in formats have parameter grouping features or offer static parameter mappings to controller pages. For NKS we have done dynamic mapping of parameters onto controller pages. This was such a complex effort that I do not think it's possible to satisfactorily generalise this in a plug-in format. Creating inter-parameter dependencies is just that invitation to mess things up. But I digress, parameter mappings don't require the burden of asynchronous or serialised messaging between UI and DSP part of a plug-in.

Which brings up alternative GUIs that pose different workflows and such. I've already read that people commonly wouldn't expect that more than one of those would be attached at a time. We already do that as part of our built-in UI toolkit. I don't know how JUCE and iPlug and VSTGUI do things, but I absolutely think that "skin support" is something that has nothing to do with the plug-in API and everything with the UI toolkit. Maybe a plug-in format can let the host know about the options available, purpose, size, features... Yet, here again, no need for asynchrono...

[arne]

It didn't come up here, so I will give you an idea why separation of DSP and UI controller is something natural. You all heard about plug-in delay compensation. If you think about how it is done for a moment you should see that this is done by running the audio "earlier" thru the DSP than you hear it. Just place a plug-in with a latency of 200ms into the last position of the chain. All plug-ins before it will process audio more than 200ms before it goes thru the speakers. Now if you don't separate your logic your UI will show changes from automation or DSP analysis data 200ms too early. You can think of it as separate time domains you should treat differently.
And Paul, I was not at the NAAM show when GMPI was discussed, just for the record.

[mystran]

It didn't come up here, so I will give you an idea why separation of DSP and UI controller is something natural. You all heard about plug-in delay compensation. If you think about how it is done for a moment you should see that this is done by running the audio "earlier" thru the DSP than you hear it. Just place a plug-in with a latency of 200ms into the last position of the chain.

The correct solution to this problem is to let the plugin know what the offset between the audio processing and the GUI processing is [edit: eg. by providing this into process() as part of the other timing information].

[mr.ardour]

And Paul, I was not at the NAAM show when GMPI was discussed, just for the record.

Sorry for the mistake. Some from extremely "high up" at Steinberg was there, nevertheless, and not surprisingly was adamant that Steinberg would not turn control of the VST spec over to any kind of 3rd party. Maybe you have some idea of who that would have been.

[arne]

It didn't come up here, so I will give you an idea why separation of DSP and UI controller is something natural. You all heard about plug-in delay compensation. If you think about how it is done for a moment you should see that this is done by running the audio "earlier" thru the DSP than you hear it. Just place a plug-in with a latency of 200ms into the last position of the chain.

The correct solution to this problem is to let the plugin know what the offset between the audio processing and the GUI processing is [edit: eg. by providing this into process() as part of the other timing information].

Sure, as long as I know all plug-in formats do it. But nearly all plug-ins not separating DSP and UI fail in this case. How about your plug-ins?

[arne]

And Paul, I was not at the NAAM show when GMPI was discussed, just for the record.

Sorry for the mistake. Some from extremely "high up" at Steinberg was there, nevertheless, and not surprisingly was adamant that Steinberg would not turn control of the VST spec over to any kind of 3rd party. Maybe you have some idea of who that would have been.

If it was someone "high up" than it could not be me . If someone had asked me at that time I would have said that merging VST and AU and using something like GMPI would benefit us all. But history was written differently and no one asked me.

[Blue Cat Audio]

It didn't come up here, so I will give you an idea why separation of DSP and UI controller is something natural. You all heard about plug-in delay compensation. If you think about how it is done for a moment you should see that this is done by running the audio "earlier" thru the DSP than you hear it. Just place a plug-in with a latency of 200ms into the last position of the chain. All plug-ins before it will process audio more than 200ms before it goes thru the speakers. Now if you don't separate your logic your UI will show changes from automation or DSP analysis data 200ms too early. You can think of it as separate time domains you should treat differently.

When you have latency in the chain, it will always be there, no matter what the software is trying to do to hide it. If you are using a MIDI controller or the GUI of a plug-in to change the value of a parameter, changes will happen later anyway.

Thinking that you can eliminate latency using compensation is NOT a good idea. If you display the value of parameters & automation with latency, it conflicts with user actions (that are NOT compensated). So all in all you are just moving the latency away from its source and it is probably causing more problems that it is solving...

[KenjiDeVries]

May I ask how DSP/UI separation would solve the latency issue? If the DSP is the one preparing audio and sending it to the UI, it'll still be sending it 200ms too early. Or am I getting it wrong? How does the UI know it has to delay it or what the right point in time is?

edit: BCA was faster

[mystran]

May I ask how DSP/UI separation would solve the latency issue? If the DSP is the one preparing audio and sending it to the UI, it'll still be sending it 200ms too early. Or am I getting it wrong? How does the UI know it has to delay it or what the right point in time is?

You basically need a wall-clock delta from the entry to the process() call to the estimate time the buffer is going to land in the driver. With such a delta, you can then send your audio in timestamped chunks and keep a GUI side buffer to compensate the latency.

This is a feature people want, but in general the information is not available to plugins and apparently some hosts are not even capable of providing it (eg. see https://forum.juce.com/t/how-to-determi ... ency/40252), although strictly speaking knowing the audio latency is not enough, because you also need to account for things like host render-ahead, hence the wall-clock delta.

[arne]

It didn't come up here, so I will give you an idea why separation of DSP and UI controller is something natural. You all heard about plug-in delay compensation. If you think about how it is done for a moment you should see that this is done by running the audio "earlier" thru the DSP than you hear it. Just place a plug-in with a latency of 200ms into the last position of the chain. All plug-ins before it will process audio more than 200ms before it goes thru the speakers. Now if you don't separate your logic your UI will show changes from automation or DSP analysis data 200ms too early. You can think of it as separate time domains you should treat differently.

When you have latency in the chain, it will always be there, no matter what the software is trying to do to hide it. If you are using a MIDI controller or the GUI of a plug-in to change the value of a parameter, changes will happen later anyway.

Thinking that you can eliminate latency using compensation is NOT a good idea. If you display the value of parameters & automation with latency, it conflicts with user actions (that are NOT compensated). So all in all you are just moving the latency away from its source and it is probably causing more problems that it is solving...

Exactly, if you don't separate your domains you will have this issue you describe. The host can solve this conflicts because it knows if a parameter is automated or if a parameter is currently controlled via a remote device.

If you go away from live input and into the mixing stage the professional engineer in front of the DAW wants to see what she or he hears. If the metering of the DAW would show early or if the faders would move not in time the engineer would be extremely irritated. This is part of a professional software today.