KVR :: DSP and Plug-in Development » Guidance needed for GUI coding [View Original Topic]
There are 33 posts in this topic.

---

camsr - Thu Apr 05, 2012 3:15 am

---

I want to start adding a custom functionality to my plugin ideas, but I am not sure where to invest my time in development of the GUI.

I would ultimately prefer to use vector based graphics for dynamic objects like meters and controls, but I am not sure how much overhead this would have. Also, I cannot stomach the look of the Windows dialogs, but I know I need something here that is crossplatform.

So, here is a simple list of the way things should work:
Vector graphics for dynamic objects, that can draw curves.
Programmable input handling for keyboard and mouse.
Can support crossplatform CUSTOM File dialogs with settings for default folder and favorites.


So far, Direct2D seems to be the candidate for windows. But I don't want to leave Mac users hanging, as many people have a need for plugins on the Mac. I know Mac uses OpenGL for hardware acceleration, but this is exactly the opposite of Direct2D, and would require twice the code, too much for myself.

Without hardware acceleration, things would be fine if it could be efficient. I do not like JUCE because of the license. I prefer to stay closed source at this time. Also, I don't like the way it handles mouse input.

So what options are there?

---

Caco - Thu Apr 05, 2012 3:32 am

---

Well, you could buy a license for JUCE so that you can do closed source

---

mystran - Thu Apr 05, 2012 4:18 am

---

Direct2D is ridiculously slow (actually even MORE so with "acceleration") and Vista+ only. On top of that Direct3D (which Direct2D uses for acceleration) causes all kinds of graphics glitches in some hosts; it's not really designed for plugins. Even if you don't mind the crazy API (of D2D; native D3D API is fine) it's more trouble than it's worth.

Unlike Direct3D, OpenGL actually DOES work perfectly fine on Windows with pretty much any host as long as you're careful about your OpenGL contexts.

---

camsr - Thu Apr 05, 2012 5:15 am

---

I would have thought Direct2D would have less glitches. But I am not interested so much in the 3D aspect. I was just trying AriesVerb and this is OpenGL. The performance was very good, with no sign of graphical errors except for the 3D filter plot, where polygons would jump around at a certain view angle. I did notice a fixed editor refresh rate. As most GDI+ based plugins refresh the editor at the host buffer rate, this stood out. Is this a pitfall of OpenGL inside a host, or is it implementation specific?

I tried looking up for myself a long time ago, how to handle UI input when using OpenGL, but I didn't find much. What are some recomendations for mouse and keyboard handling? I need to be able to have text input boxes and buttons and knobs/sliders.

---

mystran - Thu Apr 05, 2012 6:23 am

---

Yeah, I also thought D3D would work better until I actually tried. It seems that D3D is only really designed for the situation where it has complete control over drawing.

Refresh rate is really up to you. Doing host-buffer refresh is not necessarily a good idea though (eg for 8ms buffers you'd do 125Hz which is about 5 times too much already). I suggest doing refresh with WM_TIMER interval timer set at 20-25Hz or so; if CPU is very busy you'll miss some updates but that's really what you want anyway. Obviously if you don't need real-time visualization then you can also just update whenever contents change.

If you want good, stable vector images, you probably should not try to use OpenGL triangle rasterization directly. Even with anti-aliasing the quality can vary quite a bit. On the bright-side you get bilinear texture filtering for free so depending on what you want to do that might not actually be a problem. Or you could just draw shapes into a texture on CPU, then use OpenGL for blending (and possibly shading). Obviously depends on what you want to do.

---

rw - Thu Apr 05, 2012 7:39 am

---

That really depends. Intel (integrated) OGL support is generally lousy but you can safely target OGL 2.0 features without too many problems (YMMV).

However, if you wish to dive into hardware acceleration and not isolate your user base then it would be wise to implement multiple render paths to support a wide variety of hardware and select an appropriate one based on the user's hardware. Abstracting your low-level rendering code from high-level GUI draw calls will make this a breeze, as well as opening up many optimization strategies with regards to batching and combining geometry calls etcetera.

Direct3D support on Windows completely trumps OGL support but of course this means no cross-platform rendering code "straight out of the box". Again, supporting multiple rendering paths could incorporate multiple APIs but for VST GUI rendering (very lightweight in the grand scheme of things) this is probably using a sledgehammer to crack a nut. However, my background is in real-time 3D rendering and I have little experience with VST development so I will take you at your word with regards to D3D being problematic for VST GUI development.

---

rw - Thu Apr 05, 2012 8:16 am

---

OpenGL is a low level graphics API. It has no concept of user input, and quite rightly so. You should not be mixing rendering and UI code together as neither system should have any knowledge of each other so there should be no coupling between them. I do something like this for GUI development:

1) Grab UI
2) Let GUI manager decide which widget the UI should be applicable to
3) Perform any logic in response to UI
4) Set widget to "dirty" and thus needs to be redrawn
5) GUI manager iterates over widgets periodically and adds dirty widget's API-agnostic drawing information to scene manager
...
n) Scene manager builds drawing list from API-agnostic drawing information, optimizing where applicable
n+1) Graphics system operates on said list and compiles the necessary data to be sent to the GPU using API-specific calls and data structures, optimizing this data where applicable

Notice that the GUI system doesn't directly make any draw calls, especially widgets not making any draw calls themselves. By abstracting the low-level API draw calls from the GUI you give the scene manager (which handles the drawable items that actually get rendered) an opportunity to batch draw calls and optimize where possible. Likewise, the graphics system actually takes the list of API-agnostic draw calls from the scene manager and compiles the necessary data structures in whatever format the graphics API (OpenGL /DirectX/GDI and so on) requires, optimizing where possible. This allows you to swap and change your graphics API to suite the end user's capabilities, e.g. you could support oldschool video cards with low versions of OGL such as fixed function rendering (or even bypass OpenGL completely and use the OS's graphics API such as GDI/GDI+ on windows) or cards with shader support and specific (modern) OGL features and so on.

This is just some food for thought. If you're only doing GUI development then you can simplify things by integrating the GUI manager and scene manager together. Even this might seem heavy handed for simple GUI rendering (especially for largely static "scenes" such as plugin GUIs) as you won't really be doing any heavy work in terms of taxing the GPU or require high frame rates. However, IMO abstracting the graphics API from your GUI at least gives you flexibility if you ever need to change APIs or use specific API features that may or may not be available on all hardware. Having your GUI framework code littered with API-specific drawing calls is not only highly questionable practice but can also lead to headaches in terms of state m-a-n-a-g-e-m-e-n-t (<- this word is "spam" apparently?) and tightly coupling your generic GUI code with a specific API.

---

mystran - Thu Apr 05, 2012 8:27 am

---

I'm not talking about features. I'm targeting pretty much OGL 1.1 with a few selected extensions (eg texture_env_combine). What I'm talking about is actual redraw problems: D3D content not clipping properly, or not drawing properly or not invalidating windows below properly or whatever else non-sense, when the containing window or one it's siblings doesn't quite happen to do what D3D expects. Slightly different issues depending on whether you use D3D9, D3D9Ex or D3D10+ and whether you have Aero running or not, but none of them quite work. And that's on ONE system (this isn't anything esoteric; normal Win7 + NV GTX570). You don't even need to draw anything: just replace your WM_PAINT with Clear+Present (and Begin/EndScene and ValidateRect and all that other crap that you need to render an empty frame) and you already run into weirdness. Happens with D2D too.

Compare my experience with OpenGL: works.

---

rw - Thu Apr 05, 2012 8:34 am

---

You can safely support at least OGL 2.0 in this day and age, doing away with fixed function pipeline and the awful texture combiners. All that stuff is emulated by the driver anyway.

---

camsr - Thu Apr 05, 2012 8:56 am

---

rw, Opengl works as a renderer only, and the VST requires a UI. How would you implement it on top of the VST, so it is able to isolate keystrokes and mouse clicks within regions? I have seen this is many video games in my life, so I know it must not be too complicated. I believe the most popular solution was directinput, but if this could be done in something crossplatform, it would be better.

In the end of this idea, I would like to see my GUI work something like this...
A round knob with a text readout in the center.
The ability to update that text readout while turning the knob, and while the host automates it's parameter.
And the ability to right click/shift click on the knob to input a text value directly, with the text updated.

---

camsr - Thu Apr 05, 2012 9:13 am

---

My idea is to refresh the editor at buffer rate. For users with onboard audio only, this could mean refresh times of 190ms. I don't know if it would be too much trouble to use v-sync or not, knowing how that is driver dependent. It would be cool to see the editor refresh linked to the monitor refresh rate if the buffer is faster than.

Is there any chance the editor could fall out of sync with the host?

---

mystran - Thu Apr 05, 2012 9:27 am

---

Actually turns out some people are still using cards (eg G550) that don't even support texture_env_combine. I'm sure OGL2.0 gives you a fairly wide user base, but I like to keep as much backwards compatibility as reasonably possible. It's not like it matters much for GUI drawing anyway.

---

mystran - Thu Apr 05, 2012 9:41 am

---

You can sync to v-sync, sure, but good luck getting that to work in two plugins at the same time. So in practice it's a BAD idea. Also for purposes of visualization 60Hz (or whatever) is total overkill anyway. Even video really only needs about 25Hz. Just because it's hardware accelerated doesn't mean it's free so I'd just use something reasonable and save the rest of the CPU for more important stuff (like audio).

Just to put things in perspective, lots of console games these days run at 30Hz (eg half vsync). Those that run at refresh (eg 60Hz) usually do so in mostly because higher refresh rate means less input lag from user to screen. If it takes 3 frames (note that this is pretty much the best case and doesn't include whatever latency the monitor itself has) from the button press to gun shooting, then at 60Hz it's ~50ms and at 30Hz it's roughly 100ms; this difference is noticeable where as 30Hz vs 60Hz refresh really isn't.

As for staying in sync, you should really deal with the situation that GUI refresh can't keep up with the audio. Sooner or later you'll run out of CPU in GUI thread anyway, whether that's because a random background task or just too much audio code. While it's not necessarily something that happens in "normal operation" you should still be prepared so you don't crash or cause a system lockup or anything else stupid like that. Normal thing to do is to drop video frames; most of the time anybody won't even notice unless you drop several frames in a row or your framerate drops for a longer period of time.

---

camsr - Thu Apr 05, 2012 10:02 am

---

Okay that is a good point. Is this why no one uses it?

---

rw - Thu Apr 05, 2012 10:33 am

---

Don't use DirectInput, it's a wrapper for the usual window messages and even Microsoft reccomend against using it.

If you are rolling your own editor using VST 2.x then for Windows you'd create your window as a child of the host-supplied window handle (void* parent cast to HWND) and then your message pump would pass the usual OS messages to your newly-created window via the callback WNDPROC function. You'd handle your mouse/keyboard messages however you please, i.e. passing the pertinent information to your GUI whereupon it can handle the mouse/keyboard input by processing it and testing widgets to see whether the UI is pertinent to them.



Your actual GUI rendering will be independent of the GUI logic. In a simplified way, you'd take the UI and do some bounds checking for each widget to see which widget is the focus of the UI. You'd then react to that input by informing the plugin that a parameter has changed and then the plugin would do whatever function in response to the input.



Your rendering can be refreshed however you choose, it certainly need not be refreshed each time your audio block is processed. As GUIs change infrequently, you need only update the GUI in response to GUI parameter changes (and only the section of the GUI that actually needs to be changed).

---

rw - Thu Apr 05, 2012 10:39 am

---

It depends what you're drawing. I'm working on a multi-channel oscilloscope so I'm offloading as much drawing and DSP to the GPU as I can to free up the CPU for more important tasks. None of this would be possible with the fixed function pipeline and I'm not concerned with isolating the tiny fraction of pre-2000 GPUs that cannot handle such duties. That's not to say that it couldn't be implemented with OpenGL 1.1, rather the CPU and RAM overhead would be far greater than using OpenGL 1.1 features. For garden variety GUI rendering, I agree, OpenGL 1.1 is certainly up to the task.

---

mystran - Thu Apr 05, 2012 4:56 pm

---

Yeah ok, I admit shaders can be useful if you can take a raw buffer of audio data and transform it into a nice complex visual representation completely on GPU. If I wanted to do something like that then I'd probably drop fixed function support as well. It's not like I ever touch any state manually anyway (eg. API wrapper implements simple "shaders" that map desired functionality to texture combines and render state). In fact I don't touch the API at all directly; in my GUI lib it took me a day to write OpenGL backend to replace the original D3D backend and I can't really remember needing to change anything outside the actual backend (the two backends can even happily co-exist in the same binary so you could select one or the other at runtime; maybe some day I'll write a software-only fall-back to support the guy that's still using CL5434).

Anyway, for "basic user interfaces" I guess the only thing that could reasonably benefit from shaders would be text rendering; since it pretty much comes down to a quad per glyph (unless you cache static text directly), it's a bit wasteful on bus-bandwidth to push 4 vertices per quad when you could instance on hardware. For on-the-fly (ie not cached RGB like GDI) sub-pixels using fixed function is even more data since the whole sub-pixel crap doesn't map very nicely to texture combiners (or I'm just stupid; that's also possible) where as all of it would be trivial with simple shaders. That's something I might actually do that at some point (as an optional optimization where supported, ofcourse ).

In my "other project" (ie the reason I haven't really released much audio stuff lately, if we ignore the fact that I've been playing too much FPS games) that does a bit more on the GPU than "cheap alpha-blending", I actually don't touch the fixed function at all. Fortunately that doesn't need to run in a plugin so it can happily use D3D (and has performance requirements high enough that you can just ignore any Intel graphics even exist).

---

camsr - Fri Apr 06, 2012 9:56 am

---

I am only okay with C and not very good with C++ yet. I have got a framework up and running already but my lack of knowledge is holding me up. How do I give variables to the editor class from the plugin class? I tried using global scope but I did it wrong lol. Does it involve more inheritance than just one class from AudioEffectX and one from AEffEditor? I am using the VSTGL sdk.

---

camsr - Sun Apr 08, 2012 6:29 am

---

Testing further, I found that having multiple instances on the screen caused frames to drop between them. The more instances, the more frames dropped. Vertical sync is not the issue. I am guessing everything would have to be done in the same context, and for a VST host that might require IPC? Well I am disappoint because OGL looks amazing for plugins.

---

mystran - Sun Apr 08, 2012 8:02 am

---

If you are not creating a separate context and properly setting/restoring it every time you touch OpenGL then you're asking for trouble. All the state and resources (eg textures etc) are tied to particular OpenGL context, so you must always make sure you have the right one active or you'll be stepping on other plugins/hosts/whatever feet.

As far as dropping frames, if you try to wait (eg glFinish() or probably on SwapBuffers if you try to use VSync) then no other instance/plugin will run while your stalling the GUI thread (never EVER wait in GUI thread). This obviously also means no other plugin will update while your waiting. If you're not already using it, enable double buffering too (to make sure SwapBuffers is as low overhead as possible).

Other than that though, I don't see why there should be any problems. Obviously if your drawing is very CPU/GPU heavy and/or you do it too often, then you can generate more work than your system can handle which will obviously lead to dropped frames.

---

camsr - Sun Apr 08, 2012 2:08 pm

---

Just fixed it VSTGL didn't include GetDC and ReleaseDC where it should have. Now everything is in sync ALMOST, because of how the editor receives variables. I am just pushing the highest peaks with a constant decay follower, and I guess the editor is reading them one after the other. The discrepency is very small, but there. With a larger buffer, it is unnoticable. Seems kinda backwards but maybe I never noticed with all the normal VSTs.

One thing that is troublesome is the windows in my host (FL) when dragged get choppy. Do you think it would help to use some kind of framerate limiting?

---

camsr - Tue Apr 10, 2012 6:21 pm

---

I haven't been able to solve the problem of framerate dividing between additional instances yet. I don't know if the problem is the Timer class in VSTGL or if I am not using the context correctly. Do I need to destroy the rendering context after every update, or just release the device context? void VSTGLEditor::refreshGraphics()
{

    dc = GetDC(tempHWnd);
    wglMakeCurrent(dc, glRenderingContext);
    draw();
    swapBuffers();
    //wglDeleteContext(glRenderingContext);
    ReleaseDC(tempHWnd, dc);
}

This has the Timer interval (refresh) dividing by the number of instances (functionally). The Timer is a Singleton and as far as I can tell, uses static types to share it. The VSYNC and Antialiasing settings made no difference.

---

mystran - Wed Apr 11, 2012 7:10 am

---

You obviously need one timer per plugin instance, so singleton won't work.

As far as setting/restoring contexts (remember you also need to do this when you want to do things like load textures or whatever; I'd really recommend a RAII helper unless you never touch OpenGL outside your drawing logic)..



void VSTGLEditor::refreshGraphics()
{
  HDC hdc, hOldDC;
  HGLRC hOldRC;

  // get previous DC and render context
  hOldDC = wglGetCurrentDC();
  hOldRC = wglGetCurrentContext();

  // get DC for the hwnd of the window to draw in
  hdc = GetDC(hwnd);

  // set rendering context and DC
  wglMakeCurrent(hdc, glRC);

  // draw, swap, whatever else you want in the context
  draw();
  SwapBuffers();

  // clear errors to avoid confusing other code
  // it's fine to call this and ignore errors if you don't
  // care about them; it's just so it reports success on next call
  glGetError();

  // restore previous context and DC
  wglMakeCurrent(hOldDC, hOldRC);

  // release the DC we allocated
  ReleaseDC(hwnd, hdc);
}


Note that if you use OWN_DC (which I'd really NOT recommend) then the above might not be correct. Remember to ValidateRect() the window in response to WM_PAINT or you'll immediately get a new WM_PAINT request afterwards.

---

camsr - Wed Apr 11, 2012 10:48 am

---

wglMakeCurrentContext(), is this a depreceated function? I can't even find it in my headers and google doesn't return much, not even from opengl.org. I tried MakeCurrent instead and everything looks correct. Thanks!

ValidateRect(tempHWnd, NULL) I added this after ReleaseDC, but I don't know if it's needed?
In GLWndProc(), the case of WM_PAINT calls to idle(), and GLWndProc returns DefWindowProc(). I am confused as to why this is needed, but it looks for compatibility.

Is it the Timer's fault for the dividing framerate?! It only divides when another instance's editor is open, so I guess it is something to do with Timer. I thought the whole point of the static singleton was to prevent contexts from stepping on each others toes? Or is this unnecessary (are the OpenGL calls buffered?)

I've seen quite a few examples of multithreaded programs with multiple GL contexts in different threads (including JUCE), and everything looks in sync and full framerate! I would use JUCE for this, but I cannot afford the license and I want to sell a couple plugins I make.

Actually, I am not sure I can close source my VST with VSTGL? The License says this:
"Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the "Software"),
to deal in the Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software."

This is only the VSTGL portion or the VST itself?

---

camsr - Wed Apr 11, 2012 11:08 am

---

Here is the Timer code for anyone interested:

CPP
H

---

mystran - Wed Apr 11, 2012 1:05 pm

---

No, wglMakeCurrent() ofcourse.. sorry I had a brain malfunction while writing that.



When a window (or a region of the same) is "invalid" or "dirty" the system will keep generating WM_PAINT messages until you validate it. BeginPaint() that GDI drawing normally calls does this automatically, but otherwise you'll need to ValidateRect() or something similar.



Urgh. I guess DefWindowProc() will validate the window for you, but where do you draw then?



You should have one timer per instance, no? Normal way to do this is to setup a WM_TIMER (or similar) for the editor window. OpenGL contexts have nothing to do with timers. The OpenGL context is like a virtual device; you can have (within reason) as many as you like, and they are pretty much completely independent. The point is, any OpenGL state or resources you set/alloc/whatever belongs to a single context, so you can safely play in your own sandbox (context) without confusing other code in other sandboxes.

---

camsr - Wed Apr 11, 2012 1:32 pm

---

As far as I can tell the timer DOES create a new timer instance for each editor instance. It is using map.h with keys, but I don't know what that is yet.

---

camsr - Wed Apr 11, 2012 10:28 pm

---

So I finally figured out how to utilize the editor and effect pointers. This stuff is pretty new to me as most of my programming hasn't seen OOP and more inline.

I am trying to decouple my editor's redraw rate from the host's rate. So far, the graphics refresh is timer based, so there is no coupling there. EXCEPT, in the case of utilizing a variable that is updated inside the processReplacing() loop. Is there a way to push variables out of, or pull variables into the editor from processReplacing asynchronously? Or will processReplacing guarantee completion before other functions are called? In case you are wondering, I am attempting to make an awesome peak meter.

---

mystran - Thu Apr 12, 2012 12:38 am

---

For most purposes you can expect there to be two threads, one for audio, one for GUI. Which functions run in which threads isn't exactly very well documented, but generally you can assume that resume/suspend/processReplacing/Events run in audio thread (so only one is active at any given time) and anything to do with the editor runs in GUI thread.

Then there's stuff like set/getParameter/Label/Display which can pretty much be called in any thread at any time (and should probably be re-entrant too). There's also a lot of gray matter that one would expect (by common sense) to run in one thread or another but you probably shouldn't assume too much (eg hosts might change presets from GUI thread while processReplacing() is running or other similar stuff that you probably should be prepared for).

Anyway, basically this means that audio thread does audio processing and GUI thread does GUI interaction and drawing and these two go on completely asynchronously. The audio thread runs at high (usually RT) priority and obviously shouldn't ever have to wait for GUI thread (which usually runs at default priority; high priority threads so never wait for lower priority threads). Waiting for the audio (in GUI thread) is generally fine as long as the critical sections (ie the duration of wait) is sensible (holding a lock for the whole processReplacing() is not a good idea).

Now, you can really use either push or pull or a combination of both (which is what I do).

The way I do it is basically to keep a local copy of data in audio thread, then at the end of every processReplacing() I attempt to take a lock (but without waiting; eg POSIX calls it trylock() and in Windows you specify zero timeout so it fails if it can't succeed immediately). If the lock succeeds, the data is copied into a shared buffer (if not, we'll just try again next processReplacing() call). In GUI timer then, I take the same lock (except this time it's fine to wait since audio is higher priority) and copy the shared buffer to a GUI local buffer.

So there's the "audio local" buffer that's always up-to-date. Then there's the "shared" buffer that's updated regularly, but can sometimes miss an update. Then there's the "GUI local" copy of whatever was in the shared buffer during the last timer interrupt; the last copy is just so the "GUI side" critical section (ie duration the lock is held) is the minimum possible (eg a single memcpy of the buffer) which minimizes the chance of audio thread not being able to get the lock when it wants (one would miss a whole lot more updates if the lock was held for a complete redraw; while missing updates might sound bad it doesn't make much difference and in practice it almost never happens anyway if the critical sections are short).

If you don't mind adding a bit of latency to the pipeline you can further optimize it by adding a flag (protected by the above mentioned lock) that the GUI sets to notify that more data is needed, and the audio clears to notify GUI that new data is available. Doesn't matter for a peak meter but could be useful if you're copying more data around.

The above isn't the only way (nor do I claim it's the best way) to handle the issue, but it has the nice properties that it (1) is safe and (2) can handle arbitrary blobs of data just fine (well, as long as the amount of data isn't huge; if you needed something like last 10 seconds of audio for a scope/spectrum you might want to do some additional logic to only copy the changes, but the general idea still works).

---

camsr - Sun Apr 15, 2012 11:09 am

---

Thanks for the help mystran. I didn't know VST is already multi threaded. Is it the same on Mac OS X?

My original idea was to use a circular buffer and iterator that loops around the end of an array, with the index being incremented per sample. Is this thread safe? Because the way it works now, the values it needs from processReplacing are only received at the end of the block. I want the editor to receive those values, on its interrupt. I don't understand multi threading, but thanks for any advice.

---

mystran - Sun Apr 15, 2012 12:34 pm

---

Unfortunately it would take a bit too much effort to properly explain multi-threading and the potential problems and solutions on a forum post, so I suggest you search for some resources on the subject.. but basically it comes down to this: can you prove that it's thread-safe?

If you can't prove it, it probably isn't.

---

camsr - Sun Apr 15, 2012 2:04 pm

---

I am going to try what you suggested. I know it's a nasty thing for the cache when the code is large, but for a peak meter it might work.

---

camsr - Wed Jun 06, 2012 9:33 pm

---

Got the framerate fixed, Vsync was enabled because of a bad definition. Thanks for all the help

---

There are 33 posts in this topic.