Developing a fast and responsive GUI - is JUCE too bloated?

[innervisions]

I think the ultimate answer is, it doesn't matter what your backend is, just limiting redraws to dirty widgets is the best optimization you're going to get.

That's very true. But it also gets more complicated than that:
In many cases, for example a visualizer that has to run FFT, you can move most of the calculation outside of the paint loop to a different thread, flatten it, and only then sync it with the paint call.

[camsr]

I would imagine that OpenGL or whatever would be a lot faster overall for rendering a lot of meters. Rather than doing a CPU bitmap fill or cull, sending a single value to scale a meter's value is far faster, when only looking at what the code says it is doing. There may be other issues involved, and I presume there is a inflection point where something very simple is better done on the CPU, and as the scene becomes more complex, should migrate rendering to the GPU.

Also the bitmap size could be a factor. If the image to render was very large, using the GPU may open paths to optimization that the CPU could only hope for.

[camsr]

Think of how video games render stuff... triangles, textures, shaders... all the CPU is doing at that point (beyond loading assets) is telling the GPU how to render that stuff.

[mystran]

If using a lot of stroking, it will be slower than CPU rendering since all GPU libs are constructing paths with CPU. Skia strokes all paths on separate textures and then it uploads it to the GPU on OpenGL for example.

If you are willing to rely on hardware anti-aliasing with MSAA, then it's actually possible to just expand each segment of a polyline separately into a stroked segment on the GPU, using either primitive instancing or geometry shaders (the former typically being faster and should work on even on D3D9/SM3 level hardware). For a traditional CPU rasterizer such strokes would be inefficient (ie. it's better to spend the extra time to generate a single stroke with less total edge-segments), but it works great with either stencil+cover (reduced overdraw compared to a complex stroke) or drawing the strokes directly (all segments are now convex). The downside is that the quality of anti-aliasing is hardware dependent (ie. how much MSAA/CSAA the hardware can do) and typically not great (read: sucks for 2D) for lower-end systems.

As far as I uderstand, the Skia approach is to use a traditional scanline rasterizer on the CPU to compute a converage alpha-mask as a texture, then send these to GPU for the actual painting. The scanline algorithm on the CPU can be made quite fast (but sadly doesn't easily translate to the GPU) and only uploading the single-channel coverage textures can save some upload bandwidth compared to a full RGB image when the amount of overdraw is low. Assuming you already have a CPU rasterizer, this approach is also quite easy to implement and also works great for text when combined with a cache atlas (ie. rasterize new glyphs on the CPU once, dynamically pack them into atlas texture and draw text as simple bitmaps).

The thing to watch out for with a CPU scanline rasterization though is that they can run into quite bad performance when stroking something like an FFT spectrum where the number of edges being tracked simultaneously grows large. In some cases, when stroking such a horizontal plot, it can actually be much faster to transpose the rasterization and do it over "scan columns" instead as this greatly reduces the number of edges you need to track (even though this is usually slower for simple paths).

Another option that avoids the above problem is to use scanline rasterization with "edge flags" instead, by processing each edge separately (much better for cache and saves all the sorting) and then integrating the edge flags at the end. To avoid having to store the edge-flags at a high-resolution, something similar to BLEP-synthesis can be done to anti-alias the edge-flags directly (ie. you add fractional contributions to two pixels). This approach is very fast, but suffers from artifacts (loss of anti-aliasing) where multiple edges of the same winding direction overlap within a single pixel (which sadly is often the case with complex strokes as computing strokes without internal overlaps is potentially even slower than using a rasterizer that deals with overlaps correctly).

[Urs]

Slow to paint to screen on macOS, especially if you use P3 screen

As this has been a large part of my recent work, let me share some findings:

The main problem on macOS is the size of the update rectangle. I've benchmarked the rectangles we get in drawRect vs. the rectangles we wanted to draw, and with some unfortunate coincidences macOS wanted us to draw 10 x more pixels than necessary. When optimizing this aggressively, the OS will start erasing background rectangles from 11.0, some already from 10.15. A good compromise is asynchronous layered drawing (layer and view opaque), using displayIfNeededInRect on the NSView (!), not CALayer, and only paint the rectangles that were asked for, and only if they are in the update rectangle. If your rectangles don't span the borders of the udpate rectangle, paint the whole thing.

The second problem is ColorSpace. If one doesn't use the same ColorSpace as the NSScreen one draws to, things turn to slow motion. I started to simply reload all images and redo all offscreen contexts if the ColourSpace changes.

The third problem is bitmap graphics. If the backing scale factor is 2 (retina), it is excessively faster using CGImages directly. If the backing scale factor is 1, it's a lot faster using CGLayers. If using latter, one should cache them at the size one is painting them.

If one uses offscreen contexts for pixel access, one should avoid using CGContext methods to draw complex paths. It's extremely slow. I've rolled my own path renderer which is a dozen times faster. There are good libraries for that, I suppose...

We're still in the process of finding best practices. Unfortunately the documentation is outdated and does not tell anyone about the differences between systems and how the combinations of methods interact.

Regarding the FPS issue, I think what happened was that early examples of VST used a 25 Hz timer or so to do background tasks. In our case, maybe others, this became the UI refresh rate. As we're speeding up things on Mac, we're also optionally deploying a 60Hz timer, which leads to a much smoother experience.

[mystran]

The main problem on macOS is the size of the update rectangle. [...]
The second problem is ColorSpace. [...]
The third problem is bitmap graphics. [...]

I just track my own rectangles, upload any CPU rendered stuff into a texture (updating just the changed regions) and composite by drawing a single triangle that covers the whole window using OpenGL or Metal. It works great and if you track the alpha-channel you can composite CPU rendering on top of GPU rendering or the other way around such that you don't even need to pick one or the other.

[Urs]

The main problem on macOS is the size of the update rectangle. [...]
The second problem is ColorSpace. [...]
The third problem is bitmap graphics. [...]

I just track my own rectangles, upload any CPU rendered stuff into a texture (updating just the changed regions) and composite by drawing a single triangle that covers the whole window using OpenGL or Metal. It works great and if you track the alpha-channel you can composite CPU rendering on top of GPU rendering or the other way around such that you don't even need to pick one or the other.

Yes, that's similar to what we do on Windows (draw in our own bitmap, then just blit that).

However, I started my efforts into plug-in development pretty much by developing my own UI framework based on Quartz. Traditionally (pre-Mojave I'd say), using Quartz/CoreGraphics was very fast and easy to maintain. It's only since recently that it became more or less unpredictable on Mac. That is, they apparently stopped caching frequently used bitmaps on the GPU when the bitmap is not aligned to resolution and color profile.

In the meantime I've implemented everything we need from a CGContext myself, except for font rendering. On Windows we use Freetype for this, but we have not yet worked out how to do font substitution for unicode (as we don't want to bloat our downloads with complete fonts). On macOS, sticking to the Quartz road, we get font rendering and substitution for free, and very fast. I'm not sure how well this performs with offscreen contexts, but in my experience this might become a bottleneck if we tried the texture method.

[Guillaume Piolat]

@u-he are you aware of the recent drawRect: problem too?
Some resources:
https://developer.apple.com/forums/thread/663256
https://gist.github.com/lukaskubanek/9a ... 28f2635779

Still wondering what are the best practice...

[Urs]

getRectsbeingDrawn almost always returns 1 rect spanning the entire drawRect.

I had instances where once every 30 seconds or so I'd suddenly get 20+ rects in there for about 5 seconds, while all the other times it just gives me a count of 1.

This is on Catalina.

The only reliable thing that works for me is displayRectIfNeeded immediately followed by needsDisplay=YES, while maintaining my own list of rectangles (plus layers etc. like I wrote). I have tested this with Catalina on 4k (Mac Pro) and Big Sur on Retina (Intel & M1).

Addendum: The second link does worry me... never heard of these backing contents flags...

[kingozrecords]

I would imagine that OpenGL or whatever would be a lot faster overall for rendering a lot of meters. Rather than doing a CPU bitmap fill or cull, sending a single value to scale a meter's value is far faster, when only looking at what the code says it is doing. There may be other issues involved, and I presume there is a inflection point where something very simple is better done on the CPU, and as the scene becomes more complex, should migrate rendering to the GPU.

Also the bitmap size could be a factor. If the image to render was very large, using the GPU may open paths to optimization that the CPU could only hope for.

It's hard to know who is to blame, but opengl builds for audio plug-ins will often crash. Whether or not it is due to the fact that a few developers used it wrong or whether or not it's the integration is to blame is anyone's guess.

I find using dependencies a guessing game, and because so many others will get it wrong; how can you be, in the customer's eyes the one who got it right?

Companies like audiod3ck (which got removed from pluginboutique for bad customer service) for instance have the same problem, and there is no fallback. Many computers do not have open gl because it's less popular. 2 years ago though, it was the world.

A dependency on a trend imo.

[Guillaume Piolat]

getRectsbeingDrawn almost always returns 1 rect spanning the entire drawRect.

In macOS >= 10.15, but in 10.14 it returned multiple rectangles. It was actually useful for quick rendering, so Apple removed it.

[Urs]

getRectsbeingDrawn almost always returns 1 rect spanning the entire drawRect.

In macOS >= 10.15, but in 10.14 it returned multiple rectangles. It was actually useful for quick rendering, so Apple removed it.

Ah, that's good to know, thank you. I've been so focussed on 10.15 and 11.0, I haven't even checked this on a machine with any OS prior.

[mystran]

It's hard to know who is to blame, but opengl builds for audio plug-ins will often crash. Whether or not it is due to the fact that a few developers used it wrong or whether or not it's the integration is to blame is anyone's guess.

It's probably either really poor drivers or some buggy plugin. Note that if some plugin doesn't manage it's OpenGL context correctly, then it might not always be obvious which plugin is causing the issue.

With correctly written plugins and a half-decent driver, you really shouldn't see any crashes.

Many computers do not have open gl because it's less popular.

This should never be an issue on macOS (and even though it's "deprecated" it's apparently still supposed to work even with M1). It used to be an issue on some old Windows systems (eg. some D3D9-class hardware, typically laptops with Intel-integrated graphics), but it really shouldn't be an issue anymore with any reasonably modern system. [edit: That said, on Windows you can also just use GDI BitBlt if you're software rendering, because unlike Apple's "solutions" it actually works well. On the other hand, if you want to use a GPU for rendering, then the type of hardware that doesn't support OpenGL generally is not up to the task anyway.]

[camsr]

I would imagine that OpenGL or whatever would be a lot faster overall for rendering a lot of meters. Rather than doing a CPU bitmap fill or cull, sending a single value to scale a meter's value is far faster, when only looking at what the code says it is doing. There may be other issues involved, and I presume there is a inflection point where something very simple is better done on the CPU, and as the scene becomes more complex, should migrate rendering to the GPU.

Also the bitmap size could be a factor. If the image to render was very large, using the GPU may open paths to optimization that the CPU could only hope for.

It's hard to know who is to blame, but opengl builds for audio plug-ins will often crash. Whether or not it is due to the fact that a few developers used it wrong or whether or not it's the integration is to blame is anyone's guess.

I find using dependencies a guessing game, and because so many others will get it wrong; how can you be, in the customer's eyes the one who got it right?

Companies like audiod3ck (which got removed from pluginboutique for bad customer service) for instance have the same problem, and there is no fallback. Many computers do not have open gl because it's less popular. 2 years ago though, it was the world.

A dependency on a trend imo.

The problem sounds then, like because all these computers we use have the same graphics chip, but the OSes and drivers want to do stuff differently. That's a real problem that OpenGL was built to solve (IIRC). Now it looks like Apple wats to be different again. However, Vulkan seems to be the next iteration of OpenGL, and AFAIK if the hardware supports Vulkan, the OS does also.

[camsr]

On the other hand, if you want to use a GPU for rendering, then the type of hardware that doesn't support OpenGL generally is not up to the task anyway.

Then forget even 60hz at that point also? Not quite sure what your point is here, unless you think the plugins should also run on non-x86 platforms without the last decade's graphics.