Possibly wrong realtime prioritization causing xruns

goli
KVRer
8 posts since 24 Apr, 2019

Post Thu Apr 25, 2019 4:32 am

I think I've found a strange behaviour regarding u-he VSTs on linux.

Short backstory:
I've been evaluating u-he synths (Hive seems to be the winner) and trying to optimize my linux system in regards to minimizing xruns.

I've applied some system optimizations as stated within these sources:

- http://subversion.ffado.org/wiki/IrqPriorities
- https://wiki.archlinux.org/index.php/Professional_audio
- https://forum.cockos.com/showpost.php?p ... count=1824

The important bits are the following:
1) Installed package realtime-privileges and added myself to the realtime group.
This created the following:

Code: Select all

~ cat /etc/security/limits.d/99-realtime-privileges.conf
@realtime - rtprio 98
@realtime - memlock unlimited
which let's me create processes with realtime priority up to 98.

(an alternative way, using the audio group, seems to be the following: http://subversion.ffado.org/wiki/UsingJ ... scheduling)

2) Established a certain IRQ priorization order.
Running

Code: Select all

ps -eLo rtprio,cls,pid,pri,nice,cmd | grep "FF" | sort -r
gives the following (minus the comments):

Image

This list shows all processes that are scheduled with "SCHED_FIFO" (indicated by "FF" part) (related to realtime priority scheduling):
- kernel stuff: highest priority things that seem to need to be there
- priorization (ex-/implicitly set):
- 95: rtc (not sure if this is still needed)
- 92: the IRQ associated with the USB port my audio interface is connected to (Motu 624)
- 70: jack daemon (set via priority parameter)
- 65: Reaper (DAW); implicitly set by itself
- the rest: all the other things that should be of lower realtime priority

All this to make sure the sound interface, jack and the DAW (in that order) are of higher importance than anything else (apart from kernel stuff).
This should help in avoiding xruns, which would happen if the audio interface or jack don't get adequat processing resources in time when they need it.

Using a program called _htop_ we can see all processes associated with Reaper:

Image

For this I've switched to tree view ('t') and selected the reaper process and all its child processes ('c') (press 'h' for help; 'U' to unselect).
Just note that there are already a bunch of child processes in a project without any loaded VSTs.


Loading non u-he VSTs:
If I load a non u-he synth (i.e. Cadmium; also tried Redux and OBDX), a couple more child processes are created:

Image

All those child processes not previously marked "yellow-ish" are new.

Listing all realtime scheduled processes with the "ps -eLo ..." command from above gives as no change compared to before,
which is how it should be. Regardless of any VST loaded we want our soundcard and jack to be of higher priority.


Loading u-he VSTs:
If I only load a u-he VST (in this case Hive Rev 8296), I get the following in 'htop':

Image

Once again a bunch of new child processes.

Now let's list all realtime scheduled processes with the "ps -eLo ..." command from above;

Image

Many of those newly created child processes are of highest user realtime priority "98" (in regards to highest for user; see start of post).
Therefore these rank above the sound interface and jack, which is not what we want in regards to realtime priorization (avoiding xruns).

Even worse: Loading more VST instances creates more such child processes, which in this case are all declared of higher importance than
the sound interface and jack.


Further tests:
- other u-he synths: same behaviour can be observed with Diva (rev: 8272) and Podolski (1.2.1)
- other DAWs: same behaviour can be observed with Bitwig (2.5.0) and Ardour (5.12). So it doesn's seem to be DAW related (or every DAW does it wrong...).

Final thoughts:
Maybe I'm wrong here and I have a flaw in my understanding of realtime priorization. Feel free to correct me.
Or maybe this really is a strange behaviour by the current u-he linux VSTs, which might actually provoke xruns on
systems that are set up this way in regards to realtime priorization.

Further background information:
- using manjaro linux (5.0.7-1-MANJARO #1 SMP PREEMPT)
- Reaper (5.974)

@u-he: Is all of this intentional? If so, what's the reasoning?
@other readers: If you have a system which is optimized in this way, maybe you could run a quick test yourself to verify or disprove this observation.

glokraw
KVRAF
7341 posts since 6 Oct, 2004

Re: Possibly wrong realtime prioritization causing xruns

Post Thu Apr 25, 2019 2:48 pm

Hi, and welcome to kvr! If you have a priority set in qjackctl, it is different
from the priority in etc/limits.conf. 99 max in limits.conf and 89 in qjackctl.
The ratio should be utilized, or set more conservatively.
Try 90 and 70 for awhile

There are important settings in Reaper buffering panel,
including setting the number of cpu's used. Try using less than all of them
for awhile. Reaper seems to distribute tasks to cpu threads very nicely.
.
There also may be issues with cooling fans, network and wireless
setup, and intensive graphics. Without your system details,
there's not much to go on. U-he synths can be used even in
rather stock linux setups. I just recorded some busy Hive and Bazille,
pushing my older i7 a bit, but no audible xruns. kernel 4.4.0-21-generic,
and the jackd2 2:1.9.12 from kxstudio

Are your xruns audible? I wouldn't worry if you can't hear them
in your recordings.

Where does the 'is all of this intentional' question come from? Time to optimize and
clean out some cobwebs somewhere closer to home, methinks :wink:

User avatar
lunardigs
KVRian
531 posts since 6 Jun, 2016 from Austin, Texas

Re: Possibly wrong realtime prioritization causing xruns

Post Fri Apr 26, 2019 2:02 pm

The biggest factor concerning DSP performance and xruns for me is setting my CPU governor to "performance" mode.

Code: Select all

cpupower frequency-set --governor performance
I'm running Fedora, so the package in question to adjust the CPU governor is "kernel-tools".

Code: Select all

#Install the utility
sudo dnf install kernel-tools

#Check your current cpu frequency configuration
sudo cpupower --info

#Check available governors
sudo cpupower frequency-info --governors

#Set the performance option if necessary
sudo cpupower frequency-set --governor cpufreq_performance
Meanwhile, my Jack priority is set to 70 and this works just fine. Also, my system's "real-time priority" starts at 70, per this command:

Code: Select all

sudo ulimit -r -l
I could increase the priority within /etc/security/limits.d/95-jack.conf, although I haven't needed to; the CPU governor makes the biggest difference for me.

goli
KVRer
8 posts since 24 Apr, 2019

Re: Possibly wrong realtime prioritization causing xruns

Post Sat Apr 27, 2019 2:22 pm

glokraw wrote:
Thu Apr 25, 2019 2:48 pm
Hi, and welcome to kvr! If you have a priority set in qjackctl, it is different
from the priority in etc/limits.conf. 99 max in limits.conf and 89 in qjackctl.
The ratio should be utilized, or set more conservatively.
Try 90 and 70 for awhile

There are important settings in Reaper buffering panel,
including setting the number of cpu's used. Try using less than all of them
for awhile. Reaper seems to distribute tasks to cpu threads very nicely.
.
There also may be issues with cooling fans, network and wireless
setup, and intensive graphics. Without your system details,
there's not much to go on. U-he synths can be used even in
rather stock linux setups. I just recorded some busy Hive and Bazille,
pushing my older i7 a bit, but no audible xruns. kernel 4.4.0-21-generic,
and the jackd2 2:1.9.12 from kxstudio

Are your xruns audible? I wouldn't worry if you can't hear them
in your recordings.

Where does the 'is all of this intentional' question come from? Time to optimize and
clean out some cobwebs somewhere closer to home, methinks :wink:
Thanks for the welcome!
I had max allowed user rtprio set to 98 due to it being the default config value after installing package 'realtime-privileges". Additionally it matches this guide, so I thought it should work. Anyway, I gave your suggestion a try. The priority change sadly made no noticeable difference. Regarding decreasing the assigned cores within Reaper I haven't done any heavy load tests yet.

I didn't really pay attention to the buffer settings in Reaper before you mentioned it, so thanks for that!
Image
I'd imagine that in theory setting this thread priority to something below highest should have an effect to the priority of the worker threads spawned by Reaper when loading an u-he VST.
But local tests have shown that it doesn't change anything currently: the worker threads are still of highest priority in accordance to configured user rt-prio regardless of changes made to this setting. But this might be a bug.

Some system details:
Image

lunardigs wrote:
Fri Apr 26, 2019 2:02 pm
The biggest factor concerning DSP performance and xruns for me is setting my CPU governor to "performance" mode.

Code: Select all

cpupower frequency-set --governor performance
I'm running Fedora, so the package in question to adjust the CPU governor is "kernel-tools".

Code: Select all

#Install the utility
sudo dnf install kernel-tools

#Check your current cpu frequency configuration
sudo cpupower --info

#Check available governors
sudo cpupower frequency-info --governors

#Set the performance option if necessary
sudo cpupower frequency-set --governor cpufreq_performance
Meanwhile, my Jack priority is set to 70 and this works just fine. Also, my system's "real-time priority" starts at 70, per this command:

Code: Select all

sudo ulimit -r -l
I could increase the priority within /etc/security/limits.d/95-jack.conf, although I haven't needed to; the CPU governor makes the biggest difference for me.
Thanks for the tip, but I've already tried that. I looks like it helps somewhat, so I'll keep that in mind when doing audio work.

So I've been doing a lot of reading and quite some experiments.
The thing that's bugging me the most is that even if I:
- close Reaper
- close web browser and other applications
- disable unneeded background services (syncthing, system update checker, ...)
- launch a minimal window manager (openbox only) instead of Xfce desktop
- stop network connectivity with

Code: Select all

nmcli networking off
- set cpu governor to performance
- disable pulseaudio via

Code: Select all

systemctl --user stop pulseaudio.socket
systemctl --user stop pulseaudio.service
- have practically only jack running via cadence with a sample rate of 44100 or 48000, period of 3 and a buffer size of 128 (and even 256)

I still get the occasional xrun out of nowhere after like 10-15 minutes while letting the PC idling and doing nothing.
Even tried it with a realtime kernel (4.19.31_rt18-1 provided by manjaro) and still get these sporadic xruns.

So I've probably gone down way too far into this "xrun-optimization"-rabbit hole.

One thing I'd like to test is if the full disk encryption I have on this system makes a problem. But there's no quick way to disable it; reinstall without it would be needed it seems.
Any prior experience that this makes a problem?

Furthermore I've found a couple things that could help in tracing this issue:
- https://www.alsa-project.org/wiki/XRUN_Debug
- https://lttng.org/blog/2016/01/06/monit ... latencies/
- http://people.redhat.com/williams/laten ... -howto.txt

The first two require special kernel settings, which would require building my own kernel, which I have no experience with yet (first try yesterday failed to boot).
Regarding test with cyclictest: I tried some, but I don't see how this actually helps in determining the real cause behind these sporadic xruns. These seem to be more of a general system test.

Any ideas on what could be the cause for such sporadic xruns out of nowhere?
Is there a reliable way to determine the cause of these xruns?

glokraw
KVRAF
7341 posts since 6 Oct, 2004

Re: Possibly wrong realtime prioritization causing xruns

Post Sun Apr 28, 2019 6:32 am

Unless you're hearing tons of pops/clicks
in your recordings, I wouldn't worry too much.
It seems like a solid productive system with lots
of good configuration steps in place. Priority settings
in the modern hardware era seem more precautionary than productive.
Stock kernels today have a decades advancements in tow.
You could chase the last xrun til you collapse in a heap,
but it will be more fun producing music.
There are so many wonderful sounds in the
U-he collections, it would be a shame to miss them
while analyzing an occasional inaudible dropout. :dog:

Have you verified that there are no audio irqs gettting
shared with other hardware? You could try making a text file
in /etc/modprobe.d named blacklist.conf, and add the name
of your motherboard audio kernel module.

Several linux users are very happy
with their Native Instruments Komplete Audio 6 usb interface.
Also, the Diva demo features mild crackling noise as the
demo limitation, and that's fooled quite a few more people
than will admit to it :hihi:
Cheers

kytdkut
KVRer
3 posts since 7 Nov, 2012

Re: Possibly wrong realtime prioritization causing xruns

Post Wed May 08, 2019 9:19 am

AUTO-ADMIN: Non-MP3, WAV, OGG, SoundCloud, YouTube, Vimeo, Twitter and Facebook links in this post have been protected automatically. Once the member reaches 5 posts the links will function as normal.
Hi, about the "tryng to optimize my linux system" part,

A little question first: are you running jack server in synchronous mode? You can set this option in cadence (configure -> engine -> server synchronous mode).

When you get below 128*3 things start to get tricky, but in my system (i7-6700k) I can get "stable" 16*3, and I even could do 16*3 on my old i3 laptop (with Renoise).

What I've discovered in all these years of tinkering with linux is that all those priority options are important but only to some extent. For me the most important options are:
- performance governor
- a cpu scheduler other than CFS (MuQSS or BMQ)

Also close your web browser :ud:

I dont think stopping network connectivity does anything... Perhaps because I'm using ConnMan? It is very lightweight.

You could also use something like Ananicy (https://github.com/Nefelim4ag/Ananicy (https://github.com/Nefelim4ag/Ananicy)) to set the nice and ionice values and scheduling policy of all the software you run on your computer automatically (just not the audio related software as it sets its scheduling policy and nice & prio values on itself).

I could go on and on about tweaks to your system, I would first recommend switching to another kernel linux-zen if you need CFS, linux-ck for MuQSS or pf-kernel (https://gitlab.com/post-factum/pf-kernel/wikis/README (https://gitlab.com/post-factum/pf-kernel/wikis/README)) (current personal preference, BMQ scheduler & other cool stuff).
I hope those are available on Fedora, if not you can compile it yourself; using modprobe-db speeds up compilation times a lot (*a lot*).

I hope I'm not adding too much noise to the thread.

Cheers!

abique
KVRian
739 posts since 26 May, 2013 from France, Montpellier

Re: Possibly wrong realtime prioritization causing xruns

Post Mon May 13, 2019 11:56 pm

Hi,

Thank you for this *very detailed* bug report!

The u-he plugins creates their own thread pool to process voices in parallel, and we need those to be as much realtime as your audio sound card because if they don't process on time, the buffer might miss the deadline. Both the audio driver and the u-he plugins are in the same boat regarding the buffer's deadline. On top of that the u-he threads should be sleeping if they don't have anything to do.

That being said, I will revisit that part of the plugin to check if there is no mistake somewhere...

Cheers,
Alex

Jack Winter
KVRer
28 posts since 23 Dec, 2017

Re: Possibly wrong realtime prioritization causing xruns

Post Tue May 14, 2019 5:07 am

I don't think there is a good way to determine at what priority to run such threads, except maybe checking at what priority the audio processing is called from the host. Maybe you can make the priority level user configurable? IMO setting the priority to the maximum possible is probably not the right solution..

abique
KVRian
739 posts since 26 May, 2013 from France, Montpellier

Re: Possibly wrong realtime prioritization causing xruns

Post Tue May 14, 2019 5:55 am

I'm changing it to be rlim_max - 10, does it make everyone happy?

Jack Winter
KVRer
28 posts since 23 Dec, 2017

Re: Possibly wrong realtime prioritization causing xruns

Post Tue May 14, 2019 9:25 am

No :) It's probably better but IMO it would need some thought about how to best do this..

For instance I have JACK running at rlim_max - 18, and reaper's audio processing threads at rlim_max - 24.

You could conceivably do something like this in the audio process callback:
pthread_getschedparam(pthread_self(), &policy, &param);

Then use that class and priority for the extra threads. That would make them similar to the app's audio processing thread. Or maybe if it's a realtime thread lower it by one, so that they run at a lower priority. That's the only automatic way I can think of.

If you are free to divulge the information, how do the plugins do this on Windows?

Another idea would be to let the user specify it via environment variables?

abique
KVRian
739 posts since 26 May, 2013 from France, Montpellier

Re: Possibly wrong realtime prioritization causing xruns

Post Tue May 14, 2019 12:58 pm

Then, here is what I can do:
- if there is an environment variale UHE_RT_PRIO we use it, if it fails we fallback to the default value
- default value is rlim_max - 24

Querying the audio thread's priority is smart but we need to create the thread pool before it starts processing, and I don't want to make that thing too complicated because it only matters if the system is overloaded right? Which is not a good situation anyway.

Jack Winter
KVRer
28 posts since 23 Dec, 2017

Re: Possibly wrong realtime prioritization causing xruns

Post Wed May 15, 2019 1:16 am

That sound good to me, rlim_max - 24 is probably a much better default.

Giving us an environment variable to control the priority makes it possible for us all to fine tune the priority. Doing it automatically would probably be more elegant, but I know of no other way than the "trick" mentioned above, which really is somewhat hacky and not a very clean solution.

Thumbs up from me in any case :)

Jack Winter
KVRer
28 posts since 23 Dec, 2017

Re: Possibly wrong realtime prioritization causing xruns

Post Wed May 15, 2019 1:19 am

Of course you'll have to consider what to do when rlim_max is lower than 25... There are no standard settings for this over the various distributions :S

abique
KVRian
739 posts since 26 May, 2013 from France, Montpellier

Re: Possibly wrong realtime prioritization causing xruns

Post Wed May 15, 2019 2:20 am

This is the fallback:

Code: Select all

    auto imax = static_cast<int>(rlim.rlim_max);
    params.sched_priority = std::max<int>(imax - 24, (imax * 3) / 4);
    pthread_setschedparam(pthread_self(), SCHED_FIFO, &params);

Jack Winter
KVRer
28 posts since 23 Dec, 2017

Re: Possibly wrong realtime prioritization causing xruns

Post Wed May 15, 2019 10:06 am

I think that would work.

Thanks for porting and caring about feedback!

Return to “u-he Linux support”