Solving MIDI input jitter on MacOS

[Kapsy]

Some DAWs insist on playing live MIDI notes "as soon as possible" which effectively means accurate timestamps are ignored live and everything rounded to the beginning of the next available block. This lowers the average latency slightly, but means there is one full block of jitter. Personally I find such jitter highly irritating, because it's essentially random so you can't really compensate (not to mention that what you hear on later playbacks don't match what you heard when you were playing). In a DAW that insists on doing this, even 256 samples at 44.1kHz feels kinda terrible for me, where as if we add pure latency (eg. with a delay plugin) with no jitter whatsoever I can easily tolerate at least another 20ms just fine.

I'm building a simple MIDI input for my instrument with recording and playback, and I'm trying to address the very issue of MIDI jitter as outlined here by mystran.

Currently I'm doing this by taking the current host timestamp at the start of the AURenderCallback, and using the difference with the MIDI events timestamp to calculate an anti jitter frame offset for the to-instrument event.

However the inTimeStamp provided by AURenderCallback has me confused. It seems to provide a "buffer to analog signal" time (about 13ms in the future for a 512 frame buffer), but I can't figure out a way to use that meaningfully, or if I should be at all. Any advice/alternative approaches are appreciated.

FWIW my current solution seems to be giving very consistent results. I'm just wondering if there's a more "correct" way (using CoreAudio/CoreMIDI frameworks or otherwise).

Code: Select all

afi:507511, afi_delta:22052
afi:551612, afi_delta:22050
afi:573662, afi_delta:22050
afi:595709, afi_delta:22047
afi:617760, afi_delta:22051
afi:639810, afi_delta:22050
(Spin lock timer producing MIDI events on a dedicated thread every 500ms)

afi:187310, afi_delta:221
afi:187529, afi_delta:219
afi:187749, afi_delta:220
afi:187970, afi_delta:221
afi:188191, afi_delta:221
afi:188411, afi_delta:220
(Spin lock timer producing MIDI events on a dedicated thread every 5ms)

afi:322470, afi_delta:881
afi:323352, afi_delta:882
afi:324235, afi_delta:883
afi:325117, afi_delta:882
afi:325999, afi_delta:882
afi:326881, afi_delta:882
(Spin lock timer producing MIDI events on a dedicated thread every 20ms)

Note the block size on my MBP is 512 frames @44.1kHz.

[mystran]

Your post is not entirely clear about exactly what is going on. Also whatever you're doing a "spin lock timer" is probably not a good idea, because especially on macOS the scheduler will get angry at you and stop scheduling your spinning thread and then everything breaks.

That said, if you're writing a plugin, then you can't do anything about this. The host sends you blocks to render and the host sends you MIDI events and both of these have timestamps and that's the end of it.

If you're writing a standalone program, then you have audio timestamps and you have MIDI timestamps. If these are using different clocks, the first thing is to convert the MIDI timestamps to the audio clock so that we can compare the two, but let's assume we've got that far, then the timestamp for a block we're processing would be expected to be a bit in the future (since we need to produce it before we can play it back) and the MIDI event timestamps would be expected to be some time in the past, because we can't expect to see them before they've arrived.

So there's now this 'gap' between the earliest possible MIDI event we might have received and the earliest possible audio time we can actually process that event, so somehow we must offset (=delay) the events so that their timestamps are always at least as big as the current audio time. The jitter issue I was talking about happens when you offset each event just enough to move it to the beginning of the block (ie. if eventTime < audioTime, set eventTime to audioTime). The problem is that different events are delayed by different amounts here. To avoid the jitter, you instead need to figure out a midiOffset that's just large enough that when we add it to every MIDI event timestamps, then we have eventTime+midiOffset >= audioTime so that we can process them, but we use the same offset for every event whether or not they even need it.

But again.. if you're a plugin, then the MIDI event timestamps you receive have already been massaged by host, they should never be in the past, so we should never need any offsets and if the host used different offsets for different events so that there is jitter then there's absolutely nothing you can do about it other than bug the host developers.

[Kapsy]

Thanks for your reply. Sorry I should have been clearer. I'm writing my own standalone program so I do have access to raw MIDI input (via CoreMIDI). The spinlock is just for testing and won't be a part of anything in production. It creates MIDI on/off events at known regular intervals on a separate thread so I can then measure the audio frame delta to see if my audio block offsets are accurate. I tried using usleep but it's wildly inaccurate, hence the spinlock. So you can see for instance, when I set the spinlock to 500ms the resulting interval in audio frames becomes 22050 give or take, which is about right for a 44.1kHz sample rate.

The approach you suggest is quite different to what I've done but it seems like it's worth a try. FWIW I take a system timestamp at the start of each audio block callback (AURenderCallback), so that there is always an audio timestamp in the past, to compare new MIDI event timestamps to. I then convert that to audio frames for delaying input to the instruments. I works out because calls to AURenderCallback are at regular intervals (every 11.6ms @512 frames), and the difference can just be applied in the next block, but it felt wrong using a timestamp in the past.

So what you're suggesting is applying an "delay" value to the timestamp, so that the MIDI note always occurs after the audio render callbacks timestamp (which makes sense), but still falls within the audio block? How would you go about calculating that delay? (assuming we're on a system where buffer sizes can change in while the application is running).

[mystran]

So what you're suggesting is applying an "delay" value to the timestamp, so that the MIDI note always occurs after the audio render callbacks timestamp (which makes sense), but still falls within the audio block? How would you go about calculating that delay? (assuming we're on a system where buffer sizes can change in while the application is running).

You want to calculate the delay based on the buffer size (and recalculate a new one when it changes; audio applications usually reset everything in this case anyway), because the whole point of smaller buffer size is that you get less total latency and the whole point of the delay is that we deal with latency specifically with the longer buffersizes... so basically your delay should be some multiple of the current audio blocksize. Ideally this would be exactly one block, but practically speaking you might want to add some slack, I don't know. If you're dealing with an API where the buffer can vary every callback, then use the maximum.

What to do if you need to delay an event past the next audio block (eg. because you added too much slack, or something)? What if we didn't add enough and missed anyway?

Well, let's look at a situation where we would be recording the MIDI first. Even if you don't actually record, this will make things more clear.

Our pipeline will theoretically look like input->offsetTime->record->play. If we apply enough offset to the timestamps before when write them to the sequencer in the record step, then we always record in the future and theoretically (with the caveat below) we can just play what we just recorded at the very same time, because the playcursor is (theoretically) always behind where we're adding new events.

So how do we match this for live when we're not recording and how do we deal with the possibility that sometimes we might screw up with the timestamps and end up with something in the past anyway?

We can do it like this: input->offsetTime->queue->play. So.. you take MIDI input, you add the delay offset, you dump stuff into a queue and then when you start your audio block render you drain the queue until it's either empty or the next event is beyond the block you want to render (so it should be processed next block). If you see events in the past, you bump them to the very beginning of the block (so as not to miss an event) and just eat the jitter.

And then the case where we want robust live playback AND we want to record? Here comes the ASCII art:

Code: Select all

input -> offsetTime ---> recordQueue --> sequencer
                     |
                     v
   sequencer  ----->[+]---> playQueue -> render

When user hits stop at the end of recording or we hit a loop point and want to replay the newly recorded stuff (or even once the playcursor is past the events; there are choices to be made here), we dump the contents of the recordQueue into the sequencer. Some variations are possible, but the key to solving the issue I was originally complaining is the location of the offsetTime block: we want to apply it to the live playback so that all events have consistent delay and we want these timestamps to be recorded, so that what you hear live is the same thing as what you hear when you hit play later.

Another nice thing about having queues like this and a clean merge point for two streams of data is that now these different things can live in different threads and communicate using those queues, which means we also solved the problem of on-screen-keyboard where we need to generate MIDI events from GUI thread where the user is clicking on the screen and somehow not lock the higher priority audio thread: if the queues are wait free, problem simply goes away.

ps. Bonus point: how do we figure out exactly how much delay do we need? Well, in render if you're missing events you need more offset, if the last event queued from live sources is past the block + some margin (the margin basically acts as hysteresis for an adaptive strategy) we need less offset. With this information you could even adjusts adaptively as long as you do it slowly (eg. over a timespan of seconds; in theory once your window snaps to the correct position it won't move further so it doesn't matter, but in practice we don't wanna go all bonkers if there are small hickups with CPU delays between threads or what not) so it doesn't screw up user's perception of musical time further.

[Kapsy]

Mystran, I logged onto KVR for the first time in well over a year and I realized that I completely forgot to reply to your post at the time. My apologies. Thank you very much as usual -- even though late I do appreciate it.

This is a part of my app that I have yet to fully resolve, and am definitely interested in trying your idea of an 'adaptive' strategy for figuring out a consistent MIDI delay. So I'll likely revisit this thread some time in the (hopefully) not too distant future. Thanks once again.