KVR Audio

tdmusic · Post by **tdmusic** » Thu Sep 06, 2012 11:08 am

Hi all,

Does anyone else experience issues with the timing of MIDI input to iOS apps (e.g. synths)?

Specifically, I noticed something didn't sound "right" timing-wise when playing MIDI out of Logic/Live, so I carried out a bit of an experiment, and I'm seeing quite a lot of jitter in the audio of a 16th note pattern output from my computer to my iPad. I initially suspected it could be down to using wi-fi network MIDI, but hooking up my Focusrite Saffire 6 USB's MIDI out to my Behringer BCR2000's MIDI input (connected to my iPad via CCK as a CoreMIDI interface) seemed to produce roughly equivalent results. The jitter seems to vary from app to app, but even in the best case (Animoog seems quite tight), it looks like the timing of notes varies +/-5ms from when it should arrive (and every now and then, one is way off, but I can let that slide as it seems quite rare).

I connected the Focusrite MIDI output to the Berhringer MIDI input, with both connected to my Mac, and repeated the same experiment (but recording the MIDI coming in to the Behringer rather than the audio coming out of the iPad) and it looked like the timing was much closer to what I'd expect (although not perfect - I've not got much experience with external MIDI but I know not to expect miracles

), so it looks like the issue is with the iPad part of the setup rather than the MIDI interfaces.

I suspect this is unavoidable due to a combination of iOS and app-specific issues, having found some other discussion on the internet (e.g. (http://forums.blipinteractive.co.uk/node/2140), but thought I'd check if anyone had any other input or workarounds

It's a bit of a shame as I bought the Focusrite specifically to record from my iPad and was hoping to be able to loop MIDI sequences on my computer, play with synth/FX parameters on the iPad and record the output as audio to use in a DAW/export to another iPad, but the timing doesn't seem tight enough to use this for any kind of rhythmic sequence. Not the end of the world as I can still record from apps sampler style, one note at a time, or time-correct the audio recording of a sequence, but I was hoping the MIDI timing would be tighter than it is.

Cheers,
Tom

coverup0204 · Post by **coverup0204** » Fri Oct 12, 2012 12:21 pm

I get the same issues using the iRig MIDI interface - it is very disappointing because the sounds you can produce with some of these apps is amazing, matching or exceeding hardware synths for a fraction of the cost.

16th notes at 120 bpm, or jeez, even 80 bpm shouldn't be a problem. I think it's hard to take the iPad or other iOS devices seriously for musical production if they can't even have reasonable timing.

Here is a video I made detailing the problem:

I guess we are pretty much limited to pad sounds at this point for the software instruments. In-app timing for sequence based apps like Korg iMS-20 and Nanostudio are totally fine, but I don't think they use MIDI internally.

Inter-app midi is not much better... for example, when I sequence with Phaedra, I get similarly sloppy notes. What a bummer!

I would love to know if there is a solution to this! Ideas or additional info about what could be causing the problem?[/url]

Tim Webb · Post by **Tim Webb** » Fri Oct 12, 2012 4:05 pm

That is a great video of your issues, coverup! It caught my attention so I did my own tests. I had similar issues, but it seems like they are limited to Animoog for iPhone, like you were using there. None of the other apps I tested had any serious jitter issues, including Animoog for iPad!

I've documented the results in this article:
http://discchord.com/blog/2012/10/12/mi ... n-ios.html

coverup0204 · Post by **coverup0204** » Fri Oct 12, 2012 11:06 pm

Thanks for the suggestion Tim. I followed your advice and did similar tests to document the results I am seeing here, which are poor no matter what app I use.

Method: 16th notes at 80, 120, and 160 BPM
Equipment: Ableton Live 8 out through Roland Quad-Capture into iRig MIDI adaptor, iPad 3rd Gen
Apps tried: Magellan, PPG Wavegenerator, Sunrizer (all native iPad)
Summary: Lots of jitter, although Magellan at 80BPM did better than the others. PPG Wavegenerator was the worst.

Results:
Sunrizer 160, 120, 80 BPM (in order)

PPG Wavegenerator 160, 120, 80 BPM (in order)

Magellan 160, 120, 80 BPM (in order)

Next thing to try is wireless MIDI via rtpMIDI to see if there is a difference there. Will post again with results. Tim, you are using the camera connection kit right?

SingleCell · Post by **SingleCell** » Fri Oct 12, 2012 11:44 pm

From my tests, I believe this has to do with the way a lot of apps process their audio. I've run a few test with getting Caustic running on iOS as well as with NanoStudio on iPad, and have come to the following conclusion:

I hope I can explain it clearly...

It has to do with the buffer size, i.e.: latency and the time it takes to generate the audio.

Let's assume a decent size buffer for the sake of explaining, say 10ms.

1 - At the beginning of creating a buffer's worth of audio, I look at the state of the note ON/Offs for the synth that came in from MIDI, let's say there's a new note ON, I start producing audio for that buffer's worth of audio, 10ms. It takes less than that time to create the 10ms of audio, let's say it takes 1ms. If a note OFF event comes in 3ms after the note ON, it's too late to respond to it because my audio has already been generated and that part of the program is sleeping, waiting for the soundcard to ask for new stuff. It's only 10ms AFTER that that I'll act on the Note OFF, stopping the note. So you got 7 extra ms of synth note you didn't want.

If you've got NanoStudio you can try this for yourself. My test involves a AKAI MPK Mini using the built-in arpeggiator plugged in via the CCK, this eliminates rtpMIDI network lag. First, set NanoStudio's latency setting to it's highest. With the default 1/16 at 120bpm on the AKAI and a simple 1 OSC no ENV setup in Eden, press a key. Notice the jitter in note ON/OFF. You can clearly hear it. Then switch Nanostudio to lowest latency. Try again, notice the timing is pretty tight this time.

Hope this makes sense. Basically what's happening is MIDI events get quantized to the audio buffer size. The smaller the buffer, the less quantization errors you get.

The difference in how apps respond is probably due to the different buffer sizes they use. My hypothesis is you'll probably see a direct correlation between latency in different apps and their MIDI jitter.

coverup0204 · Post by **coverup0204** » Sat Oct 13, 2012 2:45 am

SingleCell, your explanation makes sense and would explain why the notes deviate gradually and then fall back in place before drifting again, and then falling back in place like you see in my test results.

I'm sure there is a reason for this, but it seems to be a terribly lazy way to process audio - forcing note lengths to only be allowed to exist in lengths that are multiples of the audio buffer? I don't think that's the way VST softsynths are handled in a PC DAW... notes should be able to start and stop mid-buffer. Correct me if I am wrong here!

Or maybe it has to do with the update polling frequency of the iRig adaptor too? Something is not timing accurately and reducing the resolution for note on/off.

fessaboy · Post by **fessaboy** » Sat Oct 13, 2012 2:16 pm

SingleCell wrote:From my tests, I believe this has to do with the way a lot of apps process their audio. I've run a few test with getting Caustic running on iOS as well as with NanoStudio on iPad, and have come to the following conclusion:

I hope I can explain it clearly...

It has to do with the buffer size, i.e.: latency and the time it takes to generate the audio.

...

This sounds spot-on to me. In iOS, an app can request buffers of different sizes; ones I've seen are 256, 512, or 1024 samples (and probably higher multiples of 2 happen as well). What the app requests, and what it gets, can be two different things; the operating system knows the hardware it's on, and probably estimates a few things before giving the app a particular size buffer.

Periodically, iOS asks an app to fill up a buffer with audio data -- and the expectation is that there's quick turn-around on that. iOS then takes the audio, and queues it up for output. If the app doesn't return the audio buffer in time, the audio gets choppy from dropped samples (I've heard this with SampleTank in particular, and a few other apps depending on what's running in the background).

Processing audio in buffers is the right thing to do in most cases; it cuts way down on the CPU demand, which lets the device handle multiple apps running, phone calls coming in, and everything else an iPad or iPhone might do. With a 256 sample buffer, it's hard to hear a 1 buffer delay (I can't, but I don't have the best ears in the world). Once you get into the 1024 or 2048 sample range, though, it starts to get very noticeable.

WIST (someone correct me on this if I'm missing some details) is essentially sending time stamped MIDI events -- and these are sent ahead of the time that a note on/off is supposed to occur. This way, the audio buffer can get filled with data that starts at exactly the right time (even for big buffers), and everything is tightly synced. Notes being played live on an instrument can't take advantage of WIST -- but anything that's sequenced can.

** A side question -- which iPad? The iPad 3 is pretty fast, and an app may be given a smaller buffer size by the operating system. The iPad 1 might get a larger buffer -- resulting in more jitter.

coverup0204 · Post by **coverup0204** » Sun Oct 14, 2012 10:38 pm

fessaboy, I am using the iPad 3 and I get the issues you see in my video with all synth apps to a greater or lesser degree, but it is noticeable in every case and none of them are usable as synth modules in a MIDI environment right now.

The audio buffer "delay" is not the problem - if there were a consistent delay it would be fine because you can compensate for that in the DAW, like with Ableton's track delay.

The issue is that notes do not come through consistently and appear to be shifted in units relative to the buffer size. I think you were the one making the bus stop analogy on discchord, so I am sure you understand, but just for the sake of clarity...

fessaboy · Post by **fessaboy** » Tue Oct 16, 2012 1:54 am

Still seems like a buffer size issue to me. No matter what, 44.1k samples go from the app to the audio out, every second; they get chunked into 256/512/1024 sample buffers. There's generally three buffers happening at any given time (or at least this is my understanding of the iOS pipeline from the documentation).
Buffer 1 -- input audio being gathered
Buffer 2 -- audio being processed by the app (including processing of audio from a moment earlier)
Buffer 3 -- the audio buffer being played.
Each sample is 0.02267ms long; 256 of them are 5.8ms. 512 is 11.6ms, 1024 is 23.2ms.

Things move through as a pipeline; an app gets a request to "give me the next buffer", and needs to respond before the 5.8ms/11.6ms/23.2ms has expired (and preferably, much sooner). When buffer 3 has been played, buffer 2 moves up, and whatever audio was recorded in buffer 1 is passed to the app for processing.

Suppose the app is very efficient, and generates the audio needed instantly upon request. If we're using 256 sample buffers, we generate audio, hand it over to iOS, and it's 5.8ms before that audio starts coming out from the speakers.

During that 5.8ms... Suppose a note comes in, and we're supposed to play it RIGHT NOW. There's nothing that can be done until the next bus comes by -- and how long that is depends how far into our 5.8ms wait. Some notes will come at the beginning of the wait, and will have a big lag. Others come at the end, and have small lag. Any way you slice it, we've got jitter.

One potential solution is to add one buffer's worth of delay to every MIDI note that comes in -- and then start generating audio at the "right time." This would enable having things fixed with a track-based delay compensation. For live instruments, though -- not much that can be done, apart from having small buffers.

This has all been pretty interesting. I'm working on an app that responds to MIDI, and I'll put in a config switch to have it either generate audio as soon as possible (good for live playing), or with fixed delay (letting the DAW compensate for the track).

SingleCell · Post by **SingleCell** » Tue Oct 16, 2012 2:03 am

Well put. I've come to the same conclusion. You either add an extra buffer as look-ahead and deal with precise time-stamped events that will play a bit later OR you use small buffers and have a _bit_ of jitter.

I think the problem exposed here comes from apps with "large" buffers like 1024-2048 because in my experience, small buffers sizes like 256 minimize the issue enough to fool most people. I still haven't gotten that far into my MIDI handler code yet so who knows how it'll go...

Great discussion and investigation guys.

Tim Webb · Post by **Tim Webb** » Tue Oct 16, 2012 7:56 pm

I've published a new article, including more details of my testing at high (240!!!) BPM, and on different MIDI configurations. This includes lots of pictures and audio samples.

MIDI Jitter on iOS 2: The Reckoning

coverup0204 · Post by **coverup0204** » Wed Oct 17, 2012 3:57 pm

New video:

I tried Midibridge on the advice of the developer, who said it would create a MIDI buffer and deliver steadier timing to MIDI controlled synth apps... but I didn't notice any results even with a 250ms MIDI buffer... here are my tests compared with a JP-8080 for reference:

Note that I am looking at this at a finer "resolution" than Tim is, so that might be why differences are more apparent. We may be getting identical results because to be honest, with some apps like Magellan and Sunrizer that are low latency and hence low jitter, the results are not as terrible as in my last video with iPhone Animoog. They're still not up to standard though!

Again, look how the notes drift away for 3 notes and then snap back... it's definitely buffer size related right?

Leslie · Post by **Leslie** » Thu Oct 18, 2012 6:39 am

Interesting reading, but why is it that I DON'T have any latency issues whatsoever..?
Maybe it is because I DON'T have any apps running in the background when playing over MIDI...
Close everything before you run any music MIDI app to begin with, turn OFF Bluetooth, WiFi, Location services and an able "Private Browsing".
restart your iPad and all should be good by now.
Latency? - what latency... - It doesn't exist in my iOS 6 iPad 3 vocabulary

Good luck

coverup0204 · Post by **coverup0204** » Thu Oct 18, 2012 11:03 am

Tim over at discchord suggested that my bad results might be because of the iRig as opposed to his Camera Connection Kit, or something else in my setup, but when I performed the same test he did and A-B'd our waveforms I got identical results, almost down to the sample:

The blue arrows show where the notes "fall back" into place after slowly drifting away in intervals.

Tim says he is satisfied with this. I am not. I think this is a difference in taste and it depends what side you are on. I think that we ought to be able to hold our 2012 iOS synth apps to at least the same standard as a 2002 VST plugin. I know the iPad 3's hardware is not as powerful as a modern laptop or desktop, but jeez louise, just take care of that with an added mode that gives a little extra latency if need be - as long as we can have tight note timing when we want it. I'm sure people had good MIDI VST plugin timing in the early 2000s on Pentium computers - it's not an impossible or unrealistic thing to ask.

Timing is everything, and this is why people are always talking about stuff like how MPC swing is the best - timing directly impacts the feel and guts of the music itself.

To sum up, we have pretty much eliminated every single external possibility except for software implementation now. I want to know if developers can give us better timing that doesn't depend on the audio buffer size to generate notes "in time" - quotation marks because they are not in time, they jitter relative to the audio buffer size.

My suspicion is that Midibridge may indeed iron out inconsistencies in the source MIDI data jitter, but it can't do anything about the way that the synth app manages to mess up even the most consistent MIDI input because of the "live midi buffering jitter" problem described in the SoS article posted above. My guess is that a MIDI buffer needs to be implemented in the synth app itself so that we can get better timing. Magellan might be approaching acceptable timing, but trying to sequence PPG Wavegenerator, for example, still sucks because its audio buffer latency is higher, which again is not a problem on its own except for that it causes note jitter.

Leslie wrote: Latency? - what latency... - It doesn't exist in my iOS 6 iPad 3 vocabulary
Good luck

Leslie, thanks for the tip, but it doesn't make a difference for me on my iOS 6 iPad 3. I think that you are confused about the difference between latency and jitter. You absolutely *do* have latency, however small. If you're not aware of it, you're a happy camper and you can go on with your music creation! I'm jealous that it doesn't bother you. By the way, post results (either audio or waveform JPEG) of your synth apps playing 16th notes at 120-160 BPM with no jitter and I'll believe you

Thanks for trying to help.

sonicflux · Post by **sonicflux** » Thu Oct 18, 2012 1:30 pm

This is all good stuff. Personally, it doesn't bother me at the moment, but I feel the standard should be much closer to what we would expect from pc/osx softsynths.

Coverup referenced a SoS article that he said had been posted. Maybe it's in the thread, but I couldn't find it, so I googled it in case others had trouble finding it, too. Some good info in there.

iOS MIDI input timing / jitter