CLAP: The New Audio Plug-in Standard (by U-he, Bitwig and others)

[Jeff McClintock]

OK, you're veering off on a tangent. I'm gonna type this slowly. Please read it slowly.

CLAP has its own API for playing notes. 'CLAP_NOTE_DIALECT_CLAP'

'clap_event_note' includes a 'note_id' a 'key' and a ''channel'.

Code: Select all

typedef struct clap_event_note {
   clap_event_header_t header;

   int32_t note_id; // -1 if unspecified, otherwise >=0
   int16_t port_index;
   int16_t channel;  // 0..15
   int16_t key;      // 0..127
   double  velocity; // 0..1
} clap_event_note_t;

The debate we're having is: Is 'note_id' redundant? (An overcomplication).

The reason I bought MPE and MIDI 2.0 into the conversation is to provide evidence that a fully-featured system can and does exist without any need for a 'note_id'. That you can achieve all the benefits of CLAP without 'note_id'.

Solo asked a very reasonable question:

Is there a reason why PCK can’t be a note ID by itself in the standard. I mean, Is there a situation PCK is not unique enough, ie, two different playing notes are having the same port, same channel and same key? If not, then why have note ID in the first place.

This is the conversation we are having. I am not saying "throw out CLAP_NOTE_DIALECT_CLAP and use MIDI instead".

It's a very simple question. I'm not hearing the answer.

[robbert-vdh]

The debate we're having is: Is 'note_id' redundant? (An overcomplication).

The reason I bought MPE and MIDI 2.0 into the conversation is to provide evidence that a fully-featured system can and does exist without any need for a 'note_id'. That you can achieve all the benefits of CLAP without 'note_id'.

No, it is not. I still don't fully understand what you're trying to get at here, but I assume your cause for thinking note IDs might be redundant is because a CLAP_EVENT_NOTE_ON for key n is the same as a CLAP_EVENT_NOTE_ON for key (n - 1) followed by a CLAP_NOTE_EXPRESSION_TUNING event for note key (n - 1) with a tuning of +1 semitones, right? After all, the reason note IDs exist is to allow multiple voices (it should have really been called voice ID...) to map to a single port, channel, and key combination. And yes, you can use note on+tuning to have up to 128 notes/voices spread over the keyboard mapped to the same same pitch. But that only works on 12-TET. And forcing a convention where every CLAP host needs to send offset note on event plus tuning messages and every CLAP plugin needs to be able to handle those is a hack, just like MPE is. If you just add a note ID field to those events you also get overlapping voices on a single port, channel, key combination, but in a way that doesn't mandate tuning expressions/pitch bend events and that plugins can just ignore if they don't need to care about note IDs.

Is there a reason why PCK can’t be a note ID by itself in the standard. I mean, Is there a situation PCK is not unique enough, ie, two different playing notes are having the same port, same channel and same key? If not, then why have note ID in the first place.

Yes, because notes are not voices. You can't individually address two voices with the same PCK without assigning an ID to that voice. At least, without involving hacks like the tuning thing. Which again doesn't work with microtuning inside of a plugin.

[Jeff McClintock]

If you just add a note ID field to those events you also get overlapping voices on a single port, channel, key combination, but in a way that doesn't mandate tuning expressions/pitch bend events and that plugins can just ignore if they don't need to care about note IDs.

Thanks, that's a clear answer.

If I understood...
'note_id' allows existing plugins that don't support per-note-tuning to infer the pitch of a note from its key. i.e. the synth can safely assume that key # 69 is always Middle-A (440Hz).

So this is to cater to plugin writers who support only basic MIDI 1.0 kind of functionality (no mico-tuning). Which is a large group of people to be fair.

But I kind of expect, that if you are doing a quick and dirty port of a MIDI 1.0 plugin to CLAP, you would choose to use CLAPs support for plain old MIDI 1.0 ('CLAP_EVENT_MIDI') not the new-fangled method?

I appreciate your time helping me get to the bottom of this BTW. I'm pretty argumentative.

[robbert-vdh]

Thanks, that's a clear answer.

If I understood...
'note_id' allows existing plugins that don't support per-note-tuning to infer the pitch of a note from its key. i.e. the synth can safely assume that key # 69 is always Middle-A (440Hz).

So this is to cater to plugin writers who support only basic MIDI 1.0 kind of functionality (no mico-tuning). Which is a large group of people to be fair.

But I kind of expect, that if you are doing a quick and dirty port of a MIDI 1.0 plugin to CLAP, you would choose to use CLAPs support for plain old MIDI 1.0 ('CLAP_EVENT_MIDI') not the new-fangled method?

I appreciate your time helping me get to the bottom of this BTW. I'm pretty argumentative.

You can just keep using MIDI 1.0 if you were already using that, yes. It's not a hard requirement, but I'll assume most CLAP hosts will support MIDI (and if they don't, their users will surely ask for it). But because CLAP's own note events are semantically a super set of basic MIDI 1.0 functionality, you can choose to support those for your note events instead with only very minor changes. As a benefit, you'll then get more granular velocity/poly pressure, tuning events in +/- 120 semitone ranges instead of untyped pitch bends, and a couple per-voice/note expressions for volume, panning, vibrato, expression, and brightness. And of course, you'll have the option to support per-note ID modulation later if you so desire.

[baconpaul]

Two voices with the same key-number? Why do you need that? What essential user-facing feature does it support?

It supports, today, a user who has

1: a keyboard
2: a voice with a long release envelope and
3: presses the same key twice in a row while
4: wanting independent modulators on the release

While MIDI 2 does decouple the key from the pitch, and while CLAP will present that MIDI 2 message if a DAW sends it and a plugin receives it, with current systems based in MIDI 1 land, the problem solved is the one above.

[baconpaul]

'note_id' allows existing plugins that don't support per-note-tuning to infer the pitch of a note from its key. i.e. the synth can safely assume that key # 69 is always Middle-A (440Hz).

I would say 'note_id' allows plugins that assume a key press event maps to a consistent pitch but not necessarily a unique voice to share the knowledge of both voices with the host so the host can target modulation messages at a voice independent of the key with which it was launched

If I had a device which, when I pressed middle C 4 times in a row, sent 4 different note numbers each with an appropriate pitch message, then that device would have implemented something completely semantically equivalent to what a daw does when it takes the MIDI 1 stream from that keyboard device and augments it with note ids.

[baconpaul]

Is there a reason why PCK can’t be a note ID by itself in the standard. I mean, Is there a situation PCK is not unique enough, ie, two different playing notes are having the same port, same channel and same key? If not, then why have note ID in the first place.

It's a very simple question. I'm not hearing the answer.

With every single performance device I own except my linnstrument and roli, if I press the same key twice in a row, i get the same PCK. With my linnstrument and roli I get a re-use on my 16th key press. With note_id I get a re-use on my 2^32 key press.

2^32 > 16 > 1

Does that help?

[mystran]

"We don't like 'note_id' because we didn't think about it first!"

For what it's worth, I think the design is great.

ps. I know how to implement a hash-table.

[baconpaul]

"We don't like 'note_id' because we didn't think about it first!"

For what it's worth, I think the design is great.

I'm glad you like the design (I do too!) but I think that's not quite what Jeff is saying. I think he's pointing out (I think correctly) that if we had MIDI 2 devices everywhere (or perhaps if we used some basically unimplemented MIDI 1 Sysex extensions) we could have devices generate streams of data which uniquified voices at least during the depressed key phase. I think that is true. I don't own any devices which do that though. And I may be mis-understanding his point. He has a lot (a lot) of experience with this though so I feel I am well served to listen.

Beyond the keypress I don't see a way to know the voice lifetime without a collaboration with the engine, though. The 'voice termination' event which the plugin shares with the host is a very important part of polyphony also. I don't know if MIDI 2 has this concept (like I literally just don't know; i'm not passively aggressively asserting it doesn't). With this event, plus a coordinate for the voice (whether it is PCK or a physically-disconnected ID) you can do quite a lot today.

[Jeff McClintock]

The 'voice termination' event which the plugin shares with the host is a very important part of polyphony also. I don't know if MIDI 2 has this concept.

They have 'MIDI 2.0 Per-Note Management Message'.

"When a device receives a Per-Note Management message with D = 1 (Detach), all currently playing
notes and previous notes on the referenced Note Number shall no longer respond to any Per-Note
controllers."

"We don't like 'note_id' because we didn't think about it first!"

"Req 80: Voice management must be the domain of the instrument. All that
the host sees is a voice ID. Operations on a voice always use the ID." - GMPI requirements, 15 Aug 2004

[S0lo]

I’m having second thoughts. I think both note_id and voice_id may have musical value.

Note_id will allow the host to be voice agnostic and throws the note to voice assignment job upon the plug-in. Voice_id throws that job upon the host.

Why is this relevant? Think about this, a user may intentionally want to have a maximum polyphony of just 3. So that when he presses the 4th note it will be assigned to voice 1 (overlapping with note 1) and allowing this voice to act monophonically!!, doing legato with say note priority. And so on when he presses the 5th, 6th,,, etc notes (while holding all previous notes).

Allowing the host to control this, gives the user a chance to do what he wants with voice assignment regardless of what the plug-in does. On the other hand, allowing the plug-in to control this gives the developer a chance to do what he wants with some musically interesting assignments.

If we can establish an agreement that the channel in PCK refers to voice number, we would allow host control (edit: same as voice_id). While note_id gives plug-in control. Without such an agreement, the meaning of channel would be left for interpretation by both host and plugin and so there is no guarantee. (Rings a bell? VST...)

[Urs]

2004

While GMPI played no major role in CLAP's design process, it clearly states that both MIDI 1.0 and an abstract mechanism for identification of overlapping identical notes should be present. This requirement is met in CLAP with the concepts of PCK *and* NoteID*.

IMHO the requirement is met with elegance, because the consumption of those concurrent concepts is literally presented in the same structures. That is, while those concepts are concurrent, they are not implemented as "separate concepts to do the same thing" (as in other formats), they are "one thing you can do, and you can choose if you wish to support it this way or that way or both ways".

I point out the date of GMPI because clearly it predates the developments of nearly 20 years that played a role in CLAP's design process. In the meantime MPE could be seen as fulfilling that requirement, but MPE is too limiting for a plug-in standard that needs to last another 40 years. Also, from what I read, MIDI 2.0 does PCK, such that it centers on the concept of notes (with arbitrary pitch), not the concept of voices.




* (I don't think GMPI uses the term, but it describes the concept, even if it seems somewhat backward where the host "sees" a VoiceID that the plug-in provides, but it doesn't define a practical mechanism to do this, hence we dismissed it - because voice management is still in the plug-in domain. We talked about letting the plug-in assign a voice ID for a given note event, but because those present a chicken and egg challenge, it is not a practical way to implement this. It would be impossible to send a NoteOn and a Parameter Modulation event for the same note in the same event queue if the host first had to wait for a the plug-in to assign a VoiceID to that note. The advantages of the unified event queue in CLAP outweighs this requirement of GMPI, such that Note_ID generation is placed in the host domain while clearly the plug-in maintains control over voice management.)

[S0lo]

Never mind. post removed.

[S0lo]

I've addressed that earlier viewtopic.php?p=8471766#p8471766.

In short, It boils down to, do we want to have audio rate modulation or not.

True audio rate modulation was one of those things we looked at little while back, but was put on the backburner in order to prepare for the CLAP 1.0 release. CLAP's event-based approach to automation and modulation is incredibly convenient, but it doesn't scale well to full audio rate. For most things that's not a problem. You're probably using smoothing internally anyways, and Bitwig gives you one event per 64 samples which is more than granular enough for those use cases. However, if you were to use the same mechanism to send parameter events for every sample, for multiple parameters, and for multiple note IDs at the same time, then the overhead starts to add up real quick. The main problems here are the one you already mentioned, i.e. having to handle every single one of those events individually, and the amount of memory overhead involved in using events for this. The clap_event_param_mod_t struct is 56 bytes large, while the double value you care about for the audio rate modulation is only 8 bytes. That means that assuming you already know what the event is modulating in advance, almost 86% of the struct is unnecessary overhead.

So the idea was to have an extension that lets you tell the host for which parameters you support dense automation/modulation, and the host could then use an event from that extension to provide you with a dense buffer containing the rendered automation of modulation data for that parameter (and note ID combination) for every sample in the buffer. Then you can simply read the offsets from that as part of your processing loop. But this extension does not yet exist.

Ok thanks that does explain allot. I think I need to read it again

[baconpaul]

The 'voice termination' event which the plugin shares with the host is a very important part of polyphony also. I don't know if MIDI 2 has this concept.

They have 'MIDI 2.0 Per-Note Management Message'.

"When a device receives a Per-Note Management message with D = 1 (Detach), all currently playing
notes and previous notes on the referenced Note Number shall no longer respond to any Per-Note
controllers."

Thank you
So it seems to me like
Clap note - uses key to indicate pitch and rotating note id to uniquify voice
Midi 2 - uses rotating key number to indicate voice and pitch property to indicate pitch

So you could (semantically but not physically and loads of caveats etc) take a clap note stream, swap the key and note id, and have the semantic content of a midi 2 stream. (I’m using key here as shorthand for pck)

Namely midi 2 and clap both segregate the sort of musical intent (key in midi1/clap; note pitch in midi2) from the book keeping (note id in clap; key in midi 2)

Of course loads of shorthand in that but I want to make sure I understand the semantics.