Good approach to implementing modulation matrix?

[Kraku]

I'm implementing a modulation matrix for MIDI CC signal based changes and was wondering what might be the best approach to routing those signals from point A to point B.

One way to do it would be to have each destination parameter their own extra variable which would be used whenever that parameter is used in calculations. Then some outside system would keep those parameters updated as the MIDI CC's come in. This would allow multiple MIDI CCs to affect one parameter as they would simply be added together.

Example: Filter cutoff parameter has an extra variable X, which always gets added to cutoff calculations.

Another way would be to let some outside system handle and update the parameters which come from GUI and go to the synthesis engine. Whenever a GUI knob gets tweaked or MIDI CC comes in, those two are matched, added together and set into the synthesis engine to it's rightful location. This way there would be only a single variable for each parameter in the synthesis engine, but the outside system would handle the "book keeping" of all those parameters and their modulation sources/destinations.

Example: MIDI CC arrives. System checks where to route the signal: destination should be filter cutoff. The system checks the filter cutoff value (as defined by GUI knob), adds the MIDI CC amount to it and sets the value into the synthesis engine.

The first method might be simpler to implement but probably a bit slower, especially if the number of modulatable parameters gets higher.

The second method should be more efficient, but is more complex to implement.

Are there other useful ways to do this?

[Urs]

We work with a limited number of slots in the matrix, in order to keep bookkeeping manageable.

Our modules (filters, envelopes...) process in blocks of 16/32/48/64 samples. Before a module is processed, we check if one or more MM Slots are assigned to any of its parameters (that's determined beforehand, and the module has a vector of MM Slots). If so, we create a copy of const parameters on the stack, evaluate the param changes for each matrix slot on the copy of params, then process the module by passing it the copy of the parameters. Module's process funcs linearly fade parameters across the sample block (their states have params history if needed).

Advantage: Works polyphonic even, and we use this to do PolyMod in CLAP as well as internal ModMatrix (Hive etc)
Disadvantage: Limited number, say, 64 or so. Does not allow for resolution beyond sample block granularity (but this can be achieved)

Another advantage: Forces parameters to be immutable/const during processing, which really really helps mitigate common errors (multithreading, state consistency...)

[JustinJ]

Here's how I go about it. The MM is represented as an array of route objects containing source/destination enum indexes and some other parameters. Source enums refer to LFOs, ENVs, MSEGs and other source controllers. Destination enums refer to synth controls (Amp, Filter Cutoff etc).

For each active voice, I iterate through the MM referenced Sources calculating their current value. Then, I iterate through the MM routes and calculate destination values by accumulating source values. Finally, I iterate through the destinations putting the destination values into buffers per voice.

The end result of this process is that for each synth control, I have, per voice, a control block sized buffer of modulation values. All of it is calculated before I actually hit any synth module processing.

This means that the synth modules have no special processing for modulating values. In fact they don't know about the MM at all. For example, in the filter module I'll just read the modulation values for the cutoff filter (for a given voice) from the precomputed destination buffer and process accordingly. Every control value for the synth modules is accessed the same way.

Doing it this way means that the MM and the synth modules have good separation of concerns. The MM can be independently profiled easily enough. Currently I'm processing modulation values at audio rate but it's trivial to reduce to lower resolutions of processing (every 8, 16, 32, samples etc) and interpolate.

[Kraku]

Our modules (filters, envelopes...) process in blocks of 16/32/48/64 samples.
....
Module's process funcs linearly fade parameters across the sample block (their states have params history if needed).

OK, so you're giving each block a parameter value which it should be at the end of that block and then that value gets interpolated over the block that's being processed? I.e. linear interpolation over a single block for each parameter in the MM?

Does this also mean that the granularity of all events coming to your synthesis engine is always that 16/32/48/64 samples? So notes cannot start playing anywhere in between those samples, but always start at the beginning of such a block?

[Kraku]

Here's how I go about it. The MM is represented as an array of route objects containing source/destination enum indexes and some other parameters. Source enums refer to LFOs, ENVs, MSEGs and other source controllers. Destination enums refer to synth controls (Amp, Filter Cutoff etc).

For each active voice, I iterate through the MM referenced Sources calculating their current value. Then, I iterate through the MM routes and calculate destination values by accumulating source values. Finally, I iterate through the destinations putting the destination values into buffers per voice.

The end result of this process is that for each synth control, I have, per voice, a control block sized buffer of modulation values. All of it is calculated before I actually hit any synth module processing.

This means that the synth modules have no special processing for modulating values. In fact they don't know about the MM at all. For example, in the filter module I'll just read the modulation values for the cutoff filter (for a given voice) from the precomputed destination buffer and process accordingly. Every control value for the synth modules is accessed the same way.

Doing it this way means that the MM and the synth modules have good separation of concerns. The MM can be independently profiled easily enough. Currently I'm processing modulation values at audio rate but it's trivial to reduce to lower resolutions of processing (every 8, 16, 32, samples etc) and interpolate.

There are some clear similarities how I currently do things in my own system.
I also give my modules buffers containing audio rate modulation signals. The module itself doesn't know anything about the modulator. That's handled outside each module. (filter cutoff, oscillator pitch, etc.)

I'm just worried about the amount of memory that needs to be processed/copied/etc. if I do all the modulation that way. I might go with a hybrid solution where some parameters/modulation use audio rate buffers while other modulations are set at specific samples and then the module itself fades on every sample that parameter towards its new value. Dunno if that's a good idea or not yet.

[Kraku]

One approach I was toying with in my mind was to always give the module's parameters two buffers to go through:
1. The regular audio rate envelope+lfo each of them has by default.
2. Modulation matrix generated buffer.

To make the amount of processed memory as small as possible, I would now have two options:

1. Have an empty buffer which you can use as MM generated buffer when the parameter you're modulating isn't a MM destination. This means that the exact same empty buffer will be used most of the time for most of the modulatable parameters, which keeps the empty buffer data in cache. This in turn makes reading that buffer quick and efficient.

2. Give each MM destination the buffer AND the stride between each data entry in that buffer. I.e. if the buffer has 32 bit float in it, the stride (aka increment of the read pointer) will be 4 bytes. This can be used to give each module's parameter an empty buffer, which has only one 0.0 floating point value in it and pointer increment 0 for that pointer. Now for each sample the module processes, it will always increment the read pointer with 0 and read the exact same value over and over again. This way you can simply set the first value to whatever you want and the module uses that single data entry without needing to generate lots of data or trashing the cache. The downside to this approach is that you cannot really control if you want to smooth out the modulation signal: the module that uses the buffer knows nothing about those things, since it simply reads that buffer and increments the pointer with the stride you provided it with.

So while the latter idea seems nice, the downside might cause audible artefacts if the modulated values suddenly jump from one to the next in one sample.

[Urs]

Our modules (filters, envelopes...) process in blocks of 16/32/48/64 samples.
....
Module's process funcs linearly fade parameters across the sample block (their states have params history if needed).

OK, so you're giving each block a parameter value which it should be at the end of that block and then that value gets interpolated over the block that's being processed? I.e. linear interpolation over a single block for each parameter in the MM?

Does this also mean that the granularity of all events coming to your synthesis engine is always that 16/32/48/64 samples? So notes cannot start playing anywhere in between those samples, but always start at the beginning of such a block?

Yes. Note granularity is 16 in most of our plug-ins (+/-8 samples), but we have some with a granularity of 4. Latter is our internal Control Rate. Each modulation that has a fixed routing (i.e. not ModMatrix) exists in buffers evaluated at samplerate/4.

We could delay the outputs of each voice for sample accuracy, but considering that this is a lot better than MIDI originally was, and it has served me/us for 20 years, I think it's a valid approach.

That said, we're considering to switch to a granularity of 4 samples for all plug-ins - processing in blocks from 4-64 samples -, but that may be a source of unexpected issues.

[Urs]

Also of course, our modules know nothing about ModMatrix whatsoever. For them there is no difference between a parameter changed through UI, automation, preset change or ModMatrix. The linear interpolation is also used to dezipper knob movements.

[Urs]

I had thought about storing parameters as pointers to a float value and a stride of samples until the pointer is to be incremented by 1.

But then I think there are really just those few parameters where audio rate modulation makes sense, like mostly any pitch and any gain, but for instance rate parameters (derivatives of pitch or gain) don't necessarily.

So I'm thinking about giving parameters an option to live in arrays per 1/4/8/16 samples, and let the ModMatrix then write those values if appropriate.

[JustinJ]

2. Give each MM destination the buffer AND the stride between each data entry in that buffer. I.e. if the buffer has 32 bit float in it, the stride (aka increment of the read pointer) will be 4 bytes. This can be used to give each module's parameter an empty buffer, which has only one 0.0 floating point value in it and pointer increment 0 for that pointer. Now for each sample the module processes, it will always increment the read pointer with 0 and read the exact same value over and over again. This way you can simply set the first value to whatever you want and the module uses that single data entry without needing to generate lots of data or trashing the cache. The downside to this approach is that you cannot really control if you want to smooth out the modulation signal: the module that uses the buffer knows nothing about those things, since it simply reads that buffer and increments the pointer with the stride you provided it with.

That's quite similar to what I'm doing. When something isn't modulated it's just pointing to a buffer that holds the non-modulating control value.

Something that's worth mentioning is that my ModMatrix works closely with the control parameter system which handles parameter smoothing. Meaning, the values that are written to my modulated control value buffers (per voice) are the smoothed parameter value + the modulation.

This means that the synth modules don't care about smoothing values to reduce zipper noise either. They just see a stream of floats.

So, with the stride optimisation, control values that aren't smoothing and aren't modulating just have one value and a stride of 0 meaning the same value just gets read over and over. For parameters that are smoothing or are modulating, you get a block size stream of values to iterate through.

[Jeff McClintock]

That's quite similar to what I'm doing. When something isn't modulated it's just pointing to a buffer that holds the non-modulating control value.

me too

... with the stride optimisation, control values that aren't smoothing and aren't modulating just have one value and a stride of 0 meaning the same value just gets read over and over.

I have a slight variation, when something isn't modulated I set a 'silent' flag on the buffer.

This supports two kinds of processing:
A: 'Dumb' processors that always read the entire buffer.
B: 'Smart' processors that read only the first value in any 'silent' buffer (and assume that the value never changes)

Dumb processors have the advantage of being easy to write and debug. There's no 'special case' for blank buffers. This is compatible with most 'off the shelf' DSP code, so it's a fast way to prototype stuff.

Smart processors have the advantage of extreme efficiency because instead of processing an entire buffer, you can sometimes get away with processing just a single sample. At the cost of having to go back and write say two branches of the processing code (one modulated, the other not).

I like silence flags because they allow a processor to be super-smooth and sample-accurate when it matters, yet as efficient as block-accurate when it doesn't matter.

[mystran]

I'd use "constant" flags rather than "silent" flags or dynamic strides. Basically always allocate a full buffer, but then if the contents are constant, you just write it as the first value in the buffer, set the flag and leave the rest of it undefined.

Modulation mixing between signals flagged constant can maintain constant flags and modulation mixing involving mixed signals (constant vs. non-constant) can dispatch optimized code paths, yet because we always have the full buffers around (with undefined contents) at any point you can expand a constant signal into a proper signal simply by filling the rest of the buffer and clearing the flag.

Suppose filter cutoff knob is not being moved, so we initialized filter cutoff destination as constant. The pitchbend is not moving, so "note CV" is constant. Keytracking knob is not moving, so cutoff+=keytrack*note is also constant. Envelope amount knob is not moving, but envelope itself is not constant, so at this point the cutoff+=envAmount*env operations expands cutoff into a proper signal. If the user touches pitchbend, then we'll just expand the signal earlier. You probably don't want to bother trying to sort optimally at runtime, rather just pick a reasonable order where "probably more likely to be constant" stuff is done first.

When DSP modules are presented the constant flags they have two options: either dispatch a specialized code path (eg. generate by template expansions and it's mostly a matter of how much binary bloat you're willing to accept) or they can opt to flatten the buffers themselves. If you opt to flatten, then do it as late as possible so it's all in cache when the module runs and it's not even that expensive; it's usually all the modulation processing itself having to do everything over buffers that's more of a performance hog.

ps. As long as your buffers are guaranteed to always be at least 2 samples long (this can be arranged for) then we can even do away with the separate flags! The idea is that you pick a NaN value with a non-zero payload, so it's a value that the FPU never generates, then rather than storing the constant value as the first in buffer, you store that special NaN value as first and the constant payload as second. You don't actually ever load the NaN into an FPU register, you load it as an integer and branch on the special value. This scheme has the cool property (besides making sure you never burn an extra cache line on the flags) that if you ever accidentally forget to check for constant buffers, you're guaranteed to turn your floating point computations into NaNs rather than process garbage, so it's easy to catch them when testing.

[JustinJ]

I'd use "constant" flags rather than "silent" flags or dynamic strides. Basically always allocate a full buffer, but then if the contents are constant, you just write it as the first value in the buffer, set the flag and leave the rest of it undefined.

So what I do here is keep the old values for all synth control values. At the beginning of block processing in the ModMatrix I compare the current control values with the old control values. If there's a change then I set about filling the buffer for that control value with smoothing values going towards the target value. When it gets there, there's some logic to make sure that the buffer contains only the new target value on subsequent iterations and it doesn't need to be updated again unless the control value changes.

On the ModMatrix side of things, all destination buffers are pointing to a single buffer full of zeros unless they've got a source that's going to create modulated values, in which case they use their own destination buffer that subsequently gets filled with modulation values.

The final step combines the control values with the modulated values.

I like your idea of a constant flag. That'd mean that when control values aren't changed there's just one value to write rather than a buffer to fill. The ModMatrix and synth module need to take that into account. But thinking about it, on the scheme I have it'd only save filling the buffer post smoothing changes which shouldn't happen that often.

I haven't implemented a stride value but thinking about it, for non-mutating value buffers the stride could be 0 rather than 1. I'm also thinking that the stride could be useful for different modulation control rates. So advancing the stride by (bufferIndex >> 8 ) effectively advances every 16 frames. With the correct handling on modulation value generation and maybe interpolating between modulation values in the module, it'd be reasonably trivial to support different control update rates for different control parameters.

[Urs]

I couldn't abandon linear parameter interpolation within the module. Some of our plug-ins have hundreds of parameters that need to be smoothed. If I allocated block sized buffers for each parameter, I'd destroy any cache advantage I might have had through whatever memory layout.

Also, because we have multithreaded processing (on voice level) in some, modules need to be able to work on copies of parameter structures.

Instead of a "constant flag" I'd have a pointer to a float array. If that pointer is nullptr, the normal float value is valid and the parameter just linearly interpolates across the buffer (constant or knob turning). If there's modulation, the pointer isn't nullptr but points to the array of resulting parameter values. Then it's modulated and linear interpolation is on whatever stride level.

This way the buffer can be a reusable buffer passed from the ModMatrix itself, which only needs one buffer for each voice per slot, and possibly fewer if pooled economically (e.g. if max number of eligible parameters any such module has is lower than the number of ModMatrix slots).

(Ok this might be a hack, but I have done way worse)

[mystran]

I haven't implemented a stride value but thinking about it, for non-mutating value buffers the stride could be 0 rather than 1.

The problem I have with a stride approach is that it complicates things unnecessarily. Now you need all the extra logic for dealing with the strides. The key thing is to realize that this extra logic is not free in terms of performance either, especially if you try to deal with all of it in a general code path.. but specializing for all kinds of different strides is a huge increase in code-size that you really don't want.

The thing about constant vs. non-constant is that in many cases it's actually reasonable to specialize code. For example if a filter takes cutoff, resonance and some general "morph" parameter, then we'd have 2^3=8 cases of constant vs. non-constant inputs.

template <bool constantCutoff, bool constantRes, bool constantMorph> processFilter(...)

Then you can predicate things on the flags so that your filter statically skips loading new parameter values inside the per-sample loop if it's running with a constant input... and then if you had something like a trapezoidal solver where some of the system factorization stays the same when parameters don't change, the compiler can potentially hoist these out of the loop too.

There's other ways to write such templates (eg. you could template on the parameter types instead; kinda cleaner, but a tiny bit more boilerplate), but the idea here is that this way you can truly match the performance of having your DSP modules take a proper constant... including the performance gains you get from allowing the compiler to optimize based on the fact that the input is constant... but since expanding a constant into a full buffer is also trivial, we can do this selectively only where it matters (perhaps only generate a fast-path where all parameters or a certain set of parameters is constant) and if we have something like a hugely complicated oscillator that takes 9 parameters we don't necessarily need to generate 2^9=512 different variations (and the boilerplate to dispatch those).