What is KVR Audio? | Submit News | Advertise | Developer Account

Options (Affects News & Product results only):

OS:
Format:
Include:
Quick Search KVR

"Quick Search" KVR Audio's Product Database, News Items, Developer Listings, Forum Topics and videos here. For advanced Product Database searching please use the full product search. For the forum you can use the phpBB forum search.

To utilize the power of Google you can use the integrated Google Site Search.

Products 0

Developers 0

News 0

Forum 0

Videos 0

Search  

Modular Audio Engine Design

DSP, Plug-in and Host development discussion.

Moderator: Moderators (Main)

KVRer
 
24 posts since 27 Dec, 2009, from Berlin
 

Postby tonfilm; Fri Oct 04, 2013 7:49 pm

Hello dear audio devs,

i have quite a task in front of me. we will develop (with the help of other coders) an audio engine with modular audio processing modules which can be connected in a signal flow graph. similar to max/msp, pure data, bidule, audiomulch, CLAM and so on. but more focused on having VST plugins as nodes in the signal flow graph as well as low level nodes like multiply, add, routing and even code templates which compile in realtime. so some kind of modular DAW without the need of metaphors like channel, send, return and stuff like that... total free routing.

the good part is, that the graph building framework is already there, as well as a GUI mainloop and everything else... only thing i have to code is the audio engine (with a time clock), the audio signal data type, midi/time events and so on...

specially people who have experience with audio engines and their architecture would be very helpful at this stage. the first steps are not about audio algorithms, but about stuff like:

    - what kind of calculations should be done in the audio interrupt and what can be done elsewhere, what kind of scheduling is commonly used?
    - how are time, bpm and note events handled and triggered in an optimal way?
    - how is data transfered from and to the GUI mainloop with least possible latency?
    - how to handle plugin/algorithm latency
    - how to minimize latency overall
    - how do GUI elements handle latency? how is for example a cursor which runs over a soundfile synchronized with the audio that you actually hear right now.
    - ...


if you know good resources to read or open source projects which are well written, or have any other idea about the topic, just throw it at me. the only thing that's important is, that it should be state of the art 2013 code patterns for audio.

as audio library i am using NAudio which is C#/.NET style and i used the ISampleProvider interface as a start.

so far the signal format is 32-bit float -1..1 and the base class of an audio processor is something like this:

Code: Select all
      class AudioSignal   

                ...

      protected AudioSignal FUpstreamProvider; //the input node
      protected object FUpstreamLock = new object();
      protected float[] FReadBuffer = new float[1];

      private bool FNeedsRead = true; //will be set to true after each graph evaluation

       public int Read(float[] buffer, int offset, int count)
       {
          FReadBuffer = BufferHelpers.Ensure(FReadBuffer, count);
          if(FNeedsRead) //only first call per frame
          {
             this.CalcBuffer(FReadBuffer, offset, count);

             FNeedsRead = false;
          }


           for (int i = 0; i < count; i++) //each call
           {
               buffer[offset + i] = FReadBuffer[i];
           }

           return count;
       }

      protected virtual void CalcBuffer(float[] buffer, int offset, int count) //subclasses override this method
      {
         lock(FUpstreamLock)
         {
            if(FUpstreamProvider != null)
               FUpstreamProvider.Read(buffer, offset, count);
         }
      }


and a sine generator then looks like:

Code: Select all
public class SineSignal : AudioSignal
   {
      public SineSignal(double frequency)
         : base(44100)
      {
         Frequency = frequency;
      }

      private double Frequency;
      public double Gain = 0.1;
      private double TwoPi = Math.PI * 2;
      private int nSample = 0;

      protected override void CalcBuffer(float[] buffer, int offset, int count)
      {

         var sampleRate = this.WaveFormat.SampleRate;
         var multiple = TwoPi*Frequency/sampleRate;
         for (int i = 0; i < count; i++)
         {
            // Sinus Generator

            buffer[i] = (float)(Gain*Math.Sin(nSample*multiple));

            unchecked //;-)
            {
               nSample++;
            }
         }

      }

   }


first tests look ok... but now, before code complexity explodes, i would like to hear the opinions of experienced audio developers/architects.

best,
tf
Last edited by tonfilm on Sat Oct 05, 2013 1:19 pm, edited 1 time in total.
KVRian
 
504 posts since 18 Jul, 2007, from Netherlands

Postby obiwanjacobi; Fri Oct 04, 2013 10:52 pm

First, have you looked at VST.NET? It supports loading unmanaged (C++) plugins from a managed (.NET) host.

Some time ago I have tried to create a generic sound engine. I used the "pipes and filters" pattern as a basis. The idea was to have multiple threads process each (parallel) path in order to make use of the multi-core processors of today - one of the easiest ways to tap into the new hardware's capabilities. I aborted the experiment because scheduling of the .NET Task objects was not compatible with realtime audio processing (more for generic background worker stuff) and I wasn't up for writing my own Task Scheduler.

Another goal was to optimize memory usage and allocation for processing the entire chain/tree.

Do not forget MIDI! It also has to be woven into the Audio Engine and be processed by Midi and/or instrument plugins.

I see two options for syncing UI.
1) First (UI-)timer based to poll the audio engine from the UI (thread) for state. Polling is often wasteful so this wouldn't be my first choice - although coding would probably be simpler.
2) Second option is letting the audio engine post change notifications in a queue where listeners can register on (Publish/Subscriber or Event Aggregation pattern). That would decouple the UI and Audio engine a lot more and only the queue would be a common resource. If you can make it non-locking you're on the right track (not easy!).

In general I don't think you have to worry about latency in the UI. It should be fast enough on a reasonable machine. Do be ware of WPF though (you've not said what UI framework you are going to use) - it is known to gobble up a fair amount of resources - CPU probably most unwanted in this case.

As a final tip I would make your AudioSignal class Stream based because the methods you've got are exactly the same. I would go for the Stream-Reader/Writer pattern similar to what you can fin the .NET Framework for Files (and other).

Hope it helps.
Grtx, Marc Jacobi.
Blog | VST.NET | GMPI.NET | MIDI.NET
KVRer
 
24 posts since 27 Dec, 2009, from Berlin
 

Postby tonfilm; Sat Oct 05, 2013 4:31 am

hello obiwanjacobi,

yes, of course i wanted to run VST plugins with VST.NET :).
and thanks for pointing me to PFE, i saw it some time ago and wanted to dig into its code. it could be a good starting point. what is the current state of the project, is it working somehow?
threading has a bit improved in .NET 4.5 and maybe there are classes which are suitable for the pipes and filters approach. could you elaborate a bit about the problems with the .NET task scheduler and real time audio? do you hang out on IRC somewhere? maybe we could have a chat.

concerning midi, i was thinking to use the midi classes in the VST.NET library. but i see you als did a MIDI.NET library. then its probably going to be that one... :) but i am not entirely sure about the midi driver access, since we have that in our system multiple times already. but as i want the midi not to be processed in the GUI thread, i have to write it once again i fear.

the UI we have is written in Delphi and is very minimal and about 10 years old, you can have a look at it here: http://vvvv.org/screenshots
its a multimedia system for creative coding and large scale media applications. people were always complaining about the limited audio features compared to max or pure data, but i want to change that now.

so i will let the audio engine post changes in a queue and let the UI thread get it as needed with a short lock, sounds good for a start.

the .NET streams are always based on bytes, would you write one based on float or is your suggestion to inherit from Stream and extend it to float?

thanks for your input, was very helpful already.
KVRian
 
504 posts since 18 Jul, 2007, from Netherlands

Postby obiwanjacobi; Sat Oct 05, 2013 5:27 am

tonfilm wrote: and thanks for pointing me to PFE, i saw it some time ago and wanted to dig into its code. it could be a good starting point. what is the current state of the project, is it working somehow? threading has a bit improved in .NET 4.5 and maybe there are classes which are suitable for the pipes and filters approach. could you elaborate a bit about the problems with the .NET task scheduler and real time audio?


The code in the test project that is on codeplex demonstrates how it was intended. It works but the order of how the filters each get their slice is not compatible with how I had envisioned it for realtime audio. I have not looked at it again with .NET 4.5.

tonfilm wrote:do you hang out on IRC somewhere? maybe we could have a chat.
Skype:obiwanjacobi (amsterdam, netherlands time zone)

tonfilm wrote: concerning midi, i was thinking to use the midi classes in the VST.NET library. but i see you also did a MIDI.NET library. then its probably going to be that one... :) but i am not entirely sure about the midi driver access, since we have that in our system multiple times already. but as i want the midi not to be processed in the GUI thread, i have to write it once again i fear.

The Midi messages from MIDI.NET are usable on their own and do not depend on the Midi Ports. Note that VST (.NET) also defines event structures for communicating MIDI events to (or from) a plugin. In general MIDI ports (interacting via the WIN API with the driver) are system global so you should find a way to share that resource between threads if needed.

tonfilm wrote: the UI we have is written in Delphi and is very minimal and about 10 years old, you can have a look at it here: http://vvvv.org/screenshots
its a multimedia system for creative coding and large scale media applications. people were always complaining about the limited audio features compared to max or pure data, but i want to change that now.

Ah, the teapot! ;)

tonfilm wrote: the .NET streams are always based on bytes, would you write one based on float or is your suggestion to inherit from Stream and extend it to float?
Or have an AudioSampleReader and an AudioSampleWriter class that work with floats? Not sure if its worth the hassle. If it's just you, programming this then no. If this code is exposed to the (scripting?) user, then perhaps yes. Depends on what patterns you want to expose the user to.

tonfilm wrote:thanks for your input, was very helpful already.

You're welcome.
Grtx, Marc Jacobi.
Blog | VST.NET | GMPI.NET | MIDI.NET
KVRer
 
24 posts since 27 Dec, 2009, from Berlin
 

Postby tonfilm; Fri Oct 18, 2013 1:36 am

This is going pretty well so far... NAudio + VST.NET + vvvv's spreading technology make a powerful set of options for large scale audio projects:

Image

And routing with jack from/to other applications makes it even more fun:

Image

Next step, midi integration. Going to dig into MIDI.NET which looks pretty sophisticated for a 0.1 release...
KVRist
 
264 posts since 21 Jun, 2013

Postby 2DaT; Fri Oct 18, 2013 5:17 am

Well c# is not good solution for audio work. It doesnt perform well on low level operations like copying large blocks of floats, it doesnt optimise code like c++ compilers, it cant use SSE and much more :)

Maybe i am not a professional in managed code optimisation, but those little marshalling overheads became huge when functions are called like 1000 times per second ( eg. getParameter )

When i rewrote my plugin from VST.NET to c++, performance increace was about 250%.

However ,it gave me a good oppotunity to start :P
KVRer
 
24 posts since 27 Dec, 2009, from Berlin
 

Postby tonfilm; Fri Oct 18, 2013 10:36 am

hello 2DaT,

thank you for your thoughts. yes, i am aware of the performance drawbacks of C#. but i see it from a different angle. the audio engine that i want to design is going to do more the high level stuff like routing, mixing, talking to drivers and handling files. this is the perfect domain for C# as it is a very productive language and you usually have way less bugs compared to writing it in a unmanaged language.

for the buffer handling Array.Copy is almost as fast as memcpy as its only a wrapper. and for gain and basic buffer operations i just found this SSE/SIMD/AVX library which looks just awesome: http://www.yeppp.info/
going to test this next...

the engine is not focused on low level DSP algorithm design, this should of course be done in C/C++ and loaded as a plugin or as a .dll with .NET interop.

however it is well suited for rapid prototyping which is a main focus of vvvv. so you would be able to write DSP algorithms in our integrated code editor and (re-)compile it in real time while your audio is running.
KVRer
 
24 posts since 27 Dec, 2009, from Berlin
 

Postby tonfilm; Fri Oct 18, 2013 10:40 am

and i really want to be able to post links... ;-)

Moderator: Moderators (Main)

Return to DSP and Plug-in Development