Null-Testing (Paid vs. Free Compressors)

[koolkeys]

Second one:

His implementation is also flawed, plugin UI controls are not calibrated to any standard, so simply using the same visual values will not result in identical behaviour. Not really his fault, one could expect 1ms meaning 1ms, but unfortunately it doesn't, and therefore his test does not test the plugins at identical settings.

That is indeed a bold claim and I would like you to provide evidence of that, as I don't see why that would be the case.

Yeah, let me hack into all the compressors I own and I'll let you know what I find......or I could just use common sense knowing that you can indeed code your knobs differently in terms of weighting, travel distance, etc.

But here's the thing. It isn't his job to provide evidence. The person who created this ignorant claim on the Reaper forum needs to provide that. If he is trying to prove a point, he needs to prove it with all factors taken into account. Can he prove that all knobs on all plugins are created equal? Of course not. 1, because they aren't. 2, because he can't possibly have that info for individual plugins.

Look, you can defend until you are blue in the face with trying to believe that all compressors are pretty much the same. They are not. And the "test" at the Reaper forum doesn't even get close to proving what it is trying to prove. You don't have to agree with the mountain of experienced people here telling you why the tests were invalid. But that doesn't change the fact that the tests are invalid. I've seen bmaniac discuss previous tests he has done in this area, and I can tell you that he knows far more about this process than you, I, or the person on the Reaper forum. And his tests were more in depth. I wish his results were published somewhere, but I don't think they are.

Anyways, you're defensive and you don't need to be. Go ahead and use Antress plugins to make your music. Nobody is stopping you. But any claim that free Synthedit plugins are all pretty much the same in function as plugins costing much more is just bogus. The Reaper forums have always had an undertone of "stick it to the man" and "never pay for expensive tools". They are kind of like a budget version of the "don't pay for anything" part of the Linux community. Lots of great people on the Reaper forum who I have a ton of respect for, but the undertone still remains.

You don't have to like it, or agree. And I'm fine with that.

Brent

[.jon]

lollilol, I don't think your definition of "proving" is how people normally define it.

Provide evidence that GUI settings of different plugins from random developers don't necessarily result in same change in the audio Do you not understand that the knobs on different plugins don't control the same algorithms? If they would, they would all null down to infinity ffs.

I'm nt proposing it says anything about subjective quality. I'm explaining why the test setup tells nothing without a pre-existing, undisclosed and unproven ranking of the plugins.

[lolilol1975]

Yeah, let me hack into all the compressors I own and I'll let you know what I find......or I could just use common sense knowing that you can indeed code your knobs differently in terms of weighting, travel distance, etc.

Of course we don't know what the exact algorithm is. But the plugins don't show travel distance, they show milliseconds. And it is VERY easy, - or I'd rather say trivial -, to be exact about the ms: take the sampling frequency and divide.

Hence at 44100 samples/sec, 1 ms = 44 samples. At 96000 samples/sec, 1 ms = 96 samples.
You need to compress after 1 ms ? Skip the next 96 samples. You need to release after 100ms, release start at the 9601th sample. That's basically it and there is absolutely no reason to do it any other way. So why in the world would plugins do things differently here ?

[.jon]

And just for the record, I'm not talking specifically about testing compressors or even plugins, just about some general principles that concern valid testing and research which results in new, useful information. There's a lot of pitfalls in these, and a small oversight in some area can degrade the data or lead to false conclusions. Unfortunately this particular test is one big oversight in every area, and it's cool if you managed to scavenge something valuable to you personally out of the trainwreck.

Defending the test as an awesome compressor shootout is hopeless and you're just wasting time. Infinitely better test is just to listen to the compressors, use them, and pick the one that fits your budget and you like using most on your material.

[koolkeys]

Yeah, let me hack into all the compressors I own and I'll let you know what I find......or I could just use common sense knowing that you can indeed code your knobs differently in terms of weighting, travel distance, etc.

Of course we don't know what the exact algorithm is. But the plugins don't show travel distance, they show milliseconds. And it is VERY easy, - or I'd rather say trivial -, to be exact about the ms: take the sampling frequency and divide.

Hence at 44100 samples/sec, 1 ms = 44 samples. At 96000 samples/sec, 1 ms = 96 samples.
You need to compress after 1 ms ? Skip the next 96 samples. You need to release after 100ms, release start at the 9601th sample. That's basically it and there is absolutely no reason to do it any other way. So why in the world would plugins do things differently here ?

So you're saying every plugin in the test had the same knobs with the same weighting with the same settings across the board? No possibility of difference throughout the plugins? Even if the similar controls were all exactly the same across the board, you can't take into account the un-matched features between plugins.

But in the end, I don't even know why we are discussing this. Even if we DO accept his test as a null-test as being a legitimate test for his purposes, his test is at best incomplete and sterile, and in no way whatsoever conclusive. He is making claims without understanding the nature of his own tests or claims. And you have defended those tests. And you, just like him, have reached incorrect conclusions.

No, I won't prove it. Common sense proves it. Others here have given you evidence. You ignore them. So I won't waste the time trying. I have no problem with what tools you use. But I do have a problem when people try to use incomplete and irrelevant tests to try and prove a narrative that fits a specific audience while ignoring the science and practicality in their own test.

Brent

[FabienTDR]

Yeah, let me hack into all the compressors I own and I'll let you know what I find......or I could just use common sense knowing that you can indeed code your knobs differently in terms of weighting, travel distance, etc.

Of course we don't know what the exact algorithm is. But the plugins don't show travel distance, they show milliseconds. And it is VERY easy, - or I'd rather say trivial -, to be exact about the ms: take the sampling frequency and divide.

Hence at 44100 samples/sec, 1 ms = 44 samples. At 96000 samples/sec, 1 ms = 96 samples.
You need to compress after 1 ms ? Skip the next 96 samples. You need to release after 100ms, release start at the 9601th sample. That's basically it and there is absolutely no reason to do it any other way. So why in the world would plugins do things differently here ?

But that's not how timing is being specified in dynamics. You don't have a counter waiting x samples (read the manual!). The smoothing in dynamics processors (often controllable via the attack and release knobs) is, in 99% of the cases, done via a first order filter. Filters are generally asymptotic in nature, they never reach their target value. That's why we use seemingly weird forms of specification when it comes to filters. You find countless of them:

- Corner frequency specification uses the -3dB point (and of course NOT "where the filter stops". It never does).
- In other approaches, you'll see a -6dB specification (say, in sinc filter specification or Linkwitz Riley specification)

When it comes to control systems, such as dynamics compressors or whatever, we're typically defining timing of a smoothing filter not by frequency, but in a time domain centric manner: Often, the point where the filter drops 10dB, or reaches 30% or 80% or whatever. There are countless ways to do so.

Now to make the situation even worse, not all control systems are dB linear, some of them are linear in the pure sense, which in turn provokes a strong level dependency of the timing. E.g. such a compressor driven deep into GR will act much faster than the same timing setting driven only 2-3 dB into GR. Not to mention automatisms and other dynamic timing tricks.You do you compare that? You really can't that easily.

Timing specification is faaaaaaaaaaaaar more complicated than you are drawing here. It's not clear at all, if you think it is, you're for sure way off the mark.



Regarding null testing, you are crudely overestimating its value. Point is, I can create an unlimited amount of sine waves that are absolutely equal, but don't null (in the peak sense) in PCM form. All I need is to adjust the phase of the sine wave.

Similarly, I can generate an unlimited amount of totally different signals that show exactly the same loudness. All I need is an amplifier and a loudness meter.

Null testing is subject of too many false positives and false negatives. Seriously, physicists don't study for years to dance around something as trivial as A - B. This is more than ridiculous, it's fine humour. Your notion tells me that you still have a lot to learn and discover. It's normal for laymen to brutally underestimate the depth of things. A healthy humbleness (and respect for other ppl's views) is the only way out. If you don't, you'll probably soon mutate into a troll.

[lolilol1975]

Right, since I'm not a troll, I'll gracefully accept these explanations. Indeed I overlooked the filter smoothing that occurs.

[Hink]

[meloco_go]

Great post by Fabien!
I would like to further expand on few things.
First of all, a null test has definitive result in only one case -- when it really shows -inf dB difference meaning identical files. In all other scenarios (when where is quantifiable difference) it becomes a game of "if's".
A difference of -60 dB might mean audibly identical in one case and a broken file in the other (e.g. softly played low-frequency instrument vs same reconding with faint 4k ringing tone over it).
Simple low order all-pass filter (quite inaudible) will prevent two files from nulling by a great margin.
A difference of -20dB may sound closer than -40dB (happens quite often really in real world situations).
Now the bigger question is -- if there is a difference what it is?
For example, in case of EQs one might match frequency responce exactly (like Q-Clone) but EQ might have some additional coloration. So what is there to match and what sounds closer -- totally accurate frequency responce matching or matched non-linear behavior?
In case of compression same questions arise -- timing matching, GR curve matching, coloration matching...
A tricky thing with compression null testing is that attack plays very important role as mismatched transients would mostly determine the difference level. It is very easy to overlook release shape.
All in all, null test is quite a poor substitute for a proper ABX test (and even that is tricky).

[Compyfox]

Right, since I'm not a troll, I'll gracefully accept these explanations.

You've got to be f*cking kidding me.
A comment by a developer is accepted, yet from those skilled in testing software or "knowing" ins/outs of compressors are not?!


Where is the triple facepalm emoticon?!

[audiosabre]

Where is the triple facepalm emoticon?!

Sometimes you don't dig the verse or bridge, yet the chorus really resonates with you.

[FabienTDR]

Yeah, let me hack into all the compressors I own and I'll let you know what I find......or I could just use common sense knowing that you can indeed code your knobs differently in terms of weighting, travel distance, etc.

Of course we don't know what the exact algorithm is. But the plugins don't show travel distance, they show milliseconds. And it is VERY easy, - or I'd rather say trivial -, to be exact about the ms: take the sampling frequency and divide.

Hence at 44100 samples/sec, 1 ms = 44 samples. At 96000 samples/sec, 1 ms = 96 samples.
You need to compress after 1 ms ? Skip the next 96 samples. You need to release after 100ms, release start at the 9601th sample. That's basically it and there is absolutely no reason to do it any other way. So why in the world would plugins do things differently here ?

But that's not how timing is being specified in dynamics. You don't have a counter waiting x samples (read the manual!). The smoothing in dynamics processors (often controllable via the attack and release knobs) is, in 99% of the cases, done via a first order filter. Filters are generally asymptotic in nature, they never reach their target value. Remember the term "infinite impulse response"? That's why we use seemingly weird forms of specification when it comes to filters. You can find countless of them:

- Corner frequency specification uses the -3dB point (and of course NOT "where the filter stops". It never does).
- In other approaches, you'll see a -6dB specification (say, in sinc filter specification or Linkwitz Riley specification)

When it comes to control systems, such as dynamics compressors or whatever, we're typically defining timing of a smoothing filter not by frequency, but in a time domain centric manner: Often, the point where the filter drops 10dB, or reaches 30% or 80% or whatever. There are countless ways to do so. But with most conventional filters.

Now to make the situation even worse, not all control systems are dB linear, some of them are linear in the pure sense, which in turn provokes a strong level dependency of the timing. E.g. such a compressor driven deep into GR will act much faster than the same timing setting driven only 2-3 dB into GR. Not to mention automatisms and other dynamic timing tricks.You do you compare that? You really can't that easily.

Timing specification is faaaaaaaaaaaaar more complicated than you are drawing here. It's not clear at all, if you think it is, you're for sure way off the mark.



Regarding null testing, you are crudely overestimating its value. Point is, I can create an unlimited amount of sine waves that are absolutely equal, but don't null (in the peak sense) in PCM form. All I need is to adjust the phase of the sine wave.

Similarly, I can generate an unlimited amount of totally different signals that show exactly the same loudness. All I need is an amplifier and a loudness meter.

Null testing is subject of too many false positives and false negatives. Seriously, physicists don't study for years to dance around something as trivial as A - B. This is more than ridiculous, it's fine humour. Your notion tells me that you still have a lot to learn and discover. It's normal for laymen to brutally underestimate the depth of things. A healthy humbleness (and respect for other ppl's views) is the only way out. If you don't, you'll probably soon mutate into a troll.

[incubus]

I've decided that I like compressors and that they are not all the same.

[lolilol1975]

Right, since I'm not a troll, I'll gracefully accept these explanations.

You've got to be f*cking kidding me.
A comment by a developer is accepted, yet from those skilled in testing software or "knowing" ins/outs of compressors are not?!


Where is the triple facepalm emoticon?!

No, a comment with actually convincing arguments and factual informations vs waving hands. Maybe one day you'll learn the difference.

[.jon]

The only one who failed to provide convincing arguments and factual information was you.