Diva vs Analogue - a real world test

[Lotuzia]

.....
.... Maybe the thread should be called "a real word comparison" rather than "test".

Good point imho. ( Would not change what people will then make of the tread, but could have been more neutral from the beginning )

[EnGee]

Actually I think it is a scientific test, it just deals with perception. "Which would you use to make music with?" is a question about artistic preferences. "Can you identify which one of these is which?" not so much.

Is this your opinion? or the OP's or what?

How it is a scientific test? What the machines/tools she/he used to measure the sound, frequency ..etc?
Scientific? Show me numbers then!

For something to be scientific doesn't mean that it's just measured by rulers or oscilloscopes or whatever. It's a test of the state of analog emulation and whether musicians can discern a difference. The numbers that you get in this case would be the number of people that got it right or wrong, or couldn't tell a difference.

Now that part isn't being done in the best possible way since these responses are being done on a forum and everything, but what matters the most is the methodology behind the test itself. If you were to bring in a bunch of subjects and take down their all there answers it would be a good study. But we can still learn something from our own answers and the other answers given, it will just be a less accurate test than if all the response gathering was really done the right way.

If you want to say testing human perception can't be scientific, every psychologist and neuroscientist will argue against you. There are journals full of perceptual tests.

Sorry, it is simple. If you can't measure it, then how would you be able to compare it?

I'm not against this test at all. It is a nice test even if it is done for advertisement. I say, well done but to say this comparison based on scientific methods to prove that Diva is the same exactly like (or 90% or 98% ..etc) analog synth which costs 20 times! Come on!

Show me the scientific methods that were used in this test. Perception and telepathy can't be measured!

[IncarnateX]

Come on. We already dealt with the fact that a correct answer could be due to chance alone on page 1 and 2 in this thread because there only are two options. It should have been clear by then that this could not be considered a scientific test. It's an advertisement and the rest is a subjective. I actually thought people took this premise for granted.

Wrong, if you have two options and people are just guessing, you would expect a 50/50 spread in the answers. So what you look for in a test like this is whether the responses are significantly far off from the 50/50 spread. If you have 80% getting it right that isn't explained by chance. That's assuming of course that you have enough responses. As I said in another post, on a forum like this isn't a great way to collect this data. So that's a problem, but the fact that there are two options isn't.

No you are wrong. The test isn't optimal designed for that kind of conclusions because the sequence is the same all the way (ab, ab, ab)and besides as noted by yourself, you need a lot of subjects to statistically reach that conclusion and finally you are still not able to validate your own beliefs for your choice beyond chance.

[IncarnateX]

I understand your point - I am a scientist. However I did not present this as a scientific test.

I got that from the start and it is perfectly acceptable with me. You have stimulated some discussion all right so mission accomplished

[Ogopogo]

The test isn't optimal designed for that kind of conclusions because the sequence is the same all the way (ab, ab, ab)

Maybe it's not optimal but how does that invalidate it as a scientific test?

[recursive one]

Well, this is not a "scientfic" test. It doesn't have a 0 hypotesis (like "Diva is 100% indistinguishable from an analogue synth")

A 0 hypthesis is not the main hypothesis but the opposite, that the results from a test is based on chance. To prove something scientifically the result statistically has to be something that cannot be explained by chance but can be explained by the main hypothesis. There are statistic means for that butbthe design of the test has to live up to certain standards.

Besides if you agree that this is no scientific test, you should not claim that that the test is valid. It's contradiction.

Sorry, I'm not a scientist and I may be using wrong terms. My point is that there is nothing which theoretically could be proven with 100% certainity. But as a comparison this makes sence, just because you can listen to some supposedly similar sounds and can decide for yourself if they are actually similar.

[Ogopogo]

Sorry, it is simple. If you can't measure it, then how would you be able to compare it?

You measure the responses, as I said. If enough people can tell which is which, enough so that the result can't be chance, you know that there is a difference that people can perceive.

Perception and telepathy can't be measured!

So what do you imagine this is full of:

http://da.velux.com/ar-LB/PublishingIma ... 2_w280.jpg

[kvaca]

Nice and simple - can you tell which Synth is Diva and which is the OB8?

I apologize for joining in because Im only a poor guitar player and know nothing about synths in general...but I was curious enough to see whats going on in this threat which takes 10 new pages in 2 days...and although I expected myself to find zero or small differences in posted sound examples /especially after reading all previous comments from synth experts here/ theres one question which still bothers me:

Did somebody noticed the major difference in sound of test 10 on the very end of video 2=blind test/that long sound with lots of pitchbending/

If yes- do you thing its lack of software /Diva/to emulate that specific sound, or simply a mistake in creating that sound example?
I can roughly describe the difference like there are more voices in example B...

Sametime example B /Diva/ suffers strongly from antialias filtering resulting in dull and unnatural hi-end /same problem I found with many guitar amp modelers/...so surprisingly its very easy for me to tell which is software even when I have never used these synths

btw: using some added hardware artifacts /noise/ for Diva examples is clever trick

[EnGee]

Sorry, it is simple. If you can't measure it, then how would you be able to compare it?

You measure the responses, as I said. If enough people can tell which is which, enough so that the result can't be chance, you know that there is a difference that people can perceive.

Do you mean you base your test on poll? on random sample in KVR? right?

Perception and telepathy can't be measured!

So what do you imagine this is full of:

http://da.velux.com/ar-LB/PublishingIma ... 2_w280.jpg

So, you based your test on the methods in this book?
Ok, show me your research then. How did you measure every one's perception? every one that participated in this vote/poll/test or advertisement?

[SWAN808]

I understand your point - I am a scientist. However I did not present this as a scientific test. As per above I have provided the tools for this to be evaluated scientifically with wavs and ABX link. I hate to say it but I do not see forums as a place to try to solve these issues absolutely.

A/B/X is a better methodology, but that doesn't mean a/b is completely unscientific, right? It's been a while since school for me, but I think an a/b test could still be called scientific.

hi - just to clarify - you are right a blind A/B test is useful. An ABX test is the same as a blind AB test.

The reason why this test could not be called 'scientific' is because I have not provided my hypothesis and per person - it is not statistically significant because a person could get it right through chance of 50:50. However - as a larger group - it can become statistically significant.

Best is to shoot for a confidence level above 95% so that would mean 9/10 people would have to get it right or wrong. So if 9/10 got it wrong you could say that it was a significant result showing that group on KVR could not tell the difference with this set of sounds. However - its very difficult to test the full range of a synth - and we don't know much about the people who take the test. Thats why I prefer to keep it simple and for fun. KVR isn't a very controllable environment - but there is interest and discussion to be had.

Someone can easily download the wav, cut them up and put each A and B sound into an ABX tester. They listen to A and B - and then are presented with a blind file X - they then have to choose which out of A or B X is. They need to get it right 9 out of 10 times to be statistically significant to a high confidence level. There are charts around but you can also do 13 out of 15 etc...

Honestly on forums most people really aren't prepared to go this distance. You really have to spend time listening to short loops of audio to hear fine differences. I recommend it but I don't ask for it from people. Whenever I do tests now days tho I take my time. Initially it often sounds similar but you have to get used to the audio because our ear listens to the music first...then the tonal differences start to present themselves IME...

[IncarnateX]

The test isn't optimal designed for that kind of conclusions because the sequence is the same all the way (ab, ab, ab)

Maybe it's not optimal but how does that invalidate it as a scientific test?

It does not necessarily invalidate it as a scientific "design", but statistically you will not achieve the optimal certaincy about the result as if the sequences had been random. However, since we do not know what hypothesis is tested and even less about the results (which one is DIVA) we do not have much to judge any "validity" of the test from.

[Urs]

@Urs: I hope it's okay with you. I'm feeling a little as if I'm trying to bash Diva. Believe me, I am not. My apologies, if this is causing hard feelings, was not intended.

Are you kidding? He is laughing all the way down to the bank thinking about all the money he for the hundred and seventeenth time has saved on advertisement. There are simply no DEV that have ever got so much free attention as him.

God forbid him starting to whine about anything. If he do, I' ll gather the whole friggin North and invade Germany.

Well, not much need for whining. But do I feel relaxed after reading so much hostile nonsense? Not really.

In 4 anno Divae I was hoping that some developers would have gotten over it. Obviously we committed some sort of DSP treason. Shock horror - we didn't purely rely on objective measurements, we used our ears. In fact, we favoured some algorithms that were less accurate than others simply because they sounded better - even closer to the original. Still, four years have gone by and yet: Here we are. I'd wish that someone came up with more than just a few theories as to how we've done it, and postings that feel more like personal attacks than like anything else. Taylor Swift has made a great song about this. I'll shake it off

(btw. those developers who have done stuff in Diva's league usually don't need to fight... in fact I had drinks with Andy Simper just a few days ago and before that I had a nice email exchange with Mike Daliot)

As for the validity of the test... it's a great example IMHO. We chose the right feature set to cover the grounds that Diva was intended to cover. Can't express enough how grateful I am towards analoguesamples909 for doing this. It's what makes our work so rewarding

As for Vadim's statements, sure, extreme sounds require a different design than Diva. Just please don't blame *us* for not having come up with that - I still think ACE is among the best things out there for this, but it's eight years old and it predates my ability to employ engineers. I did it myself back then within the limits of my skill set. We'll eventually upgrade it and raise the bar with the knowledge we have at hand today.

[Ogopogo]

Sorry, it is simple. If you can't measure it, then how would you be able to compare it?

You measure the responses, as I said. If enough people can tell which is which, enough so that the result can't be chance, you know that there is a difference that people can perceive.

Do you mean you base your test on poll? on random sample in KVR? right?

Perception and telepathy can't be measured!

So what do you imagine this is full of:

http://da.velux.com/ar-LB/PublishingIma ... 2_w280.jpg

So, you based your test on the methods in this book?
Ok, show me your research then. How did you measure every one's perception? every one that participated in this vote/poll/test or advertisement?

EnGee, if you read my earlier post I said that kvr would not the best place. But yes, plenty of scientific research is based on polls. And for probably the third time in this case you would measure perception by seeing if enough people can accurately tell which is which.
As for "showing you my research" I don't know what you're talking about, if you want to know about perceptual science I suggest you just do some googling and you will see that it does indeed exist.

http://lmgtfy.com/?q=perceptual+science
http://lmgtfy.com/?q=perceptual+science+journal

[penguinfromdeep]

What is your guess Urs? LOL

[IncarnateX]

But as a comparison this makes sence, just because you can listen to some supposedly similar sounds and can decide for yourself if they are actually similar.

Of this we agree. Just do not mix it up with scientific designs and validity. There are loads of topics for sound discussions in it and there are still things to be concluded at a more anecdotal level.