RFC - Review Shakeup

Any problems with the site? How can we improve KVR?
User avatar
2470 posts since 24 Oct, 2000 from skelmersdale, west lancs, uk

Post Sun Oct 10, 2004 7:06 am

Good suggestions, Ben :wink:
Member 12, Studio One 4, Avenger 1.5 + 14 xps, Dune 3, XLN Audio XO, Microtonic, Pigments, Arturia 6 (barely used), Soundshaper 2, Izotope all, Valhalla, Spire, Sasquatch Kick 2, Mongoose, Panagement, Punchy Worm (turned to the right)

User avatar
9828 posts since 14 Jun, 2001 from Somewhere else, on principle

Post Sun Oct 10, 2004 4:16 pm

bobaudio wrote:Maybe its my short attention span (I didn't even read all of the posts in this thread), but I like a 0 to 5 star summary like MANY MANY websites use. If I want detail, I'll read the comments. Basically a system like CNET (pros and cons) and Amazon (Stars and Reviews).
The problem would be the same - 90% of people would give anything they review 5 out of 5 stars.
NOVAkILL 4.0 : Dell G7 (Core i7, 8GB RAM, RTX2060, Win10), UR44C, Cubase, DUNE, Thorn, TRK-01, Equator, Vacuum Pro, Arcsyn, Aparillo, Phase Plant, Pigments, Union, Analog Keys, Ultranova, Uno, Skulpt, Craft 2.

34 posts since 2 Oct, 2004

Post Tue Oct 19, 2004 8:17 am

DevonB wrote:Seriously, one more category that would be nice is

Most Underrated VST/DX?

Lot of us have favorites that NO ONE talkes about, and it might get some more chatter around some of the more obsure plugins.

Wouldn't that give away the Big Secret vst? :shock:

Voidoid Surrealist
4048 posts since 18 Sep, 2004 from Places far less tedious than this blue trainwreck...

Post Tue Oct 19, 2004 12:22 pm

I also like the idea of being able to rate the pro's and con's. I think that would be WAY more informative.

However, I'm going to be really disappointed if you end up backing down and leave out the "f**k me!" catagory :x

1174 posts since 12 Mar, 2002 from Kentucky

Post Fri Oct 22, 2004 6:00 pm

Any time line on this Ben? I have been holding off on some reviews until I see what the changes are.

All I need to be happy is one more VSTi.

3441 posts since 15 Mar, 2003

Post Wed Oct 27, 2004 10:19 pm

I think the rating numbers can be usefull for the newbees.

When I was starting out the number of available plugs was bewildering.The ratings way be somewhat arbitrary, but with enough reviews they tend to let the cream rise to the top.

The ratings allow a new user to get hold of some of the better plugs without having to spend forever reading reviews and downloading things.

461 posts since 12 Jan, 2003 from Kyoto

Post Thu Oct 28, 2004 9:26 am

Clearly the current ratings/rankings system is not perfect, but I'd hate to see the baby thrown out with the bath water. Nobody has to look at the ratings/rankings if they don't want to, but for some they're very valuable.
My main complaint with the ratings/rankings is that, as others have mentioned, they are a reflection of the time in which they were written. Synth X might have earned a deserved 9.5 when it was released in 2001 because it was a genuinely groundbreaking. But by 2004, there might be five other synths that do the same thing (maybe better or cheaper), which will get lower scores because they reflect the technology of 2004 -- while Synth X's score is probably still pretty close to 9.5 due to the weight of old ratings, and the fact that people are more likely to review new products. Synth X may have a greater claim to "greatness" than the others, but most people are probably looking for a more equal comparison between products. This also has the effect of influencing future reviews (Synth X got 9s? Synth Z is at least as good -- I'd better give it 10s! -- very soon there's nowhere to go).
There are probably many ways to reduce this problem -- monthly reviews have been mentioned. Another way would be to expire old ratings (say, those more than a year or 18 months old) or to only consider the most recent (x) ratings. Of course, the reviews & their original ratings would still be available to readers; they just wouldn't be used in calculating the "KvRers' favourites" list.
Of course, this currently is probably not tenable because there simply aren't enough people reviewing/rating stuff. Maybe this is partly because people don't think the current system is worthwhile, and partly because people are too lazy to use the current system.
One problem with ratings in general is, of course, the people who try out a copy of a product for a few minutes & judge it prematurely. Or in extreme cases, people who hold some personal grudge against the product/developer, or are shills for the company, etc. I don't think this is a big problem at KvR, largely because the system forces you to really consider your opinion -- you must write a review (not just enter scores); you must enter scores in every criterion, even if some are not important to you (or difficult to gauge). But I don't think that this would ever be a serious problem at KvR -- this isn't amazon.com; it's a niche and people don't come here on a whim. It's also a smaller community, and while you may not recognize every name signed to a review, I really think there's more of a sense of accountability for one's words. Again, there's a defense against crazy ratings -- disregard the top & bottom 10% of scores in calculating the favourites lists.
So I would suggest:
- any criterion that a score can be applied to should be optional -- there should be a choice for "not applicable" or "I prefer not to review this area" or whatever.
- you should be able to submit just scores, without a review (I'm sure many will disagree).
- the ratings could be more meaningful if accompanied by a line where people could explain their rating (EG instead of just seeing "value for money: 7", we might see "value for money: 7 / It's not cheap but its price is what I would expect for this feature set"). Yes, some people do this kind of breakdown in their reviews already.

Anyway, that was pretty verbose for a couple of modest and likely unimpressive suggestions. Sorry about that.

By the way, does anyone else think that a ranking for "most under-rated" is a bit paradoxical? I mean, if Synth Y tops that list, is it really underrated?

28 posts since 22 Oct, 2002 from Leicestershire, UK

Post Mon Nov 01, 2004 4:12 am

My concern would be that peeps wouldn't feel inclined to post any reviews if the criteria is made too long.

And yes, the current rating system does allow users to vent their frustration out on the product which is often a reflection of their inability to use them properly, but they are still a valuable factor when used alongside the program review.

Do you really believe the current system is so far out of synch and can be improved? I am not so sure.

Thanks for asking though, appreciate the consultation :)
Music is the food of the soul!

6496 posts since 26 Nov, 2004 from Frederick, MD

Post Mon Nov 29, 2004 2:13 pm

I agree that the current rating system is a bit flawed; however, I would not like to see ratings go away entirely. They make a good quick way of judging people's reactions. And wouldn't doing away with them limit our ability to sort instruments/effects/etc. by user opinion?

My proposal would be to move more toward the rating scales used in a lot of industry surveys. Keep the categories, because they are very well chosen, but instead of having a 1-10 rating for each you do something like:

This Product Is Very Stable
Not Applicable
Disagree Strongly
Disagree Somewhat
Agree Somewhat
Agree Very Strongly

In calculating an overall rating, then, you ignore the rating of "Not Applicable" thus the rating is not skewed for low ratings on something like Support when the user hasn't needed any support.

I've also often thought that there should not be an equal weight placed on the categories. Though Documentation and Support are important, to most users Stability and Sound Quality are going to be much more important. I'd way rather spend extra time learning a poorly documented product that sounds fantastic than waste time on a brilliantly documented piece of crap.

As far as the character count required in reviews goes, I'm not sure 2,000 is the right number. I wouldn't mind seeing longer reviews, but in publishing terms 2,000 characters equals 1.6 pages of print per review (250 words per page at 5 characters per word). Do you know what the average review is right now? In browsing through the reviews myself I haven't really been bothered by the size of them. They're generally of a much higher quality than music reviews on amazon.com, for example. Duh, hard to be worse than that.

Something else that might be a little helpful in gauging how much faith to put in any one review is how long the reviewer has used the program (1-3 days, 1 week, 2-3 weeks, 1 month, etc.). Initial impressions might be favorable or unfavorable, but often the test of time is what counts.

28 posts since 22 Oct, 2002 from Leicestershire, UK

Post Tue Nov 30, 2004 2:26 am

emdot_ambients makes some good points. I recently did a Vsti review and found a 2000 word max. word count difficult to achieve whilst putting over a balanced and decent enough view of the product.

And I totally agree with the emdot_ambients idea about using a different form of scoring; the one he suggests is used extensively in industry and can be weighted accordingly.
Music is the food of the soul!

6496 posts since 26 Nov, 2004 from Frederick, MD

Post Tue Nov 30, 2004 8:29 am

Character count. Character count. Not word count!

Actually I'll take back what I said about the character count. My last post had 1,646 characters not including spaces, and just over 2,000 with spaces. So, that doesn't seem too long for a review. If the reviewer makes any effort at all to address why they rate a product as they do, in each of the categories, plus add a touch of what's cool and what's not cool about the product, then 2,000 characters should be a doddle.

28 posts since 22 Oct, 2002 from Leicestershire, UK

Post Tue Nov 30, 2004 8:57 am

Sorry, I did mean 2000 characters...thats why I struggled to do justice to a Vsti in my last review. Maybe 2000 characters inc. spaces is okay for an effect but not for an instrument?
Music is the food of the soul!

User avatar
Ben [KVR]
1315 posts since 23 Oct, 2000 from Newport Pagnell, UK
KVR Audio

Post Wed Dec 01, 2004 12:55 am

Its actually a 1000 minimum and 5000 maximum at the moment, I'm not averse to upping that to 10,000.

Anyway, hopefully these changes will be implemented before Christmas.

8379 posts since 11 Apr, 2003 from now on the flat

Post Wed Dec 01, 2004 4:58 am

of the few reviews I've done, I've struggled to keep within the 5000 limit. I write them in note pad saying everything I want to, then paste it in to the box. Pretty much each time, I have had to go through and take out things.

Somtimes a whole paragraph has ended up being a rather lame sentence, or lost altogther to stay within the scope. I have found this hard, since sometimes there are a few points which I feel are all worth making, but I have to make them in a rather terse fashion.

I don't know what others experience is, but a 10,000 limit may suit better...


5 posts since 27 Nov, 2004 from Windsor CT - usa

Post Fri Dec 03, 2004 2:05 pm

:hihi: i know i just got here nstuff as far as becoming a member, ive got alot of good stuff from here in the past , im an old fart 33 www.acidplanet.com/ninjatron, check my stuff out , Im an Ex Liquid Sky member , i was formaly known as Megatron 1996, anyways i wanted to say i love this site , my eyes were wide open when i saw what this was . Sooo many people dont even know where to look for effects -you should run this site on some major search engines so more people can find this place , ive done lots of searches for effects / software and never seen this before . - never heard of half thease effects, well , im working an another explosive track , it will be up at acidplanet in a few days -im gonna deticate it to this forum / site whatever ,ill keep in touch , Ben im me ninjatron2@Hotm or Aim take it slow

Return to “Site Stuff”