Ratings “J Curve”
May 8th, 2006 by Sam Decker | Chief Marketing OfficerWe’ve talked to several advisors and prospects who expect to see a “U” shape in the distribution of their review submissions by ratings. In other words, there should be an equal number of “1” ratings as there is “5” ratings, displaying the extremes of customer opinion. We weren’t sure what to expect when Bazaarvoice got off the ground a year ago.
We didn’t know what to expect then, but we do now.
Across many clients in diverse industries this “U” curve turns out to be more like a “J” curve…almost a reverse "L" (see below). The average rating across all clients is 4.3 out of 5 stars. The distribution looks like a J, where there are more 1s than 2s, but far more 4s and 5s than the lower ratings.
Why is this? Aren’t people more likely to share their word of mouth about bad experiences? Perhaps they are more likely to share negative opinions when they have personal experiences with a company (service, sales) than the product they buy?
And perhaps customers are interested in sharing their opinion about great products they buy, because there are so many mediocre products. So there’s some satisfaction in sharing the news when we find a product we love.
We’ll learn more and share more here. But in the meantime, this “J” curve is part of the answer to the conern: “What about negative reviews?”
June 6th, 2006 at 3:32 am
[...] Ratings “J Curve” [...]
September 7th, 2006 at 6:33 am
I think the negative reviewers will be less likely to rate but will provide bad word of mouth in the real world.
This provides two possible dangers - firstly passive viewers will see the J curve effect and discount its veracity, secondly and more importantly the absence of low ratings may breed a false sense of achievement in those being rated. My philosophy has always been to look for the dissatisfied and determine if their dissatisfaction is valid - that way lies progress.
September 7th, 2006 at 7:58 am
Coincidentally, a couple weeks after we announced the J-Curve KellerFay came out with a study that found 63% of word of mouth (online or offline) is positive. The conventional wisdom that the majority of word of mouth is negative is no longer conventional nor wisdom!
September 7th, 2006 at 12:21 pm
Thanks for sharing - very counter-intuitive. And very good to know when you’re pushing to enable more openness online.
September 7th, 2006 at 4:58 pm
Hi - I originally left this comment at a blog that links here, but am reposting here as I thought you might find it useful.
This is as much of a question anchoring problem as it is a problem of certain customers responding. I’m a survey researcher, and attended a presentation recently which covered this issue (I’ve hunted, but cannot find a relevant link, sorry). The gist was that more useful, distinguishing, information could be obtained by chaninging the response options from somthing like:
Extremely good
…
Extremely bad
To something like:
Way beyond expectations
Better than expected for a good firm
About what I expect from a good firm
Not as good as expected for a good firm
Way below expectations
You’d need to play around with the wording, but the effect should be to gravitate answers to the midpoint, so that the extremes actually serve their purpose: to discriminate between the average, the really good and the really bad.
Of course, if the aim is only to weed out the really bad, then this may not matter much - the bad should show up regardless.
September 7th, 2006 at 10:28 pm
Sam,
We have approx. 18,000 user reviews of mobile phones contributed to our site and aggregated from other sources. Of those 18k 25% of reviewers gave their phone full marks, 10/10.
More on that here if you’re interested: http://www.cellphones.ca/1779
September 8th, 2006 at 7:22 am
[...] Social Networking Grows Up: Xanga Gets Fined By the FTC : : You know you’ve made it when you get slapped with a fine by the FTC. Xanga allowed thousands of under age children (<13) to register; of course, it doesn’t really address the issue that most 13 year olds will find away around any age authentication.Chinese Invade Italy — A Global Fight for Suits, Jackets and Textiles : : Globalization in reverse? Chinese have descended in a small town in Italy, and have started producing textiles IN Italy. Typical xenophobia abounds, but this is a fascinating, intimate article about a very strange twist on a very familiar topic. Not WeGooglebombing … from the source. : : Yes, if you look up “failure”, or “miserable failure”, maybe you won’t be surprised by the result. But it illustrates something key about how Google manages its search results — and the importance of linking. Here it is. File it under “old, but good aRatings Systems: Its a J-Curve for Some. : : Bazaarblog reviews their own data when it comes to “reviews” for their own products and services, and they find a J-curve distribution of data. A small amount of reviews are negative, and the overwhelming majority are positive. Interesting. What implicDaily Color Scheme : : Designing and don’ t what kind of color scheme you want to use? Check this out: a daily mix and mashup of color combinations that you may not have thought of — based on websites that exist and that are popular today. Ex: del.icio.us or google analytics [...]
September 9th, 2006 at 1:46 pm
I wonder if it would be more of a U shape if the reviews were anonymous. Perhaps people are less likely to give negative ratings or sumbit any ratings if they know their identity will be revealed. E.g., how many times have you given you waitress negative feedback on those restaurant surveys that occassionally accompany the bill? I’m running a small experiment myself WRT rating people anonymously online (www.tomslist.net). There’s only a couple hundred ratings so far and the distribution looks a bit more U-ish but it’s still quite early. We’ll see how it goes.
October 25th, 2006 at 1:12 am
Check this!
Wellbutrin
Wellbutrin
January 1st, 2007 at 1:53 pm
[...] The first article - published on the Crowdstorm blog - confirmed the opinion I had on reviews: they are great for SEO as they bring some UGC (user-generated content) to shopping sites, but are not that useful to choose a product. I had a natural thinking that consumers are eager to post reviews when they are not happy or post too positive reviews to be really useful. The article from Crowdstorm explored the “ratings J Curve”, a pattern that has been recently observed and identified by Bazaar Blog: [Crowdstorm] We’ve always said here at Crowdstorm HQ that rating products or doing reviews involving five stars is a waste of time. Now there is actual evidence coming out to support our hypothesis, starting with this post from the Bazaar Blog, entitled “the ratings J Curveâ€. What these guys are saying is that they find the average rating always ends up around 4.3 and that there are more 1’s than 2’s and far more 4’s and 5’s than the lower ratings. Their chart implies that people are more likely to share positive experiences and often go towards the 4 or the five at the end of the scale. [...]
February 3rd, 2007 at 7:19 am
Product reviews are great for the webmasters. They get a lot of content on the site free. Either by way of their own reviews, which are anyway copied from some other source, or by way of reviews written by readers. This helps a lot in SEO.
When you reads posts on any site, there is so much of contradicting views that one is bound to get confused. Moreover, you do not know which review is genuine.
I think, if you are really unhappy with a product, writing a reviews vent out your frustrations for the product. Although, you are sure that not even 1% of the site visitors will not read it. The same holds true for happy buyers of products.
February 6th, 2007 at 3:41 pm
In my experience, the take-away from a U-shape curve is not that your respondents are more “balanced”, but that your quality issues are rampant - inconsistent process, customer service, and product quality to name a few. The more the J-curve looks like a U, quality is root cause because it indicates high levels of variation among customer experiences. L’s say you’re not meeting customer expectations, while U’s suggest you are completely out of control. Conversely, the more hockey-stick the J (upward to positive scores), the greater an indication that prodcut quality, customer service, and internal processes are in sync. Virtuous cycles–by definition–work.
That said, you can’t rest on your laurels. I like the way Ben suggests variation among “happy” responses to tease out additional shades of gray, and using that as a segue for getting additional feedback.
August 9th, 2007 at 7:39 am
[...] word of mouth about brands is positive, rather than negative. As Bazaarvoice has noted in its research about the J-curve, consumers are more interested in sharing advice and recommendations with consumers about [...]
March 7th, 2008 at 12:22 pm
An obvious explanation for the J Curve.
If people are influenced by user generated reviews and I believe that they are, than customers are more likely to buy items with good reviews. Thus it stands to reason that future customers will also likely be satisfied with their purchase. In this situation the accurate representation of a product and its relative price should normally drive a J Curve.
Also it is most likely that products or companies whose customer satisfaction is inconsistent would typically not use a feedback system that would demonstrate their relative poor satisfaction levels.