Why Stars, Likes and Thumbs Don't Work For B2B

 In All Things Productivity, Blog

XKCD Understanding Ratings

Isn’t this xkcd.com graphic the coolest summary of star rating systems?  

Human driven ratings are an attempt to distil multiple inputs about a certain object (e.g. books, films, food) down to a single number, or a thumbs up or thumbs down. The idea of numerically summarizing something is great – it helps you decide what is good or meets your tastes, while saving you time and removing indecision. The problem is, human powered ratings systems don’t work.


Peter Calder argues that an opinion is impossible to reduce to a number, and that people should read reviews, not look at ratings. There are many other great pieces on the issues of human powered ratings systems.  Check out goodfil.ms, and Evan Hamilton’s great post on J-curves.  

A rating scale might have 100 points on it, or just 5, but all the useful ratings are crammed into the top third, and suddenly things are looking like XKCD’s example above.

The Problem with getting People to rate things

There’s lots of research and writing on this topic. The problem essentially comes down to:

  1. People only rate things if they love them or hate them. If you love the hairdryer you bought you might go online & give it a 5-star rating. If it broke immediately you’ll angrily seek out the ratings system to punish it with a single star. But if it was just ok? You’re not going to do anything.
  2. People are lazy. If you’re highlighting the top-rated items on your site, you’re unlikely to ever get ratings on the other items.

This leads to rating systems looking like a J-curve.


But wait, it gets even worse

How? Well because of the 1% rule.

The 1% rule is a rule of thumb pertaining to participation, stating that only 1% of the users of a website actively create or rate content, while the other 99% of the participants only lurk. Andrew Hubbard goes into more detail on his blog and presents some statistics that highlight the problem. Andrew documents the story of two apps, Hotel Tonight and Twindr. Hotel Tonight announced they had achieved 4 million downloads on iOS. As of December 2013 they had approximately 6000 reviews. That equates to just 0.15% of users leaving a review.

In the case of Twindr, Andrew estimates that for every 18000 downloads there are only 19 reviews. That equals just over 0.1%, or roughly 1 out of every 1000 users reviewed the app.

Yelp cover this phenomenon in more detail on their official blog and you can read more on Quora and Wikipedia

What we have learned?

  1. Human powered rating systems don’t work
  2. Less than 1% of people participate in rating systems

Rating systems within enterprise applications

With the shortcomings of human powered rating systems firmly established, wouldn’t it be crazy to put such a system into an enterprise application? Yes, but it still happens!

The biggest challenge for enterprise applications is getting people to just use them, nevermind getting users to be active with the social sharing functions. Enterprise workforces are especially hard to convince that there is value in this activity. Tamara Schenk from CSO Insights, a division of MHI Global, says the value of ratings and likes is precisely zero. Tamara also highlights that she has discovered content creators trying to game the system by rating their content high and their colleagues’ content low.

A Better Way

Think about how Google ranks the web and try to apply that to how you rate content, or anything else. Google uses more than 200 signals when ranking a webpage.

How Google Works

At Docurated, we use DocuRank to rate content. The DocuRank algorithm incorporates many signals including author, freshness, subject matter, and content usage analytics such as views, shares, time, downloads, ROI, searches, sorts, and filters. Using these inputs DocuRank then presents users with the right piece of content for their specific scenario.

Docurated tracks how each piece of content is used at an atomic level. Each content piece, or object, is an individual atomic element that can be manipulated and used. For example, a 25-page PDF is broken down into individual pages, and then each page is broken down into the parts that make up the page. Each piece is then indexed using Docurated’s proprietary algorithm. Unlike a lot of other products, which require you to tell them what the content is about (a.k.a. tagging), the Docurated solution tells you what the content is about.

There are three main components to DocuRank:

  1. Mining signals and using that data as a technique to gauge the relevance of a piece of content. Examples being views, clicks, downloads, re-use, modifications, creator, ROI etc….
  2. Capturing the data and signals from all content repositories. Focusing on a single silo or information store will cover only a small subset of content and signals and will provide distorted results. It would be the equivalent of Google only crawling a small subset of web content. The more complete the corpus of content being analyzed, the more relevant and impactful the results and vice versa.
  3. Understanding signals and relevance down to the page and object level within a document. Note, content, in this case, is any artifact, it could be the entire document, or it could be a specific piece of text, image, chart within the document

Like Google’s PageRank algorithm and how it uses web signals to determine what websites are relevant, DocuRank leverages a combination of object based signals across the entire corpus of enterprise data to offer a powerful solution to managing, understanding, and using content.

A Final Thought

Human-driven content rating systems are inaccurate and distort the true quality and relevancy of a document or deck. The next time you see a human powered rating system in an enterprise application please email the product manager and let them know they are wasting resources on something that will not be used, and if it is used the result will be inaccurate data.


Related Resources

Recent Posts

Leave a Comment

Start typing and press Enter to search