One of today’s most prominent concepts in product development is user-centered design. Who are your customers? And what are their specific needs and experiences? User-centered design is the foundation of “design thinking,” an approach to creating products and services that begins by asking those types of questions. And over the past twenty years, a parallel trend has emerged that could seemingly provide answers: online retail and customer reviews.

Customer reviews have become an omnipresent feature of online shopping. Go to any large retailer—Best Buy, Nike, Bed Bath & Beyond, etc. —and you’re sure to find a “Write a Review” button where customers can submit the pros, cons, ratings, and general feedback about any given product. Though this customer-generated data is widely used, both by customers who want to make the best product purchases, and by businesses who want to optimize their online presence with positive customer feedback, do we really know how useful this data is? Evidence suggests online reviewer feedback suffers from significant biases and inaccuracies that could significantly undermine their value.

Following the Leader

Amazon, the undeniable leader in online retailing, made the online reviewing process mainstream. But from the get go, online reviews have been attained through suspect means. In fact, I used to work for a prominent online reviews company that was hired by client companies to set up review templates on their websites. My bosses would joke that the reason we had jobs was because everyone wanted to copy Amazon. And why wouldn’t they? Copying Amazon’s online experience was the best way to stay competitive as consumers began to demand more information and convenience.

Companies quickly learned how to game the review system by sending free or highly discounted products to key customers in effort to get positive reviews. The products with the most positive reviews placed higher in Amazon’s search pages. Although Amazon finally took steps to discontinue this practice in 2016, many other aggregate sites had already copied Amazon’s original review model, and misleading subsidized reviews are still common across most platforms. At my customer reviews company, we’d set up email blasts for clients who wished to solicit reviews to post on their sites. The vast majority of all reviews collected were positive, with very few negative reviews. Often, these solicited reviews would promise a gift card for a customer’s participation, sweetening the reviewer to send positive feedback. Though my company had an “authenticity policy” to never take down negative reviews from client sites, there were often very few negative reviews in the first place. Many in the company believed this was due to clients targeting customers they knew were most satisfied to bolster their image.

Caught Red-Handed

Some online retailers are not as subtle. There have been instances where misleading positive reviews have been brought to light. Todd Rutherford, a former press release writer who sought professional reviewers of books he was hired to promote, realized he could cut out the middleman by charging money to write reviews himself. He would charge $99 for one review, $499 for 20 and $999 for 50. He eventually published 4,531 reviews and at one point pulled in $28,000 per month.

He was able to do this because Rutherford’s clients found these reviews helpful. Authors who receive more reviews on Amazon tend to sell more books than those who do not, and customers are likely to spend 31% more on a product with “excellent” (5 star) reviews. His scheme eventually folded when Google realized what he was doing and refused to sell him any more ad space. Amazon eventually caught wind as well, but Rutherford is ultimately just one small drop in an ocean of fake reviews. University of Illinois data mining expert Bing Liu estimates 1 in 3 online reviews are fake and are near impossible to discern from authentic ones.

Here to Stay

Despite glaring accuracy issues, online reviews are more popular than ever. Statistics show that they’re regularly used and trusted by consumers to influence their purchases. One study has found that 88% of surveyed consumers trust online reviews as much as a personal recommendation, and 72% of consumers say positive reviews will make them trust a business more. And millennials, in particular, trust user-generated content 50% more than other media. Food and service businesses in particular seem to live or die by their Yelp status, as over 140 million monthly users visit the site to compare customer experiences before making purchases themselves.

Since they clearly aren’t going anywhere, how do we ensure higher quality reviews? Would forcing review aggregators to never solicit reviews help? Some say no. In 2017, Yelp announced a “Don’t Ask” policy that would demote a business’s page lower in the platform’s search results if it’s proven the business attempted to solicit or “artificially inflate… search rankings and online reputations.” They cited a Northwestern University study that found “customers who are prompted (by an email) to write a review, submit, on average, up to 0.5 star higher ratings than self-motivated web reviewers.” But critics of Yelp’s policy say it accomplishes the exact opposite of its goal to remove bias, because unsolicited reviews tend to be more negative, citing the same Northwestern study that states, “self-motivated reviewers are more likely to be dissatisfied.” They question whether Yelp’s new policy will actually reduce inaccuracy or simply skew toward negative reviews – another form of bias.

From a consumer perspective, customer reviews might be an indication of how well or poorly a product works, but they might also be sugar-coated, commissioned write-ups resulting from a bribe, so it’s probably a good idea to take them with a grain of salt. and never fully depend on them for costly items. From a business perspective, it’s perhaps best to regard customer reviews as cosmetically relevant - nice window dressings for online surfers curiously perusing your products. However, if a business is searching for client feedback in order to influence their product roadmapping, they should consider other means of gathering client opinions. As explained in the upcoming CMR article “Design Roadmapping in an Uncertain World: Implementing a Customer Experience-Focused Strategy,” bringing customers into an organization to work side-by-side in setting future directions and embedding customer orientation in a company could be beneficial. This “outside innovation” approach would make innovation more predictable as it aims to create experiences customers seek that will not change at the same rate technology changes. With online reviews still in their teen years and with some maturing to do, a more hands on approach for relevant data is still a better strategy.