Can You Trust Online Beauty Reviews?

Racked is no longer publishing. Thank you to everyone who read our work over the years. The archives will remain available here; for new stories, head over to Vox.com, where our staff is covering consumer culture for The Goods by Vox. You can also see what we’re up to by signing up here.

Let’s say you’re in the market for a new mascara. You’ve heard about Lancôme’s new Monsieur Big, so you pop over to Sephora.com to read some reviews. The first five reviews listed all award the product five stars. Encouraging! Then when you read them, you discover that four of the five feature language like “received this product for free for testing purposes” and “I received this product complimentary from Influenster.”

Gifting in the beauty industry is obviously nothing new, but over the past couple years, there has been a rise in influencer and consumer programs that seed products in return for reviews, an important part of a shopper’s decision-making process. According to Bill Tancer, the author of Everyone’s a Critic: Winning Customers in a Review-Driven World, about 89 percent to 90 percent of all consumers use reviews to make purchase decisions, up from 72 percent in 2014. “It is the most influential thing in terms of making a purchase decision,” he says. “Much more influential than what a sales clerk would recommend and even more influential than what friends and family would recommend.”

Various third-party companies currently work with beauty brands to get free products into the hands of reviewers. Often those brands will then require them to review the products on Sephora.com or even Amazon in addition to their own social channels.

One such company is Influenster, which was founded in 2010. It features its own review platform, which has 3 million members. They can review whatever products they want there, and are not limited to beauty. But there’s also a program, VoxBox [note: VoxBox is not affiliated with Vox Media, Racked’s parent company], that matches brands with select Influenster reviewers to provide free merchandise to review. Reviewers are expected to fill out a survey about the products, and people can then review it on their own social-media channels. “They’re not told what to say. There’s encouragement of, ‘Hey what do you think? What are your thoughts on it?’” says Elizabeth Scherle, Influenster’s founder. Influenster gives all this data back to the brands, who pay for this service.

Brands can also request, as a condition of getting the free stuff, that reviewers leave reviews on other channels, including Sephora.com. These Influenster reviews are ending up on the beauty retailer with increasing frequency, as one cynical YouTuber recently noted while she was searching for reviews of Marc Jacobs’s Dew Drops Coconut Gel Highlighter. (You can see a big concentration of Influenster reviews in April for that product.) Reviewers do have to disclose via several mechanisms that they received the product free for review, so their affiliation with Influenster is obvious.

While Influenster (and similar services like BzzAgent and Crowdtap) facilitates product reviews among so-called regular people, Octoly, which works exclusively with beauty companies, goes after micro-influencers. This is a different category than the influencers who have 1 million-plus followers, who get paid six figures to promote products, and who are whisked away to Fiji by brands. Many are up-and-coming, with fewer than 100,000 followers. If you meet certain follower criteria and are accepted by Octoly, you can “shop” at the stores of 150 different beauty brands with store credits. If the brand then accepts you for a “campaign,” it will send you products to test. Once you review them on your social media platforms or vlog, credits are put back into your account and you can go “shopping” for more free products. Octoly works with 3,200 influencers in the US (and 7,000 total globally, including in France and Spain) and with brands like YSL, Clarins, OPI, and Lancôme.

Octoly’s vice president of sales , Philippe Garnier, stresses that he expects honest reviews from the testers. “What the brands are doing right now is they are looking for authenticity and engagement and to get that, you need to work with the smaller influencers that really have passion,” he says. “I’d say they’re a new kind of journalist. They just want to do their job, and to do their job they need the product for free…. We want them to keep their own voice. They can give good reviews or negative reviews.” And can a company choose not to give more product to someone who writes a negative review? “Absolutely. They can totally decide to do that.”

Krystle, 32, started reviewing products on her Instagram page, where she has over 19,000 followers. She also recently started a blog called KatHouse Beauty. She has received products via both Octoly and Influenster, and estimates that she has reviewed almost 20 free products, including some from Lancôme, Kevyn Aucoin, Givenchy, Ofra, ByTerry, and OPI. She says she tries to give both the pros and cons of product when she reviews, like when she criticized the packaging on a lipstick. She sometimes won’t post a review on a product she hates, instead choosing to just return it if it’s something she purchased.

Krystle explains why programs like Influenster and Octoly are important to her. “Beauty blogging can get a little expensive because you want to stay on top of the trends,” she says. Certain beauty companies may not send free PR products to smaller influencers the way they do to the more popular influencers and to beauty editors in the media, including those here at Racked.

After the reviews are written, the brands on Octoly can license the influencers’ content to use on their own pages. It’s common sense to assume that negative or mixed reviews won’t be picked up by brands for licensing. A small percentage of brands ask that an influencer leave a review on Sephora or Amazon, though an Octoly representative clarified that these aren’t tracked and stressed that brands don’t tell reviewers they must leave a positive review. Reviewers are taught to use the hashtags #gifted, #honestreview, and #octolyfamily in an effort to meet FTC guidelines, though recent crackdowns on celebrities highlight that the wording should be really obvious.

Sephora is aware of the so-called incentivized reviews and monitors them. “As part of our terms and conditions for leaving a review, our users must disclose if they have been sent promotional samples or select our ‘received as free sample’ button,” a Sephora representative said via email. “Samples are a part of the beauty industry and are one of many ways that brands market to potential clients. Our goal with the Sephora Ratings and Reviews is that regardless of how you discovered a product, your share of your experience is authentic, unpaid, and unbiased.”

It’s hard to determine how unbiased they are, though. Incentivized reviews, at least for Amazon, have become the online review equivalent of fake news — call them fake reviews. Amazon has long had an issue with “review clubs” on Facebook and elsewhere that allow brands to give people free products, sometimes with explicit instructions to leave high ratings. A lot of people buy beauty products on Amazon, so it’s worth reading those reviews critically, too.

About 40 percent of Amazon reviews are “unreliable,” according to analysis by the founders of Fakespot, a free online tool that allows you to enter an Amazon review and receive an estimation of how trustworthy it is. “In the simplest terms, when the product is put into our engine, it filters every review left for the product, but it also goes into every reviewer and reviews and analyzes every review that reviewer has ever posted,” says Ming Ooi, the co-founder and chief strategy officer of Fakespot. It looks for certain words, patterns, and reviewer activity level.

Another company, ReviewMeta, provides a similar service. In June of last year, it published a data analysis of over 7 million Amazon reviews. The results indicated that reviewers who wrote “incentivized reviews,” meaning they received the product for free or at a discount for review purposes, were more likely to leave a positive review. The data showed a 0.38 star increase, but as the authors note, “Considering that the average product on Amazon is rated around 4.4 stars, a boost from 4.36 to 4.74 stars can mean the difference between a mediocre product and a top rated product.” It can boost a product from the 54th percentile to the 94th percentile. In general, “incentivized reviewers are 12 times less likely to give a 1-star rating than non-incentivized reviews, and almost four times less likely to leave a critical review in general.”

While Amazon had previously allowed incentivized reviews to be posted with proper disclosure, in October 2016, shortly after the ReviewMeta analysis, it banned all incentivized reviews except for those run through the Amazon Vine program, an invite-only review platform. It then went on a review purge. Another recent analysis, however, shows that there are still many “fraudulent” reviews there.

Even with the Amazon review purge, you can still find questionable reviews there. Baebody Eye Gel has over 7,000 reviews, an average 4.5 star rating, and is a consistent top 20 Amazon beauty bestseller. The company has only 218 Instagram followers and its “about” page is light on details about the company and vague about its ingredients. (The first one listed on the package is “organic herbal infusion.”) Plugging it into Fakespot provides some revealing results: Fakespot gives its reviews a “C” rating. Even more telling is the fact that it gives the parent company, Baebody, an “F” rating, noting that “the Fakespot algorithm considers 60 percent of those reviews to be unreliable.” This doesn’t mean the product is bad, just that some of the reviews might be questionable in some way. Baebody Eye Gel fails on ReviewMeta also. The super popular and also mysterious Aztec Secret clay mask, by comparison, gets an “A” rating for the trustworthiness of its reviews.

While ReviewMeta’s analysis is applicable to Amazon only and did not drill down specifically to beauty, there are certainly corollaries that one can draw in the beauty world. Tancer confirms that, not surprisingly, getting free products can affect how a person reviews something: “There’s always going to be the fear that that goes away… and could drive the decision to only leave positive reviews.” In his book, he calls this type of person the “Benevolent Reviewer.” (This is obviously different from a situation in which a reviewer gets paid for a positive review.)

Beauty reviews are especially tricky, even without incentives, because as Margaret Eby pointed out here on Racked, “Makeup is hard to be objective about. After all, everyone has different skin chemistry and aesthetic preferences.”

And Tancer notes that “No one’s got a rock-solid number, but in my estimation it’s only about 10 percent to 15 percent of the population who are actually writing reviews.” Then there’s the question of whether reviews on retailers trying to sell you stuff are trustworthy once you filter out incentivized reviews.

“I tend to trust reviews on sites like Makeup Alley and Reddit more than on Sephora itself,” says Libby, 30. “I guess out of a vague sense that they could censor negative reviews if they wanted to. I also think there’s probably a bias toward the extremes with reviews — it’s hard to get revved up to leave a review for a product where your verdict is just ‘meh.’”

Sephora, while much beloved by its devoted customer base, does face some skepticism when it comes to reviews and which ones its moderators publish. Grace, 40, a self-proclaimed makeup junkie, is in two makeup Facebook groups. “I absolutely trust the opinions of the ladies in the group because I know they are unbiased. I do read Sephora reviews too, but with a huge grain of salt,” she says, noting that she suspects the retailer removes some negative reviews. “Sometimes they let some slightly negative reviews get published and I do find those useful.” (In June, Sephora took heat from reviewers for allegedly removing negative reviews about its $20 Play VIB Rouge sample box, then for removing the comments about removing reviews. It released another of these boxes in August, but it appears negative reviews were allowed to stay, with the overall rating tanking at 2.5 stars prior to the product being removed completely after it sold out.)

Sephora says explicitly in its review guidelines page, “We reserve the right not to post your review or to withdraw any posted review for any reason.” It does state specifically that obscenities, spam, references to other products in reviews, and comments critical of other users will not be posted.

A Sephora representative said via email about its policy: “The goal of Sephora Ratings and Reviews is authentic client experiences that can inform fellow beauty fans on the product solutions and benefits based on their concerns. Reviews are monitored and if needed, removed if they are truly not centered around the product experience, therefore not a benefit to the larger community.”

Ulta’s review section doesn’t get quite the heat that Sephora’s does, either in number of overall reviews per product or reviewer sass, but at first glance it seems that reviews skew mostly 4-star or higher (as do Sephora’s), with a poorly overall reviewed product here or there. “Maintaining the integrity of the reviews is extremely important to us. We do not remove or censor reviews that offer the honest opinions of our guests,” says Prama Bhatt, Ulta’s senior vice president of digital and e-commerce. Ulta has similar review restrictions as Sephora and will remove them based on those criteria. So do retailers remove negative reviews? Maybe sometimes.

Reviews are clearly crucial for brands. Lancôme has “seen great results” from using Octoly and Influenster, according to Doreen Arbel, senior vice president of marketing and Kia Lowe, vice president of Sephora sales and marketing, both at Lancôme. “The pool of testers within Influenster tend to be huge beauty junkies that shop at Sephora,” they wrote via email. “Sephora reviews for the brand are important given it’s a ‘go-to’ search engine destination for beauty products… Getting non-biased feedback on our products is critically important!”

Small indie beauty brands are the ones that may suffer the most in this environment. “The most important thing for any brand is exposure. Because if you don’t have exposure, you don’t have sales,” says Alexis Irene, the founder of Static Nails, a high-end line of reusable pop-on nails that launched in May 2016 and is carried in Urban Outfitters, Lulus, and HSN, and that will soon be in Forever 21’s new Riley Rose beauty boutiques. “It will always be difficult to compete against companies that spend tens of millions of dollars on influencers and traditional campaigns to shove their brand down everyone’s throat,” she says.

During Static Nails’ partnership with Sephora, Irene said her social team once reached out to two people who had independently praised her brand on Instagram and offered a $5 gift card to leave a review on Sephora.com. “Across the board, reviews are really important,” she says. “If a brand has a lot of reviews, it makes your brand look bigger and more credible, which results in more people trying your products… It was when we were brand new to Sephora and only had a few reviews. There were tons of people raving about Static Nails on Instagram, but no one leaving us reviews.” She hasn’t done anything like that since then, but says, “I do regret not being more proactive about getting reviews. For small brands like mine, positive ones mean the world and can help the brand grow, where constructive ones can shed light on areas to improve, which is extremely important as well.”

Irene is probably right that the number of reviews is important. A study published last month in the journal Psychological Science gave support to this theory. In it, the authors found that when presented with two products with the same average Amazon star ratings, study participants always chose the product that had more reviews. This held true even in cases where those reviews resulted in average lower star ratings, a situation that statistically indicates that you should probably stay away from that product since many more people decided it was mediocre enough to warn other people about.

In this complex review environment, with so many backchannel deals happening, and with a product category as emotional as beauty, how does one know what to believe? Tancer has some advice. “If you want to make the most informed decision, I think you’re best served by looking at a cross section and number of different platforms before making your decision.” That, and buy from places with good return policies.

Update: September 23rd, 2017, 11:01 a.m.

This article has been corrected to indicate that Philippe Garnier is Octoly’s VP of sales, not a founder.