RateBeer is Dead, Long Live Untappd
RateBeer.com is pretty much a dead tool. There are still users, of course. And they do have an excellent design for beer ratings. My impression is that RateBeer raters are experienced and serious beer drinkers. But compared to other beer rating sites, RateBeer is just the foam on the pint. They have so few regular raters that they are hardly worth talking about in the bigger picture of craft beer life. The fact that they still get attention is, I conjecture, a hangover from the fact that many people dominant in the beer and beer writing industries started rating there when RateBeer.com was new. Long-time users have their beer histories logged there and so they haven’t made the switch to Untappd. But these days, Untappd is where it is at, and to rely on anything else in understanding the big picture in beer is to open oneself up to mistakes based on statistically flawed samples.
Earlier today, I tweeted that RateBeer is dead and got immediate push back, so I decided to write this up to explain the grounds for my opinion. Sometimes even 280 characters is not enough.
I first dove into the details of ratings on RateBeer when a really terrible article came out a couple of years ago claiming to rank the best American cities for beer drinkers. It was so wildly bad that I looked into its methodology and discovered that they had relied on RateBeer data. So I compared the data on RateBeer and on Untappd and found my gut feeling confirmed: the RateBeer ratings were skewed and very incomplete. The explanation, I think, is essentially statistical. There simply aren’t enough ratings on RateBeer to get an accurate picture of the beer industry, or of the opinions of beer drinkers. The community of active Untappd beer raters is just so much bigger that RateBeer has, in my opinion, become nearly worthless.
That’s a strong claim, I know. To prove the point, consider the following comparisons, all data for which were gathered on February 1, 2018. [In response to a reader’s suggestion, I’ve also included Beer Advocate data in the analysis below. That rating site, too, has too little traffic to allow good claims about what beer drinkers as a whole really think.]
Let’s start with a beer that could make a case to be one of the most important beers in the history of craft beer.
Samuel Adams Boston Lager
Untappd: 821856 check-ins (386891 unique), 335106 ratings (avg. 3.43)
RateBeer: 4237 ratings (avg. 3.22)
Beer Advocate: 9297 ratings (avg. 3.74)
In this case, RateBeer has about half of one percent of the ratings on Untappd, and Beer Advocate with 1.1%.
Lest one worry that this is an effect of Boston Lager being “too popular” to be rated by “the experts” on RateBeer (a beer snob effect, if you like), consider this pretty obscure Sam Adams beer:
Samuel Adams Coffee Black Lager
Untappd: 4669 check-ins (4400 unique), 3954 ratings (avg 3.54)
Ratebeer: 37 ratings (avg. 3.46)
Beer Advocate: 34 ratings (avg 3.67)
Again, RateBeer has only about 0.79% of the number of ratings that Untappd has; Beer Advocate, 0.72%.
Okay, so maybe it is an East Coast thing. But it is not:
Stone Brewing Co.
Untappd: 1153 beers listed
RateBeer: 604 beers listed
This is an extreme case of a phenomenon that you’ll find all over RateBeer and Untappd: RateBeer often has a very incomplete listing of the beers produced by a given brewery. My explanation of this is that RateBeer raters are such a small proportion of the beer drinking population that there just aren’t enough of them to input all the beers. Of course, breweries themselves also maintain their Untappd pages, which helps ensure completeness and accuracy. The way RateBeer is set up, breweries aren’t involved, at least as I understand it.
Here’s a particular beer for comparison:
Stone Ghost Hammer IPA
Untappd: 56766 check-ins (49903 unique), 45674 ratings (avg 3.74)
RateBeer: 209 ratings (avg. 3.47)
Beer Advocate: 454 ratings (avg 3.89)
That’s RateBeer with 0.36% of the number of ratings as are on Untappd; Beer Advocate, 0.8%.
I’ve checked this for a lot of San Diego breweries, and the effect is even more pronounced at the smaller breweries. Again, this is probably because there are relatively so few RateBeer users that they end up checking in a smaller proportion of the beers a brewery produces, and the smaller places attract a proportionately smaller percentage of that already small total number of raters.
The effect is also clearly visible in macros.
Bud Light
Untappd: 1.43M check-ins (401714 unique), 327542 ratings (avg. 2.26)
RateBeer: 2729 ratings (avg. 1.22)
Beer Advocate: 6009 ratings (avg 1.86)
Coors Banquet Beer
Untappd: 13825 check-ins (11060 unique), 9457 ratings (avg 2.92)
RateBeer: 1421 ratings (avg. 1.72)
Beer Advocate: 2967 (avg. (2.59)
That’s almost a laughable disparity: RateBeer has less that 0.17% of the number of ratings that Untappd has. One could argue that the macros don’t matter to craft beer drinkers. But my point here is to show that the paucity of RateBeer (and Beer Advocate) ratings is a general phenomenon across all beer types and sectors of the industry.
To that end, the next examples show that it isn’t just an American phenomenon:
Cloudwater Brew Co.
Untappd: 226 beers listed.
RateBeer: 308 beers listed.
Cloudwater/Other Half Life
Untappd: 2825 ratings (avg 3.96)
RateBeer: 70 ratings (avg. 3.77)
Beer Advocate: 6 ratings (avg. 4.07)
Cloudwater/Bluejacket Twilight
Untappd: 325 ratings (avg 3.64)
RateBeer: 10 ratings (avg. 3.86)
Beer Advocate: not found
Cloudwater/Brew Dog New England V2 (the most-rated Cloudwater beer on RateBeer)
Untappd: 5388 ratings (avg 4.12)
RateBeer: 111 ratings (avg. 3.84)
Beer Advocate: 15 ratings (avg. 4.17)
Fuller’s London Pride
Untappd: 190007 check-ins (127935 unique), 114004 ratings (avg. 3.43)
RateBeer: 2312 ratings (avg. 3.39); 514 additional ratings for the cask version (avg. 3.36)
Beer Advocate: 2587 ratings (avg. 3.92)
So, in British beer, too, whether we are talking about an elite brewery’s small-batch collaborations or the flagship beer at a very large brewery, RateBeer has a tiny proportion of the ratings that Untappd has; ditto for Beer Advocate.
[Note added August 31, 2018: Regular Budweiser has 317,840 ratings on Untappd (avg. 2.54) and a measly 3,684 (avg. 1.48) on Ratebeer. Brewdog Punk IPA has 2,168 ratings on Ratebeer (avg. 3.72) and 256,116 ratings on Untappd (avg. 3.8). The difference between Ratebeer’s scale score of 0 for Budweiser and 97 for Punk IPA thus amplifies what is really a significantly smaller difference, while at the same time exaggerating just how highly rated Punk IPA actually is. Since their own drinkers don’t seem to use Ratebeer much at all compared to Untappd, it seems odd that they used Ratebeer in their advertising.]
The RateBeer Top 50 Beers list is humorous to me. The number of ratings it includes is miniscule compared to the number of serious beer drinkers. The number one beer, Topping Goliath Kentucky Brunch, has only 129 ratings: there are ten times more ratings for it on Untappd. (Its average Untappd rating is actually higher: 4.85 vs. 4.53.) The second place beer, Westvletern 12, has 3435 ratings on RateBeer, the most in the top 50 list, whereas it has 133032 ratings on Untappd (and 6455 ratings on Beer Advocate). If you are worried that this is comparing the lifetime total on Untappd vs the annual total on RateBeer, (a) nope, that’s the lifetime total on RateBeer, too, and (b) there were 1262 check-ins of Westvleteren 12 on Untappd in just the last month anyway. Number 7, AleSmith Speedway Stout, has 857 RateBeer ratings, and 148546 check-ins (99121 ratings) on Untappd. Number 19, Pliny the Elder, has 2689 ratings on RateBeer, 14700 ratings on Beer Advocate, and more than half a million check-ins on Untappd. I could keep going, but by now you get it. RateBeer and Beer Advocate simply have a tiny fraction of the users Untappd has.
It would be interesting to do a statistical analysis of the ratings of individual beers across the three sites. (I’m looking at you, @BryanDRoth.) From what I’ve seen, my prediction is that RateBeer ratings are more extreme: good beers (particularly famous good beers) get higher ratings and bad beers (especially macros) get worse ratings. This could have two causes—first, that the larger group on Untappd means ratings have greater spread and hence that they average out to lower values, and second that RateBeer users are biased towards giving higher ratings to higher-rated beers and lower ratings to lower-rated beers (perhaps trying to prove their cred or something). There’s a third possible explanation that I don’t put much stock in, namely that Untappd raters are less good at rating beer. While I would guess that there is a greater proportion of casual users on Untappd and a greater proportion of people we would call expert raters on RateBeer, there are just so many more people on Untappd that there are probably more expert raters (in pure numbers) on Untappd.
Whatever the result of such an analysis of the ratings across platforms, Untappd has clearly won the battle for users. It is far and away the more used tool for rating beers.
The entire BeerAdvocate.com site, not just the beer ratings, lands at the 4333rd most accessed web page in the USA (4.71M monthly users averaged over the past six months). Ratebeer.com is 17517th (2.99M monthly users averaged over the past six months). Untappd.com is 8465th (3.07M monthly users averaged over the past six months), but keep in mind that the vast majority of Untappd traffic goes through the app, not the website. Ratebeer.com’s nearly 3M monthly visitors is clearly nothing to sneeze at: I can see now why AB InBev was interested in buying a minority stake. But those visitors are getting lower-quality data than they could be getting on Untappd.
