LIES, DAMN LIES, AND STATISTICS

I love baseball. That’s not surprising since I was born and raised in the Boston area and was legally required to worship at the altar of Fenway Park. As a little Masshole, the Red Sox and Fenway held a magical allure. But that allure, like a lot of things in life, started to wane as the weight of the world slowly crushed the spirit and optimism out of me like it was a weighted door and I was Giles Corey (look it up). Fortunately, one man single-handedly saved my interest in baseball. That man was Bill James.

James is the patron saint of baseball statistics. When you read about modern stats like OPS, VORP, ERA+…what you’re reading about is the fruit of James’ labor. He essentially created the statistical revolution in baseball, much like Jim Koch and Pete Slosberg reinvented modern American craft brewing. Today, almost every Major League team (except the Royals apparently) incorporate complex statistical analysis in their talent evaluation.

I find this stuff fascinating. I love the idea of using real, quantitative measures to attempt to find “truths” in messy fields. Of course, the fields where stats tend to dominate are the fields where lots of money is at stake. There’s a reason baseball and finance make use of the most complicated statistical analysis known to man…there are billions of dollars in play. If we, as a society, devoted the same statistical resources to something like global warming or poverty, we’d probably solve those issues in a month. But there’s no money in that, so it ain’t happening.

When the magic of baseball began to dissipate as I got older, I found renewed passion for the sport through following the numbers. Baseball is a numbers game, of course…and by delving into the stats, the game all of a sudden had many more layers of depth than it did when I was a kid. I’ve always been interested in the way stats can be applied to other fields as well…so I decided to try some number-crunching on Aleheads.

My goal was to determine the Best Brewery in the US (the subject of a recent post) through a very simple mathematical approach. The problem with this plan should be obvious. The quality of a beer is highly subjective, thus determining the quality of the brewer is equally troubling. So where would one turn if one wanted to make a noble, but ultimately doomed effort to quantitatively judge a brewery?

This issue presented itself when I was attempting to convince Sudsy that Founders was just as good as Three Floyds. I had selected Founders as my favorite midwestern brewery and Sudsy was so apoplectic that he threatened to duel me with sabers at dawn. We both quickly realized that we didn’t have any sabers…and we live 1,300 miles apart…also it’s February and dawn is just too damn cold…plus I’m fairly certain dueling is illegal. Once that was off the table, I realized I needed some cold, hard math to show Sudsy that I wasn’t just talking out of my ass (which, of course, I was).

There are a number of great resources on-line for beer rankings, but for my money, BeerAdvocate is the best. Sure, there are plenty of rank and file tasters on there that don’t add much to the proceedings, but overall, I think the site has more “professional” tasters than many others and, as such, the rankings on BA tend to be more accurate than elsewhere. So I used their site for my raw data.

Thankfully, BeerAdvocate quickly allows you to pull the rankings for every beer from every brewery. I simply made a spreadsheet for Three Floyds and Founders and sorted both breweries by the grades given to their various beers. Then I assigned a GPA to each beer based on the letter grade (from 4.33 for an A+ down to a 1.33 for a D+…there was only one of those…these are excellent breweries after all). I then averaged all the grades out for an overall GPA. The problem with that approach, which I quickly realized, is that it gives every beer equal weight. This wouldn’t matter if all the beers were fairly graded, but many of the brews on BeerAdvocate have only been rated a handful of times. You simply can’t give equal weight to a beer that only 3 people have written about when another brew might have 1,000 ratings. I decided to cull out the outliers by eliminating any beer with less than 10 rankings. 10 rankings may not be much, but I think it’s enough to give a decent approximation of how good a beer is (generally speaking, such a small sample size will give you roughly a 30% margin of error, but because the respondents are mostly knowledgeable Aleheads, I think it’s considerably less than that…maybe 10-15%).

Once the brews with too few ratings had been pulled, I recalculated the GPA and considered the final result my “true” overall GPA. The results are on an earlier post, but to recap, Three Floyds finished with a very solid 3.60 GPA and Founders with a 3.63. A virtual tie. This was a fun exercise, so I decided to repeat it with a number of other breweries. I’ll post the full list after this, but here’s a little recap…

I selected 46 breweries altogether. These were breweries the Aleheads had talked about frequently as being potential candidates for best in America. I made sure they all had at least 20 beers rated on BeerAdvocate (that includes “retired” beers). That number may seem arbitrary, but I didn’t think it was fair to include a brewery that might make only three or four beers since their rankings could be severely skewed in either direction (each beer would have carried too much weight in their overall GPA).

I obviously didn’t include every great brewery…there are many excellent ones I didn’t rate and I plan on adding to the list over time. My goal was to pick the breweries that I thought had the best shot at being number one overall and I hope I at least addressed all of those. Some highlights:

• The overall average GPA of all 46 breweries was 3.40. A solid B+.
• The median GPA was 3.39.
• The lowest scoring brewery was Magic Hat…no surprise there since they brew some funky, fruit offerings that a lot of Aleheads shun (still a great brewery though).
• The two median breweries were Bells and Goose Island…that feels about right to me. Both are great breweries that aren’t quite at that “world-class” level but have the potential.
• The top five are dominated by breweries that are fairly provincial, with one exception. What I mean is, these are breweries whose beers are difficult to find outside of their home region. As such, they tend to be ranked high for two reasons:
  • 1. They’re generally only being ranked by locals…and locals almost always favor their hometown breweries.
  • 2. The rankings NOT done by locals are done by beer geeks who went out of their way to get their hands on the beers. So they’ll be inclined to give higher rankings as well to psychologically justify the obstacles they overcame to taste those beers. That’s not to say the top breweries aren’t really the top breweries…just want to make sure our readers know to take everything with a grain of salt.

Those top five are:

1. Russian River (rarely seen outside of Northern California…everything they produce is gold, Jerry…gold!) – GPA 3.76
2. AleSmith (very tough to get outside California…their Speedway Stout and IPA are legendary) – GPA 3.73
3. Surly (the brewery we knew the least about and the only non-California one in the Top 5…Ripped and I each have had only one offering by them…would definitely like to have more) – GPA 3.72
4. Lost Abbey (simply great stuff…getting easier to get your hands on, but still fairly obscure) – GPA 3.72
5. Stone (the one “non-provincial” brewery…we at Aleheads find them very overrated, but they’re a darling of the beer world with great marketing, cool-named beers, and some of their beers are no-question, hands-down winners) – GPA 3.68

The full list will be following shortly.