One of the things that the big coffee chains do well is consistency. Or at least that’s the conventional wisdom, lifted straight from the fast food chain playbook. A McDonald’s french fry is pretty much the same everywhere, and the coffee served at a big chain is little different. Or so we’ve all been told.

For example, a Long Island Press article cited in our last post quoted local roast master, Greg Heinz: “Starbucks does a lot of things very well. It maintains consistency nationwide.” The article’s author then later goes on to say, “Just like any chain, Starbucks cannot exist without uniformity.”

A superautomatic Schaerer Verismo machine: consistent in theory but not in practice Or can it? We’ve always felt that the coffee quality can be quite erratic between one Starbucks and another. Lately we’ve been digging into the data we’ve collected over the years to validate some of our assumptions about coffee and quality. What we found supported our hunch that some of the biggest coffee chains are actually pretty lousy at consistency and uniformity.

Using our ratings data to measure quality consistency among cafés

Below is a table we’ve compiled by keying off some of the fields in our database from thousands of reviewed espresso shots. Each row represents the aggregate espresso shot reviews for a given chain, a given coffee roaster, a given espresso machine manufacturer, or a given cup manufacturer — reflecting a few choices we made for illustration.

Each row (or sample set) shows the number of cafés, high espresso score, low score, average score, median score, and standard deviation for all the associated ratings. For our consistency evaluation here, the key is the standard deviation — which is a simplistic measure of the spread in espresso scores for a given grouping.

Variable in common # reviews High Low Average Median Standard deviation
Starbucks chain 22 6.20 2.60 5.13 5.25 1.05
Peet’s Coffee & Tea chain 23 7.80 4.60 6.71 6.80 0.74
Blue Bottle Coffee Co. coffee 15 8.60 6.10 7.69 7.90 0.70
Mr. Espresso coffee 83 8.60 3.00 6.05 6.30 1.17
Illy coffee 72 8.30 2.20 5.73 5.80 1.33
Paper cups only 185 7.80 1.40 5.19 5.40 1.56
Delco cups 46 8.00 3.50 5.94 6.20 1.12
La Spaziale machine 144 8.20 1.50 5.52 5.80 1.42
La Marzocco machine 84 9.40 4.60 7.16 7.40 1.09

For example, the first row represents all reviewed Starbucks. The data suggest that most reviewed Starbucks — about 68 percent, assuming a normal distribution — have an espresso rating score that’s within 1.05 rating points of the average for all Starbucks reviewed (here that’s 5.13).

Now compare this 1.05 with the other example rows in the table. For example, all reviewed Peet’s Coffee & Tea outlets have a standard deviation of 0.74. This suggests a much narrower variation in their espresso scores — and hence better consistency and predictability.

Coffee quality consistency among independent cafés can be stronger than within the big chains themselves

All reviewed cafés using Blue Bottle Coffee beans may have very different owners but score an even lower 0.70 standard deviation. These cafés may only share a bean supplier and some of standards for freshness and access to common consulting and training, but their espresso scores are significantly more consistent among each other than the cafés under a single Starbucks ownership — or even Peet’s.

Surely, a single quality dimension does not represent the breadth of possible flavor profiles, body weights, and crema textures that might also factor into a consistency analysis. But this data refutes the conventional wisdom that Starbucks, for example, provides a consistently dependable and uniform level of beverage quality. Even with their complete supply and delivery chain standardization, Starbucks fails to produce espresso quality that’s as consistent as a number of independent cafés that have only a coffee bean supplier in common.

The Starbucks brand or Delco cups came out to be about equal as marks of consistency Our data also suggests that the Starbucks brand is no better a determinant of quality consistency than whether the café uses a La Marzocco espresso machine or Delco cups. This is a critical point to understand, so let’s put this another way:

The Starbucks brand predicts consistency of espresso quality no better than if all the random cafés we surveyed that use Delco cups decided to re-brand themselves as “The Delco-branded Cups Coffeehouse chain.”

Given the investments Starbucks has made in coffee bean and roasting consistency, standardized push-button espresso machines, and standardized training, we don’t see how you can interpret this data as anything short of a complete failure for the company to deliver on the brand expectations of quality consistency.

If Starbucks’ consistency isn’t in their prepared coffee, then it is likely a psychological perception: the consumer’s brand association, the consistency of the Starbucks beverage menu, Starbucks’ own ridiculous names for drinks and their sizes, and the familiar environment of its coffee shops.

Seeking out chains in the hopes of managing risk

The Waikiki Sheraton's Honolulu Coffee Company and Peet's Coffee & Tea

We recently came across a blog post by Chris Pirillo, a tech-geek/blogger/ex-TV-show-host who practically wet his underpants because he found a Peet’s Coffee & Tea chain store in his Sheraton Waikiki hotel. This when located in the very same building is an outlet of the Honolulu Coffee Company — which sources beans from Hawaii’s own Greenwell Farms, operates one of the few Mistral Triplette espresso machines in the world, and last scored as high as the best Peet’s we’ve ever been to anywhere. (Famed Baltimore coffee podcaster, Jay Caragay, disputes the quality there as dubious — but we stick by our last, albeit five-year-old, rating.)

Starbucks opens in Utrecht in the Netherlands - and it may as well have been a visit by Barack Obama Meanwhile, a friend in the Netherlands told us about a Starbucks that opened in Utrecht a week ago. When he passed by over the weekend, he said, “The line was unholy.” In a country where coffee shops legally sell marijuana by the menu, lines are out the door for one that peddles the double-tall, four-pump vanilla caramel macchiato.

What makes people seek out mass-produced mediocrity over independent, higher quality options just as conveniently nearby? What made an old boss of mine seek out a Pizza Hut for dinner when we were in the heart of London together on business travel with time to kill? These questions are woven into why we started six years ago.

In the case of coffee, you can largely dismiss the argument that chains provide a convenient shorthand to take out the risk in quality versus an unknown. As our data suggests here, big coffee chains can be less consistent among themselves than independent cafés with a common coffee supplier are to each other.

This is the part of the article where we’re supposed to have a snappy, revelatory answer to these questions. But we’ve got nothin’ — other than the consumer comfort with big coffee chains likely says more about their environment and sales practices than it does about the quality of the actual products they sell.