Breaking the Chains: Dispelling the Myth of Consistency at the Big Coffee Chains
Posted by TheShot on 15 Aug 2009 | Tagged as: Café Society, Consumer Trends, Quality Issues, Starbucks
One of the things that the big coffee chains do well is consistency. Or at least that’s the conventional wisdom, lifted straight from the fast food chain playbook. A McDonald’s french fry is pretty much the same everywhere, and the coffee served at a big chain is little different. Or so we’ve all been told.
For example, a Long Island Press article cited in our last post quoted local roast master, Greg Heinz: “Starbucks does a lot of things very well. It maintains consistency nationwide.” The article’s author then later goes on to say, “Just like any chain, Starbucks cannot exist without uniformity.”
Or can it? We’ve always felt that the coffee quality can be quite erratic between one Starbucks and another. Lately we’ve been digging into the CoffeeRatings.com data we’ve collected over the years to validate some of our assumptions about coffee and quality. What we found supported our hunch that some of the biggest coffee chains are actually pretty lousy at consistency and uniformity.
Using our ratings data to measure quality consistency among cafés
Below is a table we’ve compiled by keying off some of the fields in our database from thousands of reviewed espresso shots. Each row represents the aggregate espresso shot reviews for a given chain, a given coffee roaster, a given espresso machine manufacturer, or a given cup manufacturer — reflecting a few choices we made for illustration.
Each row (or sample set) shows the number of cafés, high espresso score, low score, average score, median score, and standard deviation for all the associated ratings. For our consistency evaluation here, the key is the standard deviation — which is a simplistic measure of the spread in espresso scores for a given grouping.
| Variable in common | # reviews | High | Low | Average | Median | Standard deviation |
|---|---|---|---|---|---|---|
| Starbucks chain | 22 | 6.20 | 2.60 | 5.13 | 5.25 | 1.05 |
| Peet’s Coffee & Tea chain | 23 | 7.80 | 4.60 | 6.71 | 6.80 | 0.74 |
| Blue Bottle Coffee Co. coffee | 15 | 8.60 | 6.10 | 7.69 | 7.90 | 0.70 |
| Mr. Espresso coffee | 83 | 8.60 | 3.00 | 6.05 | 6.30 | 1.17 |
| Illy coffee | 72 | 8.30 | 2.20 | 5.73 | 5.80 | 1.33 |
| Paper cups only | 185 | 7.80 | 1.40 | 5.19 | 5.40 | 1.56 |
| Delco cups | 46 | 8.00 | 3.50 | 5.94 | 6.20 | 1.12 |
| La Spaziale machine | 144 | 8.20 | 1.50 | 5.52 | 5.80 | 1.42 |
| La Marzocco machine | 84 | 9.40 | 4.60 | 7.16 | 7.40 | 1.09 |
For example, the first row represents all reviewed Starbucks. The data suggest that most reviewed Starbucks — about 68 percent, assuming a normal distribution — have an espresso rating score that’s within 1.05 rating points of the average for all Starbucks reviewed (here that’s 5.13).
Now compare this 1.05 with the other example rows in the table. For example, all reviewed Peet’s Coffee & Tea outlets have a standard deviation of 0.74. This suggests a much narrower variation in their espresso scores — and hence better consistency and predictability.
Coffee quality consistency among independent cafés can be stronger than within the big chains themselves
All reviewed cafés using Blue Bottle Coffee beans may have very different owners but score an even lower 0.70 standard deviation. These cafés may only share a bean supplier and some of standards for freshness and access to common consulting and training, but their espresso scores are significantly more consistent among each other than the cafés under a single Starbucks ownership — or even Peet’s.
Surely, a single quality dimension does not represent the breadth of possible flavor profiles, body weights, and crema textures that might also factor into a consistency analysis. But this data refutes the conventional wisdom that Starbucks, for example, provides a consistently dependable and uniform level of beverage quality. Even with their complete supply and delivery chain standardization, Starbucks fails to produce espresso quality that’s as consistent as a number of independent cafés that have only a coffee bean supplier in common.
Our data also suggests that the Starbucks brand is no better a determinant of quality consistency than whether the café uses a La Marzocco espresso machine or Delco cups. This is a critical point to understand, so let’s put this another way:
The Starbucks brand predicts consistency of espresso quality no better than if all the random cafés we surveyed that use Delco cups decided to re-brand themselves as “The Delco-branded Cups Coffeehouse chain.”
Given the investments Starbucks has made in coffee bean and roasting consistency, standardized push-button espresso machines, and standardized training, we don’t see how you can interpret this data as anything short of a complete failure for the company to deliver on the brand expectations of quality consistency.
If Starbucks’ consistency isn’t in their prepared coffee, then it is likely a psychological perception: the consumer’s brand association, the consistency of the Starbucks beverage menu, Starbucks’ own ridiculous names for drinks and their sizes, and the familiar environment of its coffee shops.
Seeking out chains in the hopes of managing risk
We recently came across a blog post by Chris Pirillo, a tech-geek/blogger/ex-TV-show-host who practically wet his underpants because he found a Peet’s Coffee & Tea chain store in his Sheraton Waikiki hotel. This when located in the very same building is an outlet of the Honolulu Coffee Company — which sources beans from Hawaii’s own Greenwell Farms, operates one of the few Mistral Triplette espresso machines in the world, and last scored as high as the best Peet’s we’ve ever been to anywhere. (Famed Baltimore coffee podcaster, Jay Caragay, disputes the quality there as dubious — but we stick by our last, albeit five-year-old, rating.)
Meanwhile, a friend in the Netherlands told us about a Starbucks that opened in Utrecht a week ago. When he passed by over the weekend, he said, “The line was unholy.” In a country where coffee shops legally sell marijuana by the menu, lines are out the door for one that peddles the double-tall, four-pump vanilla caramel macchiato.
What makes people seek out mass-produced mediocrity over independent, higher quality options just as conveniently nearby? What made an old boss of mine seek out a Pizza Hut for dinner when we were in the heart of London together on business travel with time to kill? These questions are woven into why we started CoffeeRatings.com six years ago.
In the case of coffee, you can largely dismiss the argument that chains provide a convenient shorthand to take out the risk in quality versus an unknown. As our data suggests here, big coffee chains can be less consistent among themselves than independent cafés with a common coffee supplier are to each other.
This is the part of the article where we’re supposed to have a snappy, revelatory answer to these questions. But we’ve got nothin’ — other than the consumer comfort with big coffee chains likely says more about their environment and sales practices than it does about the quality of the actual products they sell.
9 Comments »
[...] starts a post on TheShot, the blog for CoffeeRatings.com. With thousands of reviews of both chains and independent coffee [...]
“Myth of Consistency.” Enjoyed the insights.
I paused at the first paragraph, hung up on the third sentence. A McDonalds french fry is a McDonalds french fry… that’s true. It’s easy to shut off your brain and just assume that it’s the same every time.
But when I stop to really think about it, sometimes my french fries at McDonalds are disappointing. Sometimes they’re over-salted. Sometimes they’re clearly not just out of the fryer, and they’re limp and a little soggy. Sometimes they’re TOO hot, and I stop myself from popping it in my mouth immediately, lest I burn my mouth. Sometimes they’re not quite cooked as thoroughly as I tend to expect them. I know that others can relate.
People usually say, of big chains, “You know what to expect.” What you don’t necessarily know is exactly what you’re actually gonna get. It’s the expectations that have been homogenized.
Great information. There is definately a difference in espresso shots from one Starbuck’s to the next. I think, even with all their training and standardization, the technicians are still human and still have their own preferences. The major differences in taste are likely due to this fact as well as the flavor and quality of water going into the final product.
Thanks for sharing this data.
-Tommy
Your mathematical rigor applied to coffee is impressive. And satisfying precisely because it captures the point where I remember enough statistics to follow it but without leaving me behind
Thanks, numerodix — particularly as our cited intrepid “reporter” from our European desk in Utrecht.
I must say that Cindy and I were excited for this post, having spotted your reaction to our article. I have to say that even with my engineering background, your statistical approach is more or less sound and leaves me satisfied with your point. You leave me thinking about what causes the inconsistency, and I’m tempted to blame the difference in training that a barista receives at a Starbucks versus an independent coffee shop. I’m sure I don’t need to tell you all of the factors I highly skilled barista takes into account when preparing an espresso, and I am fairly certain Starbucks does not provide that level of training, especially considering the super-automatic espresso machines they currently use. So, while I am convinced of your point, that Starbucks does not serve a consistent shot, I still have my suspicions that they roast with the intention of consistency.
No question that Starbucks achieves consistency in their roasting and, perhaps more to the point, their bean sourcing. In fact, Starbucks bean sourcing consistency closes off any practical access to some of the best coffee bean growers in the world: namely, if a region cannot produce their coffee lots in a volume large enough to supply a blend sold at thousands of Starbucks cafés, Starbucks cannot carry them.
So if it’s not bean supplies and roasting, what might be at the root of their inconsistency? You could point to the barista training, but their Verismo and Mastrena machines are designed to be push-buttoned and idiot-proof. Milk-frothing and syrup-pouring are some of the only variants they have that isn’t machine-controlled, and our ratings are based on straight espresso shots.
This leaves equipment tuning, cleaning and maintenance as one of the biggest suspects for why Starbucks’ quality can be so erratic. You might also suspect the shelf-life, storage, and turnover of Starbucks’ coffee at each café as well.
As a former Starbucks worker, this sounds right:
“This leaves equipment tuning, cleaning and maintenance as one of the biggest suspects for why Starbucks’ quality can be so erratic. You might also suspect the shelf-life, storage, and turnover of Starbucks’ coffee at each café as well.”
The only time I saw people calibrating or getting inside the guts of the superautomatic espresso machines was when one of them broke. There was a sticker saying that each shot should take a certain length (forget the exact number of seconds) but I can’t remember that duration ever being checked. Maybe the manager did it when I wasn’t there but who knows.
We had high volume so you wouldn’t think that beans would sit around for very long, but unfinished 5-pound bags of beans were stored in cabinets behind the bar overnight with no special airtight containers.
“What makes people seek out mass-produced mediocrity over independent, higher quality options just as conveniently nearby?”
Well, in case of the Utrecht opening, maybe the coffee tradition/culture locally is different.
In my country (Indonesia), the prevailing so-called “coffee culture” are usually traditional and more grass-root based. For example in the northern most province in the Island of Sumatra, the Aceh province, the coffee tradition there is to get outside the house 11PM in the night to the nearest “warung kopi” (coffee street-hawker). There they would start chatting and socializing with lotsa people into the morning.
Well, no cafe included. Starbucks in Indonesia are mainly patronized by middle-to-upper class society who can afford a $2-4 glass of coffee. That’s a lot for a country where the same class/strata can eat a decent meal for their standard for less than $4, drink included. So a coffee that costs the same as the meal? Well it’s “elite”!