The Scoring Conundrum

[PBeattie]

The latest issue of Roast Magazine has a two part article on scoring that really gets my wheels turning. If you haven’t had a chance to read it here is a link:

http://www.roastmagazine.com/currentiss ... ciple.html

This article with Part 1 by Shawn Steiman and Part 2 by Ken Davids offers two very good perspectives on the use of the 100 point scale as it pertains to coffee.

In my mind what has always been a bit of a quandary is the separation, which both articles speak to, between the preferences of the consuming public and those of professionals. It seems to me that the majority of the scoring that takes place in coffee is done by Green Coffee buyers, and the position of Green Coffee buyer being usually the most sought after job in a roasting company, is usually filled by the most knowledgeable coffee professional.

Unfortunately, the most knowledgeable person in any profession is probably the far removed from the preferences and tenancies of the average consumer. For example (pardon the analogy) if you were to take a music professional they may be completely enraptured by the styling of John Coltrane and have no idea why or how somebody could really enjoy Kenny G.

I see this in coffee, and Ken David’s touches on it, that a particularly uneventful coffee to a professional that may score an 82 on the SCAA form can be the most delicious coffee to a consumer. What this coffee lacks in brightness, complexity, and and character to the professional, will be described by the consumer as smooth, mild, and comforting. And yet from a marketing standpoint, if that bag of coffee were to be emblazoned with a big sticker that said “Cup Score 82” that consumer most likely would never have purchased the coffee or had what they may describe as one of the best coffees they have ever had.

I find that green coffee buyers and roasters alike are much more likely to allow their knowledge of coffee producing and processing to impact their perception of what is “good” or deserves a high score. A great green coffee buyer knows the characteristics that a rare variety or high altitude or perfect processing will impart on the cup, and will therefore reward these characteristics with points.

I’ve often wondered, should a roasting company have two separate scoring systems for a coffee? One for and by the Green Coffee/Roasting departments and another that is awarded by Joe coffee drinker in the company. In this way what the Green Coffee buyer awards an 82, Joe may award a 91?

Thoughts?

[c.hallien]

I was interested to find at my previous position that the roasters that cupped with daily were very well calibrated and almost always scoring lower than the other cuppers on staff. My take was that the other departments, especially those involved in the promotion of the products, tended to see all the positives that the coffee had to offer meanwhile the roasting crew tended to look at what didn't happen in the roasting process, we discussed the influence of the roast and whether or not we could get more development with some modifications to the roast profile, change in airflow etc. The roasting department viewed the coffee as their product and tried to determine if they were successful in achieving the highest potential in development of flavors and intrinsic quality. We were very honest about the level of quality we were dealing with for each coffee and whether we nailed the roast or had more work to do...because it directly impacted our next steps. Measurements needed to be as accurate as possible in order to really understand if we made an improvement the next time.
"Answer the question being asked" ...a colleague always responds to me with this one and it needs to be considered. Dennis and I are 2 of 4 people who are active graders for CQI (certified Q-Graders) and The Exchange (licensed NYBOT/ICE graders). When I enter the cupping room at The Exchange there are no numeric assignments for individual attributes, no 100-pt scale to reference. When I sit down to cup for CQI or CoE it is a totally different approach, numbers are required and attributes considered with nuances making huge impacts on overall scores. I first learned to cup at a lab that provided outside independent analysis to our clients. Occassionally I rejected a coffee that I actually favored because it didn't meet the flavor profile of the product and consistency was the goal. Our cupping skills were for hire but our preferences weren't. In the world of sensory evaluation it is tremendously important to know what it is you are being asked to evaluate and why, is it a preference test, sample to standard, ranking, descriptive analysis or for the sake of calibration with a group of peers? I believe that there are a lot of really good cuppers in the industry but probably very few that are versatile enough, or perhaps willing to be, to take into consideration the question being asked to set aside their preferences when asked to do so and evaluate a coffee in the context that they are presented with.

[PBeattie]

Thought provoking, very C.

So the important thing is that before cupping it is determined what is in question with this coffee?

To support your point in reference to Roasters cupping, I have always felt that a given coffee would have a window of opportunity for score that within reason would be around 6-8 points depending on roast (assuming it isn't being butchered). So if the Coffee Buyer scores the sample an 85, the roasting staff may have a corrective score of anywhere from 81 to 88 based on the development of that coffee as it pertains to the roast profile.

I was just speaking to a friend of mine who has a past in the wine industry, he said that they face similar dilemmas as a particular vineyard may have amazing grapes (parallel with sample green coffee score), these grapes are sent to multiple wineries (parallel with roast score), then the wines are graded by wine enthusiast or wine spectator (parallel with a consumer score).

Does this mean that if the question before the cupping is not "What score will be published on the packaging?" that a score is really not necessary? After all if the question is consistency then taste profile, moisture, density, defects or lack there of, would be paramount and a score is theoretically useless.

So what are the most common pre-cupping questions?

"Does this fill the role within this particular blend?"

"Does this match taste profile for this particular origin within my lineup?"

"Is this a high scoring coffee?"

What else could be asked?

[c.hallien]

Case in point, the other day we were qualifiying a new roaster (machine). We had the "standard" which worked at as the reference sample and cupped a series of trial roasts against the standard. The question was not which was better but which was most like the standard. My actual preference was different from the standard but in this case we were trying to match an existing product and roast profile. Something we should all really consider sometimes...if we went with my preference for all products then we wouldn't have a dark roast in the house aside from some espresso blends that still wouldn't be that dark. Also what is preferred on the cupping table doesn't always translate to the application, this is true of sample roasts as well as production trial roasts. As a standard practice I would encourage everyone to trial roast their coffees, rank them in order of preference, then brew them in a variety of styles and do the same, I've had preferences change dramatically. Also these preferences change over time and a repeat cupping 1 week later may also see some changes. It is important to know the application of the coffee when evaluating it.

The industry is so large and the evaluation of coffee is done for any number of reasons. In the third party lab we used a number of sensory evaluation techniques such as difference from control, sample to standard, triangulation, preference and affective testing. This may sound sterile but in making major decisions for large brands there is a bit more involved than a couple of cuppers throwing out "86's" often the cupping incorporates statitics and repition to ensure consistency in the product (the hallmark for most large companies). The cuppings in my current position are
based on;
1. Green - purchasing and approving green coffee (the coffee being tendered against our contracts) relative quality against expectation and cost of the coffee. We use an internal rating sytem that approximates sample to standard but with a mental reference to quality expectations basis the origin/ time of year / current supply.
2. W.I.P - evaluating the roast to determine if the roasting process was sound. degree of roast and influence or roast and flavor development are as expected for each of the products.
3. Finished Product / Positive Release - has the integrity of the final product been maintained throughout the process. There are several stages in the roasting, packaging process where coffee quality can be comprimised. Prior to releasing any coffee we cup representaive samples to enure product integrity, basically a sign off on each of our products.

With specialty coffee we regularly used the 100 pt scale to have relative quality marker on each of the coffees. We plotted a trend line for each lot and used the scores to predict when coffee quality would decrease to a level that we would need to take some action; eithet review the roast profile or make blend adjustments. Some coffees hold up very well over time while other don't seem to hold up well for very long. Scoring the coffees numerically allowed us to use the information proactively in many cases as well as make adjustments to the purchases for the next year.

There's a lot going on in the mind when cupping ... I don't like a lot of distractions unless we're just cupping some interesting coffees