Roasters Guild Makes Available Beta Version of Roast Evaluation Form

Share |

Postby lilyk » Fri May 01, 2015 9:39 am

Meet the new RG Roast Evaluation Form! Now you can evaluate your roasts with a tool designed for the specific purpose of looking at the impact of the roast on a coffee.

Download it here: http://roastersguild.org/PDF/Roast-Evaluation-Form.pdf

*Please note that this form is in the beta testing phase. If you have any feedback, please let us know in the comments below!

RGEC Roast Evaluation From Use and Protocol

Purpose

The Roasters Guild of the Specialty Coffee Association of America recommends these standards for evaluating roasted coffee using the Roast Evaluation Form. These guidelines will ensure the ability to most accurately assess the quality of the coffee.

Necessary Equipment
Clean, no interfering aromas
Cupping glasses with lids
Grinder
Cupping tables
Cupping spoons
Quiet
Hot water equipment
Comfortable temperature
Forms and other paperwork
Limited distractions (no phones, etc.)
Pencils and clipboards

Cupping Glasses | Cupping vessels shall be of tempered glass or ceramic material. They shall be between 7 and 9 fluid ounces (207 ml to 266 ml), with a top diameter of between 3 and 3.5 inches (76 - 89 mm). All cups used shall be of identical volume, dimensions and material of manufacture, and have lids.

Sample Preparation

When they reach room temperature (app. 75º F or 20º C), samples should be stored in airtight containers or non-permeable bags until cupping to minimize exposure to air and prevent contamination. Samples should be stored in a cool dark place, but not refrigerated or frozen.

Determining Measurements

The optimum ratio is 8.25 grams of coffee per 150 ml of water, as this conforms to the mid-point of the optimum balance recipes for the Golden Cup. Determine the volume of water in the selected cupping glass and adjust weight of coffee to this ratio within +/- .25 grams.

Cupping Preparation

Sample should be ground immediately prior to cupping, no more than 15 minutes before infusion with water. If this is not possible, samples should be covered and infused not more than 30 minutes after grinding.

Samples should be weighed out as WHOLE BEANS to the predetermined ratio (see above for ratio) for the appropriate cup fluid volume.

Grind particle size should be slightly coarser than typically used for paper filter drip brewing, with 70% to 75% of the particles passing through a U.S. Standard size 20 mesh sieve. At least 5 cups from each sample should be prepared to evaluate sample uniformity.

Each cup of sample should be ground by running a cleansing quantity of the sample through the grinder, and then grinding each cup's batch individually into the cupping glasses, ensuring that the whole and consistent quantity of sample gets deposited into each cup. A lid should be placed on each cup immediately after grinding.

Pouring

Water used for cupping should be clean and odor free, but not distilled or softened. Ideal Total Dissolve Solids are 125-175 ppm, but should not be less than 100 ppm or more than 250 ppm.

The water should be freshly drawn and brought to approximately 200º F (93ºC) at the time it is poured onto the ground coffee. *Temperature needs to be adjusted to elevation

The hot water should be poured directly onto the measured grounds to the rim of the cup, making sure to wet all of the grounds. The grounds to steep undisturbed for a period of 3-5 minutes before evaluation.

Sample Evaluation

Sensory testing is done for three reasons:

To determine the actual sensory differences between samples
To describe the flavor of samples
To determine preference of products

No one test can effectively address all of these, but they have common aspects. It is important for the evaluator to know the purpose of the test and how results will be used. The purpose of this roast evaluation protocol is the determination of the cupper’s perception of roast quality. The quality of specific flavor attributes is analyzed, and then drawing on the cupper’s previous experience, samples are rated on a numeric scale. The scores between samples can then be compared.

Coffees that receive higher scores should be noticeably better than coffees that receive lower scores.

The Cupping Form provides a means of recording important flavor attributes for coffee: Fragrance/Aroma, Flavor, Aftertaste, Acidity, Body, Sweetness, Balance, Overall, and Roasted Defects. The specific flavor attributes are positive scores of quality reflecting a judgment rating by the cupper; Defects are negative scores denoting unpleasant flavor sensations; the Overall score is based on the flavor experience of the individual cupper as a personal appraisal. These are rated on a 16-point scale representing levels of quality in quarter point increments between numeric values from 6 to 9. These levels are:

Quality Scale
6.00 – Good 7.00 - Very Good 8.00 – Excellent 9.00 - Outstanding
6.25 7.25 8.25 9.25
6.5 7.5 8.5 9.5
6.75 7.75 8.75 9.75

Theoretically the above scale ranges from a minimum value of 0 to a maximum value of 10 points. The lower end of the scale is below specialty grade.

Evaluation Procedure | Samples should first be visually inspected for roast color. This is marked on the sheet and may be used as a reference during the rating of specific flavor attributes. The sequence of rating each attribute is based on the flavor perception changes caused by decreasing temperature of the coffee as it cools:

Step #1 – Fragrance/Aroma
Within 15 minutes after samples have been ground, the dry fragrance of the samples should be evaluated by lifting the lid and sniffing the dry grounds.
After infusing with water, the crust is left unbroken for at least 3 minutes but not more than 5 minutes. Breaking of the crust is done by stirring 3 times, then allowing the foam to run down the back of the spoon while gently sniffing. The Fragrance/Aroma score is then marked on the basis of dry and wet evaluation.

Step #2 – Flavor, Aftertaste, Acidity, Body, Sweetness and Balance
When the sample has cooled to 160º F (71º C), in about 8-10 minutes from infusion, evaluation of the liquor should begin. The liquor is aspirated into the mouth in such a way as to cover as much area as possible, especially the tongue and upper palate. Because the retro nasal vapors are at their maximum intensity at these elevated temperatures, Flavor and Aftertaste are rated at this point.

As the coffee continues to cool (160º F - 140º F), the Acidity, Body, Sweetness, and Balance are rated next. Sweetness is the quality of the development of sugars from roasting that are present in the sample. Balance is the cupper’s assessment of how well the Flavor, Aftertaste, Acidity, and Body fit together in a synergistic combination.

The cupper’s preference for the different attributes is evaluated at several different temperatures (2 or 3 times) as the sample cools. To rate the sample on the 16-point scale, circle the appropriate tick-mark on the cupping form. If a change is made (if a sample gains or loses some of its perceived quality due to temperature changes), re-mark the horizontal scale and draw an arrow to indicate the direction of the final score.

Evaluation of the liquor should cease when the sample reaches 70º F (21º C) and the Overall score is determined by the cupper and given to the sample as “Cupper’s Points” based on ALL of the combined attributes.

Step #3 - Scoring
After evaluating the samples, all the scores are added as describe in the “Scoring” section below and the Total Score is written in the upper right hand box.

Individual Component Scores

The attribute score is recorded in the appropriate box on the Roast Evaluation form. On some of the positive attributes, there are two tick-mark scales. The vertical (up and down) scales are used to rank the intensity of the listed sensory component and are marked for the evaluator’s record. In the case of the vertical scale for the Sweetness category, the scale is used as a reference not as intensity, but as the development of sugars during roasting (from sweet grains, to candy, to chocolate). The horizontal (left to right) scales are used to rate the panelist’s perception of relative quality of the particular component based upon their perception of the sample and experiential understanding of quality.

Each of these attributes is described more fully as follows:

Fragrance/Aroma | The aromatic aspects include Fragrance (defined as the smell of the ground coffee when still dry) and Aroma (the smell of the coffee when infused with hot water). One can evaluate this at three distinct steps in the cupping process: (1) sniffing the grounds placed into the cup before pouring water onto the coffee; (2) sniffing the aromas released while breaking the crust; and (3) sniffing the aromas released as the coffee steeps. Specific aromas can be noted under “qualities” and the intensity of the dry, break, and wet aroma aspects noted on the 5-point vertical scales. The score finally given should reflect the preference of all three aspects of a sample’s Fragrance/Aroma. The score for the sample’s Fragrance/Aroma is divided by two (÷2) because it is an attribute more of the green coffee and less a product of roasting.

Flavor | Flavor represents the coffee's principal character, the "mid-range" notes, in between the first impressions given by the coffee's first aroma and acidity to its final aftertaste. It is a combined impression of all the gustatory (taste bud) sensations and retro-nasal aromas that go from the mouth to nose. The score given for Flavor should account for the intensity, quality and complexity of its combined taste and aroma, experienced when the coffee is slurped into the mouth vigorously so as to involve the entire palate in the evaluation. The score for the sample’s Flavor is divided by two (÷2) because it is a attribute more of the green coffee and less a product of roasting.


Aftertaste | Aftertaste is defined as the length of positive flavor (taste and aroma)
qualities emanating from the back of the palate and remaining after the coffee is
expectorated or swallowed. If the aftertaste were short or unpleasant, a lower score would be given.

Acidity | Acidity is often described as "brightness" when favorable or “sour” when unfavorable. At its best, acidity contributes to a coffee's liveliness, sweetness, and fresh- fruit character and is almost immediately experienced and evaluated when the coffee is first slurped into the mouth. Acidity that is overly intense or dominating may be unpleasant, however, and excessive acidity may not be appropriate to the flavor profile of the sample. The final score marked on the horizontal tick-mark scale should reflect the panelist’s perceived quality for the Acidity relative to the expected flavor profile based on origin characteristics and/or other factors (degree of roast, intended use, etc.). Coffees expected to be high in Acidity, such as a Kenya coffee, or coffees expected to be low in Acidity, such as a Sumatra coffee, can receive equally high preference scores although their intensity rankings will be quite different. The score for the sample’s Acidity is multiplied by two (x2) because how the coffee was roasted has a great effect on the perceived Acidity of the sample.

Body | The quality of Body is based upon the tactile feeling of the liquid in the mouth, especially as perceived between the tongue and roof of the mouth. Most samples with heavy Body may also receive a high score in terms of quality due to the presence of brew colloids and sucrose. Some samples with lighter Body may also have a pleasant feeling in the mouth, however. Coffees expected to be high in Body, such as a Sumatra coffee, or coffees expected to be low in Body, such as a Mexican coffee, can receive equally high preference scores although their intensity rankings will be quite different. The score for the sample’s Body is multiplied by two (x2) because how the coffee was roasted has a great effect on the perceived Body of the sample.

Sweetness | Sweetness refers to a pleasing fullness of flavor as well as any obvious sweetness and its perception is the result of the presence of certain carbohydrates. The opposite of sweetness in this context is sour, astringency or “green” flavors. This category is directly affected by how the coffee was roasted and how the sugars were developed during roasting and the score is multiplied by two (x2).

Balance | How all the various aspects of Flavor, Aftertaste, Acidity, Sweetness, and Body of the sample work together and complement or contrast to each other is Balance. If the sample is lacking in certain aroma or taste attributes or if some attributes are overpowering, the Balance score would be reduced.

Overall | The “overall” scoring aspect is meant to reflect the holistically integrated rating of the sample as perceived by the individual panelist. A sample with many highly pleasant aspects, but not quite “measuring up” would receive a lower rating. A coffee that met expectations as to its character and reflected particular origin flavor qualities would receive a high score. An exemplary example of preferred characteristics not fully reflected in the individual score of the individual attributes might receive an even higher score. This is the step where the panelists make their personal appraisal.

Defects | There are multiple Defects that may arise from the roasting process can affect the quality of the sample negatively. They are scored on a zero to five scale based on the intensity of the Defect as perceived by the cupper. A score of zero would mean that the Defect was not present and a score of five would indicate that the Defect was overwhelming the sample. These Defects are: Underdevelopment, Overdevelopment, Baked, Scorched, Tipped, and Faced.

Underdevelopment most often happens with coffee with high moisture content. Sometimes this is because it did not “rest” or “condition” in parchment for long enough at origin or the dry mill. The symptoms include being scorched on the outside, but underdeveloped on the inside. Noticeably reduced expansion may be visible on the surface. Tastes: aggressive acidity and flavors at the front of the palate with no finish or aftertaste.

Overdevelopment basically means that too much has been done to the coffee, and it implies there is nothing of interest left in the bean. All acidity and flavor have been muted.

Baking is usually caused by trying to extend the roast for too long at too low of a temperature (e.g. when you slow down the “ramping” of temperature too much). It can still look the same color but it takes too long to where you want it to go and caramelization has stalled. The baking defect tastes like popcorn, or hard cereal/oat flavors.

Scorching happens at the beginning of the roast and is when the green coffee is burnt before any enzymatic or sugar browning. Dark specks (burn marks) may be present on the green coffee before any other browning reactions have occurred, but wont be visible on the roasted coffee. Tastes burnt, smoky, and roasty.

Tipping is caused when the heat is applied to the bean at excessive speed.
The fast rate of heat causes the moisture in the bean to rapidly escape from the bean at its weakest point. The moisture blows out the end of the bean leaving a broken off and/or burned part at the end of the seam. It tastes like a burnt, over baked biscuit, and darker roasted may have skunkiness.

Facing is caused by two main reasons: One is an overloaded roaster and the beans have more exposure to conductive heat and the exposed surfaces that are in contact with the drum become burnt (faced). The other reason is an excessive amount of heat is applied to a roast and the outer surfaces of the bean are taking on heat faster than the inside can develop. This causes the outside of the bean to burn. Tastes burnt, roasty, smoky. Facing in a light roast will have these characteristics with the lighter roast flavor profile

Final Scoring
The Final Score is calculated by first summing the individual scores given for each of the primary attributes in the box marked “Total Score.” Defects are then subtracted from the “Total Score” to arrive at a “Final Score.” The following Scoring Key has proven to be a meaningful way to describe the range of coffee quality for the Final Score.

The Total Score Quality Classification will need to be revised after extensive testing of the Roast Evaluation Form.

Total Score Quality Classification
90-100 Outstanding
85-89.99 Excellent Specialty
80-84.99 Very Good
> 70.0 Below Specialty Quality Not Specialty ? (This is very open to discussion)
lilyk
 
Posts: 98
Joined: Fri Feb 26, 2010 12:53 pm
First Name: Lily
Last Name: Kubota
City: Long Beach
State: CA
Zip Code: 90802
Company: SCAA

Postby N3Roaster » Tue Jul 14, 2015 12:41 pm

I've just tried using this form to evaluate 6 different roasts of a coffee as part of a little informal experiment, so I have some thoughts on this. Bear in mind that this is coming from having only that small amount of experience with the form. I'll be continuing with this little experiment and using this form throughout so it's possible that my views on this might change a bit with further experience.

The first thing that comes to mind looking at this is that it's neat and potentially useful, but the second thing that comes to mind are the questions, "what problem does this really solve?" and "where would it make sense to use this?" The purpose states that it's for evaluating roasted coffee, but to what end? It's clear that this isn't for green evaluation and isn't intended as something to replace the SCAA cupping form. Is it useful for product development? Maybe. It seems a bit too heavy for that and doesn't seem to be a good fit for the way I do that, but it's plausible. Is it for quality assurance? There I'm much less interested in the quality of the roast and more interested in the question of if the sensory qualities match the product specification. It might provide a little more information on if the spec should be changed, but here again it doesn't seem necessary or a particularly good fit. What about customer facing applications? I don't think anybody is looking at this and thinking that's a good idea. Training applications? This might be suitable as a replacement for the simplified cupping form in RP112/120. If the instructors feel comfortable with that it might be worth trying at retreat and seeing how the feedback there is. It seems like a very good fit for well controlled roasting experiments, and this is how I'm currently attempting to use it. Maybe competition judging? There are some challenges there as so much of the scoring is relative to expectations for the coffee, it would seem difficult to do a credible blind evaluation without being overwhelmed by mere preference for particular origin characteristics that aren't strictly a function of roast quality if the coffee used is not kept constant. If there is only one coffee involved then it might work well there. I think there needs to be a lot more guidance on what kind of work flow it's envisioned that this would be useful in because in terms of utility in day to day operations to me at least this looks like a solution in search of a problem. I'd like to be convinced otherwise.

Scores with this form come out lower than I would expect. This might be a matter of training and calibration, but giving category scores in the 7s and 8s (which according to the instructions should be "good" and "very good" produces a total score in the 70s, so the total score quality classification seems overly compressed at the top. In these first 6 cups only one coffee got over 80 (84.75) with scores mainly in the 8-9 range for the categories (which, remember, is in the "very good" to "excellent" range) and I feel that sample really ought to have been considered excellent. Perhaps "Overall" is under-valued, but my gut reaction is that that's a bad way to fix this. The scoring disconnect compared with green evaluation scoring seems to mainly be because on the SCAA cupping form there are three categories that are, for most specialty coffees I've tasted, basically automatic 10s. There isn't anything like that here. Maybe people need to skew scores higher when using this form, but in a lot of companies it's the same people doing green and finished product evaluations so expecting that seems counter-productive. A better solution is to extend what's considered good in the total score interpretation to better reflect what the form is asking. Maybe 90-100 outstanding, 80-89.875 excellent, 70-79.875 very good. That seems to line up better comparing my impression of the coffees with the total scores.

I won't comment here about typos in the instructions as I've read an updated version on the RGCC Trello board and some desired corrections there aside from noting that the version of the instructions there puts the cutoff for specialty at 80, leading me to believe that the 70 here is a typo and not the alternate interpretation that 70-79.875 should be considered good.
Neal
N3Roaster
 
Posts: 66
Joined: Tue Mar 16, 2010 4:30 pm
First Name: Neal
Last Name: Wilson
City: Racine
State: WI
Zip Code: 53405
Company: Wilson's Coffee & Tea
Occupation: Roastmaster
Roasting Since (Year): 2000
Website: https://typica.us
Location: Racine, WI
Twitter: N3Roaster
YouTube: N3Roaster


Return to “%s” What's New?

Who is online

Users browsing this forum: No registered users and 1 guest

cron