Tag Archives: wine judging

Wine competitions, judging, and blind luck

Wine competitions, judging, and blind luckOr, as the co-author of a new study told me: “Consumers should disregard results from wine competitions, because it’s a matter of luck whether a wine gets a gold medal.”

That’s the conclusion of Robert Hodgson, a winemaker and statistician whose paper (written with SMU’s Jing Cao) is called “Criteria for Accrediting Expert Wine Judges” and appears in the current issue of The Journal of Wine Economics. It says that those of us who judge wine competitions, including some of the world’s best-known wine experts, are ordinary at best. And most of us aren’t ordinary.


… [M]any judges who fail the test have vast professional experience in the wine industry. This leads to us to question the basic premise that experts are able to provide consistent evaluations in wine competitions and, hence, that wine competitions do not provide reliable recommendations of wine quality.

The report is the culmination of research started at the California State Fair wine competition at the end of the last decade. The competition’s organizers wanted to see if judging was consistent; that is, did the same wine receive the same medal from the same judge if the judge tasted it more than once during the event? The initial results, which showed that there was little consistency, were confirmed in the current study.

More than confirmed, actually. Just two of the 37 judges who worked the competition in 2010, 2011, and 2012 met the study’s criteria to be an expert; that is, that they gave the same wine the same medal (within statistical variation) each time they tasted it. Even more amazing, 17 of the 37 were so inconsistent that their ratings were statistically meaningless. In other words, presented with Picasso’s Guernica, most of the judges would have given a masterpiece of 20th century art three different medals if they saw it three different times.

“This is not a reflection on the judges as people, and I don’t mean that kind of criticism,” says Hodgson. “But the task assigned them as wine judges was beyond their capabilities.”

Which, given the nature of wine competitions, makes more sense than many doubters want to believe. Could the problem be with the system, and not the judges? Is it possible to be consistent when judges taste 100 wines day? Or when they taste flight after flight of something like zinfandel, which is notoriously difficult to judge under the best circumstances?

When I asked him this, Hodgson agreed, but added: “But we don’t see an alternative. But it is an inherent problem. You just want to see the competitions give the judges sufficient time to do it.”

Perhaps. But my experience, after a decade of judging regularly, is that the results seem better (allowing for this um-mathematical approach) when I judge fewer wines. That means that the competition is smaller, or that the organizers have hired more judges. Maybe that’s where the next line of study should go, determining if judging fewer wines leads to more consistent results.

Colorado Governor’s Cup 2014

Colorado Governor's Cup 2014Ten years ago, when I first tasted Colorado wine, I spent much of my time being polite. As in, “This is nice. Thank you for letting me taste it.”

Those days are long gone, as was amply demonstrated last weekend during judging for the fifth annual Colorado Governor’s Cup. The red wines were exceptionally strong, and though the whites weren’t as good, they were technically sound and professionally made. In the regional wine business, that’s an accomplishment.

The best reds were cabernet franc and petit verdot, two Bordeaux grapes that do well in Colorado and that the state’s winemakers have taken to with enthusiasm (and especially cab franc). My panel gave a gold and double gold to cab francs, and a gold to a petit verdot. And the best wine of the competition was a petit verdot, from Canyon Wind Cellars. The results are here.

The wines were varietally correct, but also distinctive and reflected Colorado’s terroir — not a lot of fruit, more dry than a California wine, yet complex and very long. This is not an easy style of wine to make, but the state’s winemakers have made great progress figuring out how to work with their terroir over the past decade.

Finally, a few words about my pal Doug Caskey, who oversees the Colorado Wine Board and has run the competition since it started. One reason I enjoy judging this event so much is that Doug brings together judges who understand that Colorado wine isn’t French wine or California wine and isn’t supposed to taste like it came from those places. Sadly, too many judges downgrade wines that are “different,” which has nothing to do with quality, but with a preconceived notion about what wine is supposed to taste like that borders on snobbery and elitism.

The two people on my panel, Tynan Szvetecz and Sarah Moore, were terrific in this respect, and it was a pleasure to judge with them. I’m always lucky to work with people who put up with my idiosyncrasies, and they were no exception.

Critics Challenge 2013

Critics Challenge 2013Two things stand out after judging the 10th annual version of this competition last weekend: First, that local wine has come a long way when it enters — let alone does well at — a top-notch California event like this. Second, that the challenge’s unique format is not only a tremendous amount of fun, but offers one possible solution for all of the handwringing about inconsistent competition judging and results.

More, after the jump:

Continue reading

Lone Star International Wine Competition 2013

We saw the future of Texas wine during one of the championship flights at the 30th annual competition this week, where there wasn ?t a chardonnay in sight. The five wines competing for best white from Texas were all outstanding, and each was worthy of winning — two Rhone blends, a viognier, a roussanne, and an albarino.

Can I write, finally and after 20 years, that Texas producers and growers have figured this thing out? More, after the jump:

Continue reading

The Dallas Morning News and TexSom Wine Competition 2013

Midway during the second day of judging at this year ?s event, one of the other judges smiled at me and said, ?This must make you feel good. ?

And it did. My panel was judging red wines from the United States made in places that weren ?t California, Oregon, and Washington. And they had been ? dare I say? ? spectacular. We gave five silvers and a gold; the day before, we had awarded gold medals to chardonnays from Idaho’s Pend d’Oreille and Michigan.

The other judge, Paul Lukacs of Baltimore, knew about my work with DrinkLocalWine and my enthusiasm for regional producers. And, as someone who judges regularly, he also knew the quality of these wines can be uneven. And while we had some clunkers during the two-day event (we judged all regional wine), the overall quality was as good ? if not better ? than at any competition I have judged. There was even a nifty red from South Dakota’s Belle Joli made with the marquette grape. More, after the jump:

Continue reading

Indy International 2012

Let's get the apologies out of the way first. First, to Bernie Parker of Indiana's Oliver Winery, who was the fourth judge on our panel at last week's 21st annual competition. I'm afraid he had to listen to the Wine Curmudgeon bellyache more than once about the quality of the wine we were judging, which is something no one should have to put up with as often as he had to put up with it.

Second, to two wines from our group that probably deserved double golds. They were an auxerrois from Canada's Gehringer Brothers, and a sparkling wine made with frontenac gris from Four Brothers in Minnesota, and each was better made than similar wines we tasted during the best of show taste-offs. As noted here before, judging is not an exact science, and these wines reminded me of that. More, after the jump:

Continue reading

Off to Indy

I ?m judging the 21st annual Indy International Wine Competition this week, so I might be a little slow answering emails and moderating comments (which is still on since TypePad ?s success in blocking the spambots has been intermittent at best).

I enjoy judging Indy because the people are top-notch, and I get to see old pals every year; many of the grapes used to make the we judge wine are just as odd as I am; and it ?s probably the premier regional wine competition in the world (though, as I am always reminded by the organizers, it ?s much more than just a regional competition).

I ?ll have a full report next week when I get back. Of course, I will be tweeting. Sigh.