In Defense of Meteorologists: Part 2
By Steve Tracton
In my post last week, I commented upon the misleading (and a few flat-out wrong) statements about weather forecasting that appeared in a post on the New York Times Freakonomics blog. These statements were made as part of a study by a father and his fifth-grade daughter of temperature and precipitation forecasts by Kansas City broadcast meteorologists and the National Weather Service.
The study was done with good intentions and apparent rigor, yet there were significant flaws that made the results and conclusions highly questionable. Furthermore, the post included comments by some of the broadcasters and their station managers that were exaggerated, easily misconstrued or invalid, and were generalized to erroneously describe all weather forecasters and the science of meteorology in general.
Let's take a closer look at some of this misinformation, the most glaring of which was this comment credited to one of the TV weathercasters: "We have no idea what's going to happen (in the weather) beyond three days out."
Keep reading for further defense of meteorologists. See our full forecast through the weekend, and UnitedCast for the forecast for tonight's game at RFK.
Any true assessment of forecast skill must take into account what is being predicted (e.g., temperature, clouds, precipitation), location and scale.
You may have heard the mantra that today's five-day forecasts are as good as three-day forecasts were about 20 years ago. This is generally true, for example, for temperatures and winter storms. And in fact, predictability of larger-scale features of the atmosphere -- such as jet stream position and location of low-pressure and high-pressure areas in the middle to upper atmosphere -- might eventually extend to two weeks or so (from five to seven days currently).
But for smaller-scale features such as summertime showers, thunderstorms and associated rainfall amounts, predictability likely will never exceed 12-24 hours (from around a few to 12 hours now).
Probably the only thing certain about weather forecasting (and other fields, too, such as economics) is that there will always be some uncertainty. The degree of uncertainty may range from being too small to make a difference for most purposes to so large that it renders a forecast no more valuable than a random number generator.
Too often, however, forecasts are provided as "deterministic," single-valued quantities, such as a predicted high temperature of 75 degrees for seven days from now, without any measure of forecast confidence (CWG being a notable exception, of course!) or range of possible error (e.g., +/- 3°).
The uncertainty at longer ranges is usually larger (wider envelope of possible scenarios, akin to the "cone of uncertainty" used in hurricane track forecasts) than for the shorter range (narrower envelop). Generally speaking, the larger (smaller) the uncertainty, the less (more) likely a particular scenario within the envelope will be correct. Thus, it should not be surprising that a single-value forecast several (five to seven) days ahead will usually differ and be less accurate than one closer (two to three days) to the time in question, as was noted in the study posted on Freakonomics.
What's most important, though, is not whether a deterministic, longer-range forecast changes with time - which it almost invariably will - but whether the forecaster communicates the confidence associated with a prediction.
Forecast confidence would be easier to explain if it only depended on lead time. However, chaos theory, or shall we say the butterflies or perhaps an allergy-induced sneeze -- can shake things up in unexpected ways, such that sometimes a seven-day forecast can be as good as a three-day forecast, or, conversely, a three-day forecast may be as poor as a seven-day forecast. And the relative confidence at any given lead time can vary significantly from one weather scenario to another (for additional insight, refer to my presentation summarized in the latest D.C. Chapter of the American Meteorological Society newsletter.
The scenario-dependent level of confidence can be estimated based on the comparison of output from different forecast models, forecaster experience and ensemble predictions, which are multiple runs of the same forecast model using slightly different initial conditions in each run. Naturally, one hopes the confidence level applied to forecasts are reliable -- that is, higher-confidence forecasts are more likely to verify well as compared to lower -- confidence forecasts.
So, what about these two conclusions from the Freakonomics post?:
"No forecaster is ever better than just assuming it won't rain."
"For all days beyond the next day out, viewers would be better off flipping a coin to predict rainfall."
Sure, the former may be true when forecasting for San Diego during the dry season, when one can be close to 100-percent confident in predicting no rain day after day, regardless of whether the forecast is for tomorrow or for seven days from now. However, one could certainly not get away with such a forecast for D.C. (or Kansas City). Predicting precipitation in these locations, especially in summer, is one of the most difficult challenges encountered by forecasters.
In acknowledgment of this difficulty, precipitation forecasts have long been expressed in probabilistic terms. Used properly, probabilities retain more skill than a deterministic yes/no forecast. As for predicting rainfall by flipping a coin, it turns out that forecasting the climatological probability (about 30% in the D.C. area) of precipitation, even when confidence is low, is a much better approach than the yes/no forecast given by a coin toss. Beware, though, a 30 percent chance of rain is intended to mean there is a 30 percent chance that a given location will receive .01" or more of precipitation. This is a pretty low threshold, which could be produced by anything ranging from a period of light drizzle to a storm that drops several inches of rain in a couple hours.
Bottom line: From the perspective of both research and application, meteorology is one of the most challenging of all sciences. We've come a long way, but are still far from the understanding and modeling capabilities necessary for coming even close to the ultimate objective of practical forecast skill reaching theoretical limitations. Forecasts are not now, nor will they ever be, perfect in all respects. But even an imperfect forecast can be of significant value to users if accompanied by appropriate confidence and uncertainty information.
The fact of the matter is, current capabilities of weather forecasting can't be accurately represented by the kind of blanket statements and conclusions put forth by the Freakonomics weather study. While researchers and forecasters alike readily acknowledge there is much still to learn, we should not underestimate our increasing knowledge of the atmosphere, nor undermine the credibility of the vast majority of professional forecasters.
Forecasters can do their part by characterizing and communicating forecast confidence and uncertainty using all of the information and technology at their disposal. Meanwhile, hopefully the average weather forecast consumer is able to discriminate between erroneous and exaggerated pronouncements versus realistic expectations and limitations of weather forecasts.
Such is essential for maximizing the value of forecasts in making weather-dependent decisions, including whether to wrap their kids up in rain gear, planning a weekend outing, or preparing for potential snowstorm or hurricane.
The author is chair of the D.C. Chapter of the American Meteorological Society, and is working on a book about the source and nature of weather forecast uncertainty, and how users can optimize the value of forecasts by factoring uncertainty information into risk analysis and decision-making.
By Capital Weather Gang |
May 8, 2008; 11:15 AM ET
Science
Previous: UnitedCast: Cloudy With Thundershowers Possible |
Next: CommuteCast: Flooding Rains Possible Overnight
Posted by: Ryan | May 8, 2008 11:55 AM
Perhaps if weathermen/women could get the basics right, folks wouldn't be so tough on them. When I see today's high listed at 75 and it's currently 79, it doesn't take a rocket scientist to realize somebody's not trying very hard. Nor do I understand a forecast of "possible rain today" when it's been pouring outside for hours.
Posted by: Don | May 8, 2008 12:00 PM
Don -- Hmmm ... not sure where you get 79 from. Currently it's actually 67, 66, 71 at DCA, IAD and BWI, respectively. And so far most area reporting stations have reported nothing more than light rain (sometimes just cloudy) so far today.
Posted by: Capital Weather Gang | May 8, 2008 12:11 PM
I wasn't talking about today specifically regarding the temp. or the rain. It's just a general observation I've made several times. I have also observed forecasts that give one days's temp range as 40-50 and the next day 60-70 (again, the numbers aren't exact, just an example), which would actually be impossible.
Posted by: Don | May 8, 2008 12:39 PM
For example, if you go to weather.com and search for the current weather in Oak Ridge, TN, it shows a current temp as 77 and today's high as 75. How does that happen?
Posted by: Don | May 8, 2008 12:43 PM
In addition to the time constraints Ryan mentioned, people, in general, have a very poor understanding probabilities and uncertainties to begin with so I'm not sure added them to a forecast would be all that helpful.
Posted by: John | May 8, 2008 1:22 PM
A high of 40-50 followed by a high of 60-70 is not impossible.
Posted by: Steve, Capital Weather Gang | May 8, 2008 2:05 PM
Steve,
You are absolutely right. However, if you reread my post, you'll see I didn't say a "high" range, I said a temperature range, which would be impossible. If the high the first day is 50, the entire temperature range the next day can not be 60-70. The temperature must pass through the 50's to get to 60, which is outside the range of both forecasts.
Posted by: Don | May 8, 2008 2:18 PM
I think we just showed that what is perceived is not always the same as what was forecast.
Posted by: Steve, Capital Weather Gang | May 8, 2008 3:13 PM
Don, I think it's a mistake to assume people in general have difficulty in understanding probabilities and uncertainties. The real issue is how to the information is conveyed by forecasters and interpreted by people in the context of weather related decisions they encounter. How to deal with unceratinties in weather forecasts is currently at the forefront of attention by the National Weather Service and American Meteorological Society. The most effective approach is highly user dependent. For example the need and requirements for the public, emergency managers, agriculture, avaiation, etc. can be very different.
Aside from economics where uncertainty reigns (as mentioned in the posting), an area most people encounter uncertainty is in medicine. See my article in the latest DC-AMS Newsletter (http://www.dc-ams.org/
newsletters/07-08/
MarApr08Newsletter.pdf)
Posted by: Steve Tracton | May 8, 2008 3:26 PM
Steve Tracton gives a excellent, useful cogent discusssion and answers to the tired cliches and plain errors and misinformation as in the original Freakonomics post and responses and other "rants about the weatherman" for lack of a better term. Some observations of my own.
1. No matter how much the science and predictability advances, people's expectations will continue to outpace what is possible. A 24 hour forecast accurate close to 90% of the time. . .we want 95% accuracy. 95% accuracy in 10 years. . .we want 99% accuracy. Talk to your parents or grandparents about how accurate forecasts (and more importantly tornado and hurricane warnings) were 40-70 years ago.
2. People do get probabilities. We conducted an on-line survey a few years ago with sample graphics asking if including the PoP would help with viewers weather related decisions. Of 1000+ responses 95% wanted to see the PoP. In another question survey with 4 multiple choice questions 70% of respondents knew the correct definition/usage of PoP. When Howie Mandel talks about probabilities on "Deal or No Deal" . . .people get it. We find most everyone likes to see the PoP on our graphics. . .people get it.
3. Don't confuse weathercasters or weather presenters with meteorologists. The comment, "We have no idea what will happen beyond 3 days. . ." might be attributed to a "meteorologist" but the statement only indicates igorance of the science and the advances of the science. Many weather presenters have solid training in meteorology and overall I believe the U.S. public is well served by excellent weather presentations. But the outrageous quote will often get the print as they say.
4. In response to "Ryan" (no relation) we (NBC4) purposely show the weather icon graphics as 2 items, a 1-4 day "forecast" and a day 5-7 "outlook" with only general numbers (50s, 70s etc.) to show the uncertainty in an extended outlook. When the skill threshold was shown to be out to day 7 many years ago, we first showed only descriptive elements (colder, warm, humid etc.) and no numbers at all. Again to communicate the uncertainty. No station or news manager has ever directed what we can and can not show in our weather presentation. Spirited discussions and compromises . . .sure. Directives "You will do this or else". . .never.
5. Finally (at least for now) we should not be consumed with evaluating a forecast as which is "most accurate". How do you evaluate a partly sunny/partly cloudy icon for example. Or if forecast X is 1 degree more "accurate" than forecast Y does it really matter?
Rather, the generation, communication and user decision of a weather forecast should be how we look at what should be a continuous process. A 100% "accurate" forecast that results in a poor decision because of ineffective communication is a useless forecast. If no one retains what I have just said about tomorrow's weather, I haven't done my job and I get fired . . .just not tomorrow I hope. Effective communication of the forecast and the uncertainity, which may include forecaster confidence, should enable users to make the best weather related decision. That should be our common goal whether it is a forecast for the next hour, the next day or the next 50 years.
Bob Ryan
Chief Meteorologist
NBC4
Posted by: Bob Ryan | May 8, 2008 8:35 PM
Bob: Thanks for your response! NBC4 is one of the only TV stations I've seen split the 7-day as you do, or depart from the standard deterministic long-range forecast in any way. It's encouraging and I'm glad you've chosen to stand up for the science.
Posted by: Ryan | May 8, 2008 9:20 PM
Bob - thanks for the information. I would be interested in how the participants in your survey were selected? And how representative of the general population? For example, other research indicates problems with correctly applying probabilities in their decision making. It could be multiple choice definition/usage of PoP is sufficiently different than other decision making tasks to account for such good performance. If you want to look at another game show problem to compare to Deal or No Deal - look at the research around "The Monty Hall Problem". TMHP is a classic example of people applying probabilities incorrectly.
Posted by: John | May 8, 2008 10:17 PM
Thanks, Bob, for your great set of comments - Right On!
For the record Bob is at the forefront of recognizing, promoting and otherwise being pro-active in stressing the importance and value of including information on uncertainties in weather forecasts. Bob was a major contributor to the National Academy of Science publication, "Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts" (http://books.nap.edu/openbook.php?record_id=11699&page=R1
- note that there is no charge to download the PDF version)
Posted by: Steve Tracton | May 9, 2008 8:34 AM
John,
Our recent surveys were done on-line . . .so certainly not completely randon but we did do some other more random surveys years ago, some may remember. The point is I think most people do get PoPs if not the 100% exact correct definition (some meteorologists get it wrong) but almost everyone felt it helpful to them in making a weather related decision and that is part of what the forecasting "process" should be. A process that is forecast, communication and decison. Bob Ryan
Posted by: Bob Ryan | May 9, 2008 10:26 PM
Steve - thanks for the link to that report. Very interesting reading. Section 2 on Uncertainty in Decision Making and its recommendations regarding the various needs, wants and processes of users was excellent.
Bob I agree with you completely about the whole "process" as being important. And the generalized process of computation-communication-decision is the key in many fields not just forecasting.
Posted by: John | May 10, 2008 9:43 PM
Post a Comment
We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.
User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.
I'm sure a lot of TV mets would love to add an indication of uncertainty in days 3-7 in their 7-day forecasts but would quickly be overruled by news directors and other higher-ups. Until this changes, I think blogs and other forms of non-traditional communication can serve the purpose to let viewers know about these uncertainties and other details of the forecast that they can't get across in their 3 minutes of alloted time.
BTW, the Freakonomics blog is an excellent read and is highly recommended. It is part of my daily blog-reading regimen.