Monday, January 19, 2015

New York Fed on Forecasting: "What Does Disagreement Tell Us about Uncertainty?"

From the Federal Reserve Bank of New York's Liberty Street Economics blog:
Uncertainty is of considerable interest for understanding the behavior of individuals as well as the movements in key macroeconomic and financial variables. Despite its importance, direct measures of uncertainty aren’t widely available. Because of this data limitation, a common practice is to use survey-based measures of forecast dispersion—reflecting disagreement among respondents—to proxy for uncertainty. Is this a reliable practice? Here, we review the distinction between disagreement and uncertainty as concepts, and show that this conceptual distinction carries over to their empirical counterparts, suggesting that disagreement is not generally a good proxy for uncertainty. 
Blog-Disagreement-iStock_000017906875Large_450
Zarnowitz LambroseThe terms “disagreement” and “uncertainty” refer to very different concepts. Disagreement refers to a collection of forecasts or point predictions and the nature of their clustering around each other—the more disperse the forecasts, the greater the extent of disagreement among the survey respondents. On the other hand, uncertainty refers to the distribution of the probabilities that a respondent attaches to the different possible outcomes of the forecasted variable—the more confidence held by a respondent, the tighter this distribution is and the lower the respondent’s uncertainty.

        Although disagreement and uncertainty are different concepts, some commentators have drawn a connection between the two. That is, episodes characterized by high (low) disagreement are viewed as indicative of high (low) uncertainty shared by the respondents. This assumption provides the basis to use disagreement as a proxy for uncertainty when measures of the latter magnitude aren’t available.

        Is the assumed positive association between disagreement and uncertainty plausible? Yes. Is it necessarily true? No. To understand why either case is possible, we can look at the following figure previously discussed in this paper by Victor Zarnowitz and Louis Lambros:

        The illustration on the left shows the forecasts and associated probability distributions of two hypothetical survey respondents—respondent A and respondent B. The close proximity of the forecasts (ŷA and ŷB) indicates low disagreement, while the tight distributions around each forecast indicate low uncertainty. However, the illustration on the right depicts another possible situation. Here the forecasts are unchanged, so disagreement remains low. But the probability distributions around each forecast are now much wider, indicating high uncertainty. The figure is important for two reasons. First, it bears directly on the question of the reliability of disagreement as a proxy for uncertainty. While the conditions depicted in the left illustration might justify this practice, the conditions depicted in the right illustration would not. Second, it shows that the dispersion of forecasts by itself is not necessarily informative about the level of uncertainty across respondents. Because most surveys only report the respondents’ forecasts, observing the degree to which the forecasts cluster together can’t tell you whether the illustration on the left or on the right in the figure is the more relevant situation....MORE