Skill vs. Luck in Wall Street

This value may arise from analyst earnings forecasts, which are a major input into analyst stock recommendations.[2] [3] Loh and Mian find that investors who follow the investment advice (i.e., buy/sell recommendations) of the most accurate earnings forecasters earn abnormal returns.[4] This contemporaneous relation is motivated by theories of fundamental analysis (e.g., Ohlson[5])—if the analyst’s estimates of future earnings (inputs into a valuation model) are better, the analyst can potentially translate them into better target prices or stock recommendations (valuation outputs).

For the contemporaneous result to hold trivially in the predictive setting, however, forecast accuracy must be persistent—the impact of skill vs. luck.[6] In this study we seek to gain a better understanding of the persistence in the relative earnings forecast accuracy of Wall Street All Star analysts over time and whether it is related to future stock recommendation profitability. In other words, we are interested whether it is possible for individual investors to identify in advance analysts who are truly skilled and not just lucky.

A Measure Of Stock Picking Skill

To measure a stock picking skill, we develop a two-step process. We begin by calculating a measure of analyst forecast accuracy for all analyst-firm-year observations in our sample. A novel feature of our accuracy measure is that we control for forecasting complexity—the dispersion in analyst beliefs about a firm’s future earnings. We argue that if forecasting complexity was not controlled for in this manner, analysts covering firms whose earnings are more predictable would be classified as accurate, while analysts covering firms with less predictable earnings would be classified as inaccurate. In short, we would be following the recommendations of analysts covering firms with predictable earnings, rather than following the recommendations of accurate forecasters across all firms. Our measure, analyst i’s forecast accuracy of firm j’s earnings per share (EPS) for period t, Accuracy_i,j,t., is defined as:

where Actual_j,t is firm j’s actual EPS for period t, Forecast_i,j,t is analyst i’s forecast of firm j’s EPS for period t, abs is the absolute value operator, and stdev_j,t [Forecast_i,j,t)] is the standard deviation analyst one-year ahead earnings estimates of firm j for the fiscal period t t.[7]

Next, for each analyst i in each year t, we compute the mean value of all forecast errors across all firms covered by the analyst, denoted by Accuracy_Mean_i,t. That is, we focus not on analyst predictions for a particular firm’s earnings, but rather on the analyst-level data. This distinction between firm-level and analyst-level accuracy is crucial. An appropriate question for a portfolio manager to ask may be: “given this analyst did a great job for me on one company, should I trust him on other names?” Thus, for example, if analyst Joe’s 2005 EPS forecasts for IBM, Microsoft, and Cisco are 2, 5, and 0, and the respective actual EPS are 1, 2, and 9, then Accuracy_Mean_Joe,2005 equals 1.17 (= (abs(1-2)/1 + abs(2-5)/2 + abs(9-0)/9)/3 = (1+1.5+1)/3).

The aggregate set of analyst Accuracy_Mean_i,t results per year is then ranked from lowest to highest in terms of accuracy. Specifically, we sort all Accuracy_Mean_i,t observations into five portfolios (i.e. sets) such that portfolio (1) includes observations with the lowest Accuracy_Mean_i,t, i.e., the most accurate analysts, and portfolio (5) includes observations with the highest Accuracy_Mean_i,t, i.e., the least accurate analysts. Note that we require that each analyst covers at least three firms per year so that the mean and median calculations are meaningful. We also require three years of mean information for each analyst so that the time-series tests are meaningful. With the above sorts in hand, we can analyze analysts’ relative forecast accuracy.

We obtain analyst All-Star rankings from the annual Institutional Investor Survey. Analysts’ annual earnings forecasts, matching firms’ actual earnings, and the statistical measure of earnings forecast dispersion from the unadjusted Institutional Brokers Estimate System (I/B/E/S) Detail Files. Stock price data is from the Center for Research in Security Prices (CRSP). Our final sample includes 2,034 analyst-year observations over the period 1987-2012.[8]

Spotting a Winner

Figure 1 plots the transition probabilities across portfolio quintiles of relative forecasting accuracy after controlling for forecasting complexity.[9] The current- and subsequent-period ranks are represented by the horizontal axes, and the probability of migration is represented by the vertical axis. The area above the diagonal represents those analysts with the highest persistence in relative forecasting accuracy. The flat shape of the curve, especially the relatively flat area above the diagonal, clearly demonstrates that relative forecasting accuracy is only weakly persistent over time, that is, superior analyst forecast accuracy in the current period does not necessarily mean that this analyst will continue to outperform other analysts in future periods.

A possible explanation for our finding is that analysts’ contribution to financial markets goes beyond what empiricists can observe. The Institutional Investor survey shows that institutions do not care much about either earnings forecast accuracy or stock recommendation profitability when evaluating sell-side analysts. The number one item that institutions value in analysts is always industry and company knowledge. Thus, analysts that show persistence in the relative forecast accuracy, after controlling for forecasting complexity, may expand greater effort to understand the companies’ industries companies, and their future prospects.

Figure 1: Transition Diagram between Current and Future Quintile Ranking

In the next section we show that this reduced persistence, however, measures true forecasting ability. In other words, in the short-term analysts may experience good or bad luck (and that can overwhelm skill), but in the long-term luck tends to even out and skill determines results.

Performance of Skill vs. Luck

For the five accuracy portfolios we calculate average recommendations for each stock followed by the analysts in each portfolio. Our portfolios are long stocks in which the consensus recommendation is favorable and short stocks in which the recommendation is unfavorable. Thus, we report the returns of long–short portfolios, which are essentially zero-investment portfolios formed by buying the long portfolio and selling short the short portfolio of each analyst group.

Due to the skewed distribution of analysts’ recommendations, stocks are included in the favorable (long) portfolio when their average recommendation is greater than or equal to four (i.e. a buy recommendation or better). Stocks are included in the unfavorable (short) portfolio if the average recommendation is less than or equal to 3.5 (i.e. between a hold and underperform or worse). We exclude all stocks with an average rating that is greater than 3.5 but less than 4. Recommendations are used until they are reiterated/revised or become stale (i.e. maximum life of 6 months).

This portfolio formation process is conducted prior to the first trading day in year t and the portfolio is only rebalanced when changes in average firm recommendations, caused by the exclusion of stale recommendations or the issuance of new recommendations, moves a firm in or out of the favorable (long) or unfavorable (short) portfolios. Portfolio value-weighted returns are then calculated at the end of each trading day and compounded to monthly returns. The returns to these portfolios are adjusted for four risk factors: the market factor, the size and book-to-market factors proposed by Fama and French,[10] as well as the Carhart[11] momentum factor.

Figure 2 depicts the combined effect of the quality, alphas, of favorable and unfavorable recommendations of each persistent group after controlling for persistence in the relative forecast accuracy and after controlling for forecasting complexity. The figure shows that the high accuracy persistence portfolio produces a positive four-factor-adjusted return of 0.15 percent (p-value < 0.0001) per month. In contrast, the low accuracy persistence portfolio earns factor-adjusted returns of –0.23 percent (p-value < 0.0001), implying analysts are more likely to be inferior forecasters than superior forecasters.

Moreover, one could formulate a hedged investment strategy by buying the high accuracy portfolio and shorting the low accuracy portfolio to earn a monthly return of 0.38 percent (p-value < 0.0001). This monthly abnormal return is equivalent to a compounded annual abnormal return of 4.7 percent. To highlight the importance of our results we also report returns investors would earn by simply following the recommendations of analysts who were most accurate in the prior year. We note that abnormal returns are insignificantly different from zero without taking into account our complexity control measure. This finding is similar to that of Hall and Tacon[12] and confirms that past short-term earnings forecast accuracy alone is not a good predictor of the future profitability of stock recommendations. Overall, these results suggest that identifying persistence in superior analyst forecast accuracy based on a measure that controls for forecasting complexity is powerful in identifying future profitable recommendations.

Figure 2: Annualized Long-Short Abnormal Trading Strategy Returns

Summary and Guidance

We show that it is possible to identify ex ante Wall Street analysts who are truly skilled and not just lucky. To do so, however, investors must focus on the decision making process that analysts use and identify numerical measures that can be proxies for that process. We propose a measure of relative forecast accuracy over time controlling for forecasting complexity. Analysts that show persistence in relative forecast accuracy, after controlling for forecasting complexity, may expand greater effort to understand companies’ industries and their future prospects. This effort is reflected in the profitability of their stock recommendations. Thus, over longer periods, skill has much better chance of shinning through.

While no publicly available ranking system matches the feature of controlling for forecasting complexity, other research (e.g. Li, X.[13]) has shown that simply following the analysts whose recommendations consistently generate profits can be profitable. For example, Zacks Investment Research, Inc. assigns an All-Star Analyst Ratings to analysts on the basis of their stock-picking skills.[14] Specifically, in each industry, at the end of each year, the analysts are ranked into quintiles by the performance of their equally-weighted model portfolio during the year. Analysts that maintain a top Zacks rating (5 Stars) for multiple periods—would be a reasonable approximation for our study.