Skill vs. Luck in Wall Street
An Analysis of All-Star Analysts’ Stock Picks
Investors can track, to a hundredth of a percentage point, how individual Wall Street analyst’s stock picks perform. But they are largely powerless in determining the degree to which an analyst’s results are a function of skill—and how much they are attributable to plain luck. If we accept that a big part of investing success is luck, not skill, then investors should go with index funds in lieu of following the advise of Wall Street analysts. However, this argument is in contrast to the investment community and financial press’ interest in the selection of “All Star” analysts, and their reliance on analyst rankings. The assumption is that information provided by analysts has value for investment decisions (e.g. The Wall Street Journal, “Best on the Street: Analyst Survey”; “Institutional Investor Analyst Survey”).
This value may arise from analyst earnings forecasts, which are a major input into analyst stock recommendations.  Loh and Mian find that investors who follow the investment advice (i.e., buy/sell recommendations) of the most accurate earnings forecasters earn abnormal returns. This contemporaneous relation is motivated by theories of fundamental analysis (e.g., Ohlson)—if the analyst’s estimates of future earnings (inputs into a valuation model) are better, the analyst can potentially translate them into better target prices or stock recommendations (valuation outputs).
For the contemporaneous result to hold trivially in the predictive setting, however, forecast accuracy must be persistent—the impact of skill vs. luck. In this study we seek to gain a better understanding of the persistence in the relative earnings forecast accuracy of Wall Street All Star analysts over time and whether it is related to future stock recommendation profitability. In other words, we are interested whether it is possible for individual investors to identify in advance analysts who are truly skilled and not just lucky.
A Measure Of Stock Picking Skill
To measure a stock picking skill, we develop a two-step process. We begin by calculating a measure of analyst forecast accuracy for all analyst-firm-year observations in our sample. A novel feature of our accuracy measure is that we control for forecasting complexity—the dispersion in analyst beliefs about a firm’s future earnings. We argue that if forecasting complexity was not controlled for in this manner, analysts covering firms whose earnings are more predictable would be classified as accurate, while analysts covering firms with less predictable earnings would be classified as inaccurate. In short, we would be following the recommendations of analysts covering firms with predictable earnings, rather than following the recommendations of accurate forecasters across all firms. Our measure, analyst i’s forecast accuracy of firm j’s earnings per share (EPS) for period t, Accuracyi,j,t., is defined as:
where Actualj,t is firm j’s actual EPS for period t, Forecasti,j,t is analyst i’s forecast of firm j’s EPS for period t, abs is the absolute value operator, and stdevj,t [Forecasti,j,t)] is the standard deviation analyst one-year ahead earnings estimates of firm j for the fiscal period t t.
Next, for each analyst i in each year t, we compute the mean value of all forecast errors across all firms covered by the analyst, denoted by Accuracy_Meani,t. That is, we focus not on analyst predictions for a particular firm’s earnings, but rather on the analyst-level data. This distinction between firm-level and analyst-level accuracy is crucial. An appropriate question for a portfolio manager to ask may be: “given this analyst did a great job for me on one company, should I trust him on other names?” Thus, for example, if analyst Joe’s 2005 EPS forecasts for IBM, Microsoft, and Cisco are 2, 5, and 0, and the respective actual EPS are 1, 2, and 9, then Accuracy_MeanJoe,2005 equals 1.17 (= (abs(1-2)/1 + abs(2-5)/2 + abs(9-0)/9)/3 = (1+1.5+1)/3).
The aggregate set of analyst Accuracy_Meani,t results per year is then ranked from lowest to highest in terms of accuracy. Specifically, we sort all Accuracy_Meani,t observations into five portfolios (i.e. sets) such that portfolio (1) includes observations with the lowest Accuracy_Meani,t, i.e., the most accurate analysts, and portfolio (5) includes observations with the highest Accuracy_Meani,t, i.e., the least accurate analysts. Note that we require that each analyst covers at least three firms per year so that the mean and median calculations are meaningful. We also require three years of mean information for each analyst so that the time-series tests are meaningful. With the above sorts in hand, we can analyze analysts’ relative forecast accuracy.
We obtain analyst All-Star rankings from the annual Institutional Investor Survey. Analysts’ annual earnings forecasts, matching firms’ actual earnings, and the statistical measure of earnings forecast dispersion from the unadjusted Institutional Brokers Estimate System (I/B/E/S) Detail Files. Stock price data is from the Center for Research in Security Prices (CRSP). Our final sample includes 2,034 analyst-year observations over the period 1987-2012.
Spotting a Winner
Figure 1 plots the transition probabilities across portfolio quintiles of relative forecasting accuracy after controlling for forecasting complexity. The current- and subsequent-period ranks are represented by the horizontal axes, and the probability of migration is represented by the vertical axis. The area above the diagonal represents those analysts with the highest persistence in relative forecasting accuracy. The flat shape of the curve, especially the relatively flat area above the diagonal, clearly demonstrates that relative forecasting accuracy is only weakly persistent over time, that is, superior analyst forecast accuracy in the current period does not necessarily mean that this analyst will continue to outperform other analysts in future periods.
A possible explanation for our finding is that analysts’ contribution to financial markets goes beyond what empiricists can observe. The Institutional Investor survey shows that institutions do not care much about either earnings forecast accuracy or stock recommendation profitability when evaluating sell-side analysts. The number one item that institutions value in analysts is always industry and company knowledge. Thus, analysts that show persistence in the relative forecast accuracy, after controlling for forecasting complexity, may expand greater effort to understand the companies’ industries companies, and their future prospects.
Figure 1: Transition Diagram between Current and Future Quintile Ranking
In the next section we show that this reduced persistence, however, measures true forecasting ability. In other words, in the short-term analysts may experience good or bad luck (and that can overwhelm skill), but in the long-term luck tends to even out and skill determines results.
Performance of Skill vs. Luck
For the five accuracy portfolios we calculate average recommendations for each stock followed by the analysts in each portfolio. Our portfolios are long stocks in which the consensus recommendation is favorable and short stocks in which the recommendation is unfavorable. Thus, we report the returns of long–short portfolios, which are essentially zero-investment portfolios formed by buying the long portfolio and selling short the short portfolio of each analyst group.
Due to the skewed distribution of analysts’ recommendations, stocks are included in the favorable (long) portfolio when their average recommendation is greater than or equal to four (i.e. a buy recommendation or better). Stocks are included in the unfavorable (short) portfolio if the average recommendation is less than or equal to 3.5 (i.e. between a hold and underperform or worse). We exclude all stocks with an average rating that is greater than 3.5 but less than 4. Recommendations are used until they are reiterated/revised or become stale (i.e. maximum life of 6 months).
This portfolio formation process is conducted prior to the first trading day in year t and the portfolio is only rebalanced when changes in average firm recommendations, caused by the exclusion of stale recommendations or the issuance of new recommendations, moves a firm in or out of the favorable (long) or unfavorable (short) portfolios. Portfolio value-weighted returns are then calculated at the end of each trading day and compounded to monthly returns. The returns to these portfolios are adjusted for four risk factors: the market factor, the size and book-to-market factors proposed by Fama and French, as well as the Carhart momentum factor.
Figure 2 depicts the combined effect of the quality, alphas, of favorable and unfavorable recommendations of each persistent group after controlling for persistence in the relative forecast accuracy and after controlling for forecasting complexity. The figure shows that the high accuracy persistence portfolio produces a positive four-factor-adjusted return of 0.15 percent (p-value < 0.0001) per month. In contrast, the low accuracy persistence portfolio earns factor-adjusted returns of –0.23 percent (p-value < 0.0001), implying analysts are more likely to be inferior forecasters than superior forecasters.
Moreover, one could formulate a hedged investment strategy by buying the high accuracy portfolio and shorting the low accuracy portfolio to earn a monthly return of 0.38 percent (p-value < 0.0001). This monthly abnormal return is equivalent to a compounded annual abnormal return of 4.7 percent. To highlight the importance of our results we also report returns investors would earn by simply following the recommendations of analysts who were most accurate in the prior year. We note that abnormal returns are insignificantly different from zero without taking into account our complexity control measure. This finding is similar to that of Hall and Tacon and confirms that past short-term earnings forecast accuracy alone is not a good predictor of the future profitability of stock recommendations. Overall, these results suggest that identifying persistence in superior analyst forecast accuracy based on a measure that controls for forecasting complexity is powerful in identifying future profitable recommendations.
Figure 2: Annualized Long-Short Abnormal Trading Strategy Returns
Summary and Guidance
We show that it is possible to identify ex ante Wall Street analysts who are truly skilled and not just lucky. To do so, however, investors must focus on the decision making process that analysts use and identify numerical measures that can be proxies for that process. We propose a measure of relative forecast accuracy over time controlling for forecasting complexity. Analysts that show persistence in relative forecast accuracy, after controlling for forecasting complexity, may expand greater effort to understand companies’ industries and their future prospects. This effort is reflected in the profitability of their stock recommendations. Thus, over longer periods, skill has much better chance of shinning through.
While no publicly available ranking system matches the feature of controlling for forecasting complexity, other research (e.g. Li, X.) has shown that simply following the analysts whose recommendations consistently generate profits can be profitable. For example, Zacks Investment Research, Inc. assigns an All-Star Analyst Ratings to analysts on the basis of their stock-picking skills. Specifically, in each industry, at the end of each year, the analysts are ranked into quintiles by the performance of their equally-weighted model portfolio during the year. Analysts that maintain a top Zacks rating (5 Stars) for multiple periods—would be a reasonable approximation for our study.
 Fama, E. and French, K. (2010). “Luck vs. Skill in the Cross-Section of Mutual Fund Returns,” The Journal of Finance 65 (5): 1915-1947. Fama and French argue that fees paid for active investment management might be wasteful, and hence in the long run passive investment may be superior to active investment. However, some funds show nonzero alphas.
 Bradshaw, M. T. (2004). “How do analysts use their earnings forecasts in generating stock recommendations?” The Accounting Review 79 (1), 25-50.
 Simon, A. and Curtis, A. (2011). “The Use of Earnings Forecasts is Stock Recommendations: Are Accurate Analysts More Consistent?” Journal of Business Finance & Accounting 38 (1, 2): 119-144.
 Loh, R. K., and Mian, G. M. (2006). “Do Accurate Earnings Forecasts Facilitate Superior Investment Recommendations?” Journal of Financial Economics 80 (2): 455-483.
 Ohlson, J.A. (1995). “Earnings, Book Value and Dividends in Security Valuation,” Contemporary Accounting Research 11(2): 661-687.
 Loh and Mian (2006) note that their model cannot predict the profitability of future recommendations as they measure forecast accuracy and recommendation profitability contemporaneously.
 The standard deviation of the absolute forecast error captures the fact that the ability of an analyst to forecast a firm’s operations varies across firms. For example, predicting the performance of firms that are more transparent, have higher accruals quality, are less prone to manipulation, and have less volatile income is more simple and thus analyst forecasts for these firms are likely to demonstrate lower forecast errors. We refer to this effect as forecasting complexity.
 In their October issue, Institutional Investor Magazine selects the three best analysts for the year. The ranking is based on a survey sent to institutional investors and includes criteria such industry knowledge, client relations, earnings forecast accuracy, and recommendation profitability. Prior research suggests that All Star analysts are more accurate than their peers.
 This is a non-parametric approach which consists of simply estimating probabilities based on relative frequencies of analyst accuracy observed in two consecutive years.
 Fama, E. and French, K. (1997). “Industry Costs of Equity,” Journal of Financial Economics 43, 153-193.
 Carhart, Mark M. (1997). “On Persistence in Mutual Fund Performance,” Journal of Finance 52 (1): 57–82.
 Hall, Jason and Tacon, Paul B. (2010). “Forecast Accuracy and Stock Recommendations,” Journal of Contemporary Accounting and Economics, Vol. 6, pp. 18-33.
 Li, X. (2005). “The Persistence of Relative Performance in Stock Recommendations of Sell-side Financial Analysts,” Journal of Accounting and Economics 40 (August): 129−152.)
 Zacks Investment Research, Inc, http://www.zacks.com/research/allstar/explained.php.
About the Author(s)
Andreas Simon, PhD, is an Assistant Professor of Accounting at Pepperdine University. He studies the role of analysts as information intermediaries in capital markets, as well as using accounting information to predict stock returns. His published articles can be viewed in journals such as, The Journal of Business Finance & Accounting, Applied Financial Economics and The Journal of International Accounting Research. Professor Simon teaches Financial Accounting, Financial Statement Analysis and International Accounting at the graduate level. Dr. Simon’s previous professional experience entailed working as a Financial Analyst for Landesbank Berlin in Germany and as an Associate in advisory for PriceWaterhouseCoopers in Germany.
E. Matthew Van Winkle, PhD, is Director of Research and Principal at Voyant Advisors, an independent equity research firm which he has been associated with since 2007. Dr. Van Winkle was formerly a faculty member at the W.P. Carey School of Business at Arizona State University. His academic work includes, “Percent Accruals,” The Accounting Review and “Motives for Disclosure and Non-disclosure: A Framework and Review of the Evidence,” Accounting & Business Research. Previously, Dr. Van Winkle was an Associate with PricewaterhouseCoopers LLP. He also holds Master of Professional Accounting and a Bachelor of Arts degrees from the University of Washington and is a Certified Public Accountant.