Opening the Black Box of Analyst Forecasts

What separates a great analyst from those who are merely good? The most common measurement of analyst skill is to look at how accurate their forecasts are. But judging the expertise of an analyst purely based on their forecast outcomes can be biased. An accurate forecast can just as easily be the result of good luck. In light of this, a group of researchers, including at The Chinese University of Hong Kong (CUHK), took it upon themselves to design a new method of identifying analyst topic-specific skills and decode the black box of analyst forecasts.

“Observing outcomes can be like looking at a black box. You can observe success, but you don’t really know which skill they have employed and how much luck is involved.”

Prof. Cen Ling

The study What Do Questions Reveal? Analyst Topic-Specific Skill and Forecast Accuracy was co-conducted by Cen Ling, Associate Professor of Finance, and PhD student Han Yanru, both at The Chinese University of Hong Kong (CUHK) Business School, as well as Prof. Jarrad Harford at the University of Washington. This study looks at earnings conference calls and how analysts phrase their questions to determine whether they have topic-specific skills. It shows that, when they are dealing with new information that are associated with topics in their expertise, these analysts are likely to make more accurate forecasts than their peers.

“Observing outcomes can be like looking at a black box. You can observe success, but you don’t really know which skill they have employed and how much luck is involved. People tend to look at outcomes and make a strong inference to skill,” Prof. Cen says. “What we did in the study is to come up with a different approach to identify the type of skills involved in the decision-making process and to avoid outcome-based bias in judging people’s skills.”

Identifying Expert Analysts

The researchers tested their hypothesis on the specific but important topic of supply chain. Using data from earnings conference calls of all U.S.-listed companies, they identified supply-chain expert analysts and examine whether their forecasts were more accurate than peers when the firms they covered experienced significant supply-chain-related structural shocks. To identify supply-chain expert analysts, they looked at whether the questions an analyst raises in all earnings conference calls are associated with special supply-chain terminology such as “customer”, “supplier”, “vertical integration”, “trade credit”, and “OEM”. Very intuitively, analysts who raised the highest number of supply chain-related questions in past conference calls were identified as supply-chain expert analysts.

They found that supply chain expert analysts showed a 6 percent reduction in forecast errors on average, compared to analysts who were not experts, when the firms they cover established relationships with large publicly listed customer firms. This effect is much stronger when the supplier firm suffers a higher level of information asymmetry (for example, suppliers that are smaller and less well-known) or customer switching risk (suppliers that have more product market competitors).

“The approach we applied in identifying expert analysts is very similar to Google’s dynamic advertising technology. For example, Google uses search and Youtube watching history to identify products that its users are potentially interested in,” says Prof. Cen. “This approach is definitely not confined to the analyst setting. Anyone can develop certain skills when their interest is combined with repeated practice. Our approach just takes advantage of the human nature that one is more likely to talk about something they are interested in and good at.”

Skill Transfer and Recognition

According to the study, topic-specific skills can also be transferred to other analysts in the same brokerage firm. Taking the example of supply chain topic again, an addition of one supply-chain expert analyst is linked to an 8 percent increase in the likelihood of a junior analyst employed by the same brokerage firm becoming a supply-chain expert in three years. In addition, the study highlights that the difference in the performance of expert and non-expert analysts gradually decreases over time, which suggests that non-experts also learn from their peers.

Surprisingly, the study suggests that market participants are more capable in recognizing topic-specific skills than employers. “It’s unfortunate, but our results show that brokerage firms don’t really pay sufficient attention to the different skills that their analysts have when making stock coverage assignments,” Prof. Cen says. “It’s really in their best interests to assign firm coverage according to the skills and interests of their analysts. Not only does this mean that they allow their experts to thrive, but it just makes sense from an economic point of view because it leads to an improvement in the accuracy of their analysts.”

On the other hand, investors recognise the expertise of specialist analysts and react to their recommendations. The study finds that, after a firm establishes relationships with important customer firms, a downgrade recommendation made by a supply chain expert analyst generated an extra negative 1.3 percent return over a three-day period, when compared to recommendations made by ordinary analysts. This finding suggests that investors pay more attention to opinions of supply-chain expert analysts when supply-chain risk carries a higher weight in future performance.

Generalization of the Approach

Most importantly, although the study mainly uses supply-chain-specific skill as an example, the authors suggest that the approach of identifying topic-specific skills is generalizable to other topics, such as new product development, mergers and acquisitions, payout policies, and tax strategies.

However, Prof. Cen points out that three conditions must be met for this generalization: first, the topic that analysts are interested in must be relevant to company earnings. Second, to establish a causality, the opinions and expertise of analysts should not affect the type and timing of information arrivals. This can happen, for example, where the opinions of M&A expert analysts can affect the outcome (pass or withdrawal) of M&A deals. Finally, researchers must be able to identify analyst skills through textual analysis with a reasonable accuracy, for example through either keyword search or machine learning techniques.