Table of Contents: 

1. What is the Rater Effect? 
2. Why Does the Rater Effect Occur? 
3. The Rater Effect as a Result of Lack of Disclosure
4. Could the Rater Effect Bias Be Intentional? 
5. Does the Rater Effect in ESG Ratings Facilitate Greenwashing?
6. Tackling the Rater Effect in ESG Ratings 
7. Conclusion

Did you know that the correlation coefficient of different leading ESG raters is only 0.49? In contrast, that coefficient jumps to 0.96 for credit rating agencies. 

In plain words, there is a significant divergence of ESG ratings. And yes, we can feel the frustration rising up from the confusion over which rater provides the most trustworthy ratings. 

The divergence is a result of a variety of limitations of the ESG rating process. These range from substandard quality of data, lack of standardization of the metrics used to assess the companies, right up to outright biases among raters - either intentional or unintentional. 

The so-called ‘rater effect’ observed among ESG rating agencies is one such bias that contributes to the ongoing divergence. Studies have estimated that it could account for between 14% and 18% of rater disagreement (Berg et. al., 2019). 

What is the Rater Effect? 

The rater effect is an example of a halo effect bias. It occurs when a rater transfers the positive or negative scoring on one attribute of a company’s performance to another, which is completely unrelated. 

For example, an ESG rating agency may rate a company highly on its energy efficiency practices and then assume that if it is doing well in this aspect, then it is doing equally well in, say, water consumption efficiency, even if there is no evidence to support this. 

Why Does the Rater Effect Occur? 

There are several explanations as to why the rater effect occurs. Berg et al. (2022) discuss some of them: 

One potential explanation is that rating agencies are organized so that analysts specialize in firms rather than indicators. A firm that is perceived as good in general may be seen through a positive lens and receive better indicator scores than a firm that is perceived as bad in general.

Then, based on a discussion with S&P Global, the authors found that some raters make it impossible for firms to receive a good indicator score if they do not give an answer to the corresponding question in the questionnaire. This happens regardless of the actual indicator performance. 

Furthermore, the rater effect may occur across multiple categories due to vagueness of the metrics used to assess ESG indicators. Multiple data points are considered, but they are often not the same across companies and/or categories, which leads to incomplete and possibly inaccurate assessments. 

Billio et al. (2021) argue that the identification of sensitive data points is a crucial prerequisite for a good rating. However, the study reveals that the sources of information change from one rating agency to another. 

For example, ISS-Oekom and Bloomberg leverage on direct contact with the company, while Thomson Reuters also considers stock market registrations.

Besides, the number of assessed indicators changes among the different raters. MSCI and FTSE Russell represent the extreme cases since they assess respectively 37 and 300 ESG criteria.

As a consequence of all this, it becomes somewhat natural for ESG rating agencies to start looking for shortcuts, triggering the halo effect bias by assuming that if a company is successful in one category, it will probably be good enough in another, even if no sufficient evidence exists. 

The Rater Effect as a Result of Lack of Disclosure

It has already been established that the refusal to answer a question about its ESG performance in a certain area, is likely to lower the score of a company. However, the problem runs deeper. 

The lack of information almost always leads to assumption and subjectivity, finds a study published by Christensen et al. (2020). In the absence of clear rules for evaluative practices, it is likely that rules of thumb will be developed for the least demanding evaluative tasks. 

For example, while an ESG analyst would need to judge a specific piece of information, in the absence of that information, a simple rule can be developed. Therefore, in the absence of disclosure, raters are more likely to agree because they use similar rules of thumb and imputation techniques. 

For instance, if a single company does not disclose information on an important issue for its industry, they perceive the lack of disclosure as a bad attribute and thereby assign bad performance to the company. 

Similarly, in the absence of disclosure across most companies in an industry, they likely perceive the lack of disclosure as a sign that the issue is relatively unimportant, thereby imputing the company’s performance to be the average performance of companies in the industry. 

Could the Rater Effect Bias Be Intentional? 

In one of the first papers to scrutinize the integrity of ESG ratings and identify problems related to conflicts of interest, Tang et al. (2022) find that firms sharing the same major shareholders with the rater (“sister firms”) receive higher ESG ratings.

According to the scholars, the well-documented rating disagreements among ESG rating agencies are neither random nor innocuous. Different rating agencies are controlled by different owners, who then exert their own influence. 

The case of intentional bias is quite different from what we have been discussing so far. First, it is exactly what we just called it - intentional. Therefore, it distorts the ESG score quite, well, intentionally. 

Secondly, and on a more positive note, if an intentional bias exists, then it can be eliminated more easily, as well, with the introduction of appropriate rules.

Does the Rater Effect in ESG Ratings Facilitate Greenwashing?

Greenwashing has been identified as the main concern for investors. The practice of appearing more sustainable than you actually are and the difficulties of detecting instances of greenwashing have been a major criticism directed towards ESG ratings agencies. 

Therefore, it is worth reasoning a bit more on whether the presence of a rater effect makes it easier for companies to greenwash. 

From what has already been discussed, we venture to suggest that such a connection exists, indeed. 

How? 

By not assessing each relevant aspect that determines a company’s ESG performance properly, ESG rating agencies effectively allow themselves to be (mis)led by the assessed company’s positive results in one category. 

And when they tend to take this as an indicator of a good performance in others, this can (un)intentionally boost the business’ score. 

What appears, then, is that a company can focus on getting few aspects of its ESG efforts right, and that could be enough for ESG rating agencies to give it a high score in others, too. 

In fact, we can also say that greenwashing is inherently related to the halo effect concept, since the latter is the embodiment of the tendency for positive impressions of a person, company, brand or product in one area to positively influence one's opinion or feelings in other areas. 

In other words, when a company does enough to make itself look especially environmentally-friendly, this could convince ESG raters that there is no need to dig deeper. 

Tackling the Rater Effect in ESG Ratings 

As we mentioned earlier, the rater effect in ESG ratings may be down to companies not reporting enough information. While it is certainly a problem of ESG rating agencies that their ratings are influenced by the halo bias, the assessed companies are also to be deemed responsible if they refuse to provide quality data. 

It appears, then, that one of the reasons for the rater effect could also offer a way to resolve the issue. 

If ESG ratings are not an accurate reflection of companies’ performance partially due to the rater effect, and if these companies are not happy with the way they have been assessed, then they may need to start providing more details regarding their ESG efforts. Otherwise, their ratings will continue to be based on biased assumptions. 

ESG rating agencies also need to work towards standardizing the ESG criteria they use. It is incredibly confusing how two leading raters like MSCI and FTSE Russell can have such a large difference in the number of ESG criteria they assess - 37 and 300, respectively. 

Finally, with regards to the fact that the rater effect may be a result of an intentional bias resulting from shared shareholders between the rater and the rated company, stricter rules need to be applied to avoid such instances. 

A greater transparency regarding the ownership of the stakeholders will at least make it clear if any of these have a vested interest in rating a company higher. 

Conclusion 

The rater effect bias is an example of a halo effect bias that may sometimes be hard to take control of, simply because of our human nature. We tend to form an overall positive opinion of someone or something based on them displaying positive traits in only one or few aspects. 

Yet, the rater effect in ESG ratings is also down to reasons that can be addressed by raters. The establishment of more standardized criteria and data points can propel them to focus on indicators rather than on the companies themselves, which will take away much of the halo effect. 

The assessed businesses can also help improve the rating process by providing more detailed information about their activities, so that ESG raters have more real data to work with. 

The formation of the International Sustainability Standards Board (ISSB) by The International Financial Reporting Standards (IFRS) Foundation in November 2021 demonstrates a willingness by powerful organizations to harmonize the disclosure standards in ESG. 

In a similar vein, the European Commission is set to adopt the first set of sustainability reporting standards, developed jointly by The European Financial Reporting Advisory Group (EFRAG) and the Global Reporting Initiative (GRI) by October 31, 2022. In July 2021, the two started collaborating to construct the new EU sustainability reporting standards. 

The success of such initiatives is likely to pave the way for more objective rating of companies’ sustainability efforts. 

Find More ESG-Related Resources Below: 

What Makes ESG Ratings Problematic? 

What is the Biggest Concern for ESG Investors?  

Printing A Green Candle: The Impact of Green Contract Awards on Stock Returns

Green Public Procurement Contracts Data for ESG Analysis