CSAT or CES: Does It Matter?

Introduction

Companies are always looking for ways to increase productivity and customer loyalty. There are a number of metrics that have been developed to understand the relationship between the company and the customer. At this time, some of the most popular techniques for measuring the strength of the relationship between the company and its customers are Customer SATisfaction (CSAT), Net Promoter score (NPS) and Customer Effort Score (CES). Some of the less popular other metrics include the customer happiness index, the customer trust index and the customer experience index. The American Customer Satisfaction Index (ACSI) provides a broad-based metric which provides CSAT for multiple industries. While there are many others that are being developed and used, these three (CSAT, NPS and CES) seem to have found the most acceptance in the business community.

Metrics

The metrics for measuring the strength of the relationship between customer and company fall into two categories; namely strategic and tactical. The metrics for tactical relations generally focus on individual transactions between the company and the customer. The metrics for strategic relations generally focus on the broader picture with multiple dimensions of the relationships between the company and the customer beyond a simple transaction and may include sales, product performance, technology, etc.

This article focuses on the tactical metrics that rely on individual transactions and will be restricted to CSAT and CES. NPS has not been considered in this paper since there are many researchers and studies that have identified several flaws in the NPS system. While there are also flaws in CSAT and CES, these two measures were selected since they look at the individual transactions from very different perspectives (maximization of satisfaction verses minimization of effort).

Customer Effort Score

Many believe that CES provides better guidance for allocating resources to improve the customer relationship (specifically loyalty) than the other metrics based on limited testing. If this is the case, then CES seeks to identify the survey questions which have the strongest relationship to the overall opinion question asking for a customer effort score (CES). When surveys are used to assess CES, the questions on the survey may be identical to the questions used when measuring CSAT.

Customer SATisfaction

Likewise there are equally as many who believe that CSAT will provide better guidance for resource allocation. For this specific research the specific service activity questions used for CES will be identical to those used for CSAT since both metrics are included in each individual customer survey. The specific questions used for the survey will be provided in the section on methodology which follows. Since this article is focused on individual events, the questions on the survey reflect the activities associated with the most recent individual service event and also assesses both metrics and was collected within days of the most recent service event.

Since the ultimate objective for any company seeking to allocate resources in the most productive manner, the objective of this research is directed toward understanding which of the two metrics provides a more effective allocation of resources.

Differences

The fundamental difference between these two measures suggests that the allocation of personnel and financial resources will be different for each of the two metrics. The obvious mathematical difference is that the CES metric is looking for the solution that minimizes effort, a minimization problem. Conversely, the CSAT metric is looking for a solution that maximizes satisfaction, a maximization problem. Therefore, the two metrics have opposing objectives; namely, CES, is looking for a minimization solution while CSAT, is looking for a maximization solution.

To believe that a minimization problem will yield the same components in the same order of importance as a maximization problem does not appear likely. As a simple example, consider a customer who needs repair on his automobile. If the customer desires to minimize his effort he will probably take the auto to the nearest repair facility. On the other hand, if the customer is looking to maximize his satisfaction with repair, he may be willing to drive an additional distance to find a repair source that will provide the best solution. While this example is based only on distance, most issues include more than one variable (e.g. urgency, downtime, $, skill, etc.).

From this simple example one can see that a customer effort score would indicate convenience quality for the customer whereas a customer satisfaction score would indicate best repair. Hence, this logic suggests that it is unlikely that a low customer effort score will yield the same importance rankings of service components as the highest customer satisfaction score for the auto repair example.

Research Focus

This research focuses specifically on two product categories; namely medical electronics equipment and IT equipment. The research includes examination of the correlation of field service and technical support for each of these two product categories. Finally, the research examines the correlation between these two metrics, CSAT and CES.

Literature Discussion

Customer retention is receiving greater consideration in corporate strategy and operations performance within product support organizations. The focus on retention is generally based on (i) customers are among the most important marketing assets of the firm, (ii) there is a strong correlation relationship between the value of the customer base and the value of the firm, and (iii) customer feedback metrics play a primary role as indicators of future customer loyalty.[1] Research has shown that a positive relationship does exist between customer satisfaction (CSAT) and overall company performance, but similar longitudinal research has not been performed for other customer feedback metrics.[2] [3] Academic research continues to challenge the assertion that customer feedback metrics represent the best indicator of future firm performance.[4] Most studies on the effectiveness of customer feedback metrics look at the strategic level of the firm in order to analyze the firm’s competitive position in the marketplace. Studies at the individual company level are not often published due to the competitive value of the information in the marketplace.[5]

Predicting Future Performance

Market research has labeled these metrics (CSAT and CES) as customer feedback metrics and has opened the measurements to a broad range of alternatives.[6] Because of this relatively undefined field of customer feedback metrics, new metric models are frequently introduced. When reviewing the practitioner literature, new customer feedback metrics that promise to be “the best” indicator of future firm performance are developed on a regular basis.[7] Published research has suggested that practitioners should include marketing mix variables to better predict future performance.[8]

Retention

Some practitioners who use customer feedback metrics on a daily basis, compare individual metrics to gain insight about the best predictor of retention.[9] Regression analysis is often used but has the limitation of high levels of covariance between the independent variables.[10] Research has also found nonlinearity and asymmetry in customer satisfaction and loyalty models.[11]

Repurchase Behavior

There are a number of variables such as repurchase, intent to repurchase behavior and recommendation, all of which may also contribute to the overall opinion metric. Dixon, et al. suggests that CES is a better predictor of repurchase intentions and increased spending than NPS or CSAT.[12] [13] One of the concerns that arises with surveys of customers is that questions can be difficult for customers to answer, especially if they do not often do business in that aspect (product repair/support) of the industry.[14]

Research Hypothesis

There are four major research hypotheses to be examined in this paper.

First research hypothesis: CES and CSAT are highly correlated.

Second research hypothesis: correlations between the individual service activity survey questions will be in the same order (high to low) for both CES and CSAT

Third research hypothesis: medical electronics equipment and IT equipment will also have the same order of correlations (high to low) for individual service activity survey questions to the overall question for both CES and CSAT

Fourth research hypothesis: field service and technical support correlations provide the same order of correlations between the individual service survey questions to the overall question for both CES and CSAT.

Methodology

 Seven technical service organizations cooperated to test the CES methodology against the CSAT methodology on 12 customer satisfaction programs within the April and May 2015 regular program schedules (including daily or weekly cycles of data collection). The 12 programs represented a mix of IT and medical technology services, for both on-site and technical support (help desk) service delivery.

Survey Questions

While each participating organization has their own independent survey program, all the organizations in this research study used the same independent survey provider (SERVICE 800),[15] a leading expert in measuring real-time service quality and customer satisfaction. The same questions, the same operating rules to manage data collection, and the same collection methodology (telephone interview in native language) were strictly controlled in this study. The telephone interviews were performed in seven languages to minimize translation errors.

All questions were scored on a scale of 1 to 5 where 1 was considered poor performance and 5 was considered excellent performance. The survey questions are listed below in an abbreviated form.

1. time to arrive at location
2. time it took to complete the repair
3. technical ability
4. completeness of the repair
5. availability of parts
6. professionalism
7. how well you are kept informed
8. overall opinion of service (CSAT)
9. use again
10. recommend to others
11. customer effort (CES)

There were 7,379 customer contacts/interviews conducted across the seven participating companies. These data represent a cross-section of feedback on IT and medical technology services delivered on-site and for technical support services. The customer contacts included customers in North America as well as Europe. Whereas the results are not suggested to be an accurate representation for all organizations or all service characteristics, they are based on a statistically valid sample for this consortium of participants. There were 2,606 customer contacts for medical electronics equipment and 4,773 customer contacts for IT equipment.

Statistical Results

The following results have been derived from the data set;

Hypothesis I – CES and CSAT are highly correlated.

Result – The correlation between the two metrics for the total sample is 0.476.

This result suggests that the correlation between the two metrics is not high but not unreasonable. There are enough relationships between the individual questions and CES and CSAT that both metrics indicate a statistical relationship with each other.

Hypothesis II – The order of importance of independent service activity survey questions (in terms of correlations) will be the same and in the same order for CES and CSAT.

Result – In general, the order of relationships appears to have a similar sequence as noted in the table below:

Hypotheses ill – Medical electronics equipment and IT equipment provide the same ordering of questions (based on correlations) to the overall opinion question for both CES and CSAT

Result – The order of the relationships does not appear similar as shown in the table below:

The sequence order (weights) of the activities relationship to CES and CSAT for the two products are different except for the time to arrive.

Hypothesis IV – The order of field service and technical support questions are the same with respect to the overall opinion question of either CES or CSAT.

Result – The order of the strengths is similar as shown in the tables below:

Hence, both business functions (field service and technical support) indicate similar order of weights of service activities as important which suggests that both metrics yield approximately the same order of weights for resource allocation.

General Discussion

In the industrial/commercial literature new metrics are frequently introduced; such as NPS, CES, Customer Experience (CX), trust index, and happiness index.[16] Few of these new metrics have been rigorously tested. In scientific marketing literature, there have been numerous studies performed rigorously to evaluate some of the metrics.[17] Some practitioners use various metrics (both evaluated and experiential) in search of the best predictor of future loyalty and financial performance by comparing them in daily practice.

Observation 1 – Correlations

In addition to the four specific research hypotheses, several additional observations are worth considering. One of the more interesting observations concerns the correlations between the individual survey questions and the overall metric questions. In general, the correlations between the individual survey questions and CSAT were much higher than the correlations between individual survey questions and CES. Even with the large differences in the correlations, the ranking of the individual survey questions remain similar so that the underlying relationships appear to be robust.

Observation 2 – Keeping the Customer Informed

A second observation notes the dramatic difference between customers in North America and Europe with respect to keeping the customer informed. North American customers showed a much stronger relationship between the overall opinion question for CES and the question relating to how well the customer is kept informed. The European customers showed virtually no relationship between the overall CES metric and the value of being kept informed CR-square less than 10 percent). However, the strength of the relationship between how well the customer is kept informed and the overall CSAT metric was much greater than CES (R-square greater than 40 percent).

Observation 3 – Differences in Technical Ability

A third observation focuses on the dramatic difference between medical equipment customers and IT customers with respect to the technical ability of the customer service personnel. The IT customers showed a much stronger relationship between the overall CSAT metric versus the overall CES metric regarding the question relating to the technical ability of the customer service personnel (R-square greater than 50 percent for CSAT versus 20 percent for CES).

Observation Summary

From the two preceding observations it is clear the overall CSAT metric has a stronger correlation than the overall CES metric with some of the independent variables associated with the service event. One perspective suggests that since both the question of keeping the customer informed and the technical ability of customer service personnel are not directly related to the service being performed, the customer perception may not be as clearly demonstrated with respect to the overall CES metric.

Summary and Conclusions

The ability to perform a study covering several countries and two products under similarly controlled conditions provides an opportunity to reach some reasonable conclusions. Having a single research firm use the identical survey instrument translated into local languages dramatically reduces the more obvious survey errors that typically occur.

Whether or not a company uses CSAT or CES is not the only concern of this research. In general, the dominant factors for allocating resources were very similar when viewed from the perspective of the total sample. The same is true when looking at the metrics for both field service and technical support. However, the weighted relationships differ dramatically based on the type of equipment being supported; namely medical electronics and IT equipment. The product seems to be the dominant factor in choosing the metric.

We conclude that even though the two metrics, CSAT and CES, have opposing optimization objectives, there is not a dramatic performance difference between the two metrics. The significant exception is the impact of the hardware product being supported. When the sample is evaluated by product (medical electronics and IT) there is a dramatic difference. Because of the dramatic impact of product on the allocation of resources for support, the decision of the more appropriate metric should focus on the product.

Major Finding: CSAT and CES are not interchangeable since the characteristics of the product being serviced may have a significant effect on the order and impact of the individual service activities.

Future Research

This research focused on customer metrics that resulted from specific events. In other words the survey was designed to provide tactical direction for the allocation of resources of the customer service organization. One of the significant findings of this research was both the CSAT and CES metrics provided similar allocation priorities. By increasing the level of detail of the analysis, a major finding was that the allocation priorities were different for the two products.

There are a number of broad areas that should be considered for future research.

1. Perform a similar research from a strategic perspective to determine whether or not there is a difference in resource allocation.
2. Perform a similar research study from a perspective to include one or more of the other metrics identified (NPS, CX, customer trust, etc.).
3. Perform a longitudinal study on CSAT and CES to determine the predictive power of the tactical results on customer loyalty.

There are several additional research opportunities that should be considered from the simple data collected for this study; namely:

1. Perform additional analyses to determine whether or not geography (North America versus Europe) plays a significant role for either of the two metrics.
2. Perform additional analyses to determine whether or not there are significant differences in the metrics (CSAT and CES) by individual country in which the equipment is located,
3. Use other statistical techniques to determine whether or not the allocation of resources is the same (consider using factor analysis and structural equation modeling).

As the role of the customer becomes more and more important to the sustainability of the firm, effective allocation of resources will provide a significant advantage to customer service organizations.