Caterpillar’s Employee Assistance Program: Evaluating the Workplace Effects of EAP Services
Employee assistance programs (EAPs) have earned a core role in the enterprise employee benefits portfolio for reasons that appear intuitive or self-evident: unresolved personal problems are commonplace and costly in the workplace, and resolving these problems leads to healthier and more productive workers. But as EAPs have expanded and diversified, this self-evident assessment is no longer sufficient to guide management decisions about if or how an EAP “fits” in the employee benefits package or as a component of a wellness and productivity strategy.
Historically, the employee assistance (EA) field has measured “effectiveness” by reporting on basic metrics (such as utilization rates) and “feel good” assessments (such as positive anecdotes or testimonials). But most human resources and benefits purchasers agree that these measures alone are inadequate and that EA providers need to credibly demonstrate how their interventions affect the workplace. Few EA providers have assessed the effects of an EAP on work-related outcomes using validated measures and acceptable methods; instead, the focus has been on inputs such as “counting heads” and dissecting operational processes rather than outputs, actual results, or impacts.
To fill this gap in providers’ ability to objectively assess the effect of an EAP on workrelated outcomes, a validated measurement tool specifically designed for EAP evaluation — the Workplace Outcome Suite (WOS) (see image - click to enlarge) — was developed. Caterpillar Inc., a Fortune 100 company based in Peoria, Illinois, was the first employer to test the effects of its EAP service using the WOS. This article explains how the WOS is used and profiles the results for Caterpillar’s EAP.
Caterpillar’s EAP Model
The Caterpillar EAP utilizes an internal/external hybrid model. This model features onsite EAP counselors who are housed within the medical departments of the 13 larger Caterpillar facilities in the U.S., combined with an external, offsite vendor that sets up EAP services via a centralized call center and then delivers EAP counseling through a national network of contracted affiliate counselors. Employees who have easy access to an onsite EAP office can seek EAP services either onsite or offsite, but other employees and family members tend to utilize offsite services. Both onsite and offsite EAP services are coordinated by an internal EAP manager who is an employee of Caterpillar and is based in the corporate medical department. Given the pros and cons of strictly internal or external models, Caterpillar views the hybrid solution as blending the advantages of both. More than 90% of all counseling and referral interventions occur face-to-face with either an onsite master’s-level counselor or an offsite master’s- or doctoral-level clinician in a private office setting.
To evaluate the effects of the EAP, self-reported data were collected from 561 U.S.-based Caterpillar employees who had been EAP clients. The data, including workplace performance measures before and after utilization of the EAP, were collected over a 30-month period. The outcome measurement tool, the Workplace Outcome Suite (WOS), is a 25-item measure specifically developed for use with EAPs by the Division of Commercial Science at Chestnut Global Partners. The data were aggregated from Caterpillar EAP users and facilities throughout the U.S. to provide a picture of the workplace effects of Caterpillar’s EAP.
Workplace Outcome Suite
The WOS is short, precise, and well suited for studies that compare employee measures before and after use of an EAP. It uses a Likert-type rating scale that examines various components of the effects of personal issues in relation to four aspects of workplace functioning and in relation to overall life satisfaction. The WOS demonstrates acceptable psychometric properties, including validity and reliability, is able to detect change over time, and has a manageable administrative and respondent burden. The WOS is copyrighted but can be used free of charge by EA providers with signing of a license agreement. The measure can be downloaded online. Currently more than 400 EA providers have signed license agreements to use the WOS.
The 25-item WOS used in this study contains five scales that measure concepts at the heart of understanding EAP effectiveness: absenteeism, presenteeism, work engagement, workplace distress, and life satisfaction. Each of the five scales has five items. Two separate validation studies tested the reliability of the scales, the structural validity of the items, and the construct validity of the unit-weighted scale scores. The results of these studies support the use of the WOS to evaluate workplace effects of EAP and provide evidence that the WOS does indeed measure the five constructs it is intended to measure. A short, five-item version of the original 25-item WOS has been developed and validated and works well, but it is not as “sharp” or sensitive in detecting change, compared to the 25-item version.
The Five Workplace Outcome Scheme Scales Are:
1. The Work Absenteeism scale assesses the number of hours absent due to the employee’s personal concern(s). Example item: For the period of the past 30 days, please total the number of hours your personal concern caused you to miss work altogether.
In each of the other four scales, respondents were instructed to read statements, to reflect on how the statement applies to what they may have done or felt at work and at home over the past 30-day period, and to indicate the degree to which they agree with each statement. Each item was rated on a 1-5 Likert-type rating scale, where 1 indicates strongly disagree, 2 indicates somewhat agree, 3 indicates neutral, 4 indicates somewhat agree, and 5 indicates strongly agree.
2. The Work Presenteeism scale addresses decrements to productivity even though the employee was not physically absent but nonetheless was not working at his or her optimum due to unresolved personal problem(s). Example item: My personal problems kept me from concentrating on my work.
3. The Work Engagement scale refers to the extent to which the employee was passionate about his or her job. Example item: I am often eager to get to the work site to start the day.
4. The Workplace Distress scale looks at the employee’s feelings of distress about being at the work site. Example item: I dread going into work.
5. The Life Satisfaction scale addresses the employee’s general sense of well-being. Example item: So far, my life seems to be going very well.
The WOS is copyrighted but can be used free of charge by EA providers with the signing of a license agreement. The measure can be downloaded online here. Currently more than 400 EA providers have signed license agreements to use the WOS.
Study Design
A repeated-measures design was used to examine possible changes over time in the five WOS scales, from before use of EAP services to after use of EAP services. Initial data collection was embedded into the routine EAP intake. The follow-up survey, conducted 90 days after service provision, was not disruptive to the EAP clinical process or client experience. A goal of the study was to see if an improvement in work performance after use of the EAP persisted over time, so data for the period after EAP utilization were not collected immediately after the final EAP session. Instead, these data were collected after a 3-month follow-up period.
A methodological advantage of using before-and-after data is that it allows each person to serve as his or her own control for other factors, such as demographic and clinical characteristics. This type of “pre-post” singlegroup study design is commonly used in applied research evaluations in the EA field. However, the study lacked a comparison group of other employees at the same organizations who had similar issues relevant to EAP services but who had not used EAP services.
Only the employee clients of an EAP service (and not family members or dependents) were included in the study because of the interest in the relationship between EAP intervention and work performance. The recruitment of participants involved finding employees willing to complete the evaluation. Clients were not offered an incentive to participate in the study and were allowed to drop out of the study at any time. Possible biases in the findings resulting from a lack of random sampling and employee self-selection as study participants could not be empirically examined.
An analysis was performed to compare mean scores for each WOS scale before and after use of the EAP to detect differences and percent change over time.
Positive Impact of EAP Services
Change in the expected direction to show an improvement following use of EAP services was found for each WOS scale, and a statistically significant change (at the p < .0001 level) was found for all scales except for Work Engagement.
The Absenteeism scale revealed an average of 3.78 fewer hours of missed work, or 36.3% improvement. Employees reported their personal problems interfered less with their effectiveness at work after use of the EAP (a 22.9% improvement in scores on the Presenteeism scale). Work Engagement was virtually unchanged. This may be attributed to the fact that the EAP has a limited ability to alter work conditions. After use of the EAP, employees on average reported feeling less distress about going to work (9.6% improvement in scores on the Work Distress scale). Finally, the change in level of Life Satisfaction was positive (8.5% improvement).
These outcomes suggest that Caterpillar’s EAP services are associated with improvements in three workplace areas and overall life satisfaction, with particularly large effects for absenteeism and presenteeism. Employers that use a hybrid EAP model with an emphasis on face-to-face services can probably expect comparable outcomes.
These results are further evidence of the positive impact EAPs can have on business-relevant variables. In some other unpublished EAP outcome studies, changes in the WOS scale scores from before to after use of an EAP have been even larger. These findings have led Caterpillar to question whether these differences are due to the characteristics of the company’s EAP or its delivery, the way in which the outcome studies were conducted, or other unique attributes in the Caterpillar population.
Further Research
The question still remains of whether the results translate into meaningful real-world evidence and into observable, even monetized, results for employer purchasers of EAPs. Absenteeism can be monetized, but the other constructs cannot be reliably turned into economic outcomes, although there is wide acknowledgement that there are indirect cost savings when presenteeism and workplace distress are reduced. Caterpillar uses these data as one type of evidence to demonstrate the effectiveness and value of the EAP. The WOS studies can also be used to set programmatic goals and targets for process improvements. Caterpillar has even begun to use the WOS data to examine effects for specific subpopulations and to explore specific questions about the Caterpillar EAP, including whether onsite and offsite services demonstrate differing outcomes.
John Pompe, PsyD, SPHR, is assistant medical director at Caterpillar Inc.
David A. Sharar, PhD is managing director at Chestnut Global Partners (CGP) and a research scientist at Chestnut’s Division of Commercial Science. More information on the WOS can be found here.
Monica Ratcliff, LCSW, PHR, is manager, EAP and Disability, at Caterpillar Inc.
About Caterpillar
Caterpillar is the world’s leading manufacturer of construction and mining equipment, diesel and natural gas engines, industrial turbines and diesel-electric locomotives. Caterpillar is also a leading financial services provider through Caterpillar Financial Services.
Last Updated: January 2015