Make Program Evaluation a New Year’s Resolution in 2016

By Margaret Lane
Principal
ChaseLane Consulting

Original Publish Date: January 11, 2016

What if you implemented a new pilot healthcare project last year and could have learned after just nine months that your pilot was exceeding expectations and within budget, allowing you to accelerate a decision to reallocate resources and expand the pilot?

As increasing concerns over declining reimbursement and resource allocation make it essential to evaluate the effectiveness of healthcare programs today, organizations looking for a fresh way to assess and make decisions about their initiatives in 2016 may want to consider using program evaluation.

By doing so, you can determine if an initiative is meeting its stated objectives before funds have been allocated for a project that isn’t achieving results – or worse yet, for a project that could have been improved for success had you known what wasn’t working and understood the context.

“The shift from fee for service payment to outcome based reimbursement is creating a trend for healthcare organizations to be more evidence-based,” stated Jeanette Treiber, an evaluation expert and program manager in Public Health Sciences at the University of California, Davis. “As this becomes the norm and payment is based on outcomes, these outcomes need to be measured. This makes program evaluation important for healthcare today.”

What is Program Evaluation?

A “program” (or “initiative”) is a set of activities developed to accomplish one or more goals, such as a set of services to reach a defined group, a new EHR system, or ambulatory care center.

Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of a program to make judgments, improve effectiveness, and/or inform decisions about future program development. It is a way to confirm whether programs have a positive impact and should be continued by identifying their strengths and weaknesses.

Evaluation findings should be used to make decisions about program design, implementation and effectiveness. The benefits to using program evaluation include:

Roadmap development to achieve long-term outcomes and provide visual models that can be shared, for example in the form of a logic model.¹
Alignment of activities with resources, facilitating decision-making on resource allocation, such as what to close, continue, duplicate, and expand, or even when to affiliate or stay independent.
Validation of areas of program success (what’s working and why) and areas of disconnect.

Does Program Evaluation Have an ROI?

According to Kara Crohn, PhD, Managing Consultant with EMI Consulting, there may be substantial economic benefits to program evaluation.

“Program evaluation can help identify gaps in your process and places where you can gain efficiency or reduce your risk of making errors. When you improve workflow efficiency, speed up processing, root out errors, and identify new opportunities, you can save time and money while improving your outcomes,” Kara explained. “Avoided cost is another potential important economic return. If you build evaluation into a pilot program and find out the program isn’t working well enough, then you can avoid the cost of fully implementing an unsuccessful program.”

Program Evaluation in Healthcare

Program evaluation can be effective for both for-profit as well as non-profit healthcare organizations. Non-profits who receive outside funding need to show they are producing something of value. To receive funding, you need the data to show that what you are doing is successful.

Other non-profit as well as for-profit companies are competing for patients, so they need data to show potential customers that they are doing a good job, especially in areas like patient satisfaction and efficiency.

Treiber at UC Davis states: “Healthcare organizations use program evaluation in two major areas. One is in the actual work they are doing – if a particular approach the healthcare organization is trying out is working. The second is the organizational evaluation – workflow, cost-effectiveness, the organizational part of the work.”

Examples of programs that you may use program evaluation for include:

Prevention Programs: Are they doing what we hypothesized?
Patient Care Procedures: Are they done correctly? Are they done in the way we intended? What, if any, is the level of error?

An evaluation starts with identifying the most important questions the organization needs to answer and then applies data collection methods best suited to address those questions.

Data collection methods are wide-ranging, and include questionnaires, surveys, interviews, focus groups, and case studies. In many cases, it is helpful to incorporate several data collection methods for the most accurate results.

What Type of Program Evaluation is Right for Your Organization?

The type of program evaluation you choose depends on what you want to learn about your program. Some used by healthcare and other organizations include:

Proof of Concept: Validates the likelihood of program success based on similar program models.
Implementation: Confirms if a program has been implemented as planned, validates if it is operating as designed and identifies needed changes
Process: Assesses the effectiveness and efficiency of program processes, identifying opportunities to leverage good practices and optimize processes where inefficiencies occur.
Outcomes: Determines if clients are receiving the benefits of program which can be helpful in determining to continue/expand the program.

Many healthcare organizations develop programs along a lifecycle process. Program evaluation aligns with this approach because it can assist at each phase of the process.

Here is an example of how it can work and help with decision-making at each phase:

Planning: Organizations can create logic models depicting resources, activities, outputs and outcomes. Logic models are useful for analyzing the feasibility of using designated resources to conduct specific activities.
Pilot Program Development: Evaluation can provide early feedback and continuous improvement input to quickly make modifications to the program design and implementation plan. This can be helpful in the transition to new delivery and payment models.
Fully Implemented Programs: Evaluations can assess the degree to which services/products/ interventions are meeting the targeted population’s needs and process efficiency.
The End of Funding Cycles/Key Milestones/Program Shutdown: Evaluate what was achieved and how, what lessons were learned and what can be replicated in what context.

Three Easy Steps to Get Started in 2016

Once you are ready to evaluate a program, you may want to consult with a professional evaluator to help prioritize evaluation objectives and determine what is feasible to research. Even an hour could save significant time and money later on.

Focus on your priorities and what you need to know in the near future. What decisions do you want to be able to make as a result of the evaluation and what information is needed to make those decisions?
Complete a feasibility check to confirm what data is needed, if you can access the data, the cost, and stakeholder buy-in.
Begin to identify your evaluation questions and choose the program evaluation that is right for your organization.

Resources

There are many resources on the Internet to assist with basic program evaluation.

The W.K Kellogg Foundation's Evaluation Resources
The W.K Kellogg Evaluation Handbook
The W.K. Kellogg Logic Model Development Guide
American Evaluation Association’s “Find an Evaluator”

¹A logic model is a chart that shows your resources, activities, outputs, and outcomes, or impacts.

Margaret Lane is a health care consultant and co-founder of ChaseLane Consulting. ChaseLane Consulting provides full-service strategic and business services planning and marketing support to healthcare organizations. She previously served as in VP roles at The Regence Group in Products, Planning, and Public Policy. Margaret also serves on the board of the Physicians and Dentists Credit Bureau.

Contact Margaret at margaret@chaselaneconsulting.com. Visit the ChaseLane Consulting web site at www.chaselaneconsulting.com.