When helping to evaluate mental health and behavioral health programs, my theory is that all programs have a mixture of participants who are thriving, striving and sliding (out the door) in the program. When developing program metrics, most start with metrics of ‘how many participants did we serve’, ‘who did we serve demographically’ and ‘how long did we serve participants’ – perhaps with an extra metric of ‘how much did we spend’. However, does this tell you what you are doing well?
Program performance evaluations and ‘performance data’ can be a scary thing for programs who have traditionally taken more of a head-counting, service-hour counting and cost-counting approach to KPI. Many programs are scared or don’t know how to ask about performance. What will the data say about performance? What if I don’t like what I see?
Here is a risk-free way to begin using your data to look at program performance. Consider identifying your program strengths by determining who is thriving in your program. Let’s be honest, not all programs or all staff matches work well for every participant. Many programs see 1/3 of their participants drop out of the program before staying long enough to see success – the aka ‘early exits’. However, programs have strengths which can be measured by describing participants who are thriving in the program. Here are some steps to get started in celebrating the positives of a program’s performance.
1. Create one or more definitions of success for the program and name it something simple. Let’s start by looking at only those who have exited the program and have a complete episode of care – this way we have the complete picture for the episode. Here are some examples (Note: These are just examples. Each program will have a very specific definition of success based on the theories underlying the program.)
- Graduated – anyone who had a planned discharge, met >75% of their goals, maintained or improved on all measured well-being domains and completed a satisfaction survey showing >85% satisfaction.
- Prospered – anyone who stayed in the program for six months or longer, met >25% of their goals and improved on at least one measured well-being domain.
- Engaged – anyone who stayed in the program for six months or longer.
2. Using the definition of success, split your discharged population into two groups: those who met the definition of success and those who did not. If most of your population falls into either category, then rethink your definition. It may be too high of a bar or too low. Start with everyone who discharged in the last year and the year before – two complete years for two comparison groups.We are going to focus on those who met the definition of success to describe “thriving”. Remember, those who did not meet this definition of success include two more groups (striving and sliding). We are not going to deconstruct that group in this exercise. We want to celebrate success first, then we will go back and identify how we can increase our ‘thriving’ group by adjusting our program to better meet the needs of those who are striving and sliding in the program.
3. Describe characteristics of participants who thrived in your program, based on your definition of success. To do this, create a table to compare those in the thriving group to the remainder of the population discharged in the year. Identify statistically significant differences and describe who is thriving in your program. See the example below. I used the table to organize the data by descriptors for comparison. From this, we can see that the children’s mental health program has strengths in serving male youth, youth referred by the crisis center, and youth with need for adjustment to trauma, emotional dysregulation and self-harm/suicide ideation. Awesome – let’s communicate this!
For communicating of program strengths, I would keep the table as part of the analysis which could be referenced, but I would present a more simplified story as seen in the graph below the table. In preparing to communicate the findings, I would describe the program strengths by showing the proportion of participants in each descriptive category who met the definition of success, in this case thriving graduates of the program. Awesome – celebrate your strengths!
4. Lastly, look at year over year changes and set a goal for next year. In the example, we can see that 80% of participants who had a need related to adjustment to trauma thrived in our program. Next year, let’s aim to make that 90%. Communicate the goals and set up supports to achieve the goal. That might mean more training for trauma-informed care or it might mean identifying those staff with strength in this area and pairing up as mentors with staff who are experiencing lower success rates for participants with this need. You can do CQI (continuous quality improvement) by further improving on your program’s strengths!
Today we have focused on celebrating program performance strengths. In other words, evaluating program performance is not just about pointing out program weaknesses. It is a good idea to use the program strengths to build upon the program needs. However, after a round of celebrating strengths, it will be time to go back and describe those who are still striving to succeed and those who are sliding out the door as early exits or moving to higher levels of care. In the same way we have identified goals and supports for building on strengths, we can identify goals and supports for program needs. My hope is that if we start by focusing on our strengths, we will feel empowered to tackle our program needs. In the end, the goal is for everyone who enters the program to thrive.