This is the second part in a two part series. You can access the first part here.
What is the Family First Prevention Services Act (FFPSA)?
The Family First Prevention Services Act (FFPSA) is a landmark federally supported opportunity to provide more supports and services to maintain children and youth in home-like settings with their natural caregivers and families. Certain requirements of the Act necessitate program monitoring and evaluation of services provided. This blog describes how Opeeka’s Person-Centered Intelligence Solution (P-CIS /pieces/) supports agencies to meet these requirements. P-CIS is an online assessment software designed to help organizations, agencies, and their staff members to plan and guide person-centered care while measuring progress along personal trajectories of safety, permanency and success.
What does P-CIS do?
- Provides the ability to capture family circumstances in meaningful ways resulting in reliable, valid, and consistently administered outcome measures for all persons receiving services.
- Enables Real-time monitoring features to ensure that care results in positive outcomes relative to practices, services, programs, populations, and agencies.
- Delivers insights features to help to monitor level of care algorithms, placement decisions, program fidelity indicators, population needs, population outcomes, costs and more.
P-CIS can quantify reductions in need for foster care and improvements in child well-being related to each service component so that agencies can put more resources toward service components that work, while dialing back efforts towards those with little impact. This results in a more efficient system of care providing increased quality of care with improved outcomes.
Applying the Research and Evaluation Provisions of the Family First Prevention Services Act
What does the act say?
“With respect to the services and programs specified in subparagraphs (A) and (B) of paragraph (1), information on the specific promising, supported, or well-supported practices the State plans to use to provide the services or programs, including a description of—
- how the State plans to implement the services or programs, including how implementation of the services or programs will be continuously monitored to ensure fidelity to the practice model and to determine outcomes achieved and how information learned from the monitoring will be used to refine and improve practices;
- the target population for the services or programs; and
- how each service or program provided will be evaluated through a well-designed and rigorous process, which may consist of an ongoing, cross-site evaluation approved by the Secretary.”
What is fidelity?
“We keep talking about this term fidelity to the model, and that’s what we mean when we talk about fidelity, the model, it’s not adding something because it feels right or feels good. It’s sticking to the evidence based model that’s out there with the full complement of resources that are supposed to come with it. Whether that’s a certain degree, a certain training, a certain certification, a certain set of trainings, whatever that is and they there, you don’t have to be a masters level therapist to be able to do all of this, but there are certain requirements that people have to go through to be able to do this practice. So when we talk about fidelity, the model, we’re just talking about sticking close to the intent of the model and the way it was designed, so it can remain in evidence based practice…. So the longer game is…working with frontline staff to understand why fidelity is important, why filling out this form is important.” –Chris Groeber, Professor of Social Work, University of South Florida, Florida Institute for Child Welfare podcast series, Season 3, Episode 2.
How to monitor program fidelity for the FFPSA?
To monitor program fidelity to the practice model, a state can implement fidelity assessments for staff and persons in care. One of the most reliable ways to monitor fidelity is to ask the people about their experience in care. Fidelity assessments ask children/youth in care or their caregivers, for example, if they received a written prevention plan, if they met with a care team member at least weekly and if they felt that their voice was heard. Simply by asking members of a care circle (i.e., children/youth, caregivers and care staff) these types of questions, it ensures that practice principles are top of mind during care. P-CIS facilitates collection of fidelity assessments through its “Invite to Complete” feature which emails questionnaires to anyone in the care circle. Recipients can securely complete the questionnaire on their desktop or phone. (Note: Recipients do not need to be registered users of the system.) Once completed, the results of the assessment appear immediately in P-CIS. In addition, the information is automatically organized and fed into P-CIS Insights – in real time – which can result in evaluation of fidelity across practices, agencies, populations, and potentially the entire state. The steps to real-time monitoring of practice fidelity are outlined below.
- Step 1: Email fidelity assessments to people for completion.
- Step 2: Recipient securely completes assessment on their devices.
- Step 3: Assessment results appear in P-CIS.
- Step 4: Insights populate from new responses for real-time monitoring.
Step 1: Email the Fidelity Assessment to a Care Circle Member for Completion
Step 2: The Recipient Securely Completes the Assessment on Their Device
Step 3: The Results of the Assessment Appear in P-CIS
Step 4: Assessments are Monitored and Evaluated in Real Time with P-CIS Insights
How can we track outcomes in real time?
All types of questionnaire responses can be merged in P-CIS Insights to track outcomes related to practice and practice fidelity. The Insights help to identify which practices are impacting outcomes and at what level of fidelity. Filters drill into insights for specific populations. As seen in Step 4, rigorous analysis (e.g., controlled logistic regression) continuously updates as data comes in, identifying what practices are impacting which outcomes. From this insight, efforts can be designed to improve practice and fidelity where they will make the most impact. What the example analysis above portrays is that all practices are important, but they impact outcomes differently.
In this example:
- Written plans including persons helping in decisions are associated with satisfaction with experience.
- Meeting weekly with inclusive decisions demonstrates the impact on satisfaction with progress.
- Among these three practices, including persons in helping to make decision for their care is the primary practice associated with how a person feels about improvements in their life.
From these results, an agency can identify care teams that may not be emphasizing these practices and can communicate to them about the evidenced importance of these practices while providing additional training and coaching.
Evaluations of Services & Programs
From the Act
The Secretary, directly or through grants, contracts, or interagency agreements, may collect data and conduct evaluations with respect to the provision of services and programs described in section 471(e) (1) for purposes of assessing the extent to which the provision of the services and programs— (A) reduces the likelihood of foster care placement; (B) increases use of kinship care arrangements; or (C) improves child well-being.
How can P-CIS assist evaluation of services and programs for the FFPSA act?
P-CIS can meet all the evaluation requirements of the Act. P-CIS Insights are powerful and customizable. Not only do they automate analyses from questionnaires and assessments, but they also connect to data sources to pull information from other electronic records into the analyses. P-CIS embeds advanced analytics powered by Python and R to uncover significant findings through higher level statistical approaches, such as regression, classification, hierarchical analysis, longitudinal analysis, and other approaches as desired. With P-CIS as the hub for evaluation, no file export is needed. Analytic staff can log into a P-CIS Analytics Space to perform most types of outcome analysis on live and historic data. The Analytic Space supports unlimited research and evaluation notebooks for analytic staff or evaluation consultants.
Because of the standardized and highly structured format of data exchange, P-CIS organizes, merges, and transforms data minutes after it is collected or exchanged. Data is then fed to analytical engines for evaluations that can be driven by selections from drop-down filters or research notebooks, alike. An agency’s analytical staff, Opeeka’s analytical staff, or a third-party consultant, such as an academic institution, can customize P-CIS Insights dashboards to automate evaluations which test specific hypotheses.
Is P-CIS HIPAA compliant?
Does P-CIS allow customization of dashboards and insights?
Custom P-CIS Insights dashboards can be published and democratized to users throughout the system. End users who log into P-CIS can have access to the custom P-CIS Insights, but only for the people for whom they can see. This means that everyone can analyze their own outcomes from the same dashboard framework – in real time. No further development is needed because P-CIS’s HIPAA compliant data access and security automatically applies democratization. At any level, the dashboards will only display information about the people that the end user can see. System administrators who roll out one dashboard will actuate insights into sub-populations within minutes after launch. Everyone across a system can speak a uniform language about outcomes for shared populations.
What is an example of a customized dashboard for the FFPSA?
For example, the Act calls for evaluations of foster care placement, use of kinship care and improved child well-being. P-CIS can automate this evaluation so that these indicators can be evaluated in real time as youth reach milestones (e.g., yearly anniversaries, final dispositions), as seen in the example. As population outcomes change, the dashboard will track the change, and drop-down filters allow end users to re-run the analysis for specific populations – for High-Throughput Evaluation (HTE).
In this example:
- The WRAP is the service which is most often associated with increases in kinship care when children/youth are placed out of home (OOH).
Can P-CIS deliver dashboard reports weekly to my managers?
Once calibrated, schedules can automate emails of the dashboards which include the latest up-to-the minute outcomes. Program evaluation does not have to require an arduous process of file data exchange to third parties, months or years after children/youth leave the system. Evaluation can happen in real-time so that nimble adjustments can be made to service offerings. The more an agency explores P-CIS Insights, the closer they get to understanding what works for whom.
Tracking Key Indicators
From the Act.
PERCENTAGE OF CANDIDATES FOR FOSTER CARE WHO DO NOT ENTER FOSTER CARE.—The percentage of candidates for foster care for whom, or on whose behalf, the services or programs are provided who do not enter foster care, including those placed with a kin caregiver outside of foster care, during the 12-month period in which the services or programs are provided and through the end of the succeeding 12-month period.
PER-CHILD SPENDING.—The total amount of expenditures made for mental health and substance abuse prevention and treatment services or in-home parent skill-based programs, respectively, for, or on behalf of, each child described in paragraph (2).
Does P-CIS support real time monitoring of cost savings due to prevention services under the FFPSA act?
P-CIS supports real-time cost disposition and cost analysis through linkages to electronic records. P-CIS Insight dashboards can be calibrated to display analytical insights on assessment and program data from P-CIS linked to any other type of data, such as services, billing, cost, staff profile, staff training, etc.
As seen in the example, P-CIS can monitor candidates for foster care who enter or do not enter foster care by their one-year anniversary in real-time. Folding in data from a multitude of other electronic systems, P-CIS can coordinate information to track the exact cost as well as cost savings. When calibrated with these external data sources, P-CIS success-focused artificial intelligence (SF-AI) models learn what services and supports improve outcomes at reduced cost. As seen in this example, we must only recognize cost savings when the diversion from foster care resulted in positive outcomes. Cost savings for poorer care is not the aim of the Act, and P-CIS can help ensure that across the nation, we are uniformly assessing cost savings only as they relate to positive outcomes.
What are the technicalities?
Information can be securely maintained in real-time via a backend integration process with an electronic health record via Opeeka’s application programming interface (API) engine. P-CIS production data is encrypted using Transparent data encryption (TDE) to protect against the threat of malicious offline activity by encrypting data at rest. P-CIS performs real-time I/O encryption and decryption of the data at the page level. Each page is decrypted when it’s read into memory and then encrypted before written to disk. In support of a universal record across a system of care, a person in care can have as many identifiers as needed, supporting unlimited linkage between records in other data sources. The Insights Analytic Space supports direct secure connection to other data repositories stored in MySQL, PostGres, Redshift, Vertica, SQL Server, Oracle, MemSQL, Sybase, Exasol, BigQuery, Snowflake, Presto, Qubole and Athena.
How do we adapt mentally to this new approach to value outcomes for the FFPSA?
“Outcomes are going to change over time. Tools are going to change over time. But the values that we hold really are what give us the impetus to decide. I want to learn about that. I’m going to be a learner. I’m going, I’m going to be somebody who’s going to make a difference. And I think the other piece that’s important on this is when we talk about empathy, which we do a lot. It’s empathy with a capital ‘E’. That means everyone. It’s not selective empathy. It’s not strategic empathy. It really is being empathic for the whole family system and taking everybody along because I think a lot of times empathy has gotten very weaponized, and we have really poor and unworthy poor. And I think we’ve got to have these really uncomfortable discussions about what our values are and build some scaffolding around us as a system so that as we build, we can start to take scaffolding down and we can grow in new areas…And I think it goes to the heart of do I value that individual and that family or I value our outcome? And if they’re mutually exclusive, we got to do something about that.” – Chris Groeber, Professor of Social Work, University of South Florida, Florida Institute for Child Welfare podcast series, Season 3, Episode 2.
The FFPSA calls for evaluation and monitoring to better serve children/youth and families at risk of foster care. P-CIS provides the ability to collect fidelity and outcome measures which funnel into real-time monitoring and evaluation dashboards. P-CIS can link and merge data from other electronic records. Through P-CIS, agencies can unify data across a system of care and learn what works for whom. By doing this in real-time, systems can be nimble and care plans can be adjusted when insights indicate better options would have more probability of success for a family’s unique circumstances. Systems can learn exactly in what circumstances each type of care results in improved child/youth well-being, improving cost savings while simultaneously improving outcomes.
Opeeka’s mission is to support person-centered care, and our goal is to help systems identify what works for children/youth and families in becoming successful.