Success factors: What is the role of research and evaluation?

The Promise movement has given rise to a range of research and evaluation efforts that can help stakeholders understand whether programs are achieving their intended goals and build a base of knowledge about what works. Sometimes these efforts are carried out by external evaluators hired by Promise programs, sometimes they are carried out by Promise staff, and sometimes they are the products of independent researchers. Evaluation need not be costly and technical, or conducted by outside experts, but it should be an integral part of any Promise initiative from the beginning.

Research and evaluation resources can be found in multiple places: Statewide Promise programs created by legislatures generally require state agencies to track progress and usage of resources. In Tennessee, for example, the comptroller’s office produces full evaluations every four years and annual updates.¹ The higher education commission also produces annual reports² that track enrollment and other statistics.

Community college–based programs usually rely on their own institutional research or enrollment management personnel to assess the impact of their tuition-free initiatives. Some cross-institutional efforts also support the community college sector by tracking legislation and promoting best practices.³

Community-based programs have the most diverse array of evaluation efforts. Most carry out their own data tracking and may post a data dashboard,⁴ while others may also create a formal evaluation plan, hire outside evaluators,⁵ or partner with academics,⁶ especially those at local universities, to do more formal evaluations.

Information generated through research and evaluation can inform an array of stakeholders, including program administrators and staff, funders, policymakers, and community partners. Such information can reveal the impact a program is having on its target population and generate insights to help improve program delivery. It also can be used to identify effective, high-quality practices that should be scaled up or replicated.

Evaluations also produce data that can help build support for a program. In addition to providing feedback around implementation and program rules, Promise evaluation results have been used to demonstrate student impacts, such as institutional enrollment increases and stronger student and family engagement in higher education. These findings have been leveraged to solicit funding from donors, to build support in the business sector for investing in sector pathways programs or hosting internships, and to garner political support in the state context.

Types of evaluations
Evaluations take different forms depending on their purpose. Some evaluation efforts provide feedback to program administrators, allowing them to improve programming or implementation efforts (these are sometimes known as process evaluations). Others assess the outcomes of a Promise program and may address issues such as who is being served, how students are progressing through higher education, and ultimately what impact the Promise program has on individuals and their communities (these are sometimes known as impact evaluations).

Not all evaluations shed light on the effects of a Promise program. To assess causal impact (whether the Promise program itself resulted in the changes observed), a comparison group or counterfactual is required to answer the question, “What would the situation be if this initiative had not occurred?” The gold standard in evaluation is a randomized control trial (RCT), where a statistically identical control group is monitored to assess the impact of a treatment. RCTs are difficult in the Promise arena, where programs are designed to reach large cohorts of students; however, when resources are limited and Promise programs are being rolled out slowly (in a pilot phase or at a limited number of schools), randomization is a possibility. Evaluators have used quasi-experimental strategies to assess the causal impact of Promise programs. Causal research designs can help explain cause and effect and thus predict outcomes. However, such rigorous approaches are not always needed to produce useful feedback and demonstrate effectiveness. Sometimes it makes sense to simply track changes in the number of students served or the number of services delivered. Other times, interviews and focus groups can be useful in understanding how implementation is proceeding and how it can be improved.

Launching an evaluation
Evaluation is not something that should come late in the process as a “secret sauce” added at the end to reveal how an initiative has performed. Rather, evaluation is a tool through which stakeholders can better understand their work and create, review, and modify interventions in real time to best meet program goals.

Ideally, planning for evaluation will begin during the design phase of a Promise program. Evaluators and researchers can assist stakeholders in identifying goals, metrics, and timelines, and establishing data collection procedures that are implemented from the start. (For example, due to federal privacy protections, students and families must consent to having their data used for evaluation purposes, and such consents are easiest to obtain if built into the Promise application process.) While stakeholders may benefit from consulting or contracting with a third-party evaluator or researcher outside the Promise organization, evaluation efforts can be carried out by program staff members themselves. Any evaluation effort will be most successful if stakeholders understand the value and purpose of tracking data and examining processes and outcomes and buy into the evaluation process from the beginning.

Knowing your starting point is essential. Evaluation must reflect a shared understanding of program goals: What is the need the program is trying to meet, and how is the initiative expected to meet that need? Evaluators and program administrators must also understand the population they are serving: What kind of interventions are likely to be successful in which contexts? The broader ecosystem should also be part of formulating goals—a provider scan is useful so that services (e.g., success coaching, mentoring, pathway supports, etc.) are not duplicated. Establishing a system to collect baseline data is also helpful so that evaluators can establish a pre- and post-intervention analysis, if needed.