First Independent Review Finds AI Tutor Helped Second Graders, Not Older Students

Second graders gained from an AI reading tutor, older students didn't

In the WestEd evaluation of outcomes-based contracts, an AI-driven reading tutor produced a clear but narrow win. Second graders who used the software were substantially more likely to reach proficiency on their state's reading assessment than statistically similar classmates who did not. The same program failed to lift scores for older elementary students. Brittany Miller, executive director of the Center for Outcomes Based Contracting, which commissioned the study, said the split outcome shows what works for whom and under what conditions. Miller also argued that schools are often paying for services without knowing if they help, and that not every child needs 30 minutes of every ed tech product each week. The finding gives districts a concrete reason to target the AI tutor to early elementary grades and stop buying it for older ones.

First independent evaluation covered eight districts and 10 interventions

WestEd, a San Francisco-based nonprofit research organization, followed early-adopter districts in California, Florida, Mississippi and Texas from August 2024 to March 2026. The researchers hoped to evaluate 10 tutoring and ed-tech interventions delivered under outcomes-based contracts. Only four could be analyzed with the rigorous cutoff comparison method, which compared students who scored just below an eligibility threshold and received the intervention with statistically similar students who scored just above and did not. Three of those four showed measurable academic gains for students. The fourth, an online tutoring program, was meant to run most of the school year but lasted only about two months, too short to expect improvement. The mix of positive and null findings gives schools a realistic picture of what these contracts can deliver — but also a cautionary one. The study does not prove the contracts themselves caused the gains, because every district also received roughly $80,000 in coaching and technical assistance to design and monitor the contracts. That support, not the financial incentives alone, could explain the improvements, and the evaluation could not separate the two.

Why the cutoff method matters for isolating a program's effect

To separate the impact of one intervention from all the other factors that shape student achievement, WestEd used a regression discontinuity approach. It compared students who scored just below an eligibility cutoff and received tutoring or software with students who scored just above and did not get it. Because the two groups are statistically similar, the difference in later test scores can be attributed to the intervention. This works only when schools follow the cutoff strictly. Of the 10 interventions intended for evaluation, only four met that condition. The method effectively creates a natural experiment, said Sean Tanner, a WestEd researcher on the evaluation team. He finds the outcomes-based contracting model compelling because it builds these natural experiments into the purchasing process.

Six interventions couldn't be judged because schools broke their own rules

For the other six interventions in the WestEd evaluation, clean comparisons were impossible because school districts did what they often do. They filled empty tutoring seats with higher achieving students who were not supposed to receive tutoring, or they failed to get eligible students to show up for after-school programs. One district allowed every first grader to use reading software if they scored anything less than perfect on a kindergarten assessment, which made the treatment group too broad to serve as a valid comparison. These implementation failures are themselves a finding: outcomes-based contracts only work when schools enforce eligibility and ensure the intended students actually receive the service. The researchers could not determine whether any of those six interventions helped or hurt students, because the data could not support a causal claim. Yet the failures also show how difficult it can be to run a controlled experiment in real school settings, where student attendance and administrative decisions often interfere with research protocols.

Schools started holding regular data meetings and tracking attendance

WestEd researchers observed notable shifts in day-to-day practice under the new contracts. Schools and vendors began meeting regularly to review student data, checking whether students actually attended tutoring sessions, and tracking progress toward agreed-upon goals. Teachers and school leaders put more energy into making sure students showed up, and they used the data to make mid-course corrections, adjusting schedules or content when attendance or performance lagged. This level of monitoring is a direct response to the financial incentive, because payment depends on usage and outcomes. However, the evaluation cannot separate the effect of the contract terms from the intensive coaching districts received. Each participating district got roughly $80,000 worth of coaching and technical assistance to help design the contracts, establish outcome measures, monitor progress, and troubleshoot implementation problems. It's possible that the coaching and technical assistance alone, without the financial penalties, would have prompted the same vigilance. The study design cannot answer that question yet, but it does show that outcomes-based contracts are associated with closer attention to implementation.

The $80,000 coaching confounder

Every district in the WestEd evaluation received roughly $80,000 worth of coaching and technical assistance from the nonprofit that commissioned the study. That support helped design the contracts, set outcome measures, monitor progress and troubleshoot problems. As a result, the evaluation tested outcomes-based contracting plus intensive support, not the contracts in isolation. It is plausible that the coaching alone could have led schools to monitor attendance more closely, hold data meetings and make mid-course corrections. The finding of positive academic gains in three interventions could therefore stem from the money spent on support, the contractual incentives, or both. The same organization is now experimenting with cheaper models that provide less than $10,000 per district, which future evaluations will use to tease apart the cause.

No cost savings found, and some vendors lost money

WestEd did not conduct a formal cost-effectiveness analysis, but interviews with participating districts suggested overall spending was similar to traditional contracts. Some districts absorbed extra costs when they failed to meet their own responsibilities, such as ensuring students attended tutoring sessions. At the same time, some vendors reported losing money when outcomes fell short. The financial risk is meant to be shared, but in practice it sometimes fell unevenly. The organization that promoted these contracts and commissioned the evaluation is now testing leaner support models under $10,000 per district to see if districts can get results without the heavy assistance that accompanied the initial evaluation. Whether outcomes-based contracting actually saves money remains an open question, as the first independent evaluation found no evidence of lower spending.

Data systems built for contracts spilled over to other programs

WestEd researchers observed an unexpected spillover effect. Once districts built data systems to monitor tutoring attendance and student progress for outcomes-based contracts, they began using those same systems to track other interventions that weren't covered by the contracts. This infrastructure, originally created to satisfy contractual reporting requirements, became a general tool for instructional management. For example, the same dashboards that showed whether tutoring students were showing up and improving could be applied to other programs, giving school leaders a way to see in real time which services were being used and whether they seemed to help. It suggests that even if a particular contract fails to improve student outcomes, the monitoring capacity it forces districts to develop can have lasting value beyond the original purpose. The researchers noted that contracts can do more than enforce payments; they can change how districts monitor, manage and learn from the programs they buy. This is a concrete benefit unrelated to the financial incentives themselves, and it may help schools make smarter decisions about all their vendors, not just those covered by outcomes-based agreements.

Under-resourced districts may struggle to make outcomes-based contracts work

The WestEd researchers observed that resource-strained districts are likely to have a much harder time implementing outcomes-based contracting. The approach requires staff time to design contracts, build data systems, monitor attendance, and meet regularly with vendors. Districts with limited administrative capacity may not be able to sustain that level of attention. While the eight early-adopter districts in the study had access to $80,000 in coaching each, that assistance is not widely available. Without it, the burden falls on already stretched administrators. Sean Tanner, a WestEd researcher, said the model is compelling because it builds natural experiments into the process, but he acknowledged that not every intervention will work with or without a contract. The question is whether districts can learn quickly enough to adjust.

From healthcare to classrooms: Michigan's $50 million mandate

Outcomes-based contracting has long been used in healthcare, where insurers often pay drug companies less if patients don't improve. Since the pandemic, the practice has spread rapidly into education, especially for tutoring and educational technology. A new Michigan law requires schools to use this type of contract to tap into a $50 million fund for online or virtual tutoring. School districts in Florida, Texas, California and more than 20 other states are also experimenting with the approach, according to the Center for Outcomes Based Contracting, a nonprofit organization that promotes the practice. The idea is to share risk between schools and vendors, and to create incentives for both sides to pay attention to whether an intervention is working. The latest evaluation shows promise but also cautions that the outcomes depend heavily on implementation and support.