Denver Public Schools pledged to pay tutoring vendors based on their results. Did it work?

The concept of outcomes-based contracting is catching on at a time when school districts across the country have more cash to spend and bigger gaps to close.

Pandemic-era disruptions caused many students to miss key lessons, which prompted the federal government to invest billions of dollars of COVID-19 relief funding in America’s schools.

Tutoring quickly emerged as a leading research-based strategy to catch students up — especially high-impact or high-dosage tutoring, which DPS defined as 36 hours per student.

Colorado lawmakers set aside nearly $5 million in state funding in 2021 for grants to school districts to set up high-impact tutoring programs, and the State Board of Education pumped even more federal COVID relief aid, known as ESSER, into the program.

Denver Public Schools, the state’s largest district, applied for the grants and won. The tutoring began in fall 2021 and ramped up last school year when DPS signed contracts with two companies: Cignition and University Instructors. But the program was still pilot-size, serving only about 1,500 students total, or about 2% of all students in DPS.

Younger students made less progress

University Instructors struggled the most to meet the benchmarks in its contract.

In the 2022-23 school year, the Virginia-based company provided in-person literacy tutoring to DPS students in kindergarten through third grade. Its contract was for a maximum of $1.2 million: $900 per student in base pay with the possibility of $1,500 per student in payments based on hitting target outcomes.

The outcomes were based on the mechanics of reading: Did students’ fluency improve, as measured by a test called iStation? How about their vocabulary or phonemic awareness?

The answer for many students was no — or at least not enough to meet the benchmarks in the contract. For example, about half of the 641 students tutored by University Instructors met the benchmark in fluency, but only 17% met the benchmark in vocabulary, Thompson said.

University Instructors will likely be paid about $826,000, or about 68% of the maximum in its contract, according to calculations by Thompson’s staff.

The company did not respond to messages seeking comment for this story.

Staffing challenges contributed to the results, Thompson said. University Instructors struggled at times to hire qualified local tutors and provide substitutes when tutors were out, she said.

Another hiccup was more technical. Not all DPS schools use the iStation test that University Instructors’ target outcomes were based on. Thompson’s staff tried to approximate whether students who took other tests met the benchmarks, but she said that wasn’t always possible.

Online tutoring was more successful

Cignition fared better. District records show DPS paid the California-based company $1.25 million to provide online math tutoring to students in third through eighth grade in 2022-23. Cignition’s contract with DPS was for up to $1.3 million, and the company served 924 students.

Cignition had four outcomes it was trying to achieve: two based on students’ confidence about math, as measured by surveys before and after tutoring, and two based on students’ academic growth, as measured by test scores before and after tutoring. The company was paid a base rate of $720 per student and could earn $940 per student on top of that if it met all targets.

In an interview, Cignition provided a detailed breakdown of its results. The majority of students reported higher confidence, with as many as 89% meeting one of the survey-based benchmarks. Fewer students — 72% — met the academic benchmarks, the company said.

Michael Cohen, founder and CEO of Cignition, said he’s proud of the outcomes.

“We care about quality,” he said. “We’re there to help their students that are struggling the most. Some of those students are really, really struggling, and we do everything we can for every student to bring them up as far as they can possibly get in that school year. There’s going to be a range. Not every last one will get to the highest possible grade.”

Unlike University Instructors, Cignition did not struggle with staffing, according to both the company and DPS. Its model calls for one tutor, who can live anywhere in the country, to work online with a group of four students, giving that group undivided attention.

But Cignition did report issues with student attendance and schools occasionally canceling virtual tutoring sessions. While DPS was aiming to provide students with at least 36 hours of tutoring, Cignition said 50 hours is the gold standard. Only about 10% of DPS students logged 50 hours, the company said. About half of the students logged 25 hours.

At a time when other school districts across the country have had trouble with external tutoring companies, the state grant allowed DPS to try high-impact tutoring relatively risk-free — an opportunity that Thompson said will inform the district’s tutoring strategy going forward.

“Because of the grant, we were able to try these things and learn what works and what doesn’t,” she said. “Now as we plan for what tutoring will look like with Denver Public Schools’ money, we can think about all the things we learned and do it differently.”

One aspect DPS will likely keep, Thompson said, is outcomes-based contracting. While the concept has been around for years in industries such as health care and construction, it’s new in K-12 education, with about 13 school districts actively participating, said Brittany Miller, the director of outcomes-based contracting for the Georgia-based Southern Education Foundation.

Before Miller worked for the foundation, she worked for DPS and helped set up the outcomes-based tutoring contracts. The benefit, she said, is that school districts have a tangible way to judge whether the results are worth the millions of dollars they spend on external vendors.

“There is a lack of infrastructure in K-12 education, particularly in the procurement process, to say, ‘After we spent these funds, what happened for kids?’” Miller said. “This shores up a lot of that.”

Miller said outcomes-based contracting benefits vendors, too, because it sets clear expectations rather than the fuzzy goals that companies sometimes complain about. It also gives the companies the opportunity to earn more money for good performance.

Toni Rader, vice president of learning quality and operations for Cignition, said the company has been doing outcomes-based contracts with districts since 2021.

“We love to do outcomes-based contracts,” Rader said. “It’s helpful for all parties involved, because it makes it clear what we’re shooting for.”

As for DPS, its state grant goes through this school year. But Thompson said the dollar amount is much lower this year, and there are new restrictions. DPS will have just $400,000 to spend, and only on middle school math tutoring, for which the district will request proposals soon.