What guards the number

In 2020 Coursera invested in Rosalyn and asked us for one thing: cheap proctoring at scale. The exams inside Coursera for Campus were low-stakes and the volume they talked about was in the hundreds of thousands. Our fully loaded cost per exam was three or four dollars. That had to come down by a lot, and for about a year, bringing it down was the main thing the company worked on.

There were three ideas on the table: analyze the video at low resolution, analyze only random parts of it instead of all of it, or suppress events so the AI flagged less. We tried the random sampling first. The cost came down.

Then we red-teamed it. We had people sit exams and cheat in ways we knew about, and watched what the system caught. It missed violations. A violation is a short moment inside a long recording, and if you only look at random slices of the recording, some of those moments fall between the slices. We found the misses by having someone watch the whole video, which was the thing we were trying to stop paying for.

That was the first time I saw what happens when cost sits inside the target. The team was doing what we had asked, which was to bring cost down, and the fastest way to bring cost down is to look at less. Once you accept looking at less there is no floor. If random slices are fine, single frames are cheaper still. Every step down looks like a win on the number you are steering by, and the customer gets a cheater who passed.

We kept the cost target. What changed was that a cost cut only counted if it survived the red team. Under that rule, the thing that worked was two models instead of one. The first is a cheap sensor. It does not try to understand what is happening in the video. It looks for spikes and deviations, anything that looks different from what came before, and it is cheap enough to run many copies of on one machine. When the sensor sees a spike, it hands that stretch of video to the second model, the expensive one that does the real analysis. We kept some random sampling on top of that. It did the job.

The second piece was re-architecting the platform so one instance could process many video streams at once without losing quality, which meant rebuilding video ingestion too, since ingestion was its own cost center. Cost per exam ended up under thirty cents, and capacity went from twenty concurrent exams to ten thousand. We got there by holding security fixed and cutting cost around it. The year we tried it the other way around, we got a product that missed cheating.

Two numbers at the top

Nobody at Rosalyn argued for a composite metric. What happened was less deliberate than that. We had a rough idea of a north star: an exam is delivered, it completes, the review happens, the customer accepts the result. Call it close rate. Then we had cost, which after Coursera was the thing we were working on, and the obvious question was how to get cost in there.

We ended up tracking it as a second top-line number next to close rate. That did the same damage a composite does. We had two metrics and no north star. Whatever we were steering by, it was no longer the close rate, and nobody had decided that.

It took a lot of going back and forth to land on the structure I would use anywhere now. The north star is still defensible close rate. The exam is delivered, it completes, every flag gets a decision from a human proctor, and the institution records a verdict. If any of those fails, the session does not count.

Cost is a constraint. It has a ceiling and the team has to stay under it. Nobody gets rewarded for getting further below the ceiling than the ceiling requires, because the ways to do that are the ways we already tried. Student experience is another constraint, a hard one, because you can produce a clean record from an exam students hate and the close rate will not show it.

Security is the falsifier, and that came straight out of the Coursera year. We already ran red-team sessions as a security practice. The change was giving them a veto: if close rate goes up and the red team finds we are catching less, the close rate is wrong, and we stop steering by it until recall recovers. That is different from having two north stars. Two top metrics get traded against each other and the organization quotes whichever one is up. A veto can't be averaged away.

The input that isn't ours

Of the four things a session needs before it counts, the hardest to get accepted was the last one, the institution recording a verdict. The first three are on us. The verdict is on the customer. The AI raises a flag, a human proctor confirms it, and then it sits in a queue waiting for a professor, and a lot of professors would not look at it. If a session came back with a lot of flags they would ignore the queue, or accept everything as is without opening it. Neither of those is a verdict.

We set up a reminder: if a record had been sitting unreviewed for three days, the reviewer got notified. That improved the review rate a lot. It is still a challenge, and we are rebuilding the review portal now so a reviewer sees every confirmed event from a session in one place and can move between them, instead of reviewing events one at a time with the audio playing. It is mostly a UX change. But the verdict is the only input that depends on someone outside the company, and it is the one that decides whether the north star is real.

Cost as a north star

Until this week I would have said there is no business where cost should be the north star. I got pushed on that while writing this, with Costco, Southwest, and AWS as the counterexamples, and I think I was wrong. Cost can be the target. If Costco's north star is bringing the cost of goods down, that can work, but then there have to be hard constraints around quality and standards, and a falsifier: someone checks the goods against the standard, and if what is on the shelf is low quality, the cost number stops counting until that is fixed. Same for Southwest and for AWS pricing. What does not work is cost at the top with nothing guarding it. That is what we ran for a year at Rosalyn, and it produced a cheaper product that missed cheating.

If I had one thing to teach a new head of product about metrics, it is that they are harder to get right than they look, and the right answer is sometimes counterintuitive. Be deliberate about the north star and about anything composite, because that choice sets the tone for the whole product. When you catch yourself saying there are two north stars, or asking how to fit some metric into the north star, stop. Not everything has to fit. Some things that feel like they belong at the top are constraints. Some are falsifiers. Have that frame in mind.