When Sequential says 90% worth watching, you can check it

How sure is it, really?

You post a session to the engineering channel. Watch put it near the top. An engineer asks: "How sure is it?" You have a percentage. You do not have an answer.

Where the percentage comes from. About 30 minutes after a session ends, the judge reads a text version of it: its events, a few session facts and, when measured, its error counts. It answers each of your questions with a likelihood. It never watches the replay. Worth watching is one of those answers. Watch adds it to the Watch score, with the other weighted answers.

A percentage that you cannot check asks for your trust. Calibration lets you check it.

One sentence that you can show

The sentence quotes the stated percentage, the observed rate and the number of checks. It says one of three things:

• Is calibrated. It is right about as often as it says.
• Claims more than it delivers. It is right less often than it says.
• Is more right than it claims. This under-confidence can push down sessions that the ranking should show near the top.

Green shows "is calibrated". Amber shows both flags, with the same weight.

What it compares. The verdict compares the average stated percentage with how often your team agreed. An average can hide a split. So the tab also draws a bar for each ten-point range that has 5 checks or more. A split of 99% and 51% shows there, not in the sentence. Each bar shows the number of checks, the observed rate and a 95% interval.

How it works

1. 01

   Rate the call.

   On a Watch row, give a thumbs up or thumbs down: was Worth watching right? "Saved" confirms it in place. One person’s thumb on one session is one check. If that person changes their mind, the latest thumb counts.

2. 02

   Let the checks add up.

   Below 30 checks, you see the count. You get no early verdict and no chart.

3. 03

   Read the verdict.

   At 30 checks on the current wording, you get the sentence.

Thumbs measure. They never judge a session again and never change where it ranks. To fix a bad verdict, a person rewords the question.

The rules it holds itself to

• No verdict below 30 checks. Old counts never make up the shortfall. 6 checks now and 200 before show as two numbers, never as 206.
• A margin sized to the sample. The margin depends on the number of checks. At 27 of 30 (90%), the range is about 74% to 97%. At 450 of 500, it is about 87% to 92%. With few checks, only a large gap gets a flag.
• Both ways of being wrong get a flag. Over-claiming and under-claiming get the same amber, at the same weight.
• It names its own bias, every time. "These counts come only from sessions someone checked, and people check what the ranking showed them. Window: the last 90 days."

The hosted preview keeps 7 days of data by default, and deletes checks after about 15 days. So on the preview, the 30 checks must come in that time.

Example: Northlight Outfitters, our fictional demo shop

Northlight’s product lead gives a Watch row a thumbs up. The tab then shows: "Worth watching — 11 of 30 checks needed on this wording."

Before the team works through the top of Watch, Northlight’s agent checks the call behind that ranking.

you    › Can we trust the Watch ranking?

agent  › get_calibration { "days": 90 }
       ← Worth watching, current wording: 11 checked.
         The page needs 30 before a verdict, so 19 to go.

The agent gets the raw counts for the current wording from the same function as the tab. So you can do the calculation again. Your agent reads it over MCP with a project access token, from Claude Code or another MCP client.

Next, an admin rewords the question, tests it in Try it and saves. The new wording starts its own count. The old wording keeps its own line, for example "Your rubric v2, saved 22 September". The two counts are never merged.

Good to know

• We do not publish an accuracy figure. The Calibration tab measures, on your sessions, how often your team agreed with the stated percentage.
• It covers one call today. Only Worth watching collects thumbs. For other questions, run Try it on sessions whose outcome you know.
• No verdict on day one. It needs 30 checks on the current wording, from sessions judged since the last rubric save. Sequential does not judge sessions with fewer than 3 meaningful events.
• It cannot see what nobody checked. A low-ranked session that nobody rated never counts.
• One wording covers every question. Any change to what the judge reads restarts the counts for every question. A built-in rubric update does the same for projects that have not customised their questions. To rename or reorder questions resets nothing.
• You cannot change the numbers. 30 checks for a verdict and 5 checks for a range bar. The tab reads the last 90 days of checks that Sequential still keeps.

Questions