Feedback and calibration of the AI judge

• "Watch said this session was worth watching. Was it?"
• "How often does the team disagree with Watch's worth-watching call?"
• "When a question says 80%, is it right about 80% of the time on our sessions, with our wording?"
• "We reworded a question last month. Is the new wording better calibrated than the old one?"

• Sequential stores each thumb with the stated probability and the wording that the judge used.
• In the product today, only one question gets thumbs: the question with the Worth watching role, on Watch rows. So the advisory and the Calibration page report on that question.
• Thumbs on Ask results (Match) count only in your own score.

• Anyone who can open the project can give thumbs: owners, admins, members and viewers.
• Every role can read the Insights and Calibration tabs.
• Only owners and admins can edit the rubric.

Sequential links every judgment and every thumb to the exact wording that the judge read.

A change to anything the judge reads starts a new wording:

• a question's text or criteria
• its options or levels
• its option keys
• adding or removing a question that is on
• turning a question on or off

A question that is turned off is not part of the wording.

The judge never sees these, so a change to them keeps the current wording:

• question labels
• Summary wording (optional)
• Role
• Watch score %
• question order

There is one exception. Sequential sends an option's label to the judge when its criteria are blank. Sequential always sends a scale level's label.

• After your first check, the header line ends with "Right N of M you checked".
• The Calibration page compares your thumbs with the stated probability. You see that probability in the badge's tooltip.
• If you hover over a thumb, it shows what it records: "Yes, this was right …" or "No, this was wrong …".
• If you click the other thumb, your latest thumb replaces the earlier one.
• You cannot clear a thumb. If you click the same thumb again, Sequential records it again.

• For example: "People disagreed with 6 of 14 checks in the last 30 days — 4 under this wording, 2 under an earlier one."
• The advisory only reports. It has no controls.
• The line sits under the Label field, below the question's key. The key is a short code, for example worth_watching.
• On a list or scale question, a second sentence can name one answer. This happens if at least 60% of the disagreements land on that answer. For example: "Most often when it said «Just browsing»."
• The Insights card of the Insights tab lists every question that has a line. It is the second card on the tab, below Background judging. For example: "Two rules are drawing a lot of disagreement: …".

What the page has	What you see
No checks	"No sessions have been checked yet. Use 👍/👎 on Watch to start building this page's evidence."
Under 30 checks on the current wording	A count only, for example "Worth watching — 11 of 30 checks needed on this wording." No verdict and no chart.
30 checks or more	One of three verdicts, and a bar for each range with at least 5 checks

• A question appears on the page when it has at least one check in the last 90 days.
• Verdicts: each verdict compares the stated probability with a 95% interval around how often people agreed.
  • "is calibrated": it is right about as often as it says.
  • "claims more than it delivers": it is over-confident.
  • "is more right than it claims": it is under-confident. The page flags this as strongly as over-confidence, because under-confidence "can bury sessions the ranking should be surfacing".
• Bars: under the verdict, the page shows one bar for each ten-point range of stated probability that has at least 5 checks. A smaller range reads "too few to chart".
• Earlier wordings: each earlier wording has its own line with its own counts. It gets its own verdict when it has 30 checks. The page never pools earlier wordings into the current wording's figures.
• The "Across all wordings" line is context only.

• Only the Worth watching question collects thumbs today, so this loop can check only that question.
• To fix a bad verdict, reword the question.
• The three tests in Rubric:
  • Can the judge answer it from the session?
  • Does it ask one thing?
  • Does it ask about something the judge can observe, not infer?
• In Try it, pick a Session, then click Run.

• The advice in step 3 applies to every question. To test a new wording for another question, use Try it on sessions whose outcome you already know.
• If only the text on screen reads badly, edit the question's Label or Summary wording (optional). Neither starts a new wording, so Sequential keeps the counts.
• Option and scale-level labels are different: see "What starts a new wording" above.

This example is from Sequential's own testing. The first built-in "Hit an error" question asked: "Judging from events, does a software error stop the user from completing an action?"

On a session with rage clicks and no error data, the judge, under that wording, leaned clearly towards yes. It reasoned from behaviour to a cause it could not see.

The current wording adds "and the session's error counts (…; absent when not measured)". With that, the same session comes back as can't-tell.

• Sessions already judged keep their stored judgment. Watch marks them "judged with an earlier rubric".
• The exception is the replay player's Insights tab, which Watch from m:ss opens. If you open it on an earlier session, Sequential judges that session again under the new wording. It uses the capped on-demand budget.
• The Watch row and its thumbs keep the earlier judgment.
• Every question's Calibration counts start again. The old wording gets its own line.
• A thumb on a session judged before the save counts toward the earlier wording. So the new wording's 30 checks come from sessions judged after the save.

• Without a Worth watching question, the editor warns: "No question has the Worth-watching role — the badge on the player's Insights tab is off." It does not say that thumbs stop too.
• If you remove the question or turn it off, sessions judged afterwards have no thumbs.
• If you only clear its role, sessions under the current wording lose their thumbs too. Rows under an earlier wording that had the role keep their thumbs.
• In both cases, the current wording stops collecting checks.

• Sequential sets these numbers: 30 checks for a verdict, 5 checks per bar, and, for the advisory, three disagreements, 34%, and 60% to name an answer.
• Your score is a count, not a calibration measure. It covers only the current wording plus Ask thumbs.
• The split in an advisory line ("4 under this wording, 2 under an earlier one") shows how much of it comes from an earlier wording.
• A change to Watch score % keeps the wording. It does change which sessions reach the top of Watch, so it changes which sessions people check.
• In Watch and Ask, sessions judged earlier keep their stored judgment. Only the replay player's Insights tab judges a session again, when you open it.