# Feedback and calibration of the AI judge

> Mark the AI judge's answers right or wrong with 👍/👎. See where your team disagrees, and whether the stated probability is correct on your sessions.

> **In short**
>
> - A 👍 or 👎 records whether the judge's answer was right. It never re-judges a session and never changes where a session ranks.
> - Today, only the question with the **Worth watching** role collects thumbs, on Watch rows. Thumbs on Ask results count only in your own score.
> - The advisory on the **Insights** tab shows questions that people often disagree with, over the last 30 days.
> - The **Calibration** tab checks if a question is right as often as it says. It needs 30 checks on the current wording.
> - If you reword one question, the Calibration counts restart for every question.

## What it is for

Sequential judges each session with enough activity against your project's rubric: the list of questions under **Settings** → **Projects** → your project → **Insights**.

Feedback shows if those judgments are correct on your sessions, and which questions need new wording.

### Questions feedback answers

- "Watch said this session was worth watching. Was it?"
- "How often does the team disagree with Watch's worth-watching call?"
- "When a question says 80%, is it right about 80% of the time on our sessions, with our wording?"
- "We reworded a question last month. Is the new wording better calibrated than the old one?"

## How it works

This page calls a 👍 or 👎 on a judged session a check. Three views read the thumbs:

| View | Where | Whose thumbs | Covers |
|---|---|---|---|
| Your score | The Watch header: "Right N of M you checked" | Yours | The current wording, plus your Ask thumbs |
| Advisory | Under each question on the **Insights** tab | Everyone's | The last 30 days |
| Calibration | The **Calibration** tab | Everyone's | The last 90 days |

### What a thumb stores, and which question gets thumbs

- Sequential stores each thumb with the stated probability and the wording that the judge used.
- In the product today, only one question gets thumbs: the question with the **Worth watching** role, on Watch rows. So the advisory and the Calibration page report on that question.
- Thumbs on **Ask** results (**Match**) count only in your own score.

### Who can give and read feedback

- Anyone who can open the project can give thumbs: owners, admins, members and viewers.
- Every role can read the **Insights** and **Calibration** tabs.
- Only owners and admins can edit the rubric.

> **A new wording restarts the counts**
>
> Sequential tracks one wording for the whole rubric, not one for each question. If you reword one question, the Calibration counts restart for every question. Your score restarts too. An update to the built-in rubric does the same on a project that has not customised it.

### What starts a new wording

Sequential links every judgment and every thumb to the exact wording that the judge read.

A change to anything the judge reads starts a new wording:

- a question's text or criteria
- its options or levels
- its option keys
- adding or removing a question that is on
- turning a question on or off

A question that is turned off is not part of the wording.

The judge never sees these, so a change to them keeps the current wording:

- question labels
- **Summary wording (optional)**
- **Role**
- **Watch score %**
- question order

There is one exception. Sequential sends an option's label to the judge when its criteria are blank. Sequential always sends a scale level's label.

## Give feedback on Watch

1. Open **Sessions** → **Watch**.
2. Open a session from its row.
3. Return to the row. Decide if the session was worth watching.
4. If it was, click 👍 at the right of the row. If it was not, click 👎. **Saved** appears for a moment.

If Sequential cannot save your click, you see "Could not save your verdict". Answer the same question whatever badge the row shows: **Watch now**, **Watch** or **Maybe**, or **Review now** or **Review** for a session whose recording you cannot play now.

### Exact thumb behaviour

- After your first check, the header line ends with "Right N of M you checked".
- The Calibration page compares your thumbs with the stated probability. You see that probability in the badge's tooltip.
- If you hover over a thumb, it shows what it records: "Yes, this was right …" or "No, this was wrong …".
- If you click the other thumb, your latest thumb replaces the earlier one.
- You cannot clear a thumb. If you click the same thumb again, Sequential records it again.

## Read the advisory

Open **Settings** → **Projects** → your project → **Insights**. A muted line appears under a question's **Label** field when both of these are true for the last 30 days:

- at least three checks disagreed with the question
- those disagreements are at least 34% of its checks

### More about the advisory

- For example: "People disagreed with 6 of 14 checks in the last 30 days — 4 under this wording, 2 under an earlier one."
- The advisory only reports. It has no controls.
- The line sits under the **Label** field, below the question's key. The key is a short code, for example `worth_watching`.
- On a list or scale question, a second sentence can name one answer. This happens if at least 60% of the disagreements land on that answer. For example: "Most often when it said «Just browsing»."
- The **Insights** card of the **Insights** tab lists every question that has a line. It is the second card on the tab, below **Background judging**. For example: "Two rules are drawing a lot of disagreement: …".

## Read the Calibration page

Open **Settings** → **Projects** → your project → **Calibration**. For each question, the page checks: when it said 90%, was it right about 90% of the time on your sessions, with your current wording?

A question gets a verdict only after 30 checks on the current wording. Before that, the page shows a count.

The top of the page always reads: "These counts come only from sessions someone checked, and people check what the ranking showed them." So the sample is not random. It is the sessions that Watch showed to people.

### What the page shows at each count

| What the page has | What you see |
|---|---|
| No checks | "No sessions have been checked yet. Use 👍/👎 on Watch to start building this page's evidence." |
| Under 30 checks on the current wording | A count only, for example "Worth watching — 11 of 30 checks needed on this wording." No verdict and no chart. |
| 30 checks or more | One of three verdicts, and a bar for each range with at least 5 checks |

### Verdicts, bars and earlier wordings

- A question appears on the page when it has at least one check in the last 90 days.
- **Verdicts**: each verdict compares the stated probability with a 95% interval around how often people agreed.
  - "is calibrated": it is right about as often as it says.
  - "claims more than it delivers": it is over-confident.
  - "is more right than it claims": it is under-confident. The page flags this as strongly as over-confidence, because under-confidence "can bury sessions the ranking should be surfacing".
- **Bars**: under the verdict, the page shows one bar for each ten-point range of stated probability that has at least 5 checks. A smaller range reads "too few to chart".
- **Earlier wordings**: each earlier wording has its own line with its own counts. It gets its own verdict when it has 30 checks. The page never pools earlier wordings into the current wording's figures.
- The "Across all wordings" line is context only.

## Improve a question

Act when a question is over-confident or under-confident repeatedly, not after one wrong session.

The usual cause of a bad verdict is wording that asks for something the judge cannot see. The judge reads your labelled events, a few session facts and, when they were measured, error counts. It never sees the replay.

1. Wait for a calibration verdict on the current wording. An advisory line is a reason to look, not proof.
2. Check the question against the three tests in [Rubric](/docs/rubric).
3. If the question asks about a cause, name the evidence for that cause.
4. Optional, for owners and admins: test the draft in **Try it** on the **Insights** tab. Each run counts against the project's daily allowance.
5. Click **Save**. The change applies to sessions judged from now on.
6. Keep checking on Watch. Compare the new wording's verdict with the earlier wording's line.

### The three tests and Try it

- Only the Worth watching question collects thumbs today, so this loop can check only that question.
- To fix a bad verdict, reword the question.
- The three tests in [Rubric](/docs/rubric):
  - Can the judge answer it from the session?
  - Does it ask one thing?
  - Does it ask about something the judge can observe, not infer?
- In **Try it**, pick a **Session**, then click **Run**.

### Other questions and label-only edits

- The advice in step 3 applies to every question. To test a new wording for another question, use **Try it** on sessions whose outcome you already know.
- If only the text on screen reads badly, edit the question's **Label** or **Summary wording (optional)**. Neither starts a new wording, so Sequential keeps the counts.
- Option and scale-level labels are different: see "What starts a new wording" above.

### Example: naming the evidence

This example is from Sequential's own testing. The first built-in "Hit an error" question asked: "Judging from `events`, does a software error stop the user from completing an action?"

On a session with rage clicks and no error data, the judge, under that wording, leaned clearly towards yes. It reasoned from behaviour to a cause it could not see.

The current wording adds "and the session's error counts (…; absent when not measured)". With that, the same session comes back as can't-tell.

### What happens to sessions judged before a save

- Sessions already judged keep their stored judgment. Watch marks them "judged with an earlier rubric".
- The exception is the replay player's **Insights** tab, which **Watch from m:ss** opens. If you open it on an earlier session, Sequential judges that session again under the new wording. It uses the capped on-demand budget.
- The Watch row and its thumbs keep the earlier judgment.
- Every question's Calibration counts start again. The old wording gets its own line.
- A thumb on a session judged before the save counts toward the earlier wording. So the new wording's 30 checks come from sessions judged after the save.

## What it needs

- **Judged sessions.** Sequential judges a session about 30 minutes after it ends, if it has at least three meaningful events. Shorter sessions get no judgment and no thumbs.
- **A Worth watching question that is turned on.** Watch rows show thumbs only for this question. Without it, the current wording collects no checks. The editor warning for a missing role does not say that thumbs stop too.
- **Evidence the judge can read.** Mark milestones with [custom events](/docs/custom-events). If your questions ask about errors, turn on [DevTools capture](/docs/devtools).

### Removing the Worth watching role

- Without a Worth watching question, the editor warns: "No question has the Worth-watching role — the badge on the player's Insights tab is off." It does not say that thumbs stop too.
- If you remove the question or turn it off, sessions judged afterwards have no thumbs.
- If you only clear its role, sessions under the current wording lose their thumbs too. Rows under an earlier wording that had the role keep their thumbs.
- In both cases, the current wording stops collecting checks.

## Limits

- Thumbs exist only on Watch rows, for Worth watching, and on Ask results. The replay player's **Insights** tab and a Watch row's expanded view have no thumbs.
- After a rewording, rows judged before the save show blank thumbs, even if you checked them. Sequential keeps your thumb under the earlier wording.
- If you click a thumb on such a row, your score changes on screen. When you reload the page, the change is gone.
- Worth watching is a yes/no question, so the advisory never shows its "Most often when it said" sentence.
- An advisory line counts every wording in its 30 days. So it can reflect a replaced wording.
- Sequential does not judge sessions again in bulk after a rubric change. So the replay and the Watch row can show different answers.
- You cannot change the numbers for verdicts, bars and the advisory for your project.
- If you open the replay player's **Insights** tab on a session judged before a rubric save, Sequential judges it again. This uses the capped on-demand budget.
- If you add, remove, turn on or turn off a question, the Calibration counts also restart for every question.
- You cannot clear a thumb. If you click the same thumb again, Sequential records it again.

### More about the limits

- Sequential sets these numbers: 30 checks for a verdict, 5 checks per bar, and, for the advisory, three disagreements, 34%, and 60% to name an answer.
- Your score is a count, not a calibration measure. It covers only the current wording plus Ask thumbs.
- The split in an advisory line ("4 under this wording, 2 under an earlier one") shows how much of it comes from an earlier wording.
- A change to **Watch score %** keeps the wording. It does change which sessions reach the top of Watch, so it changes which sessions people check.
- In Watch and Ask, sessions judged earlier keep their stored judgment. Only the replay player's **Insights** tab judges a session again, when you open it.

## Related

- [Rubric](/docs/rubric)
- [Reading a verdict](/docs/reading-verdicts)
- [Watch](/docs/watch)
- [Judgments](/docs/judgments)
- [Ask](/docs/ask)
- [Roles and permissions](/docs/roles-and-permissions)