AI judging › Basics
Reading an AI verdict
A verdict is the AI judge's forecast of how likely each answer is. Learn what it means, why "unclear" is not a failure, and how to check it in the replay.
On this page
A verdict is a probability
A verdict is the judge's answer to one question in your rubric. The judge says how likely each answer is.
To see how often a question is right, open Settings → Projects → your project → Calibration (Feedback and calibration).
What a verdict holds
Sequential does not decide what happened in a session.
- For a choice question such as Intent, the verdict is its most likely option, with that likelihood.
- For a yes-or-no question such as Hit an error, the verdict is the likelihood of yes.
The session that you look at can be the one time in ten that a 90% answer is wrong.
What Calibration compares
Calibration compares the stated confidence with your team's 👍/👎, on your sessions and your wording.
- The 👍/👎 on Watch rows rate Worth watching, so that is the one question it has evidence for.
- Sequential records the 👍/👎 on Ask results. They count towards your own score.
Unclear means the judge could not tell
Unclear means that the session's events did not settle the question. The evidence was not there, or it pointed both ways.
- Unclear is an answer. It is not an error and it is not no. Sequential keeps "I don't know" and "I don't think so" apart on purpose.
- Unclear is a reason to look, not to skip. In Watch, Sequential does not drop these sessions (Watch: sessions worth your time).
Why unclear is not no
Unclear does not say that the user failed or that your product broke.
Take Hit an error on a session where someone clicked "Save" again and again, and nothing measured errors. The honest answer is "can't tell". The default question is worded to get that answer.
If Sequential sent a zero for errors that nobody counted, the judge would tend to answer no. The evidence does not support that answer.
In Watch, the sessions that the judge could not place stay separate from the ranking.
Verdicts depend on your events
The judge reads a short text version of the session's events and a few facts. It never watches the recording. If your events do not say something, the judge cannot know it.
Wrong events are worse than missing events. The judge cannot tell a mislabelled event from a true one.
What weak events look like to the judge
- A click with no text or id reads as "element". A page view with no path or screen name reads as "(unnamed screen)" (Events).
- Without a custom event such as
invite_sent, the judge can only infer whether someone finished (Send custom events). - If an event is mislabelled, the judge answers from wrong information, with the same confidence.
How a long session is cut
Sequential does not judge a session with fewer than 3 meaningful events. It cuts a longer session to at most 255 events, by kind:
- The cut drops plain clicks first.
- Then it drops custom events.
- Then it drops identify events.
Errors, rage clicks, dead clicks, form submits and page views survive longest. The Insights tab also keeps u-turns (where someone went back). It keeps them longer than form submits.
To see if a verdict was cut:
- If a Watch row has +N or More, click it to open the details. Hover over it to see "Based on X of Y events".
- The Insights tab shows "Judged on X of Y events." only when its own run was cut.
Missing error counts stay missing
The session's error counts are JavaScript errors, failed requests and console errors, from the recording's DevTools capture (DevTools in replays). If Sequential did not measure them, it never sends a zero in their place.
| What Sequential knows | What the judge reads |
|---|---|
| It counted all of it | The exact count, zero included |
| It found some, but could not count all of it | "at least N" |
| Anything else | Nothing |
When a count is missing
The default questions that use these counts are Outcome, Friction, Hit an error and Product bug. These questions tell the judge that a missing count means "not measured", not "nothing went wrong".
"Anything else" includes these cases:
- The capture was off.
- Your SDK does not report it.
- There is no recording.
- The session is from before the counts existed.
- An incomplete count found none.
The replay's Insights tab never receives these counts. This is one reason why it can disagree with Watch about the same session.
Check a customised rubric
The default cause questions name their evidence: the session's error counts. If a cause question does not name its evidence, the judge reasons from behaviour. That answer can sound certain.
A customised rubric keeps its saved wording. It does not get later changes to the defaults.
Open each cause question in your rubric. Check that it names the error counts in the same words as the defaults:
"the session's error counts (
js_errors,failed_requests,console_errors; absent when not measured)"If a question does not name them, add that text.
An owner or admin can reset the questions to the defaults instead (Customise the rubric).
Reset to the defaults
If you are an owner or admin, you can click Reset to default in Settings → Projects → your project → Insights. This replaces your questions with the defaults.
Why a struggling session can still rank high
Some questions ask about a cause. For example: did an error stop the user, or is something in your product broken?
If a user struggles and nothing measured errors, the question tells the judge that the counts are missing, not zero. With no count, the judge cannot say that an error happened. It also cannot say that no error happened.
The session can still rank high. Worth watching asks whether a product manager should see the session, not why it went wrong.
The judge answers what it is asked.
Check a verdict
- Click Watch in the left rail. If another tab is open, choose the Watch tab.
- On the session's row, click Watch from m:ss. On a wide screen, select the session and click Watch from m:ss in the preview.
- Press play.
- If the moment is wrong, go back a few seconds. Or click the event in the replay's Events tab.
- Compare what you saw with the verdict, not with the Insights tab. It can disagree with the verdict (Session insights).
- If the replay makes the answer obvious but the verdict was unclear or wrong, look at what the events said.
- Use the 👍 or 👎 at the end of the row to say whether the session was worth watching. Your answer counts towards Calibration. It never re-judges the session.
Buttons, moments and what to look for
- After you click Watch from m:ss, the replay opens half a second before that time, with the Insights tab showing.
- In step 6, look for a click with no text, a milestone with no event, or errors that nobody measured.
- If the session has no clear moment, the button reads Open replay.
- If you cannot play the recording now, the button reads Open events. Examples are a session with no recording, a recording that Sequential is still processing and an expired recording. Check the events instead of the replay.
- Watch from counts from the session's first event, not the start of the recording. The replay can start early or late for this reason.
- The Insights tab's moments jump to events too. They come from the tab's own judgment, so they may not match the verdict's moments, or they may be missing.
- On hover, the 👍 and 👎 read "Yes, this was right" and "No, this was wrong".
Limits
- Sequential does not judge a session with fewer than 3 meaningful events.
- Sequential cuts a long session to at most 255 events before the judge reads it. The cut can remove a milestone such as
invite_sent. - Without DevTools capture, questions about errors rest on the error events that your SDK sends.
- In the app today, Calibration has evidence only for Worth watching, because only the 👍/👎 on Watch rows feed it. The 👍/👎 on Ask results do not.
- The Insights tab does not receive error counts.