Customise the AI judge's rubric

The default rubric has no questions about your product's own moments. It asks generic questions about a session:

• What did the user come to do?
• Did they get it done?
• How much friction did they hit?
• Did an error stop them?
• Is the session worth watching?

Another example of a question about your product's own moments:

• Did the user reach the bulk-upload step?

Sequential asks your questions of every session it judges after you save.

1. For a verdict or scale, rename the two placeholder answers.
2. Give each answer a one-line criterion.
3. To add more answers, click + option or + level.
4. For a signal or moment, open Criteria (optional).
5. Fill in Yes when and No when.
6. If needed, choose a Role. See Roles and what they drive, below.
7. On a signal or scale, set Watch score %, or leave it empty. See How the Watch score is calculated, below.

The judge reads only some parts of a question. The other parts control how Sequential shows and ranks the answers. See What the judge reads, below. For the other editor controls, see Other controls in the editor, below.

The text version of a session contains:

• Labelled events: pages, clicks, form submits, errors, rage and dead clicks, and your custom events.
• A few session facts, such as duration and device.
• Error counts, when Sequential captured them.

From each question, the judge reads the question text, the criteria, each option's key and each level's label. If an option has no criteria, Sequential sends its label.

These parts never reach the judge: question labels, Summary wording (optional), roles, Watch score % and question order. They only decide how Sequential shows and ranks the answers.

Each role fits one kind of question. Only one question can hold each role.

A question without a role still appears in the Insights tab when it has something to show for that session. Sequential leaves out a signal the judge rated unlikely, or a moment it did not find. It does not show them as a no. So if a question is missing there, the judge gave it a low answer. The judge did not skip it.

The editor warns when no enabled question holds Intent, Outcome, Worth watching or Struggle.

The Worth watching question is the only question that Watch rows show thumbs-up and thumbs-down for. Those thumbs are the evidence that Feedback and calibration works from. The editor warning for Worth watching mentions only the badge, but the thumbs need the question too.

Signals and scales can have a weight. The Watch score of a session is the total of its weighted answers. Watch sorts sessions by it.

The answer is a signal's chance of yes, or how far up a scale the session sits. For each weighted answer, Sequential multiplies the answer by its weight. Then it adds the results.

The default weights are:

Your project's weights are in the Watch score % field of each card. Ask uses the Watch score to choose which sessions to search first.

Sequential does not rescale the weights. The badges on Watch, such as Watch now (Review now for a session whose recording you cannot play now), use fixed cut-offs. A total above 100% puts too many sessions in the top badge. A total well below 100% can put the top badge out of reach for every session.

Before you save, the footer counts the changes. For example: "Unsaved changes · 1 reworded, 1 added — applies to sessions judged from now on".

On Intent, Outcome and Friction questions, Summary wording (optional) sets what each answer says in the summary. On other questions it has no effect.

• Intent: the wording finishes a sentence that the product starts, such as "An identified user arrived on /pricing and …". Write a verb phrase, such as "came to upgrade".
• Outcome and Friction: the wording starts the sentence.

• The keys are hidden at first. Turn on Show keys to see the key under each question's label and beside each option. The key follows the label until you save. After that, the key stays fixed.
• When the rubric is customised, the header shows Customised followed by the version number, who saved it and when.
• If someone else saved after you opened the page, Save fails. The footer says who changed the rubric and when.
• Reload shows their version and drops your unsaved edits.

• A proposal ends after 14 days, or when someone changes the questions after the agent read them. Then Apply says "The questions changed since. Ask your agent again."
• A new proposal from the same agent replaces its open proposal.
• With unsaved edits in the editor, Apply waits: "Save or discard your edits first."
• Each project gets at most 5 proposals a UTC day, from agents and members together, and at most 10 tests on a session. Each test spends from the Try it allowance.
• If Sequential refuses the texts of one agent 3 times in a UTC day, that agent cannot propose again until the next day.
• If Sequential cannot check the texts, it refuses the proposal and the agent can try later.
• On the Agents page, a connection that can propose says "Can propose Insights question changes".

Each save that changes something makes a new version. The Audit log records who changed the rubric, when, and how many questions changed. From 12 October 2026, the History card lists the last 20 versions, newest first, for example "Version 7 · 2 hours ago · Ana". The name is the person who saved. A version that Sequential support saved says "Sequential support". A version that came from an agent's proposal says "Proposed by token <name>, applied by <person>".

Owners and admins see Restore version N on each earlier version. It asks "Save version N's questions as a new version?" Restore saves the questions of that version as a new version. The history keeps both versions, and Sequential does not judge any session again. The new version says "restored version N".

• With unsaved edits, the restore waits: "Save or discard your edits before you restore."
• If someone changed the questions after you loaded them: "The questions changed since you loaded them. Reload the page, then restore again."

Reset to default appears when the rubric is customised. Any saved change makes the rubric customised, even a label or a Watch score %. The reset asks you to confirm. It drops any unsaved edits and counts as a new version.

A customised rubric is a snapshot. When the shipped default improves, projects on the default get the new wording. Customised projects keep their wording until they reset.

Try it judges your unsaved draft on one recent session. Sequential saves nothing. Try it shows the result in the same form as the Insights tab of the player. Each run uses a daily allowance of its own.

• The Session list shows up to 20 sessions from the last 7 days, newest first. Each has at least 3 events.
• The list checks only the newest 1,000 sessions. If the range has more, and fewer than 20 of the 1,000 have 3 events, the list says "Older sessions are not in this list."
• Try it reads the session's stored events, as background judging does.
• Its daily allowance is separate from the allowances of background judging and Ask.
• The footer of each run shows how long it took, for example "1.2 s".
• When the allowance is used up, you see "Today’s judgment budget is used up" with the reset time.
• If Sequential cannot read the allowance, Try it does not run and spends nothing. You see "Preview failed" and "Can't test the rubric right now. Try again shortly.", with Retry. Try again after about 30 seconds.
• Each person can run up to 10 previews a minute.
• Run is off, with a reason beside it, when Try it cannot run: "Insights aren't set up on this server" when the server has no AI key, or "Fix the questions first" when the draft has an error.

The judge answers every question it gets.

• Answerable from the session. The judge sees labelled events, a few session facts and error counts. The session does not contain an answer to "Did they read the documentation carefully?".
• One thing per question. "Did they hit an error and give up?" asks about two facts. No single answer fits both. The default rubric asks them separately, as Hit an error and Outcome.
• About what happened, not a feeling. "Was the user frustrated?" asks for a feeling. "Did they click the same control repeatedly?" asks for something that happened.

If a question asks about a cause, name the evidence for it. Give each option a one-line criterion. Add a "none of these" option to each verdict. See Example: name the evidence, below.

Some edits change what the judge reads. Sequential treats each of these as a new wording of the whole rubric. See Roles, scores and calibration after a change, below.

The first shipped Hit an error question read: "Judging from events, does a software error stop the user from completing an action?"

On a session with rage clicks and no error data, the judge gave a clearly high chance of yes. It reasoned from behaviour to a cause it could not see.

The current wording adds "and the session's error counts (js_errors, failed_requests, console_errors; absent when not measured)". With that wording, the judge answers the same session with can't tell. A session with nothing to suggest an error stays a clear no.

• The judge reads an option's key too. A saved key stays the same when you relabel the option. To change what an option means, add a new option. Do not rename the old one.
• If a question does not mention events or session in backticks, Sequential adds "Judge from events." to its end. So a question that names only the error counts ends by pointing the judge at the events.
• If you name events yourself, as the default wording does, Sequential adds nothing.

Roles. Roles apply immediately in the Insights tab, but not on Watch. A Watch row stores its Intent, Outcome and Friction answers when Sequential judges the session. If you give one of these roles to a different question, only sessions judged after the save show the new question's answer on Watch. Earlier rows keep the old question's answer, sometimes under a raw key or the wrong level name.

Rewording. Renaming a level counts as rewording, because the judge reads that label. The same applies to an option that has no criteria.

Watch score. A Watch score % change re-ranks every judged session, older ones included. A question added later adds nothing to the score of sessions judged before it existed.

Calibration. The Calibration tab, next to Insights, counts each wording separately. See Feedback and calibration. A new wording covers the whole rubric, so rewording one question restarts the counts of every question.

Insights tab. The Insights tab of the player always uses the current rubric. After a new wording, if you open an earlier session there, Sequential judges it again on demand. That uses the project's separate daily on-demand allowance. The tab can then disagree with the Watch row. See Session insights.

• Send custom events named for what your questions ask about, such as invite_sent or plan_change_started. Put the fact your question needs in the event name.
• For questions that name the error counts, you need error capture.
• Sequential drops a custom-event property before judging if a whole part of its key is a personal-looking word, such as name or account.

See What the judge drops or cuts, below.

• The judge reads at most 255 events per session. In a longer session, the judge drops plain clicks first. Then it drops custom events, before any other event.
• Sequential cuts event names to 60 characters. It cuts the properties of an event to 80 characters in total.
• A property is dropped when a whole part of its key is a personal-looking word: email, name, user, account, address, phone, card, token and similar. This includes ordinary keys such as plan_name, step_name or account_type. Use the event name, or a key that avoids those words.
• A click reaches the judge unnamed if it has neither element text nor an element id.
• A page reaches the judge unnamed if it has no URL or screen name.