How reliable the labels are
Two validation studies on how consistent and how accurate the model's labels and predicted scores are, and why enrichment is reliable.
Optional. You don't need this to get started. Back to the quickstart
Before you build an analysis on Dimensions, it’s fair to ask two things. If the model labels the same text again, does it give the same answer? And does its answer match what the customer actually said or felt? Dimension Labs ran a study on each.
All figures on this page come from the Agent Validation study (Dimension Labs docs, May 2026) unless a line says otherwise. The method is published in our research.
Repeatable: the same answer on every run
Section titled “Repeatable: the same answer on every run”The first study asked whether the model gives the same predicted score when it reads the same text again, across thousands of conversations.
The study
Section titled “The study”- Data: 9,977 fan surveys from five MLB teams.
- Task: the model read each open-text comment and predicted the 0 to 10 rating the fan gave on the same survey.
- Runs: the same prompt, three independent times, against the same text.
- Prompt: deliberately plain. No examples, no calibration, no domain-specific instructions.
- Model: GPT-4.1.
Results
Section titled “Results”| Measure | Definition | Model |
|---|---|---|
| Consensus | All three runs gave the same score | 80.9% |
| Pairwise exact | Two runs gave the same score | 87% |
| Pairwise tolerance | Two runs within 1 point of each other | 99.9% |
On a 0 to 10 scale, two runs landed within one point of each other almost every time, and all three runs agreed exactly on four conversations in five.
Recoverable: a score for every conversation
Section titled “Recoverable: a score for every conversation”Customers rarely leave a rating. A predicted score fills the gap: the model reads the conversation and estimates the rating the customer would have given. This study checked those estimates against real ratings.
The study
Section titled “The study”- Data: support conversations from Givebutter between November 2025 and April 2026. Of about 70,260 conversations in that window, 3,736 carried a usable customer satisfaction rating.
- Task: the model read each of the 3,736 rated conversations and predicted the customer’s 1 to 5 rating, without seeing the rating.
- Baseline: tone-based sentiment on the same conversations.
- Model: GPT-4.1.
3,736 of 70,260 is 5.3%, fewer than 1 conversation in 18 (calculated from the study’s counts). The other 94.7% have no rating at all. A predicted score is generated for every conversation, so it recovers a satisfaction signal for the conversations that would otherwise have none.
Results
Section titled “Results”| Measure | Definition | Model | Sentiment baseline |
|---|---|---|---|
| Correlation | Agreement with the customer’s own 1 to 5 rating (Pearson r) | 0.47 | 0.36 |
| Low-rating detection | Accuracy separating low ratings (1 to 3) from high (4 to 5) | 91% | 74% |
| Exact match | Predicted score equals the customer’s rating | 66% | not measured |
| Within 1 point | Predicted score within one point of the customer’s rating | 93% | not measured |
The predicted score tracked customers’ own ratings more closely than sentiment did, and separated satisfied conversations from dissatisfied ones more reliably.
Why labeling with a language model is reliable
Section titled “Why labeling with a language model is reliable”Enrichment is a recognition task, not a generation task, and language models are much more reliable at recognition.
| Task type | What it does | Risk of making things up |
|---|---|---|
| Generation | Writes something new: an essay, a summary, code | Higher. The model produces content that didn’t exist before. |
| Recognition | Finds whether a concept is present in existing text, and labels it | Lower. The model matches against text that is already there, though it can still return a value outside your list. See Values you didn’t define. |
Used for enrichment, a model gives you:
- Consistency. One labeler for every conversation, with no drift as volume grows.
- Context. It reads meaning and nuance, not just keywords.
- Reliability. Labels are tied to text that exists, and a verbatim Dimension can quote the exact words behind each label.
What this means for your own data
Section titled “What this means for your own data”These studies used plain prompts. Your own prompts can do better, and you should still check them:
- Test a prompt on a sample before running it on everything. See Test before you run.
- Pair judgements with quotes, so any label can be checked against the customer’s words. See Design custom Dimensions.
- Leave a score empty when there’s no signal, and add a confidence Dimension when conversations vary a lot. See Predicted scores.
- Keep collecting surveys. Comparing predicted and real ratings on the conversations that have both is your own version of the recoverable study. Ask the agent: “For conversations that have a CSAT rating, how closely does the predicted CSAT match? Show exact match, within 1 point, and correlation.”