# Chatbot and AI agent analytics

> Chatbot and AI agent KPIs: containment, escalation, task completion, journeys and bot health, with what to ask the agent for each.

A dashboard full of counts ("how many conversations?") is information. Insight comes from questions that start with *why* or *how*: why customers escalate, how they get to a successful outcome, where they give up. This page sets out what to measure for a chatbot or AI agent, how each measure is defined, and how to ask the Dimension Labs agent for it.

Many bots now run on large language models and have no intents to report on. Dimensions measure the same things from the conversation itself, so everything here works for intent-based bots, LLM bots, and a mix of both.


## Where to start

Get the data in, agree the [definitions](#definitions), then work through five areas in this order:

1. **[Define success criteria](#define-success-criteria).** Decide what a good outcome is for your bot and your business, and the targets you'll hold it to.
2. **[Map user journeys](#user-journeys-and-drop-off-points).** The stages from first message to task completion, by persona, and where people drop off.
3. **[Track task completion](#task-completion).** Success versus failure for specific tasks, happy paths and funnels.
4. **[Measure self-service and retention](#self-service-and-retention).** How much the bot resolves on its own, why it falls back, and whether people come back.
5. **[Watch bot health](#bot-health).** The signals that show where to fix the bot first.

## Get the data in

Bot transcripts go in as conversation data with `incoming: true` for the customer and `false` for the bot. Put the end user's id in `userId` on every row, the bot's messages included. See [Conversation data format](/data/conversation-format).

Send what your bot platform knows, because the model reads only the text:

| Field | Why it matters |
|---|---|
| `intent` | Only for an intent-based bot. An object: `{ "name": "TRACK_ORDER", "confidence": 0.94 }`, the same shape in the API and in a JSONL upload. Lets the agent report on intents, intent confidence and the fallback intent directly. See [Intents and events](/api/intents-and-events). |
| A bot name column | If you run several bots, name the bot on each conversation. The model can't tell which bot a conversation belonged to unless you say. |
| `conversationId` | Keeps one conversation together when a human agent takes over from the bot. |
| Handoff or outcome flags | If your platform records a transfer, a goal reached, or a "success" tag, send it as an extra field. It is more reliable than inferring it from text. |
| Survey answers | A thumbs up or down, or a 1 to 5 or 1 to 10 rating, as an extra field. |

Don't send personal information, such as the email address or phone number a user typed, in these fields or as an id. See [Personal information](/admin/security#personal-information).

A live integration is enriched every day. A file of transcripts needs a [backfill](/dimensions/backfill) before it has Dimensions. Enrichment reads the whole conversation, bot turns included, so Dimensions can use what the bot said as well as what the customer said.

## Definitions

Every measure on this page uses the definitions below. Write them into an [analysis plan](/agent/analysis-plans) or the project's `AGENTS.md`, so every report uses the same ones.

### KPIs

Key performance indicators are the high-level view of what is happening when people use your bot. A single KPI lacks context: whether an average conversation length of five minutes is good or bad depends on what the bot is for. Use a small set, backed by the supporting metrics below, so they explain each other.

| KPI | Definition | Notes |
|---|---|---|
| **Containment rate** | Conversations that ended without a human taking over, divided by all conversations | Also called the self-service rate. Use cases the bot is not meant to handle should escalate: remove those from the score or measure them separately. |
| **Escalation rate** | Conversations where the customer asked for, or was handed to, a human, divided by all conversations | 100% minus containment. Also called the transfer rate or human takeover rate (HTR). |
| **Fallback rate (FBR)** | Customer messages that triggered the fallback or "not handled" response, divided by all customer messages | Also called missed messages. A high fallback rate can point to unsatisfying flows and low satisfaction. |
| **Self-service resolution rate** | Conversations that ended without a human and with the customer's need met, divided by all conversations | Stricter than containment: a customer who gave up without a human counts as contained but not resolved. |
| **Completion rate** | Conversations that reached a defined successful end, divided by all conversations | You define the successful end for each task. See [Define success criteria](#define-success-criteria). |
| **Satisfaction score** | Customers who said they were satisfied, divided by customers who answered the survey | Usually a yes or no question or a 1 to 5 or 1 to 10 scale. Only a few customers answer. A [predicted CSAT](/dimensions/predicted-scores) fills the gap for the rest. |
| **Intent confidence** | The 0 to 1 score your bot's language model assigns to its intent match | Below a threshold, the bot triggers fallback. See the note below. |
| **Goal completion rate (GCR)** | Goals the bot reached, divided by the goals you set | For example, if the goal is to resolve 50 requests in a day and the bot resolves 40, the GCR is 80%. |

:::note[About intent confidence]
Confidence scores come from your bot's own language model and are sent to Dimension Labs with the message, in `intent.confidence`. They are an upstream signal. Low or uneven confidence on an intent can mean false positives: messages triggering the wrong response, often because the intent has too little training data. That intent may need to be clarified, split, trained further or removed. Low confidence tends to show up before lower containment and customers not getting the answers they want.
:::

### Reading containment and fallback together

| Containment | Fallback | What it usually means |
|---|---|---|
| Low | High | The bot isn't trained well enough to recognize and answer what customers ask. |
| Low | Low | The bot understands customers but can't fully resolve their issues. Look at what happens after it understands. |
| High | Low | Generally a high-performing bot. Check satisfaction and completion to confirm it is actually helping. |

### KPIs for specific use cases

**Sales and lead generation**

- **Conversion rate:** conversations that end in the action you want, such as a purchase or a booking, divided by all conversations.
- **Leads captured:** conversations where the user gave contact details. Count it from a flag your platform sets; don't send the details themselves.
- **Retention rate:** users who come back to the bot within a set period, divided by all users. Read it in context: a customer who returns the next day with the same issue is a sign the bot didn't help them.

**Support**

- **Average response time:** how quickly customers get a first response. A bot as the first point of contact should bring it down.

### Supporting metrics

These serve as KPIs themselves or as the data that explains a KPI. Most can be computed from the transcripts and their metadata, and the agent writes the query. The rest need a Dimension.

| Metric | Definition | How it's computed |
|---|---|---|
| **User volume** | Users who used the bot in a period | Distinct `userId` |
| **Engaged users** | Users who sent at least one message | Distinct `userId` with an incoming message |
| **New users** | Users who have never used the bot before | First conversation falls in the period |
| **Bounce rate** | Users who saw the bot but didn't interact | Needs page or widget views from outside the transcripts, uploaded as business data. A user who never wrote has no conversation. |
| **Top intents** | The intents users raise most often | `intent.name`, or a contact-reason Dimension for LLM bots |
| **Intent groups** | Intents grouped into broader themes | Ask the agent to group intents or contact reasons |
| **Average intents per user** | Engagement and complexity | Intents divided by users |
| **Conversations** | All conversations with the bot | Distinct `sessionId` |
| **Engaged conversations** | Conversations where the user actively took part | At least one incoming message, or more than a set number |
| **Conversations per user** | How often users come back | Conversations divided by users |
| **Messages per conversation** | Conversation depth | Messages divided by conversations |
| **Conversation length** | Time from first to last message | From `dimensionlabs_timestamp`, in milliseconds |
| **Messages** | All messages in a period, split into incoming (from users) and outgoing (from the bot) | Row count, by `incoming` |
| **Total escalations** | Handoffs to a human | An escalation Dimension, or your platform's transfer flag |
| **Escalation type** | Planned handoffs (the flow is designed for a human), customer requests, and bot failures | A category Dimension: `planned`, `customer_requested`, `bot_failure`, `none_detected` |
| **Looping** | Conversations that keep circling back to the same topic | A boolean Dimension, with a verbatim of the repeated request |

### Reading metrics together

- **Few new users and many conversations per user:** people are coming back to the bot. Check whether they return with new needs or the same unresolved one.
- **High fallback rate with short conversations:** customers give up after the bot fails to understand them. Look at the first message the bot didn't recognize.

## Measure it without intents

For LLM bots, or to go beyond what your bot platform reports, get the inputs to the definitions above from Dimensions:

| Measure | How to get it with Dimensions |
|---|---|
| **Escalation** | A category for the escalation type, plus a verbatim of what the customer said right before. Feeds containment and escalation rate. |
| **Failure** | The bot didn't understand, gave a wrong answer, or looped. A category for the failure type. Feeds fallback for LLM bots: count conversations with `not_understood`. |
| **Unmet need** | What the customer wanted that the bot couldn't do. A short free-text reason the agent can group. |
| **Completion** | Were all of the customer's requests met, some of them, or none? A category. Feeds completion and self-service resolution. |
| **Helpfulness** | Were the bot's answers correct, relevant and clear? An integer score. |
| **Effort** | A 1 to 5 predicted effort score. |
| **Outcome satisfaction** | A predicted CSAT on the outcome only. See [Predicted scores](/dimensions/predicted-scores). |

### An example prompt

This is a starting point to adapt, in the format of the [config reference](/dimensions/config-reference). Run it on a sample first ([Test before you run](/dimensions/test-before-you-run)).

```json
{
  "name": "extract_bot_conversation_health",
  "description": "Whether the bot resolved the customer's need, why it failed or escalated, and the customer's effort.",
  "context": "Input:\nEach record is one conversation between a customer and our virtual assistant. Customer messages and bot messages are both included. A human agent may take over partway through.\n\nRules:\n- Judge from the customer's needs and the bot's answers as written.\n- Ignore speaker labels when judging the customer's words.\n- If there is no signal, return 'none_detected'.",
  "parameters": {
    "type": "object",
    "properties": {
      "request_completion": {
        "description": "Were the customer's requests met by the end of the conversation?\n- 'full': every request was met.\n- 'partial': some requests were met, others were not.\n- 'none': no request was met.\n- 'none_detected': the customer made no request.",
        "type": "string",
        "enum": ["full", "partial", "none", "none_detected"]
      },
      "escalation_type": {
        "description": "How the conversation reached a human, if it did.\n- 'planned': the bot handed off because the flow is designed for a human.\n- 'customer_requested': the customer asked for a person.\n- 'bot_failure': the bot could not understand or resolve the issue.\n- 'none_detected': no handoff.",
        "type": "string",
        "enum": ["planned", "customer_requested", "bot_failure", "none_detected"]
      },
      "bot_failure_type": {
        "description": "The main way the bot failed, if it did.\n- 'not_understood': the bot did not recognize the request.\n- 'wrong_answer': the bot answered, but incorrectly or off topic.\n- 'looping': the conversation repeated the same exchange.\n- 'cannot_do': the bot understood but could not perform the task.\n- 'none_detected': no failure.",
        "type": "string",
        "enum": ["not_understood", "wrong_answer", "looping", "cannot_do", "none_detected"]
      },
      "unmet_need": {
        "description": "In at most 10 words, what the customer wanted that the bot did not provide. If nothing, return 'none_detected'.",
        "type": "string"
      },
      "helpfulness_1to5": {
        "description": "How helpful the bot's answers were: correct, relevant to the request, and clearly worded. 1 = not helpful, 5 = fully helpful. Judge the bot's turns only.",
        "type": "integer"
      },
      "effort_1to5": {
        "description": "How hard the customer had to work to get their need met: repeated questions, rephrasing, extra steps. 1 = effortless, 5 = very hard.",
        "type": "integer"
      },
      "escalation_verbatim": {
        "description": "The customer's exact words just before the handoff to a human, at most 30 words. If there was no handoff, return 'none_detected'.",
        "type": "string"
      }
    },
    "required": ["request_completion", "escalation_type", "bot_failure_type", "unmet_need", "helpfulness_1to5", "effort_1to5", "escalation_verbatim"]
  }
}
```

## Define success criteria

A bot's value shows only when its success is measured against your business KPIs. Align each reporting criterion to one, using the [definitions](#definitions) above:

| Criterion | Measures |
|---|---|
| **User reach and engagement** | Active user rate (how often users come back) and user growth rate (new users over time) |
| **Task completion** | Completion rate and a helpfulness score |
| **Successful journeys** | Journey success rate (journeys that reach their goal without drop-off) and fallback rate |
| **Self-service effectiveness** | Self-service resolution rate and escalation rate |
| **Bot health** | Completion rate, sentiment and escalation rate. See [Bot health](#bot-health). |

For reach, set growth and engagement thresholds from your own history: *"Chart unique and returning users by month for the last year, and tell me what a normal month-on-month change looks like."*

For completion, define what a successful end looks like for each intent or use case. Examples: the user receives a confirmation such as "Thanks for your order!" or "Thanks for signing up", or the conversation carries a "success" tag in its metadata. Use a completion Dimension where no success message exists.

For journeys, map the common journeys and their critical touchpoints. Two questions help: *what conversations are my customers already having?* and *what is the ideal conversation they should have?*

:::note[Two kinds of helpfulness]
Helpfulness can come from customers (a survey question on how helpful the bot was) or from the model (a helpfulness Dimension, as in the [example prompt](#an-example-prompt)). Survey ratings cover few conversations; the Dimension covers all of them. Say which one a report uses.
:::

### Example targets

These are suggested starting targets (Defining success criteria, Dimension Labs docs, October 2025), not industry benchmarks. Set your own from your history.

| Metric | Unit | Example target |
|---|---|---|
| User growth rate | New users per month | 10% increase |
| Completion rate | Conversations ending successfully, of all conversations | 85% |
| Journey success rate | Journeys that reach their goal, of all journeys | 90% |
| Escalation rate | Conversations needing a human, of all conversations | Under 10% |
| Self-service resolution rate | Conversations resolved by the bot, of all conversations | 80% |
| Fallback rate | Customer messages that hit fallback, of all customer messages | Under 5% |
| Helpfulness score | Helpfulness as rated by users | 4.5 out of 5 |

## User journeys and drop-off points

Journeys show the steps customers take to reach a goal in the bot: by persona, where they drop off, and what they said when they did. The agent builds them from the transcripts in time order, using intents where you have them or a step Dimension where you don't.

### The start of the journey

- **Onboarding:** how users interact with the bot from their first message.
- **First interaction success:** how often users get the information or action they need from the first response.
- **Starter messages that lead to success:** which opening messages most often end in a completed task.

*"Group conversations by the customer's first message or first intent. For each group, show the share that reached a successful end."*

### Fallback

- **Questions that lead to fallback:** which customer questions the bot fails on and hands to a human.

*"List the customer messages that most often come right before a fallback or a handoff, grouped by theme, with examples."*

### Task completion paths

- **Successful task completion:** the whole path users take to complete a task, and where they leave it.
- **Completion rates:** the share of users who complete a task from start to finish, shown as a funnel.
- **Drop-off points:** the step where users abandon the task.
- **Transcripts:** the conversations behind a drop-off, so you can read why users left.

*"For the password reset task, show a funnel from first message to confirmation, with the drop-off at each step and five example transcripts from the biggest drop-off."*

### Steps to put this in place

1. **Identify key personas.** Segment users by their characteristics and behavior: new or returning, plan, channel, language. Send these as extra fields so the agent can split by them.
2. **Outline the stages.** Map each stage from onboarding to task completion and mark the key interaction points.
3. **Find the pain points.** Find where users struggle or drop off, and what that costs.

## Task completion

Task completion is the rate at which users reach a successful end point in a conversation: confirming an appointment, making a purchase, reaching the help article they needed. It reports success versus failure for specific tasks, shows pain points and opportunities, and identifies the paths that work.

1. **Define success first.** Use your [success criteria](#define-success-criteria): a confirmation message, a success tag, a completion Dimension, or a satisfaction criterion such as a predicted CSAT of 4 or 5.
2. **Define the task's steps.** The intents, messages or Dimension values that mark each step, in order.
3. **Name the happy path.** A happy path is the journey you want customers to take for this task. Naming it lets reports show how many customers follow it and where the rest go instead.
4. **Ask for a funnel.** The share of users who reach each step, and the share who reach the end.
5. **Put it in a dashboard or report** with the other measures. See [Reports, decks and sharing](/outputs).

*"Using 'predicted CSAT of 4 or 5' as success, build a funnel for the booking task: started, gave a date, chose a time, got a confirmation. Mark the happy path and show the completion rate by week."*

## Self-service and retention

### Self-service

Track the [self-service resolution rate](#kpis), the [escalation rate](#kpis), and a breakdown of the messages the bot didn't understand. Keep the bot's knowledge current for common requests, and keep clear escalation paths for the rest.

*"Show the self-service resolution rate and escalation rate by month, and the top 20 messages the bot didn't understand, grouped by theme."*

### User retention

- **Returning user share:** users who came back to the bot on more than one occasion, divided by all users.
- **New user share:** users who used the bot for the first time, divided by all users.
- **Retention over time:** how users come back, by the month they first arrived.

A high returning share is often healthy: users come back because the bot gives them what they need. Check it against repeat contacts, though. A customer who comes back with the same issue is a sign the bot didn't solve it the first time.

*"Show monthly retention cohorts for bot users. Separately, what share of returning users came back about the same issue within 7 days?"*

## Bot health

A single health score is convenient, but you need the signals behind it to know what to fix. They show where the bot falls short, which issues to prioritize, and how to raise the self-service rate.

### The critical signals

| Signal | What it covers | Broken down into |
|---|---|---|
| **Completion** | Whether the customer's requests were met | Full completion (every request met) and partial completion (only some) |
| **Sentiment** | The emotional tone of the customer's messages | Positive indicators (satisfaction, gratitude, relief) and negative indicators (complaints, frustration) |
| **Helpfulness** | Whether the bot's answers addressed the requests (a Dimension) | Accuracy (correct and relevant) and clarity (clearly communicated) |
| **Effort** | How efficiently the issue was resolved | Response time (from message timestamps) and interaction efficiency (from message counts, or an effort Dimension) |

### Health score and drivers

From those signals, ask the agent for:

- **A health score:** one overall indicator of bot performance.
- **Top satisfaction drivers:** what most often comes with a good experience. Cross contact reason or intent with sentiment and predicted satisfaction.
- **Top dissatisfaction drivers:** the most common reasons for frustration.

:::tip[Define the health score once]
Build your own composite with the agent, for example the share of conversations with full completion, non-negative sentiment and helpfulness of 4 or more. Write the formula into your analysis plan and keep it fixed, so scores compare over time. Report it alongside the signals underneath it.
:::

### Where to improve, and what's working

- **Escalation drivers:** the categories and reasons that send customers to a human, which point to the biggest issues to fix. Rank them by their effect on an outcome, not only by volume: see [How we prove a driver](/causal).
- **Self-service drivers:** the use cases that most often end in successful self-service. Decide whether to count early drop-offs (customers who left before the bot could help) as contained, and say which choice a report makes.

### Ask for it

- *"Build a bot health dashboard: completion, sentiment, helpfulness and effort by week, with the top satisfaction and dissatisfaction drivers. Make it filterable by bot."*
- *"What drives transfers to a live agent? Rank the reasons by their effect on predicted CSAT, not by volume."*
- *"Which use cases have the highest self-service resolution rate, and which have the lowest?"*

## Questions to ask

- *Where does the bot hand off to a human most, and what did customers ask right before?*
- *Which unmet needs come up most often, and which of them come before a cancellation?*
- *Has containment changed since we switched bots on August 17?*
- *Which intents have the lowest average confidence, and what do customers say in those conversations?*
- *Do conversations that loop have lower predicted CSAT than those that escalate straight away?*
- *Turn our success criteria and bot health measures into an analysis plan I can run every month.*