Chatbot and AI agent analytics

A dashboard full of counts (“how many conversations?”) is information. Insight comes from questions that start with why or how: why customers escalate, how they get to a successful outcome, where they give up. This page sets out what to measure for a chatbot or AI agent, how each measure is defined, and how to ask the Dimension Labs agent for it.

Many bots now run on large language models and have no intents to report on. Dimensions measure the same things from the conversation itself, so everything here works for intent-based bots, LLM bots, and a mix of both.

AI agents

enrichment · one record

Raw text

I asked for a person three times before it let me through. It kept sending me the same help article, which doesn't cover my situation. Twenty minutes to change one address.

Bot conversation · handed off · resolution joined

Dimensions written to the record

• Human-Escalation Attempt3 attempts
• Bot Containmentfailed
• AI Failure Modeloop · repeated answer
• Self-Service Attemptarticle not relevant
• Effort Levelhigh · 20 minutes
• Automation Opportunityaddress change

Each colour links a phrase in the raw text to the Dimensions extracted from it. Hover either side.

Where to start

Get the data in, agree the definitions, then work through five areas in this order:

1. Define success criteria. Decide what a good outcome is for your bot and your business, and the targets you’ll hold it to.
2. Map user journeys. The stages from first message to task completion, by persona, and where people drop off.
3. Track task completion. Success versus failure for specific tasks, happy paths and funnels.
4. Measure self-service and retention. How much the bot resolves on its own, why it falls back, and whether people come back.
5. Watch bot health. The signals that show where to fix the bot first.

Get the data in

Bot transcripts go in as conversation data with incoming: true for the customer and false for the bot. Put the end user’s id in userId on every row, the bot’s messages included. See Conversation data format.

Send what your bot platform knows, because the model reads only the text:

Field	Why it matters
intent	Only for an intent-based bot. An object: { "name": "TRACK_ORDER", "confidence": 0.94 }, the same shape in the API and in a JSONL upload. Lets the agent report on intents, intent confidence and the fallback intent directly. See Intents and events.
A bot name column	If you run several bots, name the bot on each conversation. The model can’t tell which bot a conversation belonged to unless you say.
conversationId	Keeps one conversation together when a human agent takes over from the bot.
Handoff or outcome flags	If your platform records a transfer, a goal reached, or a “success” tag, send it as an extra field. It is more reliable than inferring it from text.
Survey answers	A thumbs up or down, or a 1 to 5 or 1 to 10 rating, as an extra field.

Don’t send personal information, such as the email address or phone number a user typed, in these fields or as an id. See Personal information.

A live integration is enriched every day. A file of transcripts needs a backfill before it has Dimensions. Enrichment reads the whole conversation, bot turns included, so Dimensions can use what the bot said as well as what the customer said.

Definitions

Every measure on this page uses the definitions below. Write them into an analysis plan or the project’s AGENTS.md, so every report uses the same ones.

KPIs

Key performance indicators are the high-level view of what is happening when people use your bot. A single KPI lacks context: whether an average conversation length of five minutes is good or bad depends on what the bot is for. Use a small set, backed by the supporting metrics below, so they explain each other.

KPI	Definition	Notes
Containment rate	Conversations that ended without a human taking over, divided by all conversations	Also called the self-service rate. Use cases the bot is not meant to handle should escalate: remove those from the score or measure them separately.
Escalation rate	Conversations where the customer asked for, or was handed to, a human, divided by all conversations	100% minus containment. Also called the transfer rate or human takeover rate (HTR).
Fallback rate (FBR)	Customer messages that triggered the fallback or “not handled” response, divided by all customer messages	Also called missed messages. A high fallback rate can point to unsatisfying flows and low satisfaction.
Self-service resolution rate	Conversations that ended without a human and with the customer’s need met, divided by all conversations	Stricter than containment: a customer who gave up without a human counts as contained but not resolved.
Completion rate	Conversations that reached a defined successful end, divided by all conversations	You define the successful end for each task. See Define success criteria.
Satisfaction score	Customers who said they were satisfied, divided by customers who answered the survey	Usually a yes or no question or a 1 to 5 or 1 to 10 scale. Only a few customers answer. A predicted CSAT fills the gap for the rest.
Intent confidence	The 0 to 1 score your bot’s language model assigns to its intent match	Below a threshold, the bot triggers fallback. See the note below.
Goal completion rate (GCR)	Goals the bot reached, divided by the goals you set	For example, if the goal is to resolve 50 requests in a day and the bot resolves 40, the GCR is 80%.

About intent confidence

Confidence scores come from your bot’s own language model and are sent to Dimension Labs with the message, in intent.confidence. They are an upstream signal. Low or uneven confidence on an intent can mean false positives: messages triggering the wrong response, often because the intent has too little training data. That intent may need to be clarified, split, trained further or removed. Low confidence tends to show up before lower containment and customers not getting the answers they want.

Reading containment and fallback together

Containment	Fallback	What it usually means
Low	High	The bot isn’t trained well enough to recognize and answer what customers ask.
Low	Low	The bot understands customers but can’t fully resolve their issues. Look at what happens after it understands.
High	Low	Generally a high-performing bot. Check satisfaction and completion to confirm it is actually helping.

KPIs for specific use cases

Sales and lead generation

• Conversion rate: conversations that end in the action you want, such as a purchase or a booking, divided by all conversations.
• Leads captured: conversations where the user gave contact details. Count it from a flag your platform sets; don’t send the details themselves.
• Retention rate: users who come back to the bot within a set period, divided by all users. Read it in context: a customer who returns the next day with the same issue is a sign the bot didn’t help them.

Support

• Average response time: how quickly customers get a first response. A bot as the first point of contact should bring it down.

Supporting metrics

These serve as KPIs themselves or as the data that explains a KPI. Most can be computed from the transcripts and their metadata, and the agent writes the query. The rest need a Dimension.

Metric	Definition	How it’s computed
User volume	Users who used the bot in a period	Distinct userId
Engaged users	Users who sent at least one message	Distinct userId with an incoming message
New users	Users who have never used the bot before	First conversation falls in the period
Bounce rate	Users who saw the bot but didn’t interact	Needs page or widget views from outside the transcripts, uploaded as business data. A user who never wrote has no conversation.
Top intents	The intents users raise most often	intent.name, or a contact-reason Dimension for LLM bots
Intent groups	Intents grouped into broader themes	Ask the agent to group intents or contact reasons
Average intents per user	Engagement and complexity	Intents divided by users
Conversations	All conversations with the bot	Distinct sessionId
Engaged conversations	Conversations where the user actively took part	At least one incoming message, or more than a set number
Conversations per user	How often users come back	Conversations divided by users
Messages per conversation	Conversation depth	Messages divided by conversations
Conversation length	Time from first to last message	From dimensionlabs_timestamp, in milliseconds
Messages	All messages in a period, split into incoming (from users) and outgoing (from the bot)	Row count, by incoming
Total escalations	Handoffs to a human	An escalation Dimension, or your platform’s transfer flag
Escalation type	Planned handoffs (the flow is designed for a human), customer requests, and bot failures	A category Dimension: planned, customer_requested, bot_failure, none_detected
Looping	Conversations that keep circling back to the same topic	A boolean Dimension, with a verbatim of the repeated request

Reading metrics together

• Few new users and many conversations per user: people are coming back to the bot. Check whether they return with new needs or the same unresolved one.
• High fallback rate with short conversations: customers give up after the bot fails to understand them. Look at the first message the bot didn’t recognize.

Measure it without intents

For LLM bots, or to go beyond what your bot platform reports, get the inputs to the definitions above from Dimensions:

Measure	How to get it with Dimensions
Escalation	A category for the escalation type, plus a verbatim of what the customer said right before. Feeds containment and escalation rate.
Failure	The bot didn’t understand, gave a wrong answer, or looped. A category for the failure type. Feeds fallback for LLM bots: count conversations with not_understood.
Unmet need	What the customer wanted that the bot couldn’t do. A short free-text reason the agent can group.
Completion	Were all of the customer’s requests met, some of them, or none? A category. Feeds completion and self-service resolution.
Helpfulness	Were the bot’s answers correct, relevant and clear? An integer score.
Effort	A 1 to 5 predicted effort score.
Outcome satisfaction	A predicted CSAT on the outcome only. See Predicted scores.

An example prompt

This is a starting point to adapt, in the format of the config reference. Run it on a sample first (Test before you run).

{
  "name": "extract_bot_conversation_health",
  "description": "Whether the bot resolved the customer's need, why it failed or escalated, and the customer's effort.",
  "context": "Input:\nEach record is one conversation between a customer and our virtual assistant. Customer messages and bot messages are both included. A human agent may take over partway through.\n\nRules:\n- Judge from the customer's needs and the bot's answers as written.\n- Ignore speaker labels when judging the customer's words.\n- If there is no signal, return 'none_detected'.",
  "parameters": {
    "type": "object",
    "properties": {
      "request_completion": {
        "description": "Were the customer's requests met by the end of the conversation?\n- 'full': every request was met.\n- 'partial': some requests were met, others were not.\n- 'none': no request was met.\n- 'none_detected': the customer made no request.",
        "type": "string",
        "enum": ["full", "partial", "none", "none_detected"]
      },
      "escalation_type": {
        "description": "How the conversation reached a human, if it did.\n- 'planned': the bot handed off because the flow is designed for a human.\n- 'customer_requested': the customer asked for a person.\n- 'bot_failure': the bot could not understand or resolve the issue.\n- 'none_detected': no handoff.",
        "type": "string",
        "enum": ["planned", "customer_requested", "bot_failure", "none_detected"]
      },
      "bot_failure_type": {
        "description": "The main way the bot failed, if it did.\n- 'not_understood': the bot did not recognize the request.\n- 'wrong_answer': the bot answered, but incorrectly or off topic.\n- 'looping': the conversation repeated the same exchange.\n- 'cannot_do': the bot understood but could not perform the task.\n- 'none_detected': no failure.",
        "type": "string",
        "enum": ["not_understood", "wrong_answer", "looping", "cannot_do", "none_detected"]
      },
      "unmet_need": {
        "description": "In at most 10 words, what the customer wanted that the bot did not provide. If nothing, return 'none_detected'.",
        "type": "string"
      },
      "helpfulness_1to5": {
        "description": "How helpful the bot's answers were: correct, relevant to the request, and clearly worded. 1 = not helpful, 5 = fully helpful. Judge the bot's turns only.",
        "type": "integer"
      },
      "effort_1to5": {
        "description": "How hard the customer had to work to get their need met: repeated questions, rephrasing, extra steps. 1 = effortless, 5 = very hard.",
        "type": "integer"
      },
      "escalation_verbatim": {
        "description": "The customer's exact words just before the handoff to a human, at most 30 words. If there was no handoff, return 'none_detected'.",
        "type": "string"
      }
    },
    "required": ["request_completion", "escalation_type", "bot_failure_type", "unmet_need", "helpfulness_1to5", "effort_1to5", "escalation_verbatim"]
  }
}

Define success criteria

A bot’s value shows only when its success is measured against your business KPIs. Align each reporting criterion to one, using the definitions above:

Criterion	Measures
User reach and engagement	Active user rate (how often users come back) and user growth rate (new users over time)
Task completion	Completion rate and a helpfulness score
Successful journeys	Journey success rate (journeys that reach their goal without drop-off) and fallback rate
Self-service effectiveness	Self-service resolution rate and escalation rate
Bot health	Completion rate, sentiment and escalation rate. See Bot health.

For reach, set growth and engagement thresholds from your own history: “Chart unique and returning users by month for the last year, and tell me what a normal month-on-month change looks like.”

For completion, define what a successful end looks like for each intent or use case. Examples: the user receives a confirmation such as “Thanks for your order!” or “Thanks for signing up”, or the conversation carries a “success” tag in its metadata. Use a completion Dimension where no success message exists.

For journeys, map the common journeys and their critical touchpoints. Two questions help: what conversations are my customers already having? and what is the ideal conversation they should have?

Two kinds of helpfulness

Helpfulness can come from customers (a survey question on how helpful the bot was) or from the model (a helpfulness Dimension, as in the example prompt). Survey ratings cover few conversations; the Dimension covers all of them. Say which one a report uses.

Example targets

These are suggested starting targets (Defining success criteria, Dimension Labs docs, October 2025), not industry benchmarks. Set your own from your history.

Metric	Unit	Example target
User growth rate	New users per month	10% increase
Completion rate	Conversations ending successfully, of all conversations	85%
Journey success rate	Journeys that reach their goal, of all journeys	90%
Escalation rate	Conversations needing a human, of all conversations	Under 10%
Self-service resolution rate	Conversations resolved by the bot, of all conversations	80%
Fallback rate	Customer messages that hit fallback, of all customer messages	Under 5%
Helpfulness score	Helpfulness as rated by users	4.5 out of 5

User journeys and drop-off points

Journeys show the steps customers take to reach a goal in the bot: by persona, where they drop off, and what they said when they did. The agent builds them from the transcripts in time order, using intents where you have them or a step Dimension where you don’t.

The start of the journey

• Onboarding: how users interact with the bot from their first message.
• First interaction success: how often users get the information or action they need from the first response.
• Starter messages that lead to success: which opening messages most often end in a completed task.

“Group conversations by the customer’s first message or first intent. For each group, show the share that reached a successful end.”

Fallback

• Questions that lead to fallback: which customer questions the bot fails on and hands to a human.

“List the customer messages that most often come right before a fallback or a handoff, grouped by theme, with examples.”

Task completion paths

• Successful task completion: the whole path users take to complete a task, and where they leave it.
• Completion rates: the share of users who complete a task from start to finish, shown as a funnel.
• Drop-off points: the step where users abandon the task.
• Transcripts: the conversations behind a drop-off, so you can read why users left.

“For the password reset task, show a funnel from first message to confirmation, with the drop-off at each step and five example transcripts from the biggest drop-off.”

Steps to put this in place

1. Identify key personas. Segment users by their characteristics and behavior: new or returning, plan, channel, language. Send these as extra fields so the agent can split by them.
2. Outline the stages. Map each stage from onboarding to task completion and mark the key interaction points.
3. Find the pain points. Find where users struggle or drop off, and what that costs.

Task completion

Task completion is the rate at which users reach a successful end point in a conversation: confirming an appointment, making a purchase, reaching the help article they needed. It reports success versus failure for specific tasks, shows pain points and opportunities, and identifies the paths that work.

1. Define success first. Use your success criteria: a confirmation message, a success tag, a completion Dimension, or a satisfaction criterion such as a predicted CSAT of 4 or 5.
2. Define the task’s steps. The intents, messages or Dimension values that mark each step, in order.
3. Name the happy path. A happy path is the journey you want customers to take for this task. Naming it lets reports show how many customers follow it and where the rest go instead.
4. Ask for a funnel. The share of users who reach each step, and the share who reach the end.
5. Put it in a dashboard or report with the other measures. See Reports, decks and sharing.

“Using ‘predicted CSAT of 4 or 5’ as success, build a funnel for the booking task: started, gave a date, chose a time, got a confirmation. Mark the happy path and show the completion rate by week.”

Self-service and retention

Self-service

Track the self-service resolution rate, the escalation rate, and a breakdown of the messages the bot didn’t understand. Keep the bot’s knowledge current for common requests, and keep clear escalation paths for the rest.

“Show the self-service resolution rate and escalation rate by month, and the top 20 messages the bot didn’t understand, grouped by theme.”

User retention

• Returning user share: users who came back to the bot on more than one occasion, divided by all users.
• New user share: users who used the bot for the first time, divided by all users.
• Retention over time: how users come back, by the month they first arrived.

A high returning share is often healthy: users come back because the bot gives them what they need. Check it against repeat contacts, though. A customer who comes back with the same issue is a sign the bot didn’t solve it the first time.

“Show monthly retention cohorts for bot users. Separately, what share of returning users came back about the same issue within 7 days?”

Bot health

A single health score is convenient, but you need the signals behind it to know what to fix. They show where the bot falls short, which issues to prioritize, and how to raise the self-service rate.

The critical signals

Signal	What it covers	Broken down into
Completion	Whether the customer’s requests were met	Full completion (every request met) and partial completion (only some)
Sentiment	The emotional tone of the customer’s messages	Positive indicators (satisfaction, gratitude, relief) and negative indicators (complaints, frustration)
Helpfulness	Whether the bot’s answers addressed the requests (a Dimension)	Accuracy (correct and relevant) and clarity (clearly communicated)
Effort	How efficiently the issue was resolved	Response time (from message timestamps) and interaction efficiency (from message counts, or an effort Dimension)

Health score and drivers

From those signals, ask the agent for:

• A health score: one overall indicator of bot performance.
• Top satisfaction drivers: what most often comes with a good experience. Cross contact reason or intent with sentiment and predicted satisfaction.
• Top dissatisfaction drivers: the most common reasons for frustration.

Define the health score once

Build your own composite with the agent, for example the share of conversations with full completion, non-negative sentiment and helpfulness of 4 or more. Write the formula into your analysis plan and keep it fixed, so scores compare over time. Report it alongside the signals underneath it.

Where to improve, and what’s working

• Escalation drivers: the categories and reasons that send customers to a human, which point to the biggest issues to fix. Rank them by their effect on an outcome, not only by volume: see How we prove a driver.
• Self-service drivers: the use cases that most often end in successful self-service. Decide whether to count early drop-offs (customers who left before the bot could help) as contained, and say which choice a report makes.

Ask for it

• “Build a bot health dashboard: completion, sentiment, helpfulness and effort by week, with the top satisfaction and dissatisfaction drivers. Make it filterable by bot.”
• “What drives transfers to a live agent? Rank the reasons by their effect on predicted CSAT, not by volume.”
• “Which use cases have the highest self-service resolution rate, and which have the lowest?”

Questions to ask

• Where does the bot hand off to a human most, and what did customers ask right before?
• Which unmet needs come up most often, and which of them come before a cancellation?
• Has containment changed since we switched bots on August 17?
• Which intents have the lowest average confidence, and what do customers say in those conversations?
• Do conversations that loop have lower predicted CSAT than those that escalate straight away?
• Turn our success criteria and bot health measures into an analysis plan I can run every month.