Analytics programs that deliver good value to businesses must start with Key Performance Indicators (KPIs).

KPI's are the basic building blocks of dashboards. These metrics provide basic, high-level insights into what is happening when users engage with chatbots/agents.

Individually, a single KPI lacks context. For example, whether an “average session length of 5-minutes” is good or bad is unclear. Both designers and analysts require a robust set of KPIs supported by a larger set of metrics to create a framework for analysis that delivers value.

Dimension Labs provides users with unparalleled flexibility to create a wide range of KPIs and custom metrics suited to measure the unique actions and goals of their chatbot program. Below we explore a long list of performance indicators and metrics available in the platform along with examples of how they can be employed to build frameworks.

Chatbot KPIs

Two basic KPIs for any chatbot are (1) containment rate—the percentage of users successfully served by the chatbot (the escalation rate in the example below is the inverse of this, showing the percentage of users escalated to a live agent); and (2) fallback rate—the percentage of user utterances which the chatbot does not recognize and triggers the fallback or not handled intent.

Basic KPIs: Scenario Analysis

Low Containment & High Fallback
Imagine a chatbot with a low containment rate and high fallback rate. We might assume that this bot is not adequately trained to recognize and respond to user queries.
Low Containment & Low Fallback
In contrast, a chatbot with low containment and low fallback. We assume this chatbot is well trained but unable to fully resolve these issues.
High Containment & Low Fallback: Few escalations plus low fallback generally indicates a high-performing bot. KPIs like Customer Satisfaction Score and Goal Completion Rate may offer important context.

KPI Definitions:

Containment Rate: Number of times a customer requested a human to take over the conversation to resolve their issue or question. The inverse of this metric is referred to as the “escalation rate” or “human takeover rate” (HTR).
💡 A high escalation rate likely indicates that one or more conversation flows does not adequately meet users needs because human intervention is required. Use cases not covered by the bot will correctly escalate to human takeover and these instances may be removed from the score and/or measured separately.
Fallback Rate (FBR): Number of user utterances where the bot failed to understand the user's message triggering the “not handled” intent; also referred to as “missed messages.”
💡 A high FBR could indicate that the flow is not satisfactory and yields low user satisfaction.
Satisfaction Score: Percentage of customers indicating their satisfaction with with the bot interaction, typically gathered via chat box by offering a yes/ no question or a sliding scale (1-5 or 1-10).
Intent Confidence: Confidence score assigned by the NLP model indicates how well the model feels its response matches a user intent. Scores falling below a certain threshold trigger the “not handled” intent (aka fallback)

👉 Confidence scores are determined by the chatbot itself, or more specifically by the NLP model and sent to Dimension Labs as metadata along with message text, timestamp, intent, etc. Signals that some intents may be false positives, that is, utterances are triggering the incorrect response. This may be because there is not enough training data in a given intent which provides a signal that an intent needs to be removed, clarified or trained further. value 0-1 but communicates a deeper signal about potential issues about customers not getting the answers they want, lower containment, upstream indicator

Goal Completion Rate (GCR): Rate at which the chatbot achieves specific goals the company sets. E.g., if the goal is to resolve 50 user requests in a day, and it resolves 40 of them, the bot achieves an 80% GCR.

Use Case-Specific KPIs

SALES

Conversion Rate: Percentage of interactions resulting in desired outcomes or actions.
Leads Captured: Number of users who gave their personal information, like an email address or phone number, during the conversation.
💡 Many businesses use chatbots for lead generation, specifically as a method for posting qualifying questions to vet potential leads and capture their data.
Retention Rate: Percentage of users that return and engage with the chatbot within a specific timeframe.
💡 This KPI can be used to identify if customers are truly becoming part of your funnel (and eventually converting)—but it’s context-dependent. E.g., if the customer returns the next day with the same issue they reported the previous day, the chatbot was ineffective in helping them.

SUPPORT

Average Response Time: Your chatbot will help your support team respond to live inquiries faster, by providing the first point of contact for customers. That will help you cut your average response time.

Chatbot Metrics

Depending on a businesses needs chatbot metrics may be employed as KPIs or provide supporting data which add important context to explain performance and/or contribute to surfacing actionable insights to guide bot optimization.

User Metrics

User Volume: Total users engaging with the chatbot over a given timeframe.
Engaged Users: Users that sent a message within the bot
New Users: Users that have never used the bot before
Bounce Rate: Number of users who visit the page but don't interact with the chatbot (aka unengaged users)
Top Intents by User: Shows the most frequently used intents (purposes or goals) by individual users, helping identify primary reasons for user interactions.
Intent Groups by User: Categorizes user intents into groups based on common themes or purposes, providing insights into broader interaction patterns
Ave. Intents per User: Calculates the average number of intents initiated by each user during interactions with the chatbot, indicating engagement and interaction complexity.
Ave. Sessions per User: Measures the average number of separate interactions or sessions initiated by each user, reflecting user return frequency.
Ave. Messages per User: Calculates the average number of messages exchanged between the user and the chatbot per user, indicating engagement depth.

Session Metrics

Total Sessions: Overall number of chatbot interactions
Engaged Sessions: Interactions where users actively participate or engage.
Ave. Messages per Session: Calculates the average number of messages exchanged between the user and the chatbot per interaction session, reflecting conversation depth.

Message Metrics

Total Messages: Represents the overall count of messages exchanged between users and the chatbot over a specific period, providing an overall engagement measure.
Outgoing Messages: Specifically counts the total number of messages sent by the chatbot to users, assessing proactive engagement and information dissemination.
Incoming Messages: Specifically counts the total number of messages received by the chatbot from users, providing insights into user engagement and interaction patterns.

Journey Metrics

Containment Rate: Percentage of user inquiries handled without escalation
Total Escalations: Number of escalations to a human agent.
Planned Escalations: Number of escalations suggested by chatbot where the conversation flow is designed for human takeover.
Fallback: Number of users/sessions/messages which trigger a fallback response or “not handled” intent.
Looping Behavior: User interactions repeatedly circling back to the same topic

Metrics Scenario Analysis

Low New Users + High Ave. Sessions per User: High ave sessions per user show users are returning to use the bot multiple times.
Fallback Rate + Session Length: If a user chooses to speak to a support agent too quickly, it points toward a potential issue with the conversation flow.

Explore/Learn More

Clarity Score: Confidence level compared with other confidence

Data Slicer/Livechat KPIs

Satisfaction Drivers (Categories + Sentiment)

References:

Chatbot Analytics: 13 Metrics That Every Business Should Track

Mapping chatbot metrics to business KPIs

Fallback Rate

Containment & Escalation Rate

Goal Completion Rate (GCR)

Satisfaction Score