"Resolved" by a Chatbot Often Means the Customer Left

A customer asks a retailer's chatbot where a refund is. The bot shows a help article. The customer closes the window. Two days later the customer phones, waits, and asks the same question of a clerk. The dashboard says the first contact was resolved. The clerk's log says the problem took two contacts. Nobody lied. The two systems count different things.

This is a constructed case, not a reported one. I use it to fix the idea. My thesis is narrow. Firm-reported chatbot "resolution" figures cannot be read as resolution, because the definitions I could read count a session that ended and print no repeat-contact rule. The size of the gap is not published. I can show how it works and what range it could take. I cannot measure it from outside.

The question

When a firm says its bot "resolved" 60% of conversations, what must be true for that to mean 60% of problems were solved? The customer must not return on the same issue, by any channel, inside a stated window. I call this the repeat-contact rule. For every claim I ask: resolved for whom, and for how long?

Data and where it came from

I read vendor documentation and firm announcements. I saw no call or chat content. Public data on this is thin, and firms publish favourable counts. Four vendor sources carry the weight:

• Intercom's help page on Fin resolutions [1].
• Zendesk's help pages on resolution states and on AI agent billing [2][3].
• Ada's documentation on resolution rate [4].
• Klarna's February 2024 press release. The page text did not load for me, so I read its figures through search summaries of the release [5].

Two Gartner surveys give the customer side [7][8]. One report on Klarna's later change of course gives a firm's own view of quality [6]. A summary of the CFPB's 2023 spotlight gives the regulator's view of "doom loops" [9]. Where a page did not state a rule, I write "not stated on the page I read". That is not proof the rule does not exist elsewhere.

Method

Step one: rewrite each claim as a one-line definition. Step two: ask four questions of each line. Does it count customers who leave? What window does it use? Does it check for a return by another channel? Does it need the customer to say the problem is solved? Step three: show with arithmetic what a seven-day repeat rule would do to a reported rate.

I computed step three by hand, without the Lab. The formula is below. No firm supplied the inputs. The numbers are illustrations, not estimates of any firm.

Result: four definitions in one line each

Intercom Fin. A conversation counts as resolved when the customer confirms the answer, or exits "without requesting further assistance" (an assumed resolution). If the customer later returns to the same conversation for more help, the resolution is deducted and not charged [1]. Asking for a person counts as asking for further help [1]. Silence counts as success. A return counts against it only if it comes back to the same conversation. A fresh phone call, or a new chat thread, is a different record. The page I read states no seven-day rule and no cross-channel match.

Zendesk. Zendesk bills for requests an AI agent resolved with no handoff to a human [3]. Its reporting has a "Resolved" state: a conversation that "ended with a meaningful resolution and no further questions from the customer" [2]. It also has an "Undefined" state, "often due to drop-off" [2]. Resolved means the bot saw no further question. The page I read states no window and no rule for a return on another channel.

Ada. A conversation is resolved if the bot was relevant, accurate, safe and contained, meaning no handoff to a human [4]. Idle chats end after 24 hours on web chat and after 72 hours in email [4]. Escalated conversations are marked Not Resolved at once [4]. Ada is the most careful of the four on quality, because it grades relevance and accuracy. Even so, the docs state no repeat-contact rule. A customer who got a wrong answer, left, and phoned next week can still sit in the resolved column if the grade at the time was "accurate and contained".

Klarna. The February 2024 release says the assistant handled two-thirds of chats in its first month, with 2.3 million conversations, and that errands were resolved more accurately, "leading to a 25% drop in repeat inquiries" [5]. That is the only claim in my set that mentions repeats. It prints no window and no baseline in the summaries I read. A 25% drop from what level? Inside how many days? By which channels? Without those, I cannot turn it into a resolution rate. In May 2025 the CEO said, as reported secondhand, that cost had been too dominant a factor and that quality was lower, and the firm began hiring human agents again [6]. I do not claim that proves the 2024 repeat figure was wrong. It shows that one headline number did not capture quality.

What the four have in common: each measures the end of a bot session. Ada and Zendesk add a judgement about that session. Only Intercom has any rule for a later return, and it covers one conversation thread. This is a claim about the definitions I read. It is not a claim about the firms' internal reports, which I cannot see.

What a customer survey says

Gartner surveyed 5,728 customers in December 2023. Only 14% said their issue was fully resolved in self-service [7]. For issues customers rated very simple, the figure was 36% [7]. These are customer reports, across all self-service, not a vendor bot's session count. The two numbers do not measure the same thing, so I do not subtract one from the other. They show that a customer's answer to "was it solved?" can sit far below a session count. In the same survey family, 60% of respondents named "harder to reach a human" as their top worry about AI in service [8]. The CFPB's 2023 spotlight describes loops with no path to a person [9]. I treat these as context on the path to a human, not as evidence of any vendor's rate.

The arithmetic of a repeat rule

Let $R$ be the reported resolution rate. Let $r$ be the share of "resolved" sessions whose customer comes back on the same issue inside the window, by any channel. Then the repeat-adjusted rate is:

$$R_{adj} = R \times (1 - r)$$

And if a share $p$ of attempts fail and the customer keeps trying until the problem is solved, the expected contacts per solved problem are:

$$\text{contacts per solved problem} = \frac{1}{1-p}$$

The table shows $R_{adj}$ for a reported $R$ of 60%. The values of $r$ are chosen by me to show the slope. They are not observed.

Return share r	Adjusted rate	Points lost
0%	60.0%	0.0
10%	54.0%	6.0
20%	48.0%	12.0
30%	42.0%	18.0

At $p = 0.25$, the second formula gives 1.33 contacts per solved problem. At $p = 0.5$ it gives 2.0. A bot session followed by one phone call is two contacts for one problem. Any dashboard that scores the first contact green and the second contact as a new, separate case reports one success and one fresh case. The problem is one. The count of contacts is two.

The honest range for $r$ is unknown. My prior is wide: I put 0.05 to 0.30 as a plausible band for chat bots in e-commerce and finance, with no data behind the edges. If a firm published $r$ for a seven-day window, I would replace this guess at once.

Sensitivity: which assumption moves the result most

Three assumptions matter. In order of effect:

1. The window. A repeat that arrives on day 10 is invisible to a 24-hour rule. The vendors above use windows from hours to 72 hours for idle sessions [4], and some state none. A refund question may take a week to come back as a complaint. The longer the window, the lower the adjusted rate. This is the largest lever, and it is the one firms choose themselves.

2. Cross-channel matching. Intercom's deduction applies to a return to the same conversation [1]. If most returns arrive as a phone call, the deduction misses them. Matching needs a customer ID and an issue tag across channels. Most small firms do not have both. This moves $r$ more than any other measurable thing, because it decides whether returns are seen at all.

3. Customers who never return. This runs the other way, and it is where my own method fails. A repeat rule catches only the customers who come back. A customer who gave up and left stays in the "resolved" column. So $R_{adj}$ is an upper bound on true resolution, not the truth. Some people also call twice for good reasons, such as a second question or a change of plans. My rule would treat them as failures. An issue tag helps here, but it does not remove the problem. A satisfaction score does not fix either gap, since it measures how the customer felt and not whether the problem ended.

One more caution. A faster, cheaper bot may still be a good thing even if its true resolution rate is lower than reported. The question is whether the cost per solved problem fell, not the cost per contact. I have no cost data to test that here.

What I changed my mind about

I started with the view that most vendor definitions would count silence as success. That holds for Intercom's assumed resolution [1]. It does not hold for Zendesk and Ada in the same way, since both add a judgement of the session [2][4]. So the claim "most count a session that ended" is true, but the better wording is: all four measure the session, and one tries to price a return. My confidence that firm-reported rates overstate resolution stays near 0.6. The new evidence did not move it up or down. It showed the overstatement has two causes, silence and missing windows, and they differ by vendor.

The count

One refund question. One bot session counted as resolved. One phone call two days later. That is two contacts for one solved problem, and the dashboard shows one success. If a firm printed its seven-day return share beside its resolution rate, I could replace my illustration with a measurement. Until then the honest figure is a range: 1.0 contacts per solved problem if nobody returns, and 2.0 or more for every customer who does. What would change my mind is a firm publishing the share of "resolved" bot sessions followed by a human contact on the same issue within seven days, and finding it near zero.