AI Ticket Resolution Needs Proof That the Fix Worked

|

8 Mins

|

Last updated:

|

Last updated:

Share this article


TLDR


  • Write down what proves a request is fixed before an agent is allowed to touch it.

  • Check the result in the system that owns it, and keep that check separate from the action log.

  • Give an unknown result its own active status, a deadline and a named owner.

  • Report verified resolutions next to workflow runs, because they’re two different numbers.

Most AI ticket resolution numbers are counting something narrower than they claim. When an agent marks a request resolved right after its last successful tool call, the metric records that an API accepted a change. That’s a long way from the employee being able to open the app they asked for.


You’ve probably seen the gap in your own queue. The group change went through but provisioning is still running (or quietly failed). The account is disabled but a session in one of their apps is still live, the license went to the wrong Jordan, etc. Every one of those tickets looks identical in a closed-ticket report.


For the service desk, this is a question to settle before the automation goes live. Which system gets the final say, what evidence does the ticket retain, and who takes over when the result is still unknown?


Once software can carry out the fix, somebody has to define what proves the fix worked, and that somebody shouldn’t be the agent.


I believe that definition belongs in the approval for the automation itself. If it isn’t written down before the agent runs, the agent ends up inventing its own meaning of resolved, one ticket at a time. For teams adopting AI ITSM, verification belongs in the workflow design, before the first request reaches the agent.


What Does Resolved Mean for a Software Access Request?


Take the most common case: an employee needs the editor role in an app, and your software access request process adds them to a group in Okta or Microsoft Entra ID. A successful group update tells you the group changed; whether that completes the request depends on what the application does next.


If the app provisions asynchronously, the workflow needs a later look at the actual account and role. That means the provisioning log entry for that user and app, or the app’s own user record pulled by its SCIM or external ID. The green check mark on the group step can’t tell you either of those things, and a push that errors on the app side won’t show up on the step that changed the group.


Write the closure condition at the level of the request: the approved account has the approved role in the target application. Then decide whether this request type also needs the employee to confirm, and what silence means.


That last part is where most designs go vague. If the agent asks “Did this fix it?” in Slack or Teams with a Yes button and a Still broken button, you need a rule for the employee who never answers. Close after two business days with the system check as evidence, or escalate to a person. Either can be right, as long as you pick it for the request type before the agent meets it.


A reliable system check is enough evidence for a narrow change like a group membership. A reported login problem is different, because the only real proof is that the person can sign in.


Keep these records distinct:


Record

What It Establishes

Action result

The requested operation returned a result, with its account and resource identifiers.

Observed system state

A subsequent check found the expected condition in the relevant system.

Employee confirmation

The employee reported that the original problem was fixed, when the process requires it.

Outstanding work

A required condition is missing, failed or still unknown, with someone responsible for resolving it.

Your ticket can carry all four. The action result just can’t stand in for the other three.


The Check Has to Match the Promise


Ordinary automation already separates doing from checking. AWS Systems Manager’s runbook example launches an EC2 instance, waits for it to reach the running state and then asserts that state, failing the step if the instance isn’t running. That’s a standard runbook pattern; nobody calls it paranoid.


An AI agent writes a far more convincing completion message than a runbook does, which is a good reason to tie that message to evidence you can open.


Offboarding shows why the scope of the check matters. Microsoft’s guidance on revoking access says that once an application issues its own session token, the application controls access based on its own authorization policies, and Microsoft Entra ID can’t directly revoke that token. Disabling the account blocks new sign-ins, and that’s all it promises.


So an offboarding request needs a list of in-scope apps and what proves each one is done. That usually means tokens and third-party grants revoked where the app supports it, and no sign-in or API activity in the app’s audit log after the disable timestamp. The directory account can show as disabled while two apps still show as outstanding, and the ticket should say exactly that.


The check also has to look at the right object. A lookup by display name that finds the other Sam Patel (the contractor, when you meant the full-time engineer) is a successful query and a wrong answer. Carry a stable identifier (the Entra object ID, the Okta user ID or the app’s SCIM ID) from the approved request into both the action and the verification. Then a reviewer can follow one target through the whole ticket.


Who Owns the Result When Nobody Knows?


A timeout is an unknown, and you should treat it like one. The operation may have died before it changed anything or finished while the response got lost on the way back, and retrying immediately can turn that uncertainty into a second change.


The Amazon Builders’ Library article on making retries safe with idempotent APIs covers the mechanics: an idempotent operation can be retried with no additional side effects. Adding a user to a group is close to idempotent in most directories. Whether a call that creates an account, assigns a paid license or posts a comment is safe to repeat depends on the API and its idempotency controls. Without those controls, a blind retry can create duplicate resources or repeat the same message.


Ravenna retries temporary step failures automatically (our retry docs describe up to three attempts in total), which is exactly why the integration on the other side needs to be safe to repeat. Check each API’s documented retry behavior before you let anything repeat a call.


So give the unknown result its own active state. Check the target system where you can, respect the app’s normal provisioning delay and set a deadline for handing it to a person. The exception should land in the queue’s on-call channel with the action ID, the request payload and the last observed state attached, so whoever picks it up isn’t guessing whether the change already happened.


Partial completion needs the same treatment. If access was granted but the ticket update failed, a retry has to account for the grant that already happened. If one of four required apps is still unverified, the parent request still has work on it. Parking it in Waiting forever just moves the backlog to a column nobody reviews.


Where AI Ticket Resolution Gets Counted Wrong


Check this in your own reporting first, because every dashboard downstream inherits it. We covered how resolution and deflection rates get measured in our post on autonomous ticket resolution; this is about the evidence underneath those numbers.


It applies to Ravenna too: our status documentation classifies Waiting as active and Done as resolved, with Done described as resolved but possibly still needing verification. A custom sub-status inherits its parent’s meaning, so a sub-status called “Awaiting verification” that sits under Done still counts the ticket as resolved, whatever its name says.


Tickets can also move freely between statuses, so a label on its own enforces nothing. If your numbers should only count verified outcomes, put the verification stage under an active status. Our docs recommend enforcing transitions through workflow logic, and the Status Changed trigger (which can watch for a move to Done) and the Set Status action are what you’d build that from.


Workflow monitoring covers the other half. It separates Completed from Completed with errors, and step logs show resolved inputs, raw outputs and retry attempts. That tells you what ran and what came back, which is the action record from the table above. The outcome still needs its own check.


Which evidence permits the final status change is a decision for the service owner, configured per request type and tested against the integrations you actually run.


When Does the Original Request End?


A verified result is true at a point in time. Access gets restored by another process, a configuration drifts after a clean fix, and none of that means the original ticket should stay open forever.


Give the request a clear completion boundary and give ongoing monitoring its own owner. Record what was checked and when. If a later check finds the condition changed, open or reopen linked work instead of quietly editing history. A fix that never worked and a later change that undid a verified one need different responses, and often different teams.


Pull last month’s closed access requests and check each one against the target app’s audit log. Anywhere the only evidence is a successful action record, write down what observation would have justified the close, then add a timeout and an exception owner. Report verified outcomes next to workflow completions while you make the change, and keep the unknowns in the denominator, because dropping them makes the automation look more reliable without making it any better.


If you want to see how Ravenna’s workflow records and ticket statuses map onto your closure rules, we’re happy to walk through it, or you can start with the status docs linked above.


Frequently Asked Questions


What Is Ticket Resolution?


Ticket resolution is the point where the request behind a ticket is fulfilled and the ticket can close. With AI ticket resolution, an agent interprets the request and carries out the work, so the closure rule needs evidence from the target system before the ticket counts as resolved.


How Does AI Automate Service Desk Workflows?


An AI service desk agent reads the request, identifies the request type and runs the workflow attached to it, such as an approval followed by a group change in your identity provider. The designs that hold up also run a check after the action and route anything unverified to a person.


How Can You Safely Use AI in a Service Desk?


Start with narrow request types where you can define the evidence of success, like a group membership change. Write the closure rule, the timeout and the exception owner before the agent goes live, and keep anything irreversible behind a human approval.


How Do You Measure AI Ticket Resolution?


Count verified resolutions separately from workflow completions, and track how many closed tickets get reopened. Keep unknown results in the total so the rate reflects every request the agent touched.


Modernize and automate your
service desk with Ravenna

Modernize and automate your
service desk with Ravenna