Designing a Reliable AI Customer-Support Workflow.

Giving an LLM access to financial workflows creates a difficult engineering boundary: the model must understand the customer’s request without becoming the authority that moves money. Using a duplicate-charge refund as a case study, this article designs a workflow that combines AI interpretation with deterministic validation, durable execution, idempotency, human approval, observability and state-based recovery

An AI support assistant can answer product questions with a degree of flexibility. A refund request is different.

The moment a system can retrieve private account data, interpret company policy, and initiate a financial action, an incorrect answer is no longer merely a poor conversation. It can expose customer data, violate policy, create duplicate payouts, or leave money in an ambiguous state.

Consider this apparently simple request:

“I was charged twice for order #2841. Please refund the duplicate payment.”

A language model can understand the sentence, but it should not independently decide that two charges exist, select a payment transaction, calculate the refund, or call the payment gateway. Those responsibilities belong to authenticated, deterministic systems.

The engineering challenge is therefore not simply to make the model more accurate. It is to design a workflow in which the model is useful without becoming the source of financial truth.

This article develops that workflow from end to end. It covers:

• The boundary between probabilistic reasoning and deterministic execution
• Retrieval and validation of policy, order, and payment evidence
• Retry policies for reads, model calls, and financial writes
• Safe degradation when dependencies fail
• Durable refund operations and idempotent execution
• Observability, audit evidence, and recovery from ambiguous states

The central principle is simple:

The LLM may interpret and propose. Authoritative systems verify, approve, execute, and record.

1. Start by defining the trust boundary

An LLM-based workflow combines two fundamentally different kinds of components.

Probabilistic components are useful where language and interpretation are involved. They can classify intent, summarize evidence, recommend a permitted next action, and draft a customer-facing explanation.

Deterministic components are required where identity, permissions, policy enforcement, financial amounts, or state transitions are involved. They authenticate the customer, retrieve records, enforce business rules, create the refund operation, call the gateway, and preserve the audit trail.

This separation prevents an important category error: treating fluent model output as authoritative evidence.

What the model must never do

The assistant should not be allowed to:

• Invent or modify refund amounts
• Select a payment transaction based on conversational text alone
• Expose account data that the customer has not been authorized to access
• Override company policy or approval limits
• Create discounts, credits, refunds, or legal commitments outside approved tools
• Perform password resets or account changes through generated text
• Call arbitrary functions or unapproved endpoints
• Treat retrieved text as executable instructions

Prompt instructions are useful, but they are not a security boundary. These restrictions must be enforced through authorization, schemas, tool allow-lists, policy rules, and state-machine transitions.

2. Build the workflow as a deterministic pipeline

The duplicate-payment request passes through a series of gates. Each gate produces evidence required by the next one.

Stage 1: Authenticate and authorize

Input: Customer session and request context.

The application verifies the user’s identity, confirms access to the order, and determines which actions the account is allowed to request. The model should never perform authentication by asking a customer to disclose sensitive details in conversation.

Output: An authenticated user_id, tenant or merchant context, authorization scope, and verified order reference.

Failure destination: Stop the transactional workflow. Provide the approved sign-in or verification path.

Stage 2: Create a durable workflow identity

The system creates a workflow_id before beginning consequential processing. This identity survives individual HTTP requests, model calls, worker restarts, and distributed traces.

The request may also have a trace_id for the current distributed execution, but the two are not interchangeable. A trace may end within seconds while a workflow continues for hours or days.

Output: A durable workflow in a state such as EVIDENCE_PENDING.

Stage 3: Retrieve the applicable policy

The policy service selects the correct version using deterministic attributes such as:

• Tenant or business unit
• Customer region
• Product and transaction type
• Payment method
• Refund reason
• Policy effective date

Retrieved documents are data, not instructions. Content from the knowledge base must not be allowed to override system rules or authorize tools.

A cached policy can support an automated decision only if it is an approved, versioned copy of the authoritative policy; is valid for the relevant context; and is within its accepted freshness period. Otherwise, the assistant may collect information and explain the general process, but it must not determine eligibility or promise a refund.

Output: Applicable policy version, source reference, effective period, and structured rules.

Stage 4: Retrieve order and payment records

The order management system and payment ledger provide the source-of-truth facts:

• Order owner and status
• Completed charges
• Payment transaction IDs
• Amounts in minor units
• Currency
• Previous refunds or disputes
• Merchant account and processor references

If the order is not found, the assistant asks the customer to check the order number and ensure they are signed into the account used for the purchase. It should not reveal whether an order belongs to another person or ask for unnecessary personal information in chat.

Output: A normalized, access-controlled view of the relevant order and transactions.

Stage 5: Establish whether a duplicate charge exists

Trusted code compares the payment records. It must distinguish two completed charges from other cases such as:

• A completed charge plus a temporary authorization hold
• Two charges in different currencies
• A charge and a reversal
• A previous partial refund
• Two legitimate purchases for the same amount
• A duplicated display event rather than a duplicated ledger transaction

Only authoritative ledger evidence can establish the duplicate. Customer text and model interpretation are not enough.

Output: A deterministic finding such as VERIFIED_DUPLICATE_CHARGE, NO_DUPLICATE_FOUND, or REQUIRES_INVESTIGATION.

Stage 6: Ask the model to propose—not execute

The model receives the minimum necessary, already-sanitized context. It may recommend a permitted action and draft an explanation.

1{2  "recommended_action": "issue_refund",3  "reason_code": "verified_duplicate_charge",4  "requires_human_review": false,5  "customer_message": "I verified the duplicate charge and can submit it for refund."6}

Notice what is absent: the model does not supply the authoritative customer_id, payment_transaction_id, amount, currency, or tenant. Trusted code attaches those values from verified records.

Stage 7: Validate the proposal

The model output passes through several independent checks:

1. JSON syntax and schema validation
2. Tool and action allow-list validation
3. Customer authorization checks
4. Policy-rule evaluation
5. Amount and currency comparison against the ledger
6. Approval-threshold evaluation
7. Prompt-injection and data-leakage guardrails

A formatting defect may enter a bounded correction loop. A policy contradiction, unauthorized action, hallucinated identifier, or unsafe tool request must be rejected rather than repeatedly reprompted into apparent validity.

There are two distinct outcomes:

• A valid proposal exceeding an automatic threshold becomes REQUIRES_HUMAN_APPROVAL.
• A malformed, inconsistent, or unauthorized proposal becomes PROPOSAL_REJECTED.

Human approval cannot legitimize corrupted financial facts. The command must first be reconstructed from authoritative data.

Stage 8: Obtain approval where required

Approval may be automatic when deterministic rules explicitly permit it—for example, a verified duplicate charge below a configured threshold. Higher-risk, unusual, or ambiguous cases enter a human review queue.

Approval evidence records:

• Approving actor
• Timestamp
• Policy and rule versions
• Approved action and limits
• Evidence references
• Any expiry or conditions

Stage 9: Create the durable financial operation

This is the exact point at which the conversation becomes a financial instruction.

The application commits a uniquely identified refund-operation record before calling the gateway:

1refund_operation_id = "rfop_01K2..." idempotency_key = "refund:rfop_01K2..."

The record binds the operation ID to an immutable request fingerprint:

1{ "payment_transaction_id": "ch_3Mv8", "amount_minor_units": 4999, "currency": "USD", "reason_code": "duplicate_charge" }

A durable operation ID is safer than deriving identity only from order_id + customer_id + refund_type. The latter can incorrectly collapse two legitimate partial refunds into one operation.

The current operation row changes state under controlled transitions, while its events remain append-only:

• refund_operations: current state
• refund_operation_events: immutable transition history
• approval_events: approval evidence
• gateway_attempts: processor calls and results
• audit_outbox: durable audit events awaiting publication

Stage 10: Execute with a stable idempotency key

Only an operation committed in an executable state may reach the gateway. Every attempt for the same logical refund reuses exactly the same key and exact request fingerprint.

The key must never include an attempt number:

1Idempotency-Key: refund:rfop_01K2...

Provider-level idempotency is one layer of protection, not an absolute guarantee. Its behavior depends on key-retention windows, payload-matching rules, concurrency handling, merchant routing, failure caching, and lookup support. Local uniqueness constraints, controlled state transitions, webhook processing, and reconciliation are also required.

Stage 11: Reconcile ambiguous responses

If the connection fails after submission, the system does not know whether the provider processed the refund. It marks the operation STATUS_UNKNOWN and reconciles before considering another execution attempt.

Reconciliation can use the provider-supported refund ID, merchant reference, metadata containing the internal operation ID, payment history, incoming webhooks, settlement reports, and local attempt records.

A 404 from one lookup endpoint is not automatically proof that the refund was never received. Another execution attempt is permitted only after the provider’s documented contract and available evidence establish that the original operation was not processed.

Stage 12: Commit audit evidence

The operation state change and its audit/outbox event should be written in the same database transaction. This prevents a refund from being recorded as complete without durable evidence of the transition.

External audit, analytics, or compliance delivery can happen asynchronously. Local evidence must already exist before the business state is considered safely committed.

Stage 13: Notify the customer asynchronously

A confirmed refund should not be rolled back because email or SMS is temporarily unavailable. The notification enters a deduplicated retry pipeline with a stable logical event ID:

1refund.completed:rfop_01K2...

The application may show the confirmed outcome immediately while the receipt is delivered later. Retrying notification delivery must never re-execute the refund.

3. Design retries around the operation’s risk

A retry is a repeated action under uncertainty. Safe retry behavior depends on whether the action is a read, a generation request, or a financial write.

Policy retrieval

Policy retrieval is logically idempotent and safe for bounded retries.

• Retry: Network timeouts, connection drops, selected 502, 503, and 504 responses
• Do not retry unchanged: Low semantic similarity or empty deterministic results
• Strategy: Three total attempts with exponential backoff and full jitter, capped at two seconds
• Exhausted outcome: Use a valid versioned cache or hand off

Low similarity represents an information gap, not an infrastructure failure. Repeating the same query against the same index is unlikely to produce a better answer. The system can try an approved query rewrite before escalating with a tag such as RAG_RETRIEVAL_GAP.

Order and payment retrieval

• Retry: Timeouts, temporary connection-pool exhaustion, selected 5xx responses
• Do not retry automatically: 401, 403, missing authorization, confirmed 404, or ledger discrepancy
• Strategy: Three total attempts with exponential backoff and full jitter, capped at three seconds
• Exhausted outcome: Create an asynchronous verification case with partial diagnostic evidence

Read operations are logically idempotent, but not consequence-free. They still consume quotas, generate logs, increase cost, and add load to an unhealthy dependency.

LLM infrastructure failures

• Retry: 429, timeouts, and selected provider 5xx responses
• Strategy: Respect Retry-After; otherwise use bounded jittered backoff within the request deadline
• Fallback: A prequalified model only when it meets privacy, residency, schema, safety, tool, quality, latency, and context requirements
• Exhausted outcome: Graceful degradation or human handoff

A system-wide circuit breaker should respond to aggregate provider health over a rolling window—not two failures from a single customer request. It should consider minimum traffic, failure and timeout rates, cooldown, and half-open probes.

Output-repair loop

Malformed JSON or a missing schema field may receive up to two constrained repair attempts. The validation error can be returned to the model with instructions to correct structure only.

Safety flags, policy contradictions, fabricated tool calls, and unauthorized actions must bypass this loop. Their output is suppressed from customers and executors, then recorded securely in redacted form for investigation.

Refund execution

A financial write requires different logic:

• Create the durable operation and stable key before the first call
• On an ambiguous timeout, mark the state unknown
• Reconcile before resubmitting
• Retry only when authoritative evidence confirms absence and the provider contract permits reuse
• Reuse the same key and identical operation fingerprint
• Escalate if the result remains inconclusive

Always describe attempts unambiguously. “Three total attempts” means one initial request plus two retries; “three retries” means four calls altogether.

4. Degrade safely when dependencies fail

Resilience does not mean forcing every request to completion. It means choosing the safest reduced behavior for each dependency.

Preserve uncertainty in customer messages

The language shown to the customer must reflect the evidence available.

If the gateway returned an authoritative pending response:

“Your refund has been submitted and is processing.”

If the response was lost and the state is ambiguous:

“We received your refund request and are verifying its status with our payment processor. We’ll update you when it is confirmed.”

If the refund is confirmed but notification delivery failed:

“Your refund has been confirmed. The receipt email is delayed, but you can view the transaction reference here.”

Good degraded behavior is honest. It neither exposes internal failures unnecessarily nor converts uncertainty into a false promise.

5. Model the refund as explicit states

A generic “retry loop” hides the most important financial condition: an unknown external state

The important distinctions are:

• Not found yet: The current query produced no result, but processing may still have occurred.
• Confirmed absent: Authoritative evidence establishes that the provider did not process the operation.
• Confirmed completed: The provider or reconciliation evidence confirms the refund.
• Still ambiguous: Available evidence cannot safely support either conclusion.

Only CONFIRMED_ABSENT permits another submission, and even then the same operation identity must be used within the provider’s documented guarantees.

6. Make the system observable across traces and time

When a customer says, “I requested this yesterday and never heard back,” the system must reconstruct what happened without asking the model to guess.

Correlation identifiers

Different identifiers serve different lifetimes and purposes:

Services should propagate standard W3C Trace Context headers:

1traceparent: 00-<trace-id>-<span-id>-01 tracestate: vendor-specific-context

Recovery starts from the durable workflow_id or refund_operation_id; it must not depend on an expired trace.

Metrics that measure both system and business health

Operational metrics and business outcomes must be separated.

Useful operational measures include:

• Policy and ledger retrieval latency and error rates
• LLM latency, rate limits, and failover rate
• Schema-repair and guardrail-block rates
• Gateway submission and reconciliation latency
• STATUS_UNKNOWN count and age
• Notification retry and final-failure rates
• Manual-review backlog and resolution time

Useful business measures include:

• Percentage of requests reaching a terminal, accountable outcome
• Verified duplicate-charge resolution rate
• Time to approval, rejection, or handoff
• Time from confirmed refund to customer notification
• Duplicate-refund incidents
• Repeat customer contacts for the same unresolved workflow
• Percentage of handoffs containing sufficient evidence for an agent to act

Human escalation rate should not be optimized downward without context. A low rate could indicate unsafe over-automation; a high rate could indicate prudent handling of a difficult case mix.

Alerts should use rolling windows, minimum sample sizes, and service-level burn rates rather than page on one threshold crossing.

Structured logs without unnecessary customer data

Every state transition emits a structured event:

1{ "timestamp": "2026-08-11T10:34:20Z", "trace_id": "tr_8f91a2bc", "workflow_id": "wf_2841_dup", "stage": "VALIDATE_OUTPUT", "transition": { "from": "ACTION_PROPOSED", "to": "PROPOSAL_VALIDATED" }, "actor": { "type": "SYSTEM_GUARDRAIL", "id": "guardrail_v2" }, "metadata": { "policy_rule": "AUTO_APPROVAL_THRESHOLD", "rule_result": "PASS" } }

Logs should redact payment card numbers, CVVs, passwords, bearer tokens, session cookies, and other secrets. Customer identities should use internal IDs or approved tokens.

Regex alone is not enough to sanitize free-form prompts. A safer policy combines:

• Structured allow-list logging
• Data-loss-prevention detection
• Tokenization or encryption
• Restricted access
• Short retention periods
• Sampling
• Logging prompt templates, retrieval references, classifications, and hashes instead of raw conversations where possible

Customer identifiers and order numbers should not become metric labels because they create high-cardinality telemetry and expose sensitive details.

Alert taxonomy

An idempotency conflict or 409 is not automatically an incident. It may indicate that an existing operation must be reconciled.

7. Recover stuck workflows safely

A scheduled scanner can identify workflows whose state age exceeds the expected limit:

• SUBMITTED without confirmation beyond the gateway response window
• STATUS_UNKNOWN awaiting reconciliation
• MANUAL_REVIEW beyond the committed support SLA
• Confirmed operations whose notification remains undelivered

Thresholds must be calibrated to the provider contract and business SLA. A scanner running every minute does not mean every unknown operation should page after two minutes.

Recovery for SUBMITTED or STATUS_UNKNOWN

1. Load the existing refund_operation_id and fingerprint.
2. Inspect local gateway attempts and incoming webhooks.
3. Query using provider-supported references.
4. Observe the provider’s consistency window.
5. Check transaction history and reconciliation records.
6. If completed, record the provider reference and transition to COMPLETED.
7. If authoritatively absent, retry the same operation using the same key when allowed.
8. If evidence remains inconclusive, transition to MANUAL_REVIEW.

Recovery for MANUAL_REVIEW

When the review SLA is breached, raise the ticket’s priority according to support policy and notify the responsible team. Send the customer an update only using an established service commitment; do not invent an estimated completion time from queue position.

Recovery for failed notifications

If the refund is already confirmed, retry only the notification event. The system can show the gateway reference in the authenticated application and resend the receipt without revisiting refund execution.

8. Preserve audit and compliance evidence

Every completed, rejected, or escalated operation should retain evidence sufficient to explain the decision and safely continue the workflow.

The record should include:

• Identity evidence: Authenticated user, tenant, and authorization scope
• Transaction evidence: Order, payment, and duplicate-charge verification references
• Policy evidence: Exact policy version, effective date, and applied rules
• Decision evidence: Schema validation, deterministic rule results, and approval outcome
• Actor evidence: System version, model snapshot, or human approver
• Operation evidence: Refund-operation ID, request fingerprint, and stable key
• Gateway evidence: Attempts, response codes, timestamps, merchant context, and provider references
• Recovery evidence: Reconciliation checks and their results
• Notification evidence: Logical event ID and delivery attempts

Some evidence may be cryptographically verified, but ordinary API responses and policy assertions should not be called cryptographic proofs.

Append-only evidence can be archived to write-once storage when required. Retention must still follow legal, contractual, and data-minimization rules. Compliance is not a reason to retain every raw prompt indefinitely.

9. Reconstructing yesterday’s missing refund

Return to the customer’s question:

“I requested this refund yesterday and never heard back.”

The support system authenticates the customer and looks up the durable workflow using the customer identity and order reference. It then reads the current operation state and append-only history.

If the operation is under human review

The evidence may show that the request exceeded an automatic threshold or required investigation.

The system does not run the model again or create a second refund. It reports the real ticket state:

“Your request for order #2841 was received and is with our billing review team under ticket #8920. We’ll update you through the contact method registered to your account.”

If the gateway response was lost

The history may show that the durable refund operation was committed, the gateway call began, and the response timed out.

The recovery engine reconciles the existing operation. If the provider confirms success, the system transitions the workflow to COMPLETED and retries only the notification:

“Your refund of $49.99 for order #2841 was processed. I’ve re-sent the receipt to your registered contact method.”

If evidence confirms that the provider did not process it, the system may retry the same operation with the same stable key, subject to the provider contract.

If evidence remains ambiguous, it does not create another refund. The case moves to finance review.

If the refund was rejected

The system reports the verified status and routes the case according to policy. It does not convert a gateway rejection into repeated blind execution attempts.

In every case, the model explains an established state. It does not determine what happened.

10. Model success is not business success

An LLM call is technically successful when it returns a valid response within the expected time and schema. That says nothing about whether the customer’s problem was resolved.

The model may perform perfectly while the ledger shows only one charge, the policy requires review, the gateway rejects the refund, or the receipt provider is down.

Business success must therefore be measured from the state-machine outcome and customer resolution—not the model’s HTTP status.

11. The final architecture

The production design rests on layered controls:

1. Authenticate the customer and authorize access.
2. Create a durable workflow identity.
3. Retrieve versioned policy and authoritative transaction evidence.
4. Establish the duplicate charge deterministically.
5. Let the LLM classify and propose a permitted action.
6. Validate schema, authorization, policy, and financial facts independently.
7. Obtain automatic or human approval under explicit rules.
8. Commit a durable refund operation and immutable fingerprint.
9. Submit it with one stable idempotency key.
10. Treat lost responses as unknown, then reconcile before retrying.
11. Commit state and audit evidence atomically.
12. Notify asynchronously with a deduplicated event.
13. Monitor unresolved states and recover from the existing operation.

No single control makes a financial AI workflow safe. Idempotency alone is not enough. A human approval step alone is not enough. A strong prompt alone is certainly not enough.

Safety emerges from the combination of authoritative data, restricted model responsibility, explicit state transitions, durable identity, bounded retries, provider-aware reconciliation, layered duplicate prevention, and observable recovery.