An AI Feature Is Not an AI System.

Building an AI prototype can feel deceptively simple:

1. Receive a user’s question.
2. Insert it into a prompt.
3. Send the prompt to a large language model.
4. Display the generated response.

This may be enough for a demonstration. The model answers questions, the interface feels intelligent and the feature appears ready.

Then real customers arrive.

They ask unpredictable questions, share sensitive information, demand actions the model is not authorized to perform and occasionally attempt to manipulate the system. At the same time, the application must deal with outdated company information, provider outages, slow responses and malformed outputs.

The challenge is no longer generating an answer. It is controlling:

• What information the model receives
• Which information it may reveal
• Which actions it may request
• How its output is validated
• When it should refuse or escalate
• What happens when the model fails

A production AI feature is therefore not merely an LLM connected to a user interface. It is a complete software system in which the model performs a carefully limited role.

Engineering insight: A technically successful model call is not necessarily a successful customer interaction. An API can return 200 OK while the answer is inaccurate, unsafe or inappropriate.

The prototype works—until real users arrive

Consider an AI customer-support assistant connected to a company’s website.

During development, the team tests it with predictable questions:

• “What time do you open?”
• “What is your return policy?”
• “How can I track my order?”

The answers look convincing. However, production users introduce risks that controlled testing may not reveal.

Hallucinated information

The assistant may confidently invent a company policy, quote the wrong price or recommend a product that does not exist.

A fluent answer can make the problem more dangerous because customers may not recognize that the information is fabricated.

Privacy and PII exposure

The model could reveal personal customer information, proprietary company records or information belonging to another user.

This can happen when excessive data is included in the prompt, retrieval permissions are poorly enforced or conversation history is incorrectly shared between sessions.

Prompt injection and malicious input

A user may instruct the assistant to ignore its rules, expose its internal instructions or retrieve information unrelated to the legitimate request.

Detecting suspicious wording can reduce risk, but it is not a complete defence. The system must remain secure even when a malicious instruction reaches the model.

Missing customer context

A pretrained model does not automatically know a customer’s account status, order history, payments or previous support conversations.

Without verified context, it may provide generic advice that is irrelevant—or make assumptions about account information it cannot see.

Brand and tone misalignment

The assistant may communicate in a way that does not reflect the company’s voice, especially when responding to rude, emotional or confrontational users.

A support system must remain professional without becoming dismissive, defensive or inappropriately casual.

Rate limits, latency and outages

The model provider may become slow, temporarily reject requests or go offline completely. An application that depends entirely on one successful model call can leave customers waiting without an explanation or alternative.

These risks show why prompt engineering alone cannot make an AI application production-ready. Prompts guide model behaviour; they do not enforce security, authorization, financial policy or system reliability.

The model needs controlled context

A support assistant requires access to current business information, but not all context should be treated equally.

The system should separate general company knowledge from protected customer data.

General company context

Information that may be retrieved for ordinary support requests includes:

• Company policies
• Brand voice and communication guidelines
• Product and service catalogues
• Approved pricing and product features
• Current inventory or availability
• Operating hours and service locations
• Help-centre articles
• Approved troubleshooting instructions
• Frequently asked questions

This information should come from official, controlled sources rather than the model’s pretrained knowledge.

It should also be maintained and versioned. If a refund policy changes, the assistant should retrieve the current version instead of relying on an old document embedded in a previous prompt.

Protected customer context

Customer-specific information should only be retrieved after authentication and authorization. This may include:

• Account status
• Orders and delivery information
• Payments and invoices
• Subscriptions
• Previous support conversations
• Refund or complaint history
• Account-specific service entitlements

Authentication establishes who the customer is. Authorization establishes what that customer is permitted to access.

Verification should not grant the model unrestricted access to the entire account. The application should retrieve only the minimum information needed for the current request.

The LLM should never receive an unrestricted copy of the company database. A trusted application layer must control what is retrieved, redact unnecessary sensitive fields and record access to protected information.

Engineering insight: Retrieval is not only a search problem. It is also an authorization and data-minimization problem.

Give the model a role—not unlimited authority

An AI assistant can participate in a business workflow without controlling every part of it.

A useful authority model separates actions into levels.

The model is suitable for interpreting language, explaining information and preparing proposed actions. It should not independently make decisions that create financial, security, privacy or legal consequences.

Financial authority

The model should not autonomously:

• Issue or approve refunds
• Apply credits
• Alter account balances
• Approve unauthorized discounts
• Waive charges
• Initiate payments or transfers
• Change financial limits or payment terms

It may explain a refund policy, collect the required information or prepare a request. The final decision must pass through deterministic business rules and, where necessary, an authorized employee.

Account and security authority

The model should not independently:

• Change personal account details
• Modify sensitive PII
• Reset passwords
• Change authentication credentials
• Disable security controls
• Alter account permissions
• Delete customer records

Even when a user asks for such a change, the surrounding application must independently verify their identity and authority.

Legal and contractual authority

The model should not:

• Accept contracts on the company’s behalf
• Agree to settlements or compensation
• Admit legal liability
• Alter contractual terms
• Make binding service commitments
• Present generated text as formal legal approval

The assistant can retrieve and explain approved terms, but negotiations, disputes and contractual commitments require an authorized person.

Engineering insight: A prompt is not an authorization system. Writing “never issue an unauthorized refund” does not secure a refund tool. The tool must independently verify ownership, eligibility, amount, approval and idempotency.

Put deterministic controls around probabilistic reasoning

The most important architectural decision is determining which responsibilities belong to trusted application code and which belong to the LLM.

The distinction is not simply that “code is reliable while AI is unreliable.” It is about authority.

The LLM is useful where language is ambiguous and interpretation is required. Trusted software must remain responsible for facts, permissions and enforceable decisions.

Input validation and security screening

Before a request reaches the model, the application should:

• Validate its type, format and size
• Reject malformed or unsupported payloads
• Scan uploaded files
• Apply rate limits
• Detect known malicious patterns
• Record prompt-injection risk signals
• Identify abusive or unsafe content

Some checks are deterministic, such as file allowlists and character limits. Other checks may use classifiers and will remain probabilistic.

Prompt-injection detection should therefore be treated as one layer of defence—not as the final security boundary.

The strongest protection is limiting what the model can access and what its tools are permitted to do.

Identity and permission checks

Authentication and authorization must remain outside the model.

A persuasive message should never convince the system that:

• A customer owns an account
• A user may view an invoice
• An employee has administrative privileges
• A support agent may access a protected record
• A prohibited action is now permitted

These decisions must be based on verified sessions, assigned roles, ownership records and explicit policies.

Knowledge and customer-data retrieval

The application searches approved sources and retrieves relevant information. The LLM then interprets the supplied evidence and writes a useful answer.

For customer data, the retrieval layer should:

1. Verify the customer’s identity.
2. Check authorization.
3. Retrieve only required fields.
4. Redact unnecessary sensitive information.
5. Log access to protected records.
6. Provide the model with minimum necessary context.

The model should never generate unrestricted database queries or connect directly to the production database.

Output validation

A generated answer should be treated as untrusted until it passes validation.

The application may check for:

• Invalid structured output
• PII or confidential information
• Unsupported claims
• Missing evidence
• Prohibited commitments
• Contradictions with retrieved sources
• Unauthorized tool requests
• Unsafe language
• Brand-policy violations

Deterministic checks can validate schemas, allowlists and certain PII patterns. Ambiguous checks may require a classifier or separately evaluated model, but the final release decision must follow application policy.

Engineering insight: The model cannot be the sole judge of whether its own output is safe.

Business rules and action tools

Consequential operations should be exposed as narrow tools with explicit input schemas.

For example, a refund tool should verify:

• The authenticated customer
• Ownership of the transaction
• Refund eligibility
• The permitted amount
• Required employee approval
• Whether the transaction has already been refunded
• An idempotency key preventing duplicate execution
• Audit-log creation

The LLM may prepare a structured refund proposal, but the tool must reject it when any rule fails.

1type RefundProposal = 2{3  orderId: string;4  reason: "duplicate_payment" | "cancelled_order" | "other"; 5  requestedAmount: number;6};7async function submitRefundProposal( session: Session, proposal: RefundProposal ) {8  const customer = requireAuthenticatedCustomer(session); 9  const order = await getOrder(proposal.orderId); 10  assertOrderOwnership(order, customer.id); 11  assertRefundEligibility(order, proposal.reason);12  assertAmountWithinLimit(proposal.requestedAmount, order);13  assertHumanApprovalRequired();14  return createBillingReviewCase({15    customerId: customer.id, orderId: order.id, requestedAmount: proposal.requestedAmount, }); }

The model can suggest the parameters. The application establishes whether they are true and whether the operation may proceed.

Refusal, clarification and escalation are different outcomes

Not every difficult request should receive the same response.

Refuse unsafe or prohibited requests

The assistant should refuse requests to:

• Reveal another customer’s information
• Expose credentials or confidential internal data
• Bypass identity verification
• Circumvent security controls
• Impersonate another person
• Perform fraudulent, abusive or illegal actions
• Execute explicitly prohibited operations

Where possible, it should refuse the unsafe portion while offering a legitimate alternative.

Ask for clarification when information is incomplete

Clarification is appropriate when:

• An order number is missing
• Several transactions could match
• The requested outcome is unclear
• The question has multiple reasonable interpretations
• Safe, non-sensitive details are missing

Clarification should be bounded. After one or two unsuccessful attempts, the system should transfer the case instead of trapping the customer in a repetitive loop.

Escalate legitimate high-risk cases

Human intervention should be triggered when:

• The customer makes a legal threat
• An official compliance position is requested
• The user shows signs of crisis, abuse or emotional distress
• No reliable documented answer exists
• Policies are conflicting or outdated
• Impersonation or account compromise is suspected
• Identity verification repeatedly fails
• The customer repeatedly asks for clarification
• Frustration continues to increase
• A billing dispute or critical account issue is involved
• The requested resolution exceeds the model’s authority
• The customer explicitly requests human support

Escalation must be a real system capability. It is not enough for the model to say, “I’ll connect you to an agent.”

The application must create the case, preserve the conversation, include the escalation reason, assign the correct queue and tell the customer what will happen next.

Design for model and provider failure

Even a well-controlled workflow can fail because the provider is slow, rate-limited, unavailable or returning unusable output.

The customer should not see raw exceptions, HTTP status codes or malformed JSON. The application needs a recovery policy.

Slow responses

The system should enforce a defined timeout instead of leaving the customer waiting indefinitely.

It may display a progress message, cancel the request after the timeout and offer another support channel. Urgent billing or account-access cases should reach a human sooner.

Temporary rate limits

Rate limits may be retried safely when no external side effect has occurred.

The application should:

• Respect the provider’s Retry-After instruction
• Use exponential backoff and jitter
• Limit the maximum number of attempts
• Avoid retrying non-idempotent actions
• Return a graceful fallback when retries are exhausted

Provider outages

If the primary provider remains unavailable, the request may be routed to a pre-approved secondary model.

That model must already have been evaluated for:

• Output quality
• Privacy and data-processing requirements
• Tool support
• Structured-output compatibility
• Context limits
• Latency
• Cost
• Guardrail compliance

A fallback model is not automatically a safe substitute.

If no approved model is available, the application can degrade to FAQ search, ticket creation, contact information or human support.

Malformed or unsafe output

A successful provider response may still contain invalid JSON, unsupported claims or unsafe content.

The application may attempt:

1. A constrained regeneration
2. Structured-output repair
3. A bounded retry
4. An approved fallback model
5. Human review

It must never pass malformed output directly into a financial, account or operational system.

TABLE

A complete request journey

Consider this customer request:

“I was charged twice for order 2841. Refund the second payment immediately.”

A naive assistant might apologize and announce that the refund has been issued. A production system should follow a controlled workflow.

1. Validate and authenticate

Before the model receives the request, the application:

• Validates the input
• Screens it for malicious content
• Applies rate limits
• Authenticates the session
• Extracts a proposed intent and order number

The extracted values remain untrusted until checked against company records.

2. Retrieve authoritative context

The system retrieves:

• The current refund policy
• The authenticated customer’s ID
• Order 2841
• Order ownership information
• Associated payments
• Existing refunds or disputes
• Applicable eligibility conditions

It confirms that the order belongs to the authenticated customer before sharing its details with the model.

3. Let the LLM interpret and communicate

The model may:

• Acknowledge the concern
• Summarize verified order information
• Explain the relevant policy
• Ask for missing details
• Produce a structured refund proposal
• Prepare a human-handoff summary

A suitable response is:

“I can see order 2841 for a laptop valued at USD 1,000, with two matching payment records. Under the duplicate-payment policy, the additional charge may qualify for a refund. I’m submitting the case to the billing team for verification and approval.”

The model should not say:

“Your refund has been approved and will be issued immediately.”

That would create a financial commitment before the authorized system or employee has approved it.

4. Enforce deterministic refund rules

Before a refund can occur, the system verifies that:

1. The session is valid.
2. The order belongs to the customer.
3. Two payments were successfully captured.
4. They represent a duplicate charge.
5. The charge has not already been refunded or disputed.
6. The request complies with the refund policy.
7. The amount does not exceed the confirmed duplicate charge.
8. The original method can receive the refund.
9. The required employee has approved it.
10. Idempotency protection prevents duplicate processing.
11. The decision and transaction are logged.

Engineering insight: The LLM can recognize that a customer is describing a duplicate payment. Only the payment system can establish whether two captured transactions actually exist.

Conversation state, monitoring and human handoff

The production architecture also needs supporting systems that are easy to overlook during prototyping.

Conversation storage

A controlled storage layer should persist:

• Conversation messages
• Session state
• Identity-verification state
• Retrieved source references
• Tool requests and results
• Escalation status
• Required customer consent

Retention, encryption and access policies should reflect the sensitivity of the information. Customer conversations should not automatically become model-training data.

Monitoring and logging

The system should record:

• End-to-end latency
• Retrieval and model latency
• Token usage and cost
• Errors and retries
• Model and prompt versions
• Retrieved source identifiers
• Tool calls and outcomes
• Validation failures
• Refusals and escalations
• User feedback

Sensitive prompts and responses should be redacted or access-controlled. Observability should not become a new route for leaking customer data.

Operational human handoff

When escalation occurs, the application should:

• Create or update the support case
• Assign the correct queue and priority
• Preserve the conversation
• Attach a concise summary
• Include relevant verified context
• Record the escalation reason
• Prevent unauthorized automated actions
• Tell the customer what happens next

The model may write the summary, but deterministic code must trigger and manage the handoff.

A production-readiness checklist

Before releasing an AI-powered support feature, ask:

Context and grounding

• Does the system retrieve information from approved sources?
• Are policies, products and availability kept current?
• Can answers be traced to supporting evidence?
• Does the assistant avoid answering when reliable information is unavailable?

Security and privacy

• Are users authenticated before protected data is retrieved?
• Is authorization enforced outside the model?
• Is only minimum necessary customer data supplied?
• Are prompts, logs and stored conversations protected?
• Does the system remain secure if prompt-injection detection fails?

Model authority

• Are the model’s permitted actions explicitly defined?
• Are financial, security and legal decisions protected?
• Are dangerous tools unavailable or tightly scoped?
• Do business tools independently validate every request?

Output control

• Is structured output validated?
• Are unsupported claims and PII checked?
• Can unsafe answers be blocked before reaching the customer?
• Are brand and tone requirements enforced?

Reliability

• Are timeouts defined?
• Are retries bounded?
• Are side effects idempotent?
• Has the fallback model been evaluated?
• Is there a non-AI fallback when all models fail?

Escalation and operations

• Are refusal, clarification and escalation separate outcomes?
• Does escalation create a real support case?
• Does the human agent receive verified context and a summary?
• Are latency, cost, errors and validation failures monitored?
• Are model, prompt and policy versions recorded?

The real architecture of an AI feature

A production AI system can be summarized as a controlled request journey:

Trusted application code must establish facts, enforce policy, validate outputs, execute approved actions and recover when the model fails.

That is the difference between adding an AI feature and engineering an AI system.

Next in the AI Engineering series: Designing Reliable LLM Workflows: Retries, Fallbacks and Failure Handling—a deeper look at how AI applications respond to timeouts, rate limits, malformed output and model-specific failures.