Integrating an LLM into an Existing Backend Without Breaking It

Adding an LLM to a backend in production is an integration problem before it is a machine-learning problem. Treat the model as an unreliable third-party dependency: give it timeouts, capped retries, a circuit breaker, and a fallback path. Keep it out of the synchronous request path, validate every output against a schema, and measure prompt changes with a fixture set. The rest of this article turns those rules into practice.

Why treat an LLM call like any unreliable third-party dependency?

Because that is what it is: a remote API with variable latency, occasional outages, rate limits, and a bill attached. Every discipline you already apply to a payment gateway or an email provider applies here: strict timeouts, retries with a hard cap, a circuit breaker, and a fallback that keeps the feature useful when the model is not.

In production, model APIs commonly show p95 latencies measured in seconds, not milliseconds, and they degrade under load like any shared service. Set an explicit timeout well below your users' patience, cap retries (each attempt costs tokens, so a retry storm is also a billing incident), and open a circuit breaker after consecutive failures so a provider outage does not pile up thousands of doomed calls. Above all, design the fallback first: a cached previous answer, a simpler heuristic, or an honest "unavailable right now" keeps the product usable while the model is not.

const TIMEOUT_MS = 8000;
const MAX_ATTEMPTS = 3;

async function callModel(payload) {
  for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), TIMEOUT_MS);
    try {
      const res = await fetch(MODEL_API_URL, {
        method: 'POST',
        headers: { 'content-type': 'application/json' },
        body: JSON.stringify(payload),
        signal: controller.signal,
      });
      if (res.ok) return await res.json();
      if (res.status < 500) break; // a 4xx will not succeed on retry
    } catch {
      // timeout or network error: try again if attempts remain
    } finally {
      clearTimeout(timer);
    }
  }
  return null; // caller switches to the non-AI fallback path
}

Why keep the model out of the synchronous request path?

Because a call that takes two to twenty seconds does not belong between a user click and an HTTP response. Where the product allows it, accept the request, push a job onto a queue, process it asynchronously, and notify the client when the result lands. The user waits on a spinner, not on your connection pool.

A queue between your API and the model buys you retries without user-facing latency, backpressure when the provider throttles you, and a dead-letter queue for inputs that fail repeatedly. Amazon SQS plus a worker covers most cases; we detailed the pattern in our article on event-driven systems with Lambda and SQS. Synchronous calls remain defensible for typeahead-style features, but only with a tight latency budget and an instant non-AI fallback when that budget is blown.

How do you stop non-deterministic output from corrupting your data?

By validating every model response against a schema before it touches anything durable. Free text must never reach a database write, a payment flow, or a downstream API unchecked. Ask the model for structured output, parse it, validate it, and route anything that fails validation to a fallback or a human, never onward.

import { z } from 'zod';

// The model's reply must match this contract before anything persists
const TicketTriage = z.object({
  category: z.enum(['billing', 'technical', 'account']),
  urgency: z.enum(['low', 'normal', 'high']),
  summary: z.string().max(300),
});

const parsed = TicketTriage.safeParse(JSON.parse(raw));
if (!parsed.success) {
  // Invalid output: fall back to rules and flag for review
  return triageWithRules(ticket);
}
saveTriage(parsed.data);

Define the contract with a schema library and reject anything that does not parse. The OWASP Top 10 for LLM applications lists improper output handling among the most common failure modes, and it is the easiest one to prevent. For anything customer-facing, add two more layers: a feature flag so you can turn the feature off without a deploy, and human review of a sample of outputs (or of every output, early on) before you trust the loop to run alone.

How do you know a prompt change did not make things worse?

With a small evaluation set you run before and after every prompt or model change. Twenty to fifty real inputs with expected outputs, scored automatically, catch most regressions that eyeballing three examples misses. Treat prompts like code: version them, review them, and block the deploy when the eval score drops.

Keep those anonymized inputs in the repository next to the prompt, each with its expected structured output. Score them automatically: exact match on schema fields, presence of required content, refusal rate. Run the set in CI on every prompt edit, and rerun it when the provider ships a new model version, because model updates change behavior without warning you. Decide what an acceptable score is before you change anything, otherwise every result looks fine after the fact.

How do you keep the invoice predictable?

Three levers: token budgets, caching, and model tiering. Cap input and output tokens per request, cache identical or near-identical calls, and route easy cases to a small cheap model while reserving the large one for the hard ones. Without those caps, one enthusiastic user or one retry loop decides your monthly bill.

Cap max output tokens on every call and reject oversized inputs before they reach the API. Use provider-side prompt caching for repeated system prompts and shared context: both Anthropic and OpenAI document prompt caching and discounted batch processing at the time of writing. Route by difficulty: a small model handles classification and extraction well, and escalating only the ambiguous minority to a larger model costs far less than sending everything there. Then meter spend per feature and alert on it, exactly as you would for any other infrastructure line.

What actually leaves your infrastructure?

Every prompt you send, including whatever user data you interpolated into it. Under GDPR the provider is a processor, so you need a data processing agreement, a documented legal basis, and redaction of personal data the model does not need. Assume prompts are logged somewhere unless your contract says otherwise.

Before any call, strip what the model does not need: replace names, emails, phone numbers and IBANs with placeholders, and reinject them after the response if the feature requires it. Sign the provider's data processing agreement, check whether prompts are retained and for how long, and confirm they are not used for training under your plan. Some providers offer EU processing or data-residency options; verify what your contract actually covers, not what the marketing page implies. Apply the same discipline to your own logs: a prompt in a log file is personal data too.

When is AI the wrong tool?

Whenever a deterministic solution already does the job. A regex that extracts an order number, a SQL query that ranks customers, a rules table that routes tickets: all of these are faster, cheaper, testable, and explainable. Reach for a model only when inputs are genuinely unstructured and the rules keep failing.

This sorting is the first step of our AI and automation work: a model that replaces a working regex adds latency, cost, and a new failure mode, and removes a test you could trust.

The pre-production checklist

Before the feature meets real users, verify every line:

• Timeout and capped retries on every model call, plus a circuit breaker
• A fallback path that keeps the feature working, degraded, when the provider is down
• Model calls out of the synchronous path, or a strict latency budget if they must stay
• Schema validation on every output: nothing unvalidated reaches storage or money
• An eval fixture set wired into CI, run on every prompt or model change
• Token caps, caching, model tiering, and a spend alert
• DPA signed, PII redacted before the call, retention understood
• Feature flag and a kill switch on anything customer-facing