Model-Agnostic Architecture for AI SaaS

The rewrite that should have been a config change

A typical story: an AI SaaS product launches on one provider. Calls to its SDK spread across forty files. A year later a cheaper, better-suited model appears elsewhere, or the original provider has an outage during a customer's quarter-end, or an enterprise buyer requires EU-only processing. Switching turns out to be a two-month project.

The broader business case against lock-in is covered in our piece on AI vendor lock-in. This post is about the engineering: what a provider layer looks like, what it genuinely buys you and what it cannot.

What to put behind the abstraction

• One internal interface for text generation, structured output, embeddings and, if used, speech or image calls
• Your own request types: messages, tools, output schema, limits, tenant and feature tags
• Your own response types: content, parsed structured output, tool calls, token counts, cost, latency, provider and model used
• Retries, timeouts and error mapping into a small set of your own error categories
• Logging and cost capture on every call
• Model selection by configuration, per feature and optionally per tenant

Product code then asks for 'invoice extraction with this schema' and never imports a provider SDK. Open-source gateway libraries can cover some of this, and are worth considering, but you still want your own thin interface on top so a library change is not a product change either.

What you cannot abstract away

Models are not interchangeable parts. The same prompt produces different quality, formatting and failure patterns on different models. Tool-calling behaviour differs. Context limits differ. Some models follow long instructions faithfully and some drift. Structured output support varies in strictness.

The adapter makes switching possible. The evaluation set makes switching safe. You need both, and the second is the one teams skip.

A switching process that works

1. Add the new provider adapter and confirm basic calls, errors and cost capture
2. Run the full evaluation set for each affected feature on the new model with existing prompts
3. Adjust prompts per model where scores lag, keeping them as separate versions
4. Shadow a slice of production traffic, comparing outputs without showing them to users
5. Roll out per feature or per tenant through configuration, watching quality and cost
6. Keep the old provider configured as a fallback until confidence is high

With this in place, a switch that used to take two months becomes a couple of weeks, most of it evaluation and prompt work. That is an honest estimate. Anyone promising instant, zero-effort provider swaps for non-trivial features is selling the adapter and ignoring the behaviour.

Our evaluation pipeline guide covers the testing half in more detail.

Mixing models on purpose

Once the layer exists, using several models becomes natural. Small, fast models for classification and routing. Larger ones for complex reasoning. A self-hosted open model for a tenant with strict residency requirements. A second provider as a fallback during outages.

Each extra model adds evaluation, monitoring and prompt maintenance. Two or three models in production is manageable for a small team. Eight is a part-time job for someone. Add models because a feature or customer clearly benefits, not because the layer makes it easy.

Where abstraction goes too far

Some teams abstract so aggressively that they can only use features every provider shares, which means ignoring useful capabilities such as prompt caching, batch processing or strict structured output. That is lowest-common-denominator engineering, and it costs money and quality.

A better rule: the interface is common, but adapters may expose provider-specific options that features can use when available and do without when not. Log which features depend on which specific capability so you know the real switching cost at any time.

A reasonable starting point

For a new AI SaaS product, the first version of this layer is small: one interface, one or two adapters, logging, cost capture and configuration-driven model choice. At SpiderHunts we build it in the first week of any AI product, because its absence is so expensive later. It is part of the AI integration groundwork rather than a separate project, and it fits naturally with the wider AI SaaS architecture.

Frequently asked questions