Wrapping a Model in Something Dependable

The service is more than the model call

Calling a model is a few lines. Making that call dependable — retried, validated, logged, capped and observable — is the actual work, and it is what separates a demonstration from a system.

The model call is perhaps five per cent of an AI service. The rest is everything that makes it safe to depend on.

The shape that works

1. Receive the request and validate it
2. Queue it, unless a person is genuinely waiting
3. Build the prompt from configuration, not from code
4. Call with a timeout and bounded retries
5. Validate the output against a schema before accepting it
6. Record input, context, output, model version and confidence

Force structured output

Ask for a defined structure and validate what comes back. If it does not match, retry once with a stricter instruction and then quarantine rather than accepting something malformed.

• A schema the output must satisfy
• Validation before anything downstream sees it
• One retry with a stricter prompt
• Quarantine with the raw response for diagnosis

Cost control is not optional

Keep the prompts in configuration

Prompts change more often than code and are frequently adjusted by someone who is not a developer. They belong in configuration, versioned, with the ability to roll back.

That also makes it possible to run an evaluation set against a prompt change before deploying it, which is what stops quality regressions.

Frequently asked questions