The service is more than the model call
Calling a model is a few lines. Making that call dependable — retried, validated, logged, capped and observable — is the actual work, and it is what separates a demonstration from a system.
The model call is perhaps five per cent of an AI service. The rest is everything that makes it safe to depend on.
The shape that works
- Receive the request and validate it
- Queue it, unless a person is genuinely waiting
- Build the prompt from configuration, not from code
- Call with a timeout and bounded retries
- Validate the output against a schema before accepting it
- Record input, context, output, model version and confidence
Force structured output
Ask for a defined structure and validate what comes back. If it does not match, retry once with a stricter instruction and then quarantine rather than accepting something malformed.
- A schema the output must satisfy
- Validation before anything downstream sees it
- One retry with a stricter prompt
- Quarantine with the raw response for diagnosis
Cost control is not optional
| Control | Why |
|---|---|
| Hard daily cap | A loop must not spend the budget |
| Per-caller cap | One consumer cannot exhaust it |
| Cost per request tracked | Rising cost is the early warning |
| Token limits on output | Generation is the expensive direction |
| Decided behaviour at the cap | Degrade or queue, not crash |
Keep the prompts in configuration
Prompts change more often than code and are frequently adjusted by someone who is not a developer. They belong in configuration, versioned, with the ability to roll back.
That also makes it possible to run an evaluation set against a prompt change before deploying it, which is what stops quality regressions.