AI Retainers: Ongoing Improvement After Launch
Last updated:
AI systems change after launch even if you do not
Ordinary software, left alone, mostly keeps doing what it did. AI systems do not have that luxury. The model provider updates or retires the model you use. Customers start sending a new invoice layout. A product range changes and the demand model's history stops describing the present. Nobody touched the code, and behaviour shifted anyway.
That is the main reason we offer retainers for AI work. Our general advice on keeping an AI app accurate over time explains the causes. A retainer is simply a structured way of making sure someone is watching.
Where the warranty ends and the retainer begins
Every project we deliver carries a 90-day warranty after go-live, covering anything that does not behave as the agreed specification says. The detail is in what our 90-day warranty covers. It is about defects in what we built.
A retainer covers something different: the system meeting a world that keeps moving. New document types, provider changes, gradual drift, cost creep and the improvements your team asks for once they have used the system for a few months. Those are not bugs, and they deserve a planned budget rather than a surprise quote.
What an AI retainer typically covers
- Monitoring quality measures such as acceptance, correction and escalation rates
- Running the evaluation set on a schedule and after any change
- Testing new model versions and provider changes before they reach production
- Updating prompts, extraction rules and retrieval as inputs change
- Retraining machine learning models when drift is detected or on a set schedule
- Watching cost per task and optimising routing, caching or model choice
- Security patches and dependency updates
- A block of time for small improvements agreed each month
- A short written monthly report
The mix varies. A document extraction system mostly needs input monitoring and occasional prompt updates. A demand forecasting model mostly needs drift checks and retraining. An AI agent with several tools usually needs the most attention of all.
What a month on retainer looks like
| Activity | How often | What you see |
|---|---|---|
| Automated quality and cost checks | Continuously | Alerts only when a threshold is crossed |
| Evaluation set run | Monthly and after any change | Pass or regression summary in the report |
| Review of corrections and escalations | Monthly | Patterns found and proposed fixes |
| Model or provider change testing | When changes are announced | A recommendation with evidence |
| Agreed improvements | Within the monthly time block | Released after testing, noted in report |
| Monthly report and short call | Monthly | What changed, what improved, what it cost |
Retainers come with defined service levels, so response times for problems are written down rather than implied.
Getting better each month, not merely staying up
The best retainers do more than keep things steady. After a few months of real use, the corrections your staff made are a gold mine. They show which inputs the system handles poorly, which questions it cannot answer and where people still do work by hand.
We turn that into a short list of improvements each month, each with an expected effect and a way to measure it. A typical example: an invoice extraction system sending 15% of documents to review because of one supplier's layout, where a targeted fix brings that down noticeably and saves someone an hour a day. Small, measurable, cumulative.
A retainer that only reports 'no incidents this month' is insurance. One that shows what got better is an investment.
Choosing the right level of support
We shape retainers around the system rather than selling fixed packages, but they tend to fall into three broad shapes.
- Watch. Monitoring, alerts, security updates and a quarterly evaluation run. Suits stable, low-risk systems.
- Watch and adjust. Adds monthly evaluation, model change testing and a small block of change time. Suits most AI integrations.
- Continuous improvement. Adds regular improvement work, retraining and a monthly review call. Suits systems central to operations or revenue.
Retainers are priced after we have seen the live system, because the effort depends heavily on volume, number of integrations and how fast the inputs change. Our AI integration projects include a recommendation on support level at handover.
When you do not need one
If your team has a developer comfortable with the runbook, the system is low-volume and low-risk, and the inputs rarely change, you may not need a retainer at all. Some clients keep one for six months and then run things themselves, calling SpiderHunts when something specific comes up. That is a perfectly reasonable outcome, and the handover documentation is written with it in mind.
We also take on retainers for AI systems other teams built. In that case we start with a short review of the code, prompts, accounts and any evaluation data, because a system with no tests and no monitoring needs those added before anyone can honestly promise to keep it healthy. Retaining an unmeasured system just means being the person blamed when it drifts.
Frequently asked questions
What is the difference between a warranty and an AI retainer?
Can we cancel an AI retainer?
How do you handle a model provider retiring the model we use?
Do machine learning models need retraining on a retainer?
What does the monthly AI retainer report include?
Have an AI system live and no one looking after it?
Tell us what it does and how it is performing. We will suggest the lightest level of support that keeps it healthy, and say so if it needs none.