What Makes an AI Development Partner Worth Keeping
Last updated:
The first project tells you less than the second year
Most advice about choosing an AI supplier covers the selection: portfolios, references, proposals. That matters, but almost any competent team can deliver a good first project when everyone is paying attention. What separates a partner worth keeping is what happens in the months after launch, when the model provider changes something, usage doubles, a new use case appears and nobody is in a hurry.
AI systems also change more after launch than ordinary software. Models are updated or retired, costs shift and the data drifts. A partner who was excellent at building may be poor at living with what they built.
They measure quality with evidence, not adjectives
A partner worth keeping can show you how well the system performs, on what cases, and how that has changed. That usually means an evaluation set of real examples that every change is tested against, plus production figures such as acceptance rates, escalation rates and cost per completed task.
If you ask whether a recent change improved things and the answer is 'it seems better', that is a warning. If the answer is a comparison on a few hundred cases with the regressions listed, you are in good hands. This is also how we run AI agent projects at SpiderHunts, because agents in particular can get worse in ways nobody notices without measurement.
They tell you when to stop spending
The partner who says 'this next feature will not pay for itself' is worth more than one who agrees to everything. Good partners bring you fewer, better ideas and are comfortable recommending that a feature be switched off because nobody uses it.
A supplier who has never told you no has either been very lucky with your ideas or is not really looking at them.
This matters more with AI than most software, because it is easy to keep adding clever features that each carry running costs. A document assistant that gains a summariser, a translator and a sentiment score over a year may cost several times what it did at launch, while the one feature people rely on has not improved at all. A partner worth keeping points that out before the invoice does.
They keep you able to leave
It sounds odd, but the best reason to stay with a partner is that you could easily go. That means you own the repository, the cloud accounts, the model provider accounts and the evaluation data. It means documentation exists that another team could follow, and that the handover is something they would welcome rather than dread. Our description of how we hand a project over sets out what that looks like in practice.
A partner who holds your API keys, hosts your system on their own account and keeps prompts in a private folder may be perfectly honest. They have still made leaving expensive, and that changes the relationship whether anyone intends it to or not.
They handle model changes calmly
New models appear constantly, and every release comes with claims of better performance. A partner worth keeping neither ignores them nor rushes your production system onto each one. They test a new model against your evaluation set, look at cost and latency as well as quality, and recommend a switch when the evidence supports it.
They also plan for the opposite: a model being retired or changed by the provider. A well-built system keeps model calls behind one internal layer, so a forced change is a testing exercise rather than an emergency.
Warning signs worth taking seriously
| What you notice | What it often means |
|---|---|
| Accuracy claims without an evaluation set | Nobody is measuring, so nobody knows |
| Monthly AI costs rising without explanation | No per-task cost monitoring |
| Only one engineer can answer questions | Knowledge is not documented |
| Every request is quoted as a large project | The system is hard to change, or the incentives are off |
| Updates arrive as surprises in production | Changes are not tested before release |
| You cannot get a copy of the prompts and code | You do not really own what you paid for |
One of these on its own may have a good explanation. Several together deserve a direct conversation.
A simple yearly review
Once a year, set aside an hour and go through the relationship honestly, ideally with the partner in the room for the second half.
- What did the AI systems achieve against the measures agreed at launch?
- What did they cost to run and support, and is that trend acceptable?
- Which recommendations did the partner make, and which saved money?
- Could another team take over within a month using what exists today?
- What would you change about how you work together?
Our general guide to what good looks like with a software partner covers the wider relationship. For AI specifically, the second and fourth questions are the ones most often skipped.
Frequently asked questions
How often should we review our AI development partner?
Is it risky to switch AI development partners?
What should an AI supplier report to us each month?
Should our AI partner switch to every new model release?
What should we do if our AI partner fails the review?
Not sure whether your current AI supplier is serving you well?
Send us a description of what was built and how it is performing. We will give you an independent view, and we will not pretend the answer is always to switch.
Related services
What we build for problems like this one