The Hidden Work in 'Simple' AI Features

The afternoon demo and the six-week estimate

Someone on the product team connects a language model to the support desk, asks it to summarise a ticket, and it works beautifully. It took an afternoon. Then engineering estimates six weeks to ship it, and everyone quietly wonders whether engineering is padding.

They are probably not. The demo proved the model can summarise a ticket. Shipping means proving it summarises every ticket acceptably, fails safely on the strange ones, costs what you expect, respects who can see what, and tells you when it starts getting worse. That is where the six weeks go.

Where the time actually goes

For an illustrative 'summarise this support ticket' feature in an existing SaaS product, a realistic breakdown looks something like this:

The demo is the first row. Everything else is what separates a feature from a liability.

Evaluation: the part that is skipped most

Without an evaluation set, every change to the prompt or model is judged by someone reading five outputs and feeling good about them. That works until a change that improved the five examples quietly breaks a category nobody looked at.

An evaluation set is a few hundred real inputs with an agreed idea of what a good output looks like, plus a way to score them automatically or quickly by hand. It turns 'I think it is better' into a number. It is also what lets you switch models later with confidence. We cover the approach in how we evaluate whether an AI feature is good enough.

Edge cases that turn up in the first week

• A ticket thread 400 messages long that exceeds what the model can process in one go
• Tickets containing mostly a pasted log file or a screenshot
• Customers writing in a language the prompt did not anticipate
• Personal or payment data that should not be sent to a third party, or should be masked
• Text that contains instructions aimed at the model, deliberately or not
• Empty tickets, duplicate tickets and tickets that are only an email signature

None of these appears in a demo, because demos use tidy examples. All of them appear in production within days.

Each one needs a decision, not only code. Should a very long thread be summarised in chunks, or should the feature just summarise the last twenty messages and say so? Should a ticket in Portuguese get a summary in Portuguese or English? These are product questions, and the engineering team cannot answer them alone. Getting a product owner to decide them early is one of the easiest ways to keep the estimate from growing.

Permissions, cost and failure

Permissions. If the feature can pull in related tickets or account history for context, it must respect exactly what the viewing user is allowed to see. A summary that leaks another customer's details is a data breach written in fluent English.

Cost. One summary is cheap. A summary regenerated every time anyone opens a ticket, across 50,000 tickets a month, is not. Cache results, regenerate only when the ticket changes, set spending limits and alert on anomalies.

Failure. Providers have outages and slow periods. The feature needs a timeout, a graceful message and a product that works perfectly well without it. Our post on handling AI app failures gracefully goes into the patterns.

Monitoring after launch

Traditional features either work or throw an error. AI features can degrade while technically working: summaries get longer, start missing key facts, or the provider updates a model and tone shifts. You only notice if you are measuring.

1. Log inputs, outputs, model version, latency and cost for every call, with appropriate data protection
2. Collect lightweight user feedback, such as a thumbs up or down, or whether the draft was edited heavily
3. Re-run the evaluation set on a schedule and whenever anything changes
4. Alert on spikes in cost, latency, error rate or negative feedback

What you can safely trim for a first version

Not all of this needs to be perfect on day one. For an internal-only feature used by a small team, a smaller evaluation set, basic logging and a simple spending cap can be enough to learn from real use. What you should not trim, however small the release, is permissions, a fallback and some way to measure quality. Those are the three whose absence turns a small feature into a large incident.

When SpiderHunts estimates an AI feature as part of SaaS development, we list this hidden work explicitly so the estimate can be discussed line by line rather than accepted or rejected as one number. If the budget is tight, it is far better to ship less with all the safety parts than more without them. The wider cost picture is in the hidden costs of AI ownership.

Frequently asked questions