Visual Search for eCommerce Stores

The shopper who cannot name what they want

Someone sees a lamp in a friend's flat. They do not know it is called a mushroom lamp, and typing 'round glass table light thing' into your search bar returns plugs. They leave.

Visual search handles that shopper. They photograph the lamp, or tap an image they like on your site, and get products that look similar. For style-driven catalogues it answers a question text search cannot.

For a lot of stores, though, it is a nice demo that a small share of visitors ever use. Knowing which kind of store you are is the first decision.

How visual search works under the hood

The engineering is less exotic than it sounds. It has three pieces.

1. Embeddings. A pretrained vision model converts each product image into a list of numbers that captures what it looks like. Similar-looking products end up with similar numbers.
2. A vector index. Those embeddings go into a vector search index, which can be a dedicated vector database or an extension on PostgreSQL you already run.
3. A query path. The shopper's photo is embedded the same way, the index returns the nearest products, and your usual business rules filter out anything out of stock or unavailable in their region.

The same embeddings power 'more like this' carousels on product pages, which often gets more use than photo upload because it needs no effort from the shopper.

Photo upload or 'shop the look'

If you only build one, build the similar items carousel. It is the cheapest, it needs no change in shopper behaviour, and it gives you data on whether visual similarity drives clicks in your store before you commit to more.

What makes results good or bad

• Cropping. A photo of a person in a room contains many products. Detecting and cropping the object first matters more than the choice of embedding model.
• Consistent product photography. Mixing packshots with lifestyle images in the index gives odd matches. Index packshots, or index both separately.
• Business filters. Showing a perfect match that is out of stock in the shopper's size is worse than showing a decent match that is available.
• Blending with text relevance. Similar colour and shape is not always similar product. A black boot and a black handbag can sit close together in embedding space.

Tuning these is where most of the effort goes after launch. The model itself rarely needs changing.

How much does visual search cost to build?

For a store with tens of thousands of products on a platform that allows custom search, a first version of similar-items plus photo upload is typically a project of several weeks. Embedding the catalogue is a one-off compute job that is inexpensive at that scale, and new products are embedded as they are added.

Running costs are modest: vector search queries are cheap, and the image embedding at query time is the main per-search cost. The larger ongoing cost is attention, keeping filters, merchandising rules and tracking in step with the rest of your search. If your platform is a hosted one, check what its app ecosystem already offers before commissioning custom software; a plugin that is 80% as good may be enough.

When visual search is not worth it

Stores selling on specification do not benefit. Nobody photographs a replacement filter cartridge hoping for a visual match; they type the part number. The same goes for electronics, books, supplements and most B2B catalogues.

Small catalogues also struggle. With 300 products, a shopper can browse the lot in a few minutes, and visual search has too few neighbours to return anything useful.

If your text search still returns nothing for common misspellings, fix that first. It will earn more than any visual feature.

Our post on AI-powered website search covers the text side, which in most stores is where the lost revenue actually sits.

Proving it before a full build

At SpiderHunts we would start by embedding your catalogue offline and generating similar-item lists for your top 200 products. The merchandising team reviews them in a spreadsheet. If they wince at a third of the matches, you have learned that cheaply. If they like them, put the carousel live on half your traffic and compare add-to-basket rates over a month.

A month is the minimum. Visual features tend to get a burst of curiosity clicks in the first week that flatter the numbers, and the real pattern only shows once the novelty wears off. Watch returning customers in particular; if they keep using the carousel on their second and third visits, it is earning its place.

One more practical point. Keep the embedding step separate from your storefront code, as a small service with its own index. When a better image model comes along, and one will, you re-embed the catalogue overnight and swap the index without touching the theme.

Frequently asked questions