Crawl budget and faceted navigation
Deciding which facet combinations deserve to be indexable pages and making the rest genuinely uncrawlable, rather than relying on canonical hints that still cost a fetch.
A catalogue is not a content problem, it is an engineering one. Most stores we see have thousands of products that search engines never index, because filter combinations eat the crawl budget before the products are reached. We fix that, and then make the pages worth finding.
Thanks — that’s with us.
A senior engineer reads every enquiry and will get back to you shortly.
Book a time that suits you, or send a message and get an answer sooner.
2020
Building since
1,000+
Clients served
UK
Registered company
4.6/5
Rated on Clutch
Quick answer
On a large store, the binding constraint is almost never content quality. It is crawl budget. Faceted navigation generates a combinatorial explosion of URLs, colour by size by price by brand by sort order, and a crawler spends its allowance on those instead of on your products. The symptom is a Search Console report showing far more discovered URLs than indexed ones.
The fix is structural rather than editorial. Decide which facet combinations are allowed to exist as indexable pages, because a few genuinely are valuable landing pages, and make the rest uncrawlable rather than merely canonicalised. Canonical tags are a hint, not an instruction, and a crawler will still spend budget fetching a page before reading the hint.
Category architecture is the second lever and the one most often skipped. Category pages, not product pages, are what rank for the terms with volume. A store with excellent product pages and thin, templated categories is competing for the long tail while giving away the head.
Product structured data is the third. Price, availability, GTIN, brand and reviews, agreeing with what the page shows and with what the Shopping feed says. Where those three disagree, you lose on all of them. This is also what decides whether an AI assistant can include your products in a comparison answer, which is increasingly where the shortlist is made.
What we do
Eight areas. Large stores usually need the first three before anything else is worth doing.
Deciding which facet combinations deserve to be indexable pages and making the rest genuinely uncrawlable, rather than relying on canonical hints that still cost a fetch.
Building the category and subcategory structure that holds the terms with real volume, with pages that say something rather than listing products under a heading.
Price, availability, GTIN, brand and reviews, consistent with the visible page and with the Shopping feed. Disagreement between the three is a common and invisible suppressor.
Specifications, compatibility, sizing and the questions buyers actually ask, written once properly rather than generated into near-duplicates that search engines treat as thin.
Making your products available to assistants composing comparison answers. This depends on entity clarity and structured data rather than on links, which makes it unusually reachable.
A deliberate policy rather than whatever the platform does by default: what stays, what redirects, what returns gone. Done badly this leaks equity quietly for years.
Category and product templates are where ecommerce performance is won or lost. We fix at the cause: image delivery, script weight and layout shift from late-loading elements.
URL mapping product by product, verified after launch. Ecommerce migrations are where the largest single traffic losses we see happen, and they are avoidable.
Why this matters
The products exist, the pages exist, and the crawler never arrives. No amount of content investment fixes a page that is never fetched.
How much of your catalogue is actually in the index?
For most stores the honest answer is a fraction of it. The audit tells you the number and what is consuming the rest.
In practice
Six recurring situations across UK stores.
Discovered but not indexed, at scale. Nearly always faceted navigation generating more URLs than the site has crawl allowance for, combined with weak internal linking to deep products.
Crawl spent on things that can sell
Strong product pages and thin categories, so the store ranks for exact product names and nothing broader. Category work is usually the largest single organic gain available.
Visible for the terms with volume
Redirects mapped by pattern, which handled the tidy majority and dropped everything irregular, including the highest-value legacy URLs. We rebuild from the real URL list and verify after launch.
Equity carried across
Variants, filters, sorts and tracking parameters multiplying every product. Fixed structurally so crawlers stop generating the space rather than being asked to ignore it afterwards.
One canonical URL per product
A buyer asks an assistant to compare options and gets marketplaces and review sites. Usually because the store's own product data is not machine-readable and nothing corroborates it.
Products included in comparisons
Products removed outright, or left returning soft 404s. Over several seasons this quietly removes a large part of a store's accumulated authority with nobody noticing.
A policy that preserves what was earned
How we work
Structure before content, because content on an unindexed page earns nothing.
Crawl the store, compare against what is actually indexed, and work out where the crawl budget goes. Free, in writing, and yours to keep.
Fix faceted navigation, category architecture, internal linking and the out-of-stock policy, so the crawler reaches what matters and the right pages hold the right terms.
Category and product content written where it changes the outcome, product schema made consistent with the page and the feed, and the entity work that lets assistants include you.
Indexed coverage, organic revenue and AI comparison visibility, reported monthly in writing with what moved and what did not.
Clear scope · Regular demos · Full handover
Honest fit
Catalogue scale is what makes this work pay. A small clean store may need very little of it.
Sectors
Stores where catalogue scale makes indexation the deciding factor.
FAQ
What UK retailers ask before commissioning catalogue work.
Still have a question?
Ask us directly — a senior engineer will get back to you.
Ask about your projectUsually crawl budget. Faceted navigation creates far more URLs than the site has crawl allowance for, so the crawler spends its visits on filter combinations and never reaches the products. Weak internal linking to deep pages and near-duplicate templates compound it. The fix is structural, not editorial.
Only partly. A canonical is a hint, not an instruction, and the crawler still has to fetch the page before it reads the hint, which means the crawl budget has already been spent. On large stores you need the duplicate space to be uncrawlable, not merely deduplicated after the fact.
Categories, in most cases, and it is the more commonly skipped answer. Category pages hold the terms with real search volume, while product pages win exact product names. Stores with excellent products and thin categories are competing for the long tail and giving away the head.
Before launch, ideally while URL structure is still being decided. The largest traffic losses we see come from redirects mapped by pattern, which works for the tidy majority and silently drops the irregular URLs, which are often the valuable legacy ones. Involving us afterwards means recovering rather than preventing.
It depends on whether the product is coming back. Temporarily out of stock should stay and say so; permanently discontinued should usually redirect to the nearest equivalent or the parent category; neither should quietly vanish or return a soft 404. Over several seasons the default behaviour of most platforms leaks a lot of accumulated authority.
We can make it possible and much more likely, but no one can guarantee it. It depends on your product data being machine-readable and corroborated elsewhere, which is work we can do. Whether a given assistant includes you on a given day is the assistant's decision.
Shopify, WooCommerce, Magento, BigCommerce and custom builds. Because we are a software company we can usually implement in the platform rather than hand over a list, which matters for faceted navigation, where the fix is often in the template rather than in a setting.
Yes. Google Shopping and Google Ads are covered on their own pages, and for most retailers the feed work and the organic product data work overlap heavily, so doing both together avoids one undoing the other.
It depends on the size of the site and the scope of the work, so we do not publish a fixed price: the honest range is wide. The first conversation and the audit are free, and the quote follows them, in pounds, once we both know what the work actually is.
SpiderHunts Technologies Ltd, registered at Companies House under number 16718403, with an office at 182-184 High Street N, Office 14607, London E6 2JA. You are invoiced in pounds. The engineering team works from Lahore, which we state openly rather than implying a room full of desks in London.
Start here
Tell us the store and the platform. We will crawl it, compare against what is actually indexed, work out where the crawl budget is going and tell you what it would take to fix. Free, in writing, and yours to keep.
Thanks — that’s with us.
A senior engineer reads every enquiry and will get back to you shortly.
Book a time that suits you, or send a message and get an answer sooner.