Crawlability and crawl budget
Finding what blocks or wastes a crawl: robots rules, orphaned pages, infinite filter spaces, soft 404s and redirect chains. On large sites the gain usually comes from stopping waste rather than from adding anything.
Most technical SEO reports end where the work starts. We are a software company, so we find the problem and then fix it: crawl and index diagnosis, Core Web Vitals, JavaScript rendering, site architecture, schema, migrations and log-file analysis. You get the diff, not a spreadsheet of recommendations.
Thanks, that’s with us.
A senior engineer reads every enquiry and will reply as soon as they can.
Book a time that suits you, or send a message on WhatsApp.
2020
Building since
1,000+
Clients served
UK
Registered company
4.6/5
Rated on Clutch
Quick answer
Technical SEO is the work of making sure a search engine can reach your pages, render them, understand them and choose to index them. It sits underneath content and links: no amount of either helps a page a crawler never fetches or a renderer gives up on.
The most common findings are dull and expensive. Pages excluded by a rule nobody remembers writing. Canonical tags pointing at the wrong URL after a replatform. Content that only appears once JavaScript runs, on a crawl budget that will not wait. Thousands of near-duplicate URLs generated by filters, eating the crawl allowance that should be spent on products. A migration where the redirects were mapped by pattern and quietly lost a third of the equity.
What makes us different is what happens after the audit. Most technical SEO engagements deliver a document and a meeting, and the fixes then sit in a backlog behind the next feature release. We are a software company first, so we can do the implementation: write the redirects, change the rendering, correct the schema, restructure the internal links and verify in production that the change took.
There is a newer part to this as well. AI crawlers fetch differently from Googlebot, generally without executing JavaScript, so a site that renders client-side can be fully indexed by Google and effectively invisible to ChatGPT or Perplexity. We check both. If AI visibility is the main concern, start with AI SEO.
What we do
Eight areas. An audit tells you which ones are costing you, and the fix work is prioritised by value rather than by how long the list is.
Finding what blocks or wastes a crawl: robots rules, orphaned pages, infinite filter spaces, soft 404s and redirect chains. On large sites the gain usually comes from stopping waste rather than from adding anything.
Working out why pages sit at Crawled or Discovered but not indexed. The cause is usually duplication, thin templated content, weak internal linking or a canonical conflict, and the four need different fixes.
LCP, INP and CLS fixed at the cause: image and font delivery, render-blocking scripts, layout shift from late-loading elements, and main-thread work. We measure in the field, not only in a lab tool that flatters the result.
Internal linking, hierarchy and pagination so that importance flows to the pages that should earn. This is the lever most often ignored because it touches navigation, and it is frequently the one that moves indexation.
Comparing the raw HTML against the rendered DOM to find content that only exists after scripts run. Then fixing it properly, usually by server-rendering what matters rather than hoping the renderer keeps up.
Schema that validates and also matches its page. We find duplicated properties, markup describing the wrong entity and orphaned blocks left by old plugins, and generate the replacement from the page content so it cannot drift.
URL mapping done page by page rather than by pattern, with redirects tested before launch and verified after. Most migration losses trace to a mapping that looked right in a spreadsheet and was never checked against real traffic.
Server logs show what search and AI crawlers actually fetch, as opposed to what you assume. We use them to find waste and blind spots, and to confirm GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended can reach what matters.
Why this matters
Technical faults are invisible from the front end. The site looks fine, the content is good, and the traffic quietly fails to arrive.
Not sure whether the problem is technical or something else?
That is what the audit settles. We crawl the site, compare it against what is actually indexed, and tell you plainly whether the ceiling is technical, content or authority.
In practice
Six recurring situations. These describe the pattern we see, not named client engagements.
Large catalogues where most URLs sit at Crawled, currently not indexed. The cause is almost never a single setting. It is usually near-duplicate templates, weak internal linking to deep pages, and a crawl budget spent on filter combinations.
Crawl spent on pages that can earn
A framework site where the HTML served to a crawler is close to empty. Google often copes; AI crawlers usually do not. The fix is to server-render the content that matters rather than rely on a second rendering pass.
The same content for humans and machines
Redirects mapped by URL pattern, which worked for the tidy majority and silently dropped everything irregular. We rebuild the mapping from the old URL list and real traffic data, then verify after launch rather than assuming.
Equity carried across instead of written off
Filters, sorts, session parameters and tracking tags multiplying every page. The answer is usually a combination of canonicals, parameter handling and robots rules, chosen so crawlers stop generating the space in the first place.
One canonical URL per thing
Usually images served at desktop size, fonts blocking paint, and a layout that shifts when a banner or consent bar loads late. Fixing the cause is faster than the deferral tricks that move the number without helping anyone.
Field data that matches the lab score
A site that Googlebot renders successfully and GPTBot receives as a near-empty shell. Nothing in Search Console reports this, so it goes unnoticed until someone asks why ChatGPT has never heard of the company.
Reachable by both kinds of crawler
How we work
The audit is free. What follows is scheduled by what each fix is worth, not by the order items appear in a report.
A full crawl, compared against what Search Console reports as indexed, plus a rendered-DOM comparison and server logs where available. The aim is to separate symptoms from causes before anyone writes a recommendation.
Each finding gets an estimate of what fixing it is worth and what it costs to do. Some well-known best practices are not worth the engineering time on a given site, and we will say which.
We make the changes: redirects, canonicals, rendering, schema, internal linking, performance. You approve before anything goes live, and the work arrives as reviewable changes rather than instructions for someone else.
Re-crawl, re-check indexation, confirm in logs that crawlers see the change, and watch for regressions. A fix that was deployed but did not take is the failure mode that matters most here.
Clear scope · Regular demos · Full handover
Honest fit
Technical work only pays when technical faults are what is holding the site back. Often they are not.
What you are buying
Three shapes of engagement. Most clients start in the middle column.
| Audit only | Audit + implementation | Ongoing technical partner | |
|---|---|---|---|
| You get | Findings, ranked, in writing | Findings plus the fixes done | Continuous diagnosis and fixing |
| Who implements | Your team | Us, with your approval | Us, as part of your release cycle |
| Best when | You have engineers free | The backlog is full | The site changes constantly |
| Typical length | One pass | Weeks | Monthly |
| Verification | Yours to run | We verify in production | Continuous monitoring |
| Risk it addresses | Not knowing | Knowing but never shipping | Regression after each release |
Sectors
Large, templated or frequently changing sites, where a single rule can affect thousands of pages.
FAQ
What UK buyers ask before commissioning technical work.
Still have a question?
Ask us directly. A senior engineer reads every question.
Ask about your projectIt makes sure search engines can reach your pages, render them, understand them and choose to index them. In practice that means diagnosing crawl and indexation problems, fixing rendering and performance, correcting structured data and site architecture, and handling migrations without losing equity. We also implement the fixes rather than only reporting them.
No, and that is the main difference. Most technical audits end with a document whose recommendations sit in a backlog behind product work. We are a software company, so we can write the redirects, change the rendering, fix the schema and verify in production. You can still buy audit-only if you have engineers free.
For most sites, a few days to crawl, cross-check against Search Console and review logs, then the findings in writing. Very large or unusual sites take longer. Implementation time depends entirely on what is found, and we estimate it per fix rather than as one number.
Usually one of four things: near-duplicate templated content, weak internal linking so deep pages look unimportant, a canonical conflict, or crawl budget being consumed by filter and parameter URLs. They need different fixes, which is why the diagnosis matters more than the checklist.
Yes, and it is worth involving us before launch rather than after. Most migration losses come from redirects mapped by URL pattern, which handles the tidy majority and silently drops everything irregular. We build the mapping from the real URL list and traffic data, test before launch and verify after.
Either way. We can hand over reviewable changes for your team to merge, or implement directly if you would rather. Where there is an in-house team we usually find it faster to write the change ourselves and have them review it, because the context is already in our heads.
More than before, in one specific way. AI crawlers generally do not execute JavaScript, so a client-rendered site can be fully indexed by Google and effectively invisible to ChatGPT or Perplexity. Nothing in Search Console reports that, so it goes unnoticed. We check both kinds of crawler.
No. Google decides what ranks and nobody can sell you a position. What technical work can honestly promise is that nothing on your side is preventing a page from being fetched, rendered, understood and indexed. That is a necessary condition for ranking, not a sufficient one.
Technical projects start from £2,500, and a migration from £4,000, because the mapping and the risk are larger. Ongoing technical work sits in a retainer from £1,500 a month. A small clean site is a fraction of a large catalogue, so the audit is free and the quote follows it.
SpiderHunts Technologies Ltd, registered at Companies House under number 16718403, with a registered office at 182-184 High Street N, Office 14607, London E6 2JA. You are invoiced in pounds. The whole team works remotely, which we state plainly.
Start here
Tell us the site and what you suspect. We will crawl it, compare it against what is really being indexed, check how it renders for both search and AI crawlers, and send you the findings with the fixes ranked by what they are worth. Free, and yours to keep.
Thanks, that’s with us.
A senior engineer reads every enquiry and will reply as soon as they can.
Book a time that suits you, or send a message on WhatsApp.