Technical SEO that clears the path from crawl to index

Technical SEO decides eligibility, not position

Most search advice is about competition: what to say, and how to say it better than whoever currently sits above you. Technical SEO is a duller question that comes first. Can a crawler reach the page, render it, resolve it to a single address, and work out what it is about?

That distinction changes what a fix is worth. A content improvement moves one page a few positions if it lands. A technical fix either unlocks an entire class of pages or it does not, and when it does, the effect is a step change rather than a nudge. It also tends to stay fixed, which is why it is worth doing properly once instead of repeatedly.

The other reason to take it seriously is defensive. Technical damage is almost never caused by carelessness. It is caused by a developer adding a filter, a marketer duplicating a landing page for a campaign, or a replatform renaming every URL on the site. Each decision was reasonable in isolation. The traffic goes anyway.

What a crawl audit actually finds

We crawl the site the way a search engine would — render-enabled, following redirects, using Screaming Frog SEO Spider or Sitebulb — and then reconcile that crawl against three other datasets: your XML sitemaps, the Search Console page indexing report, and, on larger sites, raw server logs.

The reconciliation is where the findings live. A URL in the sitemap that the crawl cannot reach is orphaned, because nothing on the site links to it. A URL the crawl reaches that is absent from the sitemap is either deliberate or an oversight, and most site owners cannot say which without checking. A URL that Search Console reports as “Crawled — currently not indexed” is usually a quality or duplication judgement rather than a technical fault, and attacking it with technical fixes burns a sprint for nothing. Getting that classification right is most of the value of an audit.

The output is deliberately not a list of every deviation from best practice. Ten thousand duplicate title tags are one templating decision, not ten thousand problems, and a report that presents them as ten thousand problems is padding sold by the page.

Architecture, click depth and the crawl budget question

Internal linking is where technical SEO meets web design, because click depth is a navigation decision before it is a search one. Pages buried many clicks from the homepage are recrawled less often and treated as less important, and the remedy is usually a hub page or a change to the navigation rather than anything a crawler would call technical.

Crawl budget, meanwhile, is the most over-diagnosed problem in the field. Google’s own guidance scopes it to sites with more than a million unique pages that change roughly weekly, or more than ten thousand that change daily. Below that, an accurate sitemap and a server that responds promptly is generally enough. If someone has quoted you for crawl budget optimisation on a two-hundred-page brochure site, ask them to show you the constraint.

Indexation control: three tools that get confused constantly

robots.txt, noindex and rel="canonical" do three different jobs, and using one to do another’s job is the most common self-inflicted injury we find.

robots.txt controls crawling, not indexing. Google is explicit that a page disallowed in robots.txt can still be indexed if other sites link to it, with the URL and its anchor text appearing in results even though the content was never fetched. Blocking a page in order to keep it out of search is therefore the wrong instrument, and it produces the familiar “Indexed, though blocked by robots.txt” warning.

noindex removes a page from the index, but only if the crawler can fetch it. Combining a robots.txt disallow with a noindex tag guarantees the noindex is never seen. We find this pairing on a large share of the sites we audit, almost always applied by someone who reasonably assumed the two measures would reinforce each other.

rel=“canonical” is a strong signal, not an instruction. Google’s documentation ranks redirects as the strongest canonicalisation signal, rel=canonical as strong, and sitemap inclusion as weak. When Google selects a different canonical from the one you declared, that is a disagreement rather than a bug, and the remedy is to remove the ambiguity — consistent internal linking, one version in the sitemap, one version linked from navigation — rather than to repeat the declaration more emphatically.

Faceted navigation is where all three converge, and where large catalogues lose control. A store that generates a crawlable URL for every combination of colour, size and price band can produce more URLs than it has products. The resolution is a policy decision about which facet combinations have genuine search demand and deserve indexing, which should be crawlable but not indexed, and which should never be linked at all. That policy belongs in a document rather than in a developer’s memory, and it is a large enough problem in retail that e-commerce SEO treats it as a first-class concern.

JavaScript rendering, and the gap between source and screen

Google processes JavaScript in three phases: it crawls the URL, queues the page for rendering in a headless Chromium instance, and indexes the rendered HTML. Client-side content can therefore be indexed. The caveats are what catch people out.

Links must be real anchor elements with href attributes, because that is what Googlebot extracts to discover pages; a clickable div with a click handler is not a link as far as crawling is concerned. Scripts blocked in robots.txt cannot run, so the content they would have produced does not exist. Content that only appears after a user interaction may never be seen at all. And a client-side route that draws an error message while the server returns HTTP 200 is a soft 404: it will be crawled, judged empty, and quietly dropped.

There is a wider point that has grown more important. Several crawlers that matter commercially do not execute JavaScript, including some that feed answer engines and social previews. Google itself still recommends server-side rendering or pre-rendering, and that recommendation now carries more weight than it did when Googlebot was the only audience worth planning for. Rendering strategy is a build decision as much as a search one — see web development — and it should be settled before the framework is chosen rather than argued about afterwards.

Core Web Vitals: field data beats lab scores

The thresholds are public and specific. Largest Contentful Paint within 2.5 seconds, Interaction to Next Paint at or below 200 milliseconds, Cumulative Layout Shift at or below 0.1, each assessed at the 75th percentile of real page loads and segmented by device.

That percentile is the part most reports skip. Lighthouse and PageSpeed Insights lab runs simulate one visit on a synthetic device; Chrome UX Report data is what your actual users experienced. When the two disagree — a green lab score against failing field data — the field data is what counts, and the usual explanation is that your real audience is on slower devices and worse networks than the simulation assumed.

INP is where most sites now fail, because it replaced First Input Delay with something considerably harder to pass. FID measured the responsiveness of the first interaction only. INP looks at interactions across the whole visit and reports near the worst of them, so sites that felt fine under the old metric often discover that a heavy tag manager, an expensive hydration step or a long event handler has been costing them responsiveness for years.

We are also straightforward about the return. Page experience is a genuine signal but a weak one next to relevance, so performance work rarely produces a ranking jump on its own. It produces revenue, because people leave slow pages, and it makes paid traffic cheaper by wasting less of it. That is the honest business case, and it is the stronger one anyway.

Structured data that matches the page

Structured data does not improve rankings. It makes a page eligible for search features, and it helps a search engine resolve what an entity on the page actually is. Both are worth having; neither is a growth strategy.

We implement in JSON-LD against Schema.org types, validate with the Rich Results Test, and map every property to a field the CMS genuinely holds so the markup cannot drift away from the page. That last point is a policy rather than a preference. Google’s structured data guidelines require markup to describe content visible to the reader, and mismatched or invented markup risks losing rich result eligibility or attracting a manual action. Review markup on a page with no reviews is the classic example, and it is still remarkably common.

The types worth the maintenance are those tied to a feature you can actually earn or an entity worth disambiguating: organisation and site identity, breadcrumbs, articles, products with real offer data, and local business markup where physical locations exist. Entity clarity has become more valuable as answer engines started assembling responses from structured sources, but the requirement to describe what is really on the page has not loosened to accommodate that.

Hreflang, and why it usually fails

International sites can declare localised versions three ways — HTML link elements, HTTP headers, or annotations in the XML sitemap — and all three carry the same rule: the relationship must be reciprocal. If two pages do not both point at each other, the annotations are ignored entirely. There is no partial credit for a nearly-correct implementation.

Broken implementations nearly always fail for one of four reasons: the self-referencing annotation is missing, one variant points at a URL that redirects, the annotations sit on non-canonical URLs, or a language and region code has been invented rather than taken from the valid list. The x-default value is also widely misunderstood — it is the fallback for users whose settings match none of your versions, not a synonym for the English page.

Log files, and when they earn their keep

Log file analysis tells you what Googlebot actually requested, how often, and what status codes it received, as opposed to what you believe it should have done. On a small site this is theatre. It becomes genuinely useful once crawl allocation is a real constraint, or when indexing problems persist after the obvious causes have been eliminated.

What it typically reveals: crawl activity dominated by parameter URLs, redirect chains and 404s; important templates fetched far less often than trivial ones; and pages you assumed were ignored being crawled regularly and simply not indexed, which moves the diagnosis from technical to editorial.

Migrations, where this work pays for itself outright

A migration is the one project where technical SEO has an unambiguous, measurable value: the traffic you keep. It is also the project where the work is most often scheduled after launch, which is roughly like planning the parachute during the descent.

Every existing URL is inventoried first, from a crawl, from Search Console, from analytics and from backlink data. That last source matters because a page with almost no traffic may hold the external links propping up everything around it. Each URL is then mapped to a single-hop, server-side permanent redirect pointing at the closest genuine equivalent, not at the homepage. Google advises against long redirect chains and recommends retaining redirects for at least a year, which in practice means a year after the last important external link has been updated rather than a year after launch.

Then the new sitemap is submitted, internal links are rewritten to point at final destinations rather than leaning on the redirects, and index coverage is watched closely. Some ranking movement during a move is expected. Knowing which movement is normal reprocessing and which is a genuine loss is the real skill, and it is why the audit that precedes a migration matters more than the report that follows it.

Duplicate, thin and orphaned pages

Three different problems filed under one label. Duplication is a consolidation problem: several URLs competing to represent one thing, usually caused by parameters, protocol or trailing-slash variants, or the same item published under several categories. Thin pages are a quality problem, and no amount of technical work makes them worth indexing; the honest options are to improve, merge or remove them. Orphans are a linking problem: pages nothing links to, reaching the index through the sitemap and treated with the importance that implies.

Only one of the three is solved by a canonical tag. Applying a technical fix to a quality problem is how sites end up with a beautifully tidy index and exactly the same amount of traffic.

Monitoring, because sites regress

Most of the technical damage we are called in to repair was caused by a deployment. The next most common cause is a CMS setting changed by someone who did not know it was a search decision: a staging environment promoted with its noindex tag intact, a plugin update rewriting robots.txt, a sitemap that stopped regenerating and went unnoticed for months.

Continuous monitoring costs a fraction of an audit and catches more. Scheduled crawls, alerts on robots.txt and meta robots changes, sitemap URL counts, status code distributions, and field performance tracked by template. If the choice is between one thorough audit and modest monitoring running continuously, take the monitoring. Feeding that signal into the same place as the rest of your reporting is a job for analytics engineering rather than another dashboard nobody opens.

What actually drives the cost

We do not publish figures, because the same three words describe wildly different pieces of work. The variables are not secret, and knowing them will help you compare quotes honestly.

Site size and template count. Template count matters more than page count. A large site assembled from a dozen templates is a smaller job than a small site where every page is bespoke.

Platform. A mainstream CMS with a known set of failure modes is faster to audit than a bespoke application whose rendering behaviour has to be established from first principles.

Diagnosis versus implementation. An audit that ends in a document costs less than an engagement where we ship the fixes, and the right choice depends on whether your engineering team has capacity to act. An unimplemented audit is worth nothing, so be realistic about this at the start rather than the end.

Migrations and internationalisation. Both add work that has to land on a fixed date, and both are far cheaper to plan than to repair.

Where this sits alongside the rest of the work

Technical SEO is the foundation layer of the wider organic search programme, and on its own it is rarely sufficient — a perfectly crawlable site still has to say something worth ranking. It overlaps most heavily with the build, which is why we would rather be in the room while the site is being developed than called in afterwards to explain why the new framework cost you a third of your traffic.

If you are mid-migration, or watching index coverage fall and wanting a second opinion, tell us what you are seeing and we will say whether it is a technical problem at all. Often enough it is not, and that is a cheaper answer than the alternative.