Free Audit

Home / SEO Guide / Chapter 5

Chapter 5 of 12

Technical SEO and Site Architecture

Technical SEO is the foundation everything else builds on. If search engines and AI crawlers cannot access, render, and understand your site, no amount of content quality or link building will save you.

Published
Technical SEO and site architecture - Core Web Vitals crawling indexing
Photo by AISEOShift on AISEOShift

Technical SEO is the practice of ensuring that your website is fully accessible to search engine crawlers and AI retrieval systems, that it meets performance standards that affect rankings, and that its architecture helps both humans and machines find and understand your content. It is the least visible form of SEO and often the most impactful.

A technically sound site creates the conditions for everything else to work. Your keyword-targeted content cannot rank if Google cannot crawl it. Your carefully structured on-page SEO work cannot be cited by AI systems if those systems cannot access your pages. Your internal links cannot distribute authority if your site architecture creates dead ends and orphan pages. Technical SEO is not optional. It is foundational.

Core Web Vitals and Page Performance

Core Web Vitals are a set of three performance metrics — Largest Contentful Paint (LCP), Interaction to Next Paint (INP), and Cumulative Layout Shift (CLS) — that Google uses as ranking signals to evaluate user experience. Google defines and documents these metrics on web.dev, and they are part of the broader page experience signals outlined on the Google Search Central Blog. They became an official ranking factor in 2021, and their influence has only increased since then. In 2026, pages that fail Core Web Vitals thresholds face a measurable ranking disadvantage, especially on mobile.

Largest Contentful Paint (LCP)

LCP measures how long it takes for the largest visible content element to render. This is typically a hero image, a large text block, or a video thumbnail. Google considers LCP “good” if it occurs within 2.5 seconds of when the page first started loading.

Common LCP problems include unoptimized hero images, slow server response times, render-blocking JavaScript and CSS, and excessive third-party scripts. Solutions include serving images in modern formats (WebP or AVIF) at appropriate dimensions, implementing a content delivery network, preloading critical resources, and deferring non-essential scripts.

Interaction to Next Paint (INP)

INP replaced First Input Delay as the responsiveness metric in 2024. It measures the delay between a user interaction (click, tap, or keypress) and the next visual update. Google considers an INP of 200 milliseconds or less to be “good.”

INP problems are almost always caused by heavy JavaScript execution on the main thread. Long-running scripts, excessive DOM manipulation, and poorly optimized event handlers all degrade INP. Solutions include breaking up long tasks, using web workers for heavy computation, debouncing input handlers, and reducing JavaScript bundle sizes.

Cumulative Layout Shift (CLS)

CLS measures visual stability, how much page elements shift position during loading. Users experience this as text jumping, buttons moving, or content reflowing unexpectedly. Google considers a CLS score of 0.1 or less to be “good.”

Layout shifts are caused by images without defined dimensions, ads that load after surrounding content, dynamically injected content, and web fonts that cause text to reflow. Setting explicit width and height attributes on images and iframes, reserving space for ad slots, and using font-display: swap with size-adjusted fallback fonts all reduce CLS.

Performance Monitoring

Core Web Vitals should be monitored continuously, not checked once. Google Search Console provides a Core Web Vitals report that flags pages with issues. Chrome User Experience Report (CrUX) data shows real-world performance from actual Chrome users. Lighthouse and PageSpeed Insights provide lab-based diagnostics that help identify specific problems.

The important distinction is between lab data and field data. Lab data (Lighthouse) measures performance under controlled conditions. Field data (CrUX) measures what real users actually experience. Google uses field data for ranking decisions, so field data is what ultimately matters.

Mobile-First Indexing and Responsive Design

Mobile-first indexing means Google predominantly uses the mobile version of your content for indexing and ranking. Google completed its transition to mobile-first indexing years ago. The mobile version of your site is now the primary version that Google crawls, indexes, and uses for ranking decisions. If your mobile site has less content, fewer links, or worse usability than your desktop site, your rankings reflect the mobile version.

What Mobile-First Means in Practice

Every page on your site should deliver the same content, the same internal links, and the same structured data on mobile as on desktop. Responsive design that adapts layout while maintaining content parity is the standard approach. Separate mobile URLs (m.example.com) and dynamic serving are technically supported but create additional complexity and maintenance burden.

Test your pages on actual mobile devices, not just by resizing a desktop browser. Touch target sizes, font readability, horizontal scrolling, and intrusive interstitials all affect mobile usability. Google’s mobile usability report in Search Console flags specific issues.

Mobile Page Speed

Mobile connections are typically slower and less reliable than desktop connections. Page weight matters more on mobile. Aggressive image optimization, reduced JavaScript payloads, and server-side rendering or static generation all improve mobile performance. Consider that a significant portion of your mobile audience may be on 3G or unstable 4G connections, particularly in markets outside major metro areas.

Crawlability and Crawler Management

Crawlability is the ability of search engine bots and AI crawlers to access and download your web pages. If search engines and AI systems cannot reach your pages, nothing else matters. Crawlability is the most fundamental technical SEO requirement.

Robots.txt

The robots.txt file is your primary tool for controlling crawler access. It lives at the root of your domain (example.com/robots.txt) and uses simple directives to allow or disallow specific crawlers from accessing specific paths.

In 2026, robots.txt management has become more complex because you may want different access rules for different crawlers. You might allow Googlebot full access while restricting GPTBot to certain directories, or allow PerplexityBot to crawl your blog while blocking it from your documentation. Each crawler is identified by its user-agent string, and you can specify separate rules for each.

Common robots.txt mistakes include accidentally blocking CSS and JavaScript files (which prevents Google from rendering your pages), blocking entire directories that contain important content, and using overly broad wildcard patterns that match more URLs than intended. Always test robots.txt changes in Google Search Console’s robots.txt tester before deploying.

AI-Specific Access Files

Two newer standards have emerged for communicating with AI systems, a direct result of the AI SEO shift that expanded technical SEO beyond traditional crawlers. These are covered in detail in our LLM crawler optimization guide.

ai.txt follows a similar format to robots.txt but is specifically designed for AI systems. It allows you to communicate preferences about how your content should be used, including whether it can be used for training, whether it can be cited, and what attribution you expect.

llms.txt provides a machine-readable summary of your site’s content specifically designed for LLM consumption. It typically includes a brief description of your site, your key content areas, and direct links to your most important pages. Think of it as a cover letter for AI systems: it gives them context about who you are and what you publish before they start crawling.

These files are not universally supported yet, but adoption is growing. Having them in place positions your site for the emerging standard while providing immediate benefits for the AI systems that do support them.

XML Sitemaps

An XML sitemap lists all the pages on your site that you want search engines to index. It provides metadata about each page including the last modification date, change frequency, and relative priority. While Google says it can find most pages through crawling, sitemaps accelerate discovery, especially for large sites or new pages.

Best practices for sitemaps include keeping each sitemap file under 50,000 URLs (use a sitemap index for larger sites), including only canonical URLs, updating the lastmod date only when content actually changes, and excluding URLs that are noindexed, redirected, or blocked by robots.txt.

Submit your sitemap through Google Search Console and Bing Webmaster Tools. Monitor the sitemap report to ensure that submitted URLs are being indexed as expected.

Crawl Budget Optimization

Crawl budget is the number of pages a search engine will crawl on your site within a given time period. For most sites under 10,000 pages, crawl budget is not a concern. For larger sites, optimizing crawl budget ensures that your most important pages get crawled frequently while low-value pages do not waste crawler resources.

Techniques for optimizing crawl budget include eliminating duplicate content, fixing redirect chains, removing or noindexing low-value pages, improving server response times, and using internal linking to prioritize important pages. Faceted navigation on e-commerce sites is a particularly common source of crawl budget waste, as it can generate thousands of near-duplicate URLs from filter combinations.

Indexing Controls

Indexing controls are HTML meta tags, HTTP headers, and directives that tell search engines whether to add a page to their search index. Crawling and indexing are separate processes. A page can be crawled but not indexed, or indexed but not ranked. Understanding the tools that control indexing is essential for managing which pages appear in search results.

Meta Robots Tags

The meta robots tag in a page’s head section controls how search engines index and follow the page. Common directives include noindex (do not add this page to the index), nofollow (do not follow links on this page), nosnippet (do not show a text snippet in search results), and max-snippet (limit the length of text snippets).

Use noindex for pages that should not appear in search results: internal search results, filtered views, staging pages, thin tag pages, and paginated archives beyond page one. Be careful not to noindex pages that should rank, as this is a surprisingly common mistake, often introduced during development and never removed.

Canonical Tags

The canonical tag tells search engines which URL is the definitive version of a page when multiple URLs contain the same or similar content. This consolidates ranking signals and prevents duplicate content issues.

Every indexable page should have a self-referencing canonical tag. Pages with duplicate or near-duplicate content should canonical to the preferred version. Canonical tags are hints, not directives. Google may ignore a canonical tag if it contradicts other signals like internal links or sitemaps.

Common canonical mistakes include pointing canonical tags to redirected URLs, using relative URLs instead of absolute URLs, and setting canonical tags inconsistently across HTTP/HTTPS or www/non-www versions.

Indexing for AI Systems

Traditional indexing controls (noindex, canonical) only affect traditional search engines. AI crawlers may still access and use content from noindexed pages unless you specifically block them via robots.txt. This creates potential conflicts: you might want a page to be accessible to AI systems for citation purposes but not appear in Google’s organic results, or vice versa.

Managing these distinctions requires a combination of robots.txt rules for specific AI crawlers, meta tags for traditional search engines, and potentially ai.txt directives for AI-specific preferences.

HTTPS and Security

HTTPS has been a confirmed Google ranking factor since 2014, and in 2026 there is no reason for any page to be served over HTTP. Let’s Encrypt provides free SSL certificates, and most hosting providers include HTTPS by default.

Beyond the ranking signal, HTTPS protects user data, prevents content injection by intermediaries, and signals trustworthiness to both users and AI systems. AI crawlers may deprioritize or skip HTTP-only sites as a proxy for site quality and maintenance.

Ensure your HTTPS implementation is complete: all pages, images, scripts, and resources should be served over HTTPS. Mixed content (HTTPS pages loading HTTP resources) causes browser warnings, and can cause pages to be treated as insecure. Set up permanent 301 redirects from HTTP to HTTPS for all URLs.

JavaScript Rendering and Client-Side Content

JavaScript rendering is the process by which search engines execute client-side JavaScript to discover and index dynamically generated content. JavaScript-heavy sites, particularly single-page applications built with React, Vue, or Angular, present specific challenges for both search engine crawlers and AI systems.

The Rendering Problem

Googlebot has a two-phase indexing process. In the first phase, it downloads the HTML and indexes whatever content is immediately available. In the second phase, it renders the page by executing JavaScript and indexes the dynamically generated content. The second phase may be delayed by hours or days, and some JavaScript execution fails entirely.

AI crawlers typically do not render JavaScript at all. They process the raw HTML response. If your content is rendered client-side through JavaScript, AI crawlers see an empty page. This is one of the most common and damaging technical SEO issues for modern web applications.

Solutions

Server-side rendering (SSR) generates full HTML on the server for each request. The initial response contains all content, making it immediately accessible to all crawlers. Frameworks like Next.js, Nuxt, and SvelteKit all support SSR.

Static site generation (SSG) pre-renders pages at build time, producing static HTML files that are both fast and fully crawlable. This is ideal for content-heavy sites where pages do not change on every request. Astro, Next.js, Hugo, and Eleventy are popular static generators.

Hybrid rendering combines SSR or SSG for content pages with client-side rendering for interactive elements. This approach gives you the performance and crawlability benefits of server rendering while maintaining the interactivity of client-side applications.

Dynamic rendering serves pre-rendered HTML to crawlers while serving the JavaScript version to regular users. Google officially supports this approach but has signaled that SSR or SSG are preferred long-term solutions.

Site Architecture

Site architecture is how your pages are organized and connected. Good architecture helps users find what they need, helps crawlers discover and prioritize pages, helps Google understand topical relationships, and helps AI systems map your site’s knowledge.

Flat vs. Deep Hierarchies

A flat architecture means every page is reachable within a few clicks from the homepage. A deep architecture buries pages behind many layers of navigation. For SEO, flatter is generally better. Pages closer to the homepage in terms of click depth tend to receive more crawl attention and more internal link equity.

The practical guideline is that every important page should be reachable within three clicks from the homepage. This does not mean every page belongs in the top-level navigation. It means your internal linking structure should provide multiple pathways to important content.

Topic Clusters and Pillar Pages

The topic cluster model organizes content around central pillar pages that cover broad topics comprehensively, supported by cluster pages that address specific subtopics in depth. Internal links connect the pillar to its clusters and the clusters to each other.

This architecture signals topical authority to both Google and AI systems. When your site has a pillar page on “email marketing” linked to twenty cluster pages covering specific email marketing subtopics, the entire cluster reinforces Google’s understanding that your site is a comprehensive resource on that topic. AI systems evaluating your content for citation see the same topical depth.

For detailed guidance on structuring pages within this architecture, see our post on how to structure pages for AI extraction.

URL Structure

URLs should be descriptive, concise, and logically organized. A URL like “/blog/technical-seo-audit-guide/” is more informative than “/blog/post-12847/” for both humans and machines. Use hyphens to separate words, keep URLs lowercase, and avoid unnecessary parameters.

URL structure should reflect your site’s content hierarchy. If your site covers marketing topics, a structure like “/marketing/email/subject-line-best-practices/” communicates the topical relationship. Google uses URL structure as a minor relevance signal, and clean URLs improve click-through rates in search results.

Breadcrumbs provide both a user navigation aid and a structured signal to search engines about your page hierarchy. Implement breadcrumbs with BreadcrumbList schema markup so Google can display them in search results, replacing the raw URL with a readable path.

Breadcrumbs also help AI systems understand where a page sits within your site’s topical structure. A breadcrumb path like “Home > Marketing > Email Marketing > Subject Lines” tells an AI system that this page is a specific subtopic within a broader knowledge domain.

Pagination and Infinite Scroll

For content that spans multiple pages, such as category archives, search results, or long-form articles, proper pagination implementation is essential. Use rel=“next” and rel=“prev” link elements (though Google has stated it no longer uses these for indexing, other search engines may), and ensure that paginated pages have unique, descriptive titles and are accessible through crawlable links.

Infinite scroll, which loads new content as the user scrolls, creates crawlability problems. Crawlers do not scroll. If additional content loads only through JavaScript-triggered scroll events, crawlers will only see the initial content load. Always provide a paginated HTML fallback for crawlable navigation.

Technical SEO Monitoring and Auditing

Technical SEO is not a one-time project. It requires ongoing monitoring and periodic auditing to catch issues before they affect rankings.

Essential Monitoring Tools

Google Search Console is the primary monitoring tool. Its Coverage report shows indexing issues, its Core Web Vitals report tracks performance, its Mobile Usability report flags responsive design problems, and its crawl stats show how Googlebot interacts with your site.

Server logs reveal how all crawlers interact with your site, including AI crawlers that Search Console does not report on. Analyzing server logs can identify crawl waste, blocked crawlers, and unexpected access patterns.

Audit Frequency

Run a comprehensive technical SEO audit quarterly for most sites and monthly for large or rapidly changing sites. Spot-check critical metrics weekly. Automate monitoring for Core Web Vitals, indexing status, and crawl errors to catch issues promptly.

The tools and processes described in this chapter create the foundation that all other SEO efforts build upon. In the next chapter, we cover schema markup and structured data, which bridges the gap between raw content and machine-readable information.

Frequently asked questions

What is technical SEO?

Technical SEO is the practice of optimizing a website’s infrastructure so that search engines and AI systems can crawl, render, index, and understand its content. It covers server performance, site architecture, crawl management, indexing controls, and structured data implementation. Unlike content-focused SEO, technical SEO deals with the underlying systems that make content discoverable.

What are Core Web Vitals?

Core Web Vitals are three specific metrics that Google uses to measure real-world user experience: Largest Contentful Paint (loading speed), Interaction to Next Paint (interactivity), and Cumulative Layout Shift (visual stability). They became an official Google ranking signal in 2021, and pages that fail the recommended thresholds face a measurable disadvantage in search results, especially on mobile.

Should I block AI crawlers in robots.txt?

Whether to block AI crawlers depends on your goals. If you want AI systems like ChatGPT, Perplexity, or Google AI Overviews to cite your content, you should allow their crawlers (GPTBot, PerplexityBot, ClaudeBot) access to your pages. Blocking them prevents your content from appearing in AI-generated answers, which is an increasingly significant traffic and visibility channel.

What is the difference between crawling and indexing?

Crawling is when a search engine bot discovers and downloads your web pages, while indexing is when the search engine analyzes that content and adds it to its searchable database. A page can be crawled but never indexed if it has a noindex tag, and a page cannot be indexed if it is blocked from crawling via robots.txt.

Go Deeper

These blog posts explore the topics covered in this chapter in more detail: