How Search Engines Work: Crawling, Indexing & Ranking

How Search Engines Work: Crawling, Indexing, and Ranking

Search engines work in three stages — crawling, indexing, and ranking — and Google is explicit that getting through each stage is not guaranteed. Crawling is the discovery step: Googlebot finds URLs, mostly by following links and reading sitemaps, then fetches them and renders each page with an evergreen Chromium engine. Indexing is the storage step: Google processes the fetched page, resolves which URL is canonical, and decides whether the page earns a place in the Google index. Ranking, which Google also calls serving, is the ordering step: when someone searches, Google’s ranking systems select and order the most relevant, highest-quality indexed pages for that exact query.

The single most useful thing to understand about this pipeline is that the three stages are separate gates. A URL can be crawled and never indexed, because Google judged it not worth storing. A page can be indexed and never reach page one, because the relevance and quality systems ranked other pages above it. And — the classic trap — a page can be blocked from crawling yet still appear in results if other pages link to it. So when a page is not showing up, the first diagnostic question is never “why does it not rank” but “which gate did it fail?” Table 1 lays out the three stages side by side.

This crawl → index → rank pipeline is the engine underneath everything covered in what SEO is, and it is the reason the free crawl-and-index dashboard, Google Search Console, is the first tool every practitioner reaches for. The rest of this guide goes gate by gate: the control that governs crawling (robots.txt), the discovery aid that feeds it (XML sitemaps), the large-site concern that constrains it (crawl budget), the default rendering it uses (mobile-first indexing), and the unsampled record of what actually happened (log-file analysis).

Crawling, Indexing, and Ranking Are Three Separate Gates

The most expensive mistake in technical SEO is assuming that a page which cleared one gate cleared all three. Google’s own documentation frames search as three stages and states plainly that getting through each stage is not guaranteed. That means “not showing up” is not one problem — it is three possible problems, and the fix is completely different at each. Treating a page that Google never indexed as if it were a ranking problem is how teams waste weeks tuning content on a URL that never entered the index in the first place.

The diagnostic sequence follows the pipeline. First, was the page discovered and crawled? If Googlebot never fetched it, no amount of content work helps — you fix discovery (internal links, sitemap) and crawlability (robots.txt, server status). Second, was it indexed? A crawled page reported as “Crawled – currently not indexed” failed the storage gate, usually on quality, duplication, or a stray noindex — you fix the page, not the keywords. Third, is it indexed but ranking below page one? That is the only genuine ranking problem, and it is where content relevance, links, and page experience come in. Search Console’s URL Inspection tool tells you which gate a URL is at; server logs tell you whether Googlebot ever arrived at all.

Getting your pages cleanly through the crawl and index gates is the heart of technical SEO, and the report that shows exactly which URLs cleared the indexing gate — and which were excluded, and why — is the Index Coverage report in Search Console. MB Adv Agency treats the three-gate model as the starting point of every organic-search diagnosis, because it turns a vague “the page is not working” into a precise, testable question with a specific fix.

What People Search: Demand for the Crawl, Index, and Rank Terms

The demand behind this topic is dominated by one term: robots.txt, at 15,000 US searches per month (Ahrefs Keywords Explorer, US, 2026-07-01) — by far the highest-volume query in the cluster, and also one of the hardest at KD 87. The get-started head term, how do search engines work, runs 1,500 per month at KD 44, the most winnable of the set and the reason this pillar leads with the literal three-stage answer. Table 2 shows the search demand and difficulty for the five core terms this guide consolidates.

The strategy this data dictates is direct: win the lower-difficulty get-started term, build the deepest single reference on the topic, and give each high-volume control term (robots.txt, xml sitemap, crawl budget, mobile-first indexing) its own deep-linkable section on this page. The sections that follow do exactly that, gate by gate.

robots.txt Controls Crawling, Not Indexing

The correction that earns the most citations, because so much published content still gets it wrong, is this: robots.txt controls crawling, not indexing. A robots.txt file is a Robots Exclusion Protocol file — standardized as IETF RFC 9309 in 2022 — that tells compliant crawlers which URLs not to fetch. It does not remove a page from the index. A URL disallowed in robots.txt but linked from elsewhere can still be indexed, shown as a bare URL with no snippet, because Google could not read the page to write one.

The intuitive fix backfires. To keep a page out of the index you use noindex — a meta robots tag or an X-Robots-Tag header — but Google only sees that rule if it is allowed to crawl the page. Block the page in robots.txt and Googlebot never fetches it, never sees the noindex, and the URL can linger in results indefinitely. The load-bearing rule is therefore precise: to stop crawling, use robots.txt; to stop indexing, use noindex and leave the page crawlable. Table 3 is the decoder that settles the confusion in one glance.

Because the two controls solve opposite problems, the same page should almost never carry both. If you want a low-value section fetched less, use robots.txt; if you want a specific page kept out of results, use noindex and confirm the page stays crawlable so Google can read the rule. The report that surfaces a URL indexed against your intent — or excluded when you wanted it in — is the Index Coverage report, which is where this decoder becomes a daily diagnostic rather than a piece of trivia.

The robots.txt Directives Google Supports

A robots.txt file lives at the domain root (/robots.txt), is served as UTF-8 plain text, and is built from a small set of directives. Getting the syntax right matters, because a single overbroad Disallow can hide an entire section from Googlebot. Table 4 lists the directives Google actually supports, and flags the one that many older tutorials still tell you to use but Google ignores.

The practical takeaway is that robots.txt is a scalpel for crawl control, not a switch for removing pages. Use it to keep Googlebot out of infinite faceted combinations, internal search results, and cart or checkout paths; point it at your sitemap with the Sitemap: line; and manage crawl rate through server speed and health rather than a Crawl-delay line Google never reads. The server-health and measurement plumbing that supports this — the paid-media analog of getting your technical foundation right — is the kind of work covered by server-side tracking.

XML Sitemaps: A Discovery Aid With Hard Limits

An XML sitemap is a machine-readable list of your URLs that helps Google find pages it can miss — deep, new, or sparsely linked ones — along with hints such as last-modified dates. It is a discovery aid, not an indexing guarantee: Google still decides independently what to index and how to rank it. A sitemap that lists noindex, non-canonical, redirected, or blocked URLs actively hurts, because it sends mixed signals about which URLs you actually want. Table 5 answers the limit questions people search for directly.

The capacity ladder is the teaching point: a single file caps at 50,000 URLs, and nesting up to 50,000 sitemaps inside one sitemap index addresses up to roughly 2.5 billion URLs — one file suffices for a small site, an index is the pattern for an enterprise catalog. The mistakes that hurt are structural, not size-related: keep noindex, non-canonical, redirected, and blocked URLs out. You submit and monitor sitemaps — and inspect a single URL’s crawl and index status — in Sitemaps and URL Inspection inside Search Console.

Crawl Budget Is a Large-Site Concern

Crawl budget is real, and for the overwhelming majority of sites it is not your problem. Google’s large-site guide draws the line clearly: crawl budget is worth managing only for a very large site — 1,000,000+ pages changing roughly weekly — or a medium-plus site of 10,000+ pages changing daily. Those are large ecommerce catalogs with faceted navigation, marketplaces, and big publishers. Below that, Google states plainly that most sites are crawled efficiently and never need to manage it. Table 6 puts the thresholds next to real site profiles.

Where crawl budget applies, the levers are technical: fix slow and 5xx responses, remove low-value URLs (infinite faceted combinations, session IDs, endless calendars), consolidate duplicates, and keep sitemaps and internal links pointed at the canonical URLs you actually want crawled. Where it does not apply — on a 300-page site — the honest advice is to spend the day on content and links instead. Large SaaS sites tend to hit these limits first, which is why we pair the crawl-budget conversation with our SaaS software PPC work; big ecommerce catalogs with faceted navigation are the other classic case, matched by our fashion PPC campaigns on the paid side.

HTTP Status Codes and How Googlebot Treats Them

When a page is not indexed, the status code Googlebot received is often the fastest diagnostic. A page has to return a clean 200 to be eligible for indexing; redirects, errors, and blocks each change the outcome in a specific, predictable way. Table 7 is the reference for reading “why is this page not indexed?” from the response Googlebot actually got.

Reading these outcomes correctly saves real time. A page stuck out of the index because it returns a soft 404 or a 5xx is a server problem, not a content problem; a page that resolves through a chain of 302s keeps the wrong canonical; and a section blocked in robots.txt but still surfacing as a bare URL is the crawling-versus-indexing confusion from Table 3, seen from the status side. You cross-check all of these against Google’s own view of each URL in the Index Coverage report.

Log-File Analysis: The Unsampled Crawl Truth

To know what Googlebot really did — which URLs it fetched, how often, with what status codes, and where it wasted crawl — the unsampled source of truth is your server access logs. Every request is recorded: URL, timestamp, HTTP status, user-agent, and bytes. Search Console’s Crawl Stats report is useful, but it is a sampled subset; the raw logs are complete. Because the user-agent string is trivially spoofed, verify a request is genuinely Googlebot by reverse-DNS or Google’s published IP ranges before you trust a log line. Table 8 shows what each log signal reveals.

Logs are the unsampled ground truth of every request; Crawl Stats is a useful but sampled view. This gap is how large sites find orphaned pages Googlebot never reaches, crawl waste on parameter URLs, and the real crawl frequency of their revenue pages — none of which a sampled dashboard reliably shows. The same discipline of measuring what actually happened, rather than what a dashboard estimates, is how MB Adv Agency runs the paid Google campaigns that sit alongside this organic work; keeping both accountable to real data is exactly what our PPC campaign management is built to do.

Mobile-First Indexing Is Complete

A lot of 2026 content still describes mobile-first indexing as something that is “rolling out.” It is not — it is finished. Google announced “mobile-first indexing is here” in October 2023 and removed the final Search Console crawler setting in mid-2024. The smartphone Googlebot is now the default indexing crawler, so Google indexes the mobile rendering of your pages. Table 9 traces the timeline and states the present-tense status.

The practical rule: content, structured data, and links must exist in your mobile layout. Anything that lives only in a desktop-only version — a block of text hidden on mobile, structured data injected only on the desktop template, links tucked into a desktop-only menu — is effectively invisible to Google, because the smartphone Googlebot is what it indexes. There is nothing left to “migrate to”; the work is confirming that your mobile page contains everything your desktop page does. The same corpus of crawled-and-indexed mobile pages is what feeds the newer AI surfaces covered in how AI is changing SEO.

How to Help a Page Pass All Three Gates

Because crawling, indexing, and ranking are separate gates, the way to get a page found is to walk them in order and confirm each one before moving on. The sequence below is the one MB Adv Agency uses to diagnose an organic-visibility problem, and it maps directly to the controls detailed above.

1. Make the URL discoverable. Link to the page from other pages on your site and include it in your XML sitemap, so Googlebot finds it through both links and the sitemap. 2. Confirm it is crawlable and returns 200. Check that the URL is not disallowed in robots.txt and that it returns a clean HTTP 200 — not a redirect chain, a soft 404, or a 5xx. 3. Confirm it is indexable. Ensure the page carries no stray noindex and has a clean, self-referencing canonical, so Google is allowed to store it. 4. Verify the mobile rendering. Because mobile-first indexing is the default, confirm the smartphone version contains all the content, structured data, and links present on desktop.

5. Inspect which gate it is at. Use Search Console’s URL Inspection tool to see whether the page was discovered, crawled, and indexed, so you fix the right gate instead of guessing. 6. On large sites, check the logs. Analyze server access logs to see the real crawl frequency of your revenue pages and where Googlebot wastes crawl on parameter or faceted URLs. This ordered process is why the free crawl-and-index dashboard is the center of the workflow — start with tracking performance in Google Search Console and its URL Inspection tool, and monitor discovery through Sitemaps and URL Inspection. Do the gates in order and “the page is not working” becomes a specific, fixable finding every time.

Same Corpus, New Surfaces: Mobile-First and AI Search

Two shifts define how the pipeline behaves in 2026, and neither one changes the fundamentals. The first is that mobile-first indexing is complete: the smartphone Googlebot is the default, so the version of your page that gets crawled, indexed, and judged is the mobile one. The present-tense advice is simply to make sure your mobile page contains everything your desktop page does — the same content, the same structured data, the same links. Page experience, including Core Web Vitals, feeds the same crawl-and-render pipeline; a fast, healthy site supports crawl capacity and is easier for Googlebot to work through.

The second shift is the rise of AI surfaces — AI Overviews and AI Mode — and here the key fact is that they draw from the same crawled-and-indexed corpus. An AI Overview can only cite a page that Google crawled and indexed; a page that failed the crawl or index gate is invisible to the AI answer for exactly the same reason it is invisible to the ten blue links. So the pipeline is not made obsolete by AI search — it is the prerequisite for it. Everything layered on top, and what genuinely changes for practitioners, is covered in how AI is changing SEO; the server-speed and page-experience signals that support both crawl capacity and user experience are tracked in Core Web Vitals and page experience.

The through-line MB Adv Agency keeps in front of every client is that the crawl → index → rank model is durable. Surfaces change — desktop to mobile, ten blue links to AI answers — but a page still has to be discovered, stored, and selected to appear anywhere. Get the three gates right and you are eligible across every surface Google builds on top of its index.

The Drift Corrections That Decide Whether Your Pages Get Found

SEO advice drifts month to month, and four corrections are load-bearing enough that getting them wrong quietly costs indexing and visibility. Each one is a documented Google position, not an opinion, and each maps to a specific gate in the pipeline.

Blocking a page in robots.txt does not remove it from Google. robots.txt controls crawling, not indexing; a disallowed-but-linked URL can still be indexed as a bare URL with no description. To keep a page out of results, use noindex and leave the page crawlable so Google can read the rule. Submitting an XML sitemap does not make Google index or rank your pages. A sitemap is a discovery aid — it helps Google find URLs faster; it does not guarantee indexing and has no direct effect on ranking, and listing noindex or non-canonical URLs in it actively hurts.

Not every site needs to optimize crawl budget. It is a large-site concern — Google’s thresholds are roughly 1,000,000+ pages changing weekly or 10,000+ changing daily. Time spent “optimizing crawl budget” on a small site is time not spent on content and links. rel=canonical does not force Google’s choice. It is a hint, one of several canonicalization signals alongside redirects, HTTPS, internal links, and sitemaps; Google can select a different canonical than the one you declare. Keeping these four straight is the difference between a page that clears all three gates and one that silently fails at crawl or index — and it is the same discipline that shows up when Google’s ranking systems and algorithm updates decide where an already-indexed page lands.

How Search Engines Work FAQ

Methodology & Sources

Every mechanic in this guide is documented by Google. The three-stage crawl → index → rank model, the distinction between robots.txt (crawling) and noindex (indexing), the 50,000-URL and 50 MB sitemap limits, the crawl-budget page-count thresholds, and the completion of mobile-first indexing all come from Google’s official Search Central and crawling-indexing documentation, the sitemaps protocol, and IETF RFC 9309. These are Google’s own statements, not estimates, so they are not subject to the caveats that apply to third-party benchmarks.

The only numeric datasets are search-demand figures. US and global search volumes and Keyword Difficulty for the crawl, index, and rank terms come from Ahrefs Keywords Explorer (US, 2026-07-01), and the observation that all six source URLs in this cluster recorded 0 clicks and 0 impressions comes from the pillar’s Search Console export over the 90-day window (2026-04-02 to 2026-07-01). No distribution of “crawls by status code” or “share of sites on mobile-first” is charted, because no credible public dataset for either exists — charting one would be fabrication.

MB Adv Agency has no SEO or technical-SEO service page and no SEO client benchmark dataset, so the point of view here stays qualitative: our team, and the paid Google Ads and PPC campaigns we run that complement your organic search. No client metric appears anywhere in this guide. Last updated 2026-07-01. To discuss the paid search that sits alongside your SEO, talk to our team.