Most SEO advice is a list of things to do. Very little of it explains why those things are on the list, which is why so much of it gets applied badly.
Understanding how search engines work takes about ten minutes and makes the rest of the subject considerably easier to judge. Three stages, each with its own failure modes, and almost every SEO problem is a failure in one of them.
How Search Engines Work: Crawling, Indexing and Ranking
1. discovery and crawling
Search engines find pages by following links and by reading files you provide.
A crawler arrives at a page, reads it, and follows the links it contains to find more. Your XML sitemap offers a direct list, which helps for pages that are not well linked internally.
- Where it fails. Pages with no internal links pointing at them, so nothing leads a crawler there. Rules in robots.txt blocking access, often copied from another site without being read. Server errors during a crawl. Sites so slow that crawlers fetch less per visit.
- The check. Search Console reports what has been discovered and what has not. Pages listed as discovered but not indexed are the interesting ones.
2. Indexing
Being crawled is not being indexed. This trips up more people than any other part.
After crawling, the engine processes the page, works out what it is about, and decides whether to store it. Not everything crawled gets indexed, and that is a judgement rather than a technicality.
Where it fails. Thin pages with little unique content. Near-duplicates competing with one another. Canonical tags pointing somewhere unintended. Noindex directives left over from a staging site, which is a genuinely common and genuinely painful mistake.
Content requiring JavaScript to appear adds a dependency here, since rendering happens after initial crawling and is not guaranteed to complete as expected.
The check. Search for a distinctive phrase from the page in quotation marks. If it does not return, the page is probably not indexed.
3. Ranking
When somebody searches, the engine selects from its index and orders the results.
The signals are numerous, weighted differently by query type and not fully published. Broadly they cover relevance to the query, quality and depth of content, authority signals including links, and how usable the page is.
- Where it fails. Content answering a keyword rather than a question. Pages competing with each other for the same term. Thin coverage of a subject others treat thoroughly. Poor mobile experience.
- The check. Search your target query and read what currently ranks. If those results are substantially more useful than your page, that is the answer, and no technical fix substitutes for it.
The mobile-first part
Indexing is based primarily on the mobile version of your pages.
If your mobile site shows less content than your desktop site, the reduced version is what gets assessed. Hiding content behind mobile accordions is generally fine; removing it entirely is not.
Test your pages on a real mid-range phone rather than a narrowed desktop window, which still has your office connection behind it.
What has changed, and what has not
AI-generated answers now sit above results for many queries, which changes what visibility produces rather than how pages are found. The three stages still apply, because those systems draw on content discovered and indexed the same way.
Speed still matters throughout. Where performance and environmental impact both apply, treat them as the same lever: reducing unnecessary data transfer and processing supports a faster experience while lowering estimated digital carbon, though environmental figures remain modelled rather than directly measured.
Google’s own documentation on how search works is the authoritative account and is written for a general audience.
Nobody can guarantee a position, and understanding the three stages makes it clearer why. You influence discovery and indexing quite directly. Ranking is a comparison against everyone else, and you do not control the others.
Our search engine optimisation work starts at stage one, because content improvements are wasted on pages that cannot be found.