Writing · · 4 min read
How Google finds, reads and files a website
Before a page can rank, it has to be found, read and stored. Those are three separate jobs, and a problem in any one of them can keep a good page out of search entirely.
1. Finding: crawling
Google's crawler, Googlebot, discovers pages mostly by following links: from pages it already knows, from other sites, and from the XML sitemap you give it. A page that nothing links to is hard to find. Before it fetches anything, it checks your robots.txt file, which can tell it to stay out of whole sections of the site.
2. Reading: rendering
Once a page is fetched, Google renders it much like a browser does, including running its JavaScript. Content that only appears after a click, or that depends on scripts that fail, may never be seen. Plain HTML with the important words in it is still the safest bet.
3. Filing: indexing
Then Google decides whether the page is worth keeping in its index, and which version to keep when several URLs show the same thing. A stray noindex tag, a wrong canonical, or thin, duplicated content can each stop a page at this step.
What to check on your own site
- Every important page is linked from somewhere a visitor can click.
- Your sitemap lists the pages you want found, and only those.
- robots.txt isn't blocking anything you care about.
- The main text of each page is in the HTML, not hidden behind scripts.
- Search Console's page indexing report: it shows which pages were left out, and why.
Ranking only starts after all of this. That's why the first step on any site, this one included, is making sure it can be found, read and filed properly. You can follow how that goes for this site in the Work case study.
Want me to check how your site does on all three? Get in touch.