What Is Crawling in SEO? How Googlebot Finds Your Pages

Crawling in SEO is the process where Googlebot, Google's automated crawler, discovers the URLs on your website and fetches their content so Google can decide whether to index and rank them. If you have been searching for what is crawling in SEO, the one-line answer is that crawling is Google visiting your pages. It happens before indexing and before ranking, and if it fails, the other two never start. A bakery in Kothrud can have the best cake-delivery page in Pune, but if Googlebot has never fetched that page, it does not exist as far as Google Search is concerned.

Checking crawl health is the first thing I do on any SEO project, before keywords, before content, before links. It takes about ten minutes in Google Search Console and it rules out the silent problems that keep otherwise good websites invisible. This guide explains how Googlebot discovers and fetches pages, what crawl budget really means for a small site, how to read your crawl activity in Search Console step by step, and the blockers I run into most often on Indian business websites.

What is crawling in SEO?

Crawling is the discovery and fetching stage of search. Google keeps an enormous list of URLs it knows about, and Googlebot works through that list, requesting each URL the way your browser would and saving what comes back: the HTML, the images, the CSS and the JavaScript files the page needs. New URLs get added to the list as Googlebot finds links on the pages it fetches, so a well-linked site is effectively handing Google a map of itself.

The fetched copy is what everything else is built on. Google's indexing systems read it to work out what the page is about, whether it duplicates another page, and whether it deserves a place in the index. Ranking systems then decide where an indexed page appears for a given search. If the fetch never happens, or Googlebot gets an error, a blank page or a redirect loop, there is nothing to index and nothing to rank. That is why crawling problems are so damaging: they do not lower your rankings, they remove you from the running entirely.

Crawling is also not a one-time event. Googlebot returns to pages it already knows to check for changes, more often for pages that change frequently or receive many links, less often for pages that sit still. A homepage might be refetched daily. A five-year-old blog post with no links pointing at it might be refetched every few months.

What is a crawler in SEO?

A crawler, also called a spider or a bot, is the software that does the fetching. Googlebot is Google's crawler, and it comes in two main flavours: Googlebot Smartphone, which fetches pages as a mobile phone would and has been the primary crawler for almost every site since mobile-first indexing, and Googlebot Desktop. Google also runs specialised crawlers for images, video, news and ads. Bing has Bingbot. SEO tools run their own: AhrefsBot, SemrushBot and others that build the link databases those tools sell. Since 2023 a new group has arrived, AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, which fetch pages to feed AI answer engines rather than a traditional search index.

Each crawler identifies itself with a user-agent string, and each one is supposed to read your robots.txt file first and obey it. When someone asks what is a crawler in SEO, the practical answer is this: it is the visitor you cannot see in your analytics, it behaves like a very fast, very literal browser, and it decides what search engines and AI tools know about your business. My guide on what robots.txt is in SEO covers how to give each crawler instructions.

There is a second meaning of "crawler" you will hear from SEO people: desktop tools like Screaming Frog or Sitebulb that crawl your site the way Googlebot would and report what they find. Those are simulations. They are useful for audits, and Screaming Frog is free for up to 500 URLs, which covers most small business sites in India. But they are not Googlebot, and a clean Screaming Frog crawl does not prove Google has fetched your pages. Only Search Console shows that.

Website crawl meaning: crawling versus indexing versus ranking

People search for "website crawl meaning" because the three stages of search get blurred together in everyday conversation. They are separate, and Search Console reports on them separately:

• Crawling is fetching the page. Search Console reports crawl activity in the Crawl stats report and in the Crawl section of URL Inspection.
• Indexing is processing and storing the page. The Page indexing report tells you whether each URL is indexed and, if not, why. Two statuses matter here: Discovered - currently not indexed means Google knows the URL but has not fetched it yet, and Crawled - currently not indexed means Google fetched it and chose not to index it.
• Ranking is ordering indexed pages for a query. The Performance report shows the results: impressions, clicks and average position.

Almost every "my website is not showing on Google" problem sits in one of the first two stages. If you are in that situation, read why your website is not showing on Google alongside this guide, because the diagnosis order is always the same: was it crawled, was it indexed, and only then, how does it rank.

How Googlebot discovers and fetches your pages

Googlebot needs to learn that a URL exists before it can fetch it. These are the main discovery sources, in rough order of how much I rely on them for client sites.

1. Links from pages Google already crawls. This is the original mechanism and still the most powerful. When Googlebot fetches your homepage and finds a link to your new "root canal cost in Pune" page, that page enters the queue. Internal links are the single biggest discovery lever you control, which is why an internal linking strategy matters more for small sites than most owners realise.
2. XML sitemaps. A sitemap is a list of URLs you hand to Google directly, submitted through Search Console. It does not force a crawl, but it tells Google the URL exists and when it last changed. Every site should have one, and it should contain only the canonical versions of pages you want indexed.
3. Manual submission. The URL Inspection tool in Search Console has a Request indexing button. It puts one URL into a priority queue and is the fastest way to get a new or updated page fetched, often within a day. It has a small daily quota, so it is for important pages, not bulk work.
4. Links from other websites. A mention on a Pune business directory, a supplier's site or a news article introduces your URL to Google through someone else's crawl.
5. Redirects and previously known URLs. Googlebot follows 301 and 302 redirects to their destinations, and it periodically refetches URLs it has seen before, even old or deleted ones, to confirm their status.

Once a URL is in the queue, the fetch itself follows a fixed sequence. Googlebot checks your robots.txt file to confirm the URL is allowed. It then sends an HTTP request and records the response code: 200 means success, 301 or 302 means a redirect it will follow, 404 means the page is gone, and 5xx means your server failed, which makes Googlebot back off and try later. It downloads the HTML and, if the page loads content or links with JavaScript, it queues the page for rendering, where a headless version of Chromium runs the scripts and captures the result. That rendering pass is separate and can lag behind the initial fetch, which is the root of most JavaScript-related SEO trouble.

Googlebot also decides how hard to push. If your server slows down or starts returning errors, Googlebot reduces its request rate to avoid making things worse. If the site responds quickly and consistently, it crawls more freely. Your hosting quality, in other words, directly affects how much of your site Google sees.

Crawl budget for small websites: what actually matters

Crawl budget is the number of URLs Googlebot can and wants to fetch from your site in a given period. Google describes it as two things combined: a crawl capacity limit, based on how fast your server can respond without strain, and crawl demand, based on how popular your pages are and how often they change. Big publishers and e-commerce sites with hundreds of thousands of URLs manage this carefully because Googlebot cannot get to everything.

For the sites I work on in Pune, crawl budget is almost never the problem. Google has said plainly that sites with fewer than a few thousand URLs are generally crawled efficiently. A dental clinic with 25 pages, a restaurant with 12, a coaching institute with 60: Googlebot can fetch all of that in a single afternoon. When those pages are not being crawled, something is blocking the fetch or telling Google the pages are not worth it, and no amount of budget optimisation fixes that.

Where small sites genuinely do waste crawl is through URL sprawl they never intended to create. The usual sources:

• Filter and sort parameters on a shop, where ?colour=red&sort=price turns 40 products into 4,000 URLs. I audited a Wakad D2C store whose Crawl stats showed most of Googlebot's requests going to parameter URLs no customer ever searched for.
• Duplicate versions of the same page: http and https, www and non-www, with and without a trailing slash, with and without .html. Each pair doubles the crawl for zero gain. The fix is a redirect plus a canonical tag, which I explain in what canonical URLs are.
• Internal search result pages and calendar pages that generate a new URL for every query or every month, forever.
• Tag and category archives on WordPress with one post each, which are thin duplicates of the post itself. Category pages that do earn their place still need care, which is why I wrote about schema markup for category pages separately.
• Slow servers. A page that takes four seconds to respond costs Googlebot four seconds, and Googlebot compensates by fetching less. Speed work you do for Core Web Vitals also helps crawling.

The honest priority order for a small site is: remove blockers first, kill unintended duplicates second, and treat crawl budget as a phrase you can mostly ignore until your site passes a few thousand URLs.

How to check crawl activity in Google Search Console

Search Console is the only place that shows you what Googlebot actually did on your site, as opposed to what a simulator predicts. Here is the sequence I follow on every new project. It takes ten to fifteen minutes.

Step 1: Verify your site as a Domain property

Go to search.google.com/search-console and add your site. Choose the Domain property option and verify through a DNS record at your registrar (GoDaddy, Hostinger, BigRock and the others all have a DNS panel). A Domain property covers every protocol and subdomain at once, so you see http, https, www and non-www together instead of in four separate properties. If DNS verification is beyond you, a URL-prefix property with the HTML tag method works, but make sure it matches the exact version your canonical tags use.

Step 2: Read the Page indexing report first

Under Indexing, open Pages. The top shows how many URLs are indexed and how many are not. Scroll to the "Why pages aren't indexed" table. Each row is a reason, and the reasons tell you which stage failed. "Blocked by robots.txt", "Excluded by noindex tag", "Server error (5xx)" and "Discovered - currently not indexed" are crawl-stage problems. "Crawled - currently not indexed", "Duplicate without user-selected canonical" and "Soft 404" are post-crawl problems. Click any reason to see the affected URLs. When I checked the Pune dental clinic I later moved from position #59 to the top 5, this report showed every service page indexed cleanly, so I could rule crawling out in two minutes and focus on their Google Business Profile, which was the real issue.

Step 3: Open the Crawl stats report under Settings

Click Settings at the bottom of the left menu, then Open report next to Crawl stats. This report is easy to miss because of where it lives, and it is the direct record of Googlebot's visits over the last 90 days. The three charts at the top show total crawl requests, total download size and average response time. A small business site typically sees a few dozen to a few hundred requests a day. Check the Host status section as well: it flags whether Googlebot had trouble fetching robots.txt, resolving your DNS, or connecting to your server. Any red marks there explain a crawl drop instantly.

Step 4: Break the crawl down by response, purpose and Googlebot type

Below the charts, the report groups requests four ways. By response shows how many fetches returned 200, 301, 404, 5xx and so on; a healthy site is overwhelmingly 200s. By file type separates HTML from images, scripts and CSS. By purpose splits Discovery (new URLs) from Refresh (revisits), and a site that is only ever refreshed has stopped surfacing new pages to Google. By Googlebot type shows whether Smartphone or Desktop is doing the work, and it should be Smartphone. If half your requests are 404s or redirects, you have found the crawl waste from the previous section.

Step 5: Inspect a single URL

Paste a specific page into the search bar at the top of Search Console to open URL Inspection. The result tells you whether the URL is on Google, how it was discovered (sitemap or referring page), the last crawl date and time, which Googlebot crawled it, whether the crawl was allowed by robots.txt, whether the page fetch succeeded, and whether indexing is allowed. It also shows the canonical you declared next to the one Google selected. This single screen answers most "why is this page not showing" questions.

Step 6: Test the live URL and view the rendered page

Click Test live URL and Googlebot fetches the page right now. When it finishes, click View tested page. The Screenshot tab shows what Googlebot rendered, the HTML tab shows the code it processed after JavaScript ran, and the More info tab lists page resources it could not load and any JavaScript errors. If your headline, your product list or your service menu is missing from the screenshot, Googlebot is not seeing what your visitors see, and that is the problem to fix.

Step 7: Fix the blocker, request indexing and re-check in a week

Once you have fixed whatever the reports revealed, use Request indexing on the affected page, or resubmit the sitemap if the change touches many pages. Then leave it. Come back in about a week and re-run steps 2 and 5. Crawl stats update with a delay of a day or two, and index changes can take longer, so checking every hour tells you nothing. Note the date and what you changed, because being able to say "the crawl rate doubled after the hosting move on the 3rd" is what turns SEO from guesswork into a process.

How to use Googlebot to your advantage

A lot of people search for how to use Googlebot as if it were a tool you download. It is not. Googlebot is Google's private crawler and nobody outside Google can run it. Anyone selling you "Googlebot software" is selling a site crawler with a borrowed name. What you can do is make Googlebot's job easy, watch what it does, and see your site the way it sees it. That is what using Googlebot means in practice.

• Give it a sitemap and clean internal links. Submit your XML sitemap in Search Console and make sure every page you care about is reachable within three clicks of the homepage through normal <a href> links. Buttons that load content with JavaScript, dropdowns without real links and "load more" widgets are invisible to discovery.
• Use URL Inspection as your window into Googlebot. The live test in step 6 is the closest thing to running Googlebot yourself. Use it on every new template, every redesign and every page that behaves oddly.
• Pretend to be Googlebot in Chrome. Open DevTools, find Network conditions, untick "Use browser default" under User agent and choose Googlebot Smartphone. Reload the page. If a plugin or a firewall serves something different to Googlebot, you will see it. This catches cloaking by accident, which happens more than you would think with aggressive security plugins.
• Read your server logs. Hosting panels like cPanel offer raw access logs. Filter for the Googlebot user-agent and you get a timestamped list of every URL it requested, which is more granular than Search Console. Verify suspicious entries with a reverse DNS lookup, because spam bots impersonate Googlebot constantly.
• Check the robots.txt report. Search Console shows the robots.txt versions Google has fetched and any errors. A broken robots.txt is the fastest way to lose an entire site from Google, so this report deserves a look whenever anything site-wide changes.
• Run a simulated crawl. Screaming Frog (free up to 500 URLs) for the whole site, or my free website SEO checker for a single page, will surface broken links, redirect chains, missing titles and noindex tags before Googlebot trips over them.

Common crawl blockers and how to fix each

These are the problems I actually find, in the order I find them most often.

robots.txt blocking pages or resources

A Disallow: / left over from development blocks the entire site. More subtle is blocking folders like /wp-content/ or /assets/, which stops Googlebot loading the CSS and JavaScript it needs to render the page properly. Google will not index what it cannot render correctly. Open yoursite.com/robots.txt in a browser, read every Disallow line, and remove anything that blocks pages or resources you want in search. Blocking admin areas, cart pages and internal search is fine.

noindex tags and X-Robots-Tag headers

A <meta name="robots" content="noindex"> tag tells Google not to index the page, and over time Googlebot crawls noindexed pages less and less. On WordPress this is usually the "Discourage search engines from indexing this site" checkbox under Settings, Reading, which developers tick on staging and forget to untick at launch. I have found it ticked on live business sites more than once. The same instruction can be sent as an HTTP header, which never shows in the page source, so check the "Indexing allowed?" line in URL Inspection rather than trusting your eyes. My guide to meta tags for SEO covers the robots tag alongside the others that matter. One combination to avoid: a page blocked in robots.txt that also carries noindex. Google cannot fetch the page to see the noindex, so the block wins and the URL can linger in the index with no description.

JavaScript rendering problems

Sites built with React, Vue, Angular or a page builder that loads content after the page opens can look empty to Googlebot on the first pass. Rendering happens later and sometimes fails. A coaching institute in Hinjewadi came to me with course pages that showed nothing but a header in the URL Inspection screenshot: the syllabus and fees were loaded by a script that errored under Googlebot. The fixes, in order of preference: render the content on the server so it is in the HTML, use real links for navigation, and make sure scripts do not depend on resources robots.txt blocks. Always verify with step 6 above rather than assuming.

Slow servers, errors and redirect chains

Cheap shared hosting at 3,000 rupees a year is fine for a five-page site until it is not. Response times above a couple of seconds, intermittent 5xx errors and timeouts all cause Googlebot to reduce its crawl rate, and Host status in the Crawl stats report will show it. Redirect chains, where /old goes to /new goes to /newer, waste fetches and can be dropped after a few hops. Point every redirect straight at its final destination. If speed is a broader problem, the work in my Core Web Vitals guide tends to improve crawl rate as a side effect.

Orphan pages and nofollow internal links

An orphan page has no internal links pointing to it. It might be in your sitemap, and Google might discover it that way, but a page nobody on your own site links to signals low importance, and it often sits in "Discovered - currently not indexed" for months. Link every page from at least one relevant, crawled page. And check that your internal links are not tagged rel="nofollow", which some plugins add by default; a nofollow internal link tells Google you do not vouch for your own page. I explain the tag in what a nofollow link is.

Duplicate URLs and unintended URL sprawl

Every duplicate you saw in the crawl budget section is also a blocker in disguise, because Googlebot spends its visits on copies instead of on the pages that matter. Redirect the hard duplicates, canonicalise the soft ones, and keep parameter URLs out of the sitemap. If the Page indexing report is full of "Duplicate" reasons, start there.

Images and files nobody linked

Images get crawled too, through their src attribute and through image sitemaps. Lazy-loading that hides the real image path in a data-src attribute, or galleries built entirely in JavaScript, can leave Google Images with nothing. If product or clinic photos matter to your traffic, follow the checklist in my image SEO optimisation guide so the files are discoverable and described.

What crawling problems look like on Indian small business sites

Four patterns come up again and again in Pune and across India, and each one is fixable in an afternoon.

The launched-but-hidden WordPress site. A Kothrud bakery paid a freelancer 25,000 rupees for a WordPress site, launched it, and got zero Google traffic for four months. The "Discourage search engines" box was still ticked from the staging build. Unticking it, submitting the sitemap and requesting indexing on the homepage got the site crawled the same day and indexed within the week. Nothing else changed.

The JavaScript storefront. A real estate developer's project site built on a JavaScript framework listed twelve projects, all loaded by an API call after the page opened. Googlebot's screenshot showed a spinner. The developer added server-side rendering for the project pages, and they went from "Crawled - currently not indexed" to indexed within three weeks, without touching the content.

The filter explosion. A D2C clothing store in Wakad had 60 products and 3,800 crawled URLs, because every colour, size and sort combination produced a new address. Blocking the parameter patterns in robots.txt, adding canonical tags to the base category pages and rebuilding the sitemap cut crawl requests by two thirds and moved the product pages to daily refreshes.

The site with no problem. This is the most common one, and it is worth saying. The Pune dental clinic that went from #59 to the top 5 in two months had a perfectly crawlable site. The gap was local, not technical, and the work was on their Google Business Profile, categories and reviews, which I cover in what local SEO is. Checking crawl health first meant I did not spend the client's money fixing something that was not broken. That discipline applies whichever direction the diagnosis goes.

If you are choosing a platform for a new site, crawlability should be part of the decision. My comparison of WordPress, Wix and Shopify covers how each one handles sitemaps, robots.txt and rendering out of the box. And for the broader technical foundation every owner should understand, start with technical SEO basics for business owners.

Frequently asked questions

How often does Googlebot crawl a website?

There is no fixed schedule. Googlebot decides crawl frequency per URL based on how often the page changes, how many links point to it, and how quickly your server responds. A homepage that updates weekly might be fetched every day or two, while a static contact page on a small Pune business site might be checked once a month. You can see the actual pattern for your site in the Crawl stats report under Settings in Google Search Console.

What is the difference between crawling and indexing?

Crawling is Googlebot fetching a page. Indexing is Google processing what it fetched, understanding the content, and storing it in its search index. A page can be crawled and still not indexed, which Search Console reports as 'Crawled - currently not indexed'. Ranking is the third step, where indexed pages are ordered for a specific query. Each step depends on the one before it.

Why is my page 'Discovered - currently not indexed' in Search Console?

'Discovered - currently not indexed' means Google knows the URL exists, usually from a sitemap or a link, but has not fetched it yet. On small sites this is normally a quality or priority decision rather than a crawl budget problem: Google has decided the page is not worth fetching right now. Strengthen the page with unique content, link to it from your homepage or a strong related page, and request indexing through URL Inspection once.

Can I force Googlebot to crawl my site?

Not force, but you can invite it strongly. Submit an XML sitemap in Search Console, use the Request indexing button in URL Inspection for individual pages, link to new pages from pages Google already crawls often, and make sure nothing in robots.txt or a noindex tag is blocking the fetch. Googlebot usually responds to a request indexing submission within hours to a few days, but Google does not guarantee it.

Is crawl budget a problem for a small business website?

Almost never. Google has said that sites with fewer than a few thousand URLs are generally crawled efficiently, and most clinic, restaurant, salon and service websites in India have under 100 pages. If your pages are not being crawled, the cause is usually a blocker such as robots.txt, a noindex tag, JavaScript-only content, a slow server, or pages with no internal links, not a shortage of budget. Fix those before worrying about budget.

Related guides

• What are canonical URLs? The canonical tag explained
• Schema markup for category pages: what belongs there
• What is robots.txt in SEO?
• Why is my website not showing on Google?
• Technical SEO basics every business owner should know
• Internal linking strategy guide
• Free website SEO audit in 5 minutes
• How long does SEO take?

Your next step

Open Search Console, follow the seven steps above, and write down what the Page indexing report says today. Then run your homepage through the free website SEO checker and work through the on-page SEO checklist for 2026 to fix what both of them flag. If the reports show a blocker you cannot untangle, send me the URL and I will tell you what I see.