The XML Sitemap Generator is a free GrowthGPT tool that crawls a live site from its homepage, follows internal links up to 500 pages, and builds a valid XML sitemap from what it finds, with a last-modified date where the server reports one, a priority derived from click depth, and a change frequency you choose.
Most teams use a sitemap tool to produce a file. The more valuable output is the crawl itself, because it shows you the site the way a bot sees it rather than the way your navigation implies. This guide covers running the XML Sitemap Generator, reading the depth and error data it returns, and what to do with the file once you have it. For the wider indexing picture, pair it with how to get pages indexed faster.
Image: A crawl spreading outward from a homepage node through link edges, with each ring of pages labelled by click depth and a generated XML file forming alongside
What does the XML Sitemap Generator do?
You give it a URL. It fetches that page, extracts the internal links, fetches those, and keeps going breadth-first until it reaches your page cap. For each page it records the URL, the last-modified date if the server sends one, and the click depth from your starting point. Then it assembles a standards-compliant sitemap you can copy or download.
Three controls sit behind the More options toggle:
- Max pages. 50, 100, 200, 300, or 500. Start at 50 for a fast structural read, then run the full crawl once you know the starting URL is right.
- Change frequency. Applied uniformly to every URL in the file, from always through to never.
- Priority mode. Depth scales priority from 1.0 at the homepage down toward 0.1 at your deepest pages. Flat gives every URL the same value.
The results table is sortable by URL, last modified, depth, and priority, with a filter box for narrowing to a section, plus a list of the errors the crawl hit along the way.
How do I run a useful crawl?
Step 1: Start from the canonical homepage
The crawl stays on the origin you give it, so https://www.example.com and https://example.com produce different results, and starting from the wrong one gives you a sitemap full of URLs that redirect. Confirm which host and protocol your site actually canonicalises to first, with the Redirect Checker or the Canonical Checker.
Step 2: Sort by depth before you look at anything else
This is the finding that pays for the run. Sort the table by depth descending and look at what is sitting at depth four or five. If a product page, a pricing page, or your best commercial guide is four clicks from the homepage, crawlers reach it late and treat it as peripheral, and AI retrieval systems inherit the same signal. The fix is internal links, not a sitemap entry: map the shortcuts with the Internal Linking Graph Optimizer.
Step 3: Read the missing pages, not just the listed ones
Compare the crawled list against what you know exists. Anything absent is unreachable by following links, which usually means it is orphaned, only linked from JavaScript navigation, or reachable exclusively through a filter or search form. Orphan pages are effectively invisible to discovery regardless of how good they are.
Step 4: Work the error list
The crawl reports the status codes and failures it encountered. Internal links pointing at 404s and 500s waste crawl budget and leak authority, and they are almost always trivial to fix once someone can see the list. Check the response headers of anything suspicious with the HTTP Header Checker.
Step 5: Set changefreq honestly, then stop worrying about it
Weekly is a reasonable blanket value for a site that publishes regularly. Setting everything to hourly does not make crawlers visit more often, and Google’s Search Essentials make clear that discovery follows links and quality signals rather than declarations in a file. The last-modified dates are the field worth caring about, and they come from your server rather than your preferences.
Step 6: Download, validate, and reference it in robots.txt
Download the XML, check it with the Sitemap Validator, and add the sitemap line to robots.txt using the Robots.txt Generator. That directive is read by every crawler that respects the protocol, which makes it the cheapest discovery improvement available.
Should the generated file be my production sitemap?
Usually not. A static file starts drifting the moment you publish anything, and a stale sitemap is worse than none because it advertises URLs that no longer exist. The durable setup is a dynamic sitemap built by your framework at deploy time, which is a few lines in app/sitemap.ts on Next.js and equivalent elsewhere.
| Approach | Strength | Weakness | Best for |
|---|---|---|---|
| Crawl-generated file | Shows what a bot can actually reach | Goes stale immediately | Audits, migrations, competitor structure research |
| Framework-generated sitemap | Always current, versioned with code | Lists what you believe exists, not what is reachable | Production |
| CMS plugin sitemap | Zero maintenance | Often includes tag and archive noise | Content sites with disciplined taxonomy |
The strongest use of the generated file is as a diff. Two lists, one from the crawl and one from your live sitemap, and every difference is a finding: URLs the crawler reached that your sitemap omits, and sitemap URLs that no internal link points to.
Do sitemaps matter for AI search?
Indirectly, and the mechanism is worth being precise about. AI assistants do not read your sitemap to decide what to cite. They retrieve from indexes and live fetches, so the sitemap matters only insofar as it helps a page get discovered and indexed in the first place. Retrieval agents like the ones OpenAI documents in its bots documentation follow links and fetch URLs the same way, so reachability is the shared prerequisite.
Once pages are reachable, the work shifts to whether they are worth citing. Google’s guidance on AI features is blunt that there is no special markup for AI surfaces, only clear extractable content, and the Princeton GEO study measured up to 40 percent higher visibility in generative answers for content carrying quotations, statistics, and citations. Run a GEO Audit on the pages the crawl says are reachable, and score the outcome with the AI Visibility Score.
Frequently Asked Questions
What does the XML Sitemap Generator do?
The XML Sitemap Generator is a free GrowthGPT tool that crawls a live site from its homepage, follows internal links up to 500 pages, and builds a valid XML sitemap from what it finds. Each URL gets a last-modified date where the server reports one, plus a priority derived from click depth and a change frequency you choose. You can filter and sort the crawled URLs before copying or downloading the file.
How does the crawl decide which pages to include?
It starts at the URL you enter and walks internal links breadth-first, staying on the same origin, until it hits your page cap of 50 to 500. Breadth-first means shallow pages are found before deep ones, so if the crawl stops at the cap you still get the most important part of the site. Pages reachable only through forms, filters, or JavaScript navigation will not appear.
What is the difference between depth priority and flat priority?
Depth priority scales each URL by how many clicks it sits from the homepage, so the homepage gets 1.0 and the deepest pages approach 0.1. Flat priority gives every URL the same value. Depth mode is the better default because it makes your click-depth problem visible, though search engines treat the priority field as a weak hint at best.
Do priority and changefreq actually affect rankings?
No. Google has said for years that it largely ignores both fields, and no crawler treats them as ranking signals. Their practical value is diagnostic: a crawl that assigns most of your money pages a low priority is telling you those pages sit too deep in the site, which is a real problem worth fixing.
Should I use a generated sitemap or a dynamic one?
For an audit, generate. For production, ship a dynamic sitemap your framework builds at deploy time, such as app/sitemap.ts in Next.js, so it can never fall out of sync with your content. Use the generated file to compare against the dynamic one: URLs the crawl found that your sitemap omits, and sitemap URLs the crawl could not reach, are both bugs.
Does a sitemap guarantee my pages get indexed?
No. A sitemap helps discovery, not qualification. Search engines still decide whether a page is worth indexing based on quality, duplication, and crawl budget. A sitemap is most useful for large sites, new sites with few links, and pages that are poorly linked internally, which is why fixing internal links usually beats resubmitting the file.