Sitemap: Know Which Pages Appeared, Which Disappeared, and What Traffic Left With Them

Pages disappear from websites all the time. A product sells out and its page is unpublished. A listing expires. A redesign drops a section nobody thought was important. Weeks later, organic traffic is down and nobody can say exactly which pages went or when.

Your sitemap knows which pages exist right now. It does not know which ones existed last month. The Sitemap connection is now available in AI Data Stream, on Pro and Business plans, and it adds the part the sitemap is missing: memory.

What It Does

Point the connection at your sitemap.xml or sitemap index. From then on, AI Data Stream keeps a record of every URL the sitemap lists: when it first appeared, when it was last seen, and when it dropped out. The sitemap is a public file, so there are no credentials to set up and no CMS plugin to install. It works for static builds, Shopify, headless CMSs, hand-rolled frameworks, and every site WordPress-specific tooling cannot reach.

Your AI gets four tools:

• Stats: how many URLs are listed, how many have dropped out over time, additions and removals in the last 7 and 30 days, and the largest sections of the site
• Changes: the URLs added, removed or modified in a period
• Search: URLs by path, section or lastmod range, including the ones that are gone
• Refresh: fetch the latest copy on request

The questions that matter are the ones that combine them with your other data. “Which pages did we remove in September, and how many clicks were they getting before that?” is one question in chat. The AI pulls the removed URLs from the sitemap history, looks each one up in Search Console for the weeks before it disappeared, and adds up the traffic that left with them. The new Sitemap Content Changes skill runs that analysis as a report.

Honest About When

A sitemap only tells you what is listed at the moment you read it. If a page was there on the 9th and gone on the 24th, nobody can say which day in between it was removed. Most tools would quietly pick one. We do not.

Every removal is reported as a window: still listed on one fetch, gone by the next. When the two dates are a day apart, that is a day. When they are two weeks apart, you see two weeks, and the AI is instructed to quote the range rather than invent a date.

How wide those windows get is up to you. By default the sitemap is only fetched when someone uses a tool or presses Refresh now, which costs nothing when nobody is looking but leaves the gaps as long as the silences. Turn on Refresh in the background and the sitemap is checked once every refresh interval, as often as twice a day. The checks are conditional: if your server says the sitemap has not changed, nothing is downloaded, and for a sitemap index only the child files whose own lastmod moved are fetched. Removal windows shrink to a day without hammering your server or ours.

Date a Disappearance, or Triage Your 404s

Two uses fell out of the history almost for free.

When did this page go? Give the AI a URL and it tells you the last fetch that still listed it and the first that did not. For a page a client swears they never deleted, that is a useful thing to be able to say.

Which 404s matter? Hand over a list of URLs returning 404, from Search Console’s crawl data or from Cloudflare, and the sitemap history sorts them into three piles. Pages that were listed and then removed are real deletions that probably want a redirect. URLs that were never in the sitemap are old links, scanner noise and typos you can ignore. And pages that are still listed but returning 404 are the ones your CMS is actively telling Google about and failing to serve: a bug, not a content decision.

Checking Your lastmod

While parsing sitemaps we see a lot of lastmod fields, and many of them are not worth much. Some sitemaps omit the date entirely. Others stamp every URL with the same day, or with the time the site was last built. Google and Bing have both said they only use lastmod when it is consistently accurate and ignore it once it is not, so a sitemap full of fake dates has thrown away a crawl hint for the whole site.

The connection now flags both cases. If fewer than half your URLs carry a date, or nearly all of them carry the same one, the dashboard and the AI say so and explain the fix: have the CMS emit each page’s real last-modified date. It is a note, not an error. Sitemaps without lastmod are valid and the connection works fine with them. You should just know.

Built Not to Lie to You

A connection whose whole value is historical data has one catastrophic failure mode: recording removals that never happened. A WAF challenge page, a half-written sitemap during a deploy or a timeout on a large index can all make a fetch return a fraction of the URLs that were there yesterday. Treat that as truth and thousands of live pages get marked removed, and the false history can never be unwritten.

So the connection refuses. If a fetch returns fewer than half the URLs stored from the last one, nothing is marked removed, the connection shows a warning, and you are asked to check the sitemap in a browser. Partial reads behave the same way: if a sitemap is larger than your plan stores (10,000 URLs on Pro, 50,000 on Business) or than the size limit allows, the copy is flagged as partial and URLs are never marked removed from it.

Getting Started

Open a property’s Connections tab, click Add Connection and choose Sitemap. Paste the sitemap URL, keep the default 24-hour interval, decide whether you want background refreshes, and click Test. Then ask the AI what changed this month, or turn on the dashboard widget and watch the additions and removals accumulate.

The setup guide is in the documentation. Tracking starts from the first fetch, so the sooner a site is connected, the more history it has when you need it.