AI crawlers in your server logs: GPTBot, ClaudeBot & co.
How to find GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot in your logs, verify their IPs, and read training versus live-fetch patterns.
How Microsoft Copilot grounds answers in Bing, how it differs from ChatGPT search, and how Bing Webmaster Tools, IndexNow and AI Performance get you cited.
Microsoft Copilot answers web questions by turning the user's prompt into a short Bing search, reading what Bing returns, and writing an answer with linked citations to the pages it used. So getting into Copilot's answers comes down to two things. Bing has to have your pages indexed, and your page (or a third-party page that names you) has to be the clearest source among the results. Copilot also differs from ChatGPT in one useful way: Microsoft gives publishers a first-party report of when Copilot cites them. This guide covers how the grounding works according to Microsoft's own documentation, how Copilot differs from ChatGPT search, and the six things to do.
"Copilot" covers several Microsoft products, and the differences matter. There's the consumer Copilot app and website. There's Copilot Search in Bing, launched in April 2025, which blends generated answers with classic results. And there's the work product: Microsoft 365 Copilot, now renamed Microsoft Copilot, which can reach the web when an admin and the user allow it. Microsoft documents Bing as the web source for the work Copilot, Copilot Search lives inside Bing, and Microsoft's publisher-facing AI Performance report covers "Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations". Copilot reaches people inside Windows, Edge, Bing and Office without them opening a separate AI app, which is why it's worth its own plan (our field guide to the AI engines puts it next to the others).
Microsoft has documented the mechanics for its work Copilot in unusual detail. According to Microsoft Learn, Copilot parses the prompt, picks out the terms where web information would help, and generates a search query that it sends to Bing. That query "consists of a few words informed by the user's prompt", not the whole prompt (unless the prompt is very short). In the same document's example, a prompt asking for a company's financial information and business strategy produced the two searches "Fabrikam strategy" and "Fabrikam financials". Copilot then writes its answer from what Bing returned, and in Copilot Chat it shows the exact queries it ran in the citation section.
Two things follow for publishers. First, you aren't competing on the user's long, conversational prompt. You're competing on the short searches Copilot derives from it, which look a lot like classic keyword queries. Second, the candidate pool is Bing's results for those searches. A page Bing hasn't indexed, or ranks nowhere for those terms, can't be a grounding result.
On the answer side, Microsoft says Copilot Search "cites its sources prominently". Whole sentences or passages in the answer are linked to their sources, and the cited sources are also listed at the bottom of the result. Being cited usually means a visible, clickable link, which is not the case for every engine. How the others display sources is covered in brand mentions vs citations.
Microsoft doesn't publish a ranking formula for grounding, and nobody outside Microsoft can give you one. Be wary of any "Copilot ranking factors" list that claims otherwise. Microsoft does say three useful things in public.
Brand mentions follow from that. Copilot writes its shortlist from the pages it retrieved, so the brands it names are mostly the brands those pages name. If the "best [category]" pages that rank on Bing list your competitors and not you, Copilot will tend to do the same, however good your own site is. That's why Copilot work is half Bing SEO and half third-party presence, the same pattern we see in what AI engines cite.
The two get lumped together, but what a publisher can see and control is quite different.
| Question | Microsoft Copilot | ChatGPT search |
|---|---|---|
| Where web results come from | Bing search, documented by Microsoft | OpenAI says it uses third-party search providers and partner content; it doesn't name them |
| Crawler to allow | Bingbot (Bing's index) | OAI-SearchBot |
| Fast update signal | IndexNow and sitemap lastmod, both recommended by Bing | No publisher submission channel documented |
| Citation reporting for publishers | AI Performance in Bing Webmaster Tools | No publisher report; referral traffic in your analytics only |
| Opt-out controls | NOCACHE / NOARCHIVE meta tags, plus robots.txt for Bingbot | robots.txt rules for OAI-SearchBot |
Two caveats. ChatGPT can also answer from memory without searching, so its web sources only matter on the answers where it does search. And because OpenAI doesn't name its search providers, you shouldn't assume a Bing win carries over to ChatGPT. Measure each engine separately.
Everything else depends on this. If you already use Google Search Console, Bing lets you import verified sites from Search Console, up to 100 at a time, with no separate verification step. Check URL Inspection for your key pages (homepage, pricing, top comparison pages) to confirm Bing has indexed them.
Bing says that in AI-powered search, the sitemap lastmod field "remains a key signal" for deciding which URLs to recrawl. It asks for full ISO 8601 date-times, and for lastmod to reflect when the page content actually changed, not when the sitemap was generated. If every lastmod is set to "now", Bing learns to ignore it.
IndexNow is an open protocol for telling search engines a URL was added, changed or removed, instead of waiting for a recrawl. You host a key file on your domain and send changed URLs (one per GET, or up to 10,000 per POST). According to the IndexNow FAQ, a submission to any participating endpoint is shared with all of them: Bing, Naver, Seznam.cz, Yandex, Yep and Amazon. Google doesn't participate. Submitting doesn't guarantee indexing, but it gets changes discovered sooner, and for Copilot a pricing change that Bing hasn't recrawled is a pricing change Copilot can't see. Platforms are adding it - Bing named Shopify among the adopters in May 2025 - so check your CMS or host first. If yours doesn't, the builder below produces the requests. (We use it on tryzene.com: a daily job submits every sitemap URL that changed.)
Fill in your host, a key and the URLs you just changed. You get the key file to host, a single-URL GET and the batch POST - copy them into your deploy hook or terminal.
1. Host a UTF-8 text file containing only the key at:
https://<your-host>/<your-key>.txt
Bing has two controls for AI use. According to Bing, a page tagged NOCACHE may appear in chat answers, but only its URL, title and snippet are shown. A page tagged NOARCHIVE is not included in or linked from the answers at all. Both still appear in normal Bing results. The 2026 guidelines rewrite also notes, per SEJ, that nosnippet and data-nosnippet may limit how well a page can be cited. NOARCHIVE was traditionally used just to hide a page's cached copy, so check that an old setting isn't now keeping you out of Copilot's answers.
Copilot links whole passages to their sources, so the passage is what gets judged. Answer the question in the first sentence or two of each section. Put comparisons in real HTML tables. Add an FAQ where buyers really ask those questions, and state facts like prices, limits and integrations in plain sentences. This is the same craft as citable pricing pages and clean structured data, and it matches Microsoft's own advice on headings, tables and FAQs.
Search Bing itself for the short queries Copilot is likely to generate ("best [category] software", "[rival] alternatives") and look at what ranks. Those pages are Copilot's raw material. Getting listed in them, and keeping your review profiles and directory listings accurate, is often what decides whether Copilot names you. Our comparison page guide covers the first-party side.
Unlike OpenAI, Microsoft gives publishers a free, first-party citation report. AI Performance in Bing Webmaster Tools (public preview since February 2026) shows total citations, the average number of your pages cited per day, the grounding queries that led Copilot to your content, and citation counts per page. In June 2026 Microsoft added intent and topic groupings, a period-over-period compare view, and citation share: "the percentage of citations attributed to your site out of all citations shown across all sites for that same grounding query". Microsoft calls it "an observational metric - not a ranking system or a competitive scoreboard".
Three ways to use it well:
To be clear about our own coverage: Zene's scans run on ChatGPT, Claude, Gemini, Perplexity and Grok. For Copilot, Bing Webmaster Tools is the first-party view and it costs nothing, so set it up today whatever else you use.
Copilot is unusually open about how it works. It searches Bing with a few derived words, writes from what it finds, links the passages it used and reports those citations back to you. That makes the work clear: get indexed in Bing, keep it fresh with honest sitemaps and IndexNow, don't opt out of AI use by accident, write passages worth linking, and get onto the third-party pages Bing ranks for your category. Then watch AI Performance for citations, and your own prompt tracking for whether you're actually named. Run a free Zene audit to see how the other engines answer your buyer questions while Bing's report covers Copilot.
From Bing when it needs the web. Microsoft documents that Copilot identifies the parts of a prompt that need web information, generates a short search query of a few words, sends it to Bing and writes the answer from the results, with citations to the pages it used. If Bing hasn't indexed a page, Copilot can't use it as a source.
Verify your site in Bing Webmaster Tools, submit a sitemap with honest lastmod dates, turn on IndexNow so changes are picked up quickly, and check that NOARCHIVE or NOCACHE tags aren't limiting AI use. Then write pages with direct answers, clear headings, tables and FAQs - Microsoft's own guidance - and get listed on the third-party pages Bing ranks for your category.
It helps Bing find changes sooner, which is what Copilot grounds on. IndexNow lets you notify participating engines - Bing, Naver, Seznam.cz, Yandex, Yep and Amazon - when a URL is added, updated or removed, and a submission to one is shared with all. It doesn't guarantee indexing, and Google doesn't participate. Microsoft recommends it in its guidance on AI citations.
Yes. The AI Performance report in Bing Webmaster Tools, in public preview since February 2026, shows total citations, average cited pages per day, the grounding queries that led to your content and citations per page, across Copilot, AI summaries in Bing and some partner integrations. A June 2026 update added citation share, intents, topics and a compare view. It counts citations of your pages, not mentions of your brand.
Partly. Copilot is documented to use Bing, offers IndexNow and sitemaps as fast update signals, and reports citations in Bing Webmaster Tools. OpenAI says ChatGPT search uses unnamed third-party search providers plus its own OAI-SearchBot crawler, and offers no publisher citation report. The page craft is the same - quotable, current, crawlable pages - but measure the two engines separately.
Put this guide into practice - get your free visibility score in minutes.
