Publishers block them; brands mostly shouldn't. What each crawler does, what blocking actually prevents, and a sane robots.txt policy for a SaaS.
Muhammet İLBAŞ 7 min readYour server logs already show them: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and a lengthening parade of AI user-agents. News publishers block them in droves, and you've probably seen the "block AI bots" checkbox in your CDN. We leave that checkbox off on tryzene.com, and we'd leave it off on almost any SaaS site. But the crawlers do different jobs, and the difference is what makes the call easy.
A news site's product is its text: an AI that summarizes the article is a substitute for the visit, so blocking training bots is a rational licensing position. Your marketing site is the opposite case - its entire purpose is to be read, understood and repeated. When a model summarizes your features page inside an answer, the page is doing exactly its job. A publisher can reasonably say "stop taking my content"; a marketing site is a brochure you printed hoping someone would repeat it.
This is exactly the policy on tryzene.com's own robots.txt - open marketing pages, closed app, AI crawlers welcome. We build a tool whose whole premise is that models read your site and repeat what they find; blocking them on our own domain would be malpractice.
To confirm they can actually reach and read your pages, check your site's AI readiness score. Letting the crawlers in only gets you read - it doesn't get you mentioned. Run a free audit and see what the engines actually do with the access you're giving them.
For most SaaS marketing sites, no. Your marketing site's entire purpose is to be read, understood and repeated, so a model summarising your features page inside an answer is the page doing exactly its job. News publishers block these bots because their product is the text itself, but a brochure site is the opposite case.
There are three kinds. Training crawlers like GPTBot and Google-Extended collect text for future model training, so blocking them is a bet about models that ship years from now. Retrieval crawlers like PerplexityBot index pages so live AI search can find and cite you, and on-demand fetchers like ChatGPT-User load a page in real time during an answer - blocking either of those removes you from answers today.
Open your marketing surface to everything - home, features, pricing, blog, docs and comparisons, AI crawlers included - and close your app and auth surface to every crawler, since dashboards, settings and checkout have no business being indexed. Remember that well-behaved crawlers respect robots.txt while impersonators exist, so verify against published IP ranges at the CDN if you need to enforce it, and revisit the file yearly as the crawler list grows.
How to find GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot in your logs, verify their IPs, and read training versus live-fetch patterns.
Muhammet İLBAŞ 9 minHow Microsoft Copilot grounds answers in Bing, how it differs from ChatGPT search, and how Bing Webmaster Tools, IndexNow and AI Performance get you cited.
Muhammet İLBAŞ 9 minA mention is your name in an AI answer; a citation is your page linked as a source. How engines show each, share of voice vs citation share, what moves both.
Muhammet İLBAŞ 9 minPut this guide into practice - get your free visibility score in minutes.
