robots.txt used to be a five-line configuration file most treatment center operators never touched. Allow Googlebot, disallow /wp-admin/, point to the sitemap. Done in 2019, forgotten by 2020, still working in 2024.
In 2026, robots.txt is a business decision layer. Not because the file format changed, but because a new crawler class exists that behaves differently from search crawlers and produces different business outcomes when allowed versus blocked. AI bots.
There are now more than a dozen AI bot user agents actively crawling the web, split across two distinct purposes (training future models versus fetching answers for live queries). Each is operated by a company with a different commercial relationship to the sites it crawls.
Each user agent needs its own decision. Blocking one is trivial; blocking the wrong one costs citations and referral traffic in AI answers. Allowing one is trivial; allowing the wrong one gives away content to a system that will never send a click back.
This piece walks the four bot categories, the four decision frameworks a behavioral health operator can pick from, the specific BH gotchas around PHI-adjacent content, and the pragmatic default our content and SEO practice deploys.
It sits alongside the full AI search stack and the entity SEO explainer as the technical-config layer of the same AEO discipline.
Key Takeaways
- robots.txt is a business decision layer in 2026, not just a technical config. AI bots split across training crawlers (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent) and answer crawlers (ChatGPT-User, PerplexityBot, Perplexity-User, Claude-Web, Google’s AI Mode fetcher). Each user agent needs its own decision.
- The most common failure is treating all AI bots as one category. Blocking GPTBot removes your content from OpenAI training but does not stop ChatGPT from citing your facility when a user asks about you — that is ChatGPT-User, a separate user agent. Blocking PerplexityBot removes you from Perplexity’s answer index; blocking Perplexity-User removes you from live Perplexity answers.
- The second common failure is treating AI bot blocking as a HIPAA compliance layer. It is not. HIPAA governance is server-side. robots.txt is a courtesy request that well-behaved crawlers honor and bad-faith actors ignore.
- Google’s public position (September 2023): blocking Google-Extended does not affect Google Search rankings. Blocking Googlebot does. These are separate user agents and separate decisions.
- The recommended default for most treatment centers is allow all major AI bots, disallow specific PHI-adjacent paths (patient testimonials with real names, alumni stories with identifiable details, staff bios that include personal contact info), and review server logs every 90 days.
- Portfolio operators have different economics. If parent-operator content is a competitive moat that AI training consumption erodes, a default-disallow-then-selectively-allow posture makes sense for training crawlers while keeping answer crawlers open.
DEFINITION
AI bot user agent. A named crawler identity operated by an AI company, declared via the User-agent HTTP header when the crawler fetches a URL, that a website can allow or disallow through directives in robots.txt. Distinct from search index crawlers (Googlebot, Bingbot), which index for search engines rather than AI systems.
AI bot user agents fall into two functionally distinct purposes. Training crawlers (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai) fetch URLs to feed future model training corpora. Answer crawlers (ChatGPT-User, PerplexityBot, Perplexity-User, Claude-Web, Google’s AI Mode fetcher) fetch URLs in real time when a live user asks an AI system about the entity. Blocking one class does not block the other.
COMPLIANCE REALITY
robots.txt is not a HIPAA compliance layer. Well-behaved crawlers honor it as a courtesy request. Bad-faith scrapers ignore it. Blocking AI training bots does not protect PHI, because PHI should never have been in public marketing content in the first place.
PHI protection is server-side: auth walls, no PHI in public HTML, no PHI in URLs, no PHI in form action parameters. If PHI is currently in public content on your site, the fix is removing the PHI, not blocking the crawlers who found it. Use robots.txt to shape which AI systems consume your public marketing content, not to substitute for HIPAA governance.
The four AI bot categories
The user agents that show up in server logs today fall into four functional categories. The category determines the business decision.
Category 1: Search index crawlers. Googlebot, Googlebot-Image, Googlebot-Video, Bingbot. These crawl to index for Google Search and Bing Search. Blocking them removes the site from those search engines. No treatment center should block these except from staging or dev subdomains.
Category 2: AI training crawlers. GPTBot (OpenAI), Google-Extended (Google AI/Gemini training, separate from Googlebot), ClaudeBot (Anthropic training), Applebot-Extended (Apple Intelligence training, separate from Applebot), CCBot (Common Crawl, feeds many downstream trainers), Meta-ExternalAgent (Meta AI training), Bytespider (ByteDance), cohere-ai (Cohere). Blocking these removes your content from future model training corpora but does not affect current live AI answers.
Category 3: AI answer/retrieval crawlers. ChatGPT-User (real-time fetch when a ChatGPT user asks about your facility), PerplexityBot (Perplexity’s answer index), Perplexity-User (real-time fetch for Perplexity user queries), Claude-Web (real-time retrieval for Claude), Google’s AI Mode fetcher. Blocking these removes your facility from live AI answers where citations produce referral traffic.
Category 4: SEO and analytics crawlers. AhrefsBot, SemrushBot, MJ12bot, DotBot, and similar. These crawl for third-party SEO tools. Not AI bots, but often bundled into the same robots.txt conversation because operators overwhelmed by the new AI bots default to blanket-blocking for server resource reasons rather than strategy reasons.
The strategic decisions live in Categories 2 and 3. Category 1 is table stakes. Category 4 is a separate resource-cost decision.
The AI bot decision surface at a glance
4
Bot categories: search index, AI training, AI answer, SEO/analytics
4
Decision frameworks operators can pick from (A, B, C, D)
12+
Named AI bot user agents actively crawling as of 2026
90 days
Review cadence for robots.txt and server-log user agent audit
The four decision frameworks
Treatment center operators can pick one of four decision frameworks depending on their business model, portfolio structure, and content strategy.
Framework A: Default-allow with selective PHI-adjacent disallow
Allow all AI bots (both training and answer). Add specific Disallow rules for paths that contain PHI-adjacent content: patient testimonials with real names, alumni story pages with identifiable details, family-narrative surfaces. This is the recommended default for most single-facility operators. The pre-work is inventorying which URL patterns carry PHI-adjacent content and confirming the disallow list matches.
Framework B: Allow answer crawlers, disallow training crawlers
Allow Category 3 (ChatGPT-User, PerplexityBot, Perplexity-User, Claude-Web, Google’s AI Mode fetcher) so the facility gets cited in live AI answers. Disallow Category 2 (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai). This is the framework for portfolio operators with a content moat concern and for facilities running proprietary clinical methodology content they do not want absorbed into open-web training corpora.
Framework C: Default-allow everything except explicit competitive concern
Same as Framework A, but without the PHI-adjacent disallow layer. Appropriate for facilities whose content is entirely public marketing (no testimonials, no alumni stories, no PHI-adjacent surfaces). Rare in behavioral health because most centers publish some form of patient narrative content.
Framework D: Default-disallow with selective allow
Disallow all AI bots by default, then allow only the specific user agents the operator has an explicit reason to invite in. This posture fits portfolio operators with a strong content moat concern (proprietary clinical frameworks, unique methodology, competitive positioning) that AI training consumption would erode. Requires the highest maintenance discipline of the four frameworks.
The right framework depends on where the operator sits on the spectrum from we need every citation we can get to we have a content moat worth protecting. For most single-facility operators, Framework A. For most portfolio operators, Framework B. For proprietary-methodology operators, Framework D.
Behavioral health-specific considerations
Four content types produce specific AI-bot decisions that generic robots.txt guides do not cover.
Patient testimonials with identifiable details. Names, cities, specific timelines, family situations. Even when the patient has signed a release for marketing use, the release usually does not contemplate model training consumption. Add Disallow rules for the URL patterns that hold identifiable testimonials, and confirm the patterns match the actual URL structure the site publishes those pages under.
Alumni narratives. Similar to testimonials but often longer-form and more identifiable. Some facilities publish these under /alumni/, /community/, or as blog category subdirectories. Same disallow logic applies.
Staff bios with personal contact information. Bios that list direct staff email addresses or personal phone numbers should not be in training corpora. If the bios do not include personal contact info (which they should not on a well-configured site), they can stay open because clinical credentials are the entity-graph signal you want AI systems to consume.
Program pages, service pages, and level-of-care pages. These are public marketing content and are the primary reason to allow AI bots. Blocking these removes the facility from exactly the AI answers a facility wants to appear in. Keep these open.
Insurance verification pages, VOB forms, and any URL that accepts patient information through a form should be behind auth or noindexed regardless of robots.txt. robots.txt is not a security layer.
The four common mistakes
Mistake 1: Treating robots.txt as a HIPAA compliance layer. It is not. Well-behaved crawlers honor robots.txt as a courtesy. Bad-faith scrapers ignore it. PHI protection has to be server-side: auth walls, no PHI in public content, no PHI in URLs, no PHI in form action URLs, no PHI in query strings. The robots.txt file cannot fix architecture that leaks PHI into public content.
Mistake 2: Blocking Google-Extended and thinking you blocked Google Search. Google-Extended is the AI training crawler. Googlebot is the search crawler. Google confirmed publicly (September 2023) that blocking Google-Extended does not affect Google Search rankings. These are separate user agents that make separate decisions and the confusion between the two is the single most common self-inflicted rank loss we see.
Mistake 3: Blocking GPTBot and expecting no more ChatGPT citations. GPTBot is the training crawler. ChatGPT-User is the answer crawler that fetches URLs when a live user asks ChatGPT about your facility. Blocking GPTBot removes your content from training corpora but does not prevent ChatGPT from citing your public content in real time. The two decisions are separate.
Mistake 4: Blanket-blocking every AI user agent found in a blog post. Not all bots claiming AI purpose are worth blocking. The strategic question is: what does allowing this crawler produce for the facility? Blanket-blocking Category 3 answer crawlers is the most common self-inflicted citation loss and shows up in the measurement approach we walk in 6 ways to get your rehab center cited in AI search answers.
The recommended default configuration
For most single-facility treatment centers, Framework A applied cleanly. The specific configuration:
Allow all Category 1 search crawlers (Googlebot, Bingbot, and their variants). Allow all Category 3 answer crawlers (ChatGPT-User, PerplexityBot, Perplexity-User, Claude-Web).
Allow all Category 2 training crawlers (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai).
Disallow the specific paths that carry PHI-adjacent content (/testimonials/, /patient-stories/, /alumni/, /success-stories/, whichever URL patterns the site uses). Disallow the standard non-public admin paths (/wp-admin/, /wp-login.php) with the standard /wp-admin/admin-ajax.php allowance.
For portfolio operators, Framework B with the same PHI-adjacent disallow layer, plus specific disallow rules for Category 2 training crawlers while keeping Category 3 answer crawlers open.
The robots.txt file is not a set-and-forget configuration in 2026. New AI bot user agents appear on a rolling basis. Server log review every 90 days will surface user agents the current file does not address, which then get evaluated against the framework the operator picked.
DO
- Pick one of the four frameworks (A default-allow with PHI disallow, B allow answer disallow training, C default-allow, D default-disallow selective allow) and document why.
- Keep ChatGPT-User, PerplexityBot, Perplexity-User, Claude-Web, and Google’s AI Mode fetcher open unless there is a specific reason to block them — these are the answer crawlers that produce citations.
- Add Disallow rules for the specific URL patterns that carry PHI-adjacent content (/testimonials/, /patient-stories/, /alumni/, /success-stories/) — whichever patterns your site uses.
- Review server logs every 90 days, filter by user agent, and update the file to address new AI bots that started crawling since the last review.
- Layer X-Robots-Tag with noai and noimageai on PHI-adjacent responses as a secondary defense, not the primary control.
DON’T
- Treat robots.txt as a HIPAA compliance layer — PHI governance is server-side (auth walls, no PHI in public content, no PHI in URLs).
- Block Google-Extended assuming it also blocks Google Search — Google confirmed publicly (Sept 2023) that Google-Extended is separate from Googlebot and does not affect Search rankings.
- Block GPTBot and expect no more ChatGPT citations — GPTBot is the training crawler, ChatGPT-User is the answer crawler that fetches URLs in real time when a user asks about your facility.
- Blanket-block every AI user agent you find in a blog post — blanket-blocking Category 3 answer crawlers is the most common self-inflicted citation loss.
- Rely on meta robots noindex to block AI training crawlers — it does not stop the fetch itself and is not consistently honored by all AI bots.
What to watch after publishing
Three signals should be monitored after any robots.txt change.
Server logs, filtered by user agent, to confirm the intended bots are honoring the file and to catch new user agents the file did not anticipate. The specific pattern to watch: an AI bot that starts fetching aggressively should be evaluated against the framework, not blanket-blocked in a reflex.
AI citation appearances, tracked through the measurement approach in the 6 ways to get your rehab center cited in AI search piece. If citations drop after a robots.txt change, the change probably blocked an answer crawler unintentionally. The AI Mode vs AI Overviews piece covers where the specific citation signals surface across Google’s two AI surfaces.
Google Search Console coverage reports, which will surface any Googlebot-related regression. If the robots.txt change accidentally caught a search crawler, GSC will show it as a coverage error within a few days.
Frequently Asked Questions
Should behavioral health treatment centers block AI training bots to protect patient privacy?
No. Blocking AI training bots is not a patient privacy layer. Patient privacy is protected by server-side controls: no PHI in public content, auth walls on any surface that accepts patient information, and clean URL structures that never expose patient identifiers.
Blocking GPTBot, Google-Extended, or ClaudeBot from crawling your public marketing content does not protect PHI, because PHI should never have been in that content in the first place. If PHI is currently in public content on the site, the fix is removing the PHI, not blocking the crawlers that found it.
The one legitimate PHI-adjacent case is patient testimonials with identifiable details. Even when the patient has consented to marketing use, the release usually does not contemplate model training. Disallowing /testimonials/ or the equivalent URL pattern is a reasonable addition to the file.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI’s training crawler. It fetches URLs to feed into model training corpora that will inform future versions of ChatGPT and other OpenAI models.
ChatGPT-User is OpenAI’s live browsing crawler. It fetches URLs in real time when a ChatGPT user asks about a specific facility, topic, or entity, and ChatGPT decides to browse the web to answer.
Blocking GPTBot removes your content from training data. Blocking ChatGPT-User removes your facility from live ChatGPT answers. The two decisions are separate. Most treatment centers should allow both. Some may block GPTBot but keep ChatGPT-User open, which is Framework B.
Does blocking Google-Extended hurt Google Search rankings?
No. Google confirmed publicly in September 2023 that Google-Extended is a separate crawler purpose from Googlebot and that its use has no effect on Google Search rankings or Google Search inclusion.
Google-Extended fetches URLs to inform Google’s AI training (including Gemini and the models behind Google AI Overviews). Googlebot fetches URLs to index for Google Search.
Blocking Google-Extended removes your content from Google’s AI training corpus but leaves Google Search intact. Blocking Googlebot removes your content from Google Search itself.
How often should we review our robots.txt file?
Every 90 days. New AI bot user agents launch on a rolling basis, and existing bots occasionally change behavior or rename user agents.
The specific review cadence: pull server logs, filter by user agent, identify any new AI bots that have started crawling since the last review, evaluate each against the framework the facility picked, and update the file accordingly.
Also confirm that the disallowed paths (PHI-adjacent content) still exist at those URLs and that no new PHI-adjacent content has been added at unlisted paths. Facilities that skip the 90-day review typically end up with a file that either blocks bots that no longer exist or fails to address bots that started crawling recently.
Should we use meta robots noindex instead of robots.txt disallow for AI-sensitive pages?
Different tools for different purposes. robots.txt Disallow tells the crawler not to fetch the URL at all. Meta robots noindex allows the crawler to fetch the URL but tells it not to include the URL in an index.
For AI bots, robots.txt Disallow is the appropriate tool because it prevents the fetch itself, which is what training corpora consume. Meta robots noindex does not stop the training fetch and is not consistently honored by all AI bots.
Meta robots noindex is the right tool for keeping specific URLs out of search results while still allowing the crawler to fetch them (typical use: internal search pages, thank-you pages, low-quality URL parameters). It is not the right tool for AI training moat protection.
What about the X-Robots-Tag HTTP header for AI bots?
X-Robots-Tag is a response header that carries the same instructions as meta robots but at the HTTP layer rather than in HTML. It can carry directives like noai and noimageai that some AI bots respect.
The current AI bot ecosystem’s honoring of noai and noimageai is uneven. GPTBot honors noai in the sense that OpenAI’s stated policy is to respect it, but the granular enforcement varies. Other AI bots vary widely in whether they respect it.
The pragmatic recommendation for treatment centers: use robots.txt as the primary AI bot control layer because it is more reliably honored and easier to audit. Layer X-Robots-Tag with noai, noimageai on specific PHI-adjacent responses as a secondary defense. Our get-cited-in-AI-search framework covers the citation-side measurement approach after any robots.txt change.
Trevor Gage is the Director of Marketing at Webserv, a digital marketing agency for treatment centers.
