info.link logo
Go back
Crawlers & Access

Which AI crawlers does robots.txt control? The declared bots, not user fetches or CDN blocks

Sites now block AI training crawlers more often than AI search crawlers: in a 4,242-file robots.txt sample, the main training bot was disallowed 1.60 times per allow and the main search bot 0.91. Robots.txt is one gate of three, beside CDN bot settings and the HTML the server sends.

Robots.txt controls the AI crawlers that declare themselves, and site owners now use it to treat training and search bots differently. In a September 2026 snapshot of 4,242 robots.txt files, GPTBot was disallowed 1.60 times per allow, and OAI-SearchBot about as often as allowed TechnologyChecker.io · 2026. The file is only one of three gates. User-triggered fetches can sit outside it. CDN bot settings can block crawlers the file allows. AI crawlers also read only the HTML the server sends. The sections below take the crawlers first, then each gate in turn.

AI crawlers split into training, search and user-fetch bots

There is no single AI crawler. Most vendors run separate user agents for training, for a search index and for fetches a user triggers mid-conversation. OpenAI lists GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for user-initiated actions OpenAI · 2024. Its live page now adds OAI-AdsBot, which checks landing pages submitted as ChatGPT ads.

Anthropic mirrors that split with ClaudeBot, Claude-SearchBot and Claude-User, each controllable on its own Anthropic · 2026. Perplexity runs PerplexityBot for its index and Perplexity-User for on-demand fetches Perplexity AI · 2024.

Google and Apple use control tokens rather than separate crawlers. Google-Extended governs Gemini training and grounding, and Google states it does not affect inclusion or ranking in Google Search Google · 2024. Applebot-Extended governs training of Apple's foundation models, while Applebot itself powers Siri and Spotlight Apple · 2025. Microsoft has no separate AI crawler: Copilot grounds on Bingbot data, so there is no Copilot-specific user agent to block Microsoft · 2025.

Robots.txt binds the declared crawlers, not every fetch

The user-fetch bots are where robots.txt stops being reliable. OpenAI's documentation says ChatGPT-User actions are initiated by a user, so "robots.txt rules may not apply" OpenAI · 2024. The change dates from a December 2025 revision that removed the earlier compliance commitment PPC Land · 2025. Anthropic, by contrast, states that all three of its bots respect robots.txt Anthropic · 2026.

Compliance in practice is uneven. A 40-day log study of 130 self-declared bots found that AI search crawlers rarely checked robots.txt at all arXiv · 2025. Bots overall were less likely to obey stricter directives. The study draws on one institution's logs, so the rates may not generalise. Cloudflare caught Perplexity fetching blocked test domains through a generic Chrome user agent and undisclosed IP ranges Cloudflare · 2025.

The answer-side data points the same way. Among the top 50 news sites blocking ChatGPT bots, 82.4% of those blocking OAI-SearchBot still appeared in AI citations Nero · BuzzStream · 2026. That study is cross-sectional and cannot show which route the content took.

AI crawler blocking statistics: training bots are blocked more than search bots

Sites block training crawlers more often than search crawlers. In the same Cloudflare Radar snapshot, ClaudeBot was disallowed 1.58 times per allow and Google-Extended 1.38 times, against 0.91 for OAI-SearchBot TechnologyChecker.io · 2026. The sample is about 4,200 top-domain files, not the whole web, and the vendor flags September as provisional.

The crawl-to-referral gap explains the pattern. In 2025, ClaudeBot crawled around 70,000 pages for every visitor it referred, GPTBot around 1,700 and Perplexity around 5 Cloudflare · 2025. Cloudflare's year-end figures put Anthropic's peak at 500,000 to 1 InfoQ / Cloudflare · 2025. Those ratios are the usual case for blocking training bots: they read far more than they send back.

CDN bot settings can block AI crawlers, and Googlebot, without a robots.txt change

A site's CDN can override what its robots.txt allows. Cloudflare made blocking AI bots the default for sites it hosts on 1 July 2025 MIT Technology Review · 2025. In July 2026 it replaced the single switch with three categories: Search, Agent and Training Lee · Cloudflare · 2026.

From 15 September 2026, new Cloudflare domains block Training and Agent by default on pages that show ads, while Search stays allowed. The same announcement says multi-purpose crawlers, naming Googlebot, Applebot and Bingbot, are blocked for customers who block Training Lee · Cloudflare · 2026. Site owners reported verified Googlebot and Bingbot receiving HTTP 403 errors on sitemap fetches after setting that block Montti · Search Engine Journal · 2026. The report is anecdotal and gives no incidence rate.

Cloudflare also writes preferences into robots.txt for sites that use its managed file. Its Content Signals Policy set search=yes and ai-train=no by default for 3.8 million domains Cloudflare · 2025. Those signals are preferences, not enforcement.

AI crawlers read the server-sent HTML, not what JavaScript builds

Content that appears only after JavaScript runs is invisible to the major AI crawlers. On Vercel's network in late 2024, none of the crawlers from OpenAI, Anthropic, Meta, ByteDance or Perplexity executed JavaScript Vercel + MERJ · 2025. ChatGPT spent 34.82% of its fetches on 404 pages, against 8.22% for Googlebot. The data is two years old, and crawler behaviour can change between releases.

A 2026 study shows the same gap on commercial pages. Of 77 Cloud 100 companies with public pricing pages, 10 hid at least 40% of their body content from bots Poyar · Profound; Growth Unhinged · 2026. The causes were client-side rendering, tabs, accordions and price calculators, robots.txt blocks, and iframes from uncrawlable domains. The study is a single vendor's, built on one prompt template.

AI access standards are moving from robots.txt toward signed requests and payment

Three proposals would replace the bare user-agent string with something more precise. The IETF AIPREF draft defines two usage categories, train-ai and search, each marked allow, disallow or unspecified IETF AIPREF Working Group · 2026. It is a working-group draft and does not yet reflect consensus.

Web Bot Auth lets a crawler sign its requests so a site can verify who is fetching. Google is testing it, with only some Google AI agents signing during the experiment Google · 2025. Cloudflare has moved from Pay Per Crawl to Pay Per Use experiments, which pay owners when content appears in an AI answer Conroy · Cloudflare · 2026. No pricing or results have been published, so none of the three yet changes what a robots.txt audit finds today.

FAQ
Frequently Asked Questions

Sources

Sources are tiered per our methodology & sources page.

Key finding

On its second Content Independence Day, 1 July 2026, Cloudflare announced a shift from Pay Per Crawl to Pay Per Use, paying content owners when their content actually appears in an AI answer rather than each time a crawler fetches a page. Cloudflare data shows more than 50% of crawl traffic from good bots re-fetches unchanged pages. Named partners: Ceramic.ai (pay-per-query) and You.com (pay-on-demand). No pricing disclosed.

Methodology note

First-party Cloudflare Blog post 'Making AI search smarter' by Matthew Conroy, published 1 July 2026, announcing Pay Per Use experiments and AEO reporting for participating content owners. Canonical URL confirmed and page fetched directly on 2026-08-06. An announcement of network-scale experiments, not a completed study; no pricing or results yet.

Cloudflare Blog·Accessed
Tier A — Strongest evidenceRead source

Your site, your rules: new AI traffic options for all customers (Content Independence Day 2026)

Cloudflare · Jin-Hee Lee, Bryan Becker · 2026

Key finding

Cloudflare replaced its single 'AI bots' block with three categories, available to all customers: Search (indexes to answer later, expected to return referrals), Agent (real-time action for a person) and Training (absorbed into a model). From 15 September 2026, new domains block Training and Agent by default on pages showing ads, while Search stays allowed. Multi-purpose crawlers are judged by all behaviors, so blocking Training also blocks them.

Methodology note

Official Cloudflare product and policy announcement (its second 'Content Independence Day') by Jin-Hee Lee and Bryan Becker, published 1 July 2026. Also introduces a robots.txt content-use signal (immediate/reference/full, default reference), a redefinition of 'Verified' bots, and a transitive-trust proposal using the Forwarded header (RFC 7239). Verified by direct fetch and by TechCrunch and Cloudflare's own changelog.

Cloudflare Blog·Accessed
Key finding

Anthropic operates three distinct web crawlers with separate robots.txt user agents: ClaudeBot collects content for training foundation models, Claude-User fetches pages on demand when users ask Claude a question, and Claude-SearchBot indexes content for Claude's search features. Each can be allowed or blocked independently, letting site owners opt out of training while still appearing in Claude's search answers. All three respect robots.txt and support the non-standard Crawl-delay directive.

Methodology note

Official Anthropic crawler documentation, formalised in updates throughout 2025. The page was inaccessible to direct fetch; user-agent strings, behaviour and robots.txt rules were confirmed against Anthropic's Claude Help Center article and multiple secondary references (Search Engine Journal, Search Engine Land, Search Engine Roundtable) reporting the same three-bot framework.

Anthropic Docs·Accessed
Tier A — Strongest evidenceRead source

A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab)

IETF AIPREF Working Group · 2026

Key finding

The IETF AIPREF working group is developing a standard vocabulary for websites to express how their content can be used by AI systems. The current draft defines two usage categories, train-ai and search, each of which can be marked allow, disallow, or unspecified. A site might publish train-ai=y, search=n to permit AI training while disallowing search indexing. The format is designed to plug into robots.txt and other carriers.

Methodology note

Working group Internet-Draft from the IETF AI Preferences group, edited by Paul Keller (Open Future) and Martin Thomson (Mozilla). The version reviewed is draft-ietf-aipref-vocab-06, last updated April 28, 2026, intended for Proposed Standard status. The document is a work in progress and does not yet reflect working group consensus.

IETF Datatracker·Accessed
Key finding

Google is testing Web Bot Auth, a cryptographic protocol that lets bots sign HTTP requests so sites can verify their identity beyond user-agent strings or IP address ranges. During the experimental phase, only some Google AI agents sign requests, and signatures use HTTP Message Signatures (RFC 9421) keyed to https://agent.bot.goog. Google recommends continued reliance on reverse DNS and published IP ranges as a fallback.

Methodology note

Official Google Crawling Infrastructure documentation, last updated May 4, 2026, describing Google's implementation of the IETF Web Bot Auth Internet-Draft. The page links to the IETF Working Group, a Cloudflare reference implementation on GitHub, and a feedback form. Web Bot Auth itself is still a draft specification that may change.

Google for Developers·Accessed
Key finding

Cloudflare introduced a Content Signals Policy that extends robots.txt with three new signals: search (indexing for traditional search), ai-input (use in retrieval augmented generation or AI answers), and ai-train (use for training or fine-tuning AI models). Each can be set to yes or no, or left blank. For 3.8 million domains already using Cloudflare's managed robots.txt, the company will publish search=yes, ai-train=no by default.

Methodology note

Official Cloudflare product announcement from September 24, 2025, written by Will Allen. The policy text is released under a CC0 license to encourage adoption, with a generator at ContentSignals.org. Cloudflare notes signals are preferences, not technical countermeasures, and recommends combining them with WAF and bot management rules.

Cloudflare Blog·Accessed
Key finding

AI crawlers read content far more than they send referrals back. Anthropic's ClaudeBot crawled around 70,000 pages for every visitor it referred; OpenAI's GPTBot crawled around 1,700 for every visitor; Perplexity around 5 for every visitor. Mistral was the only major AI engine where referrals outweighed crawl volume.

Methodology note

Aggregate analysis of crawl requests and referral traffic across the Cloudflare network. For each major AI crawler, the team divided pages crawled by visits sent to the same destinations during the same window, producing a crawl-to-refer ratio. Published August 2025.

Cloudflare Blog·Accessed
Key finding

Cloudflare observed Perplexity using undeclared crawlers to fetch content from sites that had blocked its known bots in robots.txt and WAF rules. When PerplexityBot was blocked, traffic rotated through a generic Chrome-on-macOS user agent, undisclosed IP ranges, and multiple ASNs, hitting tens of thousands of domains and millions of requests a day. Cloudflare de-listed Perplexity from its verified bots list and added detection rules.

Methodology note

Cloudflare blog post from August 4, 2025, by Gabriel Corral, Vaibhav Singhal, Brian Mitchell, and Reid Tatoris. The team set up brand new test domains that had never been indexed, applied robots.txt and WAF blocks, then queried Perplexity about content on those domains. Detection used machine learning plus network signals.

Cloudflare Blog·Accessed
Tier A — Strongest evidenceRead source

Which Crawlers Does Bing Use? (incl. Copilot)

Microsoft · 2025

Key finding

Bing identifies several active crawlers. Bingbot is the main web crawler for Bing search. AdIdxBot crawls pages for Bing Ads. BingPreview generates page snapshots. MicrosoftPreview supports preview cards. Copilot uses Bingbot data for grounding rather than a separate crawler. Publishers can issue user-agent specific rules in robots.txt, and Bing respects standard directives and meta robots tags for indexing control.

Methodology note

Official Bing Webmaster Tools help documentation page that lists Bing's crawlers, their purposes, and their user-agent strings. The page is the canonical reference for site owners configuring crawler rules for Microsoft search and Copilot.

Bing Webmaster·Accessed
Tier A — Strongest evidenceRead source

About Applebot (incl. Applebot-Extended)

Apple · 2025

Key finding

Apple uses one crawler, Applebot, to gather data that powers Spotlight, Siri, and Safari search. A separate user agent, Applebot-Extended, governs whether content is used to train Apple's foundation models, including Apple Intelligence. Sites can disallow Applebot-Extended in robots.txt to opt out of generative AI training while keeping content discoverable in Apple search. Applebot is identified via reverse DNS at applebot.apple.com or a published IP CIDR list.

Methodology note

Official Apple support documentation about Applebot, last updated April 25, 2025. The page details user-agent strings, robots.txt behavior, supported meta directives such as noindex, nosnippet, nofollow, and how Applebot-Extended works as a secondary control specifically for generative AI training.

Apple Support·Accessed
Tier A — Strongest evidenceRead source

Scrapers Selectively Respect robots.txt Directives

arXiv · 2025

Key finding

Audit of major AI scrapers' compliance with robots.txt directives finds that compliance is selective rather than uniform: scrapers honour blocks on some user agents and not others, even within the same vendor, and compliance changes over time as crawlers are updated. The paper provides specific evidence of non-compliance events with named vendors and dates.

Methodology note

arXiv preprint 2505.21733 (May 2025). Direct fetch on arxiv.org returned the abstract page. The paper documents specific non-compliance events with named scrapers using server-log evidence and timestamped robots.txt snapshots.

arXiv·Accessed
Tier A — Strongest evidenceRead source

The Rise of the AI Crawler (Vercel + MERJ, 1B requests)

Vercel + MERJ · 2025

Key finding

On Vercel's network in late 2024, GPTBot generated 569 million monthly requests and Anthropic's Claude generated 370 million, together roughly 20% of Googlebot's 4.5 billion. None of the major AI crawlers (OpenAI, Anthropic, Meta, ByteDance, Perplexity) executed JavaScript. ChatGPT spent 34.82% of fetches on 404 pages, Claude 34.16%, versus 8.22% for Googlebot. Server-side rendered content is far more visible to AI crawlers.

Methodology note

Joint research from Vercel and MERJ, published December 17, 2024. Data comes from monitoring nextjs.org and the Vercel network, validated against two job-board sites (Resume Library on Next.js and CV Library on a custom monolith). Microsoft Copilot was excluded because it lacks a distinct user agent. Methods follow the same approach used in MERJ's earlier Googlebot analysis.

Vercel Blog·Accessed
Key finding

Google lists its common crawlers with their robots.txt tokens and the products each affects. Google-Extended has no separate user agent; it is a control token that governs whether crawled content may train future Gemini models and be used for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google states it does not affect inclusion or ranking in Google Search.

Methodology note

Official Google Search Central documentation listing the common Googlebot variants and their robots.txt tokens. The page was inaccessible to direct fetch; user-agent strings, IP-range publication mechanism and reverse-DNS verification process were confirmed through the live developers.google.com URL referenced in search results and supporting third-party crawler reference databases.

Google Search Central·Accessed
Key finding

OpenAI documents three distinct user agents, controllable independently via robots.txt. OAI-SearchBot surfaces sites inside ChatGPT search results; blocking it removes a site from those answers but still allows navigational links. GPTBot crawls content used to train foundation models. ChatGPT-User performs on-demand fetches triggered by user prompts, Custom GPTs and GPT Actions. Sites that allow both OAI-SearchBot and GPTBot may be crawled only once, with results shared between use cases.

Methodology note

Official OpenAI developer documentation listing each crawler, its full user-agent string and its published IP-range JSON file (searchbot.json, gptbot.json, chatgpt-user.json). The page sets out which robots.txt directives apply to each agent, the approximate 24-hour lag for search opt-outs to take effect, and the policy that user-initiated ChatGPT-User requests are not governed by robots.txt.

OpenAI Developer Docs·Accessed
Key finding

Perplexity documents two user agents controllable via robots.txt: PerplexityBot indexes content for retrieval in Perplexity answers, while Perplexity-User fetches pages on demand when a user submits a query. Pages disallowed for PerplexityBot are not indexed in full, though Perplexity may still display the domain, headline and a brief factual summary. IP ranges are published at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. (agent inferred)

Methodology note

Official Perplexity crawler documentation. Original URL docs.perplexity.ai/guides/bots returns HTTP 308 redirect; the canonical content now lives at docs.perplexity.ai/docs/resources/perplexity-crawlers. User-agent strings, robots.txt behaviour and IP-publication URLs (perplexity.com/perplexitybot.json, perplexity-user.json) cross-verified via 51Degrees, Known Agents and CrawlerCheck.

Perplexity Docs·Accessed
Key finding

In Cloudflare Radar's 28 September 2026 snapshot of 4,242 robots.txt files, every training crawler is disallowed more often than allowed (GPTBot 1.60:1, ClaudeBot 1.58:1, Google-Extended 1.38:1) while OAI-SearchBot (0.91:1) is allowed as often as blocked. Over twelve months, Allow rules naming GPTBot rose from 2.51% to 7.69% of files; September's disallow drop likely reflects a sample change.

Methodology note

Vendor report by TechnologyChecker.io, first published 2026-04-03 and updated 2026-10-01, re-querying named Cloudflare Radar API endpoints (robots_txt directive, crawl_refer_ratio, crawl_purpose) and normalising counts by files parsed per snapshot. Fetched directly. Counts cover Cloudflare's sample of top domains rather than the whole web; the vendor flags September as provisional.

TechnologyChecker Blog·Accessed
Tier B — Citable with caveatsRead source

What AI agents really think about your pricing (Profound x Growth Unhinged)

Profound; Growth Unhinged · Kyle Poyar, Nikolas Laskaris · 2026

Key finding

Across about 7,600 AI responses to a pricing prompt for the Cloud 100 (29 July to 10 August 2026, six engines), company pricing pages were cited in 46% of responses but first in only 12%. Of 77 companies with public pricing pages, 57 were fully readable to bots. Plaid's pricing page was cited first 42% of the time, with billing docs in 70% and FAQs in 50% of answers.

Methodology note

Single-vendor study by Kyle Poyar of Growth Unhinged and Nikolas Laskaris of Profound, published 3 September 2026 and fetched directly. One prompt template, Cloud 100 sample, six engines. Response counts differ slightly (about 7,600 versus 7,675), and the authors say they cannot prove llms.txt causes better retrieval. Partnership disclosed.

Profound Blog·Accessed
Tier B — Citable with caveatsRead source

Report That Cloudflare AI Bot Blocking Prevents Googlebot From Indexing Sites

Search Engine Journal · Roger Montti · 2026

Key finding

Site owners report that setting Cloudflare's AI Training block returns HTTP 403 to verified Googlebot and Bingbot fetching their sitemaps, because Cloudflare classifies both as Search plus Training crawlers. Google's John Mueller asked the reporter for details. Cloudflare's own documentation confirms mixed-purpose crawlers are blocked by every training-block configuration including the legacy Block AI bots toggle, from 15 September 2026.

Methodology note

Trade-press news report by Roger Montti for Search Engine Journal, 4 August 2026, fetched directly 14 September 2026. Built on a single r/TechSEO thread plus a verbatim quote of Cloudflare's block-ai-bots documentation. The article states explicitly that the report is anecdotal and that user error is not ruled out. No sample, no incidence rate.

Search Engine Journal·Accessed
Tier B — Citable with caveatsRead source

Do News Publishers That Block AI Crawlers Get Cited Less Often by AI? (BuzzStream)

BuzzStream · Vince Nero · 2026

Key finding

Among the top 50 news sites that block ChatGPT bots, 70.6% blocking ChatGPT-User, 82.4% blocking OAI-SearchBot and 88.2% blocking GPTBot still appeared in AI citations; about 70% of ChatGPT citations came from sites blocking its retrieval bots. Cross-sectional, news-only and without a counterfactual, and the post's Google-Extended figure is inconsistent (92.3% in text, 88.2% in table).

Methodology note

Vendor study by Vince Nero (BuzzStream), published 2026-03-18 and updated 2026-04-08, using Citation Labs' XOFU tracker: 4 million citations from 3,600 prompts across ChatGPT, Gemini, AI Overviews and AI Mode in 10 industries, matched to robots.txt rules of the top 50 blocking news sites. Fetched directly; per-engine citation denominators are not fully disclosed.

BuzzStream Blog·Accessed
Key finding

Cloudflare's 2025 Radar Year in Review reports global internet traffic up 19% year on year, with Googlebot still the largest single source of crawler traffic. Crawl-to-refer ratios widened sharply: Anthropic peaked at 500,000 to 1, OpenAI at 3,700 to 1, vast numbers of crawls per referral click. Half of human web traffic now uses post-quantum encryption, and Go's share of automated API requests jumped from 12% to 20%.

Methodology note

InfoQ summary by Renato Losio of Cloudflare's sixth annual Radar Year in Review, published December 31, 2025. Data is drawn from Cloudflare's edge network and the 1.1.1.1 public DNS resolver. The InfoQ piece highlights and aggregates the figures from Cloudflare's own published Radar microsite.

InfoQ·Accessed
Key finding

On December 9, 2025, OpenAI updated its crawler documentation. ChatGPT-User, which handles user-initiated browsing inside ChatGPT, no longer commits to following robots.txt, on the basis that requests come from a user rather than an autonomous crawler. OAI-SearchBot is now described purely as a search crawler, with training data removed from its scope. GPTBot and OAI-SearchBot may also share crawl results to avoid duplicate fetches.

Methodology note

PPC Land news article by Luis Rijo summarising changes spotted by digital marketing consultant Pieter Serraris in OpenAI's public bots documentation. The article quotes OpenAI's previous and revised wording side by side and links to the live OpenAI bots reference page on platform.openai.com.

PPC Land·Accessed
Tier B — Citable with caveatsRead source

Cloudflare Will Now Block AI Bots by Default

MIT Technology Review · 2025

Key finding

Cloudflare made blocking AI bots the default for websites it hosts as of July 1, 2025. Customers can override per bot, allow verified crawlers, or charge for access via Pay Per Crawl. Media outlets including the Associated Press and Time, plus platforms like Quora and Stack Overflow, endorsed the move. CEO Matthew Prince argues current AI use of the web is breaking the publisher business model.

Methodology note

Reporting in MIT Technology Review by Peter Hall, published July 1, 2025, covering Cloudflare's announcement and including direct comment from Will Allen, Cloudflare's head of AI privacy, control, and media products. The piece also includes a contrasting view from MIT Media Lab PhD candidate Shayne Longpre on impacts to research and non-commercial use.

MIT Technology Review·Accessed

About the author Max Ackermann

Max Ackermann is founder and Managing Director of info.link, the product data platform that makes brands visible in AI search and connects every physical product to the web through GS1 Digital Link. He writes about AI search and generative engine optimization (GEO), AI-powered commerce, and how brands can structure product data for ChatGPT, Gemini, Perplexity, and retailer AI assistants like Amazon Rufus. For the past two years he has built the pipelines that put structured product data into AI answers, and run the experiments that test what actually moves AI citations.

Max has 20+ years of experience building digital products and businesses. He previously led McKinsey's Corporate Venture and Design teams across Europe, and as Managing Director of a leading US digital agency he built platforms with Nike, Google, Meta, and Airbnb. He founded the UX Design program at Central Saint Martins College, University of the Arts London, and is a Fellow of the UK's Higher Education Academy. Based in Hamburg, he works closely with GS1 on Digital Link adoption; info.link is headquartered in Hamburg and Berlin and counts GS1 Germany among its investors.

Follow Max on LinkedIn.

Interested?

From compliant digital labels to AI-verified product answers, we help leading brands ensure their products are visible and accurately represented everywhere consumers look. Book your free consultation and demo.

digital label preview
digital label preview
digital label preview
AI crawlers and robots.txt: what each bot can reach | info.link