Why does the 2026 PROEMA robots.txt pattern block GPTBot but allow OAI-SearchBot?

Because OpenAI uses two user-agents for two purposes: GPTBot ingests data to train models, OAI-SearchBot ingests data to answer in real-time inside ChatGPT-Search. Block the first to protect your IP, allow the second to stay citable.

Why does the 2026 PROEMA robots.txt pattern block GPTBot but allow OAI-SearchBot, for a brand?

OpenAI documented the split in August 2023: GPTBot = training acquisition, OAI-SearchBot = live retrieval for ChatGPT Search, ChatGPT-User = agentic plugin execution (source: developers.openai.com/api/docs/bots). Anthropic mirrors the logic: ClaudeBot (training) vs Claude-Web / Claude-User (retrieval, source: support.anthropic.com). Google split Google-Extended (training Gemini) from Googlebot (search). The split is not symbolic, it is explicitly documented and respected, confirmed by independent checks from AppearOnAI, NoHacks and Mersel in 2025-2026.

The strategic compromise of the 2026 pattern

The strategic upshot: blanket-block AI bots and you disappear from ChatGPT-Search, Perplexity, Claude. Allow everything and you feed competitors' training for free. "Our 2026 pattern allows OAI-SearchBot, PerplexityBot, Bingbot, Applebot and MistralAI-User and blocks GPTBot, ClaudeBot, Google-Extended and anthropic-ai: it is the middle path, maximum retrieval visibility with minimum training leakage," explains Lorenzo Eeman, founder of PROEMA. The doctrine is shared by a growing share of European premium media, FT, NYT, Le Monde, Reuters have publicly described comparable configurations in 2024-2025.

Traps to avoid

First trap: Bingbot crawl is shared with Copilot (Microsoft confirmed in 2024 that Copilot uses the Bing index in its backend), so blocking it kicks you out of Microsoft Search AND Copilot at once. Second trap: forgetting emerging user-agents (Applebot-Extended for Apple Intelligence, MistralAI-User for Le Chat, Meta-ExternalAgent for Meta AI), the 2026 list counts 24+ identifiable AI user-agents according to AppearOnAI and NoHacks tracking. Third trap: a malformed robots.txt (extra space after :, inconsistent casing on agent names), some strict crawlers silently ignore the directive. PROEMA ships the 2026 pattern in every Optim package, with monthly audit.