# fleetfax robots.txt
#
# Policy: fleetfax welcomes AI crawlers. The record is meant to be cited.
#
# fleetfax reads the public federal record for every US motor carrier and freight
# broker and publishes one plain carrier profile. A record nobody is allowed to
# read is not a record, so every AI crawler below is allowed the same access as
# any search engine: training crawlers, retrieval crawlers, and user-initiated
# fetches alike. No AI crawler is blocked. That is a deliberate decision with the
# tradeoffs weighed, not an oversight and not a default.
#
# Two things to know before editing:
#
#  1. robots.txt groups do NOT inherit. A bot named in its own group stops
#     reading the "*" group entirely. Every group below therefore repeats the
#     same body verbatim, and a test fails if the two bodies ever diverge.
#     Change one, change both.
#  2. The only Disallow is /feedback, a form with nothing to cite. Auth-gated
#     paths (/account, /login) are deliberately NOT disallowed here: they are
#     de-indexed by an X-Robots-Tag noindex response header, and a
#     crawler must be able to fetch a page to see that header. Disallowing them
#     would hide the noindex and could leave them indexed.
#
# User-agent tokens below were read from each operator's own published
# documentation on 2026-08-05. They change; re-verify before trusting this list.

User-agent: *
Allow: /
Disallow: /feedback

# AI crawlers: explicitly welcome.
# Same body as the "*" group above, repeated because groups do not inherit.

# OpenAI
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
# Anthropic. anthropic-ai and Claude-Web are RETIRED tokens, kept so an older
# deployment still reads a yes; ClaudeBot, Claude-User and Claude-SearchBot are
# the current three.
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: Claude-Web
# Perplexity. Perplexity-User is documented as generally ignoring robots.txt
# because a person drove the request; listed anyway, so our answer is on record.
User-agent: PerplexityBot
User-agent: Perplexity-User
# Google. Google-Extended is a permission TOKEN, not a crawler: it fetches
# nothing, and its only meaning is whether crawled content may train Gemini.
# Allowing it is our stated yes. Do not delete it as a dead entry.
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
# Apple. Applebot-Extended is likewise a permission token, not a fetcher.
User-agent: Applebot
User-agent: Applebot-Extended
# Common Crawl, the corpus behind much open model training.
User-agent: CCBot
# Meta
User-agent: Meta-ExternalAgent
User-agent: meta-externalagent
User-agent: FacebookBot
# Amazon
User-agent: Amazonbot
# Others
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: MistralAI-User
User-agent: Bytespider
User-agent: DuckAssistBot
User-agent: YouBot
User-agent: Diffbot
User-agent: Timpibot
Allow: /
Disallow: /feedback

# Where to start.
# llms.txt is the index; llms-full.txt is the grounding document, carrying the
# full signal vocabulary and the rules fleetfax phrases its findings under.
# The research feed is fleetfax's own analyses of the federal datasets.
# LLM-Content: https://www.fleetfax.com/llms.txt
# LLM-Full-Content: https://www.fleetfax.com/llms-full.txt
# Feed: https://www.fleetfax.com/research/feed.xml

Sitemap: https://www.fleetfax.com/sitemap.xml
Sitemap: https://www.fleetfax.com/carrier/sitemap-index.xml
Sitemap: https://www.fleetfax.com/carrier/directory-sitemap.xml