AgenBaseBot

Only when someone imports a website into their chatbot's knowledge, and then when that website is kept up to date (weekly or monthly, if they chose it). It never crawls the web on its own and never visits sites nobody imported.

Every request carries this user agent:

Mozilla/5.0 (compatible; AgenBaseBot/1.0; +agenbase.com/bot)

The product token is AgenBaseBot. Pages that only display with JavaScript may be loaded through a headless browser that sends the same user agent.

• Follows robots.txt: the rules for AgenBaseBot if you have any, otherwise those for *. It honors Crawl-delay (up to 5 seconds).
• Skips pages marked noindex, and pages such as carts, checkouts, logins and account areas.
• Reads only public pages (HTML, plain text and PDF) of the website that was imported.
• Makes a few requests at a time, slows down when asked (429 and Retry-After), and re-reads pages only to keep the owner's chatbot current.

If your site uses a firewall or bot protection, allow requests whose user agent contains AgenBaseBot. In Cloudflare: Security → Security rules (or WAF → Custom rules), create a rule where User Agent contains AgenBaseBot, with the action Skip.

Add this to your robots.txt:

User-agent: AgenBaseBot
Disallow: /

A page can also opt out with <meta name="AgenBaseBot" content="noindex">. Questions or concerns: write to us from the contact page.