Skip to content
PLANTELBOT

PlantelBot

The web crawler that builds the Plantel Index — the search index Plantel is building for search in the Plantel Browser.

How to recognise it

PlantelBot sends this User-Agent with every request:

PlantelBot/1.0 (+https://plantel.ai/bot)

It is operated by Plantel Inc. It does not hide behind another browser's User-Agent. We do not publish a list of IP addresses for it yet.

What it fetches

  • Your robots.txt first, before anything else on your site.
  • The sitemaps your robots.txt lists, and public HTML pages linked from pages it has already read.
  • Only http and https on the standard ports. Only HTML pages — no images, video, audio, PDFs or downloads. At most 2 MB of any one response.
  • It does not run JavaScript, sign in, fill in forms or follow links marked rel="nofollow", rel="ugc" or rel="sponsored".
  • It follows at most five redirects in a row.

From each page the Plantel Index keeps the address, title, meta description, language, the date it was read, up to 10,000 characters of its visible text, and — when the page gives them — its canonical address, icon link and publication date. Pages are read again from time to time, between once a day and once every 60 days, depending on how often they change.

How fast it goes

  • At most one request to a site every 10 seconds.
  • Slower if your robots.txt sets a Crawl-delay (in seconds).
  • If your site answers 429 Too Many Requests or 503 Service Unavailable, it stops visiting for the time given in Retry-After, and for at least 10 minutes.

The robots.txt rules it follows

PlantelBot follows the Robots Exclusion Protocol (RFC 9309):

  • It obeys the group whose User-agent line names PlantelBot (in any letter case). If there is none, it obeys the User-agent: * group.
  • Allow and Disallow rules support * (any characters) and a trailing $ (end of the address). The longest matching rule wins; if an Allow and a Disallow rule are equally long, Allow wins.
  • Crawl-delay and Sitemap lines are honoured.
  • It reads robots.txt again at least once every 24 hours while it is visiting your site.
  • If robots.txt answers with a server error (5xx) or 429, or cannot be reached, PlantelBot fetches nothing else from the site and tries robots.txt again an hour later. If robots.txt does not exist (404 or another 4xx), there are no restrictions.

How to slow it down or opt out

To keep PlantelBot off your whole site:

User-agent: PlantelBot
Disallow: /

To keep it out of part of your site:

User-agent: PlantelBot
Disallow: /private/

To make it visit less often (here, one request every 30 seconds at most):

User-agent: PlantelBot
Crawl-delay: 30

To keep a single page out of the Plantel Index, use a robots meta tag or header. PlantelBot honours noindex, nofollow and none in any of these:

<meta name="robots" content="noindex">
<meta name="plantelbot" content="noindex">
X-Robots-Tag: noindex

When PlantelBot next reads your robots.txt, pages that a new Disallow rule covers are removed from the Plantel Index. A page with noindex is removed the next time PlantelBot reads it.

Contact

If PlantelBot is causing a problem on your site, or you need a page removed quickly, write to [email protected]. Please include your site's address and, if you can, the time and path of the requests.

Last updated 2026-11-16.