PlantelBot
The web crawler that builds the Plantel Index — the search index Plantel is building for search in the Plantel Browser.
How to recognise it
PlantelBot sends this User-Agent with every request:
PlantelBot/1.0 (+https://plantel.ai/bot)
It is operated by Plantel Inc. It does not hide behind another browser's User-Agent. We do not publish a list of IP addresses for it yet.
What it fetches
- Your
robots.txtfirst, before anything else on your site. - The sitemaps your
robots.txtlists, and public HTML pages linked from pages it has already read. - Only
httpandhttpson the standard ports. Only HTML pages — no images, video, audio, PDFs or downloads. At most 2 MB of any one response. - It does not run JavaScript, sign in, fill in forms or follow links marked
rel="nofollow",rel="ugc"orrel="sponsored". - It follows at most five redirects in a row.
From each page the Plantel Index keeps the address, title, meta description, language, the date it was read, up to 10,000 characters of its visible text, and — when the page gives them — its canonical address, icon link and publication date. Pages are read again from time to time, between once a day and once every 60 days, depending on how often they change.
How fast it goes
- At most one request to a site every 10 seconds.
- Slower if your
robots.txtsets aCrawl-delay(in seconds). - If your site answers
429 Too Many Requestsor503 Service Unavailable, it stops visiting for the time given inRetry-After, and for at least 10 minutes.
The robots.txt rules it follows
PlantelBot follows the Robots Exclusion Protocol (RFC 9309):
- It obeys the group whose
User-agentline namesPlantelBot(in any letter case). If there is none, it obeys theUser-agent: *group. AllowandDisallowrules support*(any characters) and a trailing$(end of the address). The longest matching rule wins; if an Allow and a Disallow rule are equally long, Allow wins.Crawl-delayandSitemaplines are honoured.- It reads
robots.txtagain at least once every 24 hours while it is visiting your site. - If
robots.txtanswers with a server error (5xx) or 429, or cannot be reached, PlantelBot fetches nothing else from the site and triesrobots.txtagain an hour later. Ifrobots.txtdoes not exist (404 or another 4xx), there are no restrictions.
How to slow it down or opt out
To keep PlantelBot off your whole site:
User-agent: PlantelBot
Disallow: /
To keep it out of part of your site:
User-agent: PlantelBot
Disallow: /private/
To make it visit less often (here, one request every 30 seconds at most):
User-agent: PlantelBot
Crawl-delay: 30
To keep a single page out of the Plantel Index, use a robots meta tag or header. PlantelBot honours noindex, nofollow and none in any of these:
<meta name="robots" content="noindex">
<meta name="plantelbot" content="noindex">
X-Robots-Tag: noindex
When PlantelBot next reads your robots.txt, pages that a new Disallow rule covers are removed from the Plantel Index. A page with noindex is removed the next time PlantelBot reads it.
Contact
If PlantelBot is causing a problem on your site, or you need a page removed quickly, write to [email protected]. Please include your site's address and, if you can, the time and path of the requests.
Last updated 2026-11-16.