PilgudBrainBot

How to recognise it

It identifies itself in every request with this user agent:

PilgudBrainBot/0.1 (+https://pilgud.com/bot)

Some pages that need a browser to render are read through Firecrawl, a scraping service we use; those requests carry Firecrawl's own user agent and follow the same rules below.

What it reads – and what it does not

• Public pages only. It never logs in, never uses someone else's account and never gets around a paywall; a paywalled page gives it the teaser and nothing more.
• Marketing knowledge, not people. It is looking for claims about marketing – what works, what the rules are, what the numbers say. Collecting personal data is not its purpose, and personal names are not stored in what it learns.
• Not our clients' websites. Our clients' own sites are read only when a client asks us to, through a separate path, and never enter the shared knowledge base.
• Not Meta's websites. Facebook and Instagram pages are not read automatically at all.

Your opt-out is respected

• robots.txt, always. It reads your robots.txt before fetching anything and follows it – for User-agent: PilgudBrainBot and for User-agent: *. It keeps a copy for up to 24 hours, so a change reaches it within a day. This is not a setting we can switch off.
• Text-and-data-mining reservations. EU copyright law (Article 4 of the DSM Directive) allows commercial text and data mining of lawfully accessible content unless the rightsholder has reserved it in a machine-readable way. We treat a robots.txt disallow and a TDMRep reservation (/.well-known/tdmrep.json) as such a reservation, and then we do not mine the page.

To keep PilgudBrainBot off your whole site, add this to your robots.txt:

How fast it goes

At most one or two requests to your site at a time, with at least a second between them on our own fetcher. A small site will hardly notice it; if yours does, write to us and we will slow down or stop.

What we keep

• Full text of a page is kept only as long as needed for mining: while the source is in our list, plus 90 days. Pages under an open licence (for example public statistics) may be kept longer.
• Short quotes – a sentence or two, at most about 300 characters – are kept with a link to the page, so that anything the brain concludes can be traced back to where it came from.

Contact

Something wrong, or want your site excluded without editing robots.txt? Write to info@pilgud.com. We add your domain to the list PilgudBrainBot never fetches and delete the full text we keep from your pages. Pilgud is run by R.J. RADIANT OÜ, Estonia.