AgudotBot

AgudotBot is the crawler of the Agudot search engine. It reads public pages so people can find them, with a focus on Israeli sites in Hebrew and English.

How to recognise it

Every request carries this User-Agent:

AgudotBot/1.0 (+https://agudot.com/bot)

How it behaves

• Obeys robots.txt (RFC 9309), including Allow, Disallow and wildcards. Rules are re-read daily.
• Obeys Crawl-delay. Without one it waits at least one second between two requests to the same site.
• Sends one request at a time per site, and backs off automatically on 429 or 503.
• Honours noindex and nofollow in meta robots or a meta tag named agudotbot, and rel="nofollow" on links.
• Downloads at most 2MB per page and stores text only. It runs no JavaScript and submits no forms.
• Stays out of logins, shopping carts and internal search pages.

How to block or slow it down

To block it from the whole site, add to robots.txt:

User-agent: AgudotBot
Disallow: /

To slow it down or block part of the site:

User-agent: AgudotBot
Crawl-delay: 10
Disallow: /private/

To keep a single page out of the results:

<meta name="agudotbot" content="noindex">

robots.txt changes are picked up within 24 hours. Blocked or noindex pages leave the index on the next visit.