ElamurBot

The web crawler of Elamur — a search engine in development.

What it is

Elamur is building a general web search engine at elamur.ai. ElamurBot visits publicly available web pages so that they can be found in our search results and in our open catalog of websites. The search engine is not public yet; the crawler is collecting and ranking pages for it now.

ElamurBot identifies itself with this User-Agent:

ElamurBot/1.0 (+https://bot.elamur.ai/)

How it crawls

It reads and follows robots.txt (RFC 9309). Its product token is ElamurBot.
It honours Crawl-delay. Small sites are visited slowly, with several seconds between requests; the pace adapts and backs off on errors, slow responses, HTTP 429 and 503.
It sends at most about one request per second to any single IP address.
It only fetches pages with plain HTTP(S) GET: no forms, no logins, no POST.
If your robots.txt closes your site to several other search crawlers by name, ElamurBot treats the site as closed too, even if it is not named.

How to block or allow it

To keep ElamurBot away from your whole site, add this to robots.txt:

User-agent: ElamurBot
Disallow: /

To invite it explicitly, even if you limit other crawlers:

User-agent: ElamurBot
Allow: /

Changes are picked up within a day. If you prefer, write to us and we will exclude your domains by hand.

Where it crawls from

178.162.229.202
2a00:c98:2050:a05f:4::112

Contact

Questions, complaints or requests to stop crawling: ai [at] elamur [dot] ai. We answer every message.