nnov.ec-bot — the encyclopedia’s crawler

Who we are. The Nizhny Novgorod Encyclopedia is a non-commercial project about the city and its region. The crawler reads publications by local outlets in order to put a fact into an article with a link to its source: the outlet’s name, the headline and a direct link. We do not republish full texts and never show someone else’s material instead of the original — on the contrary, we send the reader to you.

How it behaves

• identifies itself in the User-Agent: nnov.ec-bot/1.0 (+https://www.nnov.ec/about-bot);
• reads and respects robots.txt;
• keeps a pause of at least 2.5 seconds between requests to the same site — and that pause is shared by all our processes, so there are never bursts;
• caches what it has read for a week and does not request the same page again;
• on a 429, 503 or 403 it stops visiting the site for at least an hour — automatically, without argument.

If the crawler is in your way

The fastest way is to block it in robots.txt:

User-agent: nnov.ec-bot
Disallow: /

It will stop coming within a day (that is how long our robots.txt cache lives). A softer option is available in the same file: close individual sections or set a Crawl-delay — the crawler will take your pause if it is longer than ours.

If you want the opposite

If it matters to your outlet that events reach the city chronicle faster and more accurately, point us to your feed the usual way — a Sitemap: line in robots.txt or a <link rel="alternate" type="application/rss+xml"> tag in your page head. The crawler will find it and take data from there instead of walking your pages: less load on you and fewer mistakes on our side.