For portal operators

PratilipiBot

The crawler FIRSetu uses to keep a copy of published FIR listings. It runs slowly and stops when your server is having a bad day.

Operated byFIRSetu
Contacthello@firsetu.com
robots.txt tokenpratilipibot
Rate on Maharashtra PoliceOne request at a time, at least 3 seconds apart, and no more than 400 an hour.
Rate on Goa PoliceOne request at a time, at least 5 seconds apart, and no more than 60 an hour.
Rate on Delhi PoliceOne request at a time, at least 5 seconds apart, and no more than 150 an hour.
User-Agent sent to Maharashtra PoliceMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
User-Agent sent to Goa PoliceMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
User-Agent sent to Delhi PoliceMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Why it exists

FIRSetu is a search interface for First Information Reports that state police forces already publish themselves. We do not hold anything a portal has not put online, and we do not change what we find.

Those portals go down. When one does, every record we have already fetched keeps working and every new lookup fails, which is a poor experience for someone who needs a document today. PratilipiBot exists so that the records are already here when that happens. “Pratilipi” means copy.

What it fetches

Only the public listing pages a visitor would reach by using the portal’s own search form, and the FIR documents those listings link to. It submits the same form fields a browser does: a district, a police station, and a date range.

It does not attempt any page behind a login, does not submit data, and does not follow links outside the published-FIR section.

How it behaves

Requests go out one at a time, with no parallelism and no burst. A full pass over every station in a state takes hours by design, because there is nothing to gain from finishing sooner.

It watches how a portal is answering. If requests start failing, it slows to a third of its usual rate; if they keep failing, it stops entirely and does not try again until the portal is answering normally. It infers this from requests it has already made, so it never polls a struggling server to ask whether it has recovered.

If you would rather it did not

Write to hello@firsetu.com and tell us what you want changed — a slower rate, different hours, particular paths left alone, or no crawling at all. We will do it, and we will not ask you to justify it.

If you would rather not write to anyone, a Disallow rule for pratilipibot in your robots.txt is enough. We check for one and act on it.

More about what FIRSetu is and is not: About, Sources, and Disclaimer.