For portal operators
PratilipiBot
The crawler FIRSetu uses to keep a copy of published FIR listings. It runs slowly and stops when your server is having a bad day.
Why it exists
FIRSetu is a search interface for First Information Reports that state police forces already publish themselves. We do not hold anything a portal has not put online, and we do not change what we find.
Those portals go down. When one does, every record we have already fetched keeps working and every new lookup fails, which is a poor experience for someone who needs a document today. PratilipiBot exists so that the records are already here when that happens. “Pratilipi” means copy.
What it fetches
Only the public listing pages a visitor would reach by using the portal’s own search form, and the FIR documents those listings link to. It submits the same form fields a browser does: a district, a police station, and a date range.
It does not attempt any page behind a login, does not submit data, and does not follow links outside the published-FIR section.
How it behaves
Requests go out one at a time, with no parallelism and no burst. A full pass over every station in a state takes hours by design, because there is nothing to gain from finishing sooner.
It watches how a portal is answering. If requests start failing, it slows to a third of its usual rate; if they keep failing, it stops entirely and does not try again until the portal is answering normally. It infers this from requests it has already made, so it never polls a struggling server to ask whether it has recovered.
If you would rather it did not
Write to hello@firsetu.com and tell us what you want changed — a slower rate, different hours, particular paths left alone, or no crawling at all. We will do it, and we will not ask you to justify it.
If you would rather not write to anyone, a Disallow rule for pratilipibot in your robots.txt is enough. We check for one and act on it.
More about what FIRSetu is and is not: About, Sources, and Disclaimer.