PratilipiBot

FIRSetu is a search interface for First Information Reports that state police forces already publish themselves. We do not hold anything a portal has not put online, and we do not change what we find.

Those portals go down. When one does, every record we have already fetched keeps working and every new lookup fails, which is a poor experience for someone who needs a document today. PratilipiBot exists so that the records are already here when that happens. “Pratilipi” means copy.

Only the public listing pages a visitor would reach by using the portal’s own search form, and the FIR documents those listings link to. It submits the same form fields a browser does: a district, a police station, and a date range.

It does not attempt any page behind a login, does not submit data, and does not follow links outside the published-FIR section.

Requests go out one at a time, capped at roughly ten a minute. There is no parallelism and no burst: a full pass over every station in a state takes hours by design, because there is nothing to gain from finishing sooner.

It watches how a portal is answering. If requests start failing, it slows to a third of its usual rate; if they keep failing, it stops entirely and does not try again until the portal is answering normally. It infers this from requests it has already made, so it never polls a struggling server to ask whether it has recovered.

Write to hello@firsetu.com and tell us what you want changed — a slower rate, different hours, particular paths left alone, or no crawling at all. We will do it, and we will not ask you to justify it.

If you would rather not write to anyone, a Disallow rule for PratilipiBot in your robots.txt is enough. We check for one and act on it.

More about what FIRSetu is and is not: About, Sources, and Disclaimer.