Lemmy newb here, not sure if this is right for this /c.
An article I found from someone who hosts their own website and micro-social network, and their experience with web-scraping robots who refuse to respect robots.txt, and how they deal with them.
I found a very large botnet in Brazil mainly and several other countries. And abuseipdb.com is not marking those IPs are a thread. We need a better solution.
I think a honeypot is a good way. Another way is to use proof of work basically on the client side. Or we need a better place to share all stupid web scraping bot IPs.
I love the idea of abuseipdb and I even contributed to it briefly. Unfortunately, even as a contributor, I don’t get enough API resources to actually use it for my own purposes without having to pay. I think the problem is simply that if you created a good enough database of abusive IPs then you’d be overwhelmed in traffic trying to pull that data out.
Not really… We do have this wonderful list(s): https://github.com/firehol/blocklist-ipsets
And my firewall is using for example the Spamhaus drop list source: https://raw.githubusercontent.com/firehol/blocklist-ipsets/refs/heads/master/spamhaus_drop.netset
So I know its possible. And hosting in a git repo like that, will scale a lot. Since tons of people using this already that way.