I’m trying to understand the bot problem in the internet and finding more ways to defend myself. One thing that I can’t seem to understand is why most bots, scrapers and crawlers seem to have residential IPs.
- Is it that ISPs are being paid by tech-bros to assign them these IPs?
- Is it that residential devices have been hacked /contain malware that does this?
- Is it trivial for companies to assign themselves residential IPs?
- Paid volunteers are doing this for AI companies?
Or is there is some other reason for this?
Obviously this is a problem because one can rotate / cycle through residential IPs and if I aggressively block each offender in my logs permanently, then the next person assigned this IP who may be a legitimate user will be unable to access my site.
if I aggressively block each offender in my logs permanently, then the next person assigned this IP who may be a legitimate user will be unable to access my site.
temp bans exist for this reason. You can use something like fail2ban for it, or that may be overkill for your purposes, but any mechanism that blocks the IP address for a short amount of time will work. My f2b blocks spammers’ IP addresses for a day, and I don’t see repeat bans which means the spammers aren’t coming back on the same IP address, so the short ban works to stop a given spam attack.
I worked as a network analyst for a provider for several years and during that time I’d say ~90% of the issue stemmed from sketchy apps/services that the user loaded from their end.
A lot of “free” VPN services will basically allow bad actors (the paid tier) to use your connection. A lot of IoT devices are also just openly available on the Internet to route through.
From the ISP perspective, we managed the roads, not your car. There is a push to blame the ISP as it’s their network, but realistically how are they meant to provide security (in the context that is being asked) to any device that gets plugged into that network? We even had business customers demand we add clauses to contracts where we would accept responsibility for any malware they sent between sites over an MPLS setup.
In the end, a lot of people seem to want this impossible scenario of the ISP managing security for them but also not inspecting their traffic.
@Mordikan @Maroon A lot of people _don’t know what their ISP does_. Many seem to think that the ISP is selling them the entire internet as a product, and so from that logic why shouldn’t the ISP be liable for whatever mayhem they get into online?
Source: worked for a little while as dial-up ISP support.
Wouldn’t even mind the option to let someone else use my connection a bit for a free VPN tbh, that is no worse than running a TOR node. What I dislike is the dishonesty side of it. Be open and honest then it’s all good.
you should not run a Tor exit relay from your home [[1]](https://community.torproject.org/relay/types-of-relays/)
<.< Let me just throw two scenarios at you real quick to show you why that’s a problem:
That third party can easily just use your connection to view and download CSAM and all external monitoring would suggest that activity was coming from within your network. Because it is.
That third party has full network access to anything your PC does (or at least that nic) which includes anything in your local network. Insecure IoT, other PCs, etc.
First scenario is no different to running a TOR exit node. The second is why it needs to be built securely, which can be done though probably isn’t in these cases.
It would be like running TOR, but not a relay, it would be like an exit node.
That should be enough to warn anyone away from using them.
Running an exit node is perfectly legal though. There would be no evidence you have done anything wrong very quickly.
You can just look at the testimonies from others who have run exit nodes. The cost of your “free” VPN is that law enforcement will constantly be in contact and investigating you because your network/machine is being used to download CSAM.
There is no “oh don’t worry, A.B.C.D is just a tor node, we can give it a pass”. Every time that happens, it has to be treated with a full investigation.
Let them waste their time investigating, actually how do they even know your address?
Ok, this I can answer personally as we did multiple cases of this happening (CSAM, bomb threats, etc) at work.
So, anonymity on the Internet is not actually a thing. Whether its an IP address or telecom switch or whatever, there is a path back to you even if only for either billing or connectivity purposes. So, for IP, we would receive a subpoena signed by a judge to hand over any and all information regarding the identify of the a given IP address (they include a long list of things whether applicable or not in the order so every potential base is covered). Once legal was able to review and handed it off to us, we take that and look at the DHCP logs to see that on a given date at a given time that the IP address was assigned as part of shelf A / slot B / port C. That shelf/slot/port combination is tied physically to an address/account. We provide the relevant logs and personal information of that user to law enforcement.
For bomb threats over the phone, telecom switches love to tell every other telecom switch who they are (again, connectivity purposes). So, when you make a call to a business/school doing that, their PBX is going to log to the millisecond when that call occurred and who the switch was. Again, subpoena and we pull the SIP logs. We can even provide the RTP/RTCP packets and reconstruct the phone call audio if the subpoena asks for that.
I can pay for internet in cash and the only details I gave them are fake and a random username. 4G internet isn’t even tied to the same location as I can move freely.
As I’m not bothered about doing cybercrimes I don’t bother doing that much about security though.
Residential routers have crappy security 'cos no-one bothers to punish ISPs for crap security practices.
Also some free VPNs come at a malware cost.
You mention crappy security practices from the ISP but then mention the user’s action (installing “free” VPNs). Why is the ISP on the hook for the user making terrible decisions?
What is the correct security practice in that instance? Fire the customer for being an idiot? Maybe just DENY IP ANY ANY on outbound traffic?
How do you protect somebody who’s intent on running themselves off a cliff?
All of the above.
Is it that ISPs are being paid by tech-bros to assign them these IPs?
Bullet Proof Hosting is a thing. Some ISPs basically advertise to criminals about their ability to evade take down orders and unwillingness to work with law enforcement. So, some infrastructure ends up on these devices. However, the IP ranges from these services often get discovered and are added to public reputation and block lists.
Along side this, cloud providers are pretty bad about policing their networks. On my own home server, I have blocked much of the Digital Ocean IP space, as it’s home to a lot of scanners, bots and other malicious traffic.
Is it that residential devices have been hacked /contain malware that does this?
This happens, a lot. The Mirai Botnet thrived on compromised home routers. People are pretty bad at updating their devices and many SOHO routers ship with some pretty bad vulnerabilities. It’s only a matter of time until someone finds an unpatched or misconfigured router and adds it to a botnet. People also get phished or install trojans all the time, adding to botnets. Darknet Diaries just had a fantastic episode on the Bayrob malware, part of which was turning infected machines into a custom botnet.
Is it trivial for companies to assign themselves residential IPs?
Some ISPs just look the other way when they get reports of malicious activity on their network. Also, attackers can force a DHCP refresh and just get a new IP when the old one seems blocked. Getting one in the first place is often as simple as signing up for service and/or compromising someone’s home PC and using it as a relay.
Paid volunteers are doing this for AI companies?
This probably happens. Afterall, we’ve already seen a company selling an AI product which was just workers in India.
Obviously this is a problem because one can rotate / cycle through residential IPs and if I aggressively block each offender in my logs permanently, then the next person assigned this IP who may be a legitimate user will be unable to access my site.
Look into Fail2Ban. This program monitors your logs and will ban IPs automatically based on criteria you set. This can include specific HTTP requests in your web logs. The ban can be permanent or can be time limited. For example, I have a container running in a cloud provider which I use to proxy requests through my ISP’s CGNAT setup. There is an NGinx reverse proxy running there and I have fail2ban watching the access log. If certain request strings are seen, the sending IP gets dumped in a permanent jail. I also have it scanning the sshd logs and banning IPs which fail to login 3 times within a short period.
It’s far from a silver bullet, but it’s something which should be running on any web facing system. Attackers will always be rattling the door knobs. There is no reason to let them keep rattling away.
I have blocked much of the Digital Ocean IP space
Over the years, I have blocked the entire probably Digital Ocean spectrum, a ton of Linode, clients.your-server.de, a long string of them. It used to make me paranoid I was setting up something wrong. I’d fire up a brand new, untouched server, and boom! Here they’d come. Then I thought, maybe it was a few Docker containers pushing analytics to a collector. Upon blocking them tho, nothing seemed to malfunction. So instead of trying mitigate each and every offending IP, I switched to a deny all until something complains posture, and block by ip ranges. I can still see them trying to gain access, but as long as I can keep them out on the edge, everything is golden.
Some ISPs basically advertise to criminals about their ability to evade take down orders and unwillingness to work with law enforcement.
There are Russian ISPs that advertise VPS services and such by the hour. I’m sure there are other ISPs in other countries that do the same. Nothing good could be coming out of that. When you consider that the internet gobbles through 14,000,000+/- petabytes per 24 hours, and 40%+/- is bot traffic, the picture gets a bit clearer, because that’s a shit load of bots, and they are sophisticated bots at that.
Yeah, a long time ago I started blocking outbound traffic to all but our laptops and it’s interesting how many things work perfectly well, but continually try to get out… the SolarPv inverter is VERY persistent to get to China due to an overly helpful installer setting it up on the wifi.
I now have a firewall alias to only allow certain devices out and then just update that alias, do whatever (usually updates) and then revert the alias.
There was an article here about one of the companies doing this, basically one of the methods is having apps on people’s smart TVs or phones that scrape the web in the background and upload to a control server. So it’s a person’s genuine device in a genuine residence in some cases.
https://thehackernews.com/2026/06/free-apps-are-quietly-turning-smart-tvs.html
Interesting article.
The company’s CEO, Or Lenchner, said a device in its network “is a device whose owner said yes, understood what they were saying yes to, and can say no again at any moment with two steps.”
That’s part of the problem. Most casual users of technology don’t know what they are saying yes to. They just want to do face swaps on sketchy apps. Even as a ‘more aware’ user, I have always considered my phone and TV to be the weakest links in my network.
I remember just recently a post about a “smart” washing machine creating over 3GB of Internet traffic per day.
Iirc that was a math error on the poster’s part.
Not the ISPs problem at all.
Blame IoT manufacturers for providing shit devices that can easily be hacked.
Theere is no difference from a technical standpoint between a business and residential IP. It’s allocation is tied to the network provider.
People install extensions onto browsers for any number of reasons without knowing what it does. It was a problem is the 90s that never went away.
As often as im Blocked as supposed bot/crawler, have you given tough to misidentifiying users as bots?
There are clear signs like massive ping rates, downloads, etc. Anubis helps block a lot of them, btut more turn up all the time.
