Many are "compensated" (in the way of software they didn't pay for), so the real question is that of disclosure (in which case many software vendors check the box in the most minimal way possible by including it as fine print during the install)
No, the question is not just disclosure. People have their bandwidth stolen, and sometimes internet access revoked due to this kind of fraud and misuse - disclosure wouldn’t solve that
Also, as a website owner, these residential proxies are a real pain. Tons and tons of abusive traffic, including people trying to exploit vulnerabilities and patently broken crawlers that send insane numbers of requests, and no real way to block it.
It's just nasty stuff. Intent matters, and if you're selling a service that's used only by the bad guys, you're a bad guy too. This is not some dual-use, maybe-we-should-accept-the-risks deal that you have with Tor.
Getting rid of malware is good. A private for-profit company exercising its power over the Internet, not so much. We should have appropriate organizations for this.
The proxies is the reason why you get spam in your Google search result, spam in your Play store (by means of fake good reviews), basically spam in anything user generated.
It directly affects Google and you, I don’t see why they should not do this.
Spam in Google search results is due to Google happily taking money from the spammers in exchange for promoting their spam, or that the spam sites benefit Google indirectly by embedding Google Ads/Analytics.
I don't see any spam in Kagi, so clearly there is a way to detect and filter it out. Google is simply not doing so because it would cut into their profits.
> These efforts to help keep the broader digital ecosystem safe supplement the protections we have to safeguard Android users on certified devices. We ensured Google Play Protect, Android’s built-in security protection, automatically warns users and removes applications known to incorporate IPIDEA SDKs, and blocks any future install attempts.
Nice to see Google Play Protect actually serving a purpose for once.
If I m not mistaken, the plaintiffs in the US v Google antitrust case in the DC Circuit tried to argue that website operators are biased toward allowing Google to crawl and against allowing other search engines to do the same
The Court rejected this argument because the plaintiffs did not present any evidence to support it
For someone who does not follow the web's history, how would one produce direct evidence that the bias exists
LLMs aren't a good indicator of success here because an LLM trained on 80% of the data is just as good as one trained on 100%, assuming the type/category of data is distributed evenly. Proxies help when you do need to get access to 100% of the data.
Yup exactly. Google must be the only one allowed to scrape the web. Google can't have any other competition. Calling it in "user's best interest" is just like their other marketing cons: "play integrity for user's security" etc
Have you got any proof of Google scraping from residential proxies users don't know about, rather than from their clearly labelled AS? Otherwise you're mixing entirely different things into one claim.
That's the whole point. Websites that try to block scraping attempts will let google scrape without any hurdle because of google's ads and search network. This gives google some advantage over new players because as a new name brand you are hardly going to convince a website to allow scraping even if your product may actually be more advantageous to the website (for example assume you made a search engine that doesn't suck like google, and aggregates links instead of copying content from your website).
Proxies in comparison can allow new players to have some playing chance. That said I doubt any legitimate & ethical business would use proxies.
I don't think parent post is claiming that Google is using other people's networks to scrape the web only that they have a strong incentive to keep other players from doing that.
No, there are other scrapers that Google doesn't block or interact with. You can even run scraping from GCP. This has nothing to do with "only Google is allowed to scrape".
They even host apps which exist for scraping data, like https://play.google.com/store/apps/details?id=com.sociallead...
Play Protect blocks malicious apps, not network traffic, so no, it obviously doesn't interfere with Google's apps.
AFAIK it also left SmartTube (an alternative YouTube client) alone until the developer got pwned and the app trojanized with this kind of SDK, and the clean versions are AFAIK again being left alone. No guarantee that it won't change in the future, of course, but so far they seem to not be abusing it.
My understanding is that routing through residential IPs is a part of the business of some VPN providers. I don't know how above board they are on this (as in notifying customers that this may happen, however buried in the usage agreement, or even allowing them to opt out).
But, my main point, is that the whole business is "on the up and up" vs some dark botnet.
> While operators of residential proxies often extol the privacy and freedom of expression benefits of residential proxies, Google Threat Intelligence Group’s (GTIG) research shows that these proxies are overwhelmingly misused by bad actors
They have a robust KYC that appears to serve, at least in large part, as a way to stay off the shit list of companies with the resources to pursue recourse.
Source: went through that process, ended up going a different route. The rep was refreshingly transparent about where they get the data, why the have the kyc process (aside from regulatory compliance).
Ended up going with a different provider who has been cheaper and very reliable, so no complaints.
Yeah, they make you do a Skype interview (or probably Zoom interview nowadays). You could call this KYC or collateral, depending on your view of the company. It does limit the nefariousness of their clientele but I doubt they do much, or any, monitoring of actual traffic after onboarding (not for compliance reasons, anyway).
I've had enough of companies saying "you're connecting from an AWS IP address, therefore you aren't allowed in, or must buy enterprise licensing". Reddit is an example which totally blocks all data to non-residential IP's.
I want exactly the same content visible no matter who you are or where you are connecting from, and a robust network of residential proxies is a stepping stone to achieving that.
If you look at the article, the network they disrupted pays software vendors per-download to sneakily turn their users into residential proxy endpoints. I'm sure that at least some of the time the user is technically agreeing to some wording buried in the ToS saying they consent to this, but it's certainly unethical. I wouldn't want to proxy traffic from random people through my home network, that's how you get legal threats from media companies or the police called to your house.
> that's how you get legal threats from media companies or the police called to your house.
Or residential proxies get so widespread that almost every house has a proxy in, and it becomes the new way the internet works - "for privacy, your data has been routed through someone else's connection at random".
> Or residential proxies get so widespread that almost every house has a proxy in, and it becomes the new way the internet works - "for privacy, your data has been routed through someone else's connection at random".
Of course they're pitching it like everything's above board, but from the article:
> While many residential proxy providers state that they source their IP addresses ethically, our analysis shows these claims are often incorrect or overstated. Many of the malicious applications we analyzed in our investigation did not disclose that they enrolled devices into the IPIDEA proxy network. Researchers have previously found uncertified and off-brand Android Open Source Project devices, such as television set top boxes, with hidden residential proxy payloads.
I live in the UK and can't view a large portion of the internet without having to submit my ID to _every_ site serving anything deemed "not safe the for the children". I had a question about a new piercing and couldn't get info on it from Reddit because of that. I try using a VPN and they're blocked too. Luckily, I work at a copmany selling proxies so I've got free proxies whenever I want, but I shouldn't _need_ to use them.
I find it funny that companies like Reddit, who make their money entirely from content produced by users for free (which is also often sourced from other parts of the internet without permission), are so against their site being scraped that they have to objectively ruin the site for everyone using it. See the API changes and killing off of third party apps.
Obviously, it's mostly for advertising purposes, but they love to talk about the load scraping puts on their site, even suing AI companies and SerpApi for it. If it's truly that bad, just offer a free API for the scrapers to use - or even an API that works out just slightly cheaper than using proxies...
My ideal internet would look something like that, all content free and accessible to everyone.
> that they have to objectively ruin the site for everyone using it. See the API changes and killing off of third party apps.
Third party app users were a very small but vocal minority. The API changes didn't drop their traffic at all. In fact, it's only gone up since then.
The datacenter IP address blocks aren't just for scrapers, it's an anti-bot measure across the board. I don't spend much time on Reddit but even the few subreddits I visited were starting to become infiltrated by obvious bot accounts doing weird karma farming operations.
Even HN routinely gets AI posting bots. It's a common technique to generate upvote rings - Make the accounts post comments so they look real enough, have the bots randomly upvote things to hide activity, and then when someone buys upvotes you have a selection of the puppet accounts upvote the targeted story. Having a lot of IP addresses and generating fake activity is key to making this work, so there's a lot of incentive to do it.
I agree that write-actions should be protected, especially now when every other person online is a bot. As for read-actions, I'll continue to profit off those being protected too but I wouldn't be too bothered if something suddenly changed and all content across the internet was a lot easier to access programmatically. I think only harm can come from that data being restricted to the huge (nefarious) companies that can pay for that data or negotiate backroom deals.
Have you considered that it’s because a new industry popped up that decided it was okay to slurp up the entire internet, repackage it, and resell it? Surely that couldn’t be why sites are trying to keep non humans out.
> I want exactly the same content visible no matter who you are or where you are connecting from
The reason those IP addresses get blocked is not because of "who" is connecting, but "what"
Traffic from datacenter address ranges to sites like Reddit is almost entirely bots and scrapers. They can put a tremendous load on your site because many will try to run their queries as fast as they can with as many IPs as they can get.
Blocking these IP addresses catches a few false positives, but it's an easy step to make botting and scraping a little more expensive. Residential proxies aren't all that expensive, but now there's a little line item bill that comes with their request volume that makes them think twice.
> We need more residential proxies, not less
Great, you can always volunteer your home IP address as a start. There are services that will pay you a nominal amount for it, even.
That's already the case (irrespective of residential proxies) because content only serves as bait for someone to hand over personal information (during signup/login) and then engage with ads.
Proxies actually help with that by facilitating mass account registration and scraping of the content without wasting a human's time "engaging" with ads.
Amazon.com now only shows you a few reviews. To see the rest you must login. Social media websites have long gated the carrots behind a login. Anandtech just took their ball and went home by going offline.
There's a company that pays you to keep their box connected to your residential router. I assume it sells residential proxy services, maybe also DDoS services, I don't know. It's aptly named Absurd Computing.
Agreed. With things people paid for and using our wifi data to build their "positioning dbs" that you can't block or turn off on your phone, without "rooting" your own device.
I don’t know. I wouldn’t have thought of myself as proxying other people’s traffic by carrying my iPhone around. (For one thing, it’s my own phone that initiates all the activity- it monitors for Apple devices, the devices don’t reach out to my phone.) I can see how you could frame it that way, though. I just thought they might be referring to something else that I didn’t know about.
I remain skeptical. I can understand how one would might see it that way, but I think it’s stretching the word proxy too far.
Devices on Apple’s Find My aren’t broadcasting anything like packets that get forwarded to a destination of their choosing. I would think that would be a necessity to call it “proxying”.
They’re just broadcasting basic information about themselves into the void. The phones report back what they’ve picked up.
That doesn’t fit the definition to me.
I absolutely don’t mind the fact that my phone is doing that. The amount of data is ridiculously minuscule. And it’s sort of a tit for tat thing. Yeah my phone does it, but so does theirs. So just like I may be helping you locate your AirTag, you would be helping me locate mine. Or any other device I own that shows up on Find My.
It’s a very close to a classic public good, with the only restriction being that you own a relevant device.
Have you tried it? Every new account will be shadowbanned and if it's shared you often get blank page 429. None of this was true before the API shutdown.
That’s not my experience, using various VPNs, public networks, Cloudflare and Apple private relays. A captcha is common when logged out but that’s about it, I have not encountered any shadow bans. I create a new account each week.
That's not the same as "blocks all data to non-residential IP's"?
>if it's shared you often get blank page 429. None of this was true before the API shutdown.
See my other comment. I agree there's a non-zero amount of VPNs that are banned from reddit, but it's also not particularly hard to find a VPN that's not banned on reddit.
Private VPS for personal VPN in Netherlands (digital ocean), then Hungary (some small local DC) — both are blocked from day one.
> You've been blocked by network security. To continue, log in to your Reddit account or use your developer token. If you think you've been blocked by mistake, file a ticket below and we'll look into it.
Proton VPN sometimes (mostly?) has this issue too. It's a bit of an hit or miss in there iirc but I have definitely seen the last message of your comment.
That's just mullvad's IP pool being banned. The other VPN providers I use aren't banned, or at least are only intermittently banned that I can easily switch to another server.
I have never interacted with a reddit employee who wasn't actively gaslighting me about the platform. Do you even use the site? I talked to a PM recently who genuinely thought the phone app was something people liked.
everything on Reddit is so locked down it’s useless. even if you do get to post something useful some basement dwelling mod will block it for an arcane interpretation of one of the subreddits 14 rules.
This blog post from the company that used promise "don't be evil", one that steals water for data centers from vilages and towns via shady deals, whose whole premise it stealing other people's stuff and claiming it as their own and locking them out and selling their data.. Who made them the arbiter of the internet? No one!!!
They just stole this and get on their high horse to tell people how to use internet? You can eff right off Google.
I still "run" a small ISP with a few thousand residential ips from my scraping days. The requirements are laughable and costs were negligible in the early 2000s.
I haven't looked at any court documents, but the WSJ article from Wednesday reported that "Last year, Google sued the anonymous operators of a network of more than 10 million internet-connected televisions, tablets and projectors, saying they had secretly pre-installed residential proxy software on them... an Ipidea spokeswoman acknowledged in an email that the company and its partners had engaged in “relatively aggressive market expansion strategies” and “conducted promotional activities in inappropriate venues (e.g., hacker forums)...”"
There was also a botnet, Kimwolf, that apparently leveraged an exploit to use the residential proxy service, so it may be related to Ipidea not shutting them down.
I'm actually a little shocked seeing that there was a WebOS variant of the residential proxying SDK endpoint. Does that mean there might be a bit more unchecked malware lurking behind the scenes in the LG ecosystem?
Personally I'm surprised they didn't have a Samsung option.
I keep my brand new LG C5 totally disconnected from the internet and use my Apple TV for movie watching. I’m not going to trust a company like LG to secure their devices.
The need for proxies in any legitimate context became obsolete with starlink being so widespread. Throw up a few terminals and you have about 500-2k cgnat IP addresses to do whatever you like.
The actual secret is to use IPv6 with varied source IPs in the same subnet, you get an insane number of IPs and 90% of anti-scraping software is not specialized enough to realize that any IP in a /64 is the same as a single IP in a /32 in IPv4.
Yes, proxies are good. Ones which you pay for and which are running legitimately, with the knowledge (and compensation) of those who run them.
Malware in random apps running on your device without your knowledge is bad.
reply