The exact adversary Anubis targets defeats it trivially.
I don't believe this is true. Anubis' targets are not LLM agents and vibecoded programs. To my understanding, Anubis main aim is to block web scrapers used by LLM companies. These are bots that just go out and slurp anything they can find. Sure, they could be powered by agents or LLMs themselves, and sure there can be work arounds. But the vast majority of these scrapers are "dumb" and their creators don't really care about bypassing the minority of sites that implement Anubis --- and that's kind of the goal here, it's kind of just annoying enough to where it doesn't really matter to the owners of the scraper that these sites can't be scraped.
Yeah, Anubis probably works fine to prevent your server to be hammered to death from residential proxy using scrapers. A request from an individual agent is not much different from its user browsing your site themselves.
However this also means, Anubis is overdesigned for what it does. Requiring a single button press and then issuing a cookie for access provides the same protection. I implemented that in Apache and that works just fine without any need for complicated proof of work.
Yeah, Anubis probably works fine to prevent your server to be hammered to death from residential proxy using scrapers. A request from an individual agent is not much different from its user browsing your site themselves.
Why though? It only helped during the brief window from it was introduced to when scrapers implemented proof of work to bypass it. A bunch of sites have already reported that Anubis doesn't work for them anymore (e.g. Codeberg). Once the scrapers implement it, the cost for them is much lower than for real users, because they just use people's smart TVs as proxies for free (play Pac-Man without ads! [1]) and because solving the PoW is much faster when using native code.
For a short time, it was a nice stopgap, but now it is wasteful.
Edit: someone correctly points out below that the TVs may only be used as proxies and not to do the PoW. I think the point still stands though, the PoW calculations are still cheap. There companies are very well capitalized and the cost of solving PoW that has to run on consumer devices is probably not even showing up on their radar compared to GPU cost for training their models.
Well. I think Anubis is not much different from any other CAPTCHA. As long as you are a small target, you get away with the simplest stuff. For your average personal website it's perfectly fine to protect the comment form with a simple "type 'orange' in this input field" style of CAPTCHA. But if everyone uses exactly that mechanism, it is suddenly worth to specifically target and defeat the CAPTCHA.
As long as Anubis was used for a few geeky homepages, it was not worth it to invest any effort into it. Codeberg OTOH is a big enough target to be interesting.
Though I wouldn't be surprised if scrapers would still specifically target their Anubis-Circumventer only at Codeberg and not care about the rest of tiny pages using it. So Anubis might still work fine for many of its users...
Yep. I actually also do care about preventing RAG access to my personal site and software, too, though. I manage that with basic useragent blocking, and with basic testing (opening up chatgpt and asking it to access my site) that works fine. Super custom setups might fall through the cracks, but. my goal is to block the lazy, not the malicious.
Anubis main aim is to block web scrapers used by LLM companies
I'm not sure what your definition of an "LLM company" is the increase in traffic we see on websites are not coming from OpenAI, Anthropic, Meta and the like but by other actors.
But the vast majority of these scrapers are "dumb" and their creators don't really care about bypassing the minority of sites that implement Anubis
Not from my experience. Modern scraping projects are very good at bypassing almost anything that comes in their way. I know it, because I'm using them myself for data mining. Anubis is trivially bypassed.
I mean, if I understand this right, Anubis is more or less intended as an anti-DoS mechanism, rather than "blocking AI agents from accessing it" (which, let's be honest, it can't reasonably do)?
On the question of "who does Anubis actually stop", though, it includes me when on mobile devices and on sites I'm not familiar with (so I don't have JS on).
For a scraping operation leveraging botnets of very low-end devices (i.e. smart TVs), it doesn't seem far-fetched that Anubis is an issue for them. https://lobste.rs/s/kpaxih/update_on_scraper_situation touches on that a little.
While Anubis usually doesn't block Dillo, sometimes sites configure it to do so, which is pretty annoying because it promotes the narrative of "if you don't have JS, you are not allowed to exist": https://paste.rs/jNgwd.png
I'm thinking if it would be possible to collect and maintain a list of all the IPs that are present in residential proxies and ban them for a while (perhaps ipset may be able to scale?). From the logs it is relatively easy to spot, because bots will crawl every single file at every commit in dillo repos, without loading any other resource (CSS or images).
I would be willing to bet that while sites only see a unique request per IP, the same IP fetches different sites. So the same IP reported multiple times will cause you to get banned. The ban should allow you to know what has happened and how to fix it, not just close your connection. Perhaps this can be done by redirecting the request to another port that serves the notice. This should also get rid of the incentive to introduce a reverse proxy backdoor in your browser extension or phone app, as it will cause users to start being banned.
It is sad that we would have to resort to a reputation system, but I'm not sure if there is any other effective measure against this. While those devices are still vulnerable, even if the botnet C&C is taken down, they will get exploited by the next attacker and you get yourself another botnet.
I don't believe this is true. Anubis' targets are not LLM agents and vibecoded programs. To my understanding, Anubis main aim is to block web scrapers used by LLM companies. These are bots that just go out and slurp anything they can find. Sure, they could be powered by agents or LLMs themselves, and sure there can be work arounds. But the vast majority of these scrapers are "dumb" and their creators don't really care about bypassing the minority of sites that implement Anubis --- and that's kind of the goal here, it's kind of just annoying enough to where it doesn't really matter to the owners of the scraper that these sites can't be scraped.
Yeah, Anubis probably works fine to prevent your server to be hammered to death from residential proxy using scrapers. A request from an individual agent is not much different from its user browsing your site themselves.
However this also means, Anubis is overdesigned for what it does. Requiring a single button press and then issuing a cookie for access provides the same protection. I implemented that in Apache and that works just fine without any need for complicated proof of work.
Why though? It only helped during the brief window from it was introduced to when scrapers implemented proof of work to bypass it. A bunch of sites have already reported that Anubis doesn't work for them anymore (e.g. Codeberg). Once the scrapers implement it, the cost for them is much lower than for real users, because they just use people's smart TVs as proxies for free (play Pac-Man without ads! [1]) and because solving the PoW is much faster when using native code.
For a short time, it was a nice stopgap, but now it is wasteful.
[1] https://spur.us/blog/smart-tv-apps-residential-proxy-sdks
Edit: someone correctly points out below that the TVs may only be used as proxies and not to do the PoW. I think the point still stands though, the PoW calculations are still cheap. There companies are very well capitalized and the cost of solving PoW that has to run on consumer devices is probably not even showing up on their radar compared to GPU cost for training their models.
Well. I think Anubis is not much different from any other CAPTCHA. As long as you are a small target, you get away with the simplest stuff. For your average personal website it's perfectly fine to protect the comment form with a simple "type 'orange' in this input field" style of CAPTCHA. But if everyone uses exactly that mechanism, it is suddenly worth to specifically target and defeat the CAPTCHA.
As long as Anubis was used for a few geeky homepages, it was not worth it to invest any effort into it. Codeberg OTOH is a big enough target to be interesting.
Though I wouldn't be surprised if scrapers would still specifically target their Anubis-Circumventer only at Codeberg and not care about the rest of tiny pages using it. So Anubis might still work fine for many of its users...
Yep. I actually also do care about preventing RAG access to my personal site and software, too, though. I manage that with basic useragent blocking, and with basic testing (opening up chatgpt and asking it to access my site) that works fine. Super custom setups might fall through the cracks, but. my goal is to block the lazy, not the malicious.
I'm not sure what your definition of an "LLM company" is the increase in traffic we see on websites are not coming from OpenAI, Anthropic, Meta and the like but by other actors.
Not from my experience. Modern scraping projects are very good at bypassing almost anything that comes in their way. I know it, because I'm using them myself for data mining. Anubis is trivially bypassed.
Other actors which sell their scraped data sets to LLM companies.
[Comment removed by author]
I mean, if I understand this right, Anubis is more or less intended as an anti-DoS mechanism, rather than "blocking AI agents from accessing it" (which, let's be honest, it can't reasonably do)?
On the question of "who does Anubis actually stop", though, it includes me when on mobile devices and on sites I'm not familiar with (so I don't have JS on).
Anubis does not filter bots, it rate-limits clients.
For a scraping operation leveraging botnets of very low-end devices (i.e. smart TVs), it doesn't seem far-fetched that Anubis is an issue for them. https://lobste.rs/s/kpaxih/update_on_scraper_situation touches on that a little.
If I'm not mistaken SmartTVs are used for proxies and are not actually doing the PoW themselves.
Oh yeah that would make more sense
While Anubis usually doesn't block Dillo, sometimes sites configure it to do so, which is pretty annoying because it promotes the narrative of "if you don't have JS, you are not allowed to exist": https://paste.rs/jNgwd.png
I'm thinking if it would be possible to collect and maintain a list of all the IPs that are present in residential proxies and ban them for a while (perhaps ipset may be able to scale?). From the logs it is relatively easy to spot, because bots will crawl every single file at every commit in dillo repos, without loading any other resource (CSS or images).
I would be willing to bet that while sites only see a unique request per IP, the same IP fetches different sites. So the same IP reported multiple times will cause you to get banned. The ban should allow you to know what has happened and how to fix it, not just close your connection. Perhaps this can be done by redirecting the request to another port that serves the notice. This should also get rid of the incentive to introduce a reverse proxy backdoor in your browser extension or phone app, as it will cause users to start being banned.
It is sad that we would have to resort to a reputation system, but I'm not sure if there is any other effective measure against this. While those devices are still vulnerable, even if the botnet C&C is taken down, they will get exploited by the next attacker and you get yourself another botnet.
So then why is this not tagged vibecoding?
because it's not about vibecoding, just as it's not tagged compilers or editors, although author used them (and
vibecodingwas removed by pushcx)