I’ve been following the struggle of bearblog developer to manage the current war between bot scrapers and people who are trying to keep a safe and human oriented internet. What is lemmy doing about bot scrapers?

Some context from bearblog dev

The great scrape

https://herman.bearblog.dev/the-great-scrape/

LLMs feed on data. Vast quantities of text are needed to train these models, which are in turn receiving valuations in the billions. This data is scraped from the broader internet, from blogs, websites, and forums, without the author’s permission and all content being opt-in by default.

Needless to say, this is unethical. But as Meta has proven, it’s much easier to ask for forgiveness than permission. It is unlikely they will be ordered to “un-train” their next generation models due to some copyright complaints.

Aggressive bots ruined my weekend

https://herman.bearblog.dev/agressive-bots/

It’s more dangerous than ever to self-host, since simple mistakes in configurations will likely be found and exploited. In the last 24 hours I’ve blocked close to 2 million malicious requests across several hundred blogs.

What’s wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I’m still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers

  • Cocodapuf@lemmy.world
    4·
    19 hours ago

    Do you realize how much extra work your browser has to do every time you visit a site that makes money on ads? All the additional scripts being run in the background, it’s astonishing. Trust me, the additional work that users’ machines have to do for this is totally insignificant when viewed in the greater context of what we actually do with computers.

    Watching a 10 minute YouTube video, that’s your computer doing more work than it would loading a million text based pages running Anubis.

    • FaceDeer@fedia.io
      1·
      18 hours ago

      Do you realize how much extra work your browser has to do every time you visit a site that makes money on ads?

      I have uBlock origin and Ghostery, so very little.

      Watching a 10 minute YouTube video, that’s your computer doing more work than it would loading a million text based pages running Anubis.

      Given that AI trainers are training on YouTube videos too, that sounds like Anubis isn’t going to impose meaningful costs on them.

      • Cocodapuf@lemmy.world
        2·
        13 hours ago

        Given that AI trainers are training on YouTube videos too, that sounds like Anubis isn’t going to impose meaningful costs on them.

        Well, does it work?

        You don’t need to guess about it, you can simply look at traffic records and see how much it changes after installing Anubis. If it works for now, great. Like all things like this, it’s a cat and mouse game.

        Also, the way your computer interprets a YouTube video and the way a scraper interprets a YouTube video may well be different. But in general, for a browser, streaming and decoding video is a relatively heavy and high bandwidth operation. Video is much higher bandwidth and has much higher CPU processing requirements than audio, which likewise is heavier and higher higher bandwidth than text. As a result, video and text barely compare, they’re totally different orders of magnitude in bandwidth and processing needs. So does an AI scraper have to do all that decoding? I actually have no idea, but there definitely could be shortcuts, ways to just avoid it. For instance, they may only care about the audio, or perhaps the transcripts are good enough for them.