Yes offering everything for free to prevent competition has been a surprisingly effective strategy for Google.

I think you’d be right that the direct cost of running the crawler and index would not be the issue. But fighting SEO to keep your results decent is probably a cost that dwarfs the basic technical cost of running the crawler and index.

And you’d need a technical security team on top of things as link farms aren’t your only risk, I’m sure there are countless ways to manipulate the algorithm to put your site on top that Google probably have multiple teams working on fighting it full time.

Many of these things would likely not be a problem for a startup, though. No one is paying SEO firms big money to get into a search index no one has heard of and hardly anyone uses, so these costs probably grow exponentially over time as you become more well known.

I’m not disputing that you might be right, but the internet archive runs a very different service. Mainly that Google needs to continuously prune their 400 billion page index because of link rot. The Internet Archive has the opposite aim, they are preserving sites that no longer exist.

I’m also not sure they even crawl. Do sites get added on user request? When looking at a medium popularity page, you see it only has a couple of scrapes a year.

None of them. At least, none that I’m aware of. I just don’t think that direct expenses are the reason that there are are only two major web search tools. I also don’t think Google and bing are good examples to point at when estimating the cost of running a complete search engine.

I would suggest direct expenses are the barrier, but perhaps crawling is not the main expense. I would be interested to know any speculations you have outside of expenses that cause a barrier?

That website claims they add 3-5 billion pages a month. Google is doing that in a day or three, as recency of information is very important in search. Plus that site claims 100 billion pages to Googles 400 billion. It’s still an impressive project.

Size isn’t everything, so the real question is: what search site uses only the common crawl index and has results on par with bing or google?

I once read that running a search crawler costs upwards of a billion dollars a year. Anyone other than Microsoft or Google running their own search index are either not getting a wide spread of the internet or they are using their own index to supplement Google or Bing results.

Can confirm, my banking app works fine.

It’s not an on/off switch. Everything we can do will lessen the impact even if it can’t be stopped.

But as others mention, real impact comes from governments and international cooperation, not individual actions. Hence why voting is so important.

This was me. New job every year, eventually stretching to every two years. Bonus is getting a pay rise each time.

stay focused

😅

I can absolutely imagine sitting around with mates and deciding to order a delivered candy bar just to see if they would do it.

I love that he’s like “I co founded Netscape and Mozilla, now I run a dance club and pizzeria”.

Dude’s living the dream.

Holy crap, I hadn’t considered that we have the technology to create articles on the fly based on search terms.

That could be a serious weapon of war - you get your site to the top through SEO, then show each user personalised propoganda. You show googlebot the genuine page but adjust the page based on what you know about the user.

Kagi does seem to cut out a lot of blogspam. I think Google is incentivised to send people to these sites with adwords ads on them.

To give credit to the people trying to make good search, the internet is so much worse than it was a decade or two ago.

Over that time the ad driven internet has encouraged low quality, high volume websites full of articles designed around common search terms.

It started before AI, but now you can drum up an article in seconds it has got much worse.

AI is driving more searches

This is hilarious. People are searching more because they aren’t getting results they need from AI, which drives ad revenue so Google’s doing great!

Yeah that’s why I called it out. It seemed like it was an important part.

There probably isn’t an actual reason why you couldn’t connect it to a wall plug in a way that replicates this setup even if a bit hacky. E.g. short cable then attach the unit to the top of the plug. It’s not that heavy and may attach fine without coming out of the wall.

How far away is the Home Assistant Voice Preview from what you’re looking for?

It doesn’t plug directly into the wall but instead uses a USB C cable (that you provide). Other than this, mine can answer questions, search the internet, turn things on and off, play music via Spotify, Jellyfin, etc. Tell me about the state of stuff in Home Assistant (temps in rooms, how the solar is doing, what’s on my shopping list and can add things, etc).

It requires you already have Home Assistant set up but it is a pretty good experience so long as you’re willing to do some amount of tinkering to make it your own.

Like other comments say it’s not general public ready but it’s pretty close and costs $69.

Around two years ago reddit effectively banned most third party apps. That was when Lemmy went from a handful of instances with 1000 or less active users (mostly those banned from reddit), to tens of thousands of users and hundreds of instances in a short space of time.

People here are saying there are dozens of people here who came from reddit, but I’d guess it’s dozens of thousands, a pretty decent proportion of active users.

This article says it did it by disguising itself as a web crawler. So can I just set my user agent to googlebot and paywalls will disappear?

I live in New Zealand and there are many 24/7 McDonalds in busy areas. Clicking randomly on their NZ map it’s pretty easy to find them: https://mcdonalds.co.nz/find-us/restaurants

It’s the same with Australia: https://mcdonalds.com.au/find-us/restaurants

Actually, the same for the US. It’s not hard to find 24/7 ones (you need to search for a city before they show on the map): https://www.mcdonalds.com/us/en-us/restaurant-locator.html