…and I did not even notice it, aside from the news here.
Maybe it was a bad idea for society to put 90% of the internet on one company’s infrastructure.
Let’s take a decentralized network and centalize it! Nothing will surely go wrong.
It’s kind of telling when Amazon services like Ring, Prime Video, and Alexa failed over pretty quickly, but everything else just didnt.
There’s no conspiracy here. It’s just highlighting that Amazon could prepare for AWS outages in a region, but since everything auth related was on us-east-1, everything else went down.
Shortcuts are costly.
It’s almost as if centralizing everything is fucking stupid.
But the bean counters said it was the best idea!
Amazon sneezes, the whole internet catches a cold.
Im still seeing services up/down the entire day at work. Services that are not even AWS like Azure are slow for some reason (probably businesses failing over to other infa). Its crazy.
None of our in office infa is having issues. Managers are talking about fail-overs all day lol.
When a handful of people own all the companies in the world, the whole world becomes a single point of failure.
Soon: “Welcome to Amazon, I love you.”
There are some downstream / knock on effects going on which can be explained…but I can’t help but wonder if today’s story is bigger than just AWS. AWS saying it was an outage of a “few hours” for DynamoDB and DNS…and that doesn’t line up all that great with what people are reporting in the wild . I’m not trying to start a conspiracy theory, just wondering what the post mortems will tell us, if anything. Obviously the suits want to keep embarrassing fuckups downplayed as much as possible.
Dave explains the “long tail” of recovery:
Its crazy, we are seeing unrelated services stop sending emails, issues with DNS, all sorts of strange stuff.
Same with us. Had to reboot/restart a number of things, and resynch clocks.
They want to keep the news of the rally over the weekend as quiet as possible.
and the epstein files. i heard it dint affect international that much, so its rather covenient.
games were affected online, plus apps, and then anyone in retail who does inventory, order writing.
Jesus Amazon is a big enough company they should be doing their own hosting not using AWS.
They’re big enough to be a utility and be nationalized!
And should be
Amazon is using their own hosting. AWS stands for Amazon Web Services. Or did I miss the joke?
I for one don’t think you missed anything
Miss
I was confused too, but I’m still glad you didn’t add the /s because that always kills the joke.
Once upon a time they insisted that Amazon had independent decision making on their providers when they were needing new infrastructure and they “always decided that AWS suited them best.”
Shockingly (/s), they stopped making that claim right about the same time they started admitting that their biggest users are all under Private Pricing Agreements.
So Roblox is the only of those companies that can handle proper region failover if us-east-1 shits the bed? You young ops have gotten soft! Learn to live by ChaosMonkey or die by the Gorilla.
Roblox doesn’t really lock down regions unless they are China. The game client can connect to any server they own. It naturally falls back to the server with the least issues without prompts and intervention. If for an even bigger example all US servers were to go offline, the client automatically redirects people to either Europe or Asia based on ping.
Yeah, but proper failover and recovery requires additional infrastructure, and that costs money.
Hopefully a bunch of risk management people are writing I-told-you-so emails to C-suites right now.
Yeah but not to worry, C-suites have pretty good filtering rules in place to never read them. Saves time, really.
And plausible deniability.
Yup, and some things which can be moved cannot be done automatically, quickly or easily…even if you are prepared. AWS is a huge suite of products and services, and there’s a lot of old legacy shit running on it. I wouldn’t punch down on the ops for this one. Cybersecurity and disaster recovery are not directly profitable, so they are almost always neglected in your average shop.
AWS is a huge suite of products and services, and there’s a lot of old legacy shit running on it.
Yup, AWS is legacy cloud. It was only recently that they set encryption by default on S3 buckets, before that they were just in the clear by default.
Cybersecurity and disaster recovery are not directly profitable, so they are almost always neglected in your average shop.
It’s never important until suddenly it’s the most important thing in the world.
So Roblox is the only of those companies that can handle proper region failover if us-east-1 shits the bed?
Either that, or the reports stopped coming in since it’s a school day.
I think they’re the only ones doing multi-cloud. If us-east-1 shits the bed, it’s a bad day for AWS in general.
Figma balls wheeeyyyy
It seems that there is not a single original thought in that head of mine.
doesn’t matter bro, they’re your thoughts and that’s what makes them awesome 😊
Figma
of your imagination?
Alright…
crushes your balls with ginger root which makes them burn
Why does Amazon use AWS?
I thought they had an in-house solution.
A lot of AWS are wrappers over said in house solutions
And there’s a push to move off of those to proper AWS
Bottom left
figma ballz hahaha gotem
Edit: ohh noooo I just saw PhobosAnomaly’s post, I am so slow
The cloud was a scam for people with more money than sense.
Most people hosting things on the cloud, like Fediverse services, would be better off selfhosting and buying the hardware themselves.
Self hosting requires an immense amount of specialized knowledge and time…
I got started self-hosting last week when I got a hold of a smal Lenovo ThinkCentre. I installed Proxmox on it and if I want to self-host something I just spin up a container or virtual machine on the Proxmox system. It’s so much easier than installing self-hosted projects on bare metal. And if you want to change things around then just disable or delete the container/vm and let Proxmox stay clean. If one container breaks the rest of the system will still function.
I could easily host Lemmy from home with Proxmox and a reverse proxy with my current setup. I am not going to because I am not interested in moderating a platform and all the responsibility that comes with it, but it’s very possible to do.
Edit: the Proxmox community helper scripts makes installing most things a breeze! I use them every opportunity I get.
https://community-scripts.github.io/ProxmoxVE/
This is the first script to start with on a fresh install
https://community-scripts.github.io/ProxmoxVE/scripts?id=post-pve-install
Cool?
It really doesn’t, and certainly not more than running something on AWS.
In fact, hosting on cloud infrastructure adds another layer of complexity.
I stand by my original assertion.
You’re just wrong…
Shit, none of my websites are slow. Fuck Amazon
Diversity is a strength when things go bad.
I’m okay to blame bozos.
