Yet another Hacker News

AWS says it can't restore some data from mideast facilities struck by Iran

107 points berkeleyjunk 1 day ago 65 comments

https://archive.is/Ay7RJ

markive 23 hours ago | parent

Does this mean that even with 3 availability zones for Amazon S3 storage, that some data is lost?

MiroslavPokorny 22 hours ago | parent

Obviously what you understand is different from the reality after you actually follow all the footnotes.

OrangeDelonge 19 hours ago | parent

Did they say its S3 data? Could also be single-az EBS or RDS.

dhx 12 hours ago | parent

For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data.

It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.

I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.

[1] https://news.ycombinator.com/item?id=49033240

[2] https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Cri...

0cf8612b2e1e 35 minutes ago | parent

As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?

testplzignore 24 minutes ago | parent

https://aws.amazon.com/compliance/data-privacy-faq/

"You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."

zmgsabst 29 minutes ago | parent

AZs weren’t meant to be disaster resistant, eg, an earthquake or hurricane could take out a whole region.

Regions were always the scale of disaster isolation on AWS.

gregw2 31 minutes ago | parent

I think so.

Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.

curuinor 23 hours ago | parent

They guaranteed 11 9's durability, didn't they?

e: Yep

https://aws.amazon.com/s3/storage-classes/

jonahx 43 minutes ago | parent

I don't think this has any teeth. They don't compensate in the event of loss afaict.

rbanffy 38 minutes ago | parent

Considering all the data they have globally, they might still be compliant.

stackskipton 33 minutes ago | parent

Even if they have payable SLA on this, most SLAs have Acts of God and Acts of War exemption.

advisedwang 26 minutes ago | parent

They say it's "designed for" 11 9s, not guaranteed.

beejiu 25 minutes ago | parent

The SLA excludes force majeure.

jameshart 19 minutes ago | parent

The footnote which says that is the design durability against equipment failure literally begins:

> In the unlikely case of the loss or damage to all or part of an AWS Availability Zone, data in a One Zone storage class may be lost. For example, events like fire and water damage could result in data loss

chews 15 hours ago | parent

I'm sorry but your data is in another castle.

Chance-Device 45 minutes ago | parent

This should have been in that super Dario game.

carefree-bob 56 minutes ago | parent

This is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.

kibwen 50 minutes ago | parent

For long-term backups you want offline cold storage in an underground facility in a friendly jurisdiction, not a datacenter.

watwut 49 minutes ago | parent

War did not randomly came. America intentionally caused it.

And has lawless goverment and unaccountable tech industry making it bad place for data.

aprilthird2021 44 minutes ago | parent

I wonder if Amazon can sue the US govt bc they basically caused this material loss to their business. I'm sure they cannot. Maybe a lawyer can explain why?

nradov 33 minutes ago | parent

In US courts you can sue anyone for anything but you might not win. The US government has sovereign immunity from most civil liability. As for the legal system in Bahrain I have no idea but hypothetically even if Amazon could somehow win a judgment they wouldn't be able to collect.

carefree-bob 8 minutes ago | parent

You don't see the EU causing a war with Russia?

Let's just revisit all the different acts of war that the EU has already done:

   * Seizing sovereign reserves
   * Seizing Russian ships on the high seas
   * Sending special forces and other "military advisers" to launch long range attacks on Russian soil. 
   * Sending in commandos to launch attacks on nuclear power plants

You really don't think this will lead to war? I think there is a 50% chance there will be open war, and that percentage increases as the EU continues to take these measures, each of which are equivalent to acts of war from the perspective of international law.

And when this happens, and EU data centers are hit, you will probably still blame America and then not make any connection with data residency requirements and the resulting chaos.

BonoboIO 49 minutes ago | parent

If you can not restore data from EU based Amazon Datacenters because it’s destroyed … you will definitely have better things to do like packing your go bag or buying the last groceries for a while.

numpad0 44 minutes ago | parent

I don't think this is a flipside, unless you're thinking from the PoV of data itself rather than its owners.

wewewedxfgdf 46 minutes ago | parent

"Use the cloud" they say.

"It's the only way." the true believers say.

"You can't run your own computer systems.", they say.

"Trust us.", they say.

“No one ever got fired for buying IBM.”

Wildberries has nothing to do with AWS.

"Massive centralisation of computing can never go wrong."

"We don't need to host our own systems, it's all safe in one of 20 global data centers."

etc etc

knorker 30 minutes ago | parent

What's you point? That if you had run your own DC in that region (because that was your business requirement) then you'd have better missile defense than AWS?

Or maybe AWS or DIY, you are always responsible for geographic diversity?

Anyone losing data over this lost it because they'd literally told AWS to only store it in one place.

shevy-java 28 minutes ago | parent

How about not bombing other countries and then acting surprised when retaliation happens? I mean clearly the problem isn't AWS as such - it is the problem that someone leading a country is totally clueless about the world. Only personal profit is in the interest of the orange clown.

culi 21 minutes ago | parent

> I mean clearly the problem isn't AWS as such

I get your main point, but just wanna point out that AWS is one of the largest military contractors in the world. They hold multi-billion dollar contracts from the DoD, USAF, CIA, and more. An estimated $4B a year in military spending goes to AWS

wewewedxfgdf 28 minutes ago | parent

The point is that AWS has the same problem as Wildberries.

There is no difference at all between Wildberries and AWS data centers.

If you don't know what Wildberries is then go watch their facilities systematically destroyed on YouTube - centralisation is a target.

If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. Is that on your risk management plan?

gleezard 25 minutes ago | parent

HN shitposting is the point. If you want actual pragmatic commentary, use Lobsters.

culi 25 minutes ago | parent

You don't need better missile defense than AWS. You don't need missile defense at all because you won't be a target. 99.999999% of the land has no missile threat on it. You are actively increasing the threat to your business by running it on the same servers that military contractors run their software on.

vkou 20 minutes ago | parent

In a war where schools and bridges and aid convoys and bread lines are targeted, anyone should consider themselves a valid target.

It's true that you are less likely to be a target than a state-sponsored tech megacorp's data centers, but the risk is still present.

noir_lord 10 minutes ago | parent

Risk is always present, We quantify it for a reason and then we mitigate if it's beyond a level we are comfortable.

Living is risk, every time I go to the shop for milk there is a non-zero chance I don't come back but the risk is so low I don't worry about it.

drnick1 9 minutes ago | parent

> What's you point?

Not the OP, but the point is that decentralized infrastructure is a lot more resilient to attacks of any kind.

justonenote 29 minutes ago | parent

this is reductionist to the point of absurdism.

It can be true that using cloud storage, using managed services, and paying a premium is still worth it for a lot of people and organizations, and while not perfect, still a hell of a lot better compared to the fully in your control tape backups that you distribute to different physical locations every week.

I'm not a particular fan of relying on one provider or vendor lock in, but to pretend they don't provide a service with failure rates that are low enough to be very useful is a very short-sighted take.

shitloadofbooks 23 minutes ago | parent

Can you post the specs on your missile defense system for your home lab?

How are you dealing with the rise of low-cast swarm attacks from drones?

Is it land-based, sea-based or space-based and at what point of the trajectory do you target and do you use jamming?

noir_lord 12 minutes ago | parent

Eh I get your point but would point out that distribution does mitigate the risk here, having all your data in DC's that can be seen from space also isn't a panacea when your next door neighbour targets them in retaliation for what your ally did.

Art9681 43 minutes ago | parent

No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.

rbanffy 38 minutes ago | parent

The more dramatic contingency you have to plan for, the more expensive the plan gets.

Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.

ares623 28 minutes ago | parent

"Daddy, where were you when the flames reached our house?"

"I was in the office, reviewing Terraform plans"

jiggawatts 6 minutes ago | parent

I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.

He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.

rokhayakebe 28 minutes ago | parent

Did local laws permit?

advisedwang 27 minutes ago | parent

That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported.

Now if this was multi-AZ S3 or whatever then this would be significant.

The article does not tell us what products were impacted.

cpncrunch 23 minutes ago | parent

But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.

yipinwong 23 minutes ago | parent

Nothing is ever real-time. Eventual consistency leads to some data are not backed up.

You talking as if this is some mom-and-pop shop that you run.

jeremyjh 18 minutes ago | parent

That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.

nixass 13 minutes ago | parent

Offsite to.. where? Sea? Data residency in Gulf states is very strict and basically nothing is leaving the countries

michael-bey 42 minutes ago | parent

What a nightmare scenario to tabletop. How do you even begin to recover from something like this?

NegativeLatency 32 minutes ago | parent

Backups in a different region?

noir_lord 15 minutes ago | parent

Works unless local laws specifically block you doing that which they do for some classes of data in some countries.

Multi-cloud in the same country (if that exists in the country and is far enough apart) maybe.

criemen 5 minutes ago | parent

Do cloud providers even share data center locations so you can assess the "far enough" bit yourself?

nixass 11 minutes ago | parent

Not applicable in Gulf states

rbanffy 42 minutes ago | parent

Well… they have a good excuse.

shevy-java 29 minutes ago | parent

How about ... stop bombing other countries? Trump is like a professional liar. From "no more forever wars" to "hey this is what must be done now" in a second. He is almost as good as Putin with regards to lies - the ultimate agent Krasnov. Minus the apparent dementia now.

Cyclone_ 10 minutes ago | parent

It's a war that he started to benefit Israel, and is now causing suffering for so many others.

rvz 28 minutes ago | parent

Backup both locally and everywhere no matter what.

burnt-resistor 28 minutes ago | parent

50%+ of companies that lose all of their data go out of business in 6 months.

DR/BCP costs are readily justified by doing a Business Impact Analysis (BIA).. budget up to some fraction of risk cost * risk probability.

And a friendly reminder that replication isn't a tested data backup.

rishikeshs 26 minutes ago | parent

I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.

jackb4040 11 minutes ago | parent

Hi, I'm from the future. You might want to consider storing the data somewhere besides an Azure datacenter in the UAE.

r_lee 7 minutes ago | parent

> the government requirements requires me to store data only in UAE!

birdatlaw 5 minutes ago | parent

turns out reading comprehension skills have still not gotten better in the future

SmirkingRevenge 12 minutes ago | parent

It was always a bad bet for billionaires like Bezos to become Trump enablers. You weren't buying a seat at the table, or the privilege of being left alone, you were just signing yourself up to be force-fed shit sandwiches over and over (And the shit-to-bread ratio gets worse as time goes on)

You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.

skybrian 8 minutes ago | parent

I wonder if they'll start adding an underground bunker to new data centers so you can put an S3 replica there?