Interesting Links for 06-10-2025

[andrewducker]

1. Scientists Reveal Biological Basis of Long Covid Brain Fog

(tags:Pandemic brain disease )

2. Fire destroys Korean government's cloud storage system, no backups available

(tags:Korea data epicfail backup )

3. Disastrous South Korean data centre fire is connected to North Korean hacking

(tags:NorthKorea korea hacking sabotage )

Comments

[aldabra]

2. How can you have a data structure that doesn't allow for backups? Why wouldn't a recursive search-and-copy work? I mean, I am right now trying to design a backup approach for some software we are using and it doesn't work in the places where we can't see and don't own the proprietary data "structures" they are using, but the Korean government presumably could in principle see what they were working with? Those 750,000 civil servants are saving files in a directory structure? You can traverse that.

[andrewducker]

All a bit suspicious, isn't it?

[simont]

Off the top of my head: if a lot of the data is confidential and any given person is only allowed to access the parts their particular job gives them a need to know, then there might be nobody who can do a single recursive trawl of the whole filesystem and catch everything? So instead you'd have to cat-herd a ton of smaller sub-organisations or perhaps even individual people into each arranging to run reliable backups of their own little bit. And I expect that would go about as well as actual cat-herding: most of those people would hear "you must do this", interpret it as "you must have sorting this out on your backlog somewhere", and then prioritise everything else above it for years on end.

Of course, someone in the datacentre with physical access to the machine could surely back up the whole lot at the layer below per-file access control – just image the whole hard disk. But it might very well be arranged that when the system is running normally nobody gets full access to everything, so that you couldn't do that without a full shutdown and reboot into Special Administrator Mode. If you were storing that much sensitive data, that might look like a good idea, to reduce the risk of a single bribed, blackmailed or disgruntled official exfiltrating the whole lot in one go. You might also make an admin-mode reboot require a huge amount of special authorisation paperwork from three branches of government and be illegal as hell to do otherwise.

Reliably backing up secrets is hard: secrets should be in as few places as possible, but backups should be in many, and there is a tradeoff. Even I – with a much smaller computing infrastructure handling at least one or two moderately important secrets – struggle with deciding how best to do it. And I don't have soundbite-influenced politicians trying to second-guess my technical decisions!

Not saying that mistakes were not made. But I don't see this as a totally incomprehensible foulup. On the contrary, an all too plausible one.

[andrewducker]

Encrypted disks are a thing nowadays. Backing up at block level, without the backups being easily accessed, is definitely possible.

No person should have easy access at that level, but automated processes should be able to manage that, with secrets injected. It's not simple, but it's all possible.

When the alternative is "A fire can destroy your country's bureaucracy" then they really should have tried harder.

[bens_dad]

Who do you trust to synchronised the keys between datacentres ?

Remember that the South Korean president was impeached for declarjng Martial Law https://en.wikipedia.org/wiki/2024_South_Korean_martial_law_crisis then the PM was suspended for failing to appoint some new judges
(overturned by a court) https://edition.cnn.com/2025/03/23/asia/south-korea-court-reinstates-prime-minister-martial-law/

[andrewducker]

You have keys where multiple different people are needed to unlock them. And you don't store the key with the backed up data.

I'm not saying that these things aren't hard - just that they are possible.

[simont]

I agree that encryption is part of the solution. Even so, the encryption key also needs to be backed up, and then you need to try to arrange that the offsite copies of the key and the encrypted data at least aren't stored together, and ideally, stored in such very different places that it would be really hard to get at both together.

But also, encryption is even more likely than ordinary data storage to be misunderstood by either politicians or the people they're trying to impress, leading to misguided attempts to interfere!

[bens_dad]

How can you have a data structure that doesn't allow for backups?
The article says no external backups. If the system is heavily loaded, keeping a remote backup synchronised with a live system is a tall order.

don't own the proprietary data "structures" they are using, but the Korean government presumably could in principle see what they were working with?

Horizon suggests to me that we cannot assume that the Korean government would have practical access to the proprietary data "structures". It could easily be mediated through the suppliers.

Those 750,000 civil servants are saving files in a directory structure? You can traverse that.

I don't know much about databases but IIUC when I left UCAM, MISD the merging university central IT teams had storage systems based on a database with no separate underlying filesystem. It isn't clear to me that those would have had a directory structure.

In my experience, cost per unit of storageincreases with size of the collection, so I would not be surprised to find that a nationwide system has cut some corners.

Re: 1.

[channelpenguin]

Very important. Means it's provably real, can be detected with high confidence and has a plausible mechanism that might be amenable to treatment. This could definitely be helpful for those dealing with disbeliving doctors, employers and insurers. Should. I hope, anyway.

[channelpenguin]

2. That's not Cloud Storage, that's On-Prem

[andrewducker]

Only if you work in the datacentre.

[channelpenguin]

Ok. Sure. I get you. I stand corrected.

It was a data centre. A private one, it seems. With no off-site backups. For gov, that's kinda unbelievable, but hey. Back in my days at BR, we had 2 data centre sites, pretty much hot swappable and a 3rd "secret" site (rumoured to be in Wales).

To me, "Cloud" implies one of the big players with all that offers. But yeah I'm sure there's ways to fuck that up too.

And too many people back in the day never tried actually restoring their backups from time to time...

[andrewducker]

It's definitely a giant fuck up/sabotage in any case. Probably both.

[bens_dad]

Network rather than storage; the story I heard was that some time (last century, I believe) Cambridge had problems with the link to JaNET, the nation university network and eventually achieved redundant links with two different suppliers.

A JCB (backhoe?) managed to break both links, because the two suppliers were using the same trench !

The entertaining part was that the break was in Bury-St-Edmunds, tens of miles in the wrong direction from the direct route.

[andrewducker]

Ouch! I wonder why the "wrong" direction was easier. I assume historical reasons of some kind!

[bens_dad]

I didn't get to ask that.

I don't know the telecom map of the UK, but I wonder whether the BT research labs at Martlesham Heath, Ipswich are relevant ?

[redbird]

After the 9/11 attacks, it was discovered that some computer communication systems that should have been triply-redundant, from different suppliers, all used cables that went under the World Trade Center. I think that was a combination of the people who paid for that should-be-redundant system not having checked that companies A, B, and C weren't using any shared cables (possibly owned by D), and not having checked the physical locations of the cables.

[andrewducker]

Oh dear.

I do think that most people are terrible at thinking things through.

[bens_dad]

Ah, the reverse of the 1996 Manchester bombing.

Some servers in buildings demolished by the bomb were still running and there was a rush to copy the data before the emergency services switched the power off.

[bens_dad]

Unless they have a very fast network, there may be (possibly unreliable) caches of data nearer to the users.
Whether reliable data can be reconstructed from these (whether quickly or cost effectively) may be less clear.

[mountainkiss]

I love how “pandemic brain disease” can equally credibly be three tags or one.

[andrewducker]

It's also the chorus to my new song "I got the brain-fogging, energy-sapping, sleep-inducing Covid Blues".

[mountainkiss]

It could even be the name of the band.

[symbioid]

I misread "Fire destroys Karen's cloud storage" and thought "That's what you get for being a Karen!"

(But not the "Karen People" who are a tribe in Myanmar)

[greenwoodside]

2. Eep. About eight years ago, I took a module on digital preservation.

Apparently the Korean government didn't.

And someone must have really hurried through the risk assessment.