Personal experience, 100%

As a producer, I can safely say that your entire job is based around one rule: the show must go on. That means you have backups. And your backups have backups. And your backup's backup's have backups.

Is it your fault when bad things happen? If you're good at what you do, probably not - especially when you consider the AMAZING number of things that can go wrong on any given day. However, it IS your fault if the bad things that inevitably happen cause the show to fail because you had no plan B.

Maybe it's a disgruntled employee with a set of passwords he really shouldn't have. Or maybe it's an outbreak of anthrax (yeah, I was on a job where that was suddenly an issue). Perhaps you're shooting on a beach and a dead body randomly washes up, suddenly turning your carefully chosen location into a crime scene crawling with cops and coroners (another true story / very rough day.) God forbid it's something really awful, like a bike messenger carrying an important batch of media getting hit by some asshole who was answering his phone instead of paying attention to the road, landing the kid in hospital, and turning your package into collateral damage (horrific moments like this, by the way, make the job seem suddenly unimportant).

Regardless, the show must go on. And if you have a good producer, it will. But losing TWO YEARS worth of work because one data center (effectively) went offline? That's seriously embarrassing. Also, very bad for credibility. And yes, reason enough to loose your stripes.

I think any professional can agree that if you're the guy at the top making decisions, everything is your responsibility. Taking the fall for something that goes wrong comes with the territory.

That's why every CEO is responsible for everything that goes on in their company, whether or not they knew every little thing that was going on. It's their responsibility to know, and put processes in place to ensure that they know, and if something still slips through the cracks, own up and take action to fix (or take the fall if it's a big enough issue).

What I think people are missing is that this doesn't absolve the perpetrator of moral, legal, and professional responsibility for a clearly malicious act. So two parties are at fault and should experience the pain in different ways. Fact remains, they should both experience pain.

As it just so happens, they are. In spades.

I can certainly understand the perspective that ultimately the responsibility for the loss will have to be borne by the folks who didn't do their homework or didn't take enough care in their backup strategy.

However, that wasn't the assertion that I was intending to respond to :) I've seen this "its not the perpetrator's fault" argument used several times in the last few weeks, the most recent that I remember was the kid who hacked the PHPFog site. The same argument was used there, and indeed was spouted off by the kid himself - it wasn't the hackers fault the site got hacked.

This line of reasoning seems dangerous to me, as it obscures a criminal or unethical activity by the ultimate result of that activity. Wrong action is wrong action, regardless of who didn't cover their bases.

Should the producer take more precautions, and will they ultimately be burned? Most assuredely, but lets not forget the reason they were burned in the first place - somebody maliciously acted against them.

Totally agreed. The incompetence of the producer is a problems for the producer's backers. It has no bearing on the obvious wrong done by the ex-employee.

A co-worker of mine once wiped out the entire CVS repository at the company we worked for because a symbolic link was placed in his home directory and he thought it was ok to 'rm -Rf' its contents. We were able to build a new repo based off of production and everyone who had changes not in production had to re-introduce those changes, but it was not the developer's fault for wiping out CVS. It was because the sysadmin had set the symbolic links for the primary repository. If you enable developers to do something bad, it isn't always their fault when they do it.

Having intent to destroy something on purpose is a different matter though, and I'm not sure that there is a good argument for it.

Did the sysadmin also forget to create a backup for the repo? I don't think it's a stretch to assume that the developers were somewhat negligent in working on a single repo that can disappear at any time.

There was no backup of the CVS repository. The production server saved us since we were able to treat it as the definitive backup. This was before Git and the concept that everyone had their own copy of the entire repository.

There is a sort of half intent by the developer. Yes, the developer had intended to delete the repository that was in his home directory, but he had no idea that it was actually a link to the master repository.

The developer had hard links or symbolic links? It must have been a hard link to do that type of damage. I find that really dangerous and that's why I never use them. Unless you are using an obscure version of UNIX, rm -rf should delete the delete and not the content of the folder it is linked to.

A hardlink would be safer in this case. If there are two hardlinks to the same file and you rm one of the hardlinks, the other hardlink still works and the file itself isn't deleted.

A symlinked directory, though, being made victim of a recursive rm command, just might result in rm traversing through the directory and deleting everything inside of it.