The Three Letters

----- Begin NetScrap(TM) -----

The Three Letters


When I was hired to run the IT department of a major company 
my predecessor left three letters in the desk that was now 
mine. Each letter was clearly labeled; System Failure #1, 
System Failure #2, System Failure #3. A post-it note was 
attached to the bundle of letters.

In case of a substantial system failure open the letters 
in order, once per failure, and they will help you through 
the problem.

I put the letters back in the desk and forgot about them.

About one year later we had a cascading server failure that 
left our corporate intranet and several important production 
servers off-line. While repairing the problem I remembered 
the letters. Curious, I opened the first letter.

"Blame me, your predecessor"

The day after we got the servers back up I was called in to 
my boss's office to explain what happened and why were down 
for so long. Taking my queue from the letter I blamed my 
predecessor. My boss was satisfied with my answer and let 
me go.

About six months down the road we had another big failure. 
This time our primary database server went down and the 
secondary was having trouble dealing with the load. I had to 
put a lot of extra hours into getting them back up and we lost 
a few transactions due to the backup server not being able to 
function under the load.

Once again, I reached into that desk drawer and opened letter #2.

"Blame the equipment"

This time I lamented to the boss about how it wasn't my fault. 
It was that backup server! If we had some good equipment to run 
on these things just would not happen. He was satisfied with 
my answer and I went back to work.

Things ran smoothly for the next 18 months. Then we got hit with 
a virus that somehow got past our firewall and wrecked havoc on 
our systems.

I opened the third letter.

"Write three letters"


----- End NetScrap(TM) -----