Who, Me? We’ve covered backups before in the annals of this column, but a bit of helpfulness that turned into a bonfire of the binaries? Start your Monday with a lesson in not taking the initiative.
Our story comes from “Harry” (not his name) who was working for a major medical products company back in the days when Windows XP was the hot new thing and software was, frankly, a bit simpler.
We’ve also turned the Regomizer on a new hire assigned to Harry. We’ll call him “James” since we imagine he’d rather his true identity never be made public considering what happened…
However, back to Harry for now. Harry was a line engineer, responsible for ensuring “the enormous series of complex machines spat out the product in just the right way to avoid any defective products getting out of the factory.”
This was important, not just because of the whole medical product thing, but because certain countries operated a three-strikes-and-you’re-out rule for products with issues turning up in the market. So those machines had to be in tip-top shape, along with the software that ran them.
Harry was given a new hire, James, to train up on the equipment. “New hires would usually be fresh engineering graduates,” he said, “and things went as they usually went, having a test line where mistakes could be made, and, importantly, used a simplified software stack, running on Windows XP (of course).”
When training was complete, James could be let loose on the real thing. He was expected to know the lines inside and out but, crucially, not the software stack. That was very much a black-box affair, updated manually when a CD-ROM arrived with updates.
The time came to do the updates, and James was instructed to run them while the machines on the lines were being cleaned and recalibrated.
James was the only one on the night shift and, looking at the update and thinking he knew a bit about PCs, decided to use his initiative. He ran the updates on all the lines. He then decided to see if he could make the PCs a bit more efficient. The disks could always benefit from more headroom, so he ran a utility to look for duplicate files.
Oh my, there were lots. James had full system access and so after each update removed the duplicate files. Much better.
“The lines were fired back up over the following days,” remembered Harry, “getting ready to get back to full productivity. But they didn’t.”
Instead each line popped up a console window on their displays, all with the same message:
FILE NOT FOUND
The message repeated over and over again, and the factory was eerily silent.
“Turns out,” said Harry, “that the software demanded a precise duplicate of a large database in order to perform its normal duties, but James had destroyed all of them.”
A fresh install had to be mailed out in order to get the lines restarted. Harry was hauled before the bosses but successfully protested his ignorance by virtue of simply not knowing how one would go about doing such a thing.
James was not so lucky, and found himself marched out of the door once his helpfulness was discovered.
“Access to control systems was thereafter heavily limited,” said Harry, “and updates had to be pushed by a senior engineer only, on pain of death (well, firing anyway).”
So was James truly the guilty party? Or should Harry have pointed at the mysterious boxes driving the line and said “Here be Dragons” to ward off his protégé? Have you ever taken the initiative, only for it to turn into a catastrophe? Confess all with an email to Who, Me? ®