IT Disaster Stories
The following stories inspired our slogan "Because Tomorrow is Too Late." Doing things right always seems more costly at first, BUT, when disaster strikes, that wise investment could be what saves your business.
The following stories are 100% REAL and have happened to actual companies right here in Winnipeg. Due to confidentiality, we cannot reveal their names.
The T&M Nightmare
This story is about a local company in Winnipeg who just purchased new accounting software. Their IT Provider at the time was called in to get the software installed and running. Their contract was for time and material (T&M). The company was using T&M because that was the Service Provider's only option. They would call if they wanted work done or when something didn't work. It seemed like the right choice... at the time.
One day, some of the data in the new accounting system was corrupted and jumbled around. An employee immediately called their current IT Provider. The Provider contacted the accounting software's tech support to determine the best course of action. After some diagnostics, they said the quickest and least expensive way to go was to restore the accounting data from the night before.
When the IT Provider tried, they realized the backups were still backing up the old accounting data and not the new one. They had no way of restoring the data from the night before because it was never backed up. The accounting software company offered to try to fix all of the corrupted data (which would not be free) – with no guarantee the data could even be retrieved.
The President of the IT Provider rushed in for an emergency meeting with the company's CEO to explain what happened to their data.
The CEO asked the IT Provider: "Why didn't you change the backups to back up the new accounting system?"
The IT Provider's response: "You only paid us to install the accounting system, not to change the backups."
In not so nice terms, the CEO told the IT provider to leave the boardroom. Then, he immediately called us as we had recently been speaking with him about true managed services. The CEO wanted to see if we could sort the situation out. The only solution to the problem was to hire the accounting software company to fix as much as they could and then rekey the missing information. The out-of-pocket cost for both the software company's work and the downtime to the company was devastating.
This never would have happened if this company had used a Managed IT Services provider like us. With a monthly fixed fee, we become your IT Department. We provide ALL of the services you require and are constantly looking out for your business.
Where the 'TRUCK' are my backups?!?!
This story is about a transportation company who barely survived a data disaster.
This company regularly moves items across the Canadian and United States border. They had acquired new software that enabled them to electronically transmit required documents to the border.
Everyone was excited about the new software – and rightly so. This was an efficient system that eliminated a lot of manual paperwork – until the unthinkable happened.
One day their computers stopped working. No one could work, no trucks could cross the border, and all shipments were on hold – then the customers started calling.
In a panic, the President called in their current IT Provider for help.
The IT Provider diagnosed the problem, and then told the President of the company:
IT Provider: "Your hard drive failed and that's why the server crashed and everyone stopped working."
President: "You told me if one drive fails, we would still be running and could just replace the failed drive?"
IT Provider: "Yeah, but this was the second drive to fail. One drive failed months ago. You've been running your network with no spare drive."
President: "Why didn't you replace the first drive that failed – I think it is still under warranty?"
IT Provider: "You aren't paying us to monitor your IT systems so we didn't know. There should have been a red light or even a yellow blinking light that should have caught your attention."
President: "What lights!? No one ever told us about any lights we need to watch out for!"
The President asked for their 'Plan B'. The IT Provider replaced the failed drives and got the server up and running. Now all they had to do was restore the backup.
And now, the real horror story begins...
When the IT Provider finished restoring the data, they realized something was missing.
The backup software they were using was not configured to back up the database the new software was using. They lost all that information. This meant they now had to manually re-enter data.
President: "Didn't you check to make sure we were being backed up!?"
IT Provider: "You don't pay us to do that."
Exasperated, the client called us as we came highly recommended by another company. We responded immediately to the emergency. We couldn't fix the past, but we helped them pick up the pieces and move forward in the right direction.
We put in place a backup and disaster recovery plan that automatically backs up all files – eliminating human error. We constantly test and monitor the backups to ensure they are fully restorable all of the time. With our backup and disaster recovery plan, we've got your back.
A local Winnipeg company had a full-time, internal IT person that managed their day-to-day network. He was skilled, but being only one person, his overall knowledge was limited. The company outsourced projects the internal IT guy didn't know how to do.
This company came to us for advice on their network including their current backup and disaster recovery plan. We gave them a comprehensive plan that included:
- Making sure their onsite backup tapes were in a fireproof and secure safe
- Taking some of the backup tapes offsite each night as extra insurance
- Testing the backups to make sure they could be restored if needed
The company took our advice, but didn't want to pay for an outside company to check their backups, take them offsite, or test them. They chose to rely on their current internal IT Person to take care of it.
One day, there was a fire in the office. Their server melted into a 'blob.' Surprisingly, minus a few melted parts, the drives looked salvageable.
If their IT guy had followed our recommendations, there would have been no problem. The company would have backup tapes offsite that could have restored the data.
As it turns out, for the past 30 days, their IT guy forgot to take the backup tapes offsite each night.
Now for plan B:
We sent the damaged drives to a data recovery company to see if they could recover the lost data. Their quote was $40,000.00 – if they even could retrieve the data. The company now faced a tough decision.
- Don't pay $40,000.00 and re-key all of the lost data for the past 30 days
- Pay $40,000.00 and have all of their data restored
Either way, this company had just lost a ton of money.
The company decided it was less expensive to manually re-key all of the missing data for the past 30 days.