Many years ago, in what now seems like another lifetime, I was given the task of upgrading a customer’s software. A simple enough job. Go on-site. Perform the upgrade. A couple of hours, and I’ll be back in the office for lunch.
I arrived on site by 8:00 a.m. and the first step was to perform a backup of the database and software. Completing the backup and verifying the tape was accomplished in about 45 minutes or so. Now all I needed to do was to remove the existing installation, install the new version of the database and then restore the customer data.
Because it was a Unix system, I promoted myself to super user and proceeded to execute the command to remove the software directory. I issued a simple command that involved removing all files and folders from the current directory, and I added a modifier to not stop and ask for confirmation as the command ran.
What could possibly go wrong?
I stood watching the screen scroll as each file and directory was deleted, and then it hit me. I think my heart skipped a couple of beats as I realized I was in the wrong directory. Instead of in the application directory I was in the root directory – that is – at the lowest level of the operating system. What I was doing was deleting everything. I mean everything. As the operating system dissolved before my eyes no amount of CTRL+C commands that could stop it. After a few minutes the system ground to a halt. I had deleted enough of the operating system that there was nothing left to run.
Eventually I found enough fortitude to go to the customer and tell them what happened. I now needed to reinstall the operating system too. Ok, probably won’t get back to the office for a while longer.
A leading question
You know what’s coming next right?
The conversation went something like this as I recall it:
“John can I have the last operating system backup tape?”
“What about backup tape? Do you mean the daily one of the database?”
“No. The operating system only backup that you do weekly.”
“I’ve never done one.”
“Oh.”
“Why do you need it?”
“Well I just deleted your system – completely.”
“Why did you do that?”
“I was in the wrong directory.”
“What are you going to do now?”
“Do you have anything stronger than coffee?”
So, I had to get an operating system CD from the hardware vendor sent out by courier so I could reinstall the operating system. After that, reconfigure the system and finally complete the original task of upgrading.
So much for getting back to the office for lunch. I got home around 10:00 p.m. that evening after finally getting the customer back up and running. Fortunately, the customer was very understanding and accepted a whole day of being non-operational with incredible grace.
But what is the moral of this story?
Simple. Always backup your system and data. While I had a backup of the data that I was dealing with, I did not ensure that there was a backup of the operating system.
Daily backups aren’t optional for on-premises systems
All too often in Support we face the issue of helping customers with non-existent
backups. While we can provide the software if it gets lost, we can’t get data back. I have no doubt that everyone realizes the importance of regular data and operating system backups. But do they happen as often as they should? Probably not.
In my years with DocuWare, I have seen the categories of backup plans:
- Every day without fail
- Whenever a member of the IT staff remembers to do one
- Never done at all
Support can work some magic at times, but I have yet to discover how to recover an irreparably corrupt database or a database that has been deleted when there are no backups at all, or they are so old as to only have minimal value. With a mission critical application there is just no excuse for not having complete verified and current backups and a documented and proven disaster recovery plan.
This blog post is not about how to perform those backups – that is up to the individual customer to research and consult with industry professionals to ensure that their document management system and other software is thoroughly protected. Even the freshest backup is useless if it cannot be read when you go to use it. All backups should be verified regularly and where practical there should be multiple redundant backups of everything with at least one backup copy kept off-site.
DocuWare components that must be included in every on-premises backup plan
- DocuWare Databases
All DocuWare databases should be backed up daily.
A note on MySQL. Backing up MySQL by copying the file system folders is not a reliable method of backing it up. If the copy is taken while MySQL is running, you can bet that the files will be unusable when you need them most. Always backup MySQL with data dumps. - Operating system
Physical servers should have the system state (a full backup of the operating system) backed up regularly. Virtual machine snapshots taken daily. - Import jobs
If your import jobs are run on your DocuWare server, then a server backup will preserve those, otherwise the workstation(s) where those jobs run should also be backed up regularly. - File system
Let’s not forget the actual documents. File system backups should also be conducted daily.
Your exposure to disaster can only be limited by your latest verified backup.
Take another approach and move to the cloud
Cloud services providers assume responsibility for system upgrades and applying security patches. If the customer has accidentally deleted documents, these can be restored if necessary. Also, if documents are changed in error, previous versions can be restored.
For example, to enable a recovery, DocuWare backs up both the databases and the documents in a separate location in a Microsoft Azure data center. A copy of each document is made and stored in a continuous backup. This happens shortly after the document has been saved or modified in DocuWare. The backup after document modification creates a new copy of the document. This is saved in addition to existing document backups. By manually importing the backup data into the live system, a DocuWare organization can be completely restored in cooperation with DocuWare Support.
In addition to backing up the documents, full backups for the SQL databases are stored at a separate data center. This backup is done on weekends, usually overnight.
Whether you continue with your on-premises system or move to a cloud-based option – it’s your data, protect it!
