5 Disaster Recovery Disasters to Avoid

/in , /by

No one likes thinking about bad news. Maybe that’s why planning for disaster recovery often doesn’t get the attention it requires. But without a solid plan that’s been documented and tested, your disaster recovery process can turn into a disaster of its own. Take the steps you need to ensure you don’t experience these disaster recovery disasters:

1. Recovery site not ready

There are two backup sites you need to think about when you create your disaster recovery strategy.

The first backup site is where your backups are stored. You need your backups to be stored securely, but also to be available quickly when they’re needed. It often makes sense to keep backups in at least two locations—one, onsite at your primary data center for use in small outages, and two, at your secondary location or in the cloud for use when the primary site is unavailable.

The second backup site is where you bring up your systems when your production site is down. Whether in the cloud or a secondary data center, it’s a good idea to keep this site relatively current with applications and data to allow the recovery process to happen more quickly.

2. Backups not available

You can’t restore systems to their production state without backup copies of systems and data. The problems go beyond potential inaccessibility of the site where backups are stored. Common backup problems include data that was never backed up or backup media that has been corrupted. Another potential problem is that the data backup doesn’t let you easily isolate the exact elements to be restored, forcing you to spend time restoring files that haven’t changed.

3. Recovery process not known

In the middle of a crisis is a bad time to discover you don’t know how to restore your systems. Not being documented at all is a worst-case scenario, but even a thick binder of recovery procedures doesn’t guarantee the process will run smoothly. This isn’t a process your staff is familiar with; maybe they remember it from a once-per-year test scenario, or maybe they didn’t participate in the test and have never executed the process.

In addition, no matter how detailed the recovery documents are, they may not be a match for your situation. Sometimes you’ll need to recover just a single file or a single server, not your entire data center, and comprehensive disaster recovery plans typically focus on the biggest possible outage. You’ll have to figure out a smaller-scale recovery process on the fly.

4. Recovery process takes too long

Every minute your business isn’t operational costs money, so restoring systems as quickly as possible is critical to minimizing the impact of a disaster. You should have a recovery time objective for every system, and conduct tests that verify you can meet those requirements.

5. Recovery process is error-prone

A manual recovery process is vulnerable to user error, and every recovery process is vulnerable to poor communication.

Take steps to avoid disaster recovery disasters before you experience a crisis. Let CCS Technology Group develop a disaster recovery solution that gets you through your disaster without creating new disasters along the way. Contact us to learn more about effective disaster recovery planning.

Additional Disaster Recovery Resources

Craft An Effective Disaster Recovery Plan

The Differences Between Backups, Disaster Recovery, and Archiving Matter

5 Changes to Make When You Switch to Disaster Recovery in the Cloud

Make Sure Your Disaster Recovery Plan Isn’t Just Words on Paper

/in , /by

A written disaster recovery (DR) plan is a good start towards making sure your business can resume operations after an outage, but you won’t know how good those words are until you put them into action. Because you don’t want to find out your plan is incomplete or incorrect during a crisis, it’s important to schedule periodic disaster recovery tests to try out your plan before you need to execute it for real.

Types of Disaster Recovery Tests

There are several different ways you can test your plan:

  • Circulate for comment. Distribute the plan to everyone who would participate in it and solicit their comments and feedback.
  • Walkthrough the plan. Gather everyone who would participate in the plan in a conference room or on a conference call. Read through the plan as a group—out loud, not silently. Because there is group interaction in this approach, you’re likely to surface issues that won’t be identified when individuals read through the plan separately.
  • Tabletop testing. Similar to a walkthrough, the participants are gathered together. Rather than read through the plan in isolation, they are presented with a typical failure situation and called upon to resolve it. This can identify planning gaps and failures that are not addressed by the DR plan. It’s important to choose realistic failure scenarios and that the participants are not informed of the scenario in advance.
  • Parallel test the plan. Bring up the disaster recovery systems and test whether they can execute a day’s work. The production systems run in parallel, so the only impact on routine business is that some personnel have to perform tasks on the disaster recovery systems.
  • Failover test. Simulate a production outage by gracefully shutting down the primary servers and failing over to the secondary site. This test method impacts ordinary production work so it may be better to execute this process on a weekend or other low volume time period. This process requires additional work to bring the primary servers back online after the test is complete.

Learn more in Craft An Effective Disaster Recovery Plan.

Disaster Recovery Test Follow-up

Whichever test strategy you choose, the test process isn’t over when the final system is brought back online. After the test, the DR plan needs to be updated to reflect:

  • missing applications. It’s not uncommon for applications to be overlooked when the DR plan is written.
  • missing or incorrect steps. The processes for bringing up applications may be missing some steps, miss some dependencies, have steps in the incorrect sequence, or contain errors in the details of the commands to be executed.
  • incorrect timings. Every application should have a recovery time objective which the recovery plan attempts to meet. If the test shows recovery can’t meet those objectives, the plan needs to be revisited to determine how it can be altered.
  • missing communication. Plans often fail because important notification steps are omitted.

In addition, you should always consider how the plan would have worked if this was an actual, unscheduled outage.

Learn more in Don’t Improvise Your Way Through Disaster Recovery.

Repeat the Test

If there were major failures during the test, take time to revise the plan to reflect those problems and then schedule another test to verify the corrections. If the recovery process mostly worked as planned, you can wait until your next regularly scheduled test—usually annually, though some prefer twice annually or even quarterly—to test the update.

CCS Technology group offers disaster recovery planning services. Disaster recovery testing is an important part of your business continuity strategy. Contact CCS Technology Group to learn more about writing and testing your DR plan.

Choose the Right Backup Strategy to Meet Time and Space Requirements

/in , /by

There are multiple reasons businesses need to backup their data. You need the ability to restore data if it gets lost or corrupted, or if a disaster requires shifting processing to an alternate site. Compliance policies may require retaining copies of data for a lengthy period of time. Analytics projects may need years’ worth of history, and new software projects often require copies of production data for development and testing.

Given all these reasons for making backups, implementing an effective backup process is a critical IT function.

Types of Backups

All critical systems need to be backed up daily, but not every piece of data needs to be backed up every day. There are different kinds of backups that allow the process to run more efficiently.

  • Full backup. Every system needs a full backup to be made at least once. This is a complete copy of the data, and serves as a baseline state for the system.
  • Differential backup. A differential backup includes all the data that changed since the last full backup.
  • Incremental backup. An incremental backup includes only data that changed since the last incremental backup.

Once you’ve made a full backup, you can use either differential or incremental backups to copy only the changed data. This makes the backup process faster and requires less storage space. However, it makes the recovery process longer, as recovering means first restoring the full backup and then applying the changes on top of that.

Creating a synthetic full backup every week or month allows you to use incremental backups and shorten the recovery process. On a regular schedule, the incremental changes are applied to the last full backup. This effectively creates a current full backup that can be restored rapidly.

Backup Capabilities

In addition to the different types of backups described above, there are some backup features that can help speed recovery in specific scenarios.

  • Snapshots. A snapshot is a copy of a dataset at a specific point in time. Unlike backups, snapshots are typically stored on the same device as the original data. This makes them suitable for recovering rapidly, but you can’t recover from them if the device fails.
  • Replication. Replication copies data changes to a second site nearly instantaneously. This allows recovery with almost no downtime if the primary device fails. However, the second site only has the latest copy of the data, so it doesn’t support recovery if data is corrupted or deleted or if an older version is required.
  • Deduplication. Much data is stored in multiple places throughout an organization. Deduplication reduces the size of backups by identifying and reducing this duplicate data. However, recovery times are made longer by the need to reverse this process.

With all these options, choosing an appropriate backup strategy requires careful consideration. Contact CCS Technology Group to develop and implement a backup solution that protects your data and your business.

Additional Backup Resources

Effective Backups Need to Address These Challenges

The Differences Between Backups, Disaster Recovery, and Archiving Matter

Understand the Different Cloud Options for Your Backup and Disaster Recovery Strategy

Don’t Improvise Your Way Through Disaster Recovery

/in , /by

Given the importance of disaster recovery (DR), you don’t want to improvise through the planning—or worse, through the execution. Here are some best practices to make sure your disaster recovery follows an effective script:

1. Assign staff to disaster recovery

It sounds obvious, but if you don’t have staff assigned to disaster recovery, it isn’t anybody’s job, and it won’t get done. You need staff who are dedicated and empowered to make sure disaster recovery is properly planned. This isn’t limited to technology staff either; business employees have roles and responsibility in disaster recovery as well.

2. Develop a detailed plan

If you don’t want to improvise, you need a documented plan. The full contents of a DR plan are beyond the scope of this short blog post, but you need to start by identifying all of your IT resources. Evaluate the impact of an outage on each application and use that to determine your DR priorities. Then assess how much time you can tolerate the application being down and how much data you can afford to lose. Use those numbers to guide you in developing a cost-effective recovery strategy. Document the recovery steps in detail, and make sure the recovery plan will be available in case of a disaster.

3. Test your recovery plan

It’s far better to discover your DR plan won’t work during a test rather than during a disaster. Schedule time to test your plan, at least annually. There are different ways of approaching testing, ranging from a table read-through of the documentation to fully executing the steps to failover and resume operations at a secondary site. The more your test simulates a real disaster, the more reliable results you’ll get. Track the time it takes to recover as well as the accuracy of the documented procedures. After the test, collect feedback from all participants on what worked and what didn’t, and use it to update the document.

4. Update the plan

Changes in your business and your technology mean the plan that worked last year may not work this year. Allocate time to review and update your plan every year—even better, make updating the plan part of your change management process and don’t sign off on deployments until the recovery process is documented.

5. Don’t go it alone

For many businesses, leveraging Disaster Recovery as a Service (DRaaS) is a good choice that makes disaster recovery faster and more reliable. With DRaaS, you get a high level of automation and support from the provider to help guide you through the process of defining and implementing a recovery strategy.

Another way to avoid going it alone is to work with an IT services firm like CCS Technology Group. Our disaster recovery and business continuity services help you protect your data, reduce downtime, and survive a crisis. Contact us to learn how CCS Technology Group can help you write your disaster recovery script.

Additional Disaster Recovery Resources

Craft An Effective Disaster Recovery Plan

5 Changes to Make When You Switch to Disaster Recovery in the Cloud

Backups Are Not A Disaster Recovery Solution

Don’t Let These Obstacles Get in the Way of Your IT Security

/in , , /by

Information security should be a top priority for any business. You don’t make any money by having good information security practices, but you can lose a lot of money if you don’t: this year, the average cost per record of a data breach was $150, according to the Ponemon Institute. Multiply that number by the size of your database and you can see how the costs quickly mount up.

So if a lack of information security can be so costly, why are there so many data breaches? One reason is that it’s impossible for any defense to be 100 percent effective; there’s always the risk that one malware author will get lucky and break through. But more often, it’s because although companies know information security is important, it isn’t really a priority. There are too many obstacles that get in the way of implementing effective security:

  • Manual processes. When processes like patch updates and vulnerability scans need to be performed manually, it’s easy to make errors or neglect to apply them to some systems.
  • Complex infrastructure. Except for a brand-new startup, every business has a jumble of technology. Different hardware, different operating systems, different operating system versions, multiple software products, and cloud systems make it difficult to develop a comprehensive approach to security that can cost-effectively protect all resources.
  • Lack of budget. In most businesses, IT is a cost center, and that means limited budget that needs to be allocated between projects that help the business grow and projects that add security to protect the business.
  • Employees don’t use safe computing practices. How many computers do you walk past with passwords written down on sticky notes? Information security is everybody’s responsibility, but many companies don’t do a good job educating their non-IT employees about safe computing, including strong passwords and recognizing phishing attacks.
  • Overworked, under-trained IT staff. IT staff is often overwhelmed and spends most of its time fighting fires and putting out today’s problems. Getting training on the latest security threats and their defenses isn’t top priority and isn’t always in the budget.
  • Changing threats. The scope and source of security threats is constantly changing. It’s not just about dealing with new variants of existing malware. There are new kinds of malware, such as ransomware, which has been devastatingly effective in numerous instances. There are also new attack vectors, including mobile devices, the internet of things, and the cloud.
  • Lack of business support. Business management is focused on the business, not IT. They sometimes see information security measures, such as preparing and testing an incident response plan, as a distraction.

Security services from CCS Technology Group can help you overcome these challenges. Our proactive approach closes holes that make you vulnerable to current attacks and implements layered security and defense in depth strategies that help guard against future attacks. Contact us to learn more about how CCS Technology Group can help you protect your business.

Additional IT Security Resources

Closing the Most Common Cybersecurity Holes

The Key Features to Look for In Your Firewall

Phishing 101: What it is, how it works and how to avoid it

Effective Backups Need to Address These Challenges

/in , /by

Backups are conceptually easy, but implementing an effective backup process isn’t nearly as easy as it sounds. Making a backup process that really works has to address these challenges:

Long backup windows

Creating backups that are consistent and usable means related files can’t undergo any changes during the backup process. This may require shutting applications down for the duration of the backup process. As businesses become 24×7 operations, this downtime becomes an unacceptable impact on the business. Even a backup process that’s acceptable now may not scale effectively as the volume of data increases.

Unmonitored backups

Backups often run unattended and unmonitored. Problems with the backup may never be discovered until it’s too late to correct them. Even if backups are monitored and the support team responds to an alert, rerunning the backup after the problem is corrected can take too long and impact business operations.

Inability to restore data

The whole point of backups is to be able to restore data and get systems up and running again. Backups stored offsite may take too long to access when needed. In addition, the restore process is often untested and unfamiliar to the support teams so they struggle with it in a crisis. Even when performed smoothly, the restore process may take too long. When trying to restore older data, changes in data models or applications may cause the restore process or application to fail; successfully accessing this older data may require restoring an older version of the application as well.

Unprotected backups

Backup data media is exposed to several vulnerabilities. First, in order to be accessible, backups may be stored at the primary data site. This means any physical damage at the data center—flood, fire, or other problem—may damage the backup media or make it unavailable. The second big risk is that backups are often not encrypted. Anyone who has access to the media can access any of the data it contains.

Expensive backups

Backup media, backup storage, software licenses, and support staff all cost money. Although backups are vital, they’re also infrequently used, so managing costs and the ROI of your backup process is important.

Complicated backup management

Backups can’t always be centrally managed and tracked; depending on your backup tools, they may need to be installed, monitored, and managed on every system requiring backups.

CCS Technology Group helps our clients develop comprehensive business continuity strategies that provide backup and disaster recovery solutions to protect your critical data and applications. Contact us to learn more about how to implement a backup solution that addresses these backup challenges.

Additional Backup Resources

Craft An Effective Disaster Recovery Plan

The Differences Between Backups, Disaster Recovery, and Archiving Matter

Understand the Different Cloud Options for Your Backup and Disaster Recovery Strategy

Craft An Effective Disaster Recovery Plan

/in , , /by

If you don’t want to be scrambling in the middle of a crisis, you need a plan. Here’s what to think about as you develop your disaster recovery plan to make sure you get out of the situation and back into normal operations fast:

Communications plan

There’s bound to be lots of confusion during an incident, but you don’t want there to be any confusion about who’s in charge. Make sure your plan identifies who decides to invoke the disaster recovery plan and how this will be communicated to everyone who needs to be involved in the recovery.

Scope of potential threats

Crises come in all sizes, from a single accidentally deleted critical file to a fire that destroys your primary data center. Spend time assessing a variety of possible situations and determine how you’ll match your response to the size of the outage.

Lists of systems and people

You’ll need a complete list of all hardware and software that your business uses, as well as network diagrams. Also create a list of all the staff you’ll need to help bring the systems back online, including their contact info. Include contact info for third parties, such as vendors and partners, that may need to make changes on their side to connect to your recovery site.

Priorities and targets

It isn’t possible to bring up all systems at the same time, and it usually isn’t necessary. Take your list of systems and evaluate the priority of each system so you know where you need to focus your effort. For each system, set a specific recovery time objective and recovery point objective, specifying how rapidly you need to restore that system to operation and how much data you can afford to lose. Once you know these numbers, you can craft a recovery strategy for each application to meet those targets.

Recovery procedures

Document the details of the recovery procedures for each application, including the complete details of the commands that need to be executed. Identify the other processes the application depends on in order to start up. Include validations that allow you to confirm the application is running properly in its recovery mode.

Fallback procedures

Once the disaster is over, you’ll want to resume operations in your normal production environment. Executing fallback processes can be as complex as the disaster recovery procedure itself, so document the process to the same level of detail.

Once your disaster recovery plan is complete, schedule a test to validate that it works. Then update the plan with any corrections, clarifications, or critical information that was missed the first time around. Because your infrastructure changes continually, your plan should be a living document. When you place new resources into production, you should also update your plan to include them. The entire plan should be periodically reviewed and tested, at least annually, to make sure there are no omissions and that it works with your current infrastructure.

CCS Technology Group provides comprehensive disaster recovery services. Contact us to find out how you can make your plan more effective.

Did you know three out of four small businesses have no disaster recovery plan at all? Learn more in Why a Business Continuity Plan is Essential.

Additional Disaster Recovery Resources

7 Common Mistakes That Place Your Data in Danger

Backups Are Not A Disaster Recovery Solution

The Differences Between Backups, Disaster Recovery, and Archiving Matter

The Differences Between Backups, Disaster Recovery, and Archiving Matter

/in , /by

Backup, disaster recovery, and archiving all create or use copies of data, but they have different purposes and objectives. We’ve talked before about how backup is not disaster recovery; backups are also not an archive solution.

Know the Purpose of Backups, Disaster Recovery, and Archiving

Here’s a quick reminder of the purpose of these three processes:

Backups Are Data Copies

Backups are simply data copies; that’s all. Backups don’t do anything to the original data, and the purpose of a backup is to be able to restore the original data if something happens to it. If a file is corrupted or accidentally deleted, it can be replaced with an undamaged copy.

Disaster Recovery Isn’t Just About Data

Disasters are almost any scenario that brings down systems in a data center, including equipment failures, fires, and weather conditions. Data may be damaged and need to be restored, but first you need to get servers and possibly entire data centers back online.

Archiving Preserves Data

Archives provide unchanged historic copies of data to meet legal and compliance requirements. Unlike backup files, which may be kept for only a short while, archives are kept for the long term. You need quick access to backup files in order to restore files rapidly and minimize the impact of lost data, but archives are not used by routine business operations and can be stored in low-cost, off-site locations. Working with an archive may require using special e-discovery software that can search through large data stores to find records relevant to a legal process.

Don’t Use A Backup Tool as an Archive Tool

It may seem that you can create your archive simply by keeping your backup tapes (or other backup media) instead of recycling them. That’s a shortcut that will create many problems in the long term. Backups aren’t tagged in any way, so searching them for data is difficult. In addition, backups don’t let you easily delete data.

Why would you delete data from an archive if the purpose of an archive is to preserve data? Storage costs money, so keep data only as long as legally required. There may also be legal or other risks if older records are exposed. Making sure data is preserved and deleted appropriately requires a workflow that backup tools can’t support.

Don’t Use an Archive Tool as a Backup or Disaster Recovery Tool

An alternative would be to take the opposite approach. If your archive contains all the copies of your data, why do you need separate backup and disaster recovery tools? Couldn’t you just extract the necessary data from the archive?

First, if your archive is kept on a lower tier of storage, retrieving and restoring data can’t happen as fast as you need during an outage. More important, archives simply aren’t built to manage a data restoration process, which requires getting a specific file from a specific location on a specific data.

Although they sound similar, backups, disaster recovery, and archiving are all unique processes that require distinct tools and strategies. CCS Technology Group can help you make sure you have the right solution in place to meet specific backup, disaster recovery, and archiving needs. Contact us to learn more.

Additional Business Continuity Resources

Understand the Different Cloud Options for Your Backup and Disaster Recovery Strategy

Don’t Lose Your Files to Ransomware

5 Changes to Make When You Switch to Disaster Recovery in the Cloud

Understand the Different Cloud Options for Your Backup and Disaster Recovery Strategy

/in , /by

Effective backup requires more than simply making another copy of a file. You need to track the files you’ve backed up, provide appropriate security, and know how to restore them when needed. If you’re planning to backup files in the cloud, it’s important to know how to use the different options to get the right level of protection.

Cloud Storage

Cloud storage simply provides a remote filesystem for you to use. How you use the available space is up to you; depending on the cloud provider’s capabilities, you may be able to access it as a local filesystem. Unlike local filesystems, the capacity is unlimited, and you pay only for the capacity you use. An additional advantage of cloud storage is that cloud providers usually have several regions, allowing you to store data in a different geographic location.

Cloud Sync

Cloud sync copies folders from your local filesystem to a filesystem in the cloud. This is often used to share files so they can be used from anywhere, making them production data rather than a backup. Depending on the vendor, cloud sync may or may not allow you to access older versions of files. 

Cloud Backup

Cloud backup operates like traditional backup software, but with the cloud rather than a local filesystem as the target. The software operates on a schedule to backup changes to the cloud, with historic versions preserved. Cloud backup can be implemented with backup software running in the cloud or in your local data center. Cloud backup give you more control than cloud sync with respect to when and how data is duplicated. Cloud backup often uses compression and deduplication to reduce the space and cost of the backed-up data; it may also apply encryption for security. 

Cloud Disaster Recovery

It’s important to note that getting data out of the cloud is often more difficult and more expensive than getting data into the cloud. Cloud disaster recovery provides additional support needed to restore files and virtual machine images in case of an outage. Disaster Recovery as a Service (DRaaS) uses high levels of automation to bring systems online in the cloud rapidly.

Understanding the different capabilities between these cloud services is key to implementing an effective backup and disaster recovery strategy in the cloud. CCS Technology Group combines its cloud expertise with our business continuity insight to develop, implement, monitor, and support effective cloud-based backup and disaster recovery solutions. Contact us to learn more about how your backup can leverage the cloud to ensure a smooth backup and disaster recovery process for your business.

Additional Disaster Recovery Resources

Don’t Lose Your Files to Ransomware

5 Changes to Make When You Switch to Disaster Recovery in the Cloud

Backups Are Not A Disaster Recovery Solution

Don’t Lose Your Files to Ransomware

/in , , /by

Think about that panicky feeling you get when you lose one file. Now scale that feeling up and imagine the panic after losing all your files. That’s how you’ll feel if a ransomware attack makes it impossible for you to access any of your data.

Ransomware is a kind of malware that holds your data hostage. When you’re attacked by malware, it encrypts all your data. Since you don’t have the key, you aren’t able to read it. Typically you’re asked to make a payment in cryptocurrency in exchange for the key. If you don’t pay up by the deadline, the key is discarded and your data is lost for good.

Ransomware can be difficult and time-consuming to recover from; one town had to rely on typewriters when their computers were down after an incident. If you don’t have typewriters tucked away in a closet, here are some options to help prevent and respond to ransomware incidents.

Prevent Ransomware Attacks

It’s impossible to completely protect yourself from a ransomware attack; like any other malware, they spread through phishing and social engineering methods that trick your employees into opening dangerous attachments. Training employees is important but not foolproof.

Keeping up with your operating system patches is an important measure, as it reduces the number of vulnerabilities for hackers to exploit. You should also use antivirus software and whitelisting software to block malware and prevent unapproved applications from executing.

Ensure you have a reliable backup and disaster recovery process. This won’t prevent you from becoming a ransomware victim but will reduce the panic if you do.

Recover from a Ransomware Attack

The first thing to know about recovering from a ransomware attack is that you should never ever pay the ransom! For one thing, there’s no guarantee that you’ll receive the decryption key. Plus, once you pay ransom, you’ve shown that you’ll pay ransom, and you make yourself a target for additional ransomware attacks with bigger and bigger ransom demands.

Identify the ransomware that attacked you and see whether there’s a decryptor. This will let you recover your locked files without paying the ransom.

If there isn’t a decryptor (and it’s really not that likely you’ll find one for the exact version of the attack that victimized you), you’ll need to do a scan to remove the malware from your system and then restore files from a clean backup. Unfortunately you’ll lose any new files or modifications made between the time the backup was created and the time you were encrypted—good motivation for doing backups at least nightly. You’ll need to make sure the backup isn’t infected with the malware as well, as some ransomware can attack shared drives.

Then protect yourself from future attacks by hardening your cybersecurity strategy and making sure your backups aren’t vulnerable, perhaps by storing them in the cloud. CCS Technology Group information security services help you develop and implement an approach that protects you against ransomware and the many other common malware threats that target your systems. Contact us to learn more.