Author: Alexus Gore, Customer Experience Software Engineer at SIOS Technology Corp
Having a failover environment configured is only one part of completing the setup. Validating and verifying whether your configuration will work as expected is the next and most important step. Here are ways you can be confident your infrastructure can withstand a disaster.
Build a QA Cluster That Mirrors Production
A QA cluster is typically a replica of the production cluster that is used for testing and/or implementing any changes that need to be introduced into the cluster. With environments that contain multiple failover solutions, it’s necessary to have an environment that allows for testing to see how the environment behaves. When solutions work separately, there is sometimes an assumption that they will seamlessly work together. This may not always be the case. Finding out that solutions do not work well together could happen during a failover. If the order of operations is not tested and verified to work, you may experience a failure rather than a failover, possibly due to the order of operations. Having a QA environment is useful in mitigating and preventing issues arising when HA/DR response is needed most.
Test Failover and Switchover Scenarios
Testing is crucial in validating your failover setup. Testing helps ensure that an environment behaves properly when a failure scenario happens. Simultaneously, testing is important to verify if the behavior of the environment aligns with your expectations. There are circumstances where you may not want to begin a failover immediately, which is why it helps to understand the difference between switchover and failover testing.
Verify Logs Before Go-Live
Before going into production, always check the logs from your system and the failover solution, such as SIOS LifeKeeper for Linux. Ideally, you will want to verify that within your environment all needed systems, clusters, and infrastructure elements can communicate with each other to ensure smooth operation. Checking the logs for any anomalies is also helpful in understanding how your environment is behaving, especially during testing, should your expectations not be met.
Document Everything in a Runbook
Always have a Runbook to document procedures for the environment and to outline the tests that should be executed. Documenting the process of setting up the environment and testing the environment helps beyond having an understanding of what you are doing and the information that is being shared. Documentation makes sure processes are repeatable, but it also serves as a record that you can audit if needed. By keeping documentation of how procedures should be performed and the tests that should be verified, a paper trail can be maintained so any analyses performed after an issue opens up room for improvement.
During this phase of validating your configuration, it is not expected that everything will be 100% successful; failures are inevitable. When they do happen, a runbook provides documentation on the steps that led to the situation. Runbooks and documentation are important because they provide insight into the events that did – or did not – happen in the lead-up to an issue. With thorough documentation, you get the ability to adjust procedures and prevent these circumstances from creating an issue in your production environment.
Confirm Application Setup Meets Your Failover Solution’s Requirements
Certain failover solutions may differ in their requirements for application protection. Setting up the applications is sometimes not as simple as making sure the application itself works, but making sure the application and the failover solution work well together. Verify with the respective documentation on how applications need to be set up.
For example, the LifeKeeper for Linux documentation outlines a requirement to modify some default settings to allow proper protection of an SAP environment, as seen in the modify ASCS and ERS instance profile settings for SAP page.
Considerations like these are crucial to ensure that your HA/DR solution and protected applications work together cohesively and grant the most value from the solutions in use.
When deploying an environment, there are a lot of factors at play. With this checklist, rollouts can be much more efficient and manageable. This checklist will help ensure your environment is validated and ready for production, giving you confidence in the life of your deployment. Want an expert to check your setup? Schedule an HA Health Check with SIOS.
About the Author: Alexus Gore is a Customer Experience Software Engineer at SIOS Technology Corp. She focuses on helping customers optimize their high availability and disaster recovery environments, ensuring their systems are resilient and reliable.
