A warning appears, services stop responding, tickets begin to pile up, and someone says, “We need to do something!” That instinct to begin recovery activities immediately is understandable, as during an incident, action feels productive, while waiting can feel irresponsible. Under this kind of pressure, doing almost anything can feel better than doing nothing.
However, some of the most damaging decisions in disaster recovery are made because nobody acted, but because someone acted before understanding what had happened. The philosopher, Marcus Aurelius, often wrote about the importance of separating an event from the judgment we form about it. We first receive an impression of what has happened, and then our minds quickly form an explanation. If we do not pause to examine that explanation and the actions leading to the circumstance, we can begin reacting to an assumption as though it were a fact.
Why Impulsive Incident Response Can Increase Risk
Suppose a server becomes unreachable, for example. The immediate conclusion may be that it has failed, but all we actually know is that we cannot communicate with it. It might still be running while a network problem prevents us from seeing it.
That difference matters in a highly available environment, as moving an application to another server manually might restore service, but it could also create a situation where both servers believe they should be active. In high-availability environments, this kind of condition is often referred to as a split-brain scenario: two systems each acting as though they have ownership of the same application or resource. A response intended to improve availability for end users can introduce a risk to the application or its data.
We can observe the same problem occurring during ordinary troubleshooting of non-critical components. Restarting a service may clear an issue, but it also changes the conditions we were trying to understand. Once the restart is complete, useful evidence about the original problem may be gone. We may have restored service without learning what happened or whether it is likely to happen again.
The Difference Between an Observation and an Assumption
None of this indicates that teams should stand still during an outage, as Aurelius was not advocating indecision, and restraint should not become an excuse for delay or inaction. The point is to act based on what we know rather than what we fear might be happening. I think that distinction is easy to lose when an incident becomes stressful. People want updates, alerts continue to appear, and periods of silence on a conference call can feel longer than they really are. Someone may suggest a reboot because it worked last time a critical incident of this nature was observed.
So, the suggestion begins to sound like the plan, even though the current problem may have a different cause, yet present similar symptoms as the previous issue. Experience can help, but it can also create shortcuts in our thinking. Recognizing a familiar symptom is useful; assuming it must have the same cause as the last incident is not. Similar symptoms can come from very different problems.
A more disciplined response starts by stating only what has been confirmed. Instead of saying, “The server is down,” a better approach may be, “The server is not responding from this location.” That wording may seem like a minor detail, but it keeps the team from treating a conclusion as an observation. It also leaves room for another person to report that the server is reachable from somewhere else.
Why Disciplined Troubleshooting Matters During Outages
This is where incident response becomes more than technical knowledge. It requires control over the urge to solve the problem before the problem is understood. Sometimes, one additional check is enough to change the direction of an investigation and the next steps.
A second monitoring location might show that the application is still available internally, and a local console might confirm that a supposedly failed server is healthy but isolated. That information can prevent an unnecessary recovery action and point the team toward the actual problem.
The Role of Automation in Disaster Recovery
Well-designed automation follows a similar principle. Automation is valuable because it can respond consistently and does not need to wait for an administrator to wake up or join a call. However, speed alone does not make an automated response correct.
An automated system should act when the conditions for recovery are clear. When the available information is incomplete or contradictory, the safer behavior may be to stop and seek more evidence. Highly available systems account for this in several ways. Independent communication paths can help distinguish the failure of one connection from the loss of an entire server. Quorum or witness mechanisms can provide another perspective when systems can no longer communicate with each other. These controls are important because a system’s view of the environment may be accurate but incomplete.
How SIOS LifeKeeper Supports Smarter Recovery Decisions
LifeKeeper can support this decision-making through resource monitoring, defined dependencies, and recovery policies. The technology helps carry out an established recovery plan, but it cannot decide what level of risk is acceptable for a particular business. That judgment has to be made by people while the environment is stable, not improvised after an incident begins.
Building Better Incident Runbooks for Disaster Recovery
Clear procedures make restraint easier. A good incident runbook should help the team establish what is known before making a consequential change. It should explain how to confirm whether an application is already active elsewhere and identify who has the authority to initiate recovery.
The goal is not to remove human judgment. It is to give that judgment a reliable foundation when time is limited. We cannot eliminate uncertainty from technology; hardware will fail, networks will behave unpredictably, and applications will occasionally surprise the people who know them best. What we can do is prepare ourselves to recognize the difference between an event and our first explanation of it.
Discipline in Disaster Recovery
Aurelius returned to that idea because it applies most when circumstances are difficult. Clear judgment is easy when nothing is at stake. Its value becomes relevant when pressure makes the fastest answer feel like the only answer. During an incident, the calmest person in the room is not necessarily doing nothing. They may be making sure that the next action solves the problem that is actually happening. In incident response, discipline is not the absence of action. It is the refusal to let pressure choose the action for you.
Protect critical applications with high availability and disaster recovery solutions built for complex IT environments. Request a demo to see how SIOS LifeKeeper can help your team reduce downtime and recover with confidence.
Written by Aidan Macklen (Associate Product Support Specialist)
