Every infrastructure architect eventually hits the wall of inline security design: If our security stack crashes, do we prioritize business continuity or perimeter defense?
This decision surfaces across Web Application Firewalls (WAFs), email hygiene gateways, Next-Gen Firewalls (NGFWs), and API proxies. It is often framed as an unavoidable trade-off between two undesirable outcomes: Fail Open or Fail Closed.
What Is Fail Open vs. Fail Closed?
When an inline inspection node suffers an unhandled exception, resource exhaustion, or kernel panic, it must resolve network traffic in one of two ways:
- Fail Open (Bypass Mode): Network traffic flows directly through to the destination without inspection.
- The Upside: Users don’t experience an outage. Business transactions proceed, latency stays low, and uptime metrics stay green.
- The Downside: The perimeter vanishes. Adversaries can intentionally crash an engine (via algorithmic complexity attacks, malformed packet bursts, or memory leaks) specifically to force it into bypass and deliver payloads undetected.
- Fail Closed (Drop Mode): All traffic routed through the failed inspection appliance is dropped.
- The Upside: Security integrity remains intact. No uninspected, potentially malicious byte reaches internal systems.
- The Downside: Total denial of service. For e-commerce checkout APIs, authentication gateways, or customer-facing portals, this means immediate revenue loss and angry customers.
How This Plays Out Across Security Appliances
| Appliance Type | Fail Open Behavior | Fail Closed Behavior | The Real-World Risk |
| Email Spam / Mail Security Gateways | Bypasses AV/phishing filters; sends raw inbound mail straight to exchange/inbox. | Refuses SMTP connections (4xx temporary error or 5xx reject). | In Fail Open, a wave of ransomware or spear-phishing hits inboxes. In Fail Closed, business communication freezes, and external senders get bounce-backs. |
| Web Application Firewalls (WAFs) | Reverts to a plain layer-4/7 reverse proxy, passing SQLi, XSS, and bot traffic through. | Returns 502 Bad Gateway / 504 Gateway Timeout or drops TCP sessions. | In Fail Open, web assets are completely exposed to zero-days. In Fail Closed, customer traffic stops entirely. |
| DDoS Scrubbing / IPS Nodes | Hard bypass via optical/bypass switches; passes unfiltered traffic upstream. | Interface shuts down; link goes dark. | In Fail Open, transit pipes saturate immediately. In Fail Closed, the service goes offline—achieving the attacker’s goal anyway. |
| Identity & Authentication Proxies | Grants fallback access or drops MFA enforcement. | Rejects all logins (Access Denied). | In Fail Open, unauthorized access spikes. In Fail Closed, zero staff or customers can authenticate. |
Why “Fail Open vs. Fail Closed” Is an Architecture Antipattern
Relying on an appliance’s native fallback mechanism assumes that failure must mean compromise or downtime. It accepts single points of failure at the architectural level.
A bypass switch or a static drop rule only addresses what to do after an instance dies. It does nothing to:
- Re-establish active inspection immediately.
- Maintain synchronization of active connection states or security session tables.
- Automatically remediate the underlying service fault.
Choosing between security and uptime is usually a sign that the appliance lacks true Application-Aware High Availability (HA).
Escaping the Binary: How something like SIOS LifeKeeper Solves the Dilemma
Rather than deciding whether your crashed WAF, proxy, or MTA should blindly drop or blindly bypass traffic, SIOS LifeKeeper eliminates the dead-end entirely by keeping the inspection layer highly available.
1. Deep Application-Aware Health Monitoring
Standard hypervisors or cloud orchestrators only look at whether a virtual machine is running. If the underlying Linux OS is alive but the spam filter’s amavisd, clamav, or custom WAF inspection worker hangs in a deadlock, hypervisors see a healthy VM.
SIOS LifeKeeper includes Application Recovery Kits (ARKs) for many applications but also includes the ability to make your own custom “GenApp ARK” to provide customized monitoring that query the application layer directly:
- Is the filtering daemon responding to probe requests?
- Are memory limits approaching threshold panics?
- Are the network sockets actively processing queues?
2. Local Self-Healing First, Failover Second
Before triggering a costly cross-node failover, LifeKeeper can attempt an automated, controlled restart of the misbehaving security service. If the process is irrecoverable, it orchestrates an automated failover.
3. Zero-Data-Loss Configuration & State Replication
Security appliances often depend on dynamic blocklists, rate-limiting caches, and session states. In tandem with SIOS DataKeeper, LifeKeeper provides block-level synchronous or asynchronous storage replication without requiring expensive, shared SAN hardware. The standby node has access to the exact policy sets, certificates, and IP reputation caches the moment it assumes control.
4. Automated Virtual IP and Network Redirection
When failover occurs, LifeKeeper reallocates the cluster Virtual IP (VIP), updates cloud routing tables (e.g., across AWS AZs, Azure Availability Zones, or on-premises subnets), and starts the security services on the secondary node.
The result:
- The system behaves like “Fail Closed” from a security perspective: uninspected packets are never let through blindly.
- The system behaves like “Fail Open” from an uptime perspective: the service recovers automatically in seconds, eliminating prolonged outages.
Real-world example… Protecting ModSecurity with SIOS LifeKeeper
As a widely adopted open-source web application firewall engine embedded within Apache HTTP Server, Nginx, or IIS, ModSecurity sits directly inline to evaluate incoming HTTP traffic against malicious signatures. Consequently, when an unhandled daemon lockup or process crash occurs, infrastructure teams face a stark operational dilemma: expose the application to immediate compromise or accept a total service outage. This makes ModSecurity an ideal scenario for leveraging SIOS LifeKeeper to resolve the trade-off through robust High Availability.
Custom GenApp ARK Monitoring for Web Server Processes
Using the SIOS LifeKeeper Generic Application Recovery Kit (GenApp ARK), administrators can define custom health-check scripts tailored to ModSecurity and its host web server (Apache or Nginx):
- Regularly probe web server worker processes and verify that the ModSecurity engine is actively parsing requests without deadlocks.
- Monitor rule engine memory usage and HTTP response codes to detect silent inspection failures.
Local Service Self-Healing
If the GenApp ARK detects that the Apache or Nginx process hosting ModSecurity has hung or crashed, LifeKeeper first attempts an automated local restart of the web server and WAF module. This resolves transient software faults immediately without interrupting cluster state or switching node ownership.
Automated Failover and Rule Synchronisation
If local recovery fails, LifeKeeper automatically initiates failover to a standby node. In conjunction with SIOS DataKeeper block-level replication, custom SecRule directives, audit logs, and IP reputation tables are continuously synchronised, ensuring the secondary node assumes active WAF inspection with up-to-date protection profiles.
The Verdict
“Fail open or fail closed” is a forced choice that stems from treating security nodes as isolated, standalone appliances.
For perimeter security where data breaches and downtime carry identical financial risk, the goal should be neither. By anchoring your email security clusters, WAFs, and proprietary Linux/Windows security servers to an active-passive or active-active framework via SIOS LifeKeeper, you take the human gamble out of the equation: security stays rigid, and operations stay online.
About the Author: Aaron West is a Solutions Engineer at SIOS Technology Corp, specializing in high availability, open-source solutions, and networking. With over a decade of experience, Aaron has held leadership roles, including Head of Solutions at Loadbalancer.org, where he built and led teams in technical support and pre-sales consulting. His expertise spans network engineering, Linux, and Microsoft systems, with a strong focus on team building and customer-centric solutions. Aaron is passionate about delivering robust open-source technologies and providing tailored solutions for clients across various industries.
