PayGo (www.paygoutilities.com) is a privately held integrated utility payment solution provider that manages the largest energy company prepay programs in the United States, including: Southern Company, Duke Energy, Consumers Energy, NV Energy, Exelon and Ameren. Headquartered in Alpharetta, GA, PayGo’s Pay-as-You-Go utility programs can reach over 25 million energy customers. The company’s cash payment solution, CheckOut by PayGo, enables customers to bypass service lines and pay in regular retail checkout lanes. PayGo solutions not only increase customer satisfaction, but also reduce costs for the utility. A critical way PayGo reduces costs is by operating its platforms in the cloud.
“We have been using SIOS DataKeeper for several years now, and it has proven to be the most rock-solid piece of software we have. It just works.”
Chad Gates, Senior Director of Infrastructure and Security, PayGo
The Environment
PayGo deploys all its production systems in Amazon Web Services (AWS). Their environments utilize Elastic Compute Cloud (EC2) virtual servers with solid-state drive (SSD)-only storage to maximize disk throughput performance. There are currently four production environments in AWS with another coming online soon. For the database backend, PayGo uses SQL Server 2017 Standard Edition running on Windows Server 2012 R2, with plans to migrate to Windows Server 2019 after testing is completed.
The Challenge
| PayGo makes payments convenient and flexible for utility customers. With PayGo’s payment services, customers can use a credit/debit/ACH through our full Merchant Services Provider (MSP) payment application for either a PrePay or a PostPaid account. |
“Our backend SQL Servers hold terabytes of data that must be available 24×7,” explains Chad Gates, Senior Director of Infrastructure and Security. “As a Windows shop, we prefer to use Windows Server Failover Clustering (WFSC) for data protection and continuous operation in case of any failures. But WSFC requires some form of shared storage, like a storage area network (SAN) and that isn’t natively available in AWS.”
With AWS’s lack of shared storage, PayGo was forced to use SQL Server’s transaction logging and log shipping to protect the data. Although requiring manual intervention, this approach was acceptable for disaster recovery (DR) purposes. But it could not provide the rapid, automatic failover capability needed to ensure high availability (HA) for the company’s mission-critical applications.”We had another option, but we believed there were more cost-effective solutions,” according to Chad: “We could use the Always On Availability Groups feature in SQL Server Enterprise Edition, but that would cost us hundreds of thousands of dollars that could be spent on other mission-critical initiatives. We felt there must be a better solution, so we started looking for other options.”
The Evaluation
In its search for a capable and cost-effective HA solution, PayGo established four criteria:
- Seamless integration with Windows Server Failover Clustering
- High disk throughput performance to satisfy demanding recovery point and time objectives
- Ease of implementation and dependable ongoing operation
- Responsive technical support from the vendor
While researching available options, Chad received a recommendation from a colleague to look at SIOS Technology. When Chad did, he liked what he saw. “SIOS DataKeeper Cluster Edition overcame the problem caused by the lack of shared storage. Its use of a mirrored drive looks like shared storage to the WSFC. It was exactly what we wanted.” SIOS DataKeeper also met PayGo’s other three criteria better than any other solution considered.
The Solution
SIOS DataKeeper provides the high-performance, real-time block-level data replication needed to seamlessly integrate with WSFC, which also enables it to support Failover Cluster Instances (FCIs) in SQL Server Standard Edition. The resulting SANless cluster provides SQL Server with continuous monitoring for addressing failures at the application level, and offers the disk performance necessary for automated failover and failback. PayGo first installed SIOS DataKeeper software in its own private cloud, and later migrated the configuration to AWS. “Because SIOS DataKeeper supports private, public and hybrid cloud environments, we migrated the entire configuration, including all application software and data, easily and without any issues,” Chad recalls. PayGo currently has two SQL Server nodes in each of its four SANless HA clusters. To provide protection against localized failures, the servers are deployed in separate Availability Zones. And to ensure high transactional throughput performance, each server has two network interfaces with one dedicated to SIOS data replication. The SANless clusters employ synchronous data replication through the sub-millisecond (ms) latency connectivity AWS delivers between Availability Zones. The IT team at PayGo is currently considering adding DR protection to the HA clusters by deploying a third node in a separate AWS region. The distance involved in this case (between datacenters in Virginia and Ohio) experience a latency of 12-13 ms. While that requires asynchronous replication to ensure high throughput performance in the active node, the combined HA/DR solution would recover much quicker than what is possible with log shipping.
The Results
SIOS DataKeeper met and exceeded PayGo’s high expectations for a high availability solution, including ease of installation and operation, and responsive support. “We have been using SIOS DataKeeper for several years now, and it has proven to be the most rock-solid piece of software we have,” Chad claims. “It just works. So well in fact, we’re tempted to take it for granted.”
Because the applications are mission-critical, PayGo takes nothing for granted. Given its proven operation, including during actual failures, the IT team has minimized the ongoing testing needed for its production SANless clusters. The clusters are now tested only after changes are made to any of the hardware or software, scheduled on a monthly basis, and the test itself consists of a simple failover and failback. PayGo also upgrades only one node at a time in each cluster to simplify roll-back, if needed.
With SIOS DataKeeper performing so well, the only reason PayGo now has for upgrading to SQL Server Enterprise Edition would be outgrowing the Standard Edition’s database size limitation. That is a good problem for a growing company to have!
Get a Free Trial of SIOS DataKeeper
Learn more about SQL Server high availability on AWS