HA NGINX Load Balancing

IBM Bluemix Infrastructure How-To:
Configure Active/Passive HA NGINX Load Balancing on CentOS Servers

 

Overview

Load balancers offer a great way to automatically distribute traffic across a pool of servers.  Not only does it afford the opportunity to add or remove servers based on resource needs, but, whether a server in the pool goes down for maintenance or because it has some sort of failure, it also assures that you have as much availability for your service as possible when a server becomes unavailable.
 
There can still, however, be a single point of failure in a load balancer implementation.  If you only employ one load balancer in the setup, and if that load balancer fails, then your entire server pool could become unreachable.  The objective of this document is to demonstrate how you can create an Active/Passive Highly Available pair of load balancers utilizing some open source operating systems, like CentOS, and web server software, like NGINX.  NGINX is a powerful, open-source web server package that has many capabilities, including a load balancing functionality that has been used by companies large and small.

 

Objectives and Outcomes

This guide will take you through how to order, configure, and deploy and test an active/passive, highly-available load balancer and web server solution on IBM Bluemix Infrastructure, using virtual servers to proxy inbound HTTP traffic from the public network to the private network.  This solution utilizes two Linux-based load balancers, three Linux-based web servers, a Secondary Portable Private IP subnet, and a Secondary Portable Public IP subnet to accomplish this objective.
 
We will configure the solution to accept HTTP traffic on the public network, proxy the traffic to the private network, and keep all the servers on the same VLAN, for two specific reasons.  The first is that it keeps your web servers secure.  You can turn the public network interfaces of your web servers off, thus negating any sort of risk you may face from intrusion attempts.  The second is that by keeping everything on the same VLAN, you can take advantage of the native intra-VLAN network and avoid any unnecessary network hops, thus lowering latency and increasing performance.

Diagram

Here is a simple diagram of what we are trying to accomplish:

 

Servers and Services Needed

2 x Virtual Servers with CentOS (6 or 7) Minimal Install: These will be used to install NGINX for the load balancers.

3 x Virtual Servers with CentOS (6 or 7) LAMP Install: These will be used as the simple Web Servers.

/30 or /29 Portable Private Subnet: For Private Network load balancing, we will use these IP addresses to bind to Apache HTTPD service as listeners.  This way we can use the Primary Private IP addresses for management purposes.  It also gives us the added benefit of being able to move the Portable Private IP addresses to new servers if we have a problem with one of our web servers.

/29 Portable Public Subnet: We will use one of these IP addresses as the "floating" or "virtual" IP for our load balancers.

*Note*: This can also be accomplished using Bare Metal Servers for the NGINX Load Balancers, in place of virtual servers.  If you need more than 1Gbps throughput (the current maximum for IBM Bluemix Infrastructure Virtual Server Instances (VSIs), or if you need more compute power (CPU, RAM, etc.), our Bare Metal Server offering is available with up to Dual 10Gbps Public and Private connections, as well as multiple CPU and RAM configuration options.

 

Order Virtual Servers

For this example, we will be ordering the servers with minimal resources (1 x virtual core, 1GB RAM, and 25GB primary disk), as the services and configurations we will be loading are very light on resources.  In production, you will want to scale these resources to meet your needs.

To learn more about IBM Bluemix Virtual Server Instances (VSIs), follow this link: http://knowledgelayer.softlayer.com/topic/virtual-server.

Once servers are provisioned:

This guide uses nano as the text editor, which is included in CentOS 6.  If you're going to use CentOS 7, vi is the included text editor.  If you're fine using vi, then you can skip this step and replace all the following nano commands with equivalent vi commands, otherwise you can run this command to install nano:

sudo yum install nano

 

Order IP Addresses You Will Need

As mentioned earlier, you will want to use a Public IP address as an entry point, but proxy that incoming traffic to the Private Network connection for each web server for security, and keep them all on the same VLAN for reduced latency and increased performance.

Public IP Type

For the Public IP address, you will want to order a subnet of Secondary Portable Public IP addresses.  You will only need to use one of these for this basic setup, as it will be serving one NGINX load balancer until it fails, and then will be automatically failed over to the secondary device if/when the primary device fails to communicate.

Ordering Public IPs
  • Log into the portal at https://control.softlayer.com.
  • Navigate to the Network tab, hover over IP Management, and select Subnets.
  • In the top right-hand corner, select Order IP Addresses.
  • From the drop-down list, select Portable Public.
  • Select the number of IP addresses you wish to purchase.

*NOTE*: The number of IP addresses listed is the number of raw IP addresses you will receive.  This does not take into account the assignment of the Gateway, Network, and Broadcast IP addresses that will be reserved in this subnet.  Also, due to the use of HSRP in almost all of our data centers, an additional two IP addresses will be unusable.  Since we are only using one of these IP addresses, we will order a /29 subnet (8 IPs), which leaves us 3 usable IP addresses.

  • Click Continue.
  • Select the Public VLAN on which your NGINX servers reside.
  • Click Continue.
  • The next page is an IP justification, which our IP reviewers use to make sure we are efficiently using our IP addresses, as IPv4 addresses are now are a limited resource.
  • Once completed, check the "I agree…" box, check the "I have read…" box, and click Place Order.
  • Once the IP addresses are provisioned, we can move onto configuring our NGINX servers to use them, which we will cover later, in the Install and Configure Keepalived section.

Private IP Type

For the Private IP addresses, you will want to order a Secondary Portable Private IP subnet.  You will want to order a subnet large enough to have one for each web server.  You don't need to worry about the load balancers using Portable Private IP addresses, as you can't force NGINX to proxy load balanced traffic from a specific IP address.

Ordering Private IPs
  • Log into the portal at https://control.softlayer.com.
  • Navigate to the Network tab, hover over IP Management, and select Subnets.
  • In the top right-hand corner, select Order IP Addresses.
  • From the drop-down list, select Portable Private.
  • Select the number of IP addresses you wish to purchase.

*NOTE*: Because we will be using these IP addresses for load balancing purposes, only, we will open a ticket after the IP addresses are provisioned to convert the subnet from a Secondary on VLAN subnet to a Routed on VLAN subnet.  This allows us to do without the Gateway, Broadcast, and Network addresses, meaning we can use all of the IP addresses in the subnet without losing any.  The subnet that most closely matches your IP needs for this example is a /30 (4 IPs).

  • Click Continue.
  • Select the Private VLAN on which your NGINX and web servers live.
  • Click Continue.
  • The next page is an IP justification, which our IP reviewers use to make sure we are efficiently using our IP addresses, as they are a limited resource.
  • Once completed, check the "I agree…" box, check the "I have read…" box, and click Place Order.
  • Once the IP addresses are provisioned, we can move onto configuring our servers with these secondary IP addresses, which will be covered in the Assigning and Configuring Secondary Private IP Addresses to Servers section.

 

Assigning and Configuring Secondary Private IPs

Assign a Secondary Private IP Address to Each Web Server

By default, Bluemix virtual servers have three network interfaces:

  • 1: lo - This is the loopback interface, which we will not be modifying or working with in this tutorial.
  • 2: eth0 - This is the Private Network interface, which is the one we will be adding the secondary IP to.
  • 3: eth1 - This is the Public Network interface.  It is not needed, as we will be proxying traffic to the Private Network.  Since we won't be using the Public Network on the web servers at all, once the configurations are done this interface should be disabled in the Control Portal.
  • On each of your web servers, pick one of the Private IP addresses that we ordered in the previous section of this document.  On each server, run the following command:

sudo nano /etc/sysconfig/network-scripts/ifcfg-eth0

  • This will enter the text editor so that you can add the second IP address.
  • At the end of the file, press enter to add a new line, and type this in:

IPADDR2=[whichever IP address you have chosen for this specific machine]

  • Make sure you document which IP address will be used on which server to make troubleshooting easier should something go wrong, and save and exit the nano editor.
  • Run this command to restart the network stack on the machine, which will then pick up on the changes:

sudo service network restart

  • Repeat these steps on each of your web servers.
  • Once the network stack has restarted, and you have regained access to the machine, you will want to ping test from each server to the others to make sure that they are communicating properly.

 

Install and Configure Keepalived on NGINX VSIs

Install Keepalived

  • On each machine that will be an NGINX load balancer, run this command to install Keepalived:

sudo yum install keepalived

Configure Keepalived

  • Once you see the notification that the install is complete, use this command to edit the Keepalived configuration file:

sudo nano /etc/keepalived/keepalived.conf

  • Your output will have many lines that you don't need.  Unless you need all of the options outlined in this config file, you can delete all of the contents, and use just a small portion.
  • For the NGINX load balancer that you would like to be the primary, use this snippet.  The values you should change are highlighted:

! Configuration File for keepalived
vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.10.121
    }
}

  • For the NGINX load balancer that you would like to be the secondary, use this snippet.  Again, the values you should change are highlighted:

! Configuration File for keepalived
vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.10.121
    }
}

  • The virtual_ipaddress parameter should be changed to the same IP on both servers, and you should use one of the Secondary Public Portable IP addresses you ordered in a previous step.
  • Note that you can change the auth_pass value to any four-digit value you would like, as long as the value is the same on both servers.
  • Now you can run the command on each server to start Keepalived:

sudo service keepalived start

  • You will also want to set the services to start automatically when the machine is powered on:

sudo chkconfig keepalived on

Configuration Changes to Bind an undefined IP

Next, we will need to make a configuration change to the Linux base kernel.  By default, Linux will not allow you to define an IP address that isn't currently bound to an interface, which in turn will cause the NGINX service (that we will install in the next step) to throw an error about identifying an IP address that isn't currently mapped to an interface.  To change this, follow these steps:

  • On each NGINX load balancer, run this command:

sudo nano /etc/sysctl.conf

  • At the end of the .conf file, add a line of code that says this:

net.ipv4.ip_nonlocal_bind = 1

  • Save and exit the .conf file
  • In order to enable this configuration after the changes have been made, run this command:

sudo sysctl -p

Testing

  • You can verify that the Portable Public IP address is being assigned to each server.  On the primary server, type this command:

ip addr show eth1

  • If your configurations worked, you will see the Portable Public IP address listed as one of the connections on the eth1 interface.
  • To verify that the IP gets automatically assigned to the secondary load balancer, shut down the primary server, and run the same command on the secondary server:

ip addr show eth1

  • If the configurations on the second server are correct, then the portable IP address should now show up for the eth1 interface on the secondary load balancer.
  • Once the primary load balancer comes back online, you can verify that the IP address is reassigned to the primary server.  If your configurations work as they are supposed to, then it should automatically reassign the IP.

 

Install and Configure NGINX

Install NGINX

  • Use the following command to open and edit the NGINX repo file:

sudo nano /etc/yum.repos.d/nginx.repo

  • Enter the following into the file:

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

  • Save and exit, then use the two commands below to update the package repository, and install NGINX:

sudo yum update
sudo yum install nginx

  • Once the install is complete, you will need to start the nginx service using the following command:

sudo service nginx start

  • Test to make sure that NGINX installed correctly by opening a web browser and navigating to the machine's public IP address.  You should get a page that looks similar to this, but may vary depending on versioning:

  • If you don't see this page, make sure a firewall is not blocking the connection.

Configure NGINX

  • Edit NGINX config and enable load balancing with upstream:

sudo nano /etc/nginx/conf.d/load-balancer.conf

  • In the following example, you'll need to define the following two segments: upstream and server.

upstream backend {
server [web server 1 portable private ip address];
server [web server 2 portable private ip address];
server [web server 3 portable private ip address];
}
 
server {
listen [Portable Public IP Address you chose for Floating IP]:80;
 
location / {
proxy_pass http://backend;
    }
}

#This tells the nginx server to accept all traffic to port 80 and pass it to the upstream.  Notice the upstream name and the proxy_pass need to match.

  • The highlighted portions will be where you define the portable IP addresses you ordered and configured on the web servers in the earlier section.
  • Save the file and exit the nano editor.
  • Disable the default server configuration you tested earlier (the "Welcome to NGINX!" message).

sudo mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.disabled

  • You will need to restart the NGINX service again, in order for these changes to take effect:

sudo service nginx restart

  • The "Welcome to NGINX!" message should no longer be displayed if you try to refresh the browser you tested in earlier, or try to navigate to the load balancer's public IP address again.
  • You will also want to set the NGINX service to start automatically when the machine powers on:

sudo chkconfig nginx on

A Note on Load Balancing Methods

When configured as we have in this example, NGINX will use round-robin, if no other method is defined.  This will balance connections to each server, in turn, based on the order in which they appear in the load-balancer.conf file.  While this example uses round-robin, NGINX can employ other types of load balancing, such as:

  • Least Connections

A new request is sent to the server with the fewest current connections to clients. The relative computing capacity of each server is factored into determining which one has the least connections.

  • IP Hash

The IP address of the client is used to determine which server receives the request.

To enable the Least Connections load balancing method, add the least_conn parameter to the upstream setion of your load-balancer.conf file, as in the example below:

upstream backend {
     least_conn;
     server [web server 1 ip];
     server [web server 2 ip];
     server [web server 3 ip];
}

To enable IP Hash load balancing method, use the ip_hash parameter to the upstream section of your load-balancer.conf file, as in the example below:

upstream backend {
     ip_hash;
     server [web server 1 ip];
     server [web server 2 ip];
     server [web server 3 ip];
}

For more detailed information on the load balancing methods for NGINX, visit this link: https://www.nginx.com/resources/glossary/load-balancing/.

 

Configure Apache on Web Servers to Listen on Specific Private IP

It's been said throughout this guide that we want load balanced traffic to proxy to the servers' private IP addresses.  Now, we are going to configure the Apache web server on our CentOS web servers to listen for HTTP traffic specifically, and only, on the Secondary Private IP addresses we assigned to each server.

  • Run this command to edit the httpd.conf file:

sudo nano /etc/httpd/conf/httpd.conf

  • Find the line that defines the listen options.  By default, Apache will listen for HTTP requests on all open interfaces with the command listen 80.  To set it to listen only on the private interface, specifically the Private Portable IP address that we defined, change that command to this:

listen [web server portable ip address]:80, where you define your Portable Private IP for the specific web server you are modifying.

  • Once this is done, you will need to restart the Apache HTTPD service, using this command:

service httpd restart

  • And, since we're here, let's set the Apache HTTPD service to start automatically when the machine powers on:

chkconfig httpd on

 

Customize Apache Web Server Splash Page

Before we test this final configuration, we will want to make some changes to the default Apache web server splash page.  Customizing this will allow us to identify each of the web servers as we reload the page to test the round-robin load balancing configuration we created.  Follow these steps to create a quick, custom web page that will show up when you connect to each server:

  • On each web server, open this file in nano:

sudo nano /var/www/html/index.html

  • Once the file is open, input this line of text:
    • For the first web server:

<h1>Server 1 works!</h1>

  • For the second web server:

<h1>Server 1 works!</h1>

  • For the third web server:

<h1>Server 1 works!</h1>

  • Save and exit the nano editor on each server.
  • Restart the httpd service on each web server:

sudo service httpd restart

 

Test the Solution

It's now time to test the round-robin load balancing, and the floating IP failover solution.  You will want to have SSH sessions open for each NGINX load balancer in order to run the command to initiate the fail over.

Test the Round-Robin Load Balancing Method

  • Once those SSH connections are established, open a web browser on your local computer and navigate to the public "floating" IP you assigned to your load balancers.
  • You should see something similar to this:

Server 1 Works!

  • Now, refresh the page, and you should see something similar to this:

Server 2 works!

  • Refresh the page one more time, and you should see something similar to this:

Server 3 works!

If this is what you see, then you know your primary load balancer and web server configurations are working correctly.  Every time you refresh this page, it should cycle through the different web servers in numerical order, unless one of the web servers is not responding properly.

Test the HA Floating IP Fail Over

To test the high-availability fail over portion of this, we will manually initiate the process that keepalived should do automatically.

  • On the primary NGINX load balancer, run this command to make sure it currently has the floating IP address assigned:

ip addr show eth1

  • This should show the server's primary public IP address, as well as the floating IP address.
  • Initiate the failover by stopping the keepalived service:

sudo service keepalived stop

  • Now if you run ip addr show eth1, the floating IP should now be gone from the list.
  • On the secondary NGINX load balancer, run the ip addr show eth1 command to make sure the floating IP was assigned properly.
  • Once the floating IP failover has been confirmed, refresh the web page for the floating IP that we tested earlier a few more times, and it should cycle through the servers numerically, just as it did before.
  • To fail the load balancers back over to the primary server, run this command on the primary NGINX load balancer:

sudo service keepalived start

  • Once again, run the ip addr show eth1 command on the primary server to make sure the IP address failed back over to the primary load balancer as it should have.
  • For completeness' sake, run the ip addr show eth1 command on the secondary load balancer, as well, to make sure the floating IP is no longer listed in the eth1 stack.

 

Conclusion

If all of your testing above went as it should, then you're done building the underlying infrastructure for an Active/Passive HA NGINX Load Balancer solution.  Once the configuration and testing is done, you can build (or import) your website on the web servers, in lieu of the test message, and rest easy that your service's availability now has redundancy built in at not only the web server level, but also at the load balancer level.

 

Get Started With Your Own Deployment

If you'd like more details about how to get started on your own infrastructure with this guide, you can reach our 24/7 Sales teams at the contact information listed here: http://www.softlayer.com/contact.