Using Squid Proxy Instances for Web Service Access in Amazon VPC: An Example

Articles & Tutorials>Using Squid Proxy Instances for Web Service Access in Amazon VPC: An Example
This article provides all the necessary resources, including easy-to-use python scripts and instructions on leveraging squid proxy instances, to implement a high availability (HA) solution for accessing AWS and other web services and scaling based on throughput, eliminating the Network Address Translation instance as a choke point.

Details

Submitted By: Jinesh Varia
AWS Products Used: AWS CloudFormation, Amazon VPC
Created On: June 7, 2013 5:27 PM GMT
Last Updated: November 5, 2013 1:21 AM GMT

by Rodney Lester, AWS Professional Services Consultant

Learn how you can leverage squid proxies in a high availability (HA) solution for accessing resources from private subnets without the choke point of the Network Address Translation (NAT) instance. This article provides all the necessary resources, including easy-to-use instructions on how to create an architecture where squid instances auto scale based on inbound network traffic, eliminating potential network contention on a NAT instance.

Overview

In Amazon Virtual Private Cloud (VPC), you can use private subnets for instances that you do not want to be directly addressable from the Internet. Instances in a private subnet can access the Internet without exposing their private IP address by routing their traffic through a Network Address Translation (NAT) instance in a public subnet. A subnet can only use one NAT, however, and the largest network throughput is limited to the largest available to an instance type. This situation is depicted in the diagram below.

Figure 1a: Internet-bound traffic through a NAT instance
Figure 1b: Internet-bound traffic congested due to NAT congestion

One approach to this situation is to leverage a farm of squid proxy instances that can scale up and down with the average network traffic on each instance. This walkthrough provides instructions for building a HA scenario where squid proxy instances in separate Availability Zones (AZ) auto scale up and down with inbound network traffic.

This approach, where we use the auto scaler to launch new instances, now has a feature to automatically assign public IPs in the launch configuration.

To create this squid proxy farm, follow these steps:

  1. Create an Amazon Virtual Private Cloud.
  2. Create an Amazon Elastic Cloud Compute (EC2) role in AWS Identity and Access Management (IAM).
  3. Create an Auto Scaling launch configuration for your squid proxy instances.
  4. Create an internal Elastic Load Balancing load balancer for your squid proxy instances.
  5. Create an Auto Scaling group using this launch configuration.
  6. Create Auto Scaling scale-up and scale-down policies.
  7. Create Amazon CloudWatch alarm when the average network traffic exceeds a threshold and have that alarm trigger the scale-up policy associated with the Auto Scaling group.
  8. Create CloudWatch alarm when the average network traffic reduces below a lower threshold and have that alarm trigger the scale-down policy associated with the Auto Scaling group.

This will launch a fully functional sample stack as shown in the diagrams below:

Figure 2a: Basic 2 AZ VPC with public and private subnets
Figure 2b: Auto scaled squid instances
Figure 2c: Auto scaled squid proxy instances used by servers in the private subnet

For the purpose of this article, we will deploy the following:

  1. An AWS CloudFormation template for deploying Amazon VPC with two public subnets, two private subnets, security groups for the instances, network ACLs, route tables, and an Internet gateway.
  2. An AWS CloudFormation template for deploying an Amazon EC2 role for the EC2 Instances that will host the squid software. Also, the template deploys an Auto Scaling Launch Configuration that specifies the packages and files to install, the internal load balancer, the Auto Scaling group, the Auto Scaling policies, and the Cloud Watch Alarms that will invoke those policies.

Step 1. Create an Amazon Virtual Private Cloud

Start by provisioning our networking infrastructure using our CloudFormation automation service.

To do this, you can use this two_az_vpc.template in the AWS CloudFormation console in the AWS Management Console or click the Launch Stack button below to launch this stack

Launch now

Click the Continue button to see the parameters for this template. The default parameters will create an Amazon VPC in two Availability Zones with 2 subnets in each AZ. One subnet will be a public subnet with its default route for other than local traffic going through an Internet Gateway, and one will be a private subnet with no route to the Internet.

Figure 3: AWS CloudFormation SquidExampleTwoAZVPC Stack default parameters

Click Continue. When prompted for additional tags, click Continue as we have no immediate need for additional tagging.

You see a review of the stack. If you changed any, you can verify the change here. Click Continue to launch the stack. Click Close to close the dialog.

Select the stack named SquidExampleTwoAZVPC in the list of stacks and then click the Events tab to track the progress of the creation of the VPC.

You can click Refresh to see new events. When the event is CREATE_COMPLETE, this step is complete.

Note: If the stack creation fails, try creating it in different Availability Zones. For example, in the Specify Parameters screen of the Create Stack wizard, try changing the default zone from us-east-1d to us-east-1a.

Figure 4: AWS CloudFormation SquidExampleTwoAZVPC Stack with Create Complete

At this point, we have an Amazon VPC with 4 subnets, and 10 security groups.

Step 2. Create Amazon Autoscaling Group for Squid Proxies that Will Be Accessed via Internal Elastic Load Balancing Load Balancer and Scale Up and Down Based on NetworkIn Metric

Now that you have created the helper instance to process messages as instances are launched by the Amazon Auto Scaler, we can actually launch our squid proxy farm. We will do this with an AWS CloudFormation template that creates the following:

  1. An IAM role for the instances.
  2. An Auto Scaling launch configuration that will install the squid and keep the service running after reboot.
  3. An Elastic Load Balancing load balancer that is only available internally to the VPC that will balance the requests among the farm.
  4. An Auto Scaling group that will create instances of the launch configuration and automatically associate them with the Elastic Load Balancing load balancer.
  5. An Auto Scaling policy to increase the number of instances by 50%.
  6. An Auto Scaling policy to decrease the number of instances by 33%.
  7. An Amazon CloudWatch alarm to trigger the Auto Scaling policy up when average bandwidth exceeds 750 MB.
  8. An Amazon CloudWatch alarm to trigger the Auto Scaling policy down when average bandwidth goes below 250 MB.

To do this, you can use this squid_autoscaling.template in the AWS CloudFormation console in the AWS Management Console or click the Launch Stack button below to launch this stack.

Launch now

Click Continue to see the parameters for this template.

Figure 5: AWS CloudFormation SquidExampleAutoscaledSquidFarm Stack parameters

Click this link to open another window for the CloudFormation console.

All the parameters except for the SquidInstanceType and SquidKeyName are outputs from the first template.

Back in the other AWS CloudFormation console, select the SquidExampleTwoAZVPC stack and select the Outputs tab to view the ouptuts of this stack.

Figure 6: AWS CloudFormation SquidExampleTwoAZVPC Stack outputs

Copy the value of the VPC to the parameter VPCID. Copy the value of the DMZ subnets (between DMZ: and the first period) to the parameter SquidSubnets. Copy the value of the private subnets (between Private: and the next period) to the parameter SquidLoadBalancerSubnets. Copy the value of the Security Group SquidELBSG to the parameter SquidLBSecurityGroup. Copy the value of the Security Group SquidSG to the parameter SquidSecurityGroup.

Enter your desired SSH key pair to the parameter SquidKeyName.

Click Continue. When prompted for additional tags, click Continue there as we have no immediate need for additional tagging.

You see a review of the stack. Click Continue to launch the stack. Click Close to close the dialog.

Select the stack named SquidExampleAutoscaledSquidFarm in the stack list and then click the Events tab to track the progress of the creation of the farm.

You can click Refresh to see new events. When the event is CREATE_COMPLETE, this step is complete.

Figure 7: AWS CloudFormation SquidExampleAutoscaledSquidFarm Stack Create Complete

Step 3. Test your configuration.

All done! Now it's time to test your configuration. We will test in several steps. We'll start by testing the Auto Scaling of the squid farm for replacing unhealthy instances. We'll then add an auto scaled set of servers to download a very large file on S3 in multiple pieces, saturating the bandwidth to the instance. As we add instances to the group manually, we will observe the squid farm scale up with more instances. We will then remove instances from the auto scale group and watch the squid farm scale down as a result.

Test Basic Auto Scaling of Squid Instances

It is simple to test the basic Auto Scaling. The easiest way of doing this to open the Amazon EC2 console and click Instances in the left  navigation pane. Click this link to open the Instances page of the Amazon EC2 console.

Figure 8: Squid instances in the Amazon EC2 console

Select the check box next to one of the two Squid VPC instances, but not both. Click Actions at the top of the page and select Terminate.

Figure 9: Terminate confirmation in the Amazon EC2 console

Click Yes, Terminate to terminate the instance. As you watch, you will see the instance status change to shutting down, followed by terminated. Once it is terminated and detected to have failed by the load balancer, the auto scaler will start another instance to replace it. This should be relatively quick, within a minute. You can click the refresh icon on the upper right to see when the new instance launches. It will start with a status of pending, then progress to running. Once it is running, it will not be fully replaced until the Status Checks show 2/2 checks passed.

Figure 10: Replaced squid instance in the Amazon EC2 console

Manually set the capacity of the squid farm

You can also set the number of instances running manually in the command line. To do this, return to the CloudFormation console, select the SquidExampleAutoscaledSquidFarm stack, and click the Resources tab.

Figure 11: Replaced squid instance in the Amazon EC2 console

Scroll through the resources until you find the SquidAutoscalingGroup. The Physical ID column contains the name of the group to use on the command line. You can manually set the capacity as follows (replace the name of the group with your actual name):

[ec2-user@ip-10-0-0-11 ~]# aws autoscaling set-desired-capacity --auto-scaling-group-name SquidExampleAutoscaledSquidFarm-SquidAutoscalingGroup-AU0QLT2ZI4KL --desired-capacity 4 --region us-east-1

If you watch the Amazon EC2 console, you'll see two instances launch, but then two will shut down and enter the terminated state in the next 30 minutes. What has happened here?

Figure 12: Only two running instances in the Amazon EC2 console

Recall that the squid_autoscaling.template configured a CloudWatch alarm that fires when the NetworkIn metric is less than 250,000,000 bytes/minute. We do not have any traffic through these servers, so this alarm will fire, causing the auto scaler to terminate instances back to the minimum of the two we configured. Open the CloudWatch console and click ALARM in the left navigation pane to see alarms in the ALARM state.

Figure 13: ALARM states in the Amazon CloudWatch console

Run a Load-Inducing Auto Scaling Group

Caution: Running this test will incur a significant cost! We can stimulate the scale up alarm by creating an Auto Scaling Group that will download a very large file in an infinite loop. To saturate the bandwidth, the script will split the file into pieces and make HTTP Range GET requests. The file will not actually be written to disk, but to /dev/null.

To do this, you can use this squid_loadtest_autoscaling.template in the AWS CloudFormation console in the AWS Management Console or use the Launch Stack button below to launch this stack

Launch now

Click Continue to see the parameters for this template.

Figure 14: AWS CloudFormation SquidExampleAutoscaledLoadtestFarm Stack parameters

All the parameters except for the LoadtestKeyName and LoadtestInstanceType are outputs from the previous stacks we created. You can get the ProxyHost from the SquidExampleAutoscaledSquidFarm stack.

Open the AWS CloudFormation console in another window.

Select the stack named SquidExampleAutoscaledSquidFarm in the stack list and then click the Outputs tab to see the outputs from this stack.

Figure 15: AWS CloudFormation SquidExampleAutoscaledSquidFarm Stack outputs

Copy the value of the SquidDNSName to the parameter ProxyHost. The value of the ProxyPort can stay the default (3128).

Back in the other AWS CloudFormation console, select the SquidExampleTwoAZVPC stack to see its outputs.

Figure 16: AWS CloudFormation SquidExampleTwoAZVPC Stack outputs

Copy the value of the VPC to the parameter VPCID. Copy the list of values of the private subnets in the Subnets output to the parameter LoadtestSubnets. Copy the value of the WebServiceSG in the SecurityGroups output to the parameter LoadtestSecurityGroups.

Enter the name of your desired SSH key pair for LoadtestKeyName. The values of the LoadtestBucket, LoadtestKey, DownloadTargetBucket, and DownloadTargetKey can stay the defaults. (These are publicly available objects.)

Click Continue. When prompted for additional tags, click Continue as we have no immediate need for additional tagging.

You see a review of the stack. Click Continue to launch the stack. Click Close to close the dialog.

Select the stack named SquidExampleAutoscaledLoadtestFarm in the stack list and then click the Events tab to track the progress of the creation of the instance.

Click Refresh button to see new events. When the event is CREATE_COMPLETE, this step is complete.

Figure 17: AWS CloudFormation SquidExampleAutoscaledLoadtestFarm Stack with Create Complete

Once the stack is formed, it will start downloading the file in an infinite loop. This will saturate the bandwidth of the instances and trigger the scale up alarm. You can watch the alarm in the CloudWatch console. Observe the alarm that starts with the name SquidExampleAutoscaledSquidFarm-SquidNetworkBandwidthHighAlarm to observe when it goes into alarm.

Figure 18: Amazon CloudWatch SquidNetworkNadwidthHighAlarm in ALARM state

The auto scaler will start a new squid instance in the farm and add it to the load balancer. Once the instance is up and running, it will then balance the HTTP requests across the three instances, which will put the alarm back in the OK state.

Figure 19: Amazon CloudWatch SquidNetworkNadwidthHighAlarm in OK state

You can continue to scale up the example by using the command line to increase the number of instances in the load test Auto Scaling Group. The name of the Auto Scaling Group to use on the command line is the output from the SquidExampleAutoscaledLoadtestFarm stack.

Figure 20: AWS CloudFormation SquidExampleAutoscaledLoadtestFarm stack outputs

[ec2-user@ip-10-0-0-11 ~]# aws autoscaling set-desired-capacity --auto-scaling-group-name SquidExampleAutoscaledLoadtestFarm-LoadtestAutoscalingGroup-QYNVWM1ZK82D --desired-capacity 4 --region us-east-1

To remove all the loadtest instances, you can delete the SquidExampleAutoscaledLoadtestFarm stack. Simply go to the AWS CloudFormation console, select the SquidExampleAutoscaledLoadTestFarm stack, and then click Delete Stack.

Figure 21: AWS CloudFormation console, SquidExampleAutoscaledLoadtestFarm stack deletion confirmation

Running this test has created nine squid proxy instances in the farm. They will be deleted at 33% down capacity, every 15 minutes as the scale-down alarm fires.

Figure 22: Amazon EC2 console, squid proxy farm contents at end of load test

When the low bandwidth alarm fires, you will see the auto scaler shut down instances:

Figure 23: Amazon EC2 console, squid proxy farm Auto Scaling shutting down instances

You can watch the alarm in the CloudWatch console. Observe the alarm that starts with the name SquidExampleAutoscaledSquidFarm-SquidNetworkBandwidthLowAlarm to see when the alarm fires.

Figure 24: Amazon CloudWatch SquidNetworkNadwidthLowAlarm in ALARM state

Eventually, it will scale back to the minimum number of two servers.

Figure 25: Amazon EC2 console, the number of squid proxies back to minimum

Step 4. Remove your test environment.

To clean up the resources, you can use the AWS CloudFormation console. You must delete the SquidExampleAutoscaledSquidFarm stack before you can delete the VPC. Select the check box in the left column of the stack, and then click Delete Stack. You can click Refresh on the upper right to see when deletion is complete. Once it is deleted, you can delete the SquidExampleTwoAZVPC stack to remove the remaining resources.

Appendix A: Tuning Instance Sizes and Alarm values

The instance size chosen in this example, m1.xlarge, has High I/O performance. Your needs may require higher throughput on individual HTTP requests, so you may need Very High performance.

The alarm values used in this article - 750,000,000 and 250,000,000 - were chosen to illustrate this example. Your application may need other values tuned to your usage and cost analysis. If you change instance types on the proxies, you will definitely want to run some benchmarking to establish what values you wish to use.

Appendix B: loadTestProxy.py code and discussion

You can download the code that processes the messages upon instance launch. It is also reproduced below:

#
# Copyright 2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
# http://aws.amazon.com/apache2.0
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.

from __future__ import print_function
import boto
from boto.s3.connection import S3Connection
import os
import sys
from threading import Thread

def main():
proxy_host = sys.argv[1]
proxy_port_used = sys.argv[2]
bucket_name = sys.argv[3]
object_key_name = sys.argv[4]

s3_c = S3Connection(proxy=proxy_host, proxy_port=proxy_port_used)
bucket = s3_c.get_bucket(bucket_name)
object_key = bucket.get_key(object_key_name)
numThreads = (object_key.size / (250*1024*1024)) + 1

The top is Apache 2.0 license, followed by the import of the packages used. The main() function is the actual processing. The command line arguments are parsed. A connection is then made to the Amazon S3 endpoint through the proxy. The bucket and key are used to get the size of the object, so we can download in multiple 250 MB chunks.

 cur_thread_num = 0
threads = []
while cur_thread_num < numThreads:
range_low_calc = cur_thread_num*250*1024*1024
range_high_calc = (cur_thread_num+1)*250*1024*1024 - 1
thread_name = "s3-transfer-manager-worker-" + str(cur_thread_num)
new_thread = Thread(None, downloadS3Part, thread_name, args=(s3_c, bucket_name, object_key_name, range_low_calc, range_high_calc))
new_thread.start()
threads.append(new_thread)
cur_thread_num = cur_thread_num + 1
for thread in threads:
thread.join()
s3_c.close()

An empty list of the threads is created, and then a loop is entered to spin off the threads. Each thread is an HTTP Range GET, which uses the function downloadS3Part to perform the task. Then each thread is waited upon for completion. Once all are done, it will close the connection.

def downloadS3Part(s3_c, bucket_name, object_key_name, range_low, range_high):
bucket = s3_c.get_bucket(bucket_name)
object_key = bucket.get_key(object_key_name)
range_headers = dict({'Range' : "bytes=" + str(range_low) + "-" + str(range_high)})
local_fp = open("/dev/null", 'w')
object_key.get_file(fp=local_fp, headers=range_headers)
local_fp.close()
return

The downloadS3Part function will use an HTTP Range GET to download the part of the file and write it to . It then returns control up the stack.

if len(sys.argv) < 5:
print("Usage: " + sys.argv[0] + " ")
os._exit(os.EX_DATAERR)

while True:
try:
main()
except:
print('swallowing an exception and retrying')

The executable section will check the command line, and then enter an infinite loop downloading the file requested and swallow any exceptions due to transient errors.

©2017, Amazon Web Services, Inc. or its affiliates. All rights reserved.