Web proxy has many purposes, to name a few, some people use it for anonymity purposes, and some people want bypass network censorship, and another common usage is for CDN. Here at CoinGecko, we use web proxies to protect ourselves while consuming and sharing large data by staying anonymous. Let’s go through the steps to create high availability proxy servers with zero maintenance using Amazon Web Services and Squid.
Step 1: Launch an EC2 instance
Step 2: Reserve Elastic IPs (static ip) from AWS
In this example, We reserved 2 static ip(s) from AWS, specifically 126.96.36.199 and 188.8.131.52. You can reserved as many ip(s) you want depending on your need, each ip represents a proxy node.
Step 3: Creating a golden image (AMI) of EC2 with squid installed (EC2 > Instances)
Now, we are going to prepare a golden image for auto scaling. Now ssh into the ec2 instance created in step 1 and do the following:
sudo yum update -y sudo yum install docker -y sudo service docker start sudo usermod -a -G docker ec2-user sudo systemctl enable docker sudo amazon-linux-extras install python3 -y sudo docker run --name squid -d -p 3128:3128 --restart always --volume ~/squid.conf:/etc/squid/squid.conf datadog/squid
vim ~/squid.conf and and copy the following into the file (feel free to use your own configuration)
acl SSL_ports port 443 acl Safe_ports port 80 acl Safe_ports port 21 acl Safe_ports port 443 acl Safe_ports port 70 acl Safe_ports port 210 acl Safe_ports port 1025-65535 acl Safe_ports port 280 acl Safe_ports port 488 acl Safe_ports port 591 acl Safe_ports port 777 acl CONNECT method CONNECT #http_access deny !Safe_ports #http_access deny CONNECT !SSL_ports #http_access allow localhost manager #http_access deny manager #http_access allow localhost #http_access deny all http_access allow all http_port 3128 coredump_dir /var/spool/squid3 refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320
Now we have our ec2 instance setup to be a proxy, it is time to create an AMI.
Step 4: Create an appropriate IAM Role to allow address association. (IAM > Roles)
Step 5: Create AWS Security Group (EC2 > Security Group)
In this tutorial, I will be creating a security group that allows ALL inbound and outbound traffic for simplicity sake. Please use the appropriate security group when you are creating your own proxy cluster.
Step 6: Create AWS Launch Template (EC2 > Launch Templates)
Things to take note while creating launch template:
- For AMI, use the AMI created in Step 3
- For Security group, use the security group created in Step 5
- in Advanced tab > User data put in the following (for the valid ips, put in your comma-separated ip(s) created in Step 2
#!/bin/bash sudo pip3 install aws-ec2-assign-elastic-ip sudo /usr/local/bin/aws-ec2-assign-elastic-ip --region us-east-2 --valid-ips 184.108.40.206,220.127.116.11 sudo /usr/local/bin/aws-ec2-assign-elastic-ip --region us-east-2 --valid-ips 18.104.22.168,22.214.171.124 sudo docker container prune -f sudo docker run --name squid -d -p 3128:3128 --restart unless-stopped --volume /home/ec2-user/squid.conf:/etc/squid/squid.conf datadog/squid
Step 7: Create Auto Scaling Group (EC2 > Auto Scaling Group)
Use the launch template as the recipe to create the Auto Scaling Group. In this example, we configure the desired number of instance to be 2 since we reserved 2 ips in Step 2. Now click done and you should be able to see 2 EC2 instances being created in the dashboard
Step 8: Verify if proxy is working
In order to verify if the proxy is working, you can use https://www.proxy-checker.org/
With the above setup, auto scaling will take spawn new instance(s) using the static ip(s) we reserved and the configuration we setup from the above steps, if the node is down or terminated. Hence zero maintenance required.
13 thoughts on “How to create a self-healing web proxy cluster with AWS and Squid in 8 steps”
How do you direct all the traffic from Private Subnet destined for the internet to the EIP’s of the Proxy?
Thanks for the useful steps…
You can use a load balancer, and configure security group to allow inbound/outbound access.
Thankyou for helping
Thanks for posting this informative one
Wow didn’t know that using a proxy is so easy, well described!
You r blogs are always on the point. And really easy to understand.
It doesn’t seem like 8 steps. You explained it really beautifully. Truly amazing.
Wow, I had no idea using a proxy could be so simple! There don’t seem to be eight steps. You did a really good job of explaining things. genuinely awesome.
Thanks for sharing this useful information
Thank you for this post.
Buy Persian saffron online
Really, wonderful post. That’s such eight easy steps. Thank you!