Over the past couple of weeks, we have been working to deploy CircleCI 2.0 in our enterprise AWS environment so we can reap the benefits that 2.0 brings. The purpose of this article is to clarify some points that aren’t super clear from CircleCI’s documentation.
Getting Started
First off, props to the team at Circle for open sourcing this. It’s really awesome to see a tool like this publicly available and open to feedback.
git clone https://github.com/circleci/enterprise-setupcd enterprise-setupmake init
This will create a terraform.tfvars file for you to use. Terraform is a technology that allows us to orchestrate AWS deployments automatically so we don’t have to do each step by hand. It is quite nifty, and elegant in execution. Once we have that terraform file, I’d recommend making a branch and committing so that your company can make changes and not work on the master branch.
Let’s take a look at the terraform file.
aws_access_key = "..."
aws_secret_key = "..."
aws_region = "..."
aws_vpc_id = "..."
aws_subnet_id = "..."
aws_ssh_key_name = "..."
circle_secret_passphrase = "..."
services_instance_type = "c4.2xlarge"
builder_instance_type = "r3.4xlarge"
nomad_client_instance_type = "..."
# Use this var if you have multiple installation within one AWS region
# prefix = "..."
Most of this is pretty straight forward, but let’s hone in on some things that you may not need or know up front.
circle_secret_passphrase is optional. Make it an empty string if you don’t want it.nomad_client_instance_type has a default value if you aren’t sureprefix is super useful because you can append a constant string to the front of the instance names using this
You can refer to the Circle GitHub page above for an overview of options you can use, as well as their default values.
One last thing. If your company doesn’t want to have your CI on a public IP address, you will want to disable that functionality. Open up the circleci.tffile and set the value associate_public_ip_address to false. There is currently a PR open for this to be part of the default tfvars files.
Deploy to AWS via Terraform
Once you have filled in the above file, you may need to run terraform init in the repo. This is so terraform can keep state of your project.
First, before deploying, you should run terraform plan. This allows you to see what will be changed in AWS prior to it actually being changed. Once you have done that, and verified that it has all the changes you are expecting, you can run terraform apply to generate your new AWS instances. This may take a while, so go get a cup of your favorite drink of choice while this executes.
Adding Proxies
If you work at a large scale enterprise company, there is a good chance you have strict security restrictions around inbound and outbound traffic on your network. Unfortunately, at the time of this writing, CircleCI doesn’t take proxies into account during the setup phase. We will manually have to go into our AWS instances and do the rest by hand.
Service Box ✍️
- Log into AWS and go to your EC2 instances.
- In the filter box type the prefix you entered early in your terraform file (mine has “intuit” in it, so that’s what I used)
- In the results find the instance ending in the word “service”.
Once you have the service instance, grab the IP. You will need to SSH into the box. If you click “Connect” in AWS, it will give you instructions on how to do that. I won’t cover that here. Once you have SSH’ed into the box, it is time to use our proxy.
export http_proxy=http://your.cool.proxy
export https_proxy=https://your.cool.secure.proxy
Make sure to replace the above with your actual proxies. Once you have done that, kick off the setup script for the service box by running the code below.
Note: If you run into an issue regarding “unable to obtain lock”, reboot the instance and on start repeat the above export commands followed by sudo -E apt-get update. This will guarantee you are updated and not fighting for a lock.
curl https://get.replicated.com/docker | sudo -E bash
What is important here is the -E command. It passes the environment variables down to the script. In this case, we are setting the proxy so that any network requests made can reach outside servers.
Builders (2.0) ????
Before starting, there is a box for 1.0 builders and a box for 2.0 builders. This section only covers the 2.0 builders. We haven’t attempted to get 1.0 builders running yet, but I’m sure Circle is aware of it after working with us.
As with the Service Box, the Builder Box also needs to make a call when it starts to standup the service. Again, we need to expose the proxy we want to use so that this script can run properly. In the enterprise-setup repo we checked out at the beginning, open up the file nomad_user_data.tpl and at the top add the following:
#! /bin/sh
set -exu
export https_proxy="http://your.cool.proxy"
export http_proxy="http://your.cool.secure.proxy"
echo "-------------------------------------------"
echo " Performing System Updates"
echo "-------------------------------------------"
apt-get update && apt-get -y upgrade
...
You will also want to edit the area where the script creates the config for nomad. Search for /etc/nomad/config.hcl. You will want to edit it to include the following advertise block:
log_level = "DEBUG"
data_dir = "/opt/nomad"
datacenter = "us-east-1"
advertise {
http = "$PRIVATE_IP"
rpc = "$PRIVATE_IP"
serf = "$PRIVATE_IP"
}
Finally, we want to make sure our Docker containers have access to the outside world. Find the section in the script that says “Installing Docker” and add the following right before the:
sudo echo 'export http_proxy="http://your.cool.proxy.com:8080"' >> /etc/default/docker
sudo echo 'export https_proxy="http://your.cool.secure.proxy.com:8080"' >> /etc/default/docker
sudo service docker restart
sleep 5 # This gives us some time to let the service come back up
Once you have done that, chmod +x the script and run viasudo ./init.sh. Make sure it runs completely. If it fails at any point, just run different sections by hand. For safe measure, restart both nomad and docker via sudo service _____ restart.
Wrapping Up ????
Once you are all set with the above, head to the IP of your service box on port 8800. You can finish the setup of your new instance by walking through the steps presented to you.
I’d like to give a huge thank you to the CircleCI team for taking hours of their time to sit with me to step through these problems. Together, we worked to find the path to this state. There is currently a PR open on the enterprise-setuprepo with the fix for the http-proxy issues. I encourage you all to go contribute and be part of the discussion.
This is just some of the cool things that we work on here at Intuit. Got a cool thing you or your company is working on? Share below in the comments and let us know!