In the last blog on Docker, we talked about how we use Docker with Bamboo internally. It went into detail on how we were able to build a customized agent image, which can then be used to run build agents. Docker agents have a couple honorable advantages over configuring build agents manually. Once the docker agent image is created, it can be used to run multiple agents on a single or multiple hosts. We understand that there are many advantages of using Docker bamboo-agents, but we need to be able to implement some orchestration around it. There are many orchestrators like Docker Swarm, Kubernetes and Apache Mesos Every one of them has its own philosophy, use cases and architecture. Rancher is a tool built to simplify Docker orchestration and management.
In this blog, we will talk about how we can leverage tools such as Rancher to manage the distributed build environment across multiple hosts. Rancher makes it easy to manage containerized environments. From deploying and scaling multiple Docker clusters, to integrating the container networks and storage. The simple yet powerful user interface provides complete visibility into your infrastructure and the applications running on top of it. You can easily select, configure and deploy infrastructure services through the Rancher interface, enabling repeatable deployments of custom environments.
Below diagram shows how Rancher is used to create multiple agents on different hosts.
We use Rancher to create our dockerized bamboo-agents. Rancher makes managing and scaling the docker containers easier than ever.
Atlassian has published two images for bamboo-agents to the Docker Hub. To view/use their images, see the link below. You can either use an official bamboo-agent Docker image or you can modify the Atlassian images to suit your requirement. The customized Docker image can be retrieved from a repository/Docker hub. For the purpose of this post, I will be using a custom image named "agarwalsheetal19/atlassian-bamboo-agent".
Following are the steps to setup a Docker agent in Rancher.
From the environments menu, select “manage environments” and then create a new environment.
In this setup, we will be using an Amazon Web Services EC2 instance. Provide your access key and secret key to login to your AWS account. Select the availability zone and the VPC/Subnet.
You can either select the default "rancher-machine" group or select any existing security group you may have in your AWS account.
Now, specify the hostname for your instance and select the number of hosts needed. In this example, we are using a default ubuntu AMI.
After you click on the "Create" button, you will be able to see the hosts added. To find that, look under the Infrastructure tab then click on Hosts.
We will add a new stack for the bamboo-agent. A stack is a group of services. Stacks can be used to group together services that work simultaneously to implement an application.
Once a stack is created, we can add services to that stack. Provide the name of the service and the Docker image you would like to use. In the below example, we are using the custom agent Docker image mentioned from the beginning. Initially, we select "Run 3 containers". You can use the slider to select the number of containers you would like to run. You can always add more bamboo-agents later on, as per your specific requirement.
It is a good practice to also provide environment variables when creating the service. Here we provided the "BAMBOO_SERVER" (Bamboo master server base URL) variable to which the bamboo-agent will register once the container is started. If you are using a custom Docker image, you may have to pass additional environment variables.
A good example is when the bamboo-agent container is started. It needs to be authenticated by an administrator in the Bamboo server, so that the agent can connect to the master. By default, you will have to login to the Bamboo UI and approve the agent access. As you are dynamically scaling the agents, you can use the "security token verification." This way, you do not have to manually approve the agent once a new agent is created. You can also pass another environment variable "SECURITY_TOKEN" to the container to achieve this.
We can also create persistent volumes, that may be shared between containers for m2. ssh or any OTHER tools that need to be shared with all of the Bamboo agents.
Volume Plugin and
Volume Driver are used interchangeably in Docker and Rancher documentation. Both versions of terminology are referring to a Docker Volume Plugin. A Docker Volume Plugin provides local and/or shared persistent volumes to a Docker container.
Rancher Volume Plugins (Drivers) are currently implemented as Docker Volume Plugins and may be interacted with the
docker volume command on any host. However, depending on the storage technology, volumes may be accessed by one, some or all hosts in an environment. Rancher handles the complexity of coordinating shared volumes across hosts. Some examples are: rancher-nfs, rancher-ebs, pxd (portworx).
Once we create the service, we can see that the bamboo-agent service is running on the hosts. As shown in the below image.
When looking in the Bamboo UI, we can now see that 3 online agents are listed.
Scaling the containers up and down is easy. Under "Service", we see that our current 3 containers are running.
By simply pressing the "+" or "-" buttons we can easily scale to 6 containers. As we scale up the number of containers, we see that new containers are getting spinned up, as shown below.
Now, there are 6 bamboo-agents running in both Rancher and Bamboo.
If we go back to the Bamboo UI, the new bamboo-agents were registered to the Bamboo master server. They are now online to accept build requests.
The video below shows the process of scaling your bamboo-agents. It is a visual representation of what was briefly mentioned above about scaling.
In this blog, we went over how to build bamboo-agents with Docker and Rancher. We also deployed a Dockerized bamboo-agent which can be scaled as needed. Rancher’s cross-node networking feature allows us to seamlessly scale the bamboo-agents on multiple nodes and across multiple cloud environments, with just a few simple clicks.
The next NY DevOps Contained meetup by Addteq is coming up on December 13th! This event is all about Rancher 2.0 & Kubernetes. Learn more about the importance of containerization with Rancher.