Using Containers to Automate Your Development Environment
Beginning to work on an existing codebase can be daunting, but it can be even more time-consuming if the team hasn’t taken time to automate the creation of their development environment.
When you start working on an existing project, you’ll likely follow similar steps to the ones below to run the code on your machine:
- Download the source code
- Set up the development environment
- Build the software
- Run the software
Things can start to get complicated on step #2. A development environment is a collection of tools that a developer needs installed on their computer in order to run and work on the codebase. Having developers manually install these tools on their computers leads to a few problems:
- Time – Installing and configuring all of that software can take a long time, sometimes days for each developer
- Diversity – Each developer on your team could be running a different OS with different configurations. You won’t be able to include setup instructions for every developer’s setup.
- Version Conflicts – What happens when this project needs a certain version of a dependency, but you already have a different version installed for working on a separate project?
How do containers help?
You can remedy these complications by packaging up a common development environment for all developers on your team to download and use. Containers allow you to bundle together an environment that includes everything your application requires to run, including an OS, tools, and dependencies. Virtual machines have been the tool of choice for this in the past, but containers offer the same environment isolation with a smaller image size, faster startup, and often better performance.
To demonstrate how you can use containers for this, we’re going to build a development environment for creating, validating, and deploying CloudFormation templates.
Note: Docker is the most popular tool for building and running containers, so we’ll use it for this example. You can see Docker documentation for instructions on installing docker for your OS.
When using a container to create your development environment, you’ll need to answer two questions:
- How will I get data into and out of my container?
- How will I run commands inside of my container?
Your container image will contain an operating system and any software packages needed to work on your project. Any other files, such as source code and credentials, will live outside of the container. To make these files accessible inside the container we’ll use a docker volume.
In a container, any changes made to the file system are lost when the container exits unless those changes happen inside of a mounted volume. Any files that are created or modified inside of our source code directory will be saved outside of the container because we’re mounting it as a volume.
Running individual commands inside of a container can become tedious. For example, without a container you would run the following command to lint your CloudFormation file:
$ cfn-lint template.yaml
But to run the same command in a container you would have to use this command:
$ docker run --rm -it --workdir /mnt --volume .:/mnt my_container_name cfn-lint template.yaml
Because these commands can become so long, I recommend capturing them in a Makefile. You can also capture these commands into shell scripts, but a Makefile gives you a single place to capture all of your scripts and use variables between them. When you want to run your linter while using a Makefile, would use a command similar to:
$ make lint
A developer will need these already installed on their machine before creating their development environment:
- GNU Make
$ tree . . ├── credentials ├── template.yaml ├── Dockerfile └── Makefile
[default] region = us-west-2 output = table aws_access_key_id = 123456789012 aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
A developer’s credentials are created on the developer’s machine, not stored in version control or inside the container. This is crucial so that secrets are not exposed to the public and so that each developer can have their own set of credentials.
In AWS, credentials can be stored in a number of ways. Here I’ll store my credentials in a file that will be mounted into our development environment.
--- Description: A short description of the project Resources: Test: Type: AWS::EC2::VPC Properties: CidrBlock: "10.0.0.0/16" # The rest of your infrastructure goes here…
We’re just going to deploy a single VPC to verify that the development environment is working, but you can replace this file with your own CloudFormation template
# Start from a minimal OS containing Python 3.8 FROM python:3.8.0-alpine # Install AWS tools RUN pip install -q awscli aws-shell cfn-lint yamllint # Change to the directory where the source code should be mounted WORKDIR /project
Your Dockerfile contains instructions that tell Docker how to install the software that you want to include in your development environment
# Variables CONTAINER=aws_example SOURCE_MOUNT=/proj STACK_NAME=test CREDENTIALS_FILE=$(shell pwd)/credentials DOCKER_RUN=docker run --rm --it --volume $(shell pwd):$(SOURCE_MOUNT) --volume $(CREDENTIALS_FILE):/root/.aws/credentials:ro $(CONTAINER) # Targets .PHONY: shell shell: @$(DOCKER_RUN) aws-shell .PHONY: setup setup: @docker build -t $(CONTAINER) . .PHONY: lint lint: @$(DOCKER_RUN) yamllint template.yaml @$(DOCKER_RUN) cfn-lint template.yaml .PHONY: deploy deploy: @$(DOCKER_RUN) aws cloudformation create-stack --template-body file://template.yaml --stack-name $(STACK_NAME) .PHONY: destroy destroy: @$(DOCKER_RUN) aws cloudformation delete-stack --stack-name $(STACK_NAME) @echo "Deleted stack $(STACK_NAME)"
DOCKER_RUN variable contains the command used to run other commands inside the docker container. It contains a few parameters:
--rm– Remove the container once the command exits
-it– Run the container in “interactive” and “terminal” mode
--volume– Mount a file or directory into the container
Before you can use your container you have to build the container image:
$ make setup
This will create and save a container image on your machine. This has to be done before you can run anything within a container.
After you’ve created your container image, a typical CloudFormation workflow would look like this:
1. Validate the CloudFormation template
$ make lint
2. Deploy to AWS & manually verify that the infrastructure performs as expected
$ make deploy
3. Clean up – destroy the deployed infrastructure
$ make destroy
Congratulations, you’ve deployed and tested your AWS CloudFormation templates without installing any CloudFormation tools on your machine.
Bonus: It’s also helpful to be able to start a shell session inside of your development environment so you issue commands manually. To do this I’ve added a “shell” target to my Makefile. Here I’m using aws-shell as the shell, but you can also use
bash if you want to use non-AWS commands. Just make sure that the shell that you use is installed in your Docker image.
$ make shell First run, creating autocomplete index... Creating doc index in the background. It will be a few minutes before all documentation is available. aws>
Automating the creation of a project’s development environment enhances the productivity of everyone on the team. Containers make this easy, performant, and work on any operating system and programming language.