Testing AWS AMI with Serverspec

Generated by Bing Image Creator

Are you tired of manually configuring your AWS EC2 instances every time you need to launch a new server? Do you find it challenging to keep track of all the changes you make to your infrastructure over time? If so, you’re not alone. As your infrastructure grows, it becomes increasingly difficult to manage it efficiently and reliably.
That’s where Infrastructure as Code (IaC) comes in. With IaC, you can define your infrastructure in code, version it, and deploy it with confidence, knowing that your infrastructure is consistent and reproducible. One of the key tools for implementing IaC on AWS is Amazon Machine Images (AMIs), which provide a pre-configured, secure, and scalable environment that you can use to launch new instances.
But how can you be sure that your AMIs are properly configured and meet your requirements? That’s where testing comes in. In this article, we’ll explore how you can use Serverspec, a powerful testing framework for infrastructure automation, to test your AWS AMIs and ensure that they are working as expected.

Prerequisites

Let’s discuss the prerequisites you need in order to follow along, before we dive into the topic. First and foremost, you’ll need an AWS account with sufficient permissions to run EC2 instances, access S3 and IAM roles, and create key pairs and security groups. Additionally, I’ll be demonstrating how to run Serverspec from Jenkins, but you could also run it from your local machine (although I don’t recommend this approach). It’s worth noting that Infrastructure as Code principles emphasize the importance of building systems that are easy to recreate and dispose of, and running tests locally is a violation of these principles.

Before we proceed with the rest of the article, I want to clarify the meaning of a term that will be frequently used throughout: “Jenkins instance.” It refers not to the Jenkins in-build node, but rather to an agent executor. It’s important to never run jobs on your main Jenkins instance, but always spin up separate workers that are terminated soon after the job is complete. This is a best practice for several reasons. First, it reduces the load on the Jenkins in-build node, making it more stable and reliable. Second, it allows you to easily scale your Jenkins setup by adding more workers as needed. Finally, it provides better security by isolating the job execution environment from the Jenkins in-build node, preventing any potential security breaches from spreading to the in-build node.

Depending on how you connect to the instance launched from the AMI under test, there are different approaches to running Serverspec — it can be done with SSH or SSM.

Part 1. Running tests with SSH

When it comes to running Serverspec tests, one of the most common approaches is to use SSH. This method involves connecting to the instance under test via SSH and running the tests from other instance (e.g. Jenkins). In this way, you can ensure that the tests are executed in an environment that is basically identical to the production one.

To use SSH for running Serverspec tests, you need to have Serverspec installed on the machine from which you are connecting to the instance. If you haven’t installed Serverspec yet, you can refer to the official installation guide.

In order to run tests with SSH, we first need to configure our test suite. Serverspec uses RSpec, a popular Ruby testing framework, for its syntax and organization. Under the hood, it leverages the Net::SSH library to establish SSH connections to the target instances. The configuration options for SSH connections in Serverspec are largely the same as in Net::SSH.

Those ENV parameters will be set in Jenkins job later.

For this tutorial, we’ll have a simple test suite that verifies that the running system is Ubuntu.

Note that this test suite expects the configuration to be one level higher. Basically, the folder structure is expected to be as:

serverspec
├── specs
│ └── ubuntu_spec.rb
├── serverspec_config.rb
└── Jenkinsfile.groovy

Lastly, let’s create the orchestrator of the execution — Jenkinsfile.groovy

Let me guide you through the pipeline code.
The pipeline expects only 2 parameters to be passed: AMI_ID to test and the REGION where the AMI was built.

The first stage is to create a key-pair. Since we don’t care about its name, UUID is used to generate a pseudo-random name. Then using AWS CLI a key-pair is created. I want to note that the KeyMaterial is read with the help of the jq library since the CLI native --query call returns it with \n symbols. The KeyMaterial data is a private key and it is saved to the SSH_KEY_PATH file. And don’t forget to set permissions on the key-pair file to 400, otherwise you will run into “permissions are too open”.

In the second stage a security group is created. The Jenkins instance and the instance that we are about to start will reside in the same VPC and subnet. Thus we only need to open port 22 on the test instance for the Jenkins IP address.

The third stage is to launch an instance. We’ll use the AWS CLI to launch an instance with the specified AMI_ID and previously created security group and key-pair. The Jenkins instance will communicate with the test instance over the private IP address. That value is also an environment variable that is used in the serverspec_config file.

With all parts in place we can now run the Serverspec tests.
Once the tests are finished, we should not forget to terminate the instance, delete the key-pair and security group that we created earlier, as well as to remove the SSH key file from the Jenkins instance.

Part 2. Running tests with SSM

Another common approach for managing the servers is to use Systems Manager or SSM. It allows you to manage servers at scale, automate common maintenance tasks, and execute commands and scripts remotely. One of the benefits of using SSM is that it does not require any open ports on the server, which can help increase security by reducing the attack surface. In this section, I will show you how to use SSM to execute serverspec tests on a remote server, using the same simple suite that we tested with SSH in the previous section.

Unlike the SSH approach, the tests will now be executed on the test instance directly, without the need for an SSH connection. To use SSM, two changes are required in the Serverspec part of the process. First, since SSM doesn’t require an SSH connection, the serverspec_config.rb file is no longer needed. Second, the test file should now use the local backend, which can be configured with the set :backend, :exec line in the test file.

One of the important steps in setting up a Jenkins server to run Serverspec tests is configuring the appropriate IAM roles. IAM roles provide a secure way to control access to AWS resources, and they are essential for ensuring that your Jenkins server can access the resources it needs to run tests.

The IAM role that we’ll need to create for our Jenkins instance will have a number of policies attached to it:

Let’s take a closer look at the policies that are included in the role. There are basically two operations that we need to allow our Jenkins instance to perform. First it should be able to access the specified S3 bucket to upload and delete the test scripts. And second it should be allowed to execute SSM commands and get the results of the execution.

The IAM role for the instance should be even more restrictive — it only needs to access the specified S3 bucket.

Lastly, we should create an instance profile that will be used to launch the instance for the test. It can be done with the following CLI calls:

sh "aws iam create-instance-profile - instance-profile-name serverspec-instance"
sh "aws iam add-role-to-instance-profile - instance-profile-name serverspec-instance - role-name instance-serverspec"

Note! It is also possible to set up an IAM role and instance profile for the test instance right before the test and delete it afterwards. It makes sense especially if you are running infrequent builds.

With the IAM roles and instance profile in place, we can move on to configuring our Jenkins job to actually run the tests.

The pipeline is similar to that of the SSH case, but the way it’s executed is entirely different. Upon starting the instance, no security group is initially attached. Instead, the previously created instance-profile instance-serverspec is used.

The aws ssm send-command command is utilized to carry out each step of the process, while command-executed ensures that each execution is successful.

Before we can begin testing, we must install gem and serverspec, as well as upload the necessary Serverspec test files to an S3 bucket. Once uploaded, the files are retrieved by the test instance and the tests are executed.

After the execution is complete, the files are promptly deleted from the S3 bucket, and the instance is terminated.

Part 3. Which approach to choose?

There are pros and cons to both the SSM and SSH approaches for running tests on EC2 instances. Here’s a summary:

One of the main advantages of the SSM approach is that you can delete the SSH service from the AMI, which reduces the attack surface and enhances security. While it’s theoretically possible to do the same with the SSH approach and install it back via a bash script passed via --user-data of aws ec2 run-instances command, it’s not recommended because it alters the instance under test.

The SSM approach on the other hand does require tampering with the instance under test to run the tests, which can result in false-positives or false-negatives. In contrast, the SSH approach doesn’t have this flaw because it doesn’t require altering the instance. That alone can be a deal-breaker when choosing the way of testing.

The SSM approach is more flexible because you can run tests on instances in different subnets, VPCs, or even different AWS accounts as long as the Jenkins instance has the necessary IAM permissions. With the SSH approach, the instances need to be in the same subnet and if not, subnets should be configured correctly within the same VPC. If the instances are running in different VPCs, VPC peering is required.

However, one of the biggest disadvantages of the SSM approach is that it gives the Jenkins instance powerful capabilities to run SSM commands on any instance in the account, which can be a security risk if not properly isolated. It’s important to keep test accounts separate from production accounts to minimize this risk.

Afterwards

In conclusion, both the SSM and SSH approaches have their pros and cons when it comes to running tests on EC2 instances. Ultimately, the choice depends on your specific use case and security requirements.

Regardless of the approach, it’s crucial to have a secure environment for testing and production. Infrastructure as code (IaC) can help in achieving this goal by defining and managing your infrastructure through code. This allows you to version control and test changes before deployment, reducing the likelihood of misconfigurations and vulnerabilities.

By adopting a secure-by-design approach and implementing good security practices, you can minimize the risk of security incidents and protect your infrastructure, data and business.

Support

If you like the content you read and want to support the author — thank you very much!

Here is my Ethereum wallet for tips:

0xB34C2BcE674104a7ca1ECEbF76d21fE1099132F0