EBS Unused Volumes

Automating EBS volume clean up event-driven architecture

Before Start

EBS: Elastic Block Storage

EBS Snapshots

Goal

EBS volumes could sometimes be costly if left unused for a long time. The following table lists the pricing for EBS volumes in the us-east-1 region:

Volume price us-east-1

If an EBS volume is not attached to an EC2 instance, then it is considered to be in an available state. AWS would charge you for the provisioned storage of a volume even if you are not really using it. So then, why would you want to pay for volumes that are not being used?

Therefore, the main purpose of this function is to determine EBS volumes that have been unattached from EC2 instances, create a snapshot to persist the volume’s data, and then delete the volume to clean the environment and save money.

EBS Snapshots

Storing data in a snapshot is always cheaper than storing it in an EBS volume.

The reason is because EBS volumes are block storage that will get physically attached to EC2 instances. On the other hand, Snapshots are just copies of your data over a certain period of time.

Therefore, from an operational standpoint, it is much cheaper to keep a snapshot that just has data sitting around, rather than a full fledge disk that has more extensive functionalities.

The following table lists the prices of a snapshots in the us-east-1 region. It is noticeable that there is a significant difference in prices with respect to the EBS volumes.

Snapshot price us-east-1

Architecture

Architecture including services like CloudTrail, EventBridge and Lambda

The first part of the automation will be using CloudTrail to capture the EC2 API: DetachVolume.

The second part is to create a rule in EventBridge that will react once this API is called.

The third part is to configure a Lambda function as a target in the rule, which would be responsible of:

1. Tag the EBS volume that becomes available with key/value which would represent the days for which the volume should get cleaned up, e.g. five days ahead
2. Create another rule in EventBridge which has a schedule that matches the above key/value tag corresponding to the days for which the volume needs to be deleted

The fourth part would be invoking another Lambda function that will:

1. Take care the snapshot of the available EBS volume
2. Delete this available EBS volume

Setup (CloudTrail + EventBridge)

You can read more about setting up CloudTrail with EventBridge in my other post. This is going to be particularly useful when EventBridge rule is triggered and needs to connect to a Lambda function to perform some computation.

Logic

First Lambda — Scheduling

The scheduling Lambda function will tag the volume resource with a clean up day time, e.g. five days from now. This will be used just to know the general information about when the available volume would get deleted from the environment.

There is a datetime_to_cron function that would convert the datetime to a cron expression. This would then be later used for when scheduling the event in EventBridge.

Code part 1: tagging and using cron function conversion

timedelta(minutes=2) represents the time that the volume should be kept around before cleaning up. This could also be seconds, days, months or years depending on your requirement.

After tagging the volume with the cleanup day, the next step is to create a rule in EventBridge. For this, we need the following three boto3 API methods:

• put_rule() and put_targets() for configuring EventBridge
• add_permissions() for setting the proper permissions in Lambda.

The put_rule() API takes a ScheduledExpression as attribute. This is where we use the schedule variable that gets assigned the cron expression.

One thing to notice here is that we are passing the volume_id to the event as custom target input. This is intentionally used so that later in the other Lambda function, we would be able to determine which volumes we have to work with.

Passing custom inputs to the events can be powerful when working with event-driven systems where data needs to flow from one point to the other.

Code part 2: scheduling the event with EventBridge

Second Lambda — Snapshot / EBS Delete

This function will first take a snapshot from the EBS volume that is not being used, and then delete this volume.

However, it would be nice to have some idea what this snapshot represents. It could get quite confusing and chaotic when searching for a snapshot that is not easily recognised.

Therefore, in this first part of the function below we are taking the Name tags from the EBS volume and assign it to a dictionary (name_tag) so that we can use it later on the next stage.

Code part 3: pulling the Name tags from the EBS volume

This second part of the function will take care of creating the snapshot, with the same Name tags from the EBS volume, and then delete the volume.

Code part 4: creating snapshot with tags and deleting EBS volume

Full code can be found in the reference.

Let me know if I can help in anything. Subscribe if you want to see more of this content.

Go Build 🔥

Reference

aws_devops/storage/block/unused-ebs-volumes at master · edreinoso/aws_devops

High-Performance Block Storage- Amazon EBS Pricing - Amazon Web Services