EBS Volume Extend

Ed Reinoso

4 min readSep 30, 2020

Before Reading

CloudWatch agents for monitoring disk and memory need to be installed
At the moment, this automation only works with XFS file system volumes
This automation only works up to the Partition level — not Logical yet
/etc/fstab needs to have a specific format for the script to run successfully
IAM permissions to execute service interactions are really important

Tech Stack

Scripting languages: Shell Script, Python

Libraries and tools: Jq (shell), Boto3 (python)

Infrastructure used: Lambda Functions, SSM Documents (more details below)

Goal

The main purpose of this automation is to enable EBS volume expansion without worrying about the underlaying tasks that need to be executed.

Logic

This automation is composed by 4 main parts:

2 Shell Scripts on the servers

One to pull EBS volume’s data information
One to check the EBS volume’s space left

1 Lambda function

1 SSM Document — this contains a shell script for extending the volume

Pulling data from volumes (1st Shell Script)

The main purpose of this scripting part is to recognize EBS volume(s) attached to the current instance. It leverages the Jq tool, which helps manipulate JSON objects using Shell Scripting.

From this script, the most important information to understand is that a JSON file is going to be created with some arguments (path, instanceId, volumeId, volumeSize, device, mountPoint, and mountNumber). These arguments are going to be crucial to serve as parameters for the next script.

For this script to run, there are two important steps that need to be considered:

AWS SDK have to be installed on the instance, if not using an Amazon Linux AMI
The instance needs to have the proper IAM permissions to execute describe-volume CLI command

Checking space left on volumes (2nd Shell Script)

The main purpose of this script is to monitor EBS volume(s), and to invoke the Lambda function if the volume is 80% above its capacity. This script will be reading the data from the JSON file that was created in the previous step to evaluate and execute proper EBS expansion.

It is essential to have the CloudWatch agents installed on the instance since they will determine the capacity of each EBS volume. The following command will provide the disk space utilized:

~/aws-scripts-mon/mon-put-instance-data.pl --disk-path="$path" --disk-space-util --verbose

Another important factor is to check the value of the variable mountNumber. This variable will provide an insight on whether the volume has partitions. Therefore, if the value is an empty string “ “, it means that the there are no partitions associated with the disk. Else, the partition is going to be expanded along with the disk itself.

The following command will invoke the Lambda Function, passing the necessary parameters — obtained from previous script— as payload to perform the extension:

aws lambda invoke --function-name EBS_Extend --payload '{"document": "extend_ebs_volume","ebsInfo":[{"instanceId":"'$instanceId'", "volumeId":"'$volumeId'","ebsSize":"'$size'","path":"'$path'","mountPoint":"'$mountPoint'","mountNumber":"'$mountNumber'"}]}' response.json

Again, for this to work without errors, the necessary permissions have to be set so that EC2 can invoke Lambda Functions.

Lambda function to call SSM document

The main purpose of this lambda function is:

a) Increase the EBS volume size by 5 gib (this variable can be changed).

b) Call SSM to run the shell script that will extend the disk.

The event will hold the data obtained from the previous shell scripts — volumeId, size, path, etc. This is crucial when calling the modify_volume() and send_command().

This is the permissions needed to execute this Lambda function, which needs to send the command to the SSM as well as the instance ID which volumes are going to be modified.

{
   "Version": "2012-10-17",
   "Statement": [
    {
       "Sid": "VisualEditor0",
       "Effect": "Allow",
       "Action": "ssm:SendCommand",
       "Resource": [
             "arn:aws:ssm:us-east-1:130193131803:document/extend_ebs_volume",
             "arn:aws:ec2:us-east-1:130193131803:instance/${aws_instance.ec2.id}"
         ]
      }
   ]
}

SSM document to auto extend EBS volume

The main purpose of this part is to extend a volume based on the parameters that are passed from the Lambda Function.

This syntax is very specific for SSM Documents to run commands on EC2 instances. I am leaving some documentation on how to write an SSM document; nonetheless, we should just be focusing on the following script:

#!/bin/bashvolumePath='{{VolumePath}}'
mountPoint='{{MountPoint}}'
mountNumber='{{MountNumber}}'
typeOfVolume=`cat /etc/fstab | grep -w $volumePath | rev | cut -d \" \" -f 7 | rev`if [ $typeOfVolume == \"xfs\" ]; then
   if [ $mountNumber != \"empty\" ]; then
      sleep 3
      growpart /dev/$mountPoint $mountNumber
   fi
   xfs_growfs -d $volumePath >> /home/ec2-user/volumeType.txt
fi

Local variables — volumePath, mountPoint, mountNumber — are getting declared based on parameters that were passed.

TypeOfVolume variable will look at the /etc/fstab and cut the output to obtain the type of volume to grow. One thing to consider about this, the fstab needs to be formatted the following way:

UUID=91ab5f73-23b0-4204-bbe0-813fc53eb4b5  /data xfs  defaults,nofail  0  2

As of right now, the steps to get the type of volume are hardcoded based on the number of spaces in the output (7). Hence, this is why it’s important to have the /etc/fstab exactly as above in order to avoid unexpected behaviors in the automation.

Lastly, there is going to be an if statement checking whether the volume to be extended has a partition through the mountNumber. This information is passed from the Lambda Function. In case that the it does, the script will grow the partition based on the variable mountNumber. This variable is the partition’s number for which it is mounted to the instance.