Automating AWS Resource Cleanup with Terraform and Python

With the exponential growth of cloud computing, AWS (Amazon Web Services) has become one of the most popular platforms for hosting and managing IT resources. Yet many developers, myself included, run into a common problem: forgetting about resources created in a test account. Resources such as EC2 instances, RDS databases and S3 buckets, when left unmanaged, can lead to unexpected costs and, in some cases, data loss. In this article, we'll look at how to automate listing and removing these resources across every AWS Region, in a safe and efficient way.

Tools and Technologies Used

To tackle the problem of managing unused resources on AWS, we'll use a powerful combination of tools: Terraform and Python with the Boto3 library. Terraform is an infrastructure as code (IaC) tool that lets you create, modify and remove resources declaratively. It will give us the ability to list and manage EC2, RDS and S3 resources across all AWS Regions. Alongside Terraform, we'll use Boto3, a Python library that lets us interact with AWS services, making it possible to automate resource removal.

Implementing the Automation

To implement the automation, we start by defining the EC2, RDS and S3 resources in the Terraform file main.tf. Next, we set up an aws_regions variable containing all the AWS Regions we want to cover. With Terraform configured, we create a Python script called list_resources.py that uses Boto3 to list the resources in each Region and, when needed, remove them. The script runs sequentially to make sure every unused resource in every Region of the AWS test account is identified and removed, minimizing the costs and risks of forgotten resources.

To automate listing and removing EC2, RDS and S3 resources across all AWS Regions using Terraform and Python, you can follow the steps below:

1. Initial setup:

Make sure Terraform is installed and configured on your local machine. You'll also need access credentials for your AWS account, set either as environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) or through the AWS CLI configuration file.

2. Listing and Removing Resources:

Here we'll use Terraform to list and remove EC2, RDS and S3 resources across all AWS Regions. Where needed, we'll use Python with the Boto3 library to help remove the resources.

2.1. Create the main.tf file with the following content:

provider "aws" {
  region = "us-east-1"  # Defina uma região aqui para inicializar o provider
}

# Recursos EC2
resource "aws_instance" "existing_instances" {
  count = 0
}

# Recursos RDS
resource "aws_db_instance" "existing_db_instances" {
  count = 0
}

# Recursos S3
resource "aws_s3_bucket" "existing_s3_buckets" {
  count = 0
}

2.2. Create the variables.tf file with the following content:

variable "aws_regions" {
  type    = list(string)
  default = ["us-east-1", "us-west-1", "us-west-2", "eu-west-1", "eu-central-1", "ap-southeast-1", "ap-southeast-2", "ap-northeast-1", "sa-east-1"]
}

Add or remove Regions from the aws_regions list as needed.

2.3. Create the outputs.tf file with the following content:

output "existing_instances" {
  value = aws_instance.existing_instances[*].id
}
output "existing_db_instances" {
  value = aws_db_instance.existing_db_instances[*].identifier
}
output "existing_s3_buckets" {
  value = aws_s3_bucket.existing_s3_buckets[*].bucket
}

2.4. Create the list_resources.py file with the following content:

import boto3
import os
def get_all_regions():
    ec2 = boto3.client('ec2', region_name='us-east-1')
    response = ec2.describe_regions()
    return [region['RegionName'] for region in response['Regions']]
def list_ec2_instances(region):
    ec2 = boto3.resource('ec2', region_name=region)
    instances = ec2.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    return list(instances)
def list_rds_instances(region):
    rds = boto3.client('rds', region_name=region)
    response = rds.describe_db_instances()
    return response['DBInstances']
def list_s3_buckets(region):
    s3 = boto3.client('s3', region_name=region)
    response = s3.list_buckets()
    return response['Buckets']
def delete_ec2_instances(instances):
    for instance in instances:
        instance.terminate()
def delete_rds_instances(instances):
    for instance in instances:
        rds = boto3.client('rds', region_name=instance['DBInstanceIdentifier'].split(':')[0])
        rds.delete_db_instance(DBInstanceIdentifier=instance['DBInstanceIdentifier'], SkipFinalSnapshot=True)
def delete_s3_buckets(buckets):
    for bucket in buckets:
        s3 = boto3.resource('s3', region_name=bucket['LocationConstraint'])
        bucket = s3.Bucket(bucket['Name'])
        bucket.objects.all().delete()
        bucket.delete()
if __name__ == "__main__":
    regions = get_all_regions()
    for region in regions:
        print(f"Listing resources in region {region}")
        # List EC2 instances
        ec2_instances = list_ec2_instances(region)
        print(f"EC2 instances found: {len(ec2_instances)}")
        if ec2_instances:
            delete_ec2_instances(ec2_instances)
        # List RDS instances
        rds_instances = list_rds_instances(region)
        print(f"RDS instances found: {len(rds_instances)}")
        if rds_instances:
            delete_rds_instances(rds_instances)
        # List S3 buckets
        s3_buckets = list_s3_buckets(region)
        print(f"S3 buckets found: {len(s3_buckets)}")
        if s3_buckets:
            delete_s3_buckets(s3_buckets)
        print(f"Resources deleted in region {region}")

3. Running it:

1. Run terraform init to initialize the Terraform directory.
2. Then run terraform apply to create the configuration and list all existing resources across all Regions.
3. Finally, run the Python script list_resources.py to remove the EC2, RDS and S3 resources across all Regions.

Notes:

• Be extremely careful when removing resources: it's irreversible and can cause permanent data loss. Make sure you're using a test account and that the resources being removed are really the ones you want gone.

Automating AWS resource cleanup with Terraform and Python is a valuable solution for developers who want to keep their test accounts tidy and efficient. Combining Terraform's power for infrastructure configuration with Boto3 for interacting with AWS makes the process safe and controlled.

With this approach, developers can focus more on building applications and less on managing resources by hand, saving both time and money. It also ensures the test account is always ready for new projects, avoiding potential conflicts with previously provisioned resources. In short, automating AWS resource cleanup is a best practice for maximizing efficiency and organization in cloud projects.

See you in the next post! =)