END-TOD AWS Settings Setup RDS with Basks Host using Terreform

Introduction

Any data pipe, data sources – especially for information – are the spine. To imitate me with a logical pipe, I needed a safe, reliable data for the data to work as a true Thuntream ETL Jobs.

Rather than offering this, I chose change everything use SoilIt is accompanied by modern data engineering and the best practices. This is not only long but also to ensure the environment may not be easily mature, enlarged, or destroyed by one command – as productivity. And if you work with the AWS free tier, this is very important – verify the automatic you can clean everything without forgetting the device that may flow.

Requirements

Following and this project, you will need the following tools and setup:

• Soil y'nf
• AWS CLI & IAM SETUP
  • Add AWS CLI
  • Create an IAM user with a formal access to permission:
    • Attach Policy AdministratorAccess (or create a customer policy that has limited permissions to create all resources included)
    • Download the access key ID and a secret access key
  • Prepare AWS CLI
• AWS KEY PAIR
  It is required for SSH in Basktion Host. You can create one of the AWS console under EC2> pairs.
• Unix based area (Linux / Macos, or WSL of Windows)
  This ensures complying with the shell document and Terreform commands.

Starting: Upbuilding

Let's move in a way to build a Safe and Database AS Database using the terraform.

Infrastructure View

This is the full environment benefits, the style productivity is used by Terradorm. The following resources will be created:

Furnace

• A Custom VPC with cidr block (10.0.0.0/16Selected
• Two Secret Subnets In different locations for availability (in the form of RDS)
• One Subbet for Public (For a basstionkeeper)
• Internet Gate including Path tables By a public subbet route
• A DB Subnet group Shipment of Rulti-Az Rds

Lazzatory

• A Basking Ec2 status In a public subbet
  • Used to decrease in private relatives and reach the database safely
  • Provision of customary security group allows only Port 22 (SSH))

Storage

• A An example of mySQL RDS
  • Sent with private stripes (not available from public internet)
  • Prepared for a dedicated security group that only allows access to the draft host

Security

• Security Groups:
  • Bastition Sg: Allows the Bound SSH (Port 22) from your IP
  • RDS SG: Allows the Bound Mysql (port 3306) from SG

Automation

• A Script of setup (setup.sh) That:
  • Shipping Terreform Verses

Various Design Design for Terreform

I broke infrastructure in modules such as network, basstion and RDS. This allows me to reuse, measure and test components differently.

The next drawing indicates how the terraffor understands and how different how different the variety of infrastructure, where each place represents the app or module.

This toes help to ensure that:

• Resources are properly connected (eg the condition of RDS depends on private subnets),
• Modules separated but collaborated (eg. network, bastionbeside rds),
• No rounds.

To maintain the configuration mentioned above, I planned the project correctly and gave definitions of each part to clarify their roles within the setback.

.
├── data
│   └── mysqlsampledatabase.sql       # Sample dataset to be imported into the RDS database
├── scripts
│   └── setup.sh                      # Bash script to export environment variables (TF_VAR_*), fetch dynamic values, and upload Glue scripts (optional)
└── terraform
    ├── modules                       # Reusable infrastructure modules
    │   ├── bastion
    │   │   ├── compute.tf            # Defines EC2 instance configuration for the Bastion host
    │   │   ├── network.tf            # Uses data sources to reference existing public subnet and VPC (does not create them)
    │   │   ├── outputs.tf            # Outputs Bastion host public IP address
    │   │   └── variables.tf          # Input variables required by the Bastion module (AMI ID, key pair name, etc.)
    │   ├── network
    │   │   ├── network.tf            # Provisions VPC, public/private subnets, Internet gateway, and route tables
    │   │   ├── outputs.tf            # Exposes VPC ID, subnet IDs, and route table IDs for downstream modules
    │   │   └── variables.tf          # Input variables like CIDR blocks and availability zones
    │   └── rds
    │       ├── network.tf            # Defines DB subnet group using private subnet IDs
    │       ├── outputs.tf            # Outputs RDS endpoint and security group for other modules to consume
    │       ├── rds.tf                # Provisions a MySQL RDS instance inside private subnets
    │       └── variables.tf          # Input variables such as DB name, username, password, and instance size
    └── rds-bastion                   # Root Terraform configuration
        ├── backend.tf                # Configures the Terraform backend (e.g., local or remote state file location)
        ├── main.tf                   # Top-level orchestrator file that connects and wires up all modules
        ├── outputs.tf                # Consolidates and re-exports outputs from the modules (e.g., Bastion IP, DB endpoint)
        ├── provider.tf               # Defines the AWS provider and required version
        └── variables.tf              # Project-wide variables passed to modules and referenced across files

With a modular structure in place, the main.tf The file is located in the rds-bastion The directory makes as a Orchestrator. Together the main components: Network, Redabase Redabase, and the host of putting. Each module is placed in the required installation, most of them described in variables.tf or pass through the environment variations (TF_VAR_*).

module "network" {
  source                = "../modules/network"
  region                = var.region
  project_name          = var.project_name
  availability_zone_1   = var.availability_zone_1
  availability_zone_2   = var.availability_zone_2
  vpc_cidr              = var.vpc_cidr
  public_subnet_cidr    = var.public_subnet_cidr
  private_subnet_cidr_1 = var.private_subnet_cidr_1
  private_subnet_cidr_2 = var.private_subnet_cidr_2
}


module "bastion" {
  source = "../modules/bastion"
  region              = var.region
  vpc_id              = module.network.vpc_id
  public_subnet_1     = module.network.public_subnet_id
  availability_zone_1 = var.availability_zone_1
  project_name        = var.project_name

  instance_type = var.instance_type
  key_name      = var.key_name
  ami_id        = var.ami_id

}


module "rds" {
  source              = "../modules/rds"
  region              = var.region
  project_name        = var.project_name
  vpc_id              = module.network.vpc_id
  private_subnet_1    = module.network.private_subnet_id_1
  private_subnet_2    = module.network.private_subnet_id_2
  availability_zone_1 = var.availability_zone_1
  availability_zone_2 = var.availability_zone_2

  db_name       = var.db_name
  db_username   = var.db_username
  db_password   = var.db_password
  bastion_sg_id = module.bastion.bastion_sg_id
}

In this setting program, each part of infrastructure is transmitted but is connected with a good explanation Input and consequences.

For example, after providing with VPC and Subnets in network Module, Returns their IDs using outputThen pass them as The transfers of the installation In some modules they are like rds including bastion. This avoids working hard and empowering Terreform is a change Tell her leaning including Build a dependency graph in.

In some cases (such as inadequate in bastion Module), I use data Resources Reference Existing Resources created by previous modules, instead of repeating or repetitive.

This page Depending on between modules It depends on the right Explanation and Display from pre-built modules. These results have been referred as a subtiture variety depending on the dependent, implementation of the terraform forms to form an internal dependence graph and orchestterate the appropriate creation order.

For example, network guide disclose VPC ID and subnet IDs using outputs.tf. These treasures are still it is eaten In lower varieties like rds including bastion through main.tf Root configuration file.

Below how this applies to operation:

Inside modules/network/outputs.tf:

output "vpc_id" {
  description = "ID of the VPC"
  value       = aws_vpc.main.id
}

Inside modules/bastion/variables.tf:

variable "vpc_id" {
  description = "ID of the VPC"
  type        = string
}

Inside modules/bastion/network.tf:

data "aws_vpc" "main" {
  id = var.vpc_id
}

Provision of RDS status, created Two secret subnets in different locations of availabilityAs AWS need At least two of two in different AZS setting up the DB Subnet group.

Although I met this correct configuration requirement, the Multi-Az Shipment During the creation of RDS Stay within free tier limits including Avoid Extra Costs. This setup is still imitating the formation of a production rate while remains the costs of development and testing.

Shipment of the work done

With all modules with the appropriate string by installation and outcomes, and the Incrial infrastructure installed on blocks and re-activate, the next step Change the Providing Process. Instead of passing the transforming manually, help text setup.sh used for the necessary natural exports (TF_VAR_*).

When the Setup text is ripe, post infrastructure is easier as running a few terraform commands.

source scripts/setup.sh
cd terraform/rds-bastion
terraform init
terraform plan
terraform apply

Submitting a process of submission of Terraform shipment, created a service document (setup.sh) that is exported to the necessary natural variables using TF_VAR_ To compose the meeting. Terraform automatically takes the combined variables TF_VAR_So this method avoids difficult prices inside .tf files or need handwriting all the time.

#!/bin/bash
set -e
export de_project=""
export AWS_DEFAULT_REGION=""

# Define the variables to manage
declare -A TF_VARS=(
  ["TF_VAR_project_name"]="$de_project"
  ["TF_VAR_region"]="$AWS_DEFAULT_REGION"
  ["TF_VAR_availability_zone_1"]="us-east-1a"
  ["TF_VAR_availability_zone_2"]="us-east-1b"

  ["TF_VAR_ami_id"]=""
  ["TF_VAR_key_name"]=""
  ["TF_VAR_db_username"]=""
  ["TF_VAR_db_password"]=""
  ["TF_VAR_db_name"]=""
)

for var in "${!TF_VARS[@]}"; do
    value="${TF_VARS[$var]}"
    if grep -q "^export $var=" "$HOME/.bashrc"; then
        sed -i "s|^export $var=.*|export $var=$value|" "$HOME/.bashrc"
    else
        echo "export $var=$value" >> "$HOME/.bashrc"
    fi
done

# Source updated .bashrc to make changes available immediately in this shell
source "$HOME/.bashrc"

After running terraform applyTerreform will provide all the specified sources – VPC, subnets, tables, tables, example of RDS, and the host. When the process has successfully completed, you will see the output price similar to the following:

Apply complete! Resources: 12 added, 0 changed, 0 destroyed.

Outputs:

bastion_public_ip      = ""
bastion_sg_id          = ""
db_endpoint            = ":3306"
instance_public_dns    = ""
rds_db_name            = ""
vpc_id                 = ""
vpc_name               = ""

This exit is defined in the outputs.tf files your modules and sent out of the root module (rds-bastion/outputs.tf). They are important:

• SSH-ING EMSIVER THE BAST
• Connecting securely in private RDS status
• To strengthen the creation of service

Connecting to RDS with Basktion Host and Seeding Database

Now as infrastructure is supplied, the next step is to Seed MySQL database can be handled in the RDS example. As the database is inner a private subnetWe cannot achieve directly from our local machine. Instead, we will use Basking Ec2 status As a jump guard to:

• Transfer sample data (mysqlsampledatabase.sql) In basktion.

• Connect from Basktion to RDS status.

• Import SQL data to launch a database.

You can submit the top two directions from directory of Terreform Main guide and submit SQL content to the remote EC2 (Bastion) after reading the internal SQL file.

cd ../.. 
cat data/mysqlsampledatabase.sql | ssh -i your-key.pem ec2-user@ 'cat > ~/mysqlsampledatabase.sql'

When the data is copied in the example of the Basking EC2, the next step is to move to a long machine and:

ssh -i ~/.ssh/new-key.pem ec2-user@

After connecting, you can use MySQL client (already installed when using it mariadb105 In the setting of your EC2) importing a SQL file in your RDS database:

mysql -h  -P 3306 -u  -p < mysqlsampledatabase.sql 

Enter the password when prompted.

When the import is complete, you can connect to the database of RDS MySQL and to ensure that the database and its tables are successfully created.

Run the following command from within the host auditor:

mysql -h  -P 3306 -u  -p 

After entering your password, you can write the available information and tables:

To ensure that the dataset is well admitted in the RDS condition, I ran with a simple question:

This returned a line from customers Table, to confirm that:

• Data and tables are successfully created
• Sample sample is set for seed in RDS Stance
• Bastion Personap Setup and Rirts CRDS is active as targeted

This ends infrastructure setup and the import process.

Destroying infrastructure

If you have finished checking or showing your setup, it is important to destroy AWS resources to Avoid unnecessary cases.

Since Terreform was provided in the use of Terreform, all infrastructure is simple as using one command after walking in your root configuration indicator:

cd terraform/rds-bastion
terraform destroy

Store

In this project, I showed to provide protective and productive data infrastructure Soil surprised. Instead of disclosing database on the public Internet, I used good habits by setting an example of RDS in Secret subnetsonly accessible with The tenser of putting to a public subbet.

By creating the project with The Configuration of Normal TerreformI made sure that each network, database, and host host Integrated equallyBe careful, and it's easy to manage. I also showed that the interior of the Terradorm graph of dependency Hosting Updates and the order of service creation works outside of seams.

Due to infrastructure as a code (IAC), all nature can be allowed or demolished with one command, making it good ETL Prototyping, The practice of data engineeringor Proof-of Pipeline concept. Most importantly, this automatic helps protect the unexpected costs that allowing you to spend all resources when you are done.

You can find the perfect source code, Terreform Configuration, and Setting Scriptures in my GitTub group:

Feel free to view the code, add a Republica, and adapt your AWS projects. Donations, the answer, and stars are always welcome!

What is next?

You can add this to the setup by:

• Connecting AWLA Work in the RDS performance of ETL performance.
• Adding your Database Red Reds and EC2

Progress