What is AWS RDS (Relationship Database Service)?

In this Post you will learn everything you must know end to end about AWS RDS. This tutorial will give you glimpse of each components starting from what is DB instance to scaling and multi availability zone cluster configurations and details.

Lets get started.

What is AWS RDS (Relationship Database Service)?

  • It allows you to setup relational database in the AWS Cloud. AWS RDS is managed database service.
  • It is cost effective and resizable capacity because you if you invest in your own hardware, memory, CPU and it is time consuming and very costly.
  • With AWS RDS, it manages everything starting from Scaling, availability, backups, software patching, software installing, OS patching, OS installation, hardware lifecycle, server maintenance.
  • You can define permissions of your database users and database with IAM.

Database Instance

DB instance is a database environment which you launch your database users and user created databases.

  1. You can run your database instance in various AZ’s also known as multi-AZ deployments. Amazon automatically provisions and maintains secondary standby instance in different Availability zones. With this approach the primary DB replicates the data written into it to standby instance located in another AZ. Note: Instance in secondary can also be configured as read
  2. You can attach security groups to your database instance to protect your instance.
  3. You can launch DB instance in Local zones as well by enabling local zone in Amazon EC2 console.
  4. You can use Amazon CloudWatch to monitor the status of your database instance. You can monitor the following metrics:
    1. IOPS (I/O operations per second)
    1. Latency (Submitted I/O request until completed)
    1. Throughput (Number of bytes transferred per second) to or from disk.
    1. Queue depth: how many requests are pending in the queue.
  5. DB instance has a unique DB instance identifier that a customer or a user provider and should be different in AWS Region. If you provide the DB instance identifier as testing, then your endpoint formed will be as below.
testing. <account-id><region>.rds.amazonaws.com
  • DB instance supports various DB engines such as MySQL, MariaDB, PostgreSQL, Oracle, Microsoft SQL server, Amazon Aurora database engines.
  • A DB instance can host multiple databases with multiple schemas.
  • When you create any DB instance using AWS RDS service then by default it creates a master user account, and this user has all the permissions. Note: Make sure to change the password of this master user account.
  • You can create a backup of your Database instance by creating database snapshots.  You can also store your snapshots in AWS S3 bucket.
  • You can enable IAM database authentication on your database instance so that you don’t need any password to login to the database instance.
  • You can also enable Kerberos authentication to support external authentication of database users using Kerberos and Microsoft Active directory.
  • DB Instance are billed per hour.

DB Engines

Db engines are specific software’s that runs on your DB instance such as MariaDB, Microsoft SQL server, MySQL, Oracle and Postgres.

DB Instance class

Db instance class determines the computation, memory and storage capacity of a DB instance.  AWS RDS supports three types of DB instance classes:

  • General purpose:
  • Memory optimized:
  • Burstable Performance
  1. DB instance class supports Intel Hyper threading technology which enables multiple threads to run parallelly on single Intel Xeon CPU Core. Each thread is represented as vCPU on DB Instance. For example db.m4.xlarge DB Instance class has 2 CPU Core and two threads per CPU Core which makes to total of 4 vCPU’s. Note: You can disable Intel Hyper threading by specifying a single thread per CPU core when you need a high-performance computing workload.
  2. To set the Core count and Threads per core you need to edit the processor features.
  3. Quick note: To compare the CPU capacity between different DB instance class you should use ECU (Amazon EC2 instance compute units). The amount of CPU that is allocated to a DB instance is expressed in terms of EC2 compute units.
  4. You can use EBS optimised volumes which are good for your DB instance as it provides better performance by minimizing contention between I/O and other traffic from your instance.

DB Instance Storage

You can attach EBS the block level storage volumes to a running instance. DB Instance storage comes with:

  • General purpose (SSD) [gp2 and gp3]: They are cost effective which is ideal for board range of workload on medium sized Generally, they have throughput limit of 250MB/second.
  • For GP2
    • 3 IOPS for each GB with min 100 IOPS (I/O Operations per second)
    • 16000 IOPS for 5.34TB is max limit in gp2  
    • Throughput is max 250MB/sec where throughput is how fast the storage volume can perform read and write.
  • For GP3
    • Up to 32000 IOPS
  • Provisioned IOPS (PIOPS) [io1]: They are used when you need low I/O Latency, consistent I/O throughput. These are suited for production environments.
    • For io1 – up to 256000 (IOPS) and throughput up to 4000 MB/s
    • Note: Benefits of using provisioned IOPS are
      • Increase number of I/O requests that system cab process.
      • Decreased latency because less I/O requests will be in queue.
      • Faster response time and high database throughput.

Blue/Green Deployments

Blue/Green deployments copies database environments in a separate environment. You can make changes in staging environment and then later push those changes in production environments. Blue/ Green deployments are only available for RDS for MariaDB and RDS for MySQL.

Working with Read Replicas

  • Updates from primary DB are copied to the read replicas.
  • You can promote read replica to be standalone DB as well in case you require sharing (Share nothing DB)
  • You can use or create read replica in different AWS Region as well.
  • How does cross region replication works?
  • IAM role of Destination must have access to Source DB Instance.
    • Source DB acts as source
    • RDS creates automated DB Snapshot of source DB
    • Copy of Snapshot starts
    • Destination read replica uses copied DB Snapshot

Cross Region Read Replicas

With Cross region read replicas you can create read replicas in a different region from the source DB instance.

Cross Region Automated Backups

You can configure DB instance to replicate snapshots and transaction logs in another AWS region.

Multi AZ Deployments

  • You can run your database instance in various AZ’s also known as multi-AZ deployments. Amazon automatically provisions and maintains secondary standby instance in different Availability zones. With this approach the primary DB replicates the data written into it to standby instance located in another AZ. Note: Instance in secondary can also be configured as read replicas.
  • You can align one standby or two standby instances.
  • When you have one standby instance it is known as Multi AZ DB instance deployment where one standby instance provides failover support but doesn’t act as read replica.
  • With Two standby instance it is known as Multi AZ DB cluster.
  • The failover mechanism automatically changes the Domain Name System (DNS) record of the DB instance to point to the standby DB instance

Note: DB instances with multi-AZ DB instance deployments can have increased write and commit latency compared to single AZ deployment.

Multi AZ DB instance deployment

In a Multi-AZ DB instance deployment, Amazon RDS automatically provisions and maintains a synchronous standby replica in a different Availability Zone.  You can’t use a standby replica to serve read traffic

If a planned or unplanned outage of your DB instance results from an infrastructure defect, Amazon RDS automatically switches to a standby replica in another Availability Zone if you have turned on Multi-AZ.

How to convert a single DB instance to Multi AZ DB instance deployment

  • Take a snapshot of primary DB instances EBS volume.
  • Creates a new volume for standby replicas from snapshot.
  • Next, turn on block level

Multi-AZ DB Cluster Deployments

  • It has one writer DB instance
  • It has two reader DB instances and allows clients to read the data.
  • AWS RDS replicates writer  
  • Data is synched from Writer instance to both the reader instances.
  • If a failover happens on of the writer instance then the reader instance acts as a automatic failover targets.  It does so by promoting a reader DB instance to a new writer DB instance. It happens automatically within 35 seconds and you can also do by going on Failover tab.

Cluster Endpoint

The cluster endpoint can write as well as read the data. The endpoint cannot be modified.

Reader Endpoint

Reader endpoint is used for reading the content from the DB cluster.

Instance Endpoint

These are used to connect to the DB instance directly to address the issues within instance or your application might require fine grained load balancing.

DB cluster parameter group

DB cluster parameter group acts as a container for engine configuration values that are applied to every DB instance in the Multi-AZ DB cluster

Replica Lag

The Difference in time between latest transaction on writer DB instance and latest applied transaction on reader instance. This could be because of high write concurrency or heavy batch updating.

How to Solve Replica Lag

You can solve the replica lag by reducing the load on your writer DB instance. You can also use Flow control to reduce the replica lag. In Flow log you can add a delay into the end of a transaction, which decreases the write throughput on writer instance. To turn on flow control use the below parameter. By default it is set to 120 seconds and you can turn off by setting to 84000 seconds or less than 120 .

Flow control works by throttling writes on the writer DB instance, which ensures that replica lag doesn’t continue to grow unbounded. Write throttling is accomplished by adding a delay. Throttling means queue or let it flow.

rpl_semi_sync_master_target_apply_lag

To check the status of flow control use below command.

SHOW GLOBAL STATUS like '%flow_control%';

DB Instance pricing

  • DB Instance are billed per hour.
  • Storage are billed per GB per month.
  • I/O requests (per 1 million requests per month.
  • Data transfer per GB in and out of your DB Instance.

AWS RDS performance troubleshooting

  1. Setup CloudWatch monitoring
  2. Enable Automatic backups
  3. If your DB requires more I/O, then to increase migrate to new instance class, convert from magnetic to general or provisioned IOPS.
  4. If you already have provisioned IOPS, consider adding more throughput capacity.
  5. If your app is caching DNS data of your instance, then make sure to set TTL value to less than 30 seconds because caching can lead to connection failures.
  6. Setup enough memory (RAM)
  7. Enable Enhanced monitoring to identify the Operating system issues
  8. Fine tune your SQL queries.
  9. Avoid tables in your database to grow too large as they impact Read and Writes.
  10. You can use options groups if you need to provide additional security for your database.
  11. You can use DB parameter group acts as a container for engine configuration values that are applied to one or more DB instances.

Tagging AWS RDS Resources

  • Tags are very helpful and are basically key value pair formats.
  • You can use Tags in IAM policies to manage access to AWS RDS resources.
  • Tags can be used to produce the detailed billing reports.
  • You can specify if you need tags to be applied to snapshots as well.
  • Tags are useful to determine which instance to be stopped, started, enable backups.

Amazon RDS Storage

Increasing DB instance storage capacity

Click on Modify in Databases and then Allocated Storage and apply immediately.  

Managing capacity automatically with Amazon RDS storage autoscaling

If workload is unpredictable then enable autoscaling for an Amazon RDS DB Instance. While creating the database engine, enable storage autoscaling and set the maximum storage threshold.

Modifying settings for Provisioned IOPS SSD storage

You can change that is reduce the amount of IOPS for your instance (throughput ) i.e read and write operations however with Provisioned IOPS SSD Storage you cannot reduce the storage size.

Monitoring Events, Logs and Streams in an Amazon RDS DB Instance.

Amazon Event Bridge: Serverless Event bus service that allows to connect apps with data from various sources.

Cloud trail logs and Cloud Watch logs are useful.

Database Activities Streams: AWS RDS push activities to Amazon Kinesis data stream

How to grant Amazon RDS to publish the notifications to the SNS topic using the IAM Policy.

The IAM Policy will be attached to the SNS service.

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "events.rds.amazonaws.com"
      },
      "Action": [
        "sns:Publish"
      ],
      "Resource": "arn:aws:sns:us-east-1:123456789012:topic_name",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:rds:us-east-1:123456789012:db:prefix-*"
        },
        "StringEquals": {
          "aws:SourceAccount": "123456789012"
        }
      }
    }
  ]
}

RDS logs

  • Amazon RDS doesn’t provide host access to the database logs on the file system of your DB instance. You can Choose the Logs & events tab to view the database log files and logs on the console itself.
  • To publish SQL Server DB logs to CloudWatch Logs from the AWS Management Console. In the Log exports section, choose the logs that you want to start publishing to CloudWatch Logs.

Note: In CloudWatch Logs, a log stream is a sequence of log events that share the same source. Each separate source of logs in CloudWatch Logs makes up a separate log stream. A log group is a group of log streams that share the same retention, monitoring, and access control settings.

  • Amazon RDS provides a REST endpoint that allows access to DB instance log files and you can find the log using REST Endpoint.
GET /v13/downloadCompleteLogFile/DBInstanceIdentifier/LogFileName HTTP/1.1
Content-type: application/json
host: rds.region.amazonaws.com
  • RDS for MySQL writes mysql-error.log to disk every 5 minutes. You can write the RDS for MySQL slow query log and the general log to a file or a database table. You can direct the general and slow query logs to tables on the DB instance by creating a DB parameter group and setting the log_output server parameter to TABLE
    • slow_query_log: To create the slow query log, set to 1. The default is 0.
    • general_log: To create the general log, set to 1. The default is 0.
    • long_query_time: To prevent fast-running queries from being logged in the slow query log

MySQL removes log files more than two weeks old. You can manually rotate the log tables with the following command line procedures, 

CALL mysql.rds_rotate_slow_log;

AWS RDS Proxy

  • RDS Proxy allows you to pool and share db connections to improve ability to scale.
  • RDS Proxy makes applications more effective to db failures by automatically connecting to Standby DB instance.
  • RDS Proxy establishes a database connection pool and reuses connections in this pool and avoids the memory and CPU overhead of opening a new database connection each time.
  • You can enable RDS Proxy for most applications with no code changes.

You can use RDS Proxy in the following scenarios.

  • Any DB instance or cluster that encounters “too many connections” errors is a good candidate for associating with a proxy.
  • For DB instances or clusters that use smaller AWS instance classes, such as T2 or T3, using a proxy can help avoid out-of-memory conditions
  • Applications that typically open and close large numbers of database connections and don’t have built-in connection pooling mechanisms are good candidates for using a proxy.

Amazon RDS for MySQL

There are two versions that are available for MySQL database engines i.e. version 8.0  and 5.7. MySQL provides the validate_password plugin for improved security. The plugin enforces password policies using parameters in the DB parameter group for your MySQL DB instance

To find the available version in MySQL which are supported:

aws rds describe-db-engine-versions --engine mysql --query *[].{Engine:Engine,EngineVersion:EngineVersion}" --output text

SSL/TLS on MySQL DB Instance

Amazon RDS installs SSL/TLS Certificate on the DB Instance. These certificate are signed by CA.  

To connect to DB instance with certificate use below command.

mysql -h mysql–instance1.123456789012.us-east-1.rds.amazonaws.com --ssl-ca=global-bundle.pem --ssl-mode=REQUIRED -P 3306 -u myadmin -p

To check if applications are using SSL.

mysql> SELECT id, user, host, connection_type

       FROM performance_schema.threads pst

       INNER JOIN information_schema.processlist isp

       ON pst.processlist_id = isp.id;

Performance improvements on MySQL RDS for Optimised reads.

  • An instance store provides temporary block-level storage for your DB instance.
  • With RDS Optimized reads some temporary objects are stored on Instance store. These objects include temp files, internal on disk temp tables, memory map files, binary logs, cached files.
  • The storage is located on Non-Volatile Memory express SSD’s that are physically attached.
  • Applications that can uses RDS for Optimized reads are:
    • Applications that run on-demand or dynamic reporting queries.
    • Applications that run analytical queries.
    • Database queries that perform grouping or ordering on non-indexed columns
  • Try to add retry logic for read only queries.
  • Avoid bulk changes in single transaction.
  • You can’t change the location of temporary objects to persistent storage (Amazon EBS) on the DB instance classes that support RDS Optimized Reads.
  • Transactions can fail when the instance store is full.
  • RDS Optimized Reads isn’t supported for multi-AZ DB cluster deployments.

Importing Data into MySQL with different data source.

  1. Existing MySQL database on premises or on Amazon EC2: Create a backup of your on-premises database, store it on Amazon S3, and then restore the backup file to a new Amazon RDS DB instance running MySQL.
  2. Any existing database: Use AWS Database Migration Service to migrate the database with minimal downtime
  3. Existing MySQL DB instance: Create a read replica for ongoing replication. Promote the read replica for one-time creation of a new DB instance.
  4. Data not stored in an existing database: Create flat files and import them using the mysqlimport utility.

Database Authentication with Amazon RDS

For PostgreSQL, use one of the following roles for a user of a specific database.

  • IAM Database authentication: assign rds_iam role to user
  • Kerberos authentication  assign rds_ad role to the user.
  • Password authentication don’t assign above roles.

Password Authentication

  • With Password authentication, database performs all the administration of user accounts. Database controls and authenticate the user accounts.

IAM Database authentication

  • IAM database authentication works with MySQL and PostgreSQL. With this authentication method, you don’t need to use a password when you connect to a DB instance

Kerberos Authentication

Benefits of using SSO and centralised authentication of database users.

Connecting to your DB instance using IAM authentication from the command line: AWS CLI and mysql client

  • In the Database authentication section, choose Password and IAM database authentication to enable IAM database authentication.
  • To allow an IAM user or role to connect to your DB instance, you must create an IAM policy.
{

   "Version": "2012-10-17",

   "Statement": [

      {

         "Effect": "Allow",

         "Action": [

             "rds-db:connect"

         ],

         "Resource": [

             "arn:aws:rds-db:us-east-2:1234567890:dbuser:db-ABCDEFGHIJKL01234/db_user"

         ]

      }

   ]

}

Create database user account using IAM authentication

CREATE USER jane_doe IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
CREATE USER db_userx;
GRANT rds_iam TO db_userx;

Generate an IAM authentication token

aws rds generate-db-auth-token --hostname rdsmysql.123456789012.us-west-2.rds.amazonaws.com --port 3306 --region us-west-2  --username jane_doe

Connecting to DB instance

mysql –host=hostName –port=portNumber –ssl-ca=full_path_to_ssl_certificate –enable-cleartext-plugin –user=userName –password=authToken

Connecting to DB using Python

import pymysql
import sys
import boto3
import os

ENDPOINT="mysqldb.123456789012.us-east-1.rds.amazonaws.com"
PORT="3306"
USER="jane_doe"
REGION="us-east-1"
DBNAME="mydb"

os.environ['LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN'] = '1'

#gets the credentials from .aws/credentials
session = boto3.Session(profile_name='default')
client = session.client('rds')
token = client.generate_db_auth_token(DBHostname=ENDPOINT, Port=PORT, DBUsername=USER, Region=REGION)
try:
    conn =  pymysql.connect(host=ENDPOINT, user=USER, passwd=token, port=PORT, database=DBNAME, ssl_ca='SSLCERTIFICATE')

    cur = conn.cursor()
    cur.execute("""SELECT now()""")
    query_results = cur.fetchall()
    print(query_results)

except Exception as e:
    print("Database connection failed due to {}".format(e))

   

Final AWS RDS Troubleshooting’s

Can’t connect to Amazon RDS DB instance

  • Check Security group
  • Check Port
  • Check internet Gateway
  • Check db name

Error – Could not connect to server: Connection timed out

  • Check hostname and port
  • Check security group
  • Telnet to the DB
  • Check the username and password

Error message “failed to retrieve account attributes, certain console functions may be impaired.”

  • Account is missing permissions, or your account hasn’t been properly set up.
  • lack permissions in your access policies to perform certain actions such as creating a DB instance

Amazon RDS DB instance outage or reboot

  • You change the backup retention period for a DB instance from 0 to a nonzero value or from a nonzero value to 0. You then set Apply Immediately to true.
  • You change the DB instance class, and Apply Immediately is set to true.
  • You change the storage type from Magnetic (Standard) to General Purpose (SSD) or Provisioned IOPS (SSD), or from Provisioned IOPS (SSD) or General Purpose (SSD) to Magnetic (Standard).

Amazon RDS DB instance running out of storage

  • Add more storage in  EBS volumes attached to the DB instance.

Amazon RDS insufficient DB instance capacity

The specific DB instance class isn’t available in the requested Availability Zone. You can try one of the following to solve the problem:

  • Retry the request with a different DB instance class.
  • Retry the request with a different Availability Zone.
  • Retry the request without specifying an explicit Availability Zone.

Maximum MySQL and MariaDB connections

  • The connection limit for a DB instance is set by default to the maximum for the DB instance class. You can limit the number of concurrent connections to any value up to the maximum number of connections allowed.
  • A MariaDB or MySQL DB instance can be placed in incompatible-parameters status for a memory limit when The DB instance is either restarted at least three time in one hour or at least five times in one day or potential memory usage of the DB instance exceeds 1.2 times the memory allocated to its DB instance class. To solve the issue:
    • Adjust the memory parameters in the DB parameter group associated with the DB instance.
    • Restart the DB instance.

How to Launch an Amazon DynamoDB tables in AWS Account

With rise in number of database it has become a big challenge to make the right selection. As data grows our database should also scale and perform equally well.

Now Organizations have started to move toward big data and working with real time applications we certainly need a non relational and a good performance database. For these types of challenges and work AWS has always been on the top and served various services which solves our problems and one such service is AWS DynamoDB which manages non-relational databases for you and can store unlimited data and perform very well. .

Table of content

  1. What is Relational database management system ?
  2. What is SQL and NO SQL database?
  3. What is Amazon DynamoDB ?
  4. Prerequisites
  5. How to Create tables in DynamoDB in AWS Account
  6. Conclusion

What is Relational database management system ?

  • Relational database is based on tables and structured data
  • They have relationships which are logically connected.
  • Oracle Database, MySQL, Microsoft SQL Server, and IBM DB2. PostgreSQL , SQLite (for mobiles) are few example of RDMS.

Figure shows Relational Database Management System based on relational model

What is SQL and NO SQL database?

SQL:

  • The full form of SQL is structured query language which is used to manage data in relational database management system i.e RDMS.
  • SQL database belongs to the relational database management system.
  • The SQL type database follow structure pattern that’s why they are suitable for static or predefined schemas.
  • They are good in solving complex queries and highly scalable in nature but in vertical direction.
  • SQL database follows table based methodology and that’s the reason they are good for applications such as accounting systems.

NoSQL:

  • The full form of NoSQL is non-sql or non-relational.
  • This database is used for dynamic storage or those kind of managements where data is not fixed or static
  • This database is not tabular in nature rather its a key pair values.
  • They are good for big data and real time web application and scalable in nature but in horizontal direction
  • Some of the NoSQL databases which are DynamoDB, Foundation DB, Infinity DB, MemcacheDB, , Oracle NoSQL Database, , Redis MongoDB, Cassandra, Scylla, HBase.

What is Amazon DynamoDB ?

DynamoDB is a NoSQL database service that means it is different from the relational database which consists of tables in tabular form. DynamoDB service has very fast performance and is very scalable. DynamoDB service is one of the AWS managed service where you don’t need to worry about capacity , workload , setup , configuration , software patches , replications or even cluster scaling.

With DynamoDB service you just need to create tables where you can add data or retrieve data otherwise DynamoDB takes care of everything else. If you wish to monitor your resources you can do it on AWS console.

Whenever there is a traffic or high request coming in DynamoDB scales up while maintaining the performance.

Basic components of Amazon DynamoDB

  • Tables: It stores data.
    • In below example we used a database table
  • Items: Items are present in table. You can store as many item you wish in a table.
    • In below example different Employee ID are items.
  • Attributes: Each items contains one or more attributes.
    • In below example office , designation and phone are attributes of EmployeeID.

{
"EmployeeID": "1"
"office": "USA"
"Designation": "Devops engineer"
"Phone": "1234567890"
}


{
"EmployeeID": "2"
"office": "UK"
"Designation": "Senior Devops Engineer"
"Phone": "0123456789"
}

To work with Amazon DynamoDB , applications will need API’s to communicate.

  • Control Plane: It allows you to create and manage DynamoDB tables.
  • Data lane: It allows you to perform actions on the data in DynamoDB tables.

Prerequisites

  • You should have AWS account with Full access permissions on DynamoDB . If you don’t have AWS account, please create a account from here AWS account.

How to Create tables in DynamoDB in AWS Account

  • Go to AWS account and search for DynamoDB on the top of the page.
  • Click on Create Table and then you need to Enter the name of the Table and primary Key
  • Now click on Organisation that is table name
  • Now click on Items
  • Add the list of items such address , designation and phone number.
  • Verify if table has required details.

So this was the first way to use AWS provided web service and directly start creating DynamoDB tables . The other way is to download it manually on your machine , setup and then create you’re tables . You can find the steps here

Conclusion

You should now have a basic knowledge about relational database management system and non relational. We also learned about Amazon DynamoDB which is NO SQL database . We also covered on how to create tables on Amazon DynamoDB service & store the data .

This tutorial consists of all the practical’s which were done on our lab server with lots of hard work and efforts. Please share the word if you like it and hoping you get benefit out of this tutorial.

How to Launch AWS Elastic beanstalk using Terraform

If you want to scale instances, align a load balancer in front of them, host a website, and store all data in the database. Nothing could be better than Amazon Elastic beanstalk, which provides a common platform.

With Elastic Beanstalk, you can quickly deploy and manage applications in the AWS Cloud without having to worry about the infrastructure that runs those applications.

In this tutorial, we will learn how to step up Amazon Elastic beanstalk using Terraform on AWS step by step and then upload the code to run one of the simple applications.

Let’s get started.

Join 48 other followers

Table of Content

  1. What is AWS Elastic beanstalk?
  2. Prerequisites
  3. Building Terraform configuration files for AWS Elastic beanstalk
  4. Deploying Terraform configuration to Launch AWS Elastic beanstalk
  5. Verifying AWS Elastic beanstalk in AWS Cloud.
  6. Conclusion

What is AWS Elastic beanstalk?

AWS Elastic Beanstalk is one of the most widely used Amazon web service tool services. It is a service that provides a platform for various languages such as python, go ruby, java, .net, PHP for hosting the application.

The only thing you need to do in elastic beanstalk is upload code, and the rest of the things such as scaling, load balancing, monitoring will be taken care of by elastic beanstalk itself.

Elastic beanstalk makes the life of developer and cloud admins or sysadmins so easy compared to setting each service individually and interlinking each other. Some of the key benefits of AWS Elastic beanstalk are:

  • It scales the applications up or down as per the required traffic.
  • As infrastructure is managed and taken care of by AWS Elastic beanstalk developers working with admins don’t need to spend much time.
  • It is fast and easy to setup
  • You can interlink with lots of other AWS services of your own choice or you can skip it such as linking of application or classic or network load balancer.

Prerequisites

  • Ubuntu machine to run terraform preferably 18.04 version + , if you don’t have any machine you can create a ec2 instance on AWS account. Recommended to have 4GB RAM and at least 5GB of drive space.
  • Ubuntu machine should have IAM role attached with AWS Elastic beanstalk creation permissions or admin rights or access key and secret key configured in AWS CLI.
  • Terraform installed on the Ubuntu Machine. Refer How to Install Terraform on an Ubuntu machine.

You may incur a small charge for creating an EC2 instance on Amazon Managed Web Service.

Building Terraform configuration files for AWS Elastic beanstalk

Now that you have Terraform installed on your machine, It’s time to build Terraform configuration files for AWS Elastic beanstalk that you will use to launch AWS Elastic beanstalk on the AWS Cloud.

Assuming you are still logged in Ubuntu machine.

  • Create a folder in opt directory and name it as terraform-elasticbeanstalk-demo and switch to this directory.
mkdir /opt/terraform-elasticbeanstalk-demo
cd /opt/terraform-elasticbeanstalk-demo
  • Create a file named main.tf in the /opt/terraform-elasticbeanstalk-demo directory and copy/paste the below content into it. The below Terraform configuration creates the AWS elastic beanstalk application and enviornment that will be required for application to be deployed.
# Create elastic beanstalk application

resource "aws_elastic_beanstalk_application" "elasticapp" {
  name = var.elasticapp
}

# Create elastic beanstalk Environment

resource "aws_elastic_beanstalk_environment" "beanstalkappenv" {
  name                = var.beanstalkappenv
  application         = aws_elastic_beanstalk_application.elasticapp.name
  solution_stack_name = var.solution_stack_name
  tier                = var.tier

  setting {
    namespace = "aws:ec2:vpc"
    name      = "VPCId"
    value     = var.vpc_id
  }
  setting {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "IamInstanceProfile"
    value     =  "aws-elasticbeanstalk-ec2-role"
  }
  setting {
    namespace = "aws:ec2:vpc"
    name      = "AssociatePublicIpAddress"
    value     =  "True"
  }

  setting {
    namespace = "aws:ec2:vpc"
    name      = "Subnets"
    value     = join(",", var.public_subnets)
  }
  setting {
    namespace = "aws:elasticbeanstalk:environment:process:default"
    name      = "MatcherHTTPCode"
    value     = "200"
  }
  setting {
    namespace = "aws:elasticbeanstalk:environment"
    name      = "LoadBalancerType"
    value     = "application"
  }
  setting {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "InstanceType"
    value     = "t2.medium"
  }
  setting {
    namespace = "aws:ec2:vpc"
    name      = "ELBScheme"
    value     = "internet facing"
  }
  setting {
    namespace = "aws:autoscaling:asg"
    name      = "MinSize"
    value     = 1
  }
  setting {
    namespace = "aws:autoscaling:asg"
    name      = "MaxSize"
    value     = 2
  }
  setting {
    namespace = "aws:elasticbeanstalk:healthreporting:system"
    name      = "SystemType"
    value     = "enhanced"
  }

}

  • Create another file named vars.tf in the /opt/terraform-elasticbeanstalk-demo directory and copy/paste the below content into it. The variable file contains all the variables that you have referred in main.tf file.
variable "elasticapp" {
  default = "myapp"
}
variable "beanstalkappenv" {
  default = "myenv"
}
variable "solution_stack_name" {
  type = string
}
variable "tier" {
  type = string
}

variable "vpc_id" {}
variable "public_subnets" {}
variable "elb_public_subnets" {}

  • Create another file named provider.tf in the /opt/terraform-elasticbeanstalk-demo directory and copy/paste the below content into it. The provider.tf file will authenticate and allows Terraform to connect to AWS cloud.
provider "aws" {
  region = "us-east-2"
}
  • Finally create one more file named terraform.tfvars in the /opt/terraform-elasticbeanstalk-demo directory and copy/paste the below content into it.
vpc_id              = "vpc-XXXXXXXXX"
Instance_type       = "t2.medium"
minsize             = 1
maxsize             = 2
public_subnets     = ["subnet-XXXXXXXXXX", "subnet-XXXXXXXXX"] # Service Subnet
elb_public_subnets = ["subnet-XXXXXXXXXX", "subnet-XXXXXXXXX"] # ELB Subnet
tier = "WebServer"
solution_stack_name= "64bit Amazon Linux 2 v3.2.0 running Python 3.8"

  • Now use tree command on your ubuntu machine and your folder structure should look something like below.
 tree command on your ubuntu machine and your folder structure
tree command on your ubuntu machine and your folder structure

Deploying Terraform configuration to Launch AWS Elastic beanstalk

Now that all Terraform configuration files are set up, these are not doing much unless you use Terraform commands and deploy them.

  • To deploy the AWS Elastic beanstalk first thing you need to do is Initialize the terraform by running terraform init command.
terraform init

As you see below, Terraform was initialized successfully; now, it’s time to run terraform plan.

 Terraform was initialized successfully
Terraform was initialized successfully
  • Next run the terraform plan command. Teraform plan command provides the information regarding what all resources will be provisioned or deleted by Terraform.
terraform plan
Running Terraform plan command
Running Terraform plan command
  • Finally run terraform apply command that actually deploy the code and provision the AWS Elastic terraform.
terraform apply

Verifying AWS Elastic beanstalk in AWS Cloud.

Great Job; terraform commands were executed successfully. Now it’s time to validate the AWS Elastic beanstalk launched in AWS Cloud.

  • Navigate to the AWS cloud and then futher in AWS Elasticbeanstalk service. After you reach elastic beanstalk screen you will see the enviornment and applciation name that you specified in terraform.tfvar file.
AWS Elasticbeanstalk service page
AWS Elasticbeanstalk service page
  • Next in AWS Elastic beanstalk service page click on the application URL and you will see something like below.
AWS Elasticbeanstalk service link
AWS Elasticbeanstalk service link

Join 48 other followers

Conclusion

In this tutorial, you learned what AWS Elastic beanstalk is and how to set up Amazon Elastic beanstalk using Terraform on AWS step by step.

Now that you have AWS Elastic beanstalk launched on AWS using Terraform, which applications do you plan to deploy on it next?

How to Launch multiple EC2 instances on AWS using Terraform count and Terraform for_each

Creating multiple AWS EC2 instances is generally the need of the project or the organization when you are asked to create dozens of AWS EC2 machines in a particular AWS account, and using AWS console will take hours to do that why not automate it using Terraform and save your hours of hard work?

There are various automated ways that can create multiple instances quickly, but automating with Terraform is way easier and more fun.

In this tutorial, you will learn how to Launch multiple AWS EC2 instances on AWS using Terraform count and Terraform for_each. Let’s dive in.

Join 48 other followers

Table of Content

  1. What is Amazon EC2 instance?
  2. Prerequisites
  3. Terraform files and Terraform directory structure
  4. Launch multiple EC2 instances using Terraform count
  5. Launch multiple EC2 instances using Terraform for_each
  6. Conclusion

What is Amazon EC2 instance?

Amazon Elastic Compute Cloud (Amazon EC2) provides the scalable capacity in the Amazon Web Services (AWS) Cloud. With AWS EC2, you don’t need to worry about the hardware and time to develop and deploy applications on the machines.

You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down the computations such as memory or CPU when needed. Also, AWS EC2 instances are safe as initially, they grant access to them using SSH keys.

Prerequisites

  • Ubuntu machine 20.04 version would be great , if you don’t have any machine you can create a AWS EC2 instance on AWS account with recommended 4GB RAM and at least 5GB of drive space.
  • Ubuntu machine should have IAM role attached with full access to create AWS secrets in the AWS Secret Manager or administrator permissions.
  • Terraform installed on the Ubuntu Machine. Refer How to Install Terraform on an Ubuntu machine.

You may incur a small charge for creating an EC2 instance on Amazon Managed Web Service.

Terraform files and Terraform directory structure

Now that you have Terraform installed. Let’s now dive into Terraform files and Terraform directory structure that will help you write the Terraform configuration files later in this tutorial.

Terraform code, that is, Terraform configuration files, are written in a tree-like structure to ease the overall understanding of code with .tf format or .tf.json or .tfvars format. These configuration files are placed inside the Terraform modules.

Terraform modules are on the top level in the hierarchy where configuration files reside. Terraform modules can further call another child to terraform modules from local directories or anywhere in disk or Terraform Registry.

Terraform contains mainly five files as main.tf , vars.tf , providers.tf , output.tf and terraform.tfvars.

  1. main.tf – Terraform main.tf file contains the main code where you define which resources you need to build, update or manage.
  2. vars.tf – Terraform vars.tf file contains the input variables which are customizable and defined inside the main.tf configuration file.
  3. output.tf : The Terraform output.tf file is the file where you declare what output paraeters you wish to fetch after Terraform has been executed that is after terraform apply command.
  4. .terraform: This directory contains cached provider , modules plugins and also contains the last known backend configuration. This is managed by terraform and created after you run terraform init command.
  5. terraform.tfvars files contains the values which are required to be passed for variables that are refered in main.tf and actually decalred in vars.tf file.
  6. providers.tf – The povider.tf is the most important file whrere you define your terraform providers such as terraform aws provider, terraform azure provider etc to authenticate with the cloud provider.

Launch multiple EC2 instances using Terraform count

Another special argument is Terraform count. By default, terraform create a single resource defined in Terraform resource block. But at times, you want to manage multiple objects of the same kind, such as creating four AWS EC2 instances of the same type in the AWS cloud without writing a separate block for each instance. Let’s learn how to use Terraform count meta argument.

This demonstration will create multiple AWS EC2 instances using Terraform count. So let’s create all the Terraform configuration files required to create multiple AWS EC2 instances on the AWS account.

  • Log in to the Ubuntu machine using your favorite SSH client.
  • Create a folder in opt directory named terraform-demo and switch to this folder. This terraform-demo folder will contain all the configuration files that Terraform needs.
mkdir /opt/terraform-demo
cd /opt/terraform-demo
  • Create main.tf file in the /opt/terraform-demo directory and copy/paste the content below. The below code creates the four identical AWS EC2 instances in AWS account using Terraform count meta argument.
resource "aws_instance" "my-machine" {
   count = 4   # Here we are creating identical 4 machines. 
   ami = var.ami
   instance_type = var.instance_type
   tags = {
      Name = "my-machine-${count.index}"
           }
}
  • Create another file named vars.tf file in the /opt/terraform-demo directory and copy/paste the content below. The vars.tf file contains all the variables that you reffered in the main.tf file.
# Creating a Variable for ami
variable "ami" {       
  type = string
}

# Creating a Variable for instance_type
variable "instance_type" {    
  type = string
}
  • Create another file named terraform.tfvars file in the /opt/terraform-demo directory and copy/paste the content below. The terraform.tfvars file contains all the values that are needed by variables declared in the var.tf file.
 ami = "ami-0742a572c2ce45ebf"
 instance_type = "t2.micro"

  • Create one more file named outputs.tf inside the /opt/terraform-demo directory and copy/paste the below content. This file contains all the outputs variables that will be used to display he output after running the terraform apply command.
output "ec2_machines" {
 # Here * indicates that there are more than one arn because count is 4   
  value = aws_instance.my-machine.*.arn 
}
 
  • Create another file and name it as provider.tf. This file allows Terraform to interact with AWS cloud using AWS API.
provider "aws" {
  region = "us-east-2"
}
  • Now your folder should have all files as shown below and should look like.
Terraform configurations and structure
Terraform configurations and structure
  • Now your files and code are ready for execution. Initialize the terraform using the terraform init command.
terraform init
Initialize the terraform using the terraform init command.
Initialize the terraform using the terraform init command.
  • Terraform initialized successfully , now its time to run the plan command which provides you the details of the deployment. Run terraform plan command to confirm if correct resources is going to provisioned or deleted.
terraform plan
Running terraform plan command
Running terraform plan command
Output of the terraform plan command
The output of the terraform plan command
  • After verification, now its time to actually deploy the code using terraform apply command.
terraform apply
Running terraform apply command
Running terraform apply command

Terraform commands terraform init→ terraform plan→ terraform apply All executed successfully. But it is important to manually verify all the four AWS instances launched in AWS.

  • Open your favorite web browser and navigate to the AWS Management Console and log in.
  • While in the Console, click on the search bar at the top, search for ‘EC2’, and click on the EC2 menu item and you should see four EC2 instances.
Four instance launched using Terraform count
Four instances launched using Terraform count

Launch multiple EC2 instances using Terraform for_each

In the previous example, you created more than four AWS instances, but all the instances contain the same attributes such as instance_type, ami, etc. But if you need to create multiple instances with different attributes, such as one instance with t2.medium and others with t2.micro types, you should consider using Terraform for_each.

Assuming you are still logged into the Ubuntu machine using your favorite SSH client.

  • Create a folder in opt directory named terraform-for_each-demo and switch to this folder. This terraform-for_each-demo folder will contain all the configuration files that Terraform needs.
mkdir /opt/terraform-for_each-demo
cd /opt/terraform-for_each-demo
  • Create main.tf file in the /opt/terraform-for_each-demo directory and copy/paste the content below. The below code creates the two AWS EC2 instances with different instance_type in AWS account using Terraform for_each argument.
resource "aws_instance" "my-machine" {
  ami = var.ami
  for_each  = {                     # for_each iterates over each key and values
      key1 = "t2.micro"             # Instance 1 will have key1 with t2.micro instance type
      key2 = "t2.medium"            # Instance 2 will have key2 with t2.medium instance type
        }
        instance_type  = each.value
	key_name       = each.key
    tags =  {
	   Name  = each.value
	}
}
  • Create another file vars.tf file in the /opt/terraform-for_each-demo directory and copy/paste the content below.
variable "tag_ec2" {
  type = list(string)
  default = ["ec21a","ec21b"]
}
                                           
variable "ami" {       # Creating a Variable for ami
  type = string
}
  • Create another file terraform.vars file in the /opt/terraform-for_each-demo directory and copy/paste the content below.
ami = "ami-0742a572c2ce45ebf"
instance_type = "t2.micro"
  • Now that you have all the Terraform configurations read for execution.
  • Next initialize the Terraform using terraform init command followed by terraform plan and finally terraform apply to deploy the changes.
terraform init 
terraform plan
terraform apply
Two instance launched using Terraform for_each
Two instances launched using Terraform for_each

Join 48 other followers

Conclusion

Terraform is a great open-source tool that provides the easiest code and configuration files to work with. Now that you know how to launch multiple AWS EC2 instances on AWS using Terraform count and Terraform for_each on Amazon Web Service.

So which argument do you plan to use in your next Terraform deployment?