How to Launch AWS Elasticsearch using Terraform (Terraform aws elasticsearch)

Machine-generated data is growing exponentially, and getting insights is important for your business so that you can search unstructured or semi-structured data on your site. You need a search analytic solution with speed, scalability, flexibility, and real-time search, and this is possible with Latest Amazon OpenSearch Service ( successor to Amazon Elasticsearch Service.

In this tutorial, you will learn about Amazon OpenSearch Service, Amazon Elasticsearch, and how to create an Amazon Elasticsearch domain using Terraform.

Let’s get started.

Join 33 other followers

Table of Content

  1. What Is Amazon Elasticsearch Service?
  2. Features of Amazon Elasticsearch Service
  3. What is Amazon OpenSearch Service?
  4. Prerequisites
  5. Terraform files and Terraform directory structure
  6. Building Terraform Configuration for AWS Elasticsearch
  7. Verify AWS Elasticsearch in Amazon Account
  8. Conclusion

What Is Amazon Elasticsearch Service?

Amazon Elasticsearch Service is a distributed search and analytics engine mainly used for log analytics, full-text search, business analytics, and operational intelligence. It performs real-time application monitoring and log analytics.

In the Amazon Elasticsearch service, you need to send the data in JSON format using the API or Logstash. Then Elasticsearch automatically stores the data and adds a searchable reference to the document in clusters index, and you can search using Elasticsearch API.

AWS Elastic search Working
AWS Elastic search Working

Amazon Elasticsearch service creates the AWS Elasticsearch clusters and nodes. If the nodes fail in the cluster, then the failed Elasticsearch nodes are automatically replaced.

Features of Amazon Elasticsearch Service

  • Amazon Elasticsearch service can scale up to 3 PB of attached storage and works with various instance types.
  • Amazon Elasticsearch easily integrates with other services such as IAM for security such as Amazon VPC , AWS S3 for loading data , AWS Cloud Watch for monitoring and AWS SNS for alerts notifications.

What is Amazon OpenSearch Service?

Amazon OpenSearch Service is a managed service that allows you to deploy, operate and scale OpenSearch clusters in Amazon Cloud. While you create the OpenSearch cluster, you can select the search engine of your choice.

Amazon OpenSearch is a fully open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. 

The latest version of OpenSearch is 1.1 and supports all elasticsearch versions, such as 7.10. 7.9 etc.

Prerequisites

Terraform files and Terraform directory structure

Now that you know what is Amazon Elastic search and Amazon OpenSearch service are. Let’s now dive into Terraform files and Terraform directory structure that will help you write the Terraform configuration files later in this tutorial.

Terraform code, that is, Terraform configuration files, are written in a tree-like structure to ease the overall understanding of code with .tf format or .tf.json or .tfvars format. These configuration files are placed inside the Terraform modules.

Terraform modules are on the top level in the hierarchy where configuration files reside. Terraform modules can further call another child to terraform modules from local directories or anywhere in disk or Terraform Registry.

Terraform contains mainly five files as main.tf , vars.tf , providers.tf , output.tf and terraform.tfvars.

  1. main.tf – Terraform main.tf file contains the main code where you define which resources you need to build, update or manage.
  2. vars.tf – Terraform vars.tf file contains the input variables which are customizable and defined inside the main.tf configuration file.
  3. output.tf : The Terraform output.tf file is the file where you declare what output paraeters you wish to fetch after Terraform has been executed that is after terraform apply command.
  4. .terraform: This directory contains cached provider , modules plugins and also contains the last known backend configuration. This is managed by terraform and created after you run terraform init command.
  5. terraform.tfvars files contains the values which are required to be passed for variables that are refered in main.tf and actually decalred in vars.tf file.
  6. providers.tf – The povider.tf is the most important file whrere you define your terraform providers such as terraform aws provider, terraform azure provider etc to authenticate with the cloud provider.

Building Terraform Configuration for AWS Elasticsearch

Now that you know what are Terraform configurations files look like and how to declare each of them. In this section, you will learn how to build Terraform configuration files for AWS Elasticsearch before running Terraform commands. Let’s get into it.

  • Log in to the Ubuntu machine using your favorite SSH client.
  • Create a folder in opt directory named terraform-Elasticsearch and switch to that folder.
mkdir /opt/terraform-Elasticsearch
cd /opt/terraform-Elasticsearch
  • Create a file named main.tf inside the /opt/terraform-Elasticsearch directory and copy/paste the below content. The below file creates the below components:
    • Creates domains are clusters with the settings, instance types, instance counts, and storage resources that you specify.
    • Creates the AWS Elasticsearch domain policy.
# Creating the Elasticsearch domain

resource "aws_elasticsearch_domain" "es" {
  domain_name           = var.domain
  elasticsearch_version = "7.10"

  cluster_config {
    instance_type = var.instance_type
  }
  snapshot_options {
    automated_snapshot_start_hour = 23
  }
  vpc_options {
    subnet_ids = ["subnet-0d8c53ffee6d4c59e"]
  }
  ebs_options {
    ebs_enabled = var.ebs_volume_size > 0 ? true : false
    volume_size = var.ebs_volume_size
    volume_type = var.volume_type
  }
  tags = {
    Domain = var.tag_domain
  }
}

# Creating the AWS Elasticsearch domain policy

resource "aws_elasticsearch_domain_policy" "main" {
  domain_name = aws_elasticsearch_domain.es.domain_name
  access_policies = <<POLICIES
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Resource": "${aws_elasticsearch_domain.es.arn}/*"
        }
    ]
}
POLICIES
}
  • Create one more file named vars.tf inside the /opt/terraform-Elasticsearch directory and copy/paste the below content. This file contains all the variables that are referred in the main.tf configuration file.
variable "domain" {
    type = string
}
variable "instance_type" {
    type = string
}
variable "tag_domain" {
    type = string
}
variable "volume_type" {
    type = string
}
variable "ebs_volume_size" {}

  • Create one more file named outputs.tf inside the /opt/terraform-Elasticsearch directory and copy/paste the below content. This file contains all the outputs variables that will be used to display he output after running the terraform apply command.
output "arn" {
    value = aws_elasticsearch_domain.es.arn
} 
output "domain_id" {
    value = aws_elasticsearch_domain.es.domain_id
} 
output "domain_name" {
    value = aws_elasticsearch_domain.es.domain_name
} 
output "endpoint" {
    value = aws_elasticsearch_domain.es.endpoint
} 
output "kibana_endpoint" {
    value = aws_elasticsearch_domain.es.kibana_endpoint
}

  • Create another file and name it as provider.tf. This file allows Terraform to interact with AWS cloud using AWS API.
provider "aws" {
  region = "us-east-2"
}
  • Create one more file terraform.tfvars inside the same folder and copy/paste the below content. This file contains the values of the variables that you declared in vars.tf file and refered in main.tf file.
domain = "newdomain" 
instance_type = "r4.large.elasticsearch"
tag_domain = "NewDomain"
volume_type = "gp2"
ebs_volume_size = 10
  • Now your folder should have all files as shown below and should look like.
 Terraform-elasticsearch folder
Terraform-elasticsearch folder
  • Now your files and code are ready for execution. Initialize the terraform using the terraform init command.
terraform init
Initializing the terraform using the terraform init command.
Initializing the terraform using the terraform init command.
  • Terraform initialized successfully , now its time to run the plan command which provides you the details of the deployment. Run terraform plan command to confirm if correct resources is going to provisioned or deleted.
terraform plan
Running the terraform plan command
Running the terraform plan command
The output of the terraform plan command
The output of the terraform plan command
  • After verification, now its time to actually deploy the code using terraform apply command.
terraform apply
Running the terraform apply command
Running the terraform apply command

Verify AWS Elasticsearch in Amazon Account

Terraform commands terraform init→ terraform plan→ terraform apply all executed successfully. But it is important to manually verify the AWS Elasticsearch domain on the AWS Management console.

  • Open your favorite web browser and navigate to the AWS Management Console and log in.
  • While in the Console, click on the search bar at the top, search for ‘Elasticsearch’, and click on the Elasticsearch menu item.
 Search for ‘Elasticsearch’ in AWS console
Search for ‘Elasticsearch’ in AWS console
  • Now you will see that the newdomain that you specified in Terraform configuration file is created succesfully.
AWS Elasticsearch domain created successfully
AWS Elasticsearch domain created successfully
  • Next, click on newdomain to check the details of the newly created domain.
Check the details of the newly created domain.
Check the details of the newly created domain.

In the new Amazon OpenSearch service, you should see something like below.

Amazon Open search domain
Amazon Open search domain

Join 33 other followers

Conclusion

In this tutorial, you learned Amazon Elasticsearch and how to create an Amazon Elasticsearch domain using Terraform.

Now that you have a strong basic understanding of AWS Elasticsearch, which documents will you upload for indexing and searching?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s