How to Launch AWS Elasticsearch using Terraform in Amazon Account

It is important to have a search engine for your website or applications. When it comes to automation of these great features such as load balancing and scalability of websites, Amazon provides it own managed service known as Amazon Elasticsearch.

In this tutorial you will learn about what is Amazon Elasticsearch and how to create a Amazon Elasticsearch domain using Terraform.

Table of Contents

  1. What Is Amazon Elasticsearch Service?
  2. Prerequisites:
  3. Terraform Configuration Files and Structure
  4. Configure Terraform files for AWS Elasticsearch
  5. Verify AWS Elasticsearch in Amazon Account
  6. Conclusion

It is important to have a search engine for your website or applications. When it comes to automation of these great features such as load balancing and scalability of websites, Amazon provides it own managed service known as Amazon Elasticsearch.

In this tutorial you will learn about what is Amazon Elasticsearch and how to create a Amazon Elasticsearch domain using Amazon Management console and then search the data using Kibana.

What Is Amazon Elasticsearch Service?

Amazon Elasticsearch Service is a managed service which deploys and scale the Elasticsearch clusters in the cloud. Elasticsearch is an open source analytical and search engine which is used to perform real time application monitoring and log analytics.

Amazon Elasticsearch service provisions all resources for Elasticsearch clusters and launches it. It also replaces the failed Elasticsearch nodes in the cluster automatically.

Features of Amazon Elasticsearch Service

  • It can scale up to 3 PB of attached storage
  • It works with various instance types.
  • It easily integrates with other services such as IAM for security for ,VPC , AWS S3 for loading data , AWS Cloud Watch for monitoring and AWS SNS for alerts notifications.

Prerequisites:

Terraform Configuration Files and Structure

Let us first understand terraform configuration files before running Terraform commands.

  • main.tf : This file contains code that create or import other AWS resources.
  • vars.tf : This file defines variable types and optionally set the values.
  • output.tf: This file helps in generating of the output of AWS resources .The output is generated after the terraform apply command is executed.
  • terraform.tfvars: This file contains the actual values of variables which we created in vars.tf
  • provider.tf: This file is very important . You need to provide the details of providers such as AWS , Oracle or Google etc. so that terraform can make the communication with the same provider and then work with resources.

Configure Terraform files for AWS Elasticsearch

In this demonstration we will create a simple Amazon Elasticsearch using Terraform from Windows machine.

  • Create a folder on your desktop on windows Machine and name it as Terraform-Elasticsearch
  • Now create a file main.tf inside the folder you’re in and paste the below content
rresource "aws_elasticsearch_domain" "es" {
  domain_name           = var.domain
  elasticsearch_version = "7.10"

  cluster_config {
    instance_type = var.instance_type
  }
  snapshot_options {
    automated_snapshot_start_hour = 23
  }
  vpc_options {
    subnet_ids = ["subnet-0d8c53ffee6d4c59e"]
  }
  ebs_options {
    ebs_enabled = var.ebs_volume_size > 0 ? true : false
    volume_size = var.ebs_volume_size
    volume_type = var.volume_type
  }
  tags = {
    Domain = var.tag_domain
  }
}


resource "aws_elasticsearch_domain_policy" "main" {
  domain_name = aws_elasticsearch_domain.es.domain_name
  access_policies = <<POLICIES
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Resource": "${aws_elasticsearch_domain.es.arn}/*"
        }
    ]
}
POLICIES
}
  • Create one more file vars.tf inside the same folder and paste the below content
variable "domain" {
    type = string
}
variable "instance_type" {
    type = string
}
variable "tag_domain" {
    type = string
}
variable "volume_type" {
    type = string
}
variable "ebs_volume_size" {}
  • Create one more file output.tf inside the same folder and paste the below content
output "arn" {
    value = aws_elasticsearch_domain.es.arn
} 
output "domain_id" {
    value = aws_elasticsearch_domain.es.domain_id
} 
output "domain_name" {
    value = aws_elasticsearch_domain.es.domain_name
} 
output "endpoint" {
    value = aws_elasticsearch_domain.es.endpoint
} 
output "kibana_endpoint" {
    value = aws_elasticsearch_domain.es.kibana_endpoint
}
  • Create one more file provider.tf inside the same folder and paste the below content:
provider "aws" {      # Defining the Provider Amazon  as we need to run this on AWS   
  region = "us-east-1"
}
  • Create one more file terraform.tfvars inside the same folder and paste the below content
domain = "newdomain" 
instance_type = "r4.large.elasticsearch"
tag_domain = "NewDomain"
volume_type = "gp2"
ebs_volume_size = 10
  • Now your files and code are ready for execution .
  • Initialize the terraform using below command.
terraform init
  • Terraform initialized successfully , now its time to see the plan which is kind of blueprint before deployment. We generally use plan to confirm if correct resources is going to provisioned or deleted.
terraform plan
  • After verification , now its time to actually deploy the code using apply.
terraform apply

Verify AWS Elasticsearch in Amazon Account

Terraform commands ( init , plan and apply ) all ran successfully. Now Lets verify it on AWS Management console of all the things were created properly.

  • Open your favorite web browser and navigate to the AWS Management Console and log in.
  • While in the Console, click on the search bar at the top, search for ‘Elasticsearch’, and click on the Elasticsearch menu item.
  • Now You will see that the newdomain is created succesfully
  • Click on newdomainto see all the details.

Conclusion

In this tutorial you learnt about what is Amazon Elasticsearch and how to create a Amazon Elasticsearch domain using Terraform

So, Now you have a strong fundamental understanding of AWS Elasticsearch , which website are you going to implement on Elasticsearch with Terraform ?

Getting Started with Amazon Elasticsearch Service and Kibana

It is important to have a search engine for your website or applications. When it comes to automation of these great features such as load balancing and scalability of websites, Amazon provides it own managed service known as Amazon Elasticsearch.

In this tutorial you will learn about what is Amazon Elasticsearch and how to create a Amazon Elasticsearch domain using Amazon Management console and then search the data using Kibana.

Table of contents

  1. What Is Amazon Elasticsearch Service?
  2. Creating the Amazon Elasticsearch Service domain
  3. Upload data to Amazon Elasticsearch for indexing
  4. Search documents using Kibana in Amazon Elasticsearch
  5. Conclusion

What Is Amazon Elasticsearch Service?

Amazon Elasticsearch Service is a managed service which deploys and scale the Elasticsearch clusters in the cloud. Elasticsearch is an open source analytical and search engine which is used to perform real time application monitoring and log analytics.

Amazon Elasticsearch service provisions all resources for Elasticsearch clusters and launches it. It also replaces the failed Elasticsearch nodes in the cluster automatically.

Features of Amazon Elasticsearch Service

  • It can scale up to 3 PB of attached storage
  • It works with various instance types.
  • It easily integrates with other services such as IAM for security for ,VPC , AWS S3 for loading data , AWS Cloud Watch for monitoring and AWS SNS for alerts notifications.

Creating the Amazon Elasticsearch Service domain

In this tutorial you will see how to create Elasticsearch cluster using Amazon Management console. Lets start.

  • Open your favorite web browser and navigate to the AWS Management Console and log in.
  • While in the Console, click on the search bar at the top, search for ‘Elasticsearch’, and click on the Elasticsearch menu item.
  • Now, one thing to note here is the name of Amazon Elasticsearch domain is same as that of Elasticsearch cluster that means domains are clusters with the settings, instance types, instance counts, and storage resources that you specify.
  • Now, click on Create a new domain.
  • Select the deployment type as development and testing domain
  • Now Under Configure domain provide the Elasticsearch domain name as “firstdomain” . A domain is the collection of resources needed to run Elasticsearch. The domain name will be part of your domain endpoint.
  • Under Data nodes, choose the t3.small.elasticsearch and ignore rest of the settings and click on NEXT
  • Under Network configuration, choose Public access. For Fine-grained access control, choose Create master user. Provide a user name as user and password as Admin@123. Fine-grained access control keeps your data safe.
  • For Domain access policy, choose Allow open access to the domain. Access policies control whether a request is accepted or rejected when it reaches the Amazon Elasticsearch Service domain
  • Now click on NEXT till end and create the domain. It takes few minutes for Domain to get Launched.
  • Click on the firstdomain Elasticsearch domain

Upload data to Amazon Elasticsearch for indexing

  • You can load streaming data into your Amazon Elasticsearch Service (Amazon ES) domain from many different sources. Some sources, like Amazon Kinesis Data Firehose and Amazon Cloud Watch Logs, have built-in support for Amazon ES. Others, like Amazon S3, Amazon Kinesis Data Streams, and Amazon DynamoDB, use AWS Lambda functions as event handlers
  • In this tutorial we will directly use a sample data to upload the data.
  • Click on the Kibana link as shown in above snapshot using the username user and password Admin@123 and then click on Add data
  • As this is just the demonstration , lets use sample data and Add e-commerce Orders.

Search documents using Kibana in Amazon Elasticsearch

Kibana is a popular open source visualization tool which works with AWS Elasticsearch service. It provides interface to monitor and search the indexes. Lets use Kibana to search the sample data which you just uploaded in AWS ES.

  • Click on Discover option from the main menu to search the data.
  • Now you will notice that Kibana will search the data and populate for you. You can modify the timelines and many other fields accordingly.

Kibana did provide the data when we searched in the dashboard using sample data which you uploaded.

Conclusion

In this tutorial you learnt about what is Amazon Elasticsearch and how to create a Amazon Elasticsearch domain using Amazon Management console. Also we learnt how to upload the sample data in AWS ES although this can be done using various ways such as S3 , Dynamo DB etc.

So, Now you have a strong fundamental understanding of AWS Elasticsearch , which site are you going to implement it in ?