Machine-generated data is growing exponentially, and getting insights is important for your business so that you can search unstructured or semi-structured data on your site. You need a search analytic solution with speed, scalability, flexibility, and real-time search, and this is possible with Latest Amazon OpenSearch Service ( successor to Amazon Elasticsearch Service.
In this tutorial, you will learn about Amazon OpenSearch Service, Amazon Elasticsearch, and how to create an Amazon Elasticsearch domain using Terraform.
Let’s get started.
Table of Content
- What Is Amazon Elasticsearch Service?
- Features of Amazon Elasticsearch Service
- What is Amazon OpenSearch Service?
- Prerequisites
- Terraform files and Terraform directory structure
- Building Terraform Configuration for AWS Elasticsearch
- Verify AWS Elasticsearch in Amazon Account
- Conclusion
What Is Amazon Elasticsearch Service?
Amazon Elasticsearch Service is a distributed search and analytics engine mainly used for log analytics, full-text search, business analytics, and operational intelligence. It performs real-time application monitoring and log analytics.
In the Amazon Elasticsearch service, you need to send the data in JSON format using the API or Logstash. Then Elasticsearch automatically stores the data and adds a searchable reference to the document in clusters index, and you can search using Elasticsearch API.
Amazon Elasticsearch service creates the AWS Elasticsearch clusters and nodes. If the nodes fail in the cluster, then the failed Elasticsearch nodes are automatically replaced.
Features of Amazon Elasticsearch Service
- Amazon Elasticsearch service can scale up to 3 PB of attached storage and works with various instance types.
- Amazon Elasticsearch easily integrates with other services such as IAM for security such as Amazon VPC , AWS S3 for loading data , AWS Cloud Watch for monitoring and AWS SNS for alerts notifications.
What is Amazon OpenSearch Service?
Amazon OpenSearch Service is a managed service that allows you to deploy, operate and scale OpenSearch clusters in Amazon Cloud. While you create the OpenSearch cluster, you can select the search engine of your choice.
Amazon OpenSearch is a fully open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis.
The latest version of OpenSearch is 1.1 and supports all elasticsearch versions, such as 7.10. 7.9 etc.
Prerequisites
- Windows Machine or Ubuntu Machine. This tutorial will use Windows Machine.
- Terraform Installed on Windows Machine or Ubuntu Machine. If you don’t have click the links Terraform on Windows Machine / Terraform on Ubuntu Machine
Terraform files and Terraform directory structure
Now that you know what is Amazon Elastic search and Amazon OpenSearch service are. Let’s now dive into Terraform files and Terraform directory structure that will help you write the Terraform configuration files later in this tutorial.
Terraform code, that is, Terraform configuration files, are written in a tree-like structure to ease the overall understanding of code with .tf format or .tf.json or .tfvars format. These configuration files are placed inside the Terraform modules.
Terraform modules are on the top level in the hierarchy where configuration files reside. Terraform modules can further call another child to terraform modules from local directories or anywhere in disk or Terraform Registry.
Terraform contains mainly five files as main.tf , vars.tf , providers.tf , output.tf and terraform.tfvars.
- main.tf – Terraform main.tf file contains the main code where you define which resources you need to build, update or manage.
- vars.tf – Terraform vars.tf file contains the input variables which are customizable and defined inside the main.tf configuration file.
- output.tf : The Terraform output.tf file is the file where you declare what output paraeters you wish to fetch after Terraform has been executed that is after terraform apply command.
- .terraform: This directory contains
cached provider, modules plugins and also contains the last known backend configuration. This is managed by terraform and created after you runterraform initcommand. - terraform.tfvars files contains the values which are required to be passed for variables that are refered in main.tf and actually decalred in vars.tf file.
- providers.tf – The povider.tf is the most important file whrere you define your terraform providers such as terraform aws provider, terraform azure provider etc to authenticate with the cloud provider.
Building Terraform Configuration for AWS Elasticsearch
Now that you know what are Terraform configurations files look like and how to declare each of them. In this section, you will learn how to build Terraform configuration files for AWS Elasticsearch before running Terraform commands. Let’s get into it.
- Log in to the Ubuntu machine using your favorite SSH client.
- Create a folder in opt directory named
terraform-Elasticsearchand switch to that folder.
mkdir /opt/terraform-Elasticsearch
cd /opt/terraform-Elasticsearch
- Create a file named main.tf inside the /opt/terraform-Elasticsearch directory and copy/paste the below content. The below file creates the below components:
- Creates domains are clusters with the settings, instance types, instance counts, and storage resources that you specify.
- Creates the AWS Elasticsearch domain policy.
# Creating the Elasticsearch domain
resource "aws_elasticsearch_domain" "es" {
domain_name = var.domain
elasticsearch_version = "7.10"
cluster_config {
instance_type = var.instance_type
}
snapshot_options {
automated_snapshot_start_hour = 23
}
vpc_options {
subnet_ids = ["subnet-0d8c53ffee6d4c59e"]
}
ebs_options {
ebs_enabled = var.ebs_volume_size > 0 ? true : false
volume_size = var.ebs_volume_size
volume_type = var.volume_type
}
tags = {
Domain = var.tag_domain
}
}
# Creating the AWS Elasticsearch domain policy
resource "aws_elasticsearch_domain_policy" "main" {
domain_name = aws_elasticsearch_domain.es.domain_name
access_policies = <<POLICIES
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "es:*",
"Principal": "*",
"Effect": "Allow",
"Resource": "${aws_elasticsearch_domain.es.arn}/*"
}
]
}
POLICIES
}
- Create one more file named vars.tf inside the /opt/terraform-Elasticsearch directory and copy/paste the below content. This file contains all the variables that are referred in the main.tf configuration file.
variable "domain" {
type = string
}
variable "instance_type" {
type = string
}
variable "tag_domain" {
type = string
}
variable "volume_type" {
type = string
}
variable "ebs_volume_size" {}
- Create one more file named outputs.tf inside the /opt/terraform-Elasticsearch directory and copy/paste the below content. This file contains all the outputs variables that will be used to display he output after running the terraform apply command.
output "arn" {
value = aws_elasticsearch_domain.es.arn
}
output "domain_id" {
value = aws_elasticsearch_domain.es.domain_id
}
output "domain_name" {
value = aws_elasticsearch_domain.es.domain_name
}
output "endpoint" {
value = aws_elasticsearch_domain.es.endpoint
}
output "kibana_endpoint" {
value = aws_elasticsearch_domain.es.kibana_endpoint
}
- Create another file and name it as provider.tf. This file allows Terraform to interact with AWS cloud using AWS API.
provider "aws" {
region = "us-east-2"
}
- Create one more file
terraform.tfvarsinside the same folder and copy/paste the below content. This file contains the values of the variables that you declared in vars.tf file and refered in main.tf file.
domain = "newdomain"
instance_type = "r4.large.elasticsearch"
tag_domain = "NewDomain"
volume_type = "gp2"
ebs_volume_size = 10
- Now your folder should have all files as shown below and should look like.
- Now your files and code are ready for execution. Initialize the terraform using the terraform init command.
terraform init
- Terraform initialized successfully , now its time to run the plan command which provides you the details of the deployment. Run terraform plan command to confirm if correct resources is going to provisioned or deleted.
terraform plan
- After verification, now its time to actually deploy the code using terraform apply command.
terraform apply
Verify AWS Elasticsearch in Amazon Account
Terraform commands terraform init→ terraform plan→ terraform apply all executed successfully. But it is important to manually verify the AWS Elasticsearch domain on the AWS Management console.
- Open your favorite web browser and navigate to the AWS Management Console and log in.
- While in the Console, click on the search bar at the top, search for ‘Elasticsearch’, and click on the Elasticsearch menu item.
- Now you will see that the newdomain that you specified in Terraform configuration file is created succesfully.
- Next, click on newdomain to check the details of the newly created domain.
In the new Amazon OpenSearch service, you should see something like below.
Conclusion
In this tutorial, you learned Amazon Elasticsearch and how to create an Amazon Elasticsearch domain using Terraform.
Now that you have a strong basic understanding of AWS Elasticsearch, which documents will you upload for indexing and searching?
DevOps Automateinfra Learning 