Prometheus is a powerful, open-source monitoring system that collects metrics from services and stores them in a time-series database. It records real-time metrics and alerts. It is written in Go Language. It allows powerful queries and great visualization. Prometheus works very well with Grafana when it comes to Dashboards and alerting notifications.
Prometheus includes a Flexible query language. Every time series contains a metrics name and set of key-value pairs called labels.
# Notation of time series
<metric name> {<label name>=<label value>,.....}
# Example
node_boot_time {instance="localhost:9000",job="node_exporter"}
A maven is a build tool used for Java projects. Maven can also be used to build and manage projects written in various languages such as C#, Ruby, Scala, and other languages. The Maven project is hosted by the Apache Software Foundation, where it was formerly part of the Jakarta Project.
The most powerful feature of Maven is download project dependency libraries automatically defined in the pom.xml. Also, configure project build cycle such as invoke junits for test sonarqube for static analysis.
Prerequisites
Java Development Kit (JDK)and Eclipse
Maven 3.3+ require JDK 1.7 or above to execute
Memory
No minimum requirement
Disk
Approximately 10MB is required for the Maven installation itself. In addition to that, additional disk space will be used for your local Maven repository. The size of your local repository will vary depending on usage but expect at least 500MB.
Operating System
No minimum requirement. Start-up scripts are included as shell scripts and Windows batch files.
Next unzip the file that you just downloaded on your machine.
Next, set the enviornmental variables for Apache Maven by going in enviormental variable and add variable name as M2_HOME and MAVEN_HOME
Next append the Maven bin directory in the PATH variable.
Next, check the Maven version
Next, configure the local maven repository by going in the conf folder and then inside the setting.xml file and update the path after creating a maven_repo folder as shown below.
Setup Maven Project
First step is to open eclipse and navigate to new project and then look for Maven as shown below.
Select Create a simple project
Provide the group ID, artifact ID as shwon below.
It will create the new folder on your eclipse as shown below.
Next, create a Java class and name it something as my-calculator
Create a new class mycalculator
After clicking on Finish you will as below.
Next, add the methods in the class.
package mycalculator_package;
public class mycalculator {
// Method to add two numbers
public int add(int a, int b) {
return a + b;
}
// Methods to multiple two numbers
public int multiple(int a, int b) {
return a * b;
}
// Methods to subtract two numbers
public int subtract(int a, int b) {
return a - b;
}
// Methods to divide two numbers
public int divide(int a, int b) {
return a % b;
}
}
Similarly create a junit by creating a test class
Next, we dont want Junit4 to be added.
Next add the Junit dependencies in the POM.xml as shown below from https://mvnrepository.com/artifact/junit/junit. This will install the dependencies of Junit and you can see it on the left side.
Next Run the Maven Project as shown below by navigating to Run as and then Maven Build. All your Java source code remains in the src folder and all the classes are pressent in the target folder.
Further run mvn clean and mvn clean test: Cleans the project and removes all files generated by the previous build and the mvn clean test cleans the target folder as well.
Next test locally on your machine some of the other commands such as mvn compile thatCompiles source code of the project.
mvn test-compile: Compiles the test source code.
mvn test: Runs tests for the project.
mvn package: Creates JAR or WAR file for the project to convert it into a distributable format.
mvn install: Deploys the packaged JAR/ WAR file to the local repository.
mvn deploy: Copies the packaged JAR/ WAR file to the remote repository after compiling, running tests and building the project.
Setting up Maven in Jenkins WAY 1
Create a new Job in Jenkins and call it Maven-JOB
Select top level Maven targets in the Build Step
Next add clean test package in the Goals and the location of pom.xml file.
Next, trigger the Jenkins Job and should see the project compiled succesfully.
Setting up Maven in Jenkins WAY 2
Install Maven Plugin using Manage Jenkins
Now create another job where you will notice the Maven Project option as shown below. Select Maven Project and enter an item name as maven-job2
Inside the Build TAB add the workspace path which is the location where you pom file is located.
Make sure to add: Resolve Dependencies during Pom parsing in the Build step
Next navigate to Manage Jenkins to Global Tool Configuration and add Maven and JDK as shown below.
Next run the Jenkins Job and you should then be able to run the Job.
yamland yml files are superset of JSON. Some of the automation tools such as ansible uses yaml based files, referred to as playbooks, to define actions you want to automate. These playbooks use the YAML format.
Working with yaml files is a fun in python , so lets get started but In order to work with yaml files in python you would require to install a PyYAML libraryas Python doesn’t contain standard library. PyYAML is a YAML parser and emitter for Python.
Run the following command to install PyYAML library in your favorite code editor terminal such as visual code studio.
pip install PyYAML
Next, create a folder with a name Python and under that create a simple YML file and name it as apache.yml and paste the below content and save it.
---
- hosts: webservers
vars:
http_port: 80
max_clients: 200
remote_user: root
tasks:
- name: ensure apache is at the latest version
yum:
name: httpd
state: latest
Next, create another file in same Pythonfolder and name it as read_write_yaml.py and paste the below python code.
Below Python script imports yaml module to work with yaml files and pprint module to get a output in well designed pattern. Next, using open() function it opens the apache.yml file and reads the data using yaml.safe_load() method. Later, using yaml.dump() you can add or write the data into it. As we are not adding any data into it , the output would result as NONE
import yaml
from pprint import pprint
with open('apache.yml', 'r') as new_file:
verify_apache = yaml.safe_load(new_file)
pprint(verify_apache)
with open('apache.yml', 'w') as new_file2:
verify_apache2 = yaml.dump(verify_apache, new_file2)
pprint(verify_apache2)
Execute the above python script using python command and you should see the below output.
[{'hosts': 'webservers',
'remote_user': 'root',
'tasks': [{'name': 'ensure apache is at the latest version',
'yum': {'name': 'httpd', 'state': 'latest'}}],
'vars': {'http_port': 80, 'max_clients': 200}}]
None
Reading and Writing a XMLfile using python
XML files used mostly for structured data. Many web system uses XML to transfer data and one of them is RSS ( Real Simple Syndication) feeds which helps in finding the latest updates on websites from various sources. Python offers XML Library.
Next, in the same Pythonfolder create a simple XML file and name it as book.xml and paste the below content and save it. XML has a tree like structure and top element is known as root and rest of them are elements
<?xml version="1.0"?>
<catalog>
<book id="bk109">
<author>Author1</author>
<title>Automate Infra Part 2</title>
<genre>Science Fiction</genre>
<price>6.95</price>
<publish_date>2000-11-02</publish_date>
<description>book1</description>
</book>
<book id="bk112">
<author>Author2</author>
<title>Automate Infra Part 1</title>
<genre>Computer</genre>
<price>49.95</price>
<publish_date>2001-04-16</publish_date>
<description>book2</description>
</book>
</catalog>
Next, create another file in same python folder and name it as read_write_xml.py and paste the below python code.
In below script, importing xml.etree.ElementTreemodule helps to work with xml files and implements a simple and efficient API for parsing and creating XML data. Next, In XML file entire tree is parsed that is it reads the book.xml file and then prints the content inside it.
import xml.etree.ElementTree as ET
tree = ET.parse('book.xml') # checking each elements
root = tree.getroot() # finding the root
print(root.tag,root.attrib)
for child in root: # Each child and its attributes
print(child.tag,child.attrib)
Execute the above python script using python command and you should see the below output.
O/P:
catalog {}
book {'id': 'bk109'}
book {'id': 'bk112'}
Reading and Writing a comma-separated values (CSV)file using python
CSV is most widely used spreadsheets. To work with these file in python you need to import the csv module. Lets learn how to read and write data into CSV.
Next, in the same Pythonfolder create a CSV file and name it as devops.csv and add the content similar to below in your file and save it.
Next, create another file in same python folder and name it as read_write_csv.py and paste the below python code.
Below script uses csv module to work with csv files. As soon as python script is executed , open() function opens the csv file and then using csv.reader() it reads it and then prints the rows according to the defined range.
import csv
with open('devops.csv' , 'r') as csv_file:
read = csv.reader(csv_file, delimiter=',')
for _ in range(5):
print(next(read))
print(read)
Execute the above python script using python command and you should see the below output.
pandas.DataFrame, which acts like a data table, similar to a very powerful spreadsheet. If you want to work on something like row or column in Spreadsheet then DataFrames is the tool for you.So lets get started by installing pip install pandas
PYTHON: DEAL With Large Files(FILE BREAKER and LINE BREAKER)
Rather than loading the whole file into memory as you have done up until now, you can read one line at a time, process the line, and then move to the next. The lines are removed from memory automatically by Python’s garbage collector, freeing up memory.
# LINE BREAKER
with open("devops.txt",mode="r") as mynewfile: # if you open any binary file such as pdf keep w as wb
with open("devops-corrected.txt", "w") as target_file:
for line in mynewfile:
print(target_file.write(line))
o/p:
Automateinfra.com
automateinfra.com/blog
automateinfra.com/solutions
# FILE BREAKER with chunk of data with number of bytes
with open('book.xml' , 'rb') as sourcefile:
while True:
chunk = sourcefile.read(1024) # break down in 1024 bytes
if chunk:
print(chunk)
else:
break
O/P:
b'<?xml version="1.0"?>\r\n<catalog>\r\n <book id="bk109">\r\n <author>Author1</author>\r\n <title>Automate Infra Part 2</title>\r\n <genre>Science Fiction</genre>\r\n <price>6.95</price>\r\n
<publish_date>2000-11-02</publish_date>\r\n <description>book1</description>\r\n </book>\r\n <book id="bk112">\r\n <author>Author2</author>\r\n <title>Automate Infra Part 1</title>\r\n <genre>Computer</genre>\r\n <price>49.95</price>\r\n <publish_date>2001-04-16</publish_date>\r\n <description>book2</description>\r\n </book>\r\n</catalog>'
PYTHON ENCRYPTION: MOST IMPORTANT TOPIC OF PYTHON FILE SYSTEM
There are many times you need to encrypt text to ensure security. In addition to Python’s built-in package hashlib, there is a widely used third-party package called cryptography
HASHLIB: Uses Hash Function and based on SHA1, SHA224, SHA384, SHA512, and RSA’s MD5 Algorithms
CRYPTOGRAPHY:
symmetric key encryption: Its based on shared keys. These algorithms include Advanced Encryption Algorithm (AES), Blowfish, Data Encryption Standard (DES), Serpent, and Twofish
asymmetric key encryption: Its based on public keys ( which are widely shared ) and private keys which is kept secretly
# Encryption using HashLib
import hashlib # Python Built in Package
line = "I like editing automateinfra.com"
bline = line.encode() # Converting into Binary string
print(bline) # Print the converted Binary string
algo = hashlib.md5() # Using the secure alogorithm using haslib object
algo.update(bline) # Applying the secure alogorithm
print("Encrypted text Message")
print(algo.digest()) # Print the Encypted string
# Encryption using Cryptography (Symmetric key encryption)
from cryptography.fernet import Fernet # Third Party Package So you would need pip install cryptography
key = Fernet.generate_key() # Generating the keys
print("Generating the keys ")
print(key) # Prining the keys
algo = Fernet(key) # Using the key AES alogo using Fenet object
message = b"I definetely like Editing AutomateInfra.com"
encrypted = algo.encrypt(message)
print("Encrypted text Message ")
print(encrypted)
print(algo.decrypt(encrypted))
# Encryption using Cryptography (ASymmetric key encryption)
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import padding ,rsa
from cryptography.hazmat.primitives import hashes
private_key = rsa.generate_private_key(public_exponent=65537,key_size=4096,backend=default_backend()) # Generating the Private Key
print(private_key) # Printing the Private Key
public_key = private_key.public_key() # Generating the Public Key
print(public_key) # Printing the Public Key
message = b"I am equally liking Editing AutomateInfra.com"
encrypted = public_key.encrypt(message,padding.OAEP(mgf=padding.MGF1(algorithm=hashes.SHA256()), algorithm=hashes.SHA256() , label=None))
print(encrypted)
decrypted = private_key.decrypt(encrypted,padding.OAEP(mgf=padding.MGF1(algorithm=hashes.SHA256()), algorithm=hashes.SHA256(), label=None))
print(decrypted)
O/P:
b'I like editing automateinfra.com'
Encrypted text Message
b'v\x84*\xe55\x01\xa4z\x05\xa2\xa2\xdb\xd1y\xa9\x07'
Generating the keys
b'7trCiXpGuCfEnXoIcsFfCGOw-u_Qkas0tv1lBM8xmQo='
Encrypted text Message
b'gAAAAABgAJcPFj-aGttg8MRJQfRYGWyOWy44u-cLWGuDhqoyyvYP1uG4oQYms8BQMr4eExpv74LIZESGvpIUY88fE0_YQCQ32JH0DZsabLOAtc00QCwV8L51WktRjzUab0Fp3jnbOeb2'
b'I definetely like Editing AutomateInfra.com'
<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x036491D8>
<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x03850E38>
b"\x8b\xec\xb0\x91\xec\xe7\x8d;\x11\xbclch\xbdVD@c\xd3J\x07'\xe9\x07\x15\x1c@=^\xd2h\xcaDL\x95\xea[\x0fv\x012\xed\xd5\xed\x0e\x9b\x93V2\x00\xba\x9c\x07\xba\x8b\xf3\xcb\x03M\xa8\xb1\x12ro\xae\xc0\xfb$\xf9\xcc\x85\xe8s\xfc`{\xfe{\x88\xd2\xc3\xffI\x90\xe3\xd2\x1e\x82\x95\xdfe<\xd5\r\x0b\xc4z\xc4\xf7\x00\xcfr\x07npm0\xd4\xc4\xa4>w\x9d]\xcf\xae7F\x91&\x93\xd5\xda\xcaR\x13A\x8ewB\xf6\xd9\xae\xce\xca\x8f\xd6\x91\x06&:\x00\xa0\x84\x05#,\x7fdA\x87\xb2\xe7\x1d\x8b*\xa15\xf8\xb0\x07\xa0n\x1e\xeaI\x02\xbaA\x88ut\x8e\x82<\xfe\xbfM\xe6F\xa3\xcc\xd4\x8b\x80PY\xb5\xd3\x14}C\xe2\x83j\xaf\x85\xa6\x9e\x19\xb2\xd9\xb8\xac\xa4\xfb\x1f\x0c\xce\x9d4\x82\x1e\xfd5\xb49\xa5\xbbL\x01~\x8fA\xee\r\xc7\x84\x9e\x0c\t\x15z\r\xfd]\x0b\xcfW\x01\xd2\x16\x17btc\xeaSl\xf5\xb0\x8a\xe2X\xe7\xa7a\xa7\xf7M\x01\xa2\x0b8\xd6\xf2\xc5c\xbf\xea\xe0\x80\x15\xde-\x98\xa1\xc8ud*\xbel2\xb5\xc8:\x92\xd5\r(_8\xbd\xcb\x80\xf1\x93\x83\xe2\x9f\xed\x82f\xd0\xb2\x8f\x1b\x9eMC\x07\xf9\x08\xb0\x00QA\xea\x93\xc7@&\x84\xff<\xde\x80@\xc8\xc6\x83O&%\x91r-\xb0\xef}\x18tU{C\xa6\x17\x97\x1b\x95g\xc5\x0e>{\xb0\x94a)\xbc)*Sq\x98\xad\xf3>\x04\x9b+x\x95&\xa6\xe6,\xb4~\xf2Y\x06,\xab'uq \x9f0\x7f\xb5\xd50\xbdp\xbb\xdf\x1c\xe9\xb1\xc4\x88y\nq\\\x85\x1e\xd8\x18M\x87\x1aU.\x918;\xcd\x10 \x9b\x11\xf9R\xd3\x8fz\xe8\xf6|C\xfb\x1f\xfd1\x19\x10:>\x1c\x06\x8e\xda\x98\xb2\xf3aa^\xa54\x03\xf8\x03\xc4\xe6\xd9mw\r\x8b\x96\xa2rJ\x03\xe7\xda\x0f\rJ-iPo!^\x8a\xdcg\x8c!L\xa4\xedY\xe5\x12\xdf\xe8\xe7\x0cE\xcd\xa2\xa2Gr\xc0\xe1\xa6\xc5\x9a\x9f\x07\x89\x84\x8b\xb7"
b'I am equally liking Editing AutomateInfra.com'
PYTHON OS MODULE:
This module will help to connect with many low level operating system calls and offers connectivity between multi -OS like Unix and Windows.
import os # Python Built in Package
print(os.listdir('.')) # List the directories
os.rename('automateinfra.txt','automateinfra_backup.txt')
os.chmod('automateinfra.txt',777) # Add the permissions to the file.
os.mkdir('/tmp/automateinfra.pdf') # Make the directory
os.rmdir('/tmp/automateinfra.pdf') # remove the directory
os.stat('b.txt') # These stats include st_mode, the file type and permissions, and st_atime, the time the item was last accessed.
cur_dir = os.getcwd() # Get the current working directory.
print(os.path.dirname(cur_dir)) # Returns the Parent Directory Path
print(os.path.split(cur_dir)) # Gives structure from Parent Directory
print(os.path.basename(cur_dir)) # Returns Base Directory
while os.path.basename(cur_dir): # Until Base Path directory is true , keep continuing
cur_dir = os.path.dirname(cur_dir) # Prints the base directory and all above parents Directory
print(cur_dir)
O/P:
C:\Users\AutomateInfra\Desktop\GIT\Python-Desktop
('C:\\Users\\AutomateInfra\\Desktop\\GIT\\Python-Desktop', 'Basics')
Basics
C:\Users\AutomateInfra\Desktop\GIT\Python-Desktop
C:\Users\AutomateInfra\Desktop\GIT
C:\Users\AutomateInfra\Desktop
C:\Users\AutomateInfra
C:\Users
C:\
import os
# Check the current working directory
file_name = "automateinfra.txt"
file_path = os.path.join(os.getcwd(), file_name)
print(f"Checking {file_path}")
if os.path.exists(file_path):
print(file_path)
# Check user home directory
home_dir = os.path.expanduser("~/") #expanduser function to get the path to the user’s home directory.
file_path = os.path.join(home_dir,file_name)
print(f"Checking {file_path}")
if os.path.exists(file_path):
print(file_path)
o/p:
Checking C:\Users\Automateinfra\Desktop\GIT\Python-Desktop\Basics\automateinfra.txt
C:\Users\Automateinfra\Desktop\GIT\Python-Desktop\Basics\automateinfra.txt
Checking C:\Users\Automateinfra/automateinfra.txt
API is an interface that allows communication between client to server to simplify the building of client-server software.
API is an software that allows two applications to talk to each other. Each time you use an app like Facebook, send an instant message, or check the weather on your phone, you’re using an API.
When you use an application on your mobile phone, the application connects to the Internet and sends data to a server. The server then retrieves that data, interprets it, performs the necessary actions and sends it back to your phone. The application then interprets that data and presents you with the information you wanted in a readable way. This is all possible with an API.
Difference between Types of API’s [ SOAP v/s REST ]
REST: Representational State Transfer. It is an lightweight and scalable service built on REST architecture. It uses HTTP protocol. It is based on architectural pattern
Elements of REST API:
Method: GET, PUT, DELETE
POST – This would be used to send the data to the server such as customer information or uploading any file using the RESTful web service. To send the data use Form parameter and body payload.
GET – This would be used to retrieve data from the server using the RESTful web service. It only extracts the data there is no change in the data. No Payload or body required. To get the data use query parameter.
PUT – This would be used to update the resources using the RESTful web service
DELETE – This would be used to delete * using the RESTful services
Request Headers: These are additional instructions that are sent along with the request
Request Body: Data is sent along with the POST request that is it wants to add a resource to the server.
Response status code: Returned along with the request such as 500, 200 etc.
Characteristics of REST
REST is an Architectural style in which a web service can only be treated as a RESTful service if it follows the constraints of being 1. Client Server 2. Stateless 3. Cacheable 4. Layered System 5. Uniform Interface
Stateless means that the state of the application is not maintained in REST .For example, if you delete a resource from a server using the DELETE command, you cannot expect that delete information to be passed to the next request. This is required so that server can process the response appropriately
The Cache concept is to help with the problem of stateless which was described in the last point. Since each server client request is independent in nature, sometimes the client might ask the server for the same request again
REST use Uniform Service locators to access to the components on the hardware device. For example, if there is an object which represents the data of an employee hosted on a URL as automateinfra.com , the below are some of URI that can exist to access them automateinfra.com/blog
SOAP: Simple Object Access Protocol.
Follows strict rules for communicate between [client-server] as it doesn’t follows what is being followed by REST follows Uniform Interface, Client-Server, Stateless, Cacheable, Layered System, Code.
SOAP was designed with a specification. It includes a WSDL file which has the required information on what the web service does in addition to the location of the web service.
The other key challenge is the size of the SOAP messages which get transferred from the client to the server. Because of the large messages, using SOAP in places where bandwidth is a constraint can be a big issue.
SOAP uses service interfaces to expose its functionality to client applications. In SOAP, the WSDL file provides the client with the necessary information which can be used to understand what services the web service can offer.
SOAP uses only XML to transfer the information or exchanging the information where as REST uses plain text, HTML , JSON and XML and more.
Application Programming Interface theory (API-theory)
When Website is owned by single owner such as Google: In that case when frontend site needs to connect to backend site then it may vary with different languages and can cause lot of compatibility issues such as frontend uses Angular and backend uses Java , so you would need API to deal with it.
When Your client needs to access data from your website then you would need to expose the API rather than exploring your entire code and packages.
When client connects to another client or server using API the transmission of data takes places using either XML or JSON which are language independent.
Jenkins is an open source automated CI/CD tool where CI stands for continuous integration and CD stands for Continuous delivery. Jenkins has its own built-in Java servlet container server which is Jetty. Jenkins can also be run in different servlet containers such as Apache tomcat or glassfish.
Jenkins is used to perform smooth and quick deployment. It can be deployed to local machine or on premises data center or any cloud.
Jenkins takes your code any sort of code such as python, java or go or JS etc. and compiles it using different compiler such as MAVEN one of the most used compiler and then builds your code in war or Zip format and sometimes as a docker Image. Finally once everything is built properly it deploy as an when required . It integrates very well with lots of third party tools.
JAVA_HOME and PATH are variables to enable your operating system to find required Java programs and utilities.
JAVA_HOME: JAVA_HOME is an (OS) environment variable that can optionally be set after either the (JDK) or (JRE) is installed. The JAVA_HOME environment variable points to the file system location where the JDK or JRE was installed. This variable should be configured on all OS’s that have a Java installation, including Windows, Ubuntu, Linux, Mac, and Android.
The JAVA_HOME environment variable is not actually used by the locally installed Java runtime. Instead, other programs installed on a desktop computer that requires a Java runtime will query the OS for the JAVA_HOME variable to find out where the runtime is installed. After the location of the JDK or JRE installation is found, those programs can initiate Java-based processes, start Java virtual machines and use command-line utilities such as the Java archive utility or the Java compiler, both of which are packaged inside the Java installation’s \bin directory.
JAVA_HOME if you installed the JDK (Java Development Kit) or
JRE_HOME if you installed the JRE (Java Runtime Environment)
PATH: Set the PATH environment variable if you want to be able to conveniently run the executables (javac.exe, java.exe, javadoc.exe, and so on) from any directory without having to type the full path of the command. If you do not set the PATH variable, you need to specify the full path to the executable every time you run it, such as:
C:\Java\jdk1.8.0\bin\javac Myprogram.java
# The following is an example of a PATH environment variable:
C:\Java\jdk1.7.0\bin;C:\Windows\System32\;C:\Windows\;C:\Windows\System32\Wbem
Installing Jenkins using msi installer on Windows Machine
MSI is an installer file that installs your program on the executing system. Setup.exe is an application (executable file) that has MSI file(s) as one of the resources. The MSI is the file extension of MSI files. They are Windows installers. An MSI file is a compressed package of installer files. It consists of all the information pertaining to adding, modifying, storing, or removing the respective software. MSI file includes data, instructions, processes, and add-ons that are necessary for the application to work normally.
EXE is short for Executable. This is any kind of binary file that can be executed. All windows programs are exe files. Prior to MSI files, all installers were EXE files. The exe is a file extension of an executable file. An executable file executes a set of instructions or a code when opening it. An executable file is compiled from source code to binary code. It can be directly executed by the Windows OS. These files are understandable by the machine, and they can be directly executed by the operating system
MSI is a file extension of windows installer which is a software component of Microsoft Windows used for the installation, maintenance, and removal of software. Whereas, exe is a file extension of an executable file that performs indicated tasks according to the encoded instructions.
Select the Port 8080 and click on Test Port and then Hit Next.
Provide the admin password from the provided Path mentioned in RED color.
Further install the plugins required for jenkins.
Next,it will prompt for First admin user. Please fill the required information and keep it safe with you , as you will use this to login.
Now Jenkins URL configuration screen will appear , keep it as it is for now.
Click on Save and Finish.
Now your Jenkins is ready , click on Start using Jenkins. Soon, you will see Jenkins Dashboard. You can create New Jobs by clicking on New Item.
Installing Jenkins using jenkins exe on Windows Machine
Similarly now install jenkins.war from jenkins URL and click on Generic Java package(.war).
Next run the command as below.
java -jar jenkins.war -http=8181
Next, copy the Jenkins password from the log output and paste it in the as you did earlier in windows msi section point (5) and follow rest of the points.
Installing jenkins on Apache Tomcat server on Windows Machine
Install the Apache Tomcat on windows machine from https://tomcat.apache.org/download-90.cgi and click on tomcat installer as per your system. This tutorial is performed on 64 bit windows machine.
Next, unzip the tomcat installation folder and copy the jenkin.war file in the webapps folder.
Next, go inside the bin folder and run the tomcat by clicking on the startup batch script.
Finally you will notice that Apache Tomcat has started and Jenkins as well.
Now, navigate to localhost:8080 URL and you should see tomcat page as shown below.
Further, navigate to localhost:8080/jenkins to redirect to Jenkins Page.
Configuring the Jenkins UI
First click on Manage Jenkins and then navigate to Configure system.
Next, add the system message and save it which should display this message on Jenkins everytime as below.
To configure the name of the Jobs add the name Pattern as below.
Next, try creating a a new Jenkins Job with random name then it will not allow you and display the error message.
Managing User’s and Permission’s in Jenkins UI
Go to Manage Jenkins and Navigate to Manage users in the Jenkins UI.
Then Create three users as shown below admin, dev, qa.
Next, Navigate to Manage Jenkins and choose Configure Global Security.
Next select Project-based Matrix Authorization Strategy and define the permissions for all users as you want.
Role Based Stratergy
In Previous section you noticed that adding all users and grnating all permissions is little tough job. So, instead create a role and add users in it. To do that first step is to install the Plugin as shown below.
Next select Role based Stratergy as shown below and define the permissions for all users as you want.
Next, navigate to Manage Jenkins and then to Manage and Assign Jenkins and then click on Manage Roles.
Add 3 Global Roles named DEV Team, QA Team and admin.
Add 2 Items Roles developers and Testers with define patterns so that Jobs names are declared accordingly.
Next, Click on Assign Role
Assigning the roles as shown below.
Conclusion
In this tutorial you learnt how to install jenkins on windows through various ways , how to configure Jenkins Dashboard UI and how to manager users and Permissions.
Knowing the hardware of the computer is very important as an IT engineer and this tutorial gives you each and every information about the same. In this tutorial learn everything about the Hardware and Software/components of the computer.
What is Computer System?
The computer contains mainly two parts: Hardware and software.The computer is a programmable electronic device that can be programmed to accept the input and then provide the output. The Computer hardware can only understand binary numbers that are 0 or 1. Computer transfer data one byte at a time.
Computer stores all data on the hard disk as 0 or 1 Binary numbers. ASCII ( American Standard Code for information interchange.
What are Computer hardware’s?
Hardware is made up of various electronic circuits and components such as I/O devices, CPU, disk, and the motherboard.
Input Devices: The input device is used to provide the input ( data, instructions) into the RAM of the computer such as
Keyboard
Trackball ( the upper part of the mouse)
light pen
OBR (Optical Bar Code reader) – This is used to scan the vertical bars, read tags.
OCR ( Optical Character reader) – This is used to detect the alphanumeric characters like reading passanger tickets, computer printed bills of credit card.
Output Devices: The output devices provides the end result which user provided as input such as monitor.
Monitor also known as VDU ( Visual display unit). It contains CRT ( Cathode ray Tube) which displays the character as output.
Ther are many different types monitors available in market such as CGA ( Color graphics adapter) , EGA (Enhanced graphic adapter) ,VGA ( Video graphics adapter), SVGA (Super VGA) which is best in market.
CPU( Central Processing Unit): CPU is the most important hardware part of the Computer which performs all the functions and execution of input data. It executes the instructions stored in the main memory. CPU has a set of electronic circuits that executes the program instructions. CPU contains its own memory that is Cache to immediately process the data,
Memory or storage: This a a storage place where all the data resides. Again there are two categories in memory i.e Primary and the secondary memory.
Primary memory: These memory are directly connected with CPU and are extremely fast such as RAM (Random Access Memories) which is volatile in nature and ROM (Read Only Memories) that is non volatile in nature. The CPU works with these memory only.
Secondary Memory: These memory are not directly connected to the CPU such as Floppy disk, CD Rom ,hard disks or RAM.
Motherboard: The motherboard is most important hardware component like a circuit board. It is the main printed circuit board ( PCB ) found in computers. CPU is installed in one of the sockets of the motherboard or directly soldered. There are slots in which memory is installed.
Buses: The data is stored in 0 or 1 binary format in register in a form of a unit. When the data needs to travel or move from one registers to other then you need a seperate wires and these wires are known as buses.
The data bus is used to move data,
Address bus to move address or memory location
Control bus to send control signals between various components of a computer.
Types of Buses
System bus transfers information between different parts inside computer system.
Control bus has two wires, set and enable. When CPU wants to read from RAM, the enable wire will be opened; when CPU wants to save information on RAM, CPU will enable the set wire.
Data bus is a two-way bus carry data commute between CPU and RAM.
Address bus is a one-way bus carry addresses from CPU to RAM.
Clock: Clock is an important component of CPU which measures and allocates fixed slot for processing each and every micro operations.
The clock speed measures the number of cycles your CPU executes per second, measured in GHz (gigahertz).
A CPU with a clock speed of 3.2 GHz executes 3.2 billion cycles per second
The CPU is allocated one or more clock cycle to complete the micro operations.
The processor base frequency refers to the CPU’s regular operating point, while the Max Turbo Frequency refers to the maximum speed the processor
The CPU executes the instructions in synchronization with the clock pulse.
The operations are performed with a speed of clock cycle per second (MHz) with a range of 4.77 MHz to 266 Mhz.
The speed of CPU is measured in terms of MIPS( Millions of instructions per second) or cycles per second
Each central processing unit has an internal clock that produces pulses at a fixed rate to synchronize all computer operations
Chipsets: Chipset handles an incredible amount of data. It is the glue that connects the microprocessor with the motherboard. It contains two basic parts northbridge (connects directly to processor via FSB i.e front side bus) and the South Bridge primarily handles the routing of traffic between the various input/output (I/O) devices on the system for which speed is not vital to the total performance, such as the disk drives (including RAID drive arrays), optical drives.
Video (Graphics) Card:
A dedicated video card (or video adapter) is an expansion card installed inside your system unit to translate binary data received from the CPU or GPU into the images you view on your monitor. It is an alternative to the integrated graphics chip. Modern video cards include ports allowing you to connect to different video equipment; also they contain their own RAM, called video memory. Video cards also come with their own processors or GPUs
Sound Cards
Sound cards attached to the motherboard and enabled your computer to record and reproduce sounds.
Most computers ship with a basic sound card, most often a 3D sound card. 3D sound is better than stereo sound
Ethernet Card/Network Cards
An Ethernet network requires that you install or attach network adapters to each computer or peripheral you want to connect to the network. Most computers come with Ethernet adapters preinstalled as network interface cards (NICs).
CPU (Central Processing Unit)
CPU is the most important hardware part of the Computer which performs all the functions and execution of input data. It executes the instructions stored in the main memory. CPU has a set of electronic circuits that executes the program instructions.
An example of a CPU is Intel 8085 which was an 8-bit microprocessor.
Computer interacts with primary storage that is the main memory for processing data and instructions. CPU contains mainly two components Arthematic Logic Unit and Control Unit.
Arthematic Logic Unit (ALU) is a digital circuit that performs all the calculations such as bitwise and mathematical operations on binary number.
Control Unit: CU controls all the activities such as transfer of data, instructions. It takes or obtains the instructions from the memory and understands it and then forward it further for execution or calculations. The control unit sends the control signals along the control bus.
Registers: These are high speed memory built into CPU chip circuits to acess or store the data immediate from the calculations or instructions performed by ALU. They act as high speed temporary memory. Registers can store two words at a time until overwritten. CPU needs to process very fast so in order for CPU to process the instructions or data from the RAM you need a place high memories in between which is Registers.
Registers work under the direction of the control unit to accept, hold, and transfer instructions or data and perform arithmetic or logical comparisons at high speed.
Types of Registers
Program Counter: Stores the address of next instruction to be executed
Accumulator: This registers temporarily stores data from ALU immediately.
Memory Address Registers: Stores the address of current instruction being executed
Memory data Registers or Memory Buffer Registers: Holds the data from that is copied from the RAM and ready for CPU to process
Below is the Image snapshot of various registers that are used in the CPU.
Cache (L2 or L3). A processor uses memory installed in the chip itself to store and speed up operations before utilizing external system RAM. This on-board memory is stored in one or more caches, which are identified L2 or L3. More powerful processors will be equipped with larger caches.
Socket Unit: On which CPU is installed on the motherboard.
Computer Architecture and its Working
The Working a Computer system comprises input operations, storage operations, data processing, and output operations.
When you press a key on your keyboard lets say ABC. The keyboard has PCB behind it which converts the alphabets ABC into the binary number and sends it to the CPU.
The other scenerio could be execution of a single program like 35 + 49
CPU component that is Control unit fetches (gets) the instruction from the RAM memory how to draw ABC ( basically like opcode, operand) using the data bus and also in the mean time asks RAM to store this in your memory until I perform execution of calculation. ( At times CPU fetches from Hard disk instead of RAM as your OS lies in the Hard disk)
Data bus brings the data and instructions in CPU’s internal memory that is registers for processing the data.
The control unit decodes the instruction (decides what it means) to machine bianry code and directs that the necessary data be moved from memory to the arithmetic/logic unit. The steps (2,3,4)together are called instruction time, or I-time.
The arithmetic/logic unit executes the arithmetic or logical instruction. That is, the ALU is given control and performs the actual operation on the data.
Thc arithmetic/logic unit stores the result of this operation in memory or in a register. ( Step 5 and 6 are Execution or E time)
The control unit eventually directs memory to release the result to an output device or a secondary storage device. The combination of I-time and E-time is called the machine cycle. To Perform all the instructions it is known as clock cycles that is executes 5 instructions and nowadays Modern CPU’s can perform millions of clock cycle per second.
All these things are happening on the circuit known as motherboard.
Step by Step function of CPU along with Memory
Another example of how a computer works
Suppose your Hard disk has 500 processes.
Suppose RAM can execute a maximum of 50 process.
Lets say: You ran a program which is an executable code (low level code that is machine code) to run 500 process that are stored in the hard disk.
Then as CPU will request RAM to provide the 50 process instructions to execute. If in case it doesnt have then it asks Hard disk to provide the instructions.
Harddisk copy the instructions to RAM and then CPU fetches from the RAM.
For Hard disk to copy 50 Process to RAM and then CPU to fetch from RAM is decided by Operating system using different algorithms such as short term scheduler or long term scheduler.
Data flow from CPU to Memory and Vice Versa
Step by Step function of CPU
The MAR stands for Memory address register which is connected to the Address Bus. It stores the memory address of an instruction. The sole function of MAR is to contain the RAM address of the instruction the CPU wants.
The MDR stands for Memory data register which is connected to the Data Bus. It holds the data that will be written to the RAM or read from the RAM. Even when ALU performs the operations the data is stored in high memory registers such as MBR or MDR
The relationship between MAR and MDR is that the MAR gives the address the data of the MDR will be read from or written to.
.
Single Core CPU v/s Multi Core CPU
Single-core CPU will only be able to process one program at a time. However, when you run multiple programs simultaneously, then a single-core processor will divide all programs into small pieces and concurrently execute with time slicing.
For EX:
P1 initiated——————————————————– P1 Ends
P2 initiated ——————————— P2 Ends
P3 Initaited —————- P3 Ends
Unlike single-core processing, it is a way in which computing tasks are divided into sub-parts, and a multicore processor (multiple CPU cores) execute each sub-task simultaneously.A dual-core CPU literally has two central processing units on the CPU chip. A quad-core CPU has four central processing units, an octa-core CPU has eight central processing units, and so on.
P1 initiated—————— P1 Ends
P2 initiated —————– P2 Ends
P3 Initaited —————– P3 Ends
Hyper Threading or Logical Processor or Threads of CPU
Threads are the virtual components or codes, which divide the physical core of a CPU into virtual multiple cores. A single CPU core can have up to 2 threads per core. For dual-core (i.e., 2 cores) it will have 4 threads, for Octal core (i.e., 8 core) it will have 16 threads, and vice-versa.
Windows’ Task Manager shows this fairly well. Here, for example, you can see that this system has one actual CPU (socket) and four cores. Hyperthreading makes each core look like two CPUs to the operating system, so it shows 8 logical processors.
Threads of Processes
The thread is created by a process. Every time you open an application, it itself creates a thread that will handle all the tasks of that specific application. Like-wise the more application you open more threads will be created.
The threads are always created by the operating system for performing a task of a specific application.
Batch Processing vs Multi programming vs vs Multiprocessing vs Multitasking vs Multithreading Operating Systems
Batch processing is the grouping of several same processing jobs to be executed one after another by a computer without any user interaction.
Multiprogramming is the ability of an OS to execute multiple programs at the same time on a single processor machine.
Multiprocessing system: When one system is connected to more than one processor which collectively works for the completion of the task.
Multithreading is a conceptual programming paradigm where a process is divided into a number of sub-processes called threads. Each thread is independent and has its own path of execution with enabled inter-thread communication.
Magnetic Storage Device
There are devices that are known to be magnetic storage devices as they have a layer of magnetic substance on their surface. These devices have a read-write assembly that converts data and instructions in the form of 0 or 1 into some form of the magnetic signal.
Floppy: The floppy disk stores the data in the form of a magnetic signal and while data is stored they are converted from 0,1 to magnetic signals. These were introduced by IBM and later were known as diskettes. They have an option called a small sliding switch called write-protect notch from which you cannot delete the data from the floppy drive.
Hard Disk: Can store a huge amount of data and have hard platters that hold magnetic medium compared to floppy disks compared to floppy and tapes that have plastic films. Information remains intact even after switching off the computer. So in computers, Operating System is installed and stored on the hard disk. As the hard disk is non volatile memory, OS does not lose on the turn off
Magnetic tapes: They are similar to tapes that you see in cassettes or video cassettes and are divided into tracks. One of the tracks is used to detect the error. They can store as much as 10GB of data. It allows only sequential order which is a disadvantage.
Zip disks: It is similar to floppy disks. They have a less magnetic coating as compared to which allows more tracks per inch on the track surface.
Optical Storage Device
In the case of Optical storage devices, the signals are stored in the form of light. So 0’s and 1’s are converted into light information. Let’s learn about some of the optical storage devices.
CD-ROM: It stands for compact disk read-only memory. When you add or write any data in the CD it is known as burning the CD. It is basically ROM where data can be read but once written cannot be rewritten or erased.
DVD-ROM: Used for high-quality video and has better storage such as 4GB to 18GB.
Conclusion
In this tutorial, you learned everything about hardware and how computers work? So with this knowledge, you are a computer hardware pro and you can easily diagnose your systems!
Are you a Python developer and trying to understand how does Python Language works? This article is for you where you will learn each and every bit and piece of Python Language. Let’s dive in!
Python
Python is a high-level language, which is used in designing, deploying, and testing at lots of places. It is consistently ranked among today’s most popular programming languages. It is also dynamic and object-oriented language but also works on procedural styles as well, and runs on all major hardware platforms. Python is an interpreted language.
High Level v/s Low Level Languages
High-Level Language: High-level language is easier to understand than is it is human readable. It is either compiled or interpreted. It consumes way more memory and is slow in execution. It is portable. It requires a compiler or interpreter for a translation.
Low-Level Language: Low-level languages are machine-friendly that is machines can read the code but not humans. It consumes less memory and is fast to execute. It cannot be ported. It requires an assembler for translation.
Interpreted v/s Compiled Language
Compiled Language: Compiled language is first compiled and then expressed in the instruction of the target machine that is machine code. For example – C, C++, C# , COBOL
Interpreted Language: An interpreter is a computer program that directly executes instructions written in a programming or scripting language, without requiring them previously to have been compiled into a machine language program and these kinds of languages are known as interpreter languages. For example JavaScript, Perl, Python, BASIC
Python vs C++/C Language Compilation Process
C++ or C Language: These Languages need compilation that means human-readable code has to be translated into Machine-readable code. The Machine code is executed by the CPU. Below is the sequence in which code execution takes place.
Human Readable is compiled.
Compilation takes place.
Compiled code generates a executable file which is in a machine code format (Understood by Hardware).
Execuation file is executed by CPU
Python Language:
Python is a high-level language
Bytecode, also termed p-code, is a form of instruction set designed for efficient execution by a software interpreter
Python code is written in .py format such as test.py.
Python code is then compiled into .pyc or .pyo format which is a byte code not a machine code ( Not understood by Machine) using Python Interpreter.
Once your program has been compiled to byte code (or the byte code has been loaded from existing .pyc files), it is shipped off for execution to something generally known as the Python Virtual Machine
Byte code is converted into machine code using PVM ( Python Virtual Machine).
Once your program has been compiled to byte code (or the byte code has been loaded from existing .pyc files), it is shipped off for execution to something generally known as the Python Virtual Machine
Now byte code that is test.pyc is further converted into machine code using virtual machine such as (10101010100010101010)
Finally Program is executed and output is displayed.
Conclusion
In this tutorial, you learnt how the python language works and interacts with Operating systems and Hardware. So, which application are you planning to build using Python?
In this Ultimate Guide as a beginner you will learn everything you should know about AWS EKS and how to manage your AWS EKS cluster ?
Common! lets begin !
Table of Content
What is AWS EKS ?
Why do you need AWS EKS than Kubernetes?
Installing tools to work with AWS EKS Cluster
Creating AWS EKS using EKSCTL command line tool
Adding one more Node group in the AWS EKS Cluster
Cluster Autoscaler
Creating and Deploying Cluster Autoscaler
Nginx Deployment on the EKS cluster when Autoscaler is enabled.
EKS Cluster Monitoring and Cloud watch Logging
What is Helm?
Creating AWS EKS Cluster Admin user
Creating Read only user for the dedicated namespace
EKS Networking
IAM and RBAC Integration in AWS EKS
Worker nodes join the cluster
How to Scale Up and Down Kubernetes Pods
Conclusion
What is AWS EKS ?
Amazon provides its own service AWS EKS where you can host kubernetes without worrying about infrastructure like kubernetes nodes, installation of kubernetes etc. It gives you a platform to host kubernetes.
Some features of Amazon EKS ( Elastic kubernetes service)
It expands and scales across many availability zones so that there is always a high availability.
It automatically scales and fix any impacted or unhealthy node.
It is interlinked with various other AWS services such as IAM, VPC , ECR & ELB etc.
It is very secure service.
How does AWS EKS service work?
First step in EKS is to create EKS cluster using AWS CLI or AWS Management console or using eksctl command line tool.
Now, next you can have your own machines EC2 where you can deploy applications or deploy to AWS Fargate which manages it for you.
Now connect to kubernetes cluster with kubectl or eksctl commands.
Finally deploy and run applications on EKS cluster.
Why do you need AWS EKS than Kubernetes?
If you are working with Kubernetes you are required to handle all the below thing yourself such as:
Create and Operate K8s clusters.
Deploy Master Nodes
Deploy Etcd
Setup CA for TLS encryption.
Setup Monitoring, AutoScaling and Auto healing.
Setup Worker Nodes.
But with AWS EKS you only need to manage worker node other all rest Masters node, etcd in high availability , API server, KubeDNS, Scheduler, Controller Manager, Cloud Controller all the things are taken care of Amazon EKS.
You need to pay0.20 US dollar per hour for your AWS EKS cluster which takes you to 144 US Dollar per month.
Installing tools to work with AWS EKS Cluster
AWS CLI: Required as a dependency of eksctl to obtain the authentication token. To install AWS cli run the below command.
pip3 install --user awscli
After you install aws cli make sure to set the access key and secret key id in aws cli so that it can create the EKS cluster.
eksctl: To setup and operate EKS cluster. To install eksctl run the below commands. Below command will download the eksctl binary in the tmp directory.
curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/v0.69.0/eksctl_Linux_amd64.tar.gz" | tar xz -C /tmp
Next, move the eksctl directory in the executable directory.
sudo mv /tmp/eksctl /usr/local/bin
To check the version of eksctl and see if it is properly install run below command.
eksctl version
kubectl: Interaction with k8s API server. To install the kubectl tool run the below first command that updates the system and installs the https package.
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
Again update the system so that it takes the effect after addition of new repository.
sudo apt-get update
Next install kubectl tool.
sudo apt-get install -y kubectl
Next, check the version of the kubectl tool by running below command.
kubectl version --short --client
IAM user and IAM role:
Create an IAM user with administrator access and use that IAM user to explore the AWS resources on the console. This is the user which also be used in the EC2 instance that you will use to manage AWS EKS cluster by passing user’s credentials in aws cli.
Also make sure to create a IAM role that you will apply on the EC2 instance from where you will manage AWS EKS and other AWS resources.
Creating AWS EKS using EKSCTL command line tool
Up to now you installed and setup the tools that are required for creating an AWS EKS Cluster. To know how to create a cluster using the eksctl command then run the help command which will provide you flags that you need to use while creating a AWS EKS cluster.
eksctl create cluster --help
Lets begin to create a EKS cluster. To do that create a file named eks.yaml and copy and paste the below content.
apiVersion is the kubernetes API version that will mange the deployment.
Kind denotes what kind of resource/object will kubernetes will create. In the below case as you need to provision cluster you should give Clusterconfig
metadata: Data that helps uniquely identify the object, including a name string, UID, and optional namespace.
nodegroups: Provide the name of node group and other details required for node group that will be used in your EKS cluster.
Now, execute the command below to create the cluster.
eksctl create cluster -f eks.yaml
Once cluster is successfully created run the below command to know the details of the cluster.
eksctl get cluster
Next, Verify the AWS EKS cluster on AWS console.
Also verify the nodes of the nodegroups that were created along with the cluster by running the below commands.
kubectl get nodes
Also, verify the Nodes on AWS console. To check the nodes navigate to EC2 instances.
Verify the nodegroups in the EKS Cluster by running the eksctl command.
eksctl get nodegroup --cluster EKS-cluster
Finally Verify the number of Pods in the EKS Cluster by running the below eksctl command.
eksctl get pods --all-namespaces
Adding one more Node group in the AWS EKS Cluster
To add another node group in EKS Cluster follow the below steps:
Create a yaml file as shown below and copy/paste the below content. In below file you will notice that previous nodegroup is already mentioned otherwise if you run this file without it it will override previous changes and remove the ng-1 node group from the cluster.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: EKS-cluster
region: us-east-1
nodeGroups:
- name: ng-1
instanceType: t2.small
desiredCapacity: 3
ssh: # use existing EC2 key
publicKeyName: testing
# Adding the another Node group nodegroup2 with min/max capacity as 3 and 5 resp.
- name: nodegroup2
minSize: 2
maxSize: 3
instancesDistribution:
maxPrice: 0.2
instanceTypes: ["t2.small", "t3.small"]
onDemandBaseCapacity: 0
onDemandPercentageAboveBaseCapacity: 50
ssh:
publicKeyName: testing
Next run the below command that will help you to create a nodegroups.
The cluster Autoscaler automatically launches additional worker nodes if more resources are needed, and shutdown worker nodes if they are underutilized. The AutoScaling works within a node group, so you should create a node group with Autoscaler feature enabled.
Cluster Autoscaler has the following features:
Cluster Autoscaler is used to scale up and down the nodes within the node group.
It runs as a deployment based on CPU and Memory utilization.
It can contain on demand and spot instances.
There are two types of scaling
Multi AZ Scaling: Node group with Multi AZ ( Stateless workload )
Single AZ Scaling: Node group with Single AZ ( Stateful workload)
Creating and Deploying Cluster Autoscaler
The main function and use of Autoscaler is it dynamically on the fly adds or removes the node within the nodegroup. The Autoscaler works as a deployment and depends on the CPU/Memory requests.
There are two types of scaling available : Multi AZ v/s Single AZ ( Stateful Workload) as EBS cannot be spread across multiple availability zone
To create the cluster Autoscaler you can add multiple nodegroups in the cluster as per need . In this examples lets consider to deploy 2 node groups with single AZ and 1 node groups across 3 AZs using spot instance with Autoscaler enabled
Create a file create and name it as autoscaler.yaml.
Verify the Pods. You should notice below that first pod is for Nodegroup1 , similarly second is for Nodegroup2 and finally the third is Autoscaler pod itself.
Nginx Deployment on the EKS cluster when Autoscaler is enabled.
To deploy the nginx application on the EKS cluster that you just created , create a yaml file and name it something which you find it convenient and copy/paste the below content into that.
By, now you have already setup EKS cluster but it is also important to monitor your EKS cluster. To monitor your cluster follow the below steps:
Create a below eks.yaml file and copy /paste below code into the file.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: EKS-cluster
region: us-east-1
nodeGroups:
- name: ng-1
instanceType: t2.small
desiredCapacity: 3
ssh: # use existing EC2 key
publicKeyName: eks-ssh-key
cloudWatch:
clusterLogging:
enableTypes: ["api", "audit", "authenticator"] # To select only few log_types
# enableTypes: ["*"] # If you need to enable all the log_types
Now apply the cluster logging by running the command.
eksctl utils update-cluster-logging --name=EKS-cluster --disable-types all
To get container metrics using cloudwatch: First add IAM policy (CloudWatchAgentServerPolicy ) to all your nodegroup(s)- to nodegroup(s) role and Deploy Cloudwatch Agent – After you deploy it will have its own namespace (cloudwatch-agent)
Now run the below command.
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/quickstart/cwagent-fluentd-quickstart.yaml | sed "s/{{cluster_name}}/EKS-course-cluster/;s/{{region_name}}/us-east-1/" | kubectl apply -f -
To check what all has been created in namespaces
kubectl get all -n amazon-cloudwatch
kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --limits=cpu=500m --expose --port=80
kubectl run --generator=run-pod/v1 -it --rm load-generator --image=busybox /bin/sh
Hit enter for command prompt
while true; do wget -q -O- http://php-apache.default.svc.cluster.local; done
What is Helm?
Helm is the package manager similar to what you have in ubuntu or python such as apt or pip. Helm contains mainly three components.
Chart: All the dependency files and application files.
Config: Any configuration that you would like to deploy.
Release: It is an running instance of a chart.
Helm Components
Helm client: Manages repository, Managing releases, Communicates with Helm library.
Helm library: It interacts with Kubernetes API server.
Installing Helm
To install helm make sure to create the directory with below commands and then change the directory
mkdir helm && cd helm
Next, add official stable helm repository which contains sample charts to install
To manage all resources in the EKS cluster you need to have dedicated users either ( Admin or Read only ) to perform tasks accordingly. Lets begin by creating an admin user first.
Create IAM user in AWS console (k8s-cluster-admin) and store the access key and secret key for this user locally on your machine.
Next, add user to configmap aws-auth section within map Users section. But before you add a user, lets find all the configmap in kube-system namespace because we need to store all the users in aws-auth.
kubectl -n kube-system get cm
Save the kubectl command in the yaml formatted file.
kubectl -n kube-system get cm aws-auth -o yaml > aws-auth-configmap.yaml
Next, edit the aws-auth-configmap.yaml and add the mapUsers with the following information:
userarn
username
groups as ( system:masters) which has admin/all permissions basically a role
Run the below command to apply the changes of newly added user.
After you apply changes you will notice that in AWS EKS you will not see any warning such as kubernetes objects cannot be accessed or something like that.
Now check if user has been properly created by running the describe command.
kubectl -n kube-system describe cm aws-auth
Next, add user to aws credentials file in dedicated section (profile) and then export it using export command or store it in aws cli command line.
export AWS_PROFILE="profile_name"
Finally check which user is currently running the aws cli commands
aws sts get-caller-identity
Creating a read only user for the dedicated namespace
Similarly, now create a read only user for AWS-EKS service. Lets follow the below steps to create a read only user and map it in configmap with IAM.
Create a namespace using below namespace.
kubectl create namespace production
Create a IAM user on AWS Console
Create a file rolebinding.yaml and add both the role and role bindings that includes the permissions that a kubernetes user will have.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: production
name: prod-viewer-role
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["*"] # can be further limited, e.g. ["deployments", "replicasets", "pods"]
verbs: ["get", "list", "watch"]
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: prod-viewer-binding
namespace: production
subjects:
- kind: User
name: prod-viewer
apiGroup: ""
roleRef:
kind: Role
name: prod-viewer-role
apiGroup: ""
Now apply the role and role bindings using the below command.
kubectl apply -f rolebinding.yaml
Next edit the yaml file and apply the changes such as userarn, role and username as you did previously.
kubectl -n kube-system get cm aws-auth -o yaml > aws-auth-configmap.yaml
Amazon VPC contains CNI Plugins from which each Pod receives IP address which is linked with ENI .
Pods have same IP address within the VPC that means inside and outside the EKS cluster.
Make sure to use maximum IP address by using CIDR/18 which has more IP address.
EC2 instance can also have limited amount of ENI/IP address that is each EC2 instance can have limited PODS ( like 36 or so according to Instance_type)
IAM and RBAC Integration in AWS EKS
Authentication is done by IAM
Authorization is done by kubernetes RBAC
You can assign RBAC directly to IAM entities.
kubectl ( USER SENDS AWS IDENTITY) >>> Connects with EKS >>> Verify AWS IDENTITY ( By Authorizing AWS Identity with Kubernetes RBAC )
Worker nodes join the cluster
When you create a worker node, assign the IAM Role and authorize that IAM Role needs to be authorized in RBAC in order to join the cluster. Add system:bootstrappers and system:nodes groups in your ConfigMap. The value for rolearn is the NodeInstanceRole and then run the below command
kubectl apply -f aws-auth.yaml
Check current state of cluster services and nodes
kubectl get svc,nodes -o wide
How to Scale Up and Down Kubernetes Pods
There are three ways of Scaling up/down the kubernetes Pods, Lets look at all of these three.
Scale the deployment to 3 replicas ( that is 3 pods will be scaled) using kubectl scale command.
Understanding the difference between high level and low level language.
Interpreted v/s Compiled Language
Introduction to Python
How Python Works ?
Python Interpreter
Python Standard Library
Python Implementations
Python Installation
Python Installation on Linux Machine
Python Installation on Windows Machine
Python Installation on MacOS
Conclusion
Understanding the difference between High & Low-level Languages
High-Level Language: High-level language is easier to understand than is it is human readable. It is either compiled or interpreted. It consumes way more memory and is slow in execution. It is portable. It requires a compiler or interpreter for a translation.
Low-Level Language: Low-level Languages are machine-friendly that is machines can read the code but not humans. It consumes less memory and is fast to execute. It cannot be ported. It requires an assembler for translation.
Interpreted v/s Compiled Language
Compiled Language: Compiled language is first compiled and then expressed in the instruction of the target machine that is machine code. For example – C, C++, C# , COBOL
Interpreted Language: An interpreter is a computer program that directly executes instructions written in a programming or scripting language, without requiring them previously to have been compiled into a machine language program and these kinds of languages are known as interpreter languages. For example JavaScript, Perl, Python, BASIC
Introduction to Python
Python is a high-level language, which is used in designing, deploying, and testing at lots of places. It is consistently ranked among today’s most popular programming languages. It is also dynamic and object-oriented language but also works on procedural styles as well, and runs on all major hardware platforms. Python is an interpreted language.
How does Python Works?
Bytecode, also termed p-code, is a form of instruction set designed for efficient execution by a software interpreter
First step is to write a Python program such as test.py
Then using Python interpreter program is in built compiled and gets converted into byte code that is test.pyc.
Python saves byte code like this as a startup speed optimization. The next time you run your program, Python will load the .pyc files and skip the compilation step, as long as you haven’t changed your source code since the byte code was last saved.
Once your program has been compiled to byte code (or the byte code has been loaded from existing .pyc files), it is shipped off for execution to something generally known as the Python Virtual Machine
Now byte code that is test.pyc is further converted into machine code using virtual machine such as (10101010100010101010)
Finally Program is executed and output is displayed.
Python Interpreter
Python includes both interpreter and compiler which is implicitly invoked.
In case of Python version 2, the Python interpreter compiles the source file such as file.py and keep it in same directory with an extension file.pyc
In case of Python version 3 : the Python interpreter compiles the source file such as file.py and keep it in subdirectory __pycache__
Python does not save the compiled bytecode when you run the script directly; rather, Python recompiles the script each time you run it.
Python saves bytecode files only for modules you import however running Python command with -B flag avoids saving compiled bytecode to disk.
You can also directly execute Python script in the Unix operating system if you add shebang inside your script.
#! /usr/bin/env python
Python Standard Library
Python standard library contains several well-designed Python modules for convenient reuse like representing data, processing text, processing data, interacting with operating systems and filesystems, and web programming. Python modules are basically Python Programs like a file (abc.py) that are imported.
There are some extension modules that allows Python code to access functionalities supplied by underlying OS or other software’s components such as GUI, database and network, XML parsing. You can also wrap existing C/C++ libraries into python extension modules.
Python Implementations
Python is more than a language, you can utilize the implementation of Python in many ways such as :
CPython:CPython is an interpreter, compiler, set of built in and optional extension modules all coded in C language. Python code is converted into bytecode before interpreting it.
IronPython: Python implementation for the Microsoft-designed Common Language Runtime (CLR), most commonly known as .NET, which is now open source and ported to Linux and macOS
PyPy:PyPy is a fast and flexible implementation of Python, coded in a subset of Python itself, able to target several lower-level languages and virtual machines using advanced techniques such as type inferencing
Jython: Python implementation for any Java Virtual Machine (JVM) compliant with Java 7 or better. With Jython, you can use all Java libraries and framework and it supports only v2 as of now.
IPython: Enhances standard CPython to make it more powerful and convenient for interactive use. IPython extends the interpreter’s capabilities by allowing abbreviated function call syntax, using question mark to query an objects documentation etc.
Python Installation
Python Installation on Linux Machine
If you are working on the Latest platforms you will find Python already installed in the systems. At times Python is not installed but binaries are available in the system which you can install using RPM tool or using APT in Linux machines and for Windows use MSI( Microsoft Installer ) .
Python V2 comes installed on macOS but you should install the latest Python version always. The popular third-party macOS open-source package manager Homebrew offers, among many other packages, excellent versions of Python, both v2 and v3
To install Homebrew, open Terminal or your favorite OS X terminal emulator and run
Verify the installation of Python using below command
Conclusion
In this tutorial, you learned a basic introduction to python, why it is interpreted, and high-level language. Also, you learned lots of details of python data types, keywords, and how python works. There were a handful of examples that you learned. Hope this tutorial will help you and if you like please share it.
If you want to analyze data for your website or applications, consider learning ELK Stack or Elastic Stack that contains Elasticsearch, logstash, and Kibana dashboard.
Elasticsearch is a powerful analytics search engine that allows you to store, index, and search the documents of all types of data in real-time. But if you need your search engine to automatically scale, load-balanced then AWS Elasticsearch (Amazon OpenSearch) is for you.
In this tutorial, you will learn what is Elastic Stack, Elasticsearch, Logstash, kibana dashboard, and finally AWS Elasticsearch from Scratch, and believe me, this tutorial will be helpful for you.
The ELK stack or Elastic Stack is used to describe a stack that contains: Elasticsearch, Logstash, and Kibana. The ELK stack allows you to aggregate logs from all your systems and applications, analyze these logs, and create visualizations for application and infrastructure monitoring, faster troubleshooting, security analytics, and more.
E = Elasticsearch: Elasticsearch is a distributed search and analytics engine built on Apache Lucene
L = Logstash: Logstash is an open-source data ingestion tool that allows you to collect data from a various sources and then transforms it and send it to your desired destination
K = Kibana: Kibana is a data visualization and exploration tool for reviewing logs and events.
ELK Stack architecture
What is Elasticsearch ?
Elasticsearch is an analytics and full-text search engine built on the Apache Lucene search engine library where the indexing, search, and analysis operations occur. Elasticsearch is a powerful analytics search engine that allows you to store, index, and search the documents of all types of data in real-time.
Even if you have structured or unstructured text numerical data, Elasticsearch can efficiently store and index it in a way that supports fast searches. Some of the features of Elasticsearch are:
Provides the search box on the website, web page or on applications.
Stores and analyze the data and metrics.
Logstash and Beats helps with collecting, aggregating the data and storing it in Elasticsearch.
Elasticsearch is used in the machine learning.
Elasticsearch stores complex data structures that have been serialized as JSON documents.
If you have multiple Elasticsearch nodes in Elasticsearch cluster then documents are distributed across the cluster and can be accessed immediately from any node.
Elasticsearch also has the ability to be schema-less, which means that documents can be indexed without explicitly specifying how to handle each of the different fields.
The Elasticsearch REST APIs support structured queries, full text queries, and complex queries that combine the two.You can access all of these search capabilities using Elasticsearch’s comprehensive JSON-style query language (Query DSL).
Elasticsearch index can be thought of as an optimized collection of documents and each document is a collection of fields, which are the key-value pairs that contain your data.
Elasticsearch index is really just a logical grouping of one or more physical shards, where each shard is actually a self-contained index.
There are two types of shards: primaries and replicas. Each document in an index belongs to one primary shard. The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can be changed at any time.
Sharding splits index or indices into smaller pieces. It is used so that more number of documents can be stored at index level, easier to fit large indices into nodes, improve query throughput. By default index have one shard and you can add more shards.
Elasticsearch Cluster
Elasticsearch provides REST API for managing your cluster and indexing and searching your data. For testing purposes, you can easily submit requests directly from the command line or through the Kibana dashboard by running the GET request in the Kibana console under dev tools, as shown below.
You can find the Elasticsearch cluster health by running the below command where _cluster is API and health is the command.
GET _cluster/health
Checking the health of the Elasticsearch cluster
To check the Elasticsearch node details using below command.
GET _cat/nodes?v
Checking the health of the elasticsearch node
To check the Elasticsearch indices configured, run the below command. You will notice kibana is also listed as indices because kibana data is also stored in elasticsearch.
GET _cat/indices
Checking the Elasticsearch indices on the elasticsearch cluster
To check the Primary and replica shards from a kibana console run the below request.
GET _cat/shards
Checking all the primary shards and replica shards in the elasticsearch cluster
QuickStart Kibana Dashboard
Kibana allows you to search the documents, observe the data and analyze the data, visualize in charts, maps, graphs, and more for the Elastic Stack in the form of a dashboard. Your data can be structured or unstructured text, numerical data, time-series data, geospatial data, logs, metrics, security events.
Kibana also manages your data, monitor the health of your Elastic Stack cluster, and control which users have access to the Kibana Dashboard.
Kibana also allows you to upload the data into the ELK stack by uploading your file and optionally importing the data into an Elasticsearch index. Let’s learn how to import the data in the kibana dashboard.
Create a file named shanky.txt and copy/paste the below content.
[ 6.487046] kernel: emc: device handler registered
[ 6.489024] kernel: rdac: device handler registered
[ 6.596669] kernel: loop0: detected capacity change from 0 to 51152
[ 6.620482] kernel: loop1: detected capacity change from 0 to 113640
[ 6.636498] kernel: loop2: detected capacity change from 0 to 137712
[ 6.668493] kernel: loop3: detected capacity change from 0 to 126632
[ 6.696335] kernel: loop4: detected capacity change from 0 to 86368
[ 6.960766] kernel: audit: type=1400 audit(1643177832.640:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=394 comm="apparmor_parser"
[ 6.965983] kernel: audit: type=1400 audit(1643177832.644:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=396 comm="apparmor_parser"
Once the file is uploaded successfully you will see the details of all code that you uploaded.
Data uploaded in the kibana
Details of the data uploaded in the kibana
Next create the elasticsearch index and click on import.
Creating the elasticsearch index on the Elasticsearch cluster
After import is successful you will see the status of your elasticsearch index as below.
Status of file upload in kibana
Next, click View index in Discover as shown in the the previous image. Now you should be able to see the logs within elasticsearch index (shankyindex).
Checking the logs in kibana with newly created index
Kibana allows you to perform the below actions such as:
Refresh, flush, and clear the cache of your indices or index.
Define the lifecycle of an index as it ages.
Define a policy for taking snapshots of your Elasticsearch cluster.
Roll up data from one or more indices into a new, compact index.
Replicate indices on a remote cluster and copy them to a local cluster.
Alerting allows you to detect conditions in different Kibana apps and trigger actions when those conditions are met.
What is Logstash?
Logstash allows you to collect the data with real-time pipelining capabilities. Logstash allows you to collect data from various sources beats and push it to the elasticsearch cluster. With Logstash, any type of event is transformed using an array of input, filter, and output plugins, further simplifying the ingestion process.
Working of Logstash
Features of Logstash
Now that you have a basic idea about Logstash, let’s look at some of the benefits of Logstash, such as:
Logstash hndle all types of logging data and easily ingest web logs like Apache, and application logs like log4j for Java.
Logstash captures other log formats like syslog, networking and firewall logs.
One of the main benefits of Logstash is to securely ingest logs with Filebeat.
What is AWS Elasticsearch or Amazon OpenSearch Service??
Amazon Elasticsearch Service or OpenSearch is a managed service that deploys and scales the Elasticsearch clusters in the cloud. Elasticsearch is an open-source analytical and search engine that performs real-time application monitoring and log analytics.
Amazon Elasticsearch service provisions all resources for Elasticsearch clusters and launches it. It also replaces the failed Elasticsearch nodes in the cluster automatically. Let’s look at some of the key features of the Amazon Elasticsearch Service.
AWS Elasticsearch or Amazon OpenSearch can scale up to 3 PB of attached storage and works with various instance types.
AWS Elasticsearch or Amazon OpenSearch easily integrates with other services such as IAM for security, VPC, AWS S3 for loading data, AWS Cloud Watch for monitoring and AWS SNS for alerts notifications.
Creating the Amazon Elasticsearch Service domain or OpenSearch Service domain
Now that you have a basic idea about the Amazon Elasticsearch Service domain or OpenSearch Service let’s create the Amazon Elasticsearch Service domain or OpenSearch Service domain using the Amazon Management console.
While in the Console, click on the search bar at the top, search for ‘Elasticsearch’, and click on the Elasticsearch menu item.
Now Elasticsearch service has been replaced with Opensearch service.
Searching for Elasticsearch service
Creating a Amazon Elasticsearch domain is same as that of Elasticsearch cluster that means domains are clusters with the settings, instance types, instance counts, and storage resources that you specify. Click on create a new domain.
Creating an Amazon Elasticsearch domain
Next, select the deployment type as Development and testing.
Choosing the deployment type.
Next, select the below settings as defined below:
For Configure domain provide the Elasticsearch domain name as “firstdomain”. A domain is the collection of resources needed to run Elasticsearch. The domain name will be part of your domain endpoint.
For Data nodes, choose the t3.small.elasticsearch and ignore rest of the settings and click on NEXT.
For Network configuration, choose Public access.
For Fine-grained access control, choose Create master user and provide username as user and password as Admin@123. Fine-grained access control keeps your data safe.
For Domain access policy, choose Allow open access to the domain. Access policies control whether a request is accepted or rejected when it reaches the Amazon Elasticsearch Service domain.
Further keep clicking on NEXT button and create the domain which takes few minutes for Domain to get Launched.
Viewing the Elasticsearch domain or Elasticcluster endpoint
After successful creation of Elasticsearch domain. Click on the firstdomain Elasticsearch domain.
Elasticsearch domain (first domain)
Uploading data in AWS Elasticsearch
You can load streaming data into your Amazon Elasticsearch Service (Amazon ES) domain from many different sources like Amazon Kinesis Data Firehose, Amazon Cloud Watch Logs, Amazon S3, Amazon Kinesis Data Streams, Amazon DynamoDB, AWS Lambda functions as event handlers.
In this tutorial you will use a sample data to upload the data. To upload the sample data go to the Elasticsearch domain URL using the username user and password Admin@123 and then click on Add data.
Adding data in Elasticsearch
Now use sample data and add e-commerce orders.
sample data to add e-commerce orders in Elasticsearch cluster
Search documents in Kibana Dashboard
Kibana is a popular open-source visualization tool that works with the AWS Elasticsearch service. It provides an interface to monitor and search the indexes. Let’s use Kibana to search the sample data you just uploaded in AWS ES.
Now in the Elasticsearch domain URL itself, Click on Discover option on the left side to search the data.
Click on the Discover option.
Now you will notice that Kibana has the data that got uploaded. You can modify the timelines and many other fields accordingly.
Viewing the data in the Kibana dashboard
Join 50 other followers
Kibana provided the data when we searched in the dashboard using the sample data you uploaded.
Conclusion
In this tutorial, you learned what Elastic Stack, Elasticsearch, Logstash, kibana dashboard, and AWS Elasticsearch from Scratch using Amazon Management console. Also, you learned t how to upload the sample data in AWS ES.
Now that you have a strong understanding of ELK Stack, Elasticsearch, kibana, and AWS Elasticsearch, which site are you planning to monitor using ELK Stack and components?
Powerful, dynamic language, with static-typing and static compilation capabilities, for the Java platform aimed at improving developer productivity .Groovy syntax is simple and easy. It saves a lot of code and effort thus increasing the productivity of developer if he had to do the same thing in Java.
In this tutorial you will learn what is groovy and how to install Groovy on windows and Linux machine . Later you will see two examples which helps you kickstart for Writing Groovy Scripts.
Table of Content
What is Groovy?
Prerequisites
How to Install Groovy on Windows Machine
How to Install Groovy on Ubuntu Machine
Groovy Syntax
Groovy Examples
Conclusion
What is Groovy?
Groovy is a Powerful static as well as dynamic language which is almost same as Java language with few differences. Groovy language is vastly used in Jenkins Pipelines. It integrates with Java libraries very well to deliver powerful enhancements and features including domain specific language authoring and scripting capabilities.
Basic Features of Groovy
Groovy supports all Java libraries and it has its own libraries as well.
It has a simple similar syntax as that of Java but in more simpler form
It has both static and dynamic nature
It has great extensibility for the language and tooling.
Last but not the least it a free open source language which is being used by lot of developers.
Prerequisites
Ubuntu 18 Machine or Windows machine
Make sure to have Java 8 plus installed on machines. To check Java version run the following command.
java --version
On Ubuntu Machine
On Windows Machine
How to Install Groovy on ubuntu Machine
Installing Groovy on Ubuntu machine is pretty straightforward. Lets Install the Groovy on Ubuntu 18 machine.
First Update the ubuntu official repository by running the apt command.
sudo apt update
Now, download the groovy installation script by running the curl command.
Now you will see windows installer package, once you click on it it will automatically download the file.
Now click on the the downloaded windows installer package and installation will begin.
Accept the license Agreement
Make sure you select Typical for Setup Type and click on Install
Now Groovy is successfully installed on windows machine. Open Groovy console from the Start menu & run a simple command to test.
Groovy Syntax
Shebang line
This allows script to run groovy scripts directly from command line provided you have groovy installed and groovy command is available on the PATH
#!/usr/bin/env groovy
println "Hello from the shebang line"
Strings
Strings are basically chain of characters. Groovy strings are written with single quotes ' or double quotes '' and even with triple quotes '''
'This is an example of single line'
"This is an example of double line"
def threequotes = '''
line1
line2
line3
'''
String interpolation
Groovy expressions can be interpolated and it is just like replacing a placeholder with its value. Placeholder in groovy are surrounded by ${} or $ . Also if you pass GString in any method where string is required then you should replace it by calling toString() on that method.
def name = "automate"
def greet = "Hello $name"
Groovy Examples
Lets now see two examples of Groovy
JsonSlurper : JsonSlurper is a class that parses JSON text or reader content into Groovy data
creating instance of the JsonSlurper class
Using the parseText function of the JsonSlurper class to parse some JSON text
access the values in the JSON string via the key.
import groovy.json.JsonSlurper
class Example {
static void main(String[] args) {
def jsonSlurper = new JsonSlurper()# creating instance of the JsonSlurper class
def object = jsonSlurper.parseText('{ "name": "John", "ID" : "1"}')
println(object.name);
println(object.ID);
}
}
Catching Exceptions
Accessing an array with an index value which is greater than the size of the array
class Example {
static void main(String[] args) {
try {
def arr = new int[3];
arr[5] = 5;
}catch(ArrayIndexOutOfBoundsException ex) {
println("Catching the Array out of Bounds exception");
}catch(Exception ex) {
println("Catching the exception");
}
println("Let's move on after the exception");
}
}
Conclusion
This tutorial is pretty straightforward and to get you started with Groovy. In this tutorial you learnt what is groovy and how to install Groovy on windows and Linux machine . Later you learnt two examples which helps you kickstart for Writing Groovy Scripts.
Well , Groovy is used at various places such as Jenkins pipelines etc.What do you plan to code with Groovy next ?
With lots of software development and testing around different applications and products you certainly need a best way to deploy it in effective and in best way. With So many microservices and code it becomes very crucial for any developer or system engineers to collaborate and make a successful product ready.
Managing the code is now very well taken care by Git which is distributed code repository but on the top of it deployment has been very effective and easily managed with the help of GitLab
In this tutorial you will learn all about GitLab , Managing Pipelines , Projects and many more which a devops engineer should know to get started.
Table of Content
What is GitLab?
Prerequisites
Creating Projects on GitLab
Creating a Repository on GitLab
Creating a Branch on GitLab
Get started with GitLab CI/CD Pipelines
Pipeline Architecture
Conclusion
What is GitLab?
Git is a distributed version control designed to handle small to large projects with speed and efficiency. On the top of Git , GitLab is fully integrated platform to manage devops lifecycle or software developments.
It is single application to manage entire DevOps lifecycle.
Prerequisites
You should have GitLab account handy. If you don’t have create it from here
Creating Projects on GitLab
GitLab projects hold all the files , folders , code and all the documents you need to build your applications.
To create a project in GitLab click on Projects on the top and then click on Create a Project
Now click on Create blank project
On the Blank project tab provide the Project name and as this is demo we will keep this repository Private.
Now Project is successfully created.
You are ready to upload files either manually create/upload on GitLab
Also you can push the files using command line by cloning the repository and adding the files as show below.
A repository is a place where you store all your code and related files. It is part of a Project. You can create multiple repositories in a single project.
To create a new repository, all you need to do is create a new project or fork an existing project. Once you create a new project, you can add new files via UI or via command line.
Creating a Branch on GitLab
By Now, you saw GitLab Project creation. By default if you add any file it will be checked in master branch.
Click on New file and then select Dockerfile and add content and then commit the file by adding the comments.
You will see that Dockerfile is now added in master branch under FirstGitLab project.
So far we created a file which by default gets added in master branch. But if you need a separate Branch click on the Branches and then hit New Branch.
Provide a name for the new branch.
Get started with GitLab CI/CDPipelines
Before you start CI/CD part on GitLab make sure to have following
runners : runners are agents that run your CI/CD jobs. To check the available runners Go to Settings > CI/CD and expand Runners. As long as you have at least one active available runner then you will be able to run the Job.
.gitlab-ci.yml file : In this file you define your CI/CD jobs , decisions which runner should take with specific conditions, structure of job and order of Jobs. Go to Project overview and then click on New file & name it as .gitlab-ci.yml
Now Paste the below content
build-job:
stage: build
script:
- echo "Hello, $GITLAB_USER_LOGIN"
test-job:
stage: test
script:
- echo "Testing CI/CD Pipeline"
deploy-job:
stage: deploy
script:
- echo "Deploy from the $CI_COMMIT_BRANCH branch"
Now Pipeline should automatically trigger for this pipeline configuration. Click on Pipelines to validate and View status of pipeline.
To view details of a job, click the job name, for example build.
Pipelines can be scheduled to run automatically as and when required.
Pipeline Architecture
Pipelines are the fundamental building blocks for CI/CD in GitLab. There are three main ways to structure your pipelines, each with their own advantages. These methods can be mixed and matched if needed:
Basic: Good for straightforward projects where all the configuration are stored at one place. This is the simplest pipeline in GitLab. It runs everything in the build stage at the same time and once all of those finish, it runs everything in the test stage the same way, and so on.
If Build A is completed it waits for BUILD B and once both are completed it moves to next TEST STAGE. Similarly if TEST B is completed it will wait for TEST A and then once both are completed they move to DEPLOY STAGE.
Directed Acyclic Graph: Good for large, complex projects that need efficient execution and you want everything to run as quickly as possible.
If Build A and TEST A both are completed it moves to next DEPLOY STAGE even if TEST B is still running
Child/Parent Pipelines: Good for monorepos and projects with lots of independently defined components. This job is run mostly using trigger keyword.
Conclusion
GitLab is the first single application for software development, security, and operations that enables continuous DevOps. GitLab makes the software lifecycle faster and improves the speed of business.
GitLab provides solutions for each of the stages of the DevOps lifecycle. So Which application are you going to build ?
Hope you had learnt a lot from this guide and helped you. If you like please share.
Python’s standard library is very extensive, offering a wide range of facilities. The library contains built-in modules (written in C) that provide access to system functionality such as file I/O that would otherwise be inaccessible to Python programmers, as well as modules written in Python that provide standardized solutions for many problems that occur in everyday programming
In this tutorial, we will learn everything which a beginner and a DevOps engineer should know in Python. We will cover the basic definition of python and some brilliant examples which will be enough to get you started with Python and for sure you will love it.
Table of content
What is Python?
Prerequisites
Python Keywords
Python Numbers
Python Strings
Python Tuple
Python Lists
Python Dictionary
Python Sets
Python variables
Python Built-in functions
Python Handling Exceptions
Python Functions
Python Searching
Conclusion
Python String
Python strings are a collection of characters surrounded by quotes ” “. There are different ways in which strings are declared such as:
str() – In this method you decalre the characters or words or data inside the double quotes.
Directly calling it in quotes – “Hello, this is method2 to display string”
Template strings – Template strings are designed to offer a simple string substitution mechanism. These built-in methods work for tasks where simple word substitutions are necessary.
from string import Template
new_value = Template("$a b c d") # a will be substituted here
x = new_value.substitute(a = "Automation")
y = new_value.substitute(a = "Automate")
print(x,y)
Some Tricky Examples of declaring string
Input String
print('This is my string 1') # Correct String
print("This is my string 2") # Correct String
# Examples of Special characters inside the String such as quotes
# print('Hello's Everyone') # Incorrect Statement
print('Hello\'s Everyone') # Correct Statement after using escape (To insert characters that are illegal in a string, use an escape character. )
print("Hello's Everyone") # Correct Statement enclose within double quotes
print('Hello "shanky') # Correct Statement
print('Hello "shanky"') # Correct Statement
# print("Hello "S"shanky") # Incorrect Statement
print("Hello ""shanky")
# No need to Escape if using triple quotes but proper use of triple quotes
print(''''This is not a string "''')
print('''Hello" how' are"" u " I am " f'ine'r''')
print('''''Hello" how' are"" u " I am " f'ine'r''')
print("""'''''Hello" how' are"" u " I am " f'ine'r""")
Output String
This is my string 1
This is my string 2
Hello's Everyone
Hello's Everyone
Hello "shanky
Hello "shanky"
Hello shanky
'This is not a string "
Hello" how' are"" u " I am " f'ine'r
''Hello" how' are"" u " I am " f'ine'r
'''''Hello" how' are"" u " I am " f'ine'r
Python Tuple
Tuples: Tuples are immutable ordered sequence of items that cannot be modified. The items of a tuple are arbitrary objects and may be of different types and allow duplicate values. For Example
# 10,20,30,30 are fixed at respective index 0,1,2,3 positions
(10,20,30,30) or (3.14,5.14,6.14)
Python Dictionaries
Dictionary are written as key:value pair , where key is an expression giving the item’s key and value is an expression giving the item’s value. A dictionary is a collection which is ordered*, changeable and does not allow duplicates.
# Dictionary with three items where x,y and z are keys.
# where x,y and z have 42, 3.14 and 7 as the values.
{'x':42, 'y':3.14, 'z':7}
Python Sets
Sets: Set stores multiple items in a single variable. It contains unordered and unindexed data. Sets cannot have two items with the same value.
{"apple", "banana", "cherry"}
Data types
Mutable or Immutable
String
Immutable (Cannot be modified)
Tuples
Immutable (Cannot be modified)
Integers
Immutable (Cannot be modified)
List
Mutable (Can be modified)
Sets
Mutable (Can be modified)
Floating point
Immutable (Cannot be modified)
Dictionaries
Mutable (Can be modified)
Python variables
Variables are stored as a information it could be number , symbol , name etc. which are used to be referenced. Lets see some of the examples of Python variables.
There are few points one must remember when using variables such as
Variables cannot start with digits
Spaces are not allowed in variables.
Avoid using Python keywords
Example 1:
In below example var is a variable and value of var is this is a variable
var="this is a variable" # Defining the variable
print(var) # Printing the value of variable
Example 2:
In below example we are declaring three variable.
first_word and second_word are storing the values
add_words is substituting the variables with values
In simple words these are key value pairs where keys can be number, string or custom object. Dictionary are represented in key value pairs separated by comma within curly braces.
map = {'key-1': 'value-1', 'key-2': 'value-2'}
You can access the particular key using following way
map['key-1']
Lets see an example to access values using get() method
my_dictionary = {'key-1': 'value-1', 'key-2': 'value-2'}
my_dictionary.get('key-1') # It will print value of key-1 which is value-1
print(my_dictionary.values()) # It will print values of each key
print(my_dictionary.keys()) # It will print keys of each value
my_dictionary.get('key-3') # It will not print anything as key-3 is missing
Python Built-in functions
There are various single line command which are already embedded in python library and those are known as built in functions. You invoke a function by typing the function name, followed by parentheses.
To check the Python version on windows or Linux machine run the following command.
python3 --version
To print the output of a program , use the print command.
print("Hello Devops")
To generate a list of number through a range built-in function run the following command.
list(range(0,10))
Handling Exceptions
Exceptions are error which causes a program to stop if not handled properly. There are many built-in exceptions, such as IOError, KeyError, and ImportError. Lets see a simple example below.
Here we defined a list of characters and stored it in a variable devops
Now, while true indicated that till the ,condition is true it will execute the try block.
.pop() is built in method to remove each item one by one.
Now in our case as soon as all the characters are removed then except block catches the IndexError and prints the message.
devops = ['d','e','v','o','p','s']
while True:
try:
devop = devops.pop()
print(devop)
except IndexError as e:
print("I think I did lot of pop ")
print(e)
break
Output:
s
p
o
v
e
d
I think I did lot of pop
pop from empty list
Python Functions
Earlier in this tutorial we have already seen that there are numerous built in function and some of them you used above. But you can define and create your own functions. Lets see the syntax of function.
Lets look at some of the Python functions examples
EXAMPLE 1
Here each argument use order of arguments to assign value which is also known as positional argument.
a and b variables are parameters which are required to run the function
1 and 2 are arguments which are used to pass the value to the function ( arguments are piece of information that’s passed from a function call to a function)
def my_function(a,b):
print(f" value of a is {a}")
print(f" value of b is {b}")
my_function(1, 2)
EXAMPLE 2:
With keyword arguments, assign each argument a default value:
def my_function(a=3,b=4):
print(f" value of a is {a}")
print(f" value of b is {b}")
my_function()
EXAMPLE 3
Passing arbitrary number of arguments. When you are not sure about the number of parameters to be passed then we call it as arbitrary. Lets look at an example
Find the Even in the string
mylist = []
def myfunc(*args): # args is to take any number of arguments together in myfunc
for item in args:
if int(item)%2 == 0:
mylist.append(item)
print(mylist)
myfunc(5,6,7,8,9)
EXAMPLE 4
IF LOOP: Find the least among two numbers if both numbers are even else return greater among both the numbers
def two_of_less(a,b): # Defining the Functionwhere a and b variables are parameters
if a%2==0 and b%2==0:
print(min(a,b)) # using built in function min()
if a%2==1 or b%2==1:
print(max(a,b)) # using built in function max()
two_of_less(2,4)
EXAMPLE 5
Write a function takes a two-word string and returns True if both words begin with same letter
def check(a):
m = a.split()
if m[0][0] == m[1][0] :
print("Both the Words in the string starts with same letter")
else:
print("Both the Words in the string don't start with same letter")
check('devops Engineer')
Python Searching
The need to match patterns in strings comes up again and again. You could be looking for an identifier in a log file or checking user input for keywords or a myriad of other cases.
Regular expressions use a string of characters to define search patterns. The Python re package offers regular expression operations similar to those found in Perl.
Lets look at example which will give you overall picture of in built functions which we can use with re module.
You can use the re.search function, which returns a re.Match object only if there is a match.
import re
import datetime
name_list = '''Ezra Sharma <esharma@automateinfra.com>,
...: Rostam Bat <rostam@automateinfra.com>,
...: Chris Taylor <ctaylor@automateinfra.com,
...: Bobbi Baio <bbaio@automateinfra.com'''
# Some commonly used ones are \w, which is equivalent to [a-zA-Z0-9_] and \d, which is equivalent to [0-9].
# You can use the + modifier to match for multiple characters:
print(re.search(r'Rostam', name_list))
print(re.search('[RB]obb[yi]', name_list))
print(re.search(r'Chr[a-z][a-z]', name_list))
print(re.search(r'[A-Za-z]+', name_list))
print(re.search(r'[A-Za-z]{5}', name_list))
print(re.search(r'[A-Za-z]{7}', name_list))
print(re.search(r'[A-Za-z]+@[a-z]+\.[a-z]+', name_list))
print(re.search(r'\w+', name_list))
print(re.search(r'\w+\@\w+\.\w+', name_list))
print(re.search(r'(\w+)\@(\w+)\.(\w+)', name_list))
As a Linux administrator it is very important to know how you are managing your applications & Software’s. Every command and every installation of packages require critical attention before executing it.
So In this Ultimate guide we will learn everything you should know about ubuntu repositories , how to add apt-repository & PPA repositories and working with ubuntu repository and apt commands.
Table of Content
What is ubuntu repository?
How to add a ubuntu repository?
Manually Adding apt-repository in ubuntu
Adding PPA Repositories
Working with Ubuntu repositories
How apt or apt-get command work with Ubuntu Repository
Conclusion
What is ubuntu repository?
APT repository is a network shared server or a local directory containing deb packages and metadata files that are readable by the APT tools. When installing packages using the Ubuntu Software Center or the command line utilities such as apt or apt-get the packages are downloaded from one or more apt software repositories.
On Ubuntu and all other Debian based distributions, the apt software repositories are defined in the /etc/apt/sources.list file or in separate files under the /etc/apt/sources.list.d/ directory.
The names of the repository files inside the /etc/apt/sources.list.d/ directory must end with .list.
How to add apt-repository in ubuntu ?
add-apt-repository is basically a python script that helps in addition of repositories in ubuntu.
Lets take a example to add a mongodb repository in ubuntu machine
add-apt-repository utility is included in software-properties-common package.
sudo apt-add-repository --yes --update ppa:ansible/ansible
# PPA is Personal Package Archive
Lets check the directory /etc/apt/sources.list.d/ has ansible PPA repository
Working with Ubuntu repositories
Repositories in ubuntu machine are basically file servers or network shares under which it has lot of packages , it could be .deb packages or files which are readable by apt or apt-get command.
/etc/apt/sources.list or
/etc/apt/sources.list.d
What does sources.list and sources.list.d contains ?
Software in Ubuntu’s repository is divided into four categories or components – main, restricted, universe and multiverse.
main: contains applications that are free software that are fully supported by ubuntu.
multiverse: contains software’s that are not free that requires license.
restricted: only to promote free software and ubuntu team cannot fix it & then provide it back to author if any issues are found.
universe: They have all the possible software’s which are free and open sourced but ubuntu don’t provide regular patch guarantee.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic main restricteddeb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic main restricted
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic universedeb-srchttp://security.ubuntu.com/ubuntu bionic-security multiverse
deb or deb-src are either .deb packages or source code
bionic , bionic-security , xenial are distributions code name.
main, restricted, universe and multiverse are repository categories.
How apt or apt-get command work with Ubuntu Repository
APT stands for Advanced Packaging Tool which performs functions such as installation of new software packages, upgrade of existing software packages, updating of the package list index, and even upgrading the entire Ubuntu system by connecting with repositories stored under /etc/apt/sources.list or /etc/apt/source.list.d/
Let us see an example of how apt command works with ubuntu repositories.
You will notice that all the above packages are already up to date and latest
Now run the apt update command to update the repositories. apt command contains three types of lines.
Hit: If there is no change in package version from the previous version
Ign: It means package is being ignored.
Get: It means it has a new version available. It will download the information about the version (not the package itself). You can see that there is download information (size in kb) with the ‘get’ line in the screenshot above.
apt update
After completion of command it provides the details if upgrade is required by any package or not. In our case it shows 37 packages can be upgraded. Lets see the list of packages which can be upgraded by running the following command.
apt list --upgradable
You can either upgrade a single package or upgrade all packages together.
To upgrade a single package use : apt install <package-name>
To upgrade all packages use :apt upgrade
Lets just update the curl package by running the apt install command and verify
apt install curl
You will notice that updating curl command upgraded 2 packages which were related to curl and rest of 35 are still not upgraded.
Now, lets upgrade rest of the 35 packages together by running apt upgrade command.
apt upgrade
Lets run apt update command again to verify if ubuntu still requires any software to be upgrade. Command output should look like “All packages are up to date”
apt update
Conclusion
In this tutorial we learnt everything about ubuntu repositories and how to add various repositories and how to work with them . Finally we saw how apt command works with ubuntu repositories.
This Ultimate Guide will give you a very strong understanding of package management which is most important thing for a Linux administrator. Hope you liked this tutorial and was helpful. Please share.
PowerShell Remoting has various benefits. It started with Windows when Windows administrators use to work remotely work with tons of windows machine over WinRM protocol. With Automation and Unix distribution spreading across the world and require by every single IT engineer, PowerShell introduced PSRemoting over SSH in PowerShell 7 to connect Windows to Linux and Linux to Windows remotely .
In this tutorial we will learn how to setup PS Remoting on windows machine and on Linux machine using PS remoting over SSH ( PowerShell 7 supported) . Finally we will connect both Windows to Linux and Linux to Windows machine. Lets get started.
Table of Content
What is PSRemoting or PowerShell Remoting Over WinRM?
What is PSRemoting or PowerShell Remoting Over SSH?
Prerequisites
Step by step set up SSH remoting on Windows
Step by step set up SSH remoting on Ubuntu
Test the OpenSSH connectivity from Windows machine to Linux using PSRemoting
Test the OpenSSH connectivity from Linux to Windows machine using PSRemoting
Conclusion
What is PSRemoting or PowerShell Remoting?
PowerShell Remoting is a feature of PowerShell. With PowerShell Remoting you can connect with a single or tons of servers at a single time.
PS Remoting Over SSH (Windows to Linux and Windows to Windows)
WS-Management or Web services management or WS-Man provides a common way for systems to access and exchange management information across the IT infrastructure.
Microsoft implemented WS-Management or Web services management or WS-Man in WinRM that is Windows Remote Management that allows hardware and operating systems, from different vendors to connect to each other. For WinRM to obtain data from remote computers, you must configure a WinRM listener. WinRM listener can work on both HTTP or HTTPS Protocols.
PS Remoting Over WinRM (Linux to Windows)
When PowerShell Remoting takes place between two servers that is one server try to run commands remotely on other server, the source server connects to destination server on WinRM Listener. To configure PSRemoting on local machine or remote machine please visit the link
What is PSRemoting or PowerShell Remoting Over SSH?
Microsoft introduced PowerShell 7 Remoting over SSH, which allows true multiplatform PowerShell remoting between Linux, macOS, and Windows. PowerShell SSH Remoting creates a PowerShell host process on the target machine as an SSH subsystem. Normally, Windows PowerShell remoting uses WinRM for connection negotiation and data transport. However, WinRM is only available on Windows-based machines. That means Linux machines can connect with windows or windows can connect to Windows over WinRM but Windows cannot connect to Linux.
With PowerShell 7 Remoting over SSH Now its possible to remote between Linux, macOS, and Windows.
PS Remoting Over SSH ( Windows to Linux , Linux to Windows)
Prerequisites
Microsoft Windows Server 2019 standard . This machine should also have PowerShell 7 installed. If you don’t have PowerShell installed please follow here to install.
Make sure you have local account setup in Windows server 2019. We will be using “automate” user.
Make sure you set the password for ubuntu user on ubuntu machine or if you have it then ignore.
Ubuntu machine with PowerShell 7 installed.
Step by step set up SSH remoting on Windows
Here we will discuss about how to setup SSH remoting on Windows Machine and run the PSRemoting commands.
Assuming you are on Windows 2019 standard machine with PowerShell 7 installed. Lets verify it once.
Before we setup SSH on windows machine & if you try to make a SSH session with Linux machine you will received an error message like this.
Next step is to install Open SSH client and server on Windows 2019 standard server. Lets use the PowerShell utility Add-WindowsCapabillity and run the commands.
Now, Edit the OpenSSH configuration sshd_config located in C:\Windows\System32\OpenSSH or you can find it in C:\ProgramData\ssh\sshd_config by adding Subsystem for PowerShell.
Test the OpenSSH connectivity from Windows machine to Linux using PSRemoting
Here now we are set with windows and ubuntu SSH remoting steps , now lets verify the SSH connectivity between from windows to ubuntu machine
Verification Method 1
Create a session and then enter into session and run commands from windows PowerShell to Linux PowerShell
New-PSSession -Hostname 54.221.35.44 -UserName ubuntu # Windows to Linux Create Session
Enter-PSSession -Hostname 54.221.35.44 -UserName ubuntu # Windows to Linux Enter Session
Verification Method 2
Create the session and then test the connectivity from Windows machine to Linux using Invoke-Command command
Test the OpenSSH connectivity from Linux to Windows machine using PSRemoting
Lets verify the SSH connectivity between from ubuntu machine to windows Machine.
Open PowerShell on ubuntu machine with following command
pwsh
Although you are on ubuntu machine lets verify the ubuntu version [Optional Step]
Now SSH into Windows machine using following command
ssh automate@3.143.233.234
Here we go , You can clearly see that we are have SSH into Windows machine successfully
Conclusion
PowerShell Remoting has various benefits. It started with Windows when Windows administrators use to work remotely work with tons of windows machine over WinRM protocol. With Automation and Unix distribution spreading across the world and require by every single IT engineer. To resolve problem of connecting windows to Linux and Linux to windows PowerShell introduced PSRemoting over SSH to connect Windows to Linux and Linux to Windows remotely with easy setups .
Hope you find this tutorial helpful. If you like please share it with your friends.
PSRemoting or PowerShell Remoting is a PowerShell based remoting which allows you to connect to one or thousands of remote computers and execute commands. PSRemoting allows you to sit at one place and execute commands on remote machine as if you are executing physically on the servers.
In this tutorial you will learn what is PS Remoting that is PowerShell Remoting and how to enable PowerShell Remoting locally and on remote machines.
Table of Content
What is PSRemoting or PowerShell Remoting?
Prerequisites
How to Enable PS Remoting Locally on system?
How to Enable PS Remoting on remote system?
Conclusion
What is PSRemoting or PowerShell Remoting?
PowerShell Remoting is a feature of PowerShell. With PowerShell Remoting you can connect with a single or tons of servers at a single time.
WS-Management or Web services management or WS-Man provides a common way for systems to access and exchange management information across the IT infrastructure.
Microsoft implemented WS-Management or Web services management or WS-Man in WinRM that is Windows Remote Management that allows hardware and operating systems, from different vendors to connect to each other. For WinRM to obtain data from remote computers, you must configure a WinRM listener. WinRM listener can work on both HTTP or HTTPS Protocols.
When PowerShell Remoting takes place between two servers that is one server try to run commands remotely on other server, the source server connects to destination server on WinRM Listener.
How to check WinRM listeners on Windows Host?
To check the WinRM listeners on windows host use the following command
winrm e winrm/config/listener
Prerequisites
Make sure you windows machine with PowerShell 7 installed . If you don’t have, Install it from here.
How to Enable PS Remoting Locally on system?
There are two ways in which you can enable PSRemoting on the local machine.
Use Enable-PSRemoting to Enable PS Remoting Locally on system
Invoke the command Enable-PSRemoting and this performs the following function
WinRM service is started
Creates listener on 5985 for HTTP
Registers and Enable PowerShell sessions
Set PowerShell sessions to allow remote sessions.
Restarts WinRM server
Enable-PSRemoting # By Default its enabled in Windows
On a Server OS, like Windows Server 2019, the firewall rule for Public networks allows on remote connections from other devices on the same network. On a client OS, like Windows 10, you will receive an error stating that you are a public network.
Command Ran on Windows 2019 server
Command Ran on Windows 10 Machine
If you want to ignore the Error message because of Network Profile on client like windows 10 use the following command
Enable-PSRemoting -SkipNetworkProfileCheck
Use WinRM to Enable PS Remoting Locally on system
We can use WinRM quickconfig command as well to enable PS Remoting on local machine
winrm quickconfig
How to Enable PS Remoting on remote system?
There are two ways in which you can enable PSRemoting on the remote machine.
Use PS exec to Enable PS Remoting on remote system
Using PS exec you can run command on remote machine after connecting to remote machine. When you run PS exec command , it initialize the PowerShell session on remote machine and then run the command.
.\psexec.exe \\3.143.113.23 -h -s powershell.exe Enable-PSRemoting -Force # 3.143.113.23 is remote machine's IP address
Use WMI to Enable PS Remoting on remotesystem
Using PowerShell and the Invoke-CimMethod cmdlet. Using the Invoke-CimMethod cmdlet, you can instruct PowerShell to connect to the remote computer over DCOM and invoke methods.
In this tutorial, you have learned what is PSRemoting and how to enable PSRemoting with various methods locally on the machine as well as remotely on the machine. This will give you great opportunity to automate with various remote machines together.
With some many windows or Linux Administrators in the world automation has always been top most requirement. PowerShell is one the most widely and command line shell which gives you string ability to perform any tasks with any remote operating system very easily.
In this tutorial we will go through basic definition of PowerShell, benefits and features of PowerShell and finally how to install latest PowerShell on both Windows and Ubuntu Machine.
Table of content
What is PowerShell?
Working with PowerShell
Install PowerShell 7.1.3 on Windows Machine
How to Install PowerShell 7.1.3 on Ubuntu Machine
Conclusion
What is PowerShell?
PowerShell is a command line tool or command line shell which helps in automation of various tasks , allows you to run scripts & helps you in managing variety of configuration. PowerShell runs on Windows , Linux and macOS
PowerShell is built on .NET Command Language Runtime that is ( CLR ) . It works currently on .NET 5.0 Framework as its runtime.
Features of PowerShell
It provides tab completion
It works with all .NET Frameworks objects
It allows pipelines of commands.
It has built support for various file formats such as JSON, CSV and XML
Working with PowerShell
PowerShell is a command line tool or command line shell which was meant for windows automation . But it has widely grown and upgraded with lots of feature and benefits. Lets check out some of the key benefits.
PowerShell can be used for cloud management such as retrieve or deploy new resources.
PowerShell can be used with Continuous integration and continuous deployment pipelines i.e.. CI/CD
PowerShell is widely used now by Devops and sysops engineers.
PowerShell comes with hundreds of preinstalled commands
PowerShell command are called cmdlets
To check the version of PowerShell , although there are various command but lets run the following
$PSVersionTable.PSVersion
Install PowerShell on Windows Machine
By default PowerShell is already present on the windows machine. To verify click on start bar and look for PowerShell.
Verify the current version of PowerShell by running the following command.
he apt software repositories are defined in the /etc/apt/sources.list file or in separate files under the /etc/apt/sources.list.d/ directory.
Register the Microsoft repository GPG keys. You will notice that as soon as we run the below command a repository is added inside /etc/apt/source.list.d directory
sudo dpkg -i packages-microsoft-prod.deb
Update the Repository again
sudo apt-get update
Enable the “universe” repositories
sudo add-apt-repository universe
Install PowerShell
sudo apt-get install -y powershell
Start PowerShell
pwsh
Lets verify PowerShell by invoking the Get-Command
Conclusion
This tutorial is pretty straightforward and to get you started with PowerShell. In this tutorial we defined what is PowerShell and what are benefits of PowerShell. Later we Installed Latest PowerShell 7.1.3 on both ubuntu and windows machine. Hope this tutorial helps you with PowerShell setup & please share it if you like.
Monitoring of application or system disk utilization has always remained a top most and crucial responsibility of any IT engineer. In the IT world with various software’s , automation and tools it is very important to keep a track of disk utilization regularly.
Having said that, In this tutorial we will show you best commands and tools to work with your disk utilization. Please follow me along to read and see these commands and their usage.
Table of content
Check Disk Space using disk free or disk filesystems command ( df )
Check Disk Space using disk usage command ( du )
Check Disk Usage using ls command
Check Disk Usage using pydf command
Check Disk Usage using Ncdu command( Ncurses Disk Usage )
Check Disk Usage using duc command
conclusion
Check Disk Space using disk free or disk filesystems command (df)
It stands for disk free. This command provides us information about the available space and used space on a file system. There are multiple parameters which can be passed along with this utility to provide additional outputs. Lets look at some of the commands from this utility.
To see all disk space available on all the mounted file systems on ubuntu machine.
df
To see all disk space available on all the mounted file systems on ubuntu machine in human readable format.
You will notice a difference in this command output and a previous. The difference is instead of 1k-blocks you will see size which is human readable.
df -h
To check the disk usage along with type of filesystem
df -T
To check disk usage of particular Filesystem
df /dev/xvda1
To check disk usage of multiple directories.
df -h /opt /var /etc /lib
To check only Percent of used disk space
df -h --output=source,pcent
To check data usage based on filesystem wise
df -h -t ext4
Check Disk Space using disk usage command ( du )
du command provides disk usage information. This command provides file and directories space utilization. Lets see some of the example .
To check disk usage of directory
du /lib # Here we are taking lib directory
To check disk usage of directory with different block size type .
M for MB
G for GB
T for TB
du -BM /var
To check disk usage according to the size
Here s represents summarize
Here k represents size in KB , you can use M, G or T and so on
Here sort represents sort
Here n represents in numerical order
Here r represents in reverse order
du -sk /opt/* | sort -nr
Check Disk Usage using ls command
ls command is used of listing of files but also provides information about disk utilized by directories and files. Lets see some of these command.
To list the files in human readable format.
ls -lh
To list the file in descending order of size of files.
ls -ls
Check Disk Usage using pydf command
pydf is a python based command-line tool which is used to display disk usage with different colors. Lets dive into command now.
To check the disk usage with pydf
pydf -h
Check Disk Usage using Ncdu command (Ncurses Disk Usage)
Ncdu is a disk utility for Unix systems. This command is text-based user interface under the [n]curses programming library.Let us see a command from Ncdu
ncdu
Check Disk Usage using duc command
Duc is a command line utility which queries the disk usage database and also create, maintain and the database.
Before we run a command using duc be sure to install duc package.
sudo apt install duc
duc is successfully installed , now lets now run a command
duc index /usr
To list the disk usage using duc command with user interface
duc ui /usr
Conclusion
There are various ways to identify and view disk usage in Linux or ubuntu operating system. In this tutorial we learnt and showed best commands and disk utilities to work with . Now are you are ready to troubleshoot disk usage issues or work with your files or application and identify the disk utilization.
Hope this tutorial gave you in depth understanding and best commands to work with disk usage . Hoping you never face any disk issues in your organization. Please share if you like.
Well AWS EBS that is Elastic block store is a very important and useful service provided by AWS. Its a permanent and shared storage and is used with various applications deployed in AWS EC2 instance. Automation is playing a vital role in provisioning or managing all the Infrastructure and related components.
Having said that , In this tutorial we will learn what is an AWS EBS , AWS EBS Snapshots and many amazing things about storage types and how to delete EBS snapshots using shell script on AWS step by step.
AWS EBS is your pendrive for instances, always use it when necessary and share with other instances.
Table of Content
What is Shell script ?
What is AWS EBS ?
What areEBS Snapshots in AWS ?
Prerequisites
Install AWS CLI Version 2 on windows machine
How to Delete EBS Snapshots from AWS account using shell script
Conclusion
What is Shell Scripting or Bash Scripting?
Shell Script is simply a text of file with various or lists of commands that are executed even on terminal or shell one by one. But in order to make thing little easier and run together as a group and in quick time we write them in single file and run it.
Main tasks which are performed by shell scripts are : file manipulation , printing text , program execution. We can include various environmental variables in script that can be used at multiple places , run programs and perform various activities are known as wrapper scripts.
A good shell script will have comments, preceded by a pound sign or hash mark, #, describing the steps. Also we can include conditions or pipe some commands to make more creative scripts.
When we execute a shell script, or function, a command interpreter goes through the ASCII text line-by-line, loop-by-loop, test-by-test, and executes each statement as each line is reached from the top to the bottom.
What is EBS ?
EBS stands for Amazon Elastic block store which is permanent storage just like your pendrive or harddisk. You can mount EBS volume to AWS EC2 instances. It is very much possible to create your own file system on top of these EBS volumes.
EBS volumes are mounted on AWS EC2 instance and are not dependent on AWS EC2 instance life. They remain persistent.
Amazon Elastic block store (EBS)
Key features of EBS
EBS can be created in any Availability zones
EBS cannot be directly attached with any instance in different Availability zone. We would need to create a Snapshot that is like a backup copy and then from that snapshot restore it to new volume and then finally use it in other Availability zone.
What are HDD and SSD storage ?
HDD ( Hard disk drive )
Hard disk drive is a old technology. They depends on spinning disks and platters to read and write data. There is a motor which spins the platter whenever any request comes to read or write the data. Platter contains tracks and each track contains severs sectors. These drives run slowly. They are less costly.
SSD ( Solid State drive )
Solid State drive is a new technology. It uses flash memory so they consume less energy and runs much faster as compared to HDD and is highly durable. It depends on electronic energy rather than mechanical energy so its easy to maintain and more efficient. They are more costly than HDD’s.
EBS are classified further into 4 types
General purpose SSD : Used in case of general use such as booting a machine or test labs.
Provisioned IOPS SSD: Used in case of scalable and high IOPS applications.
Throughput Optimized HDD: These are low cost magnetic storage which depends on throughput rather than IOPS such as EMR, data warehouse.
Cold HDD: These are also low cost magnetic storage which depends on throughput rather than IOPS.
How to create AWS EBS manually in AWS account?
You must have AWS account to create AWS EBS. If you don’t have AWS account please create from AWS account or AWS Account
Go to AWS account and on the top search for AWS EC2 service
Click on Create volume
Now fill all the details such as type of volume, size , IOPS , Tags etc.
Now click on Create volume and verify
What are EBS Snapshots in AWS ?
We just discussed about EBS that is storage. There are high chances that you might require backup to keep yourself in safe position. So basically EBS snapshots are backup of EBS volumes. There is also a option to backup your EBS with point in time snapshot which are incremental backups and these gets stored in AWS S3. This helps in saving tons of mins by keeping the snapshots with only difference to what changed in previous backup.
How to create EBS snapshots?
Go to AWS EBS console
Choose the AWS EBS volume for which you wish to create Snapshot
Add the description and Tag and then click on Create Snapshot.
Verify the Snapshot
If you wish to create AWS EBS snapshots using AWS CLI , please run the command ( Make sure you have AWS CLI installed and if not then we have explained below in this tutorial.
aws ec2 create-snapshot --volume-id <vol-1234567890> --description "My volume snapshot"
Prerequisites
AWS account to create AWS IAM user. If you don’t have AWS account please create from AWS account or AWS Account
Windows 7 or plus edition where you will execute the shell script.
Python must be installed on windows machine which will be required by AWS cli. If you want to install python on windows machine follow here
You must have Git bash already installed on your windows machine. If you don’t have install from here
Code editor for writing the shell script on windows machine. I would recommend to use visual studio code on windows machine. If you wish to install visual studio on windows machine please find steps here
In this demo , we will use shell script to launch AWS IAM user. So In order to use shell scripts from your local machine that is windows you will require AWS CLI installed and configured. So First lets install AWS CLI and then configure it.
Install AWS CLI Version 2 on windows machine
Download the installed for AWS CLI on windows machine from here
Select I accept the terms and then click next button
Do custom setup like location of installation and then click next button
Now you are ready to install the AWS CLI 2
Click finish and now verify the AWS cli
Verify the AWS version by going to command prompt and type
aws --version
Now AWS cli version 2 is successfully installed on windows machine, now its time to configure AWS credentials so that our shell script connects AWS account and execute commands.
Configure AWS Credentials by running the command on command prompt
aws configure
Enter the details such as AWS Access key , ID , region . You can skip the output format as default.
Check the location on your system C:\Users\YOUR_USER\.aws file to confirm the the AWS credentials
Now, you’re AWS credentials are configured successfully.
How to Delete EBS Snapshots from AWS account using shell script
Now we have configured AWS CLI on windows machine , its time to create our shell script to delete EBS snapshots. In this demo we will delete two AWS EBS snapshots which already exists in AWS account. Lets get started.
Create a folder on your desktop and under that create file delete-ebs-snapshots.sh
#!/usr/bin/env bash
# To check if access key is setup in your system
if ! grep -q aws_access_key_id ~/.aws/config; then
if ! grep -q aws_access_key_id ~/.aws/credentials; then
echo "AWS config not found or CLI not installed. Please run \"aws configure\"."
exit 1
fi
fi
# To Fetch all the SNAPSHOT_ID with Tag Name=myEBSvolumesnapshot
SNAPSHOTS_ID=$(aws ec2 describe-snapshots --filters Name=tag:Name,Values="myEBSvolumesnapshot" --output text | cut -f 6)
echo $SNAPSHOTS_ID
# Using For Loop Delete all Snapshots with Tag Name=myEBSvolumesnapshot
for id in $SNAPSHOTS_ID; do
aws ec2 delete-snapshot --snapshot-id "$id"
echo "Successfully deleted snapshot $id"
done
Now open visual studio code and open the location of file delete-ebs-snapshots.sh and choose terminal as Bash
Now run the script
./delete-ebs-snapshots.sh
Script ran successfully , now lets verify if AWS EBS Snapshots with Tag Name=myEBSvolumesnapshot successfully got deleted by going on AWS account.
Conclusion
In this tutorial, we demonstrated what is an AWS EBS , AWS EBS Snapshots and learnt many things about storage types and learnt how to delete EBS snapshots using shell script on AWS step by step. AWS EBS is your pendrive for instances, always use it when necessary and share with other instances.
Hope this tutorial will help you in understanding the shell script and working with AWS EBS on Amazon cloud. Please share with your friends
Docker has been a vital and very important tool for deploying your web applications securely and because of light weighted technology and they way it works has captured the market very well. Although some of the steps are manual but after deployment docker make things look very simple.
But can we automate even before the deployment takes place? So here comes terraform into play which is a infrastructure as a code tool which automate docker related work such as creation of images, containers and service with few commands.
In this tutorial we will see what is docker , terraform and how can we use docker provider in terraform to automate docker images and containers.
Table of content
What is Docker?
What is Terraform?
How to Install terraform on ubuntu machine.
What is Docker provider?
Create Docker Image, containers and docker service using docker provider on AWS using terraform
Conclusion
What is docker ?
Docker is an open source tool for developing , shipping and running applications. It has ability to run applications in loosely isolated environment using containers. Docker is an application which helps in management of containers in a very smooth and effective way. In containers you can isolate your applications. Docker is quite similar to virtual machine but it is light weighted and can be ported easily.
Containers are light weighted as they are independent of hypervisors load and configuration. They directly connect with machines ie. hosts kernel.
Prerequisites
Ubuntu machine preferably 18.04 version + , if you don’t have any machine you can create a ec2 instance on AWS account
Recommended to have 4GB RAM
At least 5GB of drive space
Ubuntu machine should have IAM role attached with full access of ec2 instance or it is always great to have administrator permissions to work with terraform demo.
You may incur a small charge for creating an EC2 instance on Amazon Managed Web Service.
What is Terraform?
Terraform is a tool for building , versioning and changing the infrastructure. Terraform is Written in GO Language and the syntax language of configuration files is hcl which stands for HashiCorp configuration language which is much easier than yaml or json.
Terraform has been in use for quite a while now . I would say its an amazing tool to build , change the infrastructure in very effective and simpler way. It’s used with variety of cloud provider such as Amazon AWS, Oracle, Microsoft Azure , Google cloud and many more. I hope you would love to learn it and utilize it.
How to Install Terraform on Ubuntu 18.04 LTS
Update your already existing system packages.
sudo apt update
Download the latest version of terraform in opt directory
Install zip package which will be required to unzip
sudo apt-get install zip -y
unzip the Terraform download zip file
unzip terraform*.zip
Move the executable to executable directory
sudo mv terraform /usr/local/bin
Verify the terraform by checking terraform command and version of terraform
terraform # To check if terraform is installed
terraform -version # To check the terraform version
This confirms that terraform has been successfully installed on ubuntu 18.04 machine.
What is Docker provider in terraform?
Docker provider helps to connect with docker images and docker containers using docker API. So in case of terraform , we would need to configure docker provider so that terraform can work with docker images and containers.
There are different ways in which docker provider can be configured. Lets see some of them now.
unix:///var/run/docker.sock is a Unix socket the docker daemon listens on. Using this Unix socket you can connect with multiple docker images and containers.
This Unix socket is also used when containers need to communicate with docker daemon such as during the mount binding.
Create a Docker Image, container and service using docker provider on AWS using terraform
Let us first understand terraform configuration files before we start creating files for our demo.
main.tf : This file contains actual terraform code to create service or any particular resource
vars.tf : This file is used to define variable types and optionally set the values.
output.tf: This file contains the output of resource we wish to store. The output are displayed
terraform.tfvars: This file contains the actual values of variables which we created in vars.tf
provider.tf: This file is very important as we provide information to terraform regarding on which cloud provider it needs to execute the code
Now, In below demo we will create docker image , container , services using docker provider. Now Lets configure terraform files which are needed for this demo. In this demo we only need one file that is main.tf to start with.
main.tf
provider "docker" { # Create a Docker Provider
host = "unix:///var/run/docker.sock"
}
resource "docker_image" "ubuntu" { # Create a Docker Image
name = "ubuntu:latest"
}
resource "docker_container" "my_container" { # Creates a Docker Container
image = docker_image.ubuntu.latest # Using same image which we created earlier
name = "my_container"
}
resource "docker_service" "my_service" { # Create a Docker Service
name = "myservice"
task_spec {
container_spec {
image = docker_image.ubuntu.latest # Using same image which we created eearlier
}
}
endpoint_spec {
ports {
target_port = "8080"
}
}
}
Now your files and code are ready for execution . Initialize the terraform
terraform init
Terraform initialized successfully ,now its time to run the terraform plan command.
Terraform plan is a sort of a blueprint before deployment to confirm if correct resources are being provisioned or deleted.
terraform plan
NOTE:
If you intend to create Docker service on same machine without having multiple nodes please run below command first so that our docker service gets created successfully using terraform
docker swarm init
After verification , now its time to actually deploy the code using apply.
terraform apply
Now lets verify all the three components one by one if they are created successfully using terraform with docker provider.
Verify docker image
docker images
Verify docker containers
docker ps -a
Verify docker service
docker service ps
Conclusion:
In this tutorial we will see what is docker , terraform and how can we use docker provider in terraform to automate docker images and containers.
Hope this tutorial will help you in understanding the terraform and provisioning the docker components using terraform. Please share with your friends if you find it useful.
With rise in number of database it has become a big challenge to make the right selection. As data grows our database should also scale and perform equally well.
Now Organizations have started to move toward big data and working with real time applications we certainly need a non relational and a good performance database. For these types of challenges and work AWS has always been on the top and served various services which solves our problems and one such service is AWS DynamoDB which manages non-relational databases for you and can store unlimited data and perform very well. .
Table of content
What is Relational database management system ?
What is SQL and NO SQL database?
What is Amazon DynamoDB ?
Prerequisites
How to Create tables in DynamoDB in AWS Account
Conclusion
What is Relational database management system ?
Relational database is based on tables and structured data
They have relationships which are logically connected.
Oracle Database, MySQL, Microsoft SQL Server, and IBM DB2. PostgreSQL , SQLite (for mobiles) are few example of RDMS.
Figure shows Relational Database Management System based on relational model
What is SQL and NO SQL database?
SQL:
The full form of SQL is structured query language which is used to manage data in relational database management system i.e RDMS.
SQL database belongs to the relational database management system.
The SQL type database follow structure pattern that’s why they are suitable for static or predefined schemas.
They are good in solving complex queries and highly scalable in nature but in vertical direction.
SQL database follows table based methodology and that’s the reason they are good for applications such as accounting systems.
NoSQL:
The full form of NoSQL is non-sql or non-relational.
This database is used for dynamic storage or those kind of managements where data is not fixed or static
This database is not tabular in nature rather its a key pair values.
They are good for big data and real time web application and scalable in nature but in horizontal direction
Some of the NoSQL databases which are DynamoDB, Foundation DB, Infinity DB, MemcacheDB, , Oracle NoSQL Database, , Redis MongoDB, Cassandra, Scylla, HBase.
What is Amazon DynamoDB ?
DynamoDB is a NoSQL database service that means it is different from the relational database which consists of tables in tabular form. DynamoDB service has very fast performance and is very scalable. DynamoDB service is one of the AWS managed service where you don’t need to worry about capacity , workload , setup , configuration , software patches , replications or even cluster scaling.
With DynamoDB service you just need to create tables where you can add data or retrieve data otherwise DynamoDB takes care of everything else. If you wish to monitor your resources you can do it on AWS console.
Whenever there is a traffic or high request coming in DynamoDB scales up while maintaining the performance.
Basic components of Amazon DynamoDB
Tables: It stores data.
In below example we used a database table
Items: Items are present in table. You can store as many item you wish in a table.
In below example different Employee ID are items.
Attributes: Each items contains one or more attributes.
In below example office , designation and phone are attributes of EmployeeID.
To work with Amazon DynamoDB , applications will need API’s to communicate.
Control Plane: It allows you to create and manage DynamoDB tables.
Data lane: It allows you to perform actions on the data in DynamoDB tables.
Prerequisites
You should have AWS account with Full access permissions on DynamoDB . If you don’t have AWS account, please create a account from here AWS account.
How to Create tables in DynamoDB in AWS Account
Go to AWS account and search for DynamoDB on the top of the page.
Click on Create Table and then you need to Enter the name of the Table and primary Key
Now click on Organisation that is table name
Now click on Items
Add the list of items such address , designation and phone number.
Verify if table has required details.
So this was the first way to use AWS provided web service and directly start creating DynamoDB tables . The other way is to download it manually on your machine , setup and then create you’re tables . You can find the steps here
Conclusion
You should now have a basic knowledge about relational database management system and non relational. We also learned about Amazon DynamoDB which is NO SQL database . We also covered on how to create tables on Amazon DynamoDB service & store the data .
This tutorial consists of all the practical’s which were done on our lab server with lots of hard work and efforts. Please share the word if you like it and hoping you get benefit out of this tutorial.
Shell is a command line interpreter and a programming language, basically what ever you are executing on terminal of your Linux machine is a shell command. There are thousands of commands which are already inbuilt such as cat , cd , ls , kill , history or pwd. The shell provides variables, flow control constructs, scripts, and functions. It also allows you to pipe commands, substitute command , do conditional testing , iterations etc. Whatever scripts you run , commands you execute are executed on shell or commonly known as Unix Shell.
There are different types of Unix shell available:
Bourne shell (sh) which is present in /bin/sh or /usr/bin/sh
Korn shell (ksh) which is present in /bin/ksh or /usr/bin/ksh
Bourne Again shell (bash) which is present in /bin/bash or /usr/bin/bash
POSIX shell (sh)
C shell (csh)
TENEX/TOPS C shell (tcsh)
To check on which shell you’re
echo $SHELL
What is Bash?
Bash is a Unix shell and also a command line interpreter. It is also known as Bourne again shell . This is improved version of Bourne shell that “sh”. This is present in almost all the operating system. It is a default login shell mostly in all Linux distributions. Also it is default login shell in Apple macOS and Solaris. Bash process shell commands. In bash you write all your commands in text format and execute commands. When bash executes any commands from other files then they can called as shell scripts.
It also contains keywords , variable , functions etc. just like sh shell . It is very much similar to Bourne shell (sh) .Latest version is bash-5.1 which was released in 2020-12-07.
To check the location of bash , you can use command.
echo $BASH
What is Shell Scripting or Bash Scripting?
Shell Script is simply a text of file with various or lists of commands that are executed even on terminal or shell one by one. But in order to make thing little easier and run together as a group and in quick time we write them in single file and run it.
Main tasks which are performed by shell scripts are : file manipulation , printing text , program execution. We can include various environmental variables in script that can be used at multiple places , run programs and perform various activities are known as wrapper scripts.
A good shell script will have comments, preceded by a pound sign or hash mark, #, describing the steps. Also we can include conditions or pipe some commands to make more creative scripts.
When we execute a shell script, or function, a command interpreter goes through the ASCII text line-by-line, loop-by-loop, test-by-test, and executes each statement as each line is reached from the top to the bottom.
How to create Shell scripts and execute it ?
Now we will create a very simple script and execute it.
Create a directory under /opt directory
mkdir script-demo
Create a file myscript.sh
touch myscript.sh
Edit the file
vi myscript.sh
Paste the code as shown in code snippet
#!/bin/bash
# This is a comment
echo Hello World, its automateinfra.com here!
echo I am using $SHELL which is a default shell.
Let us go through the code
#! is known as Shebang which is a syntax for a bash script. You can ignore this if you run your script by adding prefix bash . For example bashmyscript.sh
Here #!/bin/bash or #!/usr/bin/bash declares a Bourne-Again (Bash) shell
Similarly for Bourne shell we can use #!/bin/sh or #!/usr/bin/sh declares a Bourne shell
# is a comment
echo is a command
Grant the execution permissions
chmod + x myscript.sh
Execute the script
./myscript.sh
Script has been executed successfully.
Basic fundamentals of Shell Scripting
Shell Scripts are case sensitive
To execute a function
function function-name
{
Commands
}
You can run your scripts with specific shells as
ksh myscript.sh
bash myscript.sh
csh myscript.sh
If you are running a script in a particular location you should provide absolute path and if you are running in same directory then use “./”
/home/ubuntu/myscript.sh # Complete path
./myscript # Run in same directory
Use of if loops
if [condition]
then
command
else
command
fi
Use of for loops
for condition
do
commands
done
To Create a variable we use “$” symbol and this substitutes the variable to a value.
a = 5
echo $a
The command-line arguments $1, $2, $3,…$9 are positional parameters, with $0 pointing to the actual command, program, shell script, or function and $1, $2, $3, …$9 as the arguments to the command
Let us know see Special variables
$0 is the filename of the current script.
$n These variables correspond to the arguments with which a script was invoked.
$# The number of arguments supplied to a script.
$* All the arguments are double quoted. If a script receives two arguments, $* is equivalent to $1 $2.
$@ All the arguments are individually double quoted. If a script receives two arguments, $@ is equivalent to $1 $2.
$? The exit status of the last command executed.
$$ The process number / process ID of the current shell.
$! The process number of the last background command.
Once both the steps are completed open Visual studio.
From the dropdown menu of terminals select default shell
Then you will see Git Bash and click on it.
Now type a command to test if bash script works
hello automateinfra.com
Conclusion
You should now have a very sound knowledge of what is shell , what is Bash shell and Shell Scripting or Bash scripting. . Also we discussed how to work create a shell script and what are basic fundamentals or points one should know to getting started on bash scripting. Finally we ran bash script in windows machine on Microsoft’s Visual Studio Code.
This tutorial consists of all the practical’s which were done on our lab server with lots of hard work and efforts.
Please share the tutorial with everyone if you like it and hoping you get benefit out of this tutorial.
One of the best version control that I have used so far for managing multiple repositories and files in a smooth and distributed . The open source tool which we are talking here is Git the best control version. Please follow along to know the more on on how to work with Git , setup Git on ubuntu and windows machine & best commands and setup guide.
This tutorial is going to add a lot of valuable knowledge on Git in your pockets, Stay tuned !!
Table of Content
Getting Started with Git
Create a new git repository using graphical interface on GitHub
Creating a new git repository using command line on ubuntu machine
Getting Started with Git Commands for beginners
Summary
Getting Started with Git
What is Repository ?
Repository is a place where you keep all your source code it could be project wise or you can create it for a particular technology or you can create even for a single file, it all depends on you and your requirement. Its always a good idea to provide the name of repository according to work which you are going to do. There are some cloud based source code tools which allows you to create your repositories such as bitbucket , GitLab, GitHub ,Subversion , SVN and Mercurial etc. where you can create your own repositories.
Create a newgit repository using graphical interface on GitHub
Before we create the first repository make sure you have GitHub account created . If you don’t have please click here and Sign up.
If you have access to it then open your browser and go to GitHub website from here and click on Sign in.
Now click on New button to create a new repository
Now, provide a suitable name of the repository and you may keep it as public open to world if you wish people can see your code else keep it private. We are keeping as public as this is a demo tutorial and finally select create repository.
Now your repository is ready to be used. Please make sure you below steps which we will use later in this tutorial.
create a new repository on the command line and
push an existing repository from the command line
Creating a new git repository using command line on ubuntu machine
You must have ubuntu machine preferably 18.04 version + and if you don’t have any machine you can create a ec2 instance on AWS account
Recommended to have 4GB RAM
At least 5GB of drive space
SSH into your ubuntu machine
Now create a folder under /opt directory
cd /opt
mkdir git-demo
cd git-demo/
Initialize your new repository
git init
Create a file inside the same directory using the command
echo "My first change" > adding-new-file.txt
Now check the status of git repository using the command
git status
Add the file in git repository using the command
git add .
Again check the status of git repository using the command
git status
Commit your changes in git repository using the command
git commit -m "MY FIRST COMMIT"
Add the remote repository which we created earlier as a origin.
You should now have a very sound knowledge of what Git is , how to create Git repository using graphical mode as well as using command line tool . Also we discussed how to work with Git repositories using different commands. This tutorial consists of all the practical’s which were done on our lab server with lots of hard work and efforts.
Please share the word if you like it and hoping you get benefit out of this tutorial.
Are you tired of taking backups of your multiple files and facing syncing errors of your files ? Do you have any central place where you can store your files with proper versions and history ? Most of us faced this issue of file sync issues or how to control you same files with some updates. Git is a version control which solves all these issues & has grown the popularity for the way it works .
In this tutorial we will discuss what is version control , what are types of version controls and how to install git a version controller on windows machine and then setup to working with it.
Table of Content
What is Git ?
How to Install git version 2.31.0 on Windows machine?
Three types of control versions
Setting up Git Bash on windows machine
Summary
What is Git ?
Git is a version control which means if you have lots of file and each of these files are updated with few changes from the previous one then you need some way to handle it and this is taken care very well by Git. Sometimes you need to revert the changes back to previous state or you need to check few things of a month back , those all things are maintained by version control very well.
Three types of control versions
Local control version – In this case you take files from version control database to your local machine it’s just like a copy from one folder to other. In this case no team member knows what you are doing.
Centralized control version – In this case you can checkout files from centralized location which is better than local control as team is aware about the project and what team members are doing but in case if server goes down nobody will have full backup as in complete repository data and can cause significant loss. Example: SVN, subversion
Distributed control version: This is version control which manages everything in distributed way that means when you clone any repository it takes all data of one repository and copies on your local machine. Git is among the distributed control version
How to Install git version 2.31.0 on Windows machine
Open your browser and click on the link here so that installation starts automatically.
Now Click on the downloaded Git-2.31.0-64-bit.exe and select YES
Follow along with me and click the Next button and in most of the case select default values which are already selected
Select the location on your system where you like to place GIT installation files.
It is better to add it on Desktop
Please select the recommended
Here you will be asked to choose SSH executable and select OpenSSH. This will allow us to work with SSH connections such as logging into Linux machine.
Use MinTTY as default
Finally Git is launched on your machine and GIT bash will be available to use.
Setting up Git Bash on windows machine
Let us verify is Open SSH is working from Git Bash.
Now, you can check the config file where you can modify some configurations if required and to check that run the command below.
git config --list --show-origin
You can modify the global name while using Git by using below command
Check all your settings by using command
git config --list
Summary
You should have a basic knowledge of what Git is and how it’s different from any centralized version control systems you may have been using previously. You should also now have a working version of Git on your system that’s set up with your personal identity.
Node.js is an open source JavaScript runtime environment. Now, what is JavaScript ? Basically JavaScript is a language which is used with other languages to create a web page and add some dynamic features such as roll over and graphics.
Node.js runs as a single process without wasting much of memory and CPU and never blocks any threads or process which is why its performance is very efficient. Node.js also allows multiple connections at the same time.
With the Node.js it has become one of the most advantage for JavaScript developer as now they can create any apps utilizing it as both frontend or as a backend.
Building applications that runs in the any browser is a completely different story than than creating a Node.js application although both uses JavaScript language.
What is docker ?
Docker is an open source tool for developing , shipping and running applications. It has ability to run applications in loosely isolated environment using containers. Docker is an application which helps in management of containers in a very smooth and effective way. In containers you can isolate your applications. Docker is quite similar to virtual machine but it is light weighted and can be ported easily.
Containers are light weighted as they are independent of hypervisors load and configuration. They directly connect with machines ie. hosts kernel.
Prerequisites
Ubuntu machine preferably 18.04 version + , if you don’t have any machine you can create a ec2 instance on AWS account
You may incur a small charge for creating an EC2 instance on Amazon Managed Web Service.
How to Install node.js on ubuntu machine
Update your system packages.
sudo apt update
Lets change the directory and Download the node js package
cd /opt
sudo apt install nodejs
Install node js package manager
sudo apt install npm
Verify the node js package installation
nodejs -v
Install Nodejs Express Web Framework
Install Nodejs Express Web Framework and initialize it
npm init
package.json which got created after initializing the Nodejs framework will have all the dependencies which are required to run. Let us add one dependency which is highly recommended.
npm install express --save
CREATE NODE.JS APPLICATION
main.js
var express = require('express') //Load express module with `require` directive
var app = express()
//Define request response in root URL (/)
app.get('/', function (req, res) {
res.send('Hello Welcome to Automateinfra.com')
})
app.listen(8081, function () {
console.log('app listening on port 8081!')
})
Now Run the Node.js application locally on ubuntu machine to verify
node main.js
Create a docker file for Node.js application
Docker file is used to create a customized docker images on top of basic docker image. It is a text file that contains all the commands to build or assemble a new docker image. Using docker build command we can create new customized docker images . Its basically another layer which sits on top of docker image. Using newly built docker image we can run containers in similar way.
Create a docker file and name it as Docker file . Keep this file also in same directory as main.js
FROM node:7 # Sets the base image
WORKDIR /app # Sets the working directory in the container
COPY package.json /app # copy the dependencies file to the working directory
RUN npm install # Install dependencies
COPY . /app # Copy the content of the local src directory to the working directory
CMD node main.js # Command to run on container start
EXPOSE 8081
Build a Docker Image
Now we are ready to build our new image . So lets build our image
docker build -t nodejs-image .
You should see the docker images by now.
docker images
Run the Nodejs applicationon a Container
Now run our first container using same docker image ( nodejs-image)
docker run -d -p 8081:8081 nodejs-image
Verify if container is successfully created.
docker ps -a
Great, You have dockerized Nodejs application on a single container .
Conclusion:
In this tutorial we covered what is docker , what is Nodejs and using Nodejs application created a application on docker engine in one of the containers.
Hope this tutorial will helps you in understanding and setting up Nodejs and Nodejs applications on docker engine in ubuntu machine.
You might be already aware of difference between low level and high level language. Buts lets quickly see what are main difference between the both. High level language such as c , c ++ and python are portable , human friendly, that is they are easier to debug and understand while writing the programs. They need either compiler or interpreter to run the programs.
On the other hand, Low level language such as machine language or assembly level language they are difficult to understand and are written in assembly language which are converted using assembler.
In case of Linux distributions Python programming language by default , but in windows it does not come with it. In this tutorial, we will learn what is python which is widely used in artificial intelligence, by astronauts, by developers for creating the apps. Learning and setting python on windows is a need of the hour. So lets get started with it.
Table of content
What is Python?
Prerequisites
How to install Python on windows machine?
A simple python code to start your day
Conclusion
What is Python?
Python is a high level , oops based , interactive and a general purpose scripting programing language. Python is very easier to understand and readable. It focuses on object over functions. Python is also a interpreted language because it converts codes in machine level code even before it runs. It works on variety of protocols such as https, ftp , smtp and many more. The latest version is 3.9.2 which was released on December 2020. Python works very well with most of the such as atom, notepad ++ , vim.
Prerequisites
Windows 7 + operating system with admin rights
Command Prompt which is already present in windows operating system
Now run the executable installer which got downloaded.
Next step will prompt you to select whether to Disable path length limit , you ignore it or disable it.
Now python should be installed on your windows machine.
Verify Python Was Installed On Windows
Now Navigate to command prompt and type python
Pip is basically python package manager which is used to fetch lots of other additional modules. Pip comes with most of pythons executable. To check if it is installed use
pip -V
To verify if python environmental variables was successfully added. Search for sysdm.cpl and click on edit environmental variables.
A simple python code to start your day
Create a simple file main.py on your desktop
mystring = " This is Author of Automateinfra and its time to start learning python"
mylist = [letter for letter in mystring] # This is function which iterates over each alphabet in mystring
print(mylist) # This will print the result of mylist
Navigate to your desktop location on command prompt and use below command to run your first program in python
python main.py
Great work ! python was installed successfully and we could run our first program of python quickly.
Conclusion
In this tutorial we covered basic difference between high level language and low level language. We also discussed what is python and how to install python on windows machine to get you started in organization scripting and automation work.
Hope this tutorial will helps you in understanding and setting up Python on Windows machine.
You might have heard solar energy that right very powerful source of energy. But here we will discuss about Solr the most widely and best search tool in industry. It’s very fast and popular open source tool based on Java. From this tool we can fetch any content or perform search activity in quick seconds.
In this tutorial you’ll install using Solr on Ubuntu 18.04 . You’ll then work with Solr to index a page and later retrive using search engine.
Table of Content
What is Solr
Prerequisites
How to Install Solr 8.2.0 on Ubuntu 18.04 LTS
Conclusion
What is Solr ?
Solr is very efficient tool when it comes to search and real time indexing. It is very optimized for high volume of internet traffic. It is very highly scalable and fault tolerant. It has internal monitoring of its own instances as it publishes its own data via JMX. It provides lots of Extensible Plugin which are used for search as well as for indexing.
Solr can be installed on both windows as well as Unix based distributions. You index your data in Solr i.e. upload or put using JSON, XML or CSV or using http & later you can retrieve it using get http.
Prerequisites
Ubuntu machine preferably 18.04 version + , if you don’t have any machine you can create a ec2 instance on AWS account
Recommended to have 4GB RAM
At least 5GB of drive space
Java version 8 or 8+ ( If Java is not installed please follow me to next step else skip it )
You may incur a small charge for creating an EC2 instance on Amazon Managed Web Service.
Install Java Version 11 on ubuntu 18.04 machine
sudo apt install default-jdk # Here we are installing Java Version: Java SE 11 (LTS)
java -version # To check the Installed Java Version
which java : # It will locate executable file location which is /usr/bin/java
whereis java # It will give the location of all the files related to Java
The Installation directory of Java is /usr/lib/jvm/java-1.8.0-openjdk-amd64 and this confirms that Java is successfully installed on our ubuntu 18.04 machine. Now, let us install Solr Engine.
How to Install Solr 8.2.0 on Ubuntu 18.04 LTS
Update your system packages.
sudo apt update
Lets change the directory and Download the Solr package
cd /opt
sudo wget https://archive.apache.org/dist/lucene/solr/8.2.0/solr-8.2.0.tgz
Your downloaded package will be in tarball format , Lets extract the tarball into the folder.
sudo tar xzf solr-8.2.0.tgz solr-8.2.0/bin/install_solr_service.sh --strip-components=2
Execute the the installer
sudo ./install_solr_service.sh solr-8.2.0.tgz
Now, Solr should be up and running, you may verify by checking the status of Solr service.
sudo service solr stop
sudo service solr start
sudo service solr status
Solr should be running now, lets verify by opening a browser and enter : <ip-address>:8983/solr
Conclusion
You should now have a Apache Solr 8.2.0 instance running . Your Solr should now be ready to begin helping you manage your organization’s search.
Hope this tutorial will help you in understanding and setting up Apache Solr on ubuntu machine. Please share with your friends.
Cannot we isolate our apps so that they are independent of each other and run perfectly ? The answer is absolutely “YES”, that correct that’s very much possible with docker and containers. They provide you isolated environment and are your friend for deploying many applications with each taking its own container. You can run as many as containers in docker and are independent of each other. They all share same kernel memory.
In this tutorial we will go through a simple demonstration of a python application which will run on docker engine.
Table of content
What is Python ?
What is docker ?
Prerequisites
Create a Python flask application
Create a Docker file
Build Docker Image
Run the Python flask application Container
Conclusion
What is Python ?
Python is a language from which you create web applications and system scripts. It is a used vastly across the organizations and very easy to learn. Python apps require isolated environment to run its application very well. This is quite possible with Docker and containers which we will use in this tutorial.
If you wish to know more about python please visit our Python’s Page to learn all about Python.
What is docker ?
Docker is an open source tool for developing , shipping and running applications. It has ability to run applications in loosely isolated environment using containers. Docker is an application which helps in management of containers in a very smooth and effective way. In containers you can isolate your applications. Docker is quite similar to virtual machine but it is light weighted and can be ported easily.
Containers are light weighted as they are independent of hypervisors load and configuration. They directly connect with machines ie. hosts kernel.
Prerequisites
Ubuntu machine preferably 18.04 version + , if you don’t have any machine you can create a ec2 instance on AWS account
Create and activate a virtual environment named virt:
virtualenv venv
source virt/bin/activate
Finally install Flask
pip install flask # Install Flask from pip
Now create a text file and name it as app.py where we will write our first python flask code as below.
from flask import Flask # Importing the class flask
app = Flask(__name__) # Creating the Flask class object.
@app.route('/') # app.route informs flask about the URL to be used by function
def func(): # Creating a function
return("Iam from Automateinfra.com")
if __name__ == "__main__": # Programs starts from here.
app.run(debug=True)
Create one more file in same directory and name it as requirements.txt where we will define the dependency of flask application
Flask==1.1.1
Now our python code app.py and requirements.txt are ready for execution. Lets execute our code using below command.
python app.py
Great, so our python flask application ran successfully on our local machine. Now we need to execute same code on docker . Lets now move to docker part.
Create a docker file
Docker file is used to create a customized docker images on top of basic docker image. It is a text file that contains all the commands to build or assemble a new docker image. Using docker build command we can create new customized docker images . Its basically another layer which sits on top of docker image. Using newly built docker image we can run containers in similar way.
Create a docker file and name it as Docker file . Keep this file also in same directory as app.py and requirements.txt
FROM python:3.8 # Sets the base image
WORKDIR /code # Sets the working directory in the container
COPY requirements.txt . # copy the dependencies file to the working directory
RUN pip install -r requirements.txt # Install dependencies
COPY src/ . # Copy the content of the local src directory to the working directory
CMD [ "python", "./app.py" ] # Command to run on container start
Build docker Image
Now we are ready to build our new image . So lets build our image
docker build -t myimage .
You should see the docker images by now.
docker images
Run the Python flask application Container
Now run our first container using same docker image ( myimage)
docker run -d -p 5000:5000 myimage
Verify if container is successfully created.
docker ps -a
Conclusion
In this tutorial we covered what is docker , what is python and using python flask application created a application on docker engine in one of the containers.
Hope this tutorial will helps you in understanding and setting up Python flask and python flask applications on docker engine in ubuntu machine.
