Reading and Writing a YML file using python
yaml and yml files are superset of JSON. Some of the automation tools such as ansible uses yaml based files, referred to as playbooks, to define actions you want to automate. These playbooks use the YAML format.
Working with yaml files is a fun in python , so lets get started but In order to work with yaml files in python you would require to install a PyYAML library as Python doesn’t contain standard library. PyYAML is a YAML parser and emitter for Python.
- Run the following command to install
PyYAML libraryin your favorite code editor terminal such as visual code studio.
pip install PyYAML- Next, create a folder with a name
Pythonand under that create a simple YML file and name it as apache.yml and paste the below content and save it.
---
- hosts: webservers
vars:
http_port: 80
max_clients: 200
remote_user: root
tasks:
- name: ensure apache is at the latest version
yum:
name: httpd
state: latest
- Next, create another file in same
Pythonfolder and name it as read_write_yaml.py and paste the below python code.
Below Python script imports yaml module to work with yaml files and pprint module to get a output in well designed pattern. Next, using open() function it opens the apache.yml file and reads the data using yaml.safe_load() method. Later, using yaml.dump() you can add or write the data into it. As we are not adding any data into it , the output would result as NONE
import yaml
from pprint import pprint
with open('apache.yml', 'r') as new_file:
verify_apache = yaml.safe_load(new_file)
pprint(verify_apache)
with open('apache.yml', 'w') as new_file2:
verify_apache2 = yaml.dump(verify_apache, new_file2)
pprint(verify_apache2)
- Execute the above python script using python command and you should see the below output.
[{'hosts': 'webservers',
'remote_user': 'root',
'tasks': [{'name': 'ensure apache is at the latest version',
'yum': {'name': 'httpd', 'state': 'latest'}}],
'vars': {'http_port': 80, 'max_clients': 200}}]
None
Reading and Writing a XML file using python
XML files used mostly for structured data. Many web system uses XML to transfer data and one of them is RSS ( Real Simple Syndication) feeds which helps in finding the latest updates on websites from various sources. Python offers XML Library.
- Next, in the same
Pythonfolder create a simple XML file and name it as book.xml and paste the below content and save it. XML has a tree like structure and top element is known as root and rest of them are elements
<?xml version="1.0"?>
<catalog>
<book id="bk109">
<author>Author1</author>
<title>Automate Infra Part 2</title>
<genre>Science Fiction</genre>
<price>6.95</price>
<publish_date>2000-11-02</publish_date>
<description>book1</description>
</book>
<book id="bk112">
<author>Author2</author>
<title>Automate Infra Part 1</title>
<genre>Computer</genre>
<price>49.95</price>
<publish_date>2001-04-16</publish_date>
<description>book2</description>
</book>
</catalog>
python folder and name it as read_write_xml.py and paste the below python code.In below script, importing xml.etree.ElementTree module helps to work with xml files and implements a simple and efficient API for parsing and creating XML data. Next, In XML file entire tree is parsed that is it reads the book.xml file and then prints the content inside it.
import xml.etree.ElementTree as ET
tree = ET.parse('book.xml') # checking each elements
root = tree.getroot() # finding the root
print(root.tag,root.attrib)
for child in root: # Each child and its attributes
print(child.tag,child.attrib)
O/P:
catalog {}
book {'id': 'bk109'}
book {'id': 'bk112'}
Reading and Writing a comma-separated values (CSV) file using python
CSV is most widely used spreadsheets. To work with these file in python you need to import the csv module. Lets learn how to read and write data into CSV.
Python folder create a CSV file and name it as devops.csv and add the content similar to below in your file and save it. 
- Next, create another file in same
pythonfolder and name it as read_write_csv.py and paste the below python code.
Below script uses csv module to work with csv files. As soon as python script is executed , open() function opens the csv file and then using csv.reader() it reads it and then prints the rows according to the defined range.
import csv
with open('devops.csv' , 'r') as csv_file:
read = csv.reader(csv_file, delimiter=',')
for _ in range(5):
print(next(read))
print(read)
- Execute the above python script using python command and you should see the below output.
OUTPUT
['Date', ' PreviousUserCount', ' UserCountTotal', ' sitepage']
['02-01-2021', '61', '5336', ' automateinfra.com/blog']
['03-01-2021', '42', '5378', ' automateinfra.com/blog1']
['04-01-2021', '26', '5404', ' automateinfra.com/blog2']
['05-01-2021', '65', '5469', ' automateinfra.com/blog3']
<_csv.reader object at 0x0336A370>
Python – Pandas (Data Analysis 3rd Party Library)
pandas.DataFrame, which acts like a data table, similar to a very powerful spreadsheet. If you want to work on something like row or column in Spreadsheet then DataFrames is the tool for you. So lets get started by installing pip install pandas
import pandas as pd
df = pd.read_csv('devops.csv')
print(type(df))
print(df.head(4)) # Seeing TOP 4 rows in devops.csv file
print(df.describe()) # Statical View
O/p:
<class 'pandas.core.frame.DataFrame'>
Date PreviousUserCount UserCountTotal sitepage
0 02-01-2021 61 5336 automateinfra.com/blog
1 03-01-2021 42 5378 automateinfra.com/blog1
2 04-01-2021 26 5404 automateinfra.com/blog2
3 05-01-2021 65 5469 automateinfra.com/blog3
PreviousUserCount UserCountTotal
count 4.000000 4.000000
mean 48.500000 5396.750000
std 18.046237 55.721779
min 26.000000 5336.000000
25% 38.000000 5367.500000
50% 51.500000 5391.000000
75% 62.000000 5420.250000
max 65.000000 5469.000000
PYTHON : Regular Expressions to Search Text ( * MOSTLY USED AND IMPORTANT)
BEST TWO EXAMPLE OF SEARCHING [Can be used for Different Practices such as Analysis, HR, Sales Team and many more ]
name_list = '''Ezra Sharma <esharma@automateinfra.com>,
...: Rostam Bat <rostam@automateinfra.com>,
...: Chris Taylor <ctaylor@automateinfra.com,
...: Bobbi Baio <bbaio@automateinfra.com'''
# Some commonly used ones are \w, which is equivalent to [a-zA-Z0-9_] and \d, which is equivalent to [0-9].
# You can use the + modifier to match for multiple characters:
print(re.search(r'Rostam', name_list))
print(re.search('[RB]obb[yi]', name_list))
print(re.search(r'Chr[a-z][a-z]', name_list))
print(re.search(r'[A-Za-z]+', name_list))
print(re.search(r'[A-Za-z]{5}', name_list))
print(re.search(r'[A-Za-z]{7}', name_list))
print(re.search(r'[A-Za-z]+@[a-z]+\.[a-z]+', name_list))
print(re.search(r'\w+', name_list))
print(re.search(r'\w+\@\w+\.\w+', name_list))
print(re.search(r'(\w+)\@(\w+)\.(\w+)', name_list))
O/P:
<re.Match object; span=(49, 55), match='Rostam'>
<re.Match object; span=(147, 152), match='Bobbi'>
<re.Match object; span=(98, 103), match='Chris'>
<re.Match object; span=(0, 4), match='Ezra'>
<re.Match object; span=(5, 10), match='Sharm'>
<re.Match object; span=(13, 20), match='esharma'>
<re.Match object; span=(13, 38), match='esharma@automateinfra.com'>
<re.Match object; span=(0, 4), match='Ezra'>
<re.Match object; span=(13, 38), match='esharma@automateinfra.com'>
<re.Match object; span=(13, 38), match='esharma@automateinfra.com'>
# <IP Address> <Client Id> <User Id> <Time> <Request> <Status> <Size>
Line1 = '127.0.0.1 - Automateinfra1 [13/Nov/2021:14:43:30 -0800] "GET /assets/234 HTTP/1.0" 200 2326'
access_log = '''
127.0.0.1 - Automateinfra1 [13/Nov/2021:14:43:30 -0800] "GET /assets/234 HTTP/1.0" 200 2326
127.0.0.2 - Automateinfra2 [13/Nov/2021:14:43:30 -0800] "GET /assets/235 HTTP/1.0" 200 2324
127.0.0.3 - Automateinfra3 [13/Nov/2021:14:43:30 -0800] "GET /assets/236 HTTP/1.0" 200 2325
'''
count_ip = r'(?P<IP>\d+\.\d+\.\d+\.\d+)'
count_time = r'(?P<Time>\d\d/\w{3}/\d{4}:\d{2}:\d{2}:\d{2})'
count_clientid = r'(?P<User>".+")'
count_request = r'(?P<Request>".+")'
sol = re.search(r'(?P<IP>\d+\.\d+\.\d+\.\d+)', Line1 )
print(sol.group('IP'))
print(re.search(count_request , Line1))
print(re.search(count_time , Line1))
value = re.finditer(count_ip, access_log)
for sol in value:
print(sol.group('IP'))
O/P:
127.0.0.1
<re.Match object; span=(56, 82), match='"GET /assets/234 HTTP/1.0"'>
<re.Match object; span=(28, 48), match='13/Nov/2021:14:43:30'>
127.0.0.1
127.0.0.2
127.0.0.3
PYTHON: DEAL With Large Files (FILE BREAKER and LINE BREAKER)
Rather than loading the whole file into memory as you have done up until now, you can read one line at a time, process the line, and then move to the next. The lines are removed from memory automatically by Python’s garbage collector, freeing up memory.
# LINE BREAKER
with open("devops.txt",mode="r") as mynewfile: # if you open any binary file such as pdf keep w as wb
with open("devops-corrected.txt", "w") as target_file:
for line in mynewfile:
print(target_file.write(line))
o/p:
Automateinfra.com
automateinfra.com/blog
automateinfra.com/solutions
# FILE BREAKER with chunk of data with number of bytes
with open('book.xml' , 'rb') as sourcefile:
while True:
chunk = sourcefile.read(1024) # break down in 1024 bytes
if chunk:
print(chunk)
else:
break
O/P:
b'<?xml version="1.0"?>\r\n<catalog>\r\n <book id="bk109">\r\n <author>Author1</author>\r\n <title>Automate Infra Part 2</title>\r\n <genre>Science Fiction</genre>\r\n <price>6.95</price>\r\n
<publish_date>2000-11-02</publish_date>\r\n <description>book1</description>\r\n </book>\r\n <book id="bk112">\r\n <author>Author2</author>\r\n <title>Automate Infra Part 1</title>\r\n <genre>Computer</genre>\r\n <price>49.95</price>\r\n <publish_date>2001-04-16</publish_date>\r\n <description>book2</description>\r\n </book>\r\n</catalog>'
PYTHON ENCRYPTION: MOST IMPORTANT TOPIC OF PYTHON FILE SYSTEM
There are many times you need to encrypt text to ensure security. In addition to Python’s built-in package hashlib, there is a widely used third-party package called cryptography
HASHLIB: Uses Hash Function and based on SHA1, SHA224, SHA384, SHA512, and RSA’s MD5 Algorithms
CRYPTOGRAPHY:
symmetric key encryption: Its based on shared keys. These algorithms include Advanced Encryption Algorithm (AES), Blowfish, Data Encryption Standard (DES), Serpent, and Twofish
asymmetric key encryption: Its based on public keys ( which are widely shared ) and private keys which is kept secretly
# Encryption using HashLib
import hashlib # Python Built in Package
line = "I like editing automateinfra.com"
bline = line.encode() # Converting into Binary string
print(bline) # Print the converted Binary string
algo = hashlib.md5() # Using the secure alogorithm using haslib object
algo.update(bline) # Applying the secure alogorithm
print("Encrypted text Message")
print(algo.digest()) # Print the Encypted string
# Encryption using Cryptography (Symmetric key encryption)
from cryptography.fernet import Fernet # Third Party Package So you would need pip install cryptography
key = Fernet.generate_key() # Generating the keys
print("Generating the keys ")
print(key) # Prining the keys
algo = Fernet(key) # Using the key AES alogo using Fenet object
message = b"I definetely like Editing AutomateInfra.com"
encrypted = algo.encrypt(message)
print("Encrypted text Message ")
print(encrypted)
print(algo.decrypt(encrypted))
# Encryption using Cryptography (ASymmetric key encryption)
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import padding ,rsa
from cryptography.hazmat.primitives import hashes
private_key = rsa.generate_private_key(public_exponent=65537,key_size=4096,backend=default_backend()) # Generating the Private Key
print(private_key) # Printing the Private Key
public_key = private_key.public_key() # Generating the Public Key
print(public_key) # Printing the Public Key
message = b"I am equally liking Editing AutomateInfra.com"
encrypted = public_key.encrypt(message,padding.OAEP(mgf=padding.MGF1(algorithm=hashes.SHA256()), algorithm=hashes.SHA256() , label=None))
print(encrypted)
decrypted = private_key.decrypt(encrypted,padding.OAEP(mgf=padding.MGF1(algorithm=hashes.SHA256()), algorithm=hashes.SHA256(), label=None))
print(decrypted)
O/P:
b'I like editing automateinfra.com'
Encrypted text Message
b'v\x84*\xe55\x01\xa4z\x05\xa2\xa2\xdb\xd1y\xa9\x07'
Generating the keys
b'7trCiXpGuCfEnXoIcsFfCGOw-u_Qkas0tv1lBM8xmQo='
Encrypted text Message
b'gAAAAABgAJcPFj-aGttg8MRJQfRYGWyOWy44u-cLWGuDhqoyyvYP1uG4oQYms8BQMr4eExpv74LIZESGvpIUY88fE0_YQCQ32JH0DZsabLOAtc00QCwV8L51WktRjzUab0Fp3jnbOeb2'
b'I definetely like Editing AutomateInfra.com'
<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x036491D8>
<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x03850E38>
b"\x8b\xec\xb0\x91\xec\xe7\x8d;\x11\xbclch\xbdVD@c\xd3J\x07'\xe9\x07\x15\x1c@=^\xd2h\xcaDL\x95\xea[\x0fv\x012\xed\xd5\xed\x0e\x9b\x93V2\x00\xba\x9c\x07\xba\x8b\xf3\xcb\x03M\xa8\xb1\x12ro\xae\xc0\xfb$\xf9\xcc\x85\xe8s\xfc`{\xfe{\x88\xd2\xc3\xffI\x90\xe3\xd2\x1e\x82\x95\xdfe<\xd5\r\x0b\xc4z\xc4\xf7\x00\xcfr\x07npm0\xd4\xc4\xa4>w\x9d]\xcf\xae7F\x91&\x93\xd5\xda\xcaR\x13A\x8ewB\xf6\xd9\xae\xce\xca\x8f\xd6\x91\x06&:\x00\xa0\x84\x05#,\x7fdA\x87\xb2\xe7\x1d\x8b*\xa15\xf8\xb0\x07\xa0n\x1e\xeaI\x02\xbaA\x88ut\x8e\x82<\xfe\xbfM\xe6F\xa3\xcc\xd4\x8b\x80PY\xb5\xd3\x14}C\xe2\x83j\xaf\x85\xa6\x9e\x19\xb2\xd9\xb8\xac\xa4\xfb\x1f\x0c\xce\x9d4\x82\x1e\xfd5\xb49\xa5\xbbL\x01~\x8fA\xee\r\xc7\x84\x9e\x0c\t\x15z\r\xfd]\x0b\xcfW\x01\xd2\x16\x17btc\xeaSl\xf5\xb0\x8a\xe2X\xe7\xa7a\xa7\xf7M\x01\xa2\x0b8\xd6\xf2\xc5c\xbf\xea\xe0\x80\x15\xde-\x98\xa1\xc8ud*\xbel2\xb5\xc8:\x92\xd5\r(_8\xbd\xcb\x80\xf1\x93\x83\xe2\x9f\xed\x82f\xd0\xb2\x8f\x1b\x9eMC\x07\xf9\x08\xb0\x00QA\xea\x93\xc7@&\x84\xff<\xde\x80@\xc8\xc6\x83O&%\x91r-\xb0\xef}\x18tU{C\xa6\x17\x97\x1b\x95g\xc5\x0e>{\xb0\x94a)\xbc)*Sq\x98\xad\xf3>\x04\x9b+x\x95&\xa6\xe6,\xb4~\xf2Y\x06,\xab'uq \x9f0\x7f\xb5\xd50\xbdp\xbb\xdf\x1c\xe9\xb1\xc4\x88y\nq\\\x85\x1e\xd8\x18M\x87\x1aU.\x918;\xcd\x10 \x9b\x11\xf9R\xd3\x8fz\xe8\xf6|C\xfb\x1f\xfd1\x19\x10:>\x1c\x06\x8e\xda\x98\xb2\xf3aa^\xa54\x03\xf8\x03\xc4\xe6\xd9mw\r\x8b\x96\xa2rJ\x03\xe7\xda\x0f\rJ-iPo!^\x8a\xdcg\x8c!L\xa4\xedY\xe5\x12\xdf\xe8\xe7\x0cE\xcd\xa2\xa2Gr\xc0\xe1\xa6\xc5\x9a\x9f\x07\x89\x84\x8b\xb7"
b'I am equally liking Editing AutomateInfra.com'
PYTHON OS MODULE:
This module will help to connect with many low level operating system calls and offers connectivity between multi -OS like Unix and Windows.
import os # Python Built in Package
print(os.listdir('.')) # List the directories
os.rename('automateinfra.txt','automateinfra_backup.txt')
os.chmod('automateinfra.txt',777) # Add the permissions to the file.
os.mkdir('/tmp/automateinfra.pdf') # Make the directory
os.rmdir('/tmp/automateinfra.pdf') # remove the directory
os.stat('b.txt') # These stats include st_mode, the file type and permissions, and st_atime, the time the item was last accessed.
cur_dir = os.getcwd() # Get the current working directory.
print(os.path.dirname(cur_dir)) # Returns the Parent Directory Path
print(os.path.split(cur_dir)) # Gives structure from Parent Directory
print(os.path.basename(cur_dir)) # Returns Base Directory
while os.path.basename(cur_dir): # Until Base Path directory is true , keep continuing
cur_dir = os.path.dirname(cur_dir) # Prints the base directory and all above parents Directory
print(cur_dir)
O/P:
C:\Users\AutomateInfra\Desktop\GIT\Python-Desktop
('C:\\Users\\AutomateInfra\\Desktop\\GIT\\Python-Desktop', 'Basics')
Basics
C:\Users\AutomateInfra\Desktop\GIT\Python-Desktop
C:\Users\AutomateInfra\Desktop\GIT
C:\Users\AutomateInfra\Desktop
C:\Users\AutomateInfra
C:\Users
C:\
import os
# Check the current working directory
file_name = "automateinfra.txt"
file_path = os.path.join(os.getcwd(), file_name)
print(f"Checking {file_path}")
if os.path.exists(file_path):
print(file_path)
# Check user home directory
home_dir = os.path.expanduser("~/") #expanduser function to get the path to the user’s home directory.
file_path = os.path.join(home_dir,file_name)
print(f"Checking {file_path}")
if os.path.exists(file_path):
print(file_path)
o/p:
Checking C:\Users\Automateinfra\Desktop\GIT\Python-Desktop\Basics\automateinfra.txt
C:\Users\Automateinfra\Desktop\GIT\Python-Desktop\Basics\automateinfra.txt
Checking C:\Users\Automateinfra/automateinfra.txt
DevOps Automateinfra Learning 