Build A Test Automation Framework with me from Scratch, with Pytest, Python Selenium, Page Object Model and HTML Reports
In my previous blog, I was already talking about the test automation objectives, advantages, disadvantages, and as well as its limitations. If haven't seen it yet I encourage you to go through it first through this LINK before we start to build our first test automation framework we should first be aware of the dos and don'ts to the test automation project.
Keywords
TAF - Test Automation Framework
SUT - System Under Test
TAS - Test Automation Solution
Test Automation Framework
A test automation framework (TAF) that is easy to use, well documented, and maintainable support a consistent approach to automated test.
Dos
In order to establish an easy-to-use and maintainable TAF, the following must be done.
Implement reporting facilities. The test reports should provide information such as pass/fail/error/not run/aborted about the quality of the SUT. Reporting should provide information for the involved testers, test managers, developers, project managers, and other stakeholders to obtain an overview of the quality.
Enable easy troubleshooting. The test can fall due to failures found in the SUT, failures found in the TAS, problems with the tests themselves, or the test environment.
Address the test environment appropriately. If there is no control of the test environment and test data, test setup for tests may not meet the requirements for the test execution and it is likely to produce false execution results.
Document the automated test cases. The goals for the test automation have to be clear, e.g., which parts of the application are to be tested, to what degree, and which attributes are to be tested (functional and non-functional). This must be clearly described and documented.
Trace the automated test. TAF shall support tracing for the test automation engineer to trace the individual steps to test cases.
Enable easy maintenance. Ideally, the automated test cases should be easily maintained so that maintenance will not consume a significant part of the test automation effort. In addition, the maintenance effort needs to be in proportion to the scale of the changes made to the SUT. To do this, the cases must be easily analyzable, changeable, and expandable. Furthermore, automated testware reuse should be high to minimize the number of items requiring changes.
Keep the automated tests up-to-date. When new or changed requirements cause tests or entire test suites to fail, do not disable the failed tests - fix them.
Plan for deployment. Make sure that test scripts can be deployed, changed, and redeployed.
Retire tests as needed. Make sure that test scripts can be easily retired if they are no longer useful or necessary.
Monitor and restore the SUT. In real practice, to continually run a test case or set of test cases, the SUT must be monitored continuously, If the SUT encounters a fatal error (such as a crash), the TAF must have the capability to recover, skip the current case, and resume testing with the next case.
** The test automation code can be complex to maintain. It is not unusual to have as code for testing as the code for the SUT.**
Don'ts
In addition to the important items that should be done, there are a few that should not be done, as follow:
Do not create code that is sensitive to the interface (e.g., it would be affected by changes in the graphical interface or in non-essential parts of the API).
Do not create test automation that is sensitive to data changes or has a high dependency on particular data values.
Do not create an automation environment that is sensitive to the context (e.g., operating system date and time, operating system localization parameters, or the contents of another application).
The more success factors that are met, the fewer don'ts factor that is avoided, and the more likely the test automation project will succeed. Not all factors are required, and in practice rarely are all factors met.
Reference: The above theories were quoted from the International Software Testing Qualifications Board Advanced Level Syllabus: Test Automation Engineer version 2016
Get Started
We are going to build a Test Automation Framework from scratch with the help of Pytest, Python Selenium, and a few other python libraries.
Installation
Python Libraries
Install requires python lib
pip install selenium
pip install pytest
pip install openpyxl
pip install pytest-html
Setup Browser Driver
I am running on Linux Operating System, so the automation will be running on chrome or firefox. For Mac users, it is the same. LINK to install the browser driver. It is important to install the driver version matched with the browser version you are using (e.g., if you are using Google Chrome version 105 make sure to install the chrome driver version 105 either)
After the driver is installed, unzip it and add the driver path to Environment Variable PATH.
Linux and Mac
# cd into driver installed location
chmod +x chromedriver
mv chromedriver /usr/bin/chromedriver
# run chromedriver cmd to verify setup
chromedriver
Window
Create directory
C:\binDownload driver for window, extract and save it to
C:\binOpen Command Prompt and set the PATH with following cmd
setx PATH "C:\bin;%PATH%"
Restart Command Prompt
Verify setup with
chromedriver.exe -v
After everything is set up successfully it should be ready to build our framework. We are going to perform our testing on Guru99 Bank live project with our framework.
Framework Structure
Create a Project inside PyCharm IDE or whatever IDE you use and put the following Package and Folder inside your Project Folder.
Note: If you use other IDE besides PyCharm for every Package you need to create another empty __init__.py file inside the Package.
Note: You can use "Python Package" when you want to put some modules in there which should be importable. PyCharm will automatically create an _init_.py for the directory.
After done, we should have a project's folders like these:
PageObject Package
PageObject package is used to store each element locator and its basic action of the webpage. Our first pageObject is LoginPage:
Create LoginPage.py file under pageObjects package. Get each element locator on the LoginPage define it as variable in a LoginPage class object.
# import selenium libray
from selenium.webdriver.common.by import By
# loginPage Class Object
class LoginPage:
# userID field element locator
textbox_userId_xpath = '/html/body/form/table/tbody/tr[1]/td[2]/input'
# userID error message element locator
msg_userID_xpath = '//*[@id="message23"]'
# password field element locator
textbox_password_xpath = '/html/body/form/table/tbody/tr[2]/td[2]/input'
# password error message element locator
msg_password_xpath = '//*[@id="message18"]'
# login button element locator
btn_login_xpath = '/html/body/form/table/tbody/tr[3]/td[2]/input[1]'
# reset button element locator
btn_reset_xpath = '/html/body/form/table/tbody/tr[3]/td[2]/input[2]'
################### Constructor ###################
def __init__(self, driver):
self.driver = driver
################### Basic action on each element ###################
# input userId in userID field
def enter_userID(self, userID):
self.driver.find_element(By.XPATH, self.textbox_userId_xpath).send_keys(userID)
# get error message of userID
def get_userID_error_message(self):
actual_msg = self.driver.find_element(By.XPATH, self.msg_userID_xpath).text
return actual_msg
# input password in Password field
def enter_password(self, password):
self.driver.find_element(By.XPATH, self.textbox_password_xpath).send_keys(password)
# get error message of password
def get_password_error_message(self):
actual_msg = self.driver.find_element(By.XPATH, self.msg_password_xpath).text
return actual_msg
# click login button
def click_login_btn(self):
self.driver.find_element(By.XPATH, self.btn_login_xpath).click()
# click reset button
def click_reset_btn(self):
self.driver.find_element(By.XPATH, self.btn_reset_xpath).click()
Now we have a basic loginPage object class, let's create our first test case.
testCases Package
Setup Test Environment
For the basic understanding, we will setup our test environment (browser) to Google Chrome only. Will modify our code to add more browsers later.
To SetUp the test environment:
Under testCases Package, create a python file called conftest.py and add the following code:
from selenium import webdriver
import pytest
@pytest.fixture()
def setup():
driver = webdriver.Chrome()
return driver
Create first test case
Our first test case is "Login with valid userID and Password": Under testCases Package, create another python file called test_login.py, and add the following code:
# import our loginPage object we created under pageObject
from pageObjects.LoginPage import LoginPage
from time import sleep
# create a login test class
class Test_001_Login:
url = "https://www.demo.guru99.com/V4/"
# test case 01
def test_login_with_valid_credential(self, setup):
# start browser and go to Guru99 Bank
self.driver = setup
self.driver.get(self.url)
sleep(0.3)
# create an object of Login Page for access its method
self.lp = LoginPage(self.driver)
# enter valid userid
self.lp.enter_userID("mngr445105")
# enter valid password
self.lp.enter_password("udasEgA")
# click login button
self.lp.click_login_btn()
# get page title for our test check point
actual_page_title = self.driver.title
# our successful login page title would be "Guru99 Bank Manager HomePage"
expect_page_title = "Guru99 Bank Manager HomePage"
if actual_page_title == expect_page_title:
assert True # passed the test case
self.driver.close() # close webdriver
else:
self.driver.close()
assert False # failed the test case
Note: Test class name has to start with word Test with "T" capitalize
test case name has to start with word test with "t" lower case.
Execute test case
To run the test case, open Terminal or Command Prompt then type: pytest follow by test case file
Example:
pytest test_login.py
Example:
We have successfully executed our first test case, now let's add another negative test case to our code, "Login with invalid credential" and we expect to an error message alert.
Handling Javascript Alert Message
If you explore the Guru99 Bank a little bit more you will see that when user try to login with an invalid userID or Password it will popup a javascript alert box. So here is how to handle it:
1 - Import from selenium.webdriver.common.alert import Alert in LoginPage.py.
2 - Add function to get alert text, accept alert and dismiss alert in our LoginPage.py as following:
from selenium.webdriver.common.alert import Alert
# get actual message from alert popup
def get_alert_message(self):
alert = Alert(self.driver)
return alert.text
# accept the popup alert
def accept_alert(self):
alert = Alert(self.driver)
alert.accept()
# dismiss the popup alert
def dismiss_alert(self):
alert = Alert(self.driver)
alert.dismiss()
Add "Login with invalid credential" test case under our previous test case in test_login.py
# test case 02
def test_login_with_invalid_credential(self, setup):
# start browser and go to Guru99 Bank
self.driver = setup
self.driver.get(self.url)
sleep(0.3)
# create an object of Login Page for access its method
self.lp = LoginPage(self.driver)
# enter valid userid
self.lp.enter_userID("mngr445105")
# enter invalid password
self.lp.enter_password("invalidPassword")
# click login button
self.lp.click_login_btn()
# get popup alert message
alert_message = self.lp.get_alert_message()
# if the popup alert the following message we passed our test cases
expected_alert_message = "User or Password is not valid"
if alert_message == expected_alert_message:
assert True
self.driver.close()
else:
self.lp.accept_alert()
self.driver.close()
assert False
Execute the test cases with pytest test_login.py you will see two test cases running.
By now you should know how to create test cases and pageObject as well as how the two connected.
Let's add logging to our test cases.
Generate Logs
Under utilities package, create a logger.py and add the following code:
import logging
import sys
class LogGen:
@staticmethod
def genlog():
logger = logging.getLogger()
logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s | %(levelname)s | %(message)s',
'%m/%d/%Y %I:%M:%S %p')
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setLevel(logging.DEBUG)
stdout_handler.setFormatter(formatter)
# generate logging and store in logs.log file under log folder
file_handler = logging.FileHandler('../log/logs.log')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
logger.addHandler(stdout_handler)
return logger
To generate logging in our test cases, under Test class create a generateLog object.
We have a test class called Test_001_Login in test_login.py file. create a generateLog object Iinside the class as follow:
class Test_001_Login:
# generateLog Object
generateLog = LogGen.genlog()
def test_case_01(self, setup):
# log
self.generateLog.info("******* Stating test step 1 *******")
# YOUR TEST STEPS
def test_case_02(self, setup):
#log
self.generateLog.info("******* Stating test step 1 *******")
# YOUR TEST STEPS
........................
........................
For each test step you want to generate a logging just add self.generateLog.info("you logging message") or self.generateLog.debug("you logging message") or self.generateLog.warning("you logging message") or self.generateLog.error("you logging message") above each test step: Generate log for our first test case should be done like this:
from pageObjects.LoginPage import LoginPage
from time import sleep
from utilities.logger import LogGen # import our logger we have created
class Test_001_Login:
url = "https://www.demo.guru99.com/V4/"
generateLog = LogGen.genlog()
# test case 01
def test_login_with_valid_credential(self, setup):
# log
self.generateLog.info("********* Stard test_login_with_valid_credential ... *********")
self.driver = setup
# log
self.generateLog.info("Go to https://www.demo.guru99.com/V4/")
self.driver.get(self.url)
sleep(0.3)
self.lp = LoginPage(self.driver)
# log
self.generateLog.info("enter userID")
self.lp.enter_userID("mngr445105")
# log
self.generateLog.info("enter Password")
self.lp.enter_password("udasEgA")
# log
self.generateLog.info("click login button")
self.lp.click_reset_btn()
actual_page_title = self.driver.title
# log
self.generateLog.info("Actual page title is: %s", actual_page_title)
expect_page_title = "Guru99 Bank Manager HomePage"
# log
self.generateLog.info("Expected page title is: %s", expect_page_title)
if actual_page_title == expect_page_title:
assert True
# log
self.generateLog.info("Test Case Passed.")
self.driver.close()
else:
self.driver.close()
# log
self.generateLog.error("Test Case Failed.")
assert False
Run our test case again now we should get a logs.log file under log folder like this:
10/08/2022 01:46:30 PM | INFO | ********* Stard test_login_with_valid_credential ... *********
10/08/2022 01:46:30 PM | INFO | Go to https://www.demo.guru99.com/V4/
10/08/2022 01:46:33 PM | INFO | enter userID
10/08/2022 01:46:34 PM | INFO | enter Password
10/08/2022 01:46:34 PM | INFO | click login button
10/08/2022 01:46:35 PM | INFO | Actual page title is: Guru99 Bank Manager HomePage
10/08/2022 01:46:35 PM | INFO | Expected page title is: Guru99 Bank Manager HomePage
10/08/2022 01:46:35 PM | INFO | Test Case Passed.
Generate HTML Report
To Generate a HTML Report just run the test with the following flag:
pytest --html=../reports/reports.html test_login.py
# generate a HTML report under report folder
The HTML report with logs
We can also add test report information such as project name, project manager, test manager, tester, developers. etc., to our report by adding the following code to our conftest.py file:
def pytest_configure(config):
config._metadata['Project Name'] = 'Guru99 Bank'
config._metadata['Module'] = 'Login'
config._metadata['Project Managers'] = 'Login'
config._metadata['Test Managers'] = 'N/A'
config._metadata['Tester'] = 'Vannak Tak'
config._metadata['Developers'] = 'N/A'
After run our test case with HTML report again we will get a report like this:
Screenshot
To take a screenshot is so simple, just add self.driver.save_screenshot("../screenshots/screenshot_title.png") before your failed test step so that before your test case fail it will take a screenshot as a reference.
Let's try to fail our previous test case and take a screenshot to see how it works:
Handling the common test data that may repeat in every test case
As our test cases keep growing, it is not a good practice to keep the common test data repeating in every test case. The data such as system URL, userID, password, or other common test data might repeat in every test case, these data might be changed at some time. Once it is changed, do we have to go to every test case and manually change it one by one? It is not a good practice, imagine we have hundreds of test cases. To deal with this we have to create a file config.ini under configurations folder to store that common data, and lets every test cases call this file to use the data. That means when the data is changed we need to change it only in one place, each of our test cases will be also changed.
[common login info]
baseURl = https://www.demo.guru99.com/V4/
userID = mngr445105
password = udasEgA
Under utilities package, create another readProperty.py to read the data from config.ini file and add the following code:
import configparser
config = configparser.RawConfigParser()
config.read("../configurations/config.ini")
class ReadConfig:
@staticmethod
def getBaseURL():
url = config.get("common login info", "baseURl")
return url
@staticmethod
def getUserID():
userID = config.get("common login info", "userID")
return userID
@staticmethod
def getPassword():
password = config.get("common login info", "password")
return password
In our test_login.py import from utilities.readProperty import ReadConfig.
Replace url = "https://www.demo.guru99.com/V4/" with url = ReadConfig.getBaseURL() Our test case now should be like this:
from pageObjects.LoginPage import LoginPage
from time import sleep
from utilities.logger import LogGen # import our logger we have created
from utilities.readProperty import ReadConfig
class Test_001_Login:
url = ReadConfig.getBaseURL()
userID = ReadConfig.getUserID()
password = ReadConfig.getPassword()
generateLog = LogGen.genlog()
# test case 01
def test_login_with_valid_credential(self, setup):
# log
self.generateLog.info("********* Stard test_login_with_valid_credential ... *********")
self.driver = setup
# log
self.generateLog.info("Go to %s", self.url)
self.driver.get(self.url)
sleep(0.3)
self.lp = LoginPage(self.driver)
# log
self.generateLog.info("enter userID")
self.lp.enter_userID(self.userID)
# log
self.generateLog.info("enter Password")
self.lp.enter_password(self.password)
# log
self.generateLog.info("click login button")
self.lp.click_login_btn()
self.generateLog.info("Getting page title")
actual_page_title = self.driver.title
# log
self.generateLog.info("Actual page title is: %s", actual_page_title)
expect_page_title = "Guru99 Bank Manager HomePage"
# log
self.generateLog.info("Expected page title is: %s", expect_page_title)
if actual_page_title == expect_page_title:
assert True
# log
self.generateLog.info("Test Case Passed.")
self.driver.close()
else:
# before we failed our test we will take a screenshot as a reference
self.driver.save_screenshot("../screenshots/login_with_valid_credential.png")
self.driver.close()
# log
self.generateLog.error("Test Case Failed.")
assert False
Data Driven Test
First we need to create a file XLUtil.py under untilities folder to read data from excel file:
import openpyxl
def getRowCount(file, sheetName):
workbook = openpyxl.load_workbook(file)
sheet = workbook[sheetName]
return (sheet.max_row)
def readData(file, sheetName, rowNum, colNum):
workbook = openpyxl.load_workbook(file)
sheet = workbook[sheetName]
return sheet.cell(row=rowNum, column=colNum).value
We create another test file test_login_dataDriven.py and add the following test steps:
from pageObjects.LoginPage import LoginPage
from time import sleep
from utilities.logger import LogGen # import our logger we have created
from utilities.readProperty import ReadConfig
from utilities import XLUtil
class Test_001_Login:
url = ReadConfig.getBaseURL()
# path = ../testData/login_data.xlsx, I have config in config.ini file already
excel_file = ReadConfig.getXcel()
generateLog = LogGen.genlog()
# test case 01
def test_login_with_valid_credential(self, setup):
# log
self.generateLog.info("********* Stard test_login_with_valid_credential ... *********")
self.driver = setup
# log
self.generateLog.info("Go to %s", self.url)
self.driver.get(self.url)
sleep(0.3)
self.lp = LoginPage(self.driver)
self.rows = XLUtil.getRowCount(self.excel_file, 'Sheet1')
result = []
for r in range(2, self.rows+1):
self.userID = XLUtil.readData(self.excel_file, "Sheet1", r, 1)
self.password = XLUtil.readData(self.excel_file, "Sheet1", r, 2)
# log
self.generateLog.info("enter userID")
self.lp.enter_userID(self.userID)
# log
self.generateLog.info("enter Password")
self.lp.enter_password(self.password)
# log
self.generateLog.info("click login button")
self.lp.click_login_btn()
self.generateLog.info("Getting page title")
try:
actual_page_title = self.driver.title
except:
actual_page_title = ''
# log
self.generateLog.info("Actual page title is: %s", actual_page_title)
expect_page_title = "Guru99 Bank Manager HomePage"
# log
self.generateLog.info("Expected page title is: %s", expect_page_title)
if actual_page_title == expect_page_title:
self.generateLog.info("Test Case Passed.")
self.lp.click_logout()
self.lp.accept_alert()
result.append("passed")
else:
self.generateLog.error("Test Case Failed.")
result.append("failed")
if "failed" not in result:
assert True
self.driver.close()
else:
self.driver.close()
assert False
Note: Please make sure that the rows that is empty in the excel sheet has to be removed.
Conclusion
It is quite a bit long blog, but if you follow along I believe you have made a complete test automation framework from scratch. There is another way to read data from an online google sheet, but it is too long already for this blog. Let's play around more with this framework so you will have a clear understanding. If there is any unclear explanation from me or if I even explained it properly, feel free to ask me. I will try my best to help with what I could.
There is still a few staff I missed here, like advance test case execution run test cases with markdown. etc., Let’s do it in another blog.