Using Selenium and Python Hypothesis for Automation Testing
Himanshu Sheth
Posted On: May 8, 2019
22181 Views
17 Min Read
This article is a part of our Content Hub. For more in-depth resources, check out our content hub on Selenium Python Tutorial.
When it comes to testing a software product, various forms of testing e.g. unit testing, integration testing, release testing, etc. are performed at different stages of SDLC(Software Development Test Cycle). However, one of the major challenges faced by developers is coming up with test-cases & test-suites that can be instrumental in verifying every aspect of the code so that they are able to achieve maximum code coverage.
Coming up with unique unit test cases that can cover corner scenarios or edge cases is a huge task. Even if the developer is able to identify the scenarios, he has to write a significant amount of code to fulfill the target. Many times, this leads to copying test-cases and changing (adding/deleting) a few lines of code to come up with new test cases. This is where Hypothesis, a Python testing library can be used for creating unit tests which are easy to write and powerful when executed. Though it makes it easy to derive unit tests, the beauty lies in the edge test cases which you wouldn’t have thought about. Before we have a detailed look at Hypothesis testing in Python and Selenium, we look into an important aspect called Property-based testing which forms the basic premise of Hypothesis Python library.
If you’re new to Selenium and wondering what it is then we recommend checking out our guide – What is Selenium?
Overview Of Property-Based Testing
You should make use of Hypothesis for Python testing when you are planning to incorporate Property-based testing in your test strategy. There is a significant amount of difference between unit testing & property-based testing. In unit testing, the developer of provides input in order to verify the functionality of the code. The output is compared with the expected output and based on the comparison, the test is declared as Pass/Fail.
On the other hand, Property-based testing is about testing the program across a wide variety of inputs. For property-based testing, libraries that are equipped with generators are used to generate input values for testing. This can also be achieved using unit testing, but the developer will have to write much more code in order to accommodate different types of input values. Property-based testing was first introduced by the QuickCheck framework in Haskell In unit testing, the inputs are provided manually i.e. via code, hence there is a possibility that you may miss certain test scenarios (especially the edge case scenarios). This is where property-based testing can be instrumental for verification of code across different types & range of inputs.
- Unit testing – Provide an input (e.g. 0,1,2,…) and get a result (e.g. 1, 2, 4, ….)
- Property-based testing – Description of inputs (e.g. All ints) and description of conditions that must be held (e.g. Result is an int)
When To Use Property-Based Testing?
You might argue that doing unit-testing is sufficient for you as you are able to unearth bugs in your code using the unit-test code. Though this argument might suffice in some cases, property-based tests can be more useful in thoroughly testing your code as it even takes into account the majority of the edge test-cases. There is no clear winner as far as unit tests and property-based tests are concerned, it is important to understand the pros & cons of property-based testing.
Advantages Of Property-Based Testing
- As a developer, you can come up with an infinite number of test-cases using this approach. The number of tests that can be generated are limited by the time that you can invest in the tests & capabilities of the machine used for test-case generation.
- Powerful libraries like Hypothesis for Python testing can be used for generation of property-based tests. Using features like test strategies, a developer can come up with huge number of tests with minimal code implementation.
- As described earlier, property-based tests are best suited for development & testing of edge-case scenarios.
Disadvantages Of Property-Based Testing
- As test-cases are randomly generated using capabilities/features of libraries like Hypothesis for Python testing with Selenium, there is a possibility that some important edge-case scenario might get missed.
- There is a significant amount of learning curve involved in learning & mastering tools/libraries used for property-based testing. Hence, writing code using these libraries can sometimes be a big challenge.
- Finding properties that can match your test requirements is another challenge. However, once you have mastered the art of property-based testing (using some library like Hypothesis, for Python testing), this issue may not hold true.
It is always recommended to use Property-based testing, along with Unit testing for obtaining the maximum results.
Hypothesis A Python Testing Library
So far we have seen the advantages that property-based testing has over traditional example-based testing. In example-based testing approach, there is a test input ‘I’ passed to the function under test and the result of the test function is compared against the expected result. You may not be able to achieve complete test exhaustiveness as the implementation depends on the understanding of the developer. Your test code might not be robust enough to cover all kinds of test inputs.
Test exhaustiveness and test robustness is possible through property-based testing and Hypothesis, a Python testing library can be used for effective property tests.
Using Hypothesis you can write parameterized tests derived from a source of examples. It generates simple and comprehensible examples that can test every aspect of your code (especially where your code could fail).
Since you are able to test more edge test-cases, you can find more bugs in your code with less amount of work. We will have a detailed look at the features of Hypothesis for Python testing in subsequent sections of this article.
We will make use of pytest, and Selenium for Hypothesis Python testing. Check out our blog on pytest & Selenium WebDriver, if you are not already aware of how pytest works!
Hypothesis is compatible to run with Selenium and Python (version 2.7 onwards) and has support for popular python testing frameworks like py.test, unittest, and Nose. For implementation, we are making use of the PyCharm IDE (Community version) which can be downloaded from here. Once you have installed Selenium and Python, pytest; you should install Hypothesis for Python testing using the below command.
pip install hypothesis
Now that you have installed Hypothesis for Python testing with Selenium, let’s have a look at a very simple problem which demonstrates the shortcomings of unit testing, as well as parameterized based pytests. In the below program, we compute the addition of two numbers :
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
A simple pytest to add two numbers import pytest import pytest_html # Define a function which takes two arguments as integers and adds the two numbers def sum_of_numbers(number_1, number_2): return number_1 + number_2 # A simple test case to verify the sum_of_numbers function # Since it is a pytest testcase, the test function should start with test_ def test_verify_sum_of_numbers(): assert sum_of_numbers(2, 3) == 5 |
The implementation is self-explanatory and once the above pytest code is executed to test sum_of_numbers() API, it would result in PASS.
In order to test the sum functionality against different types of inputs, we need to follow the copy-paste mechanism where sum_of_numbers() is supplied with different input values. Since this is not a scalable approach, we make use of the parameterized fixtures feature in pytest. With parameterized fixtures, we can test more scenarios by just adding more input values to the test case.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
A simple pytest to add two numbers import pytest import selenium import pytest_html # Define a function which takes two arguments as integers and adds the two numbers def sum_of_numbers(number_1, number_2): return number_1 + number_2 # A simple test case to verify the sum_of_numbers function # Since it is a pytest testcase, the test function should start with test_ #def test_verify_sum_of_numbers(): # assert sum_of_numbers(2, 3) == 5 #A more scalable approach is to use Parameterized Fixtures @pytest.mark.parametrize('number_1, number_2, expected_output',[(1,2,3), (4,5,9), (6,-1,5), (-5,-4,-9)]) def test_verify_sum_of_numbers(number_1, number_2, expected_output): assert sum_of_numbers(number_1, number_2) == expected_output |
The output is shown below. All the test cases pass since the addition of the input numbers equates to the expected output.
Though we can add more test inputs via parameterized fixtures, there could be cases where important scenarios are missed out. Also, there could be a certain amount of ambiguity involved with the input and output variables. Take the case of sum_of_numbers() function, there could be a good amount of confusion involving its input & output. Some of them are mentioned below :
- Can the input arguments be only integers i.e. int or it can be float as well?
- What is the maximum value that the input arguments can hold and what should happen in underflow/overflow kind of scenarios?
- Can the input values be of float type, if so can it be used in combination with an int input type?
The solution to the problem that we encounter with example-based testing can be solved using Hypothesis using which you can write property-based tests. Using Hypothesis, you can write tests with test framework like pytest and test your implementation against huge set of desired input data. For more details, refer to the official documentation of Hypothesis, a Python testing library.
This certification is for professionals looking to develop advanced, hands-on expertise in Selenium automation testing with Python and take their career to the next level.
Here’s a short glimpse of the Selenium Python 101 certification from LambdaTest:
Hypothesis – Strategies, Decorators, & more
The backbone of Hypothesis is based on the famous principle ‘Most things should be easy to generate and everything should be possible’. Based on this principle, Hypothesis for Python testing offers strategies to handle most built-in types with arguments for constraining or adjusting the output. Hypothesis also offer higher-order strategies using which one can write effective test cases to handle more complex scenarios.
In simple terms, you can say that you give your requirements to the strategy module and it returns different input values (for test) based on your requirement. In the example mentioned above, the input to the strategy should be requirement of two integers. Functions for building strategies are available as a part of the hypothesis.strategies module.
Now that you have some idea about strategies in Hypothesis, we rewrite the above test code by incorporating more input data sets using Hypothesis, a Python library. The modified code is shown below :
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
''' Addition of numbers using pytest & Hypothesis ''' import pytest ''' Import the Hypothesis module ''' import hypothesis from hypothesis import given ''' Strategies are the backbone of Hypothesis. In our case, we will use the integer strategy ''' import hypothesis.strategies as strategy # Define a function which takes two arguments as integers and adds the two numbers def sum_of_numbers(number_1, number_2): return number_1 + number_2 ''' @given is the decorator ''' ''' We use the integer Strategy as testing is performed only on integer inputs ''' @given(strategy.integers(), strategy.integers()) def test_verify_sum_of_numbers(number_1, number_2): assert sum_of_numbers(number_1, number_2) == number_1 + number_2 |
In order to execute the code, you can use the –hypothesis-show-statistics option along with the normal py.test command. The command that we have used is
|
1 |
py.test --capture=no --hypothesis-show-statistics < file-name.py > |
As shown in the output snapshot below, we have carried out two test trails and each time it generated 100 different inputs. With Hypothesis, the default number of test runs that are possible is 100.
Let’s do a code-walkthrough of the addition test code which was based on the Hypothesis library. The basic function [sum_of_numbers()] that computes the sum of the two numbers if kept intact. The code which tests the sum functionality is modified to work with Hypothesis.
@given serves as the entry point in Hypothesis and the decorator helps to convert the specific test function which accepts arguments into a randomized test. In our case, only integer inputs are considered for testing. Hence, @given decorator has two arguments both of which are integer strategy. The syntax of @given is below :
hypothesis.given(*given_arguments, **given_kwargs)
More details about the @given decorator is available here
In the next line, we are importing @strategies from Hypothesis. @strategies is mainly used for generation of test data. To check out all the functions available for building strategies for performing Hypothesis testing in Python and Selenium refer to hypothesis.strategies module. In our example, we used the integer strategy. There are a number of in-built strategies in Hypothesis and you can also compose higher-order strategies for more complex inputs. Some examples of in-built strategies are :
Covering every strategy is beyond the scope of this blog, hence we recommend you to have a look at the official documentation of strategies.
Putting ‘verbose’ Option & @example Decorator To Work
As a developer, you may get confused after looking at the output of the code that used Hypothesis. In every test code, there are input arguments/input values that are used for testing. In the above example, 100 test runs were carried out; but there was no information about what were the input values to the sum_of_numbers() function. For achieving this goal, we set the Verbosity level to verbose. We need to import the @settings decorator in order to set the verbosity.
As a developer, you may get confused after looking at the output of the code that used Hypothesis. In every test code, there are input arguments/input values that are used for testing. In the above example, 100 test runs were carried out; but there was no information about what were the input values to the sum_of_numbers() function. For achieving this goal, we set the Verbosity level to verbose. We need to import the @settings decorator in order to set the verbosity.
|
1 2 3 4 5 6 7 |
…………………………………….. from hypothesis import given, settings, Verbosity ……………………………………………………………. ………………………………………………………….. @settings(verbosity=Verbosity.verbose) ……………………………………………………………. ………………………………………………………….. |
Make sure that you use the @settings decorator along with the @given decorator i.e. @settings should be set just before @given decorator. If that is not done, you would encounter an error which states ‘hypothesis.errors.InvalidArgument: Using . There is one more modification we make to our existing implementation where we extend the number of test runs to 500. This can be done by setting max_examples value of @settings object to 500.@settings on a test without @given is completely pointless’
The Verbosity & max_examples value of the @settings module have to be modified at a single place, else it results in an error (as shown in the code snippet below).
|
1 2 3 4 5 6 7 8 9 |
…………………………………….. from hypothesis import given, settings, Verbosity ……………………………………………………………. ………………………………………………………….. @settings(verbosity=Verbosity.verbose) @settings(max_examples=500) …………………………………………………………. …………………………………………………………. ………………………………………………………….. |
If you try to decorate the @settings decorator using the above implementation, you would encounter an error stating
|
1 |
hypothesis.errors.InvalidArgument: test_verify_sum_of_numbers has already been decorated with a settings object. |
The modified working implementation is below (Changes are marked in Yellow color).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
''' Addition of numbers using pytest & Hypothesis ''' import pytest ''' Import the Hypothesis module ''' import hypothesis from hypothesis import given, settings, Verbosity ''' Strategies are the backbone of Hypothesis. In our case, we will use the integer strategy ''' import hypothesis.strategies as strategy # Define a function which takes two arguments as integers and adds the two numbers def sum_of_numbers(number_1, number_2): return number_1 + number_2 ''' @given is the decorator ''' ''' We use the integer Strategy as testing is performed only on integer inputs ''' @settings(verbosity=Verbosity.verbose, max_examples=500) @given(strategy.integers(), strategy.integers()) def test_verify_sum_of_numbers(number_1, number_2): assert sum_of_numbers(number_1, number_2) == number_1 + number_2 |
Below is the screenshot of the execution where we get information about the input values used for testing and number of test runs are now 500 (instead of 100).
Though the code is testing against different range of input values, you might want to consider restricting the minimum & maximum value that the input variables can hold. You can do this by just setting the min_value & max_value for the input variables as a part of the @strategy decorator. The modified working implementation is below (Changes are marked in Yellow color).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
''' Addition of numbers using pytest & Hypothesis ''' import pytest ''' Import the Hypothesis module ''' import hypothesis from hypothesis import given, settings, Verbosity ''' Strategies are the backbone of Hypothesis. In our case, we will use the integer strategy ''' import hypothesis.strategies as strategy # Define a function which takes two arguments as integers and adds the two numbers def sum_of_numbers(number_1, number_2): return number_1 + number_2 ''' @given is the decorator ''' ''' We use the integer Strategy as testing is performed only on integer inputs ''' @settings(verbosity=Verbosity.verbose, max_examples=500) @given(strategy.integers(min_value=1, max_value=20), strategy.integers(min_value=5, max_value=100)) def test_verify_sum_of_numbers(number_1, number_2): assert sum_of_numbers(number_1, number_2) == number_1 + number_2 |
As per the changes, the variables number_1 & number_2 can hold values as per the below condition
number_2 : number_2 GTE 5 & LTE 100
We also enable the –verbose option while executing the code, the updated command is below, the output shows the effect of min_value & max_value on the input arguments (used for test).
|
1 |
py.test --capture=no –verbose --hypothesis-show-statistics < file-name.py > |
Stateful Testing With Hypothesis For Python
The major advantage of using Hypothesis as Python testing library is the automatic generation of test data that can be used for testing the code. Even when you are using the @given decorator, you have to write lot of tests. Stateful testing in Hypothesis is capable of generating entire tests along with the test data. As a developer, you have just specify the primitive actions and Hypothesis will try to find sequences that can result in a failure.
There are two types of stateful testing APIs in Hypothesis – High level API called rule-based state machine and low level API called generic state machine. Rule-based state machines are more popular as they are more user-friendly. RuleBasedStateMachine are a part of the hypothesis.stateful module.
|
1 |
class hypothesis.stateful.RuleBasedStateMachine |
|
1 |
State machines can carry a bunch of types of data called as Bundles, and there can be a set of rules that can push data out of Bundles and onto the Bundles. For more information about Rule based state machines, you can refer the official documentation of <a href="https://hypothesis.readthedocs.io/en/latest/stateful.html" rel="noopener nofollow" target="_blank">Stateful testing with Hypothesis.</a> |
Automated Cross Browser Testing Using Hypothesis With LambdaTest Selenium Grid
So far in this Hypothesis Python testing tutorial, we have covered major aspects about Hypothesis and how you can use decorators available in Hypothesis for verification of your code. Let’s have a look at how you can use pytest with Hypothesis for testing Python and Selenium in order to perform automated cross browser testing of your website/web application. Cross-browser testing is testing your website/web-app across different combinations of browsers, operating systems, and devices.
LambdaTest offers a Selenium Grid consisting 2000+ real browsers to help you perform automation testing with Selenium for browser compatibility testing. You could also perform manual cross browser testing by interacting with these browser live by the help of VMs(Virtual Machines) hosted on their cloud servers. I will demonstrate you how to leverage LambdaTest for automation testing with Python and Selenium.
For performing cross browser testing with Hypothesis, we devise a test code which tests a given URL e.g. https://www.lambdatest.com on Chrome & Firefox browser. The verification should be on the Mozilla Firefox browser version 64.0 and Google Chrome browser version 71.0. You have to launch the test URL in the respective browsers and close the browser instance once the website is loaded.
Before we have a look at the implementation, and if you are following me step-by-step then I would recommend that you create an account on LambdaTest as we would be using the Selenium Remote WebDriver on LambdaTest. Below is the overall implementation using Hypothesis, Python testing library on LambdaTest Selenium Grid.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
import pytest from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.common.keys import Keys from time import sleep import urllib3 import warnings #Set capabilities for testing on Chrome ch_caps = { "build" : "Hypothesis - Testing on Chrome", "name" : "Hypothesis - Verification of URL on Chrome", "platform" : "Windows 10", "browserName" : "Chrome", "version" : "71.0", } #Set capabilities for testing on Firefox ff_caps = { "build": "Hypothesis - Testing on Firefox", "name": "Hypothesis - Verification of URL on Firefox", "platform" : "Windows 10", "browserName" : "Firefox", "version" : "64.0", } # Visit https://accounts.lambdatest.com/profile for getting the access token user_name = "your-user-name" app_key = "access key generated from LambdaTest dashboard" class CrossBrowserSetup(object): global web_driver def __init__(self): global remote_url urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) # web_driver = webdriver.Remote(command_executor=remote_url, desired_capabilities=ch_caps) remote_url = "https://" + user_name + ":" + app_key + "@hub.lambdatest.com/wd/hub" def add(self, browsertype_1, browsertype_2): print(browsertype_1) print(browsertype_2) if (browsertype_1 == "Chrome") or (browsertype_2 == "Chrome"): web_driver = webdriver.Remote(command_executor=remote_url, desired_capabilities=ch_caps) if (browsertype_1 == "Firefox") or (browsertype_2 == "Firefox"): web_driver = webdriver.Remote(command_executor=remote_url, desired_capabilities=ff_caps) self.driver = web_driver self.driver.get("https://www.lambdatest.com") print(self.driver.title) #sleep(1) web_driver.close() web_driver.quit() # Property-based Tests from hypothesis import given, example import hypothesis.strategies as strategy @given(strategy.just("Firefox"), strategy.just("Chrome")) def test_add(browsertype_1, browsertype_2): cbt = CrossBrowserSetup() cbt.add(browsertype_1, browsertype_2) |
Since we are making use of the Selenium Grid setup on LambaTest, you would require the right combination of username & accesskey to access their grid. You can find these values at the automation dashboard by clicking on the key icon. Now, replace the variables user_name & app_key with your credentials. In the setup where we performed the test, we could execute 2 tests in parallel. Let’s do a code-walkthrough of the above implementation
In the beginning, we import the necessary packages e.g. selenium, pytest, time, urllib3, etc. Once the necessary modules are imported, we set the capabilities of the browsers on which the test would be performed. You can visit LambdaTest Desired Capabilities Generator to generate the required browser capabilities. In the add function, we initiate the required browser instance using the Remote Webdriver API. The remote Webdriver API takes two important parameters – command_executor and desired_capabilities.
command_executor is the remote URL on which the Selenium Grid is setup and desired_capabilities is the list of capabilities that should be present in the browser under test. For more in-depth information about Selenium WebDriver API and pytest, you can visit our other blogs that cover the topic in more detail.
Once the required pre-requisites are complete, we make use of the Hypothesis library to come up with the required tests for Python and Selenium. As shown in the striped implementation, the @strategy, @given & @example decorators are imported from the Hypothesis, Python testing library. The test code [test_add()] consists of two string arguments. Since the test has to be performed only on Firefox & Chrome, we use the @given decorator to limit the input arguments to “Firefox” & “Chrome”. We have used the hypothesis.strategies.just() module to fulfill the requirement.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
……………………………………………………………………… ……………………………………………………………………… ……………………………………………………………………… # Property-based Tests from hypothesis import given, example import hypothesis.strategies as strategy @given(strategy.just("Firefox"), strategy.just("Chrome")) def test_add(browsertype_1, browsertype_2): cbt = CrossBrowserSetup() cbt.add(browsertype_1, browsertype_2) |
You can execute the code using the standard command, the output is shown below :
|
1 |
py.test --capture=no --hypothesis-show-statistics < file-name.py > |
In order to verify the output, you should visit the Automation section on LambdaTest and locate the test as per the name assigned in the browser capabilities array. As this is a pytest code, make sure that the name of file should start with test_ Since the execution is done on the remote Selenium Grid, you should visit Automation Dashboard to check the status of the test, below is the screenshot of the test performed on Firefox (version 64.0).
After executing the above automation script for Hypothesis Python testing we can observe that our test ran successfully on Google Chrome & Mozilla Firefox in Parallel.
As you click on these tests from automation dashboard in LambdaTest. You will find test details.
1. Hypothesis Python Testing In Google Chrome
2. Hypothesis Python Testing In Mozilla Firefox
Conclusion
There are many scenarios where you might want to do a thorough verification of your code by testing it across different input values. Hypothesis, a Python testing library can be handy in these cases since it can generate extensive test data which can be used to perform normal tests, as well as edge type tests. Based on your requirements, you should choose the right kind strategy & decorator so that your overall effort in test code implementation & execution is reduced.
You can also use Hypothesis to optimize the tests written using pytest & unittest with Selenium. Cross browser testing for Python is also possible using pytest & Hypothesis. In a nutshell, Hypothesis is a powerful & flexible library that should be considered when you are planning to do property-based testing.
Author’s Profile
Himanshu Sheth
Himanshu Sheth is a seasoned technologist and blogger with more than 15+ years of diverse working experience. He currently works as the 'Lead Developer Evangelist' and 'Senior Manager [Technical Content Marketing]' at LambdaTest. He is very active with the startup community in Bengaluru (and down South) and loves interacting with passionate founders on his personal blog (which he has been maintaining since last 15+ years).
Blogs: 132
Got Questions? Drop them on LambdaTest Community. Visit now
