Appearing for ISTQB Advance Level exam? Decode the challenging topic "Pairwise Testing – Orthogonal Array". Master the method with this tutorial.
Pairwise Testing is a black box testing technique used in situations where the interaction of the factors cannot be so easily determined, or even when there is no interaction of the factors.
For example, consider the testing of a web application that has four features (factors) say OS, speed, browser types and anti-virus software and each of these factors has five values (options). In order to have a full coverage for the application, we need to test 5 x 5 x 5 x 5 = 625 combinations, which is not easily possible. We may test all these combinations, provided we have enough time and budget. However, what if we do not have enough time and budget? Pairwise testing is the technique we can use in such a situation.
In pairwise testing, we make sure that all possible pairs of parameter values are covered by at least one test. However, not all possible higher-order combinations such as triples, quadruples and so forth will be covered.
There are two basic models in pairwise testing
Let us discuss more on “Orthogonal Array”
Before going into the details of orthogonal array, let us write down test case by applying conventional / exhaustive testing technique
Example 1
Consider a function having three factors X, Y and Z and each factor having two options as 0 and 1
Now, let us list down all the possible combinations without considering orthogonal array
Here, as you can see there are 8 test cases / test combinations
Let us move on to the details of orthogonal array and finally will derive the test cases using orthogonal array technique for the same example
Definition of orthogonal array: A two-dimensional array constructed with special mathematical properties such that choosing any two columns in the array provides every pair combination of each number in the array
If I explain it with example 1, first we need to consider the factors X & Y, and ensure that the combinations of X & Y is covered. Second, we need to consider the factors X & Z and ensure that the combinations of X & Z is covered, and third we need to consider the factors Y & Z to ensure that all pair of combinations are covered
Orthogonal array testing is a systematic way of testing all-pair combinations of variables using orthogonal arrays. It significantly reduces the number of all combinations of variables to test all pair combinations
Let us get to know the rules of writing an orthogonal array
There are three rules for selecting an orthogonal array
1) There must be at least as many columns as factors
i.e. No of columns = No of factors
2) There must be at least enough numbers in the columns to hold the options for each factor. i.e. If you have two options for a factor, then test cases should be there for both the values and if there are three options for a factor, then test cases should be there for all the three values. If we have spare numbers that don’t map to any option for that factor can be represented by a tilde( ~ )
3) There must be at least as many rows or test cases as the product of the two largest numbers of options. For example, if one factor has 4 options and another has 3 options and yet another has 2 options, then 4 x 3 = 12 row
Now, let us apply these rules and the definitions of orthogonal array in example 1 and try to find the tests involved
1) Consider two factors X & Y. Now list down all possible combinations for the pair which will be 0 0, 0 1, 1 0 and 1 1
2) Consider the factors X & Z and list down all possible combinations for the pair and here notice to shuffle the order while writing the pairs i.e. 0 0, 0 1, 1 1 and 1 0. Note that here we changed the order of values of Z for the 3rd and 4th rows Or else we can explain in this way also as (1) Write down all the values of X similar to how it is written for the pairs X & Y (2) While writing down the values of Z, initially all the values were written as same as Y in pairs X & Y.But then later on to cover all pairs of combinations for the factors Y & Z, the values of Z in 3rd and 4th row is interchanged as highlighted below
3) Now consider the factors Y & Z. Here note that all the combinations for the pairs is now listed. Here notice that if we had not changed the order for the pairs X & Z, all combinations would not have been covered for pairs Y & Z Or else in another way, write down the values of Y from X & Y factor and values of Z from X & Z factor. If we combine this we will get the order of combinations for Y & Z
Now the table shown below are the test cases or test combinations developed using orthogonal array
Note that there are only four test cases after applying the orthogonal array technique.
Thus the number of test cases by using orthogonal array technique is reduced to four while in conventional technique the number of test cases is eight.
While considering the example 1, we have seen that only four tests are needed while testing with orthogonal array technique and we will have an impression that it missed out some tests or combinations. But if we consider a large example where in testing all combinations is impossible then we will come to know the strength of orthogonal array technique
Now, let us move on to the naming convention followed in orthogonal arrays
Consider an orthogonal array having 4 factors and each factor having 3 values each
No: of test cases as per the rules of orthogonal array = 3 x 3 = 9
The naming convention used here is L 9 (34)
Formula used in naming convention is L (no: of test cases) Count of values in each factor (number of factors)
Now, let us move on to another example in naming convention.
Consider an orthogonal array having six factors and these six factors are having 3 options each and yet another factor having 6 option.
No: of test cases = 3 x 6 = 18
As per the formula, the naming convention can be written as L 18 (3 6 6 1)
Now let us have another example for deriving the number of test cases by applying orthogonal array technique
Example 2
Company A has designed a web page with three distinct sections (Top, middle, and bottom). These sections can be individually shown or hidden by the user. The browsers on which the webpage can be viewed with are internet explorer and Firefox. The server used for launching the application can be IIS, Apache and web logic. Write down the number of test cases required by using orthogonal array technique.
Step 1
For example, consider the testing of a web application that has four features (factors) say OS, speed, browser types and anti-virus software and each of these factors has five values (options). In order to have a full coverage for the application, we need to test 5 x 5 x 5 x 5 = 625 combinations, which is not easily possible. We may test all these combinations, provided we have enough time and budget. However, what if we do not have enough time and budget? Pairwise testing is the technique we can use in such a situation.
In pairwise testing, we make sure that all possible pairs of parameter values are covered by at least one test. However, not all possible higher-order combinations such as triples, quadruples and so forth will be covered.
There are two basic models in pairwise testing
- Orthogonal Array
- All-pair Table
Let us discuss more on “Orthogonal Array”
Before going into the details of orthogonal array, let us write down test case by applying conventional / exhaustive testing technique
Example 1
Consider a function having three factors X, Y and Z and each factor having two options as 0 and 1
Now, let us list down all the possible combinations without considering orthogonal array
| Tests | Factors | ||
|---|---|---|---|
| X | Y | Z | |
| 1 | 0 | 0 | 0 |
| 2 | 0 | 0 | 1 |
| 3 | 0 | 1 | 0 |
| 4 | 0 | 1 | 1 |
| 5 | 1 | 0 | 0 |
| 6 | 1 | 0 | 1 |
| 7 | 1 | 1 | 0 |
| 8 | 1 | 1 | 1 |
Let us move on to the details of orthogonal array and finally will derive the test cases using orthogonal array technique for the same example
Definition of orthogonal array: A two-dimensional array constructed with special mathematical properties such that choosing any two columns in the array provides every pair combination of each number in the array
If I explain it with example 1, first we need to consider the factors X & Y, and ensure that the combinations of X & Y is covered. Second, we need to consider the factors X & Z and ensure that the combinations of X & Z is covered, and third we need to consider the factors Y & Z to ensure that all pair of combinations are covered
Orthogonal array testing is a systematic way of testing all-pair combinations of variables using orthogonal arrays. It significantly reduces the number of all combinations of variables to test all pair combinations
Let us get to know the rules of writing an orthogonal array
There are three rules for selecting an orthogonal array
1) There must be at least as many columns as factors
i.e. No of columns = No of factors
2) There must be at least enough numbers in the columns to hold the options for each factor. i.e. If you have two options for a factor, then test cases should be there for both the values and if there are three options for a factor, then test cases should be there for all the three values. If we have spare numbers that don’t map to any option for that factor can be represented by a tilde( ~ )
3) There must be at least as many rows or test cases as the product of the two largest numbers of options. For example, if one factor has 4 options and another has 3 options and yet another has 2 options, then 4 x 3 = 12 row
Now, let us apply these rules and the definitions of orthogonal array in example 1 and try to find the tests involved
1) Consider two factors X & Y. Now list down all possible combinations for the pair which will be 0 0, 0 1, 1 0 and 1 1
| Tests | Factors | |
|---|---|---|
| X | Y | |
| TC 1 | 0 | 0 |
| TC 2 | 0 | 1 |
| TC 3 | 1 | 0 |
| TC 4 | 1 | 1 |
| Tests | Factors | |
|---|---|---|
| X | Y | |
| TC 1 | 0 | 0 |
| TC 2 | 0 | 1 |
| TC 3 | 1 | 1 |
| TC 4 | 1 | 0 |
| Tests | Factors | |
|---|---|---|
| X | Y | |
| TC 1 | 0 | 0 |
| TC 2 | 1 | 1 |
| TC 3 | 0 | 1 |
| TC 4 | 1 | 0 |
| Tests | Factors | ||
|---|---|---|---|
| X | Y | Z | |
| TC 1 | 0 | 0 | 0 |
| TC 2 | 0 | 1 | 1 |
| TC 3 | 1 | 0 | 1 |
| TC 4 | 1 | 1 | 0 |
Thus the number of test cases by using orthogonal array technique is reduced to four while in conventional technique the number of test cases is eight.
While considering the example 1, we have seen that only four tests are needed while testing with orthogonal array technique and we will have an impression that it missed out some tests or combinations. But if we consider a large example where in testing all combinations is impossible then we will come to know the strength of orthogonal array technique
Now, let us move on to the naming convention followed in orthogonal arrays
Consider an orthogonal array having 4 factors and each factor having 3 values each
No: of test cases as per the rules of orthogonal array = 3 x 3 = 9
The naming convention used here is L 9 (34)
Formula used in naming convention is L (no: of test cases) Count of values in each factor (number of factors)
Now, let us move on to another example in naming convention.
Consider an orthogonal array having six factors and these six factors are having 3 options each and yet another factor having 6 option.
No: of test cases = 3 x 6 = 18
As per the formula, the naming convention can be written as L 18 (3 6 6 1)
Now let us have another example for deriving the number of test cases by applying orthogonal array technique
Example 2
Company A has designed a web page with three distinct sections (Top, middle, and bottom). These sections can be individually shown or hidden by the user. The browsers on which the webpage can be viewed with are internet explorer and Firefox. The server used for launching the application can be IIS, Apache and web logic. Write down the number of test cases required by using orthogonal array technique.
Step 1
First let us list down the factors and the options for each factor
The example above has four factors
(1) Section
(2) Visibility
(3) Browser
(4) Server
Options for the factors are as given below
(1) Section – Top, middle, bottom, No: of options = 3
(2) Visibility – Shown, hidden, No: of options = 2
(3) Browser – IE, Firefox, No: of options = 2
(4) Server – IIS, Apache, web logic, No: of options = 3
Step 2
Applying the rules of orthogonal array technique to derive at the number of test cases
No: of test cases = 3 x 3 = 9
Now, for the same example given above, write down the naming convention for the orthogonal array
The
above orthogonal array is having four factors out of which two factors
are having 2 options each and other two factors are having three options
each
Here, number of test cases = 3 x 3 = 9
Hence the naming convention is L 9 (2 2 3 2)
COMMENTS