Week 8 - Regression Testing

Lecture recording here.

Introduction

Regression testing is done to ensure that enhancements or defect fixes made to the software works properly and does not affect the existing functionality. Anytime software changes are made, it is important to ensure that the changes or additions work as designed, and the changes or additions do not break something that is already working and should continue to work.

Regression testing follows selective re-testing technique. Whenever the defect fixes are done, a set of test cases that need to be run to verify the defect fixes are selected by the test team. An impact analysis is done to find out what areas may get impacted due to those defect fixes. Based on the impact analysis, some more test cases are selected to take care of the impacted areas. Since this testing technique focuses on reuse of existing test cases that have already been executed, the technique is called selective re-testing. There may be situations where new test cases need to be developed to take care of some impacted areas. However, by and large, regression testing reuses the test cases that are available, as it focuses on testing the features that are already available and tested at least once already.

Videos

Assignment(s)

Assignment 3 - Performance Testing of a Custom Database

Lecture Material

• Chapter 8 of Software Testing: Principles and Practice by Gopalaswamy Ramesh, 2007

Types of Regression Testing

Before going into the types of regression testing, let us understand what a "build" means. When internal or external test teams or customers begin using a product, they report defects. These defects are analyzed by each developer who make individual defect fixes. The developers then do appropriate unit testing and check the defect fixes into a Configuration Management (CM) System. The source code for the complete product is then compiled and these defect fixes along with the existing features get consolidated into the build. A build thus becomes an aggregation of all the defect fixes and features that are present in the product. There are two types of regression testing in practice: regular regression testing and final regression testing.

A regular regression testing is done between test cycles to ensure that the defect fixes that are done and the functionality that were working with the earlier test cycles continue to work. A regular regression testing can use more than one product build for the test cases to be executed.

A final regression testing is done to validate the final build before release. The CM engineer delivers the final build with the media and other contents exactly as it would go to the customer. The final regression test cycle is conducted for a specific period of duration, which is mutually agreed upon between the development and testing teams. This is called the cook time for regression testing. Cook time is necessary to keep testing the product for a certain duration, since some of the defects (for example, memory leaks) can be unearthed only after the product has been used for a certain time duration. The product is continuously exercised for the complete duration of the cook time to ensure that such time-bound defects are identified. Some of the test cases are repeated to find out whether there are failures in the final product that will reach the customer. All the defect fixes for the release should have been completed for the build used for the final regression test cycle. The final regression test cycle is more critical than any other type or phase of testing, as this is the only testing that ensures the same build of the product that was tested reaches the customer. Regression testing can be illustrated in the below figure.

When to do Regression Testing

Regression testing is done between test cycles to find out if the software delivered is as good or better than the builds received in the past. As testing involves large amount of resources (hardware, software, and people), quick testing is needed to assess the quality of build and changes to software. It is necessary to perform regression testing when

1. A reasonable amount of initial testing is already carried out.
2. A good number of defects have been fixed.
3. Defect fixes that can produce side-effects are taken care of.

Regression testing can be performed irrespective of which test phase the product is in. A defect tracking system is used to communicate the status of defect fixes amongst the various stake-holders. When a developer fixes a defect, the defect is sent back to the test engineer for verification using the defect tracking system. The test engineer needs to take the appropriate action of closing the defect if it is fixed or reopening it if it has not been fixed properly. In this process what may get missed out are the side-effects, where a fix would have fixed the particular defect but some functionality which was working before has stopped working now. Regression testing needs to be done when a set of defect fixes are provided. To ensure that there are no side-effects, some more test cases have to be selected and defect fixes verified in the regression test cycle. Thus, before a tester can close the defect as fixed, it is important to ensure that appropriate regression tests are run and the fix produces no side-effects. It is always a good practice to initiate regression testing and verify the defect fixes. Else, when there is a side-effect or loss of functionality observed at a later point of time through testing, it will become very difficult to identify which defect fix has caused it.

It is clear that regression testing is both a planned test activity and a need-based activity and it is done between builds and test cycles. Hence, regression test is applicable to all phases in a software development life cycle (SDLC) and also to component, integration, system, and acceptance test phases.

How to do Regression Testing

The failure of regression can only be found very late in the cycle or found by the customers. Having a well-defined methodology for regression can prevent such costly misses. There are several methodologies for regression testing that are used by different organizations. The objective of this section is to explain a methodology that encompasses the majority of them. The methodology here is made of the following steps.

1. Performing an initial "Smoke" or "Sanity" test
2. Understanding the criteria for selecting the test cases
3. Classifying the test cases into different priorities
4. A methodology for selecting test cases
5. Resetting the test cases for test execution
6. Concluding the results of a regression cycle

Performing an Initial "Smoke" or "Sanity" Test

Whenever changes are made to a product, it should first be made sure that nothing basic breaks. Smoke testing consists of

1. Identifying the basic functionality that a product must satisfy;
2. Designing test cases to ensure that these basic functionality work and packaging them into a smoke test suite;
3. Ensuring that every time a product is built, this suite is run successfully before anything else is run; and
4. If this suite fails, escalating to the developers to identify the changes and perhaps change or roll back the changes to a state where the smoke test suite succeeds.

Understanding the Criteria for Selecting the Test Cases

Having performed a smoke test, the product can be assumed worthy of being subjected to further detailed tests. The question now is what tests should be run to achieve the dual objective of ensuring that the fixes work and that they do not cause unintended side-effects. There are two approaches to selecting the test cases for a regression run. First, an organization can choose to have a constant set of regression tests that are run for every build or change. In such a case, deciding what tests to run is simple. But this approach is likely to be sub-optimal because

1. In order to cover all fixes, the constant set of tests will encompass all features and tests which are not required may be run every time; and
2. A given set of defect fixes or changes may introduce problems for which there may not be ready-made test cases in the constant set. Hence, even after running all the regression test cases, introduced defects will continue to exist.

1. The defect fixes and changes made in the current build;
2. The ways to test the current changes;
3. The impact that the current changes may have on other parts of the system; and
4. The ways of testing the other impacted parts.

1. Include test cases that have produced the maximum defects in the past
2. Include test cases for a functionality in which a change has been made
3. Include test cases in which problems are reported
4. Include test cases that test the basic functionality or the core features of the product which are mandatory requirements of the customer
5. Include test cases that test the end-to-end behavior of the application or the product
6. Include test cases to test the positive test conditions
7. Includes the area which is highly visible to the users

The selection of test cases for regression testing depends more on the impact of defect fixes than the criticality of the defect itself. A minor defect can result in a major side-effect and a defect fix for a critical defect can have little or minor side-effect. Hence the test engineer needs to balance these aspects while selecting test cases for regression testing.

Classifying Test Cases

When the test cases have to be selected dynamically for each regression run, it would be worthwhile to plan for regression testing from the beginning of project, even before the test cycles start. To enable choosing the right tests for a regression run, the test cases can be classified into various priorities based on importance and customer usage. For example, we could classify the test cases into three categories, as shown in the below figure.

• Priority-0 These test cases can be called sanity test cases which check basic functionality and are run for accepting the build for further testing. They are also run when a product goes through a major change. These test cases deliver a very high project value to both to product development teams and to the customers.
• Priority-1 Uses the basic and normal setup and these test cases deliver high project value to both development team and to customers.
• Priority-2 These test cases deliver moderate project value. They are executed as part of the testing cycle and selected for regression testing on a need basis.

Methodology for Selecting Test Cases

Once the test cases are classified into different priorities, the test cases can be selected. There could be several right approaches to regression testing which need to be decided on "case to case" basis. There are several methodologies available in the industry for selecting regression test cases.

Case 1 If the criticality and impact of the defect fixes are low, then it is enough that a test engineer selects a few test cases from test case database (TCDB) - a repository that stores all the test cases that can be used for testing a product.

Case 2 If the criticality and the impact of the defect fixes are medium, then we need to execute all Priority-0 and Priority-1 test cases. If defect fixes need additional test cases (few) from Priority-2, then those test cases can also be selected and used for regression testing. Selecting Priority-2 test cases in this case is desirable but not necessary.

Case 3 If the criticality and impact of the defect fixes are high, then we need to execute all Priority-0, Priority-1 and a carefully selected subset of Priority-2 test cases.

The above cases are illustrated in the below figure.

The above methodology requires that the impact of defect fixes be analyzed for all defects. This can be a time-consuming procedure. If, for some reason, there is not enough time and the risk of not doing an impact analysis is low, then the alternative methodologies given below can be considered.

• Regress all For regression testing, all priority 0, 1, and 2 test cases are rerun. This means all the test cases in the regression test bed/suite are executed.
• Priority based regression For regression testing based on this priority, all priority 0, 1, and 2 test cases are run in order, based on the availability of time. Deciding when to stop the regression testing is based on the availability of time.
• Regress changes For regression testing using this methodology code changes are compared to the last cycle of testing and test cases are selected based on their impact on the code (gra box testing)
• Random regression Random test cases are selected and executed for this regression methodology.
• Context based dynamic regression A few Priority-0 test cases are selected, and based on the context created by the analysis of those test cases after the execution (for example, findnew defects, boundary value) and outcome, additional related cases are selected for continuing the regression testing.

Resetting the Test Cases for Regression Testing

After selecting the test cases using the above methodology, the next step is to prepare the test cases for execution. For proceeding with this step, a test case result history is needed. In a large product release involving several rounds of testing, it is very important to record what test cases were executed in which cycle, their results, and related information. This is called test case result history. A method or procedure that uses test case result history to indicate some of the test cases be selected for regression testing is called a reset procedure. Resetting a test case is nothing but setting a flag called not run or execute again in the test case database (TCDB).

Resetting of test cases, is not expected to be done often, and it needs to be done with the following considerations in mind.

1. When there is a major change in the product.
2. When there is a change in the build procedure which affects the product.
3. Large release cycle where some test cases were not executed for a long time.
4. When the prodcut is in the final regression test cycle with a few selected test cases.
5. Where there is a situation, that the expected results of the test cases could be quite different from the previous cycles.
6. The test cases relating to defect fixes and production problems need to be evaluated release after release. In case they are found to be working fine, they can be reset.
7. Whenever existing application functionality is removed, the related test cases can be reset.
8. Test cases that consistently produce a positive result can be removed.
9. Test cases relating to a few negative test conditions (not producing any defects) can be removed.

A rerun state in a test case indicates low risk and reset status represents medium to high risk for a release. Hence, close to the product release, it is a good practice to execute the "reset" test cases first before executing the "rerun" test cases. Reset is also decided on the basis of the stability of the functionalities. If you are in Priority-1 and have reached a stage of comfort level in Priority-0 (say, for example, more than 95% pass rate), then you do not reset Priority-0 test cases unless there is a major change. This is true with Priority-1 test cases when you are in the Priority-2 test phase.

Regression testing between component test cycles uses only Priority-0 test cases. For each build that enters the test, the build number is selected and all test cases in Priority-0 are reset. The test cycle starts only if all Priority-0 test cases pass.

After component testing is over, if regression is performed between integration test cycles, Priority-0 and Priority-1 test cases are executed. Priority-1 testing can use multiple builds. In this phase, the test cases are "reset" only if the criticality and impact of the defect fixes and feature additions are high. A "reset" procedure during this phase may affect all Priority-0 and Priority-1 test cases.

Priority-2 testing starts after all test cases in Priority-1 are executed with an acceptable pass percentage as defined in the test plan. In this system testphase, the test cases are "reset" only if the criticality and impact of the defect fixes and feature additions are very high. A "reset" procedure during this phase may affect Priority-0, Priority-1, and Priority-2 test cases.

Regression testing uses a good number of test cases which have already been executed and are associated with some results and assumptions on the result. A "reset" procedure gives a clear picture of how much of testing still remains, and reflects the status of regression testing. If test cases are not "reset," then the test engineers tend to report a completion rate and other results based on previous builds. This is because of the basic assumption that multiple builds are used in each phase of the testing and a gut feeling that if something passed in the past builds, it will pass in future builds also. Regression testing does not go with an assumption that "Future is an extension of the past." Resetting as a procedure removes any bias towards test cases because resetting test case results prevents the history of test cases being viewed by testers.

Concluding the Results of Regression Testing

Apart from test teams, regression test results are monitored by many people in an organization as it is done after test cycles and sometimes very close to the release date. Developers also monitor the results from regression as they would like to know how well their defect fixes work in the product. Hence, there is a need to understand a method for concluding the results of regression.

Since regression uses test cases that have already executed more than once, it is expected that 100% of those test cases pass using the same build, if defect fixes are done right. In situations where the pass percentage is not 100, the test manager can compare with the previous results of the test case to conclude whether regression was successful or not.

• If the result of a particular test case was a pass using the previous builds and a fail in the current build, then regression has failed. A new build is required and the testing must start from scratch after resetting the test cases.
• If the result of a particular test case was a fail using the previous builds and a pass in the current build, then it is safe to assume the defect fixes worked.
• If the result of a particular test case was a fail using the previous builds and a fail in the current build and if there are no defect fixes for this particular test case, it may mean that the result of this test case should not be considered for the pass percentage. This may also mean that such test cases should not be selected for regression.
• If the result of a particular test case is a fail using the previous builds but works with a documented workaround and if you are satisfied with the workaround, then it should considered as a pass for both the system test cycle and regression test cycle.
• If you are not satisfied with the workaround, then it should be considered as a fail for a system test cycle but may be considered as a pass for regression test cycle.

Best Practices in Regression Testing

Regression methodology can be applied when

1. We need to assess the quality of product between test cycles (both planned and need based);
2. We are doing a major release of a product, have executed all test cycles, and are planning a regression test cycle for defect fixes; and
3. We are doing a minor release of a product (support packs, patches, and so on) having only defect fixes, and we can plan for regression test cycles to take care of those defect fixes.

Practice 1:Regression can be used for all types of releases

There can be multiple cycles of regression testing that can be planned for every release. This applies if defect fixes come in phases or to take care of some defect fixes not working with a specific build.

Practice 2: Mapping defect identifiers with test cases improves regression Quality

When assigning a fail result to a test case during test execution, it is a good practice to enter the defect identifier(s) (from the defect tracking system) along so that you will know what test cases to be executed when a defect fix arrives. Please note that there can be multiple defects that can come out of a particular test case and a particular defect can affect more than one test case.

Even though ideally one would like to have a mapping between test cases and defects, the choice of test cases that are to be executed for taking care of side-effects of defect fixes may still remain largely a manual process as this requires knowledge of the interdependences amongst the various defect fixes.

As the time passes by and with each release of the product, the size of the regression test cases to be executed grows. It has been found that some of the defects reported by customers in the past were due to last-minute defect fixes creating side-effects. Hence, selecting the test case for regression testing is really an art and not that easy. To add to this complexity, most people want maximum returns with minimum investment on regression testing.

Practice 3: Create and execute regression test bed daily

To solve this problem, as and when there are changes made to a product, regression test cases are added or removed from an existing suite of test cases.This suite of test cases, called regression suite or regression test bed, is run when a new change is introduced to an application or a product. The automated test cases in the regression test bed can be executed along with nightly builds to ensure that the quality of the product is maintained during product development phases.

Practice 4: Ask your best test engineer to select the test cases

The knowledge of defects, products, their interdependences and a well-structured methodology are all very important to select test cases. These points stress the need for selecting the right person for the right job. The most experienced person in the team or the most talented person in the team may do a much better job of selecting the right test cases for regression than someone with less experience. Experience and talent can bring in knowledge of fragile areas in the product and impact the analysis of defects.

Practice 5: Detect defects, and protect your product from defects and defect fixes

Another aspect related to regression testing is "protecting your product from defect fixes." As discussed earlier, a defect that is classified as a minor defect may create a major impact on the product when it gets fixed into the code. It is similar to what a mosquito can do to humans (impact), even though its size is small. Hence, it is a good practice to analyze the impact of defect fixes, irrespective of size and criticality, before they are incorporated into the code. The analysis of an impact due to defect fixes is difficult due to lack of time and the complex nature of the products. Hence, it is a good practice to limit the amount of changes in the product when close to the release date. This will prevent the product from defects that may seep in through the defect fixes route, just as mosquitoes can get into the mosquito net through a small hole there. If you make a hole for a mosquito to get out of the net, it also opens the doors for new mosquitoes to come into the net. Fixing a problem without analyzing the impact can introduce a large number of defects in the product. Hence, it is important to insulate the product from defects as well as defect fixes. If defects are detected and the product is protected from defects and defect fixes, then regression testing becomes effective and efficient.

Problems and Exercises

We will complete the following problems and exercises during lecture or during lab time.

1. A product for automating payroll is slated to be released in March. The following are some of the features. For these, identify which of them would you put through smoke tests in the three months leading up to the product release in March, assuming that there will be one more maintenance release in September.
   • Screens for entering employee information
   • Statutory year-end reports
   • Calculating the pay slip details for a month
   • Screens for maintaining address of the employee
2. A product has some test cases that are rarely run, ostensibly "because of lack of time." Discuss the pros and cons of running such dormant tests as a part of regression.
3. Which of the test cases given below for inclusion in a given regression cycle?
   • Version 7 of the product is being released and there are tests cases for new features to be introduced in this version.
   • Version 6 had a number of defects in the buffer management module and Version 7 builds on top of this module.
   • Version 6 had a fairly stable (i.e., very few defects) in the user interface module and there have been very little changes in this module in Version 7.
   • Version 7 went through a Beta test cycle and during Beta tests, some important customer usage scenarios were found to uncover defects in the product.
   • The product is a database software and has features like database startup, basic SQL query and multi-site query.
4. We discussed ways of prioritizing test cases. Using the suggestions given in the text, prioritize the test cases as P0/Pl/P2.
   • A test cases for a network product that tests basic flow control and error control.
   • A test case for a database software that tests all the options of a join query.
   • A test case for a file system that checks allocation of space that is not contiguous.
   • A test case that tests the startup of OS with normal parameters.
   • A test case that tests the startup of an application in a "safe mode," something that is not done very often.
   • A test case corresponding to a feature that has undergone significant change in the current version.
   • A test case corresponding to a stable feature that has not uncovered any major defects.
5. If during regression run, you are hard pressed for time, discuss the pros and cons of the various methodologies of regress (regress all, regress changes, etc) discussed in the text.
6. A regression test engineer presented the following results. Discuss the efficacy of the choice of regression tests and of the development process:
   • The tests for print facility produced no defects in the previous version and has not produced any defects in this version.
   • The defects in tests for multi-currency continue in the current version.
   • The feature expedited delivery had new defects in the current version.