Regression Testing Out of the Blue: Facing the Unexpected

Originally published: February 18, 2016 Updated: July 01, 2022 7 min. read

Software Engineering

Quick fact: eventually all projects get to the point when full-scale regression testing should be conducted. Regression testing cycle ensures that 98% of the functionality is covered in accordance to the predetermined specs. However, even in the best-case scenario, it is a quite challenging and time-consuming task, especially when performed manually. Moreover, tension intensifies if necessary preparations could not be carried out on time. That was exactly the case with our recent regression testing project. In the following blog post, we will describe some practical lessons learned from the project.

Backstory

Our customer faced an urgent matter that had to be addressed immediately. Previously our team had integrated new functionality into the customer’s legacy software. Therefore, integration aspects affected almost the entire application. It was high time to deploy a new app version to production. Naturally, there was a need to conduct full-scale regression testing before the new version release.

Regression Testing Out of the Blue: Facing the Unexpected - Infopulse - 016517

As time was pressing, only five weeks could be given by the customer to conduct testing in four different environments. Two specialists (Senior and Junior QA Engineers) were allocated to perform the task. The customer supplied us with a great lot of old product backlog items (PBIs) – a heritage from the previous developers of this legacy software. Unfortunately, the test cases were not automated and the testing had to be conducted manually. We added some more PBIs, specifically designed for the task by our team. In total, we approached this task having 358 PBIs, which were divided into groups based on priority. To complete the project on time, we would need to go through 16 PBIs per day.

Task Execution

First, we had to design a testing plan within a constrained period of time.

The initial option was to distribute PBIs among team members. Each specialist would take one PBI and run it in each of four testing environments. Then Senior QA Engineer had to check the results presented by Junior. The same would be valid for each PBI.

We canceled this plan almost immediately for a number of reasons:

Junior would be simply unaware of all functions;
Work would be out of sync – due to Junior/Senior experience gap;
Different number of test cases in each PBI – thus different time was required to complete test runs;
By the end of the first day, we barely managed to complete 10 PBIs out of 16 planned.

We had to design a new testing model from scratch. This time we divided test environments. With two separate environments per specialist, this model had the following advantages:

Both Senior and Junior QA specialists go through all test cases, albeit in separate environments;
We read and analyzed each test case together, unpacking and solving any possible questions or issues before testing;
This helped us to reach the same speed of test cases completion;
Syncing work time immediately resulted in easier control over the processes;
Each engineer goes through all functions, nothing is missed or skipped;
Additional verification of Junior’s results became unnecessary.

Another objective, offered by Infopulse Testing Team Lead, was to track stats, namely the number of test cases in each PBI, along with the number of completed PBIs and test cases in a separate chart. If test cases were out of date, we had to determine time required to update them. We also created a list of obsolete PBIs to be removed from the Database.

Unforeseen Consequences

While the new testing plan was much better than the first one, by the end of the first week we were lagging behind by 12 PBIs despite all our efforts. Some of the legacy PBIs contained 200+ test cases. Multiplied by four test environments, that resulted in over 900 test cases per PBI! Half a day would take only to open each of 900 test cases.

As the time was running out and the load increased, we initiated a number of meetings with our client and agreed to drastically increase our work force by inviting four QA specialists from our team.

We were taking huge risks here. Too much time could have been wasted during the adaptation period. Still, as John Kennedy said, “But however close we sometimes seem to that dark and final abyss, let no man of peace and freedom despair. For he does not stand alone.” With increased forces, we saved the day – regression testing was successfully performed to its victorious end almost on time.

Conclusions

If you want to achieve good quality of testing, avoid stress and achieve your goals on time, you need to allocate a decent time-frame to the planning stage. And always keep your test cases in a good state!

Set the priorities! Test cases with the highest priority should be immediately moved to the start of the regression testing plan. This would allow for easier and more efficient planning, if, e.g., unplanned regression testing is required;
Exploratory testing has to be conducted beforehand, based on the test cases prioritization;
Instead of planning a number of completed PBIs per day, set your plan around the number of test cases;
If you have to conduct regression testing and you have to take test cases from elsewhere, take into account the probability of extraordinary situations. Be aware that test cases can be obsolete, outdated, overly detailed, etc. Learn how much time you used to update the outdated or to mark those that where obsolete – and not done by yourself. Otherwise, additional clarifications will be required in future again, and that consumes extra time and efforts;
Always try to document current state of the test cases. It is very important to track amount of time it took to update the outdated test cases. Such statistics will come in handy when planning future updates of test cases and will facilitate preparations for the next regression testing cycles;
Always mark obsolete test cases with relevant tags including the reasons for that – at least briefly, so that new specialists in future don’t waste precious time;
In the end of the sprint, out of your lessons learned, create a separate regression suite for the test cases that were required for this particular regression session. Only the most important and only positive test cases should be used. During the transfer of the project to support, the team will perform only those tests that were selected in advance.