Manual vs. Automated Software Testing: Balancing Between the Perks and Downsides

CD is hardly possible without a deeper level of automation. While software development practices are catching up, testing often becomes an operational bottleneck to increased efficiency through increased automation. The manual vs automated software testing debate isn’t new. And despite all the transformations, both approaches have their merit and place in today’s software development process.

This post offers a deep level overview of what is automated testing in software testing and how to do manual testing the right way in the age of DevOps and Continuous Development.

What is Manual Testing?

Manual testing is performed by QA engineers without the usage of any tools for automating the tests execution. Different types of case tests are created manually to identify possible flaws in the software code, as well as product design. In most cases, manual testing is not optional as every application must be checked “by hand” before any automation can take place. Despite being quite a time- and effort-demanding, manual testing remains popular among businesses. Per a recent industry survey, 42% of companies indicated that their testing activities are “mostly” or “entirely” manual.

However, certain software testing tools for manual testing can be used to speed up and improve the effectiveness of test coverage. Some of the popular manual testing tools are Postman, SoapUI, and Chrome Dev Tools. Though, these are also used in automated testing.

The Advantages of Manual Testing

• Less set up time and resources required to begin productive manual testing.
• Includes the assessment of overall User Experience (UX) e.g. when it comes to design elements, the convenience of navigation etc.
• Applicable to both small and large projects.
• Greater flexibility: you can change the course of testing in the middle of a test run to assess a certain element that was not previously considered.
• Manual testing is imperative to check for automation feasibility.

Types of Manual Testing

• Black Box Testing
• White Box Testing
• Exploratory Testing
• Integration Testing
• System Testing
• Acceptance Testing

What is Automated Testing?

As the name indicates, automated software testing assumes the use of tools to execute pre-scripted test case suites. The test results are sent for further programmatic analysis and the QA specialist can benchmark the expected vs. actual test results and review detailed reports.

However, QA engineers are not directly involved in the tests execution. They only develop and set up test cases at the preparation stage and assess the testing outcomes. Automated testing largely improves the ROI of QA as it’s aimed at reducing the number of test cases required to be run manually and repeatedly during a large project.

It is important, however, to understand the difference between automated testing and test automation.

• Automated testing assumes the usage of automation testing tools to perform specific tests instead of running them manually every single time. However, this process does not assume the complete exclusion of the human specialists and 100% reliance on technology for testing.
• Test automation is the programmed execution of predefined test cases – a set of manually defined algorithmic checks that evaluate a specific element of the software.

In the course of this post, we’ll dwell on test automation in particular, and how it should co-exist and supplement manual software testing.

What Are the Benefits of Automated Testing?

• Automated testing is approx. 70% faster than manual testing, meaning that you can run more tests to ensure better code quality and achieve faster time to market.
• Earlier bug discovery. Fixing a bug identified during Beta Testing stage is 4 times more expensive and time-consuming to fix than the one found on the Unit Testing stage.
• Automation ensures wider coverage of all the application elements. It allows businesses to attain greater consistency when it comes to coverage.
• Test automation is also reusable and can be deployed multiple times at different stages of the software development life cycle. For instance, automated regression checks can be performed after each build to ensure higher product quality.
• Increased team efficiency and morale.
• Easy and comprehensive reporting.

Why Test Automation is Essential for Continuous Delivery and Continuous Testing

As more companies embrace DevOps and continuous delivery model, new approaches to testing become necessary. CD framework assumes that your software components are always in the deployment-ready state, meaning that you can no longer postpone testing to the very end of your project. A software change has to continuously move from Development to Testing to Deployment – and that’s where continuous testing comes as an essential measure.

Continuous testing ensures that all QA activities happen in smaller batches whenever the need arises, rather than at the end of a development cycle. According to a recent survey by CA Technologies, 75% of business leaders rank continuous testing as a critical or important component of their development lifecycle.

However, continuous testing is also a massive undertaking. It requires significant investment in new automation tools (which 63% of survey respondents often found too complex and/or expensive); plus stronger communication and collaboration effort among your teams and certain changes in the workflows – 73% of respondents strongly agreeing that continuous testing requires a major shift in mindset and culture.

But those efforts do pay off. Companies with a mature continuous testing framework in place reported the following benefits:

• 2.4x more confidence in product quality;
• 1.9x more confidence in the speed of delivery;
• Organization using continuous testing are 3.9x more likely to exhibit rapid revenue growth.

Test automation is an integral step for enabling continuous testing, as it allows to streamline, track and manage the growing number of conducted test cases. While testing can be automated without integrating continuous testing, the latter is impossible to implement without test automation.

The importance of automation testing becomes paramount in this case, as it helps ensure that all teams receive timely feedback, and maintain the same levels of quality standards during the entire production pipeline.

To achieve this, your test automation initiative should be aimed at filling the next imperatives:

• Obtain quick access to the required systems and environments for testing.
• Ensure that testing data is available whenever the need arises.
• Avoid the exposure of personally identifiable information in test data.

The common risk to account for when preparing for automation and ultimately continuous testing adoption is to avoid speeding up test data acquisition process just for the sake of gaining access to some data, without considering the security and compliance risks.

Moving from manual testing to automation cannot happen in a rush or in leaps. Having a strong business case for automation and a deep understanding of the pros and cons of automated software testing is mandatory when deciding on the new testing setup.

When Not to Automate Testing?

High initial adoption costs are often named among the main disadvantages of automation testing. Certainly, the technology investment costs may look steep at first, however, in the long run, the ROI of automated software testing proves to be positive for the most mid-to-large scale projects.

Scripts can run as much as 5 times more test per hour than humans, meaning approximately 15X more code gets covered per day. This significantly reduces the average cost of a testing hour. As mentioned already, automation tests allow to identify bugs earlier and reduce the costs of fixing those.

Still, there are certain scenarios when automation may not yield you the desired ROI:

• Your team lacks the necessary skills sets. If your in-house team does not possess relevant programming background, you may want to consider either investing in their training or outsourcing automated testing to a dedicated team with relevant expertise. Both cases will assume certain investments.
• You heavily rely on Waterfall project management style. Test automation for a complete, complex product may be a time/budget-heavy endeavor to pull off.
• Creating automated test cases for tests that will be run once also makes little sense. Every automation script suite should have an extended lifetime so that its building cost was lower than the manual execution cost.
• 100% automation is a myth. Only certain areas and types of tests can be fully automated. But certain areas of testing should still be done manually – UI testing, recovery or compatibility testing. Finally, you will still need experienced staff to build, manage and analyze all the testing activities and ensure that no product aspect is left uncovered.

When to Automate Testing: The Common Types of Automated Tests

Now let’s take a closer look at when automated testing is required. Tests can be automated on several different levels, tackling the various aspects of your application. First of all, you can deploy automation to assist you during different phases of testing:

1. Unit tests.

Unit tests are aimed at checking the smallest components of your application (a unit). These are the cheapest and faster to write, and relatively easy to maintain. Plus, they offer the highest ROI as the help ensure higher code quality on a granular level.

The new test-driven development (TDD) paradigm, for instance, assumes that programmers develop unit tests prior to writing code. Once the code is done, unit tests are deployed automatically to verify it. This way your team mitigates bugs, instead of fixing them. The only drawback is that unit tests do not prevent bugs in logic. Hence, they should be paired with API tests.

2. API tests.

API tests are performed during the integration phase. They can be scheduled by either the development or testing team to be executed before the UI layer is added to the product. API tests are the prime candidates for automation as API design can be controlled through a central definition (e.g. REST APIs are defined as OpenAPI definitions) and seamlessly used throughout the software lifecycle and the automation pipeline. By 2020, Smartbear predicts that 77% of QA engineers will automate more than half of their API tests.

3. GUI tests.

GUI tests are required to make sure that your interfaces are spot on. However, automating such tests requires the most time to create and properly maintain. There are both significant advantages and disadvantages of automated testing in this case, as GUI tests tend to result in more false positives/negatives than other types of automated tests.

To further determine when to use automated testing, you should also assess the different automation options based on the type of tests:

• Regression Testing
• Smoke Testing
• Load & Performance Testing: No viable manual alternative exists
• Security Testing
• Acceptance Testing
• Integration Testing
• Build Verification Testing (BVT)

For these types of cases the benefits of test automation over manual testing are rather apparent:

• Higher consistency/coverage and more accurate results.
• Faster, parallel execution.
• Lower costs compared to manual testing.

Mastering the Balancing Act of Manual vs. Automated Testing

The major difference between manual and automated testing is speed. Manual testers are certainly not as fast as automation software and can make blunt mistakes. But automated test cases are only as good as their creators are. Without a deep understanding of the product and certain creativity employed by the developer, automation tests will fail to identify a simple issue a human tester could immediately notice (e.g. a problem with misaligned titles).

As we mentioned earlier, testing can never be automated by 100%. There will always be a certain level of human involvement present. Businesses who manage to strike the right balance between manual vs automated testing gain the most benefits.

The main case for automation should be eliminating the repetitive, time-consuming tests that your team is performing frequently, and gradually “shift-left” – incorporate testing into everyone’s thinking as early and as easily as possible, beginning with requirements definition.

By embracing a deeper level of automation, your organization can unlock a wider scope of DevOps benefits, adopt the continuous delivery/integration model and deploy code up to 30% more frequently with 50% fewer bugs.

Infopulse team invites you to schedule a discovery session to assess your current testing setup and future needs and develop a custom framework for moving away from manual testing to automation when necessary. You can learn more about our testing services here.