Integration of QA in OSS/BSS Development for Telcos: Best Practices

Lagging technological know-how and lengthy software development and testing cycles often stand in the way of steady progress. One of the crucial ways to ensure faster OSS/BSS product releases is rethinking your approach to quality assurance (QA).

The Increased Role of QA in Telecom

Routing protocol testing has traditionally been the focus of telecom testing. Ensuring secure, low-latency data transmissions from one network to another is indeed crucial for stable operations and high customer satisfaction. Many also prioritized signaling testing of SS7 and SIP protocols as both became attractive targets for hackers.

However, progressive telecom sector digitalization has significantly transformed how telecoms operate “under the hood”. Today, market leaders are not just in the business of supplying connectivity services — voice calls, SMS, Internet — but maintain a vast collection of digital products and services such as:

• Cloud-native OSS apps
• OTT media services
• Mobile money products
• Cloud storage and email accounts
• On-demand private network
• Services and products for IoT.

The digital and customer-facing nature of such offerings warrants a more comprehensive quality assurance process. In general, 93% of telecom leaders agree that testing improves customer satisfaction and reduces churn. Yet, far fewer manage to release fully bug-free software on the first try.

Unlike other industries, telecom has long relied on manual approaches to network and software testing—and expressed reluctance to adopt DevOps best practices for quality assurance. Such practices assume implementation of continuous, automated testing as quality gates in the CI/CD pipeline.

However, telecoms now feel the heightened pressure to bring new software to the market faster, release product updates more frequently, and ensure topmost security against ever-evolving threats.

For this change to happen, most will need to make a critical shift from “traditional” to “digital-first” providers — and this shift also assumes rethinking your approach to software development and testing.

Shifting from Traditional CSP to DSP

Source

The new generation of “digital” players has designed their software development process with continuous integration and continuous development (CI/CD) in mind.

As Hemant Patil, a Technology Consultant with Ericsson explains:

In telecom, the CI/CD philosophy means that software is developed, delivered, tested, accepted, and brought into operation incrementally at a far higher cadence than previously in a traditional service provider environment. Further, it creates a significant software development volume that needs validation on an increasingly dynamic network. This approach implies that continuous software validation and continuous testing must accompany continuous software delivery and deployment. 

Hemant Patil

Technology Consultant with Ericsson

Ericsson themselves shifted to a continuous approach to software delivery in 2017. After embedding CI/CD principles into their software development life cycle (SDLC), the telecom went from requiring 7 weeks to test and 6 months to deploy new software to spending only 90 minutes on testing and 3 weeks on deployment. Though timelines shrank, shipped software quality only improved.

Ericsson’s recipe for success: Fuse software development, testing, and deployment into one continuous flow and embrace automation testing tools for telecom.

However, the best part is that smaller telcos can obtain likewise massive gains by embracing a similar set of Agile QA practices.

Best Agile QA Practices for Telecoms

For over two decades, Infopulse has been supporting telecom clients through different stages of operational and technological transformations. Subsequently, we have set up a telco-focused QA expertise hub — a collection of people, processes, and technologies we leverage to ensure top-most product quality.

In this post, we share the main practices that guide our work, which other telecom tech teams could also adopt.

1. Perform Baseline Test Analytics

Testing automation is a means to solve a problem (lengthy and complex bug detection), not a solution in a nutshell. Hence, before you automate, you need to understand what types of issues you are trying to prevent and protect against.

Telecoms have an inherent disadvantage of “legacy code” and custom-built OSS/BSS systems that are hard to maintain and easy to break when introducing new system components. Adding anything new atop of such a “shaky” base is risky unless you can rapidly test how the new code will impact other system elements.

Therefore, to achieve the best outcomes from QA automation, you should always start with system requirements analysis. In our case, a telecom-specialized QA Analyst first gets down to understand data flows, reliability, security, and performance requirements. Telecom domain knowledge helps create product-specific test scenarios, covering common user behaviors and critical system performance characteristics.

Based on the initial analysis, we then map optimal testing scenarios and suggest required test types. Then, the team decides which testing scenarios should be automated.

2. Implement a Test Automation Pyramid

An Agile test automation pyramid is a schematic representation of layering various types of automated tests. The model organizes test types into three layers, based on their speed and impact on end-product quality.

The Ideal Testing Automation Pyramid

Source

A standard testing pyramid includes unit tests at the base. Unit tests help verify that individual units of source code (software components) are fit for being used in your application, i.e., they do not break the existing functionality or interfere with other software architecture decisions.

Next, comes a series of integration tests, designed to test individual software modules as a group to verify their compatibility with specified functional requirements (business rules, authentication standards, different authorization configurations, etc).

As the last step, the produced software goes through a series of end-to-end (E2E) tests, where the entire product gets tested in close to real-world scenarios.

Why do we advocate using a testing pyramid and continuous testing principles?

• Fast feedback cycle. Critical software defects get identified and fixed at early development stages. This prevents costly code re-works at later stages and bugs in production, which, in turn, require extra operational support and can increase app downtime.
• Confident refactoring. Developers can focus on improving the product functionality and shipping new features, instead of resolving earlier bugs and critical vulnerabilities.
• Better documentation. When test cases “sit” close to the code, they are easier to interpret by developers and QA engineers alike. This way newcomers can get productive much faster.
• Quality-by-design. Continuous testing prompts developers to write "clean code" with fewer dependencies on other system features or custom workarounds. Clean code is easier to understand, maintain, and then reuse for other system components. When everything is unified, building and shipping new bug-free software becomes a standard.

We also encourage all telecom teams to make Test Automation an essential part of Definition of Done (DoD) — an acceptance criterion for produced software.

3. Employ Smart Test Automation Principles

The abundance of all-purpose and telecom automation testing tools on the market makes some teams think that a 100% automation test coverage rate should be the ultimate target.

We must disagree. Do not strive to automate tests for every requirement just for the sake of hitting a benchmark. If one automation test takes 10+ hours to make but only gets deployed once per year, it can be done manually.

Instead, aim to create test cases that are:

• Self-contained, meaning different tests can be run in parallel across environments to speed up the QA process.
• Purpose-built, i.e., the test validates one thing and does it well. However, good test cases need to have reusable components so that you can replicate them when needed.

Test automation is a continuous process. Explore different scenarios and candidates until you reach the so-called plateau of diminishing returns.

Line of Diminishing Returns

Source

4. Look for QA Engineers with Domain Expertise

CI/CD is a universal methodology, used by software development teams across domains. Because of this, there are plenty of QA engineers that are familiar with test automation. Yet, far fewer pros can grasp the specifics of testing telecom applications.

To ensure comprehensive test coverage, you need to assemble a QA team that understands the complex business logic of telecom OSS/BSS products. That is why working with “generalists'' is not an option.

In practice, this translates to having people who can de-code the convoluted logic of telecom business products, system integrations, and networking requirements. A telecom-focused QA engineer can rapidly grasp the standard data flows and application usage scenarios. Then evaluate different testing approaches and develop the right set of testing scenarios.

Additionally, they must understand the security implications of working with telecom testing data, i.e., never using raw production data for testing to avoid privacy issues. As a result, you have a smaller talent pool to choose from.

5. Appoint a Dedicated Test Manager

A Test Manager takes over the ownership of all QA activities design, implementation, and execution. They sit at the helm of your automated testing efforts by progressively analyzing proposed testing scenarios and prioritizing candidates for automation.

Their main goal: advocate for the implementation of consistent QA/QC standards for all products in the telecom’s OSS/BSS portfolio.

However, a Test Manager should not be the sole bearer of responsibility for the compromised quality standards. In agile, QA is a joint, continuous process, shared between development and testing teams. The manager’s role is to ensure proper communication and collaboration between all team members and provide an aggregated view of the testing strategies to other project stakeholders. They drive the strategy, empower execution, and facilitate the QA team with day-to-day decisions and issue resolution.

Actions and real cases speak louder than words, and it is the productive team of Test Automation Specialist, Test Analyst, and Test Manager that enabled us to transform the QA practices ofour customer BICS.

From Theory to Practice: QA Excellence for BICS

In 2010, BICS and Infopulse shook hands on an IT services partnership deal. In 2016, we extended our collaboration until 2030. Over those decades, Infopulse has helped BICS  develop a modern portfolio of 60+ OSS/BSS applications, running in the cloud and hybrid environments.

Implementation of Agile QA practices has played a major role in the speed and success of new product developments. That is not just our opinion, however.

In 2021, Infopulse was selected as a finalist in European Software Testing Awards. The jury panel was impressed with Infopulse’s commitment to excellence and modern Agile transformation approaches in testing BICS's solutions that enable communications across the world.

Infopulse has taken over all QA-related processes for BICS — from resource organization to QA function management, practice updates, and issue resolution. Over the years, we have helped BICS develop an all-in-one center of QA expertise that now allows them to release competitive business products at high speed and without any compromises on quality.

We are eager to help other telecom companies improve their approach to quality assurance. Contact Infopulse to receive personalized consultation on proven approaches to telecom software testing.