The word continuous has become ubiquitous in the realm of Agile development and quality assurance. Continuous delivery, continuous integration, continuous testing, continuous deployment, continuous improvement – what do these terms really mean?

If you look at an Agile life cycle that includes steps to plan, prepare, explore, realize, test, deploy and run, continuous integration comes early on, aligned with developers. With all the pockets of new development happening in Agile environments, there’s a lot of changes being made to the code, creating a heightened need for version control. Continuous integration was born out of a need to have a central place to save and store that code change, such as GitHub, and compile that code outside of their machines in the continuous integration server. This process creates the opportunity to match all the other dependencies and things that other people are building. And that's where unit tests live. Continuous integration includes things like ADO, Jenkins, freestyle and pipeline builds. 

Continuous testing generally occurs at the realize and testing phases. This process involves testing the actual functionally to ensure it’s working end-to-end, with all of the various integrations and dependencies. The big difference between unit testing and continuous testing is that unit-level testing involves non-deployed code level testing versus deployed testing itself. In other words, continuous testing validates processes the way they actually function, interacting with other applications in a real-world environment vs. isolated in a siloed application.  

And then continuous deployment is all about moving from pre-production environments into production. It involves moving the code and connecting the applications to the databases. But without continuous testing, this step falls down. Why? Because you’re writing and pushing new code quickly and without consideration for other dependencies. Nothing shoves defects into production like a fast pipeline without adequate testing.

The combination of continuous delivery and continuous integration is often referred to as CI/CD. Continuous delivery combines the practices of continuous integration, continuous testing and continuous deployment, creating a best practices framework to support rapid, concurrent and ongoing innovation. 

With so many concurrent changes, there are some key considerations for testing in a CI/CD environment:

1. Address Version Control 

Continuous integration efforts aim to keep everything under version control. With many changes happening at once, it is important to know which version of code is being run in production or what's in a specific build. A big key consideration for CI/CD is source control and change lists. 

In traditional waterfall methodology, the process of introducing change and testing is all very linear and traditional requirements for test workloads are in place. When we move to Agile, suddenly our requirements are not documented discreetly. Often, they're just simply acceptance criteria and a story. In an advanced pipeline or advanced DevOps world, when the developers get a story saying, JIRA ADO, the associates or code check-ins assume you should know what to test. But the change lists are not highly correlated to the test management system. There isn’t that tight coupling we used to have. 

When it comes to having multiple sprints working in parallel, they may not be on the same Agile board. They may have different projects in JIRA or ADO. So now there are lots of change lists happening in parallel. That’s why we need to do unit-level testing with every build. And we need to conduct continuous testing every day because multiple people are making changes and they may not be aware of each other. You should also conduct tests in a true copy of your production environment to ensure quality for changes that are pushed into production.

2. Recognize the Impact of Change 

Understanding the overall business process is not something that lies with the developers. Developers work with granular requirements, thinking about very small features and stories. Business users think in terms of themes and how things work together for the larger function. Individuals in a development role may not realize that a change upstream affects something downstream or downstream change requires an upstream business change. That’s why the business users need to be involved in the validation process because they understand the context of how the story rolls into the features and in themes and epics. 

To that end, it’s important to note that business processes are often complex. A single process can actually be hundreds of steps. For example, a return on an item has many things that happen upstream, like inventory management, invoice payment, etc. When engaging new customers, we often hear challenges like “we really struggled with knowing exactly what to test. And because of Agile, we're working on little pieces of code at a time. We may not have that big picture.”  With rapid deployments in complex applications like SAP Fiori, Salesforce, SuccessFactors, the whole process becomes a lot more complicated itself. 

“In order to start our journey with DevOps, the first piece of the puzzle we needed was Worksoft automation. We needed to be able to do end-to-end testing across applications to get fast feedback for our developers and ensure our cloud environments were viable.” — Shane Co.

3. Identify & Resolve Defects Fast

One of the key principles of continuous integration is to commit early and often. Work on small bits of features at a time and commit them. That way if you have a breaking change, you didn't have 4,000 lines of code changing, you had four lines of code changing. So test all along with those changes, and identify and fix the errors immediately. 

If you accumulate the errors, you forget which codes you changed and it’s difficult to understand why something broke. With automated continuous testing of end-to-end business processes, your defect resolution can be faster and more effective. Scheduling functional and regression tests to run continuously in overnight or off-hours with remote execution can help keep both your project timelines and testing on track. 

“As you’re doing an Agile program and you get further and further into your program increments and your sprints, your amount of regression testing gets enormous. In past Agile projects we’ve either struggled to complete all those regression tests or do it to the depth or quality that we wanted to. So one area where we’ve already seen value with Worksoft is in those automated regression tests.” – Hershey Company

4. Test at Every Stage

What should you run in your tests when? In the early stages of development, sprint teams and developers will focus on unit-level, API tests. At this stage, things may not actually be deployed with the database in the backend. Next, layer in functional regression tests, which implies you’ve deployed the application. At this advanced stage it’s time for continuous testing. The end-to-end tests are coming into play and you’re making sure that you’re running in the databases and the servers and the clients where all the security updates and maintenance changes in place. 

Remember, developers are generally focused exclusively on unit API tests. They may not know that there was a security patch going in on Friday night, or that there was an upgrade to an ERP package like SuccessFactors. Even with continuous integration checks and balances in place, the scope of understanding the big picture is often limited for developers.

5. Engage Automation for Speed & Quality

When it comes to the difference between pre-production and production and the continuous deployment to actually move the code, the need for continuous testing is paramount. Creating a center of excellence to run tests as part of the pipeline at this more advanced stage is becoming a popular best practice. 

Rewriting and reinventing test scripts bogs down the continuous delivery process. Rather than abandon testing and sacrifice quality for speed, engaging automated testing empowers Agile organizations to keep pace with rapid change. And using Worksoft automated testing with assets that are reusable and scalable supports more efficient QA and change management, creating a significant advantage for the continuous integration lifecycle. 

“We moved from manual regression testing, once a month, to continuous testing every second day, with the same scope or even an increased scope now. We don't need to focus on the manual regression tests, so we've freed up hands for support activities and new developments. We have become more efficient and more flexible and save about 160 hours of manpower every month because we use Worksoft Certify.” – IT Central Station Review

About Worksoft

Worksoft automation empowers worldwide leaders to keep pace with persistent, pervasive change. Our Connective Automation platform connects process intelligence, test automation, and RPA to maximize efficiency and ROI for progressive enterprises. Our platform delivers continuous testing and continuous process improvement to ensure flawless execution of complex business processes. To learn more visit