These days it’s hard to have a conversation with anyone related to application delivery without the word “continuous” being brought up. If an application is not being developed, tested or deployed in some continuous fashion, the inevitable question is, “Why Not?” DevOps purists working on the software development team may say, “if you’re not testing within Continuous Integration cycles, then you’re not really testing.” Testers working within QA, who don’t want to be associated with the stigma of testing taking forever and stalling a project, will banter about the merits of Continuous Testing. But are both really needed? Are there differences in when and where Continuous Integration (CI) and Continuous Testing (CT) should be applied based on the system(s) under test?
Continuous Integration (CI) as the Best Approach
As a starting point, let’s say CI is the best approach. Every unit of code and every change is tested within the CI server. Change lists are committed to the CI Server, tests are correlated to the changes, and then tests validate the changes are functioning as designed. The change could be related to new functionality or to an update to existing functionality. The commitment of changes in this manner makes complete sense to developers who are used to working with tooling that is structured around versioning and merging changes. Since CI is used primarily in development and developers mostly write unit tests, unit tests are therefore the most common type of tests run as part of CI cycles. Each unit of code is tested with a corresponding versioned test. Tests are versioned and changed as the code is changed. The application is not deployed into production until each unit of code passes the associated test.
Continuous Integration is a newer, more modern approach to development. It delivers clear value around managing code changes and has been quickly and widely adopted by teams building custom applications. But, what does a CI change list look like when you’re dealing with a packaged application like SAP, where the software is already built and deployed?
What Does Continuous Integration Look Like for Packaged Applications?
Based on the type of packaged application and its hosting model (cloud, on-premise or hybrid), you may have the ability to add code to change functionality, or you may be limited to only making configuration changes without changing code. The question then becomes, “What does a change list look like for customizing or configuring a packaged application?” Code changes typically involve change lists whereas configuration changes may happen immediately without being checked into source control or merged like code. For configuration-only changes that can be packaged up and changes with associated unit level tests, the complexity lies in the tests moving up the stack from code level unit tests to API and end-to-end UI tests.
In both these examples defects and tests are easily traced back to the originating requirements and change list. A new requirement then generates an updated change list for the associated tests. Now, any time the requirement needs to be tested, the change list will be invoked and the tests will be verified. Many see this as the “shiny new toy” of testing and refer to it as risk-based testing. When a requirement changes, the change list is updated, and only then are the associated tests run. From a developer’s perspective, this is the pinnacle of DevOps and continuous change. Changes either fail fast or pass and automatically get pushed to production.
The Achilles’ Heel of Continuous Integration: Business Process Workflows
Continuous Integration works great when making small changes to atomic applications. But what happens when the feature or application needing changes is part of a larger business process workflow? How will this small, seemingly innocent change affect the larger end-to-end, cross-system business process? Does the change associated with the smaller, custom application that feeds into the mission critical system of the company really not need anyadditional testing beyond the smaller application’s CI development cycle? Is the smaller application’s requirements changes and change list sufficient to test all the consequences those changes may have on mission-critical, downstream systems?
The QA organization along with the business owners are crying out, “NO WAY!” Developers can’t just test the changes they are working on in isolation. Someone needs to look at the bigger picture and ensure the change is not going to break any of the upstream or downstream processes. This is where Continuous Testing moves to the forefront.
A Better Approach: Continuous Integration and Continuous Testing
Continuous Integration and Continuous Testing bolster one another to ensure small changes don’t break important business processes when no centralized control exists for the change list or manifest of changes. Typically, within large organizations there are many moving parts and multiple actors making changes. Some changes could be occurring on the front-end to custom web storefronts, while others could be on the ERP system that sits in the middle handling all of the transactions, and still others on the back-end legacy mainframe. For the end-to-end process that requires all of these systems to work together seamlessly, there is neither a “nice little” change list nor a direct correlation between the requirements and changes currently documented. Continuous Testing implies that the functional integration tests are executed against a deployed integrated systems environment like QA in contrast to Continuous Integration unit tests, which are typically executed against a code-merged, integrated DEV environment. Because there are more steps to deploying and joining end-to-end, enterprise business processes, the Continuous Testing cycle may run every day, whereas Continuous Integration tests run on every code commit or build.
When Unsure, Test Continuously.
When it’s hard to know what parts of the enterprise are changing or how the different parts interact with one another, Continuous Testing is a great approach. Running comprehensive tests continuously reduces the risk of a small change in the middle of a process causing a big failure in related, larger processes which could potentially bring down a production application or mission critical business process. Continuous Testing provides the assurance to the organization that its most important and complex, end-to-end processes are validated thoroughly and often.
Continuous Testing also helps find those “pesky” inter-application dependencies that most people in the organization can’t predict, understand or consistently document. For example, changing the date format for a localization project can “blow-up” code that does not understand how to process a timestamp with an included time zone. This could easily happen upstream or downstream of a change. To mitigate this type of risk, Continuous Testing at least daily or on demand helps narrow the scope of the problem to the configuration or code changes that were made in the last 12 hours, which dramatically reduces the size and scope of the changes that need review.
In large, complex enterprise environments both Continuous Integration and Continuous Testing are needed. They are not mutually exclusive. Organizations with enterprise-class deployments containing a mix of custom-built and packaged applications need to apply both approaches to testing. It is not possible to map every step/field in a business process to a requirement or to maintain change lists that can reliably catch every issue that may occur in a complex business process. It is alwayseasier to find that needle in the haystack if you can reduce the size of the haystack.
Along with Continuous Integration, Continuous Testing enables companies to:
- Identify unforeseen problems faster
- Narrow the scope of possible causes sooner
- Address the critical issues earlier
- Reduce defects in higher environments quicker
- Provide greater agility by allowing the business to deploy faster
Continuous Testing is the missing link between Continuous Integration and consistent delivery of high quality business value.