Avoid Overly Simple Tools When Testing Complex Systems

With so many different tools and methods in the market, how do you choose the right test data management solution (TDM)? By now, you’ve probably realized there is no simple answer to this question, especially when looking for a solution to test complex end-to-end business processes and systems like SAP that involve a mix of both web-based and traditional Windows-based UIs.

SAP enterprise landscapes are highly complex and interrelated. To get good test data, referential and logical integrity must be preserved, and not all relationships are explicit. Data has to be correlated across tests and systems. The data needs to have the context of the business process being executed, positive or negative test, standard orders, or out-of-stock orders, etc. Most test data management tools on the market generate data based on a single set of functional requirements and lack the ability to correlate data across a horizontal end-to-end business process test.

This is probably why we see most clients continue to struggle with test data. They continue to invest time and money trying to leverage tools and approaches designed for single-app, vertically focused test cases and then wonder why they continue to spend so much time and effort generating test data.

What Should You Look for in a TDM Solution?
Effective enterprise test data management solutions enable users to see the data relationships across test cases and end-to-end business systems. These solutions allow visualization of the data that is available, the data that is missing, and the data that needs to be created. They need not only to generate data for functional tests but also for horizontal end-to-end business-process tests. This means being able to support tests that cross a plethora of enterprise applications with multiple UIs, including mainframe, ERP, CRM, web, mobile, personal apps, and others.

TDM may not be simple, but evaluating a TDM solution’s ability to support end-to-end business process testing really comes down to two things:

1. Ability to Understand a Supporting Systems’ Data Perspective
The test data views need to be generated from the perspective of the system under test. Data must not only be in the right format but also be validated with the system under test at the time of the test. To ship an item, it must exist in the warehouse; it’s not enough for the data to be just in the right format. The test data dependency on the data in the supporting system must also be accounted for. Most testing tools take a very limited view into test case data and fail to do this. A Gherkin script describes the data being entered into an API or web page, but it does not describe the back-end systems’ perspective of that data.

2. Ability to Correlate Data Across the Business Process
Data must be correlated across the entire horizontal end-to-end business process. The business process needs to have the context of the job to be done, positive or negative test, standard order, or out-of-stock orders, etc. A successful order means the product in an order does exist in the warehouse, is in stock and has the right price when the test runs. In contrast, for an out-of-stock order the product has to exist, but the quantity on hand has to be a specific value.

Without this understanding of the context, the data is just optimized random values. A combinatorial math calculation like some TDMs use does not understand this context, making it too simplistic with the test case data representing only a single vertical transaction. Data without context is less valuable and, in some cases, simply useless.

For more on the various approaches to test data management and what to look for when selecting a TDM solution for testing enterprise applications like SAP, read the white paper "Enterprise Test Data Management Practicum vs. Marketing Hype".