Many companies have built simplistic, vertical TDM solutions that fall short when it comes to meeting the complex data needs of testing modern-day business processes that traverse multiple applications and have numerous data dependencies (e.g., horizontal end-to-end business processes). In the worst case, what is described as “test data management” is actually just a set of open-source buzzwords describing behavior-driven development (BDD).

In my last blog on this topic “Why Most Test Data Management (TDM) Solutions Don’t Work for End-to-End Tests” I gave a high level overview of the pros and cons of the various approaches to test data management in the market today, which include:

  • Production Data Replication
  • Data Masking
  • Synthetic Data Generation
  • Behavior-Driven Development (BDD)
  • Pairwise-Based Matching
  • Automated Self-Sufficient Test Cases

Now let’s take a closer look at BDD. The Cucumber BDD framework, with its easy-to-read Gherkin language, has become one of the most widely accepted and used open-source testing frameworks in the industry. Cucumber offers a wide range of support for programming and testing platforms, including Groovy, JavaScript, Ruby on Rails, and Selenium. Many testing solution providers are leveraging the Cucumber platform as part of their paid solutions. TestPlant has built a Gherkin integration for Eggplant Functional, and Worksoft offers a Gherkin script converter within Certify. With such a wide adoption, it is no wonder many users have also tried to adopt it as a test data generation solution for software development.

Users can provide a list of data values to select from as part of scripts and living documentation. This “list” becomes the test data repository. Using the Gherkin language, the user writes the test case, including data variables, using a simple English-language script. But within the simplicity of the script, the complications arise. The lack of detail surrounding the script leaves the meaning up for interpretation from user to user. Even though the testers understand which list they should pick the data from, they may or may not understand what each of the values of the data within the list is and what the correlations are with the other steps in the test, and the tests fail. 

Consider the following example, in which a business analyst supplies the testing team with a simple script with pick-list data for testing the order process for a bicycle.

Testing The Order Process For a Bicycle

To help the test pass, when you select a value from the pick list in a Cucumber test, you have to ensure you are picking a valid value. Does the product actually exist in the warehouse that you selected? Is there enough stock to fill the order described in the test? Will the selected data work horizontally across the business process?

Does creating and maintaining these types of pick lists really provide anyone with any value? At the end of the day, this is still manual testing and manual data generation, and it is labor intensive, tedious, and prone to human error.

We have to remember some of the guiding principles in modern IT before we just assume the data exists that the users expect. We can’t just copy production data into test anymore. Personally Identifiable Information (PII) is much broader than you think. Any customer data could fall into PII problems, which means a business person helping describe the data can’t think of the data they have in production because it is different.

Complex end-to-end process tests require the data to be synchronized across the process as you run the tests. Consider a simple order for example. How many do the various devices and applications (mobile, web, pricing, and ERP) have to sync to the supporting catalog? Typically, each system has their own system of record and just because you want to order the part does not mean the pricing is synced between the web shopping basket and the ERP.

TDM may not be simple, but evaluating a TDM solution’s ability to support end-to-end business-process testing really comes down to two things:

  • Ability to Understand Supporting Systems’ Data Perspective
  • Ability to Correlate Data Across the Business Process

For more on the various approached to test data management and what to look for when selecting a TDM solutions for testing enterprise applications like SAP read the white paper “Enterprise Test Data Management Practicum vs. Marketing Hype”.