How To Achieve Massive ROI Through SAP Test Automation

Every year, hundreds of companies seek out Worksoft^® for one simple reason: they’ve failed. Rarely is their failure explosive and public; but much more often it takes the form of deep and broad dissatisfaction with their current SAP^® management processes that has been slowly smoldering, perhaps for years, until one final event — an event that itself may be small – ignites that slow burn into a five-alarm fire for change.

How have they failed? An upgrade or enhancement delivered significantly late. Much-needed functionality getting pushed off yet again. A spate of production defects that individually may be small, but together result in an unacceptable state of affairs. The sparks are varied, but the result is the same: It’s time for a change. A change that improves cycle time. A change that reduces production defects. A change that gives control back, brings clarity and visibility, makes the most of your SAP investment and the best use of increasingly limited resources. A change that can save tens of millions of dollars in your organization.

As a leading vendor in the SAP testing space, we see these same compelling events over and over again. But the pain, the frustration and the urgency CAN be avoided and the cost savings realized sooner by understanding the underlying issues before it’s too late. Why wait for failure?

Are You a Tortoise or a Hare??

The way most companies approach functional testing falls into one of two camps. Risk averse companies test thoroughly using significant manpower. While ultimate quality may be very good, cycle time for releasing new functionality is often painfully slow. Although this may be acceptable for corporate environments with a slow rate of change, an increasing number of companies find themselves in environments where faster cycle time is demanded, or where Agile methodologies are being actively implemented.

The challenge with risk-averse, people-rich testing approaches is that they are difficult to speed up. Throwing more people at the problem is not only costly, it frequently doesn’t result in faster cycle time. As Fred Brooks, the infamous author of The Mythical Man-Month was once quoted, “Nine women can’t make a baby in one month.” Beyond a certain point, you cannot practically shorten cycle time simply by adding more testers. Communications overhead, training, license management and data management all come into play.

And yet the demand for faster cycle times continues to grow. Research firm Gartner went so far as to suggest that the best way for companies to differentiate themselves in the market when their competitors also use SAP is by using “new, more agile technologies.” ¹ And yet in the same paper they also point to the challenge this presents: “SAP core business processes are difficult to change quickly, as they were never designed to be changed quickly in the first place. They were designed to run the business reliably.” ²

This brings us to the other broad category of functional testing. Companies in this second category achieve much faster cycle times, but almost always at the cost of less thorough testing. They minimize their investments in testing manpower, preferring to test only the most critical or easy-to-test areas. They get new functionality out fast, but nearly universally experience much higher rates of defects in production. ³

The cost of this approach is more difficult to quantify than the exhaustive testing approach, but the impacts are well-known. It is generally accepted in the development world that the longer software defects go undetected, the more expensive they are to fix. Although research differs on the exact ratios, the general rule is 1:10:100. That is, if a defect costs one unit (hour, dollar, etc.) to fix in requirements and design, it costs 10 units to fix in testing (system/acceptance) and over 100 times to fix in production.

What this traditional “cost of defect” calculation generally does not take into account is the disastrous fall-out from defects in mission-critical systems in today’s enterprise: lost sales, unproductive employees, unhappy customers, demoralized IT and QA teams caught in endless break-fix cycles, and possibly even regulatory and legal issues.

Consider Automation Now

What is the solution to the predicaments of the Tortoise and the Hare? Gartner specifically recommends that SAP users evaluate automated testing software tools in their SAP change management processes to minimize testing effort. ⁴ And why not? The promise of automation is significant:

• Speed: Automated tests are fast to run, and the more tests that are completed in a given period of time, the more projects can be delivered on schedule.
• Flexibility: The execution of automated tests is not reliant on work schedules. It can be done off hours and doesn’t stop for breaks or when the day is done.
• Accuracy: Manual tests, especially of repetitive tasks — e.g., performing the same test across dozens of combinations of operating system and browsers — are more prone to error when performed manually.
• Reusability: Automated tests can be reused with much less effort than performing manual tests repeatedly.
• Maximizing People: Instead of performing repetitive manual tasks, QA resources can use their knowledge to modify and perfect the functionality of automated tests, resulting in better and more complete test coverage.

It’s All About The Tools

Traditional automated testing tools have not always delivered on this promise, leading some companies to under-utilize or even shelve them. Following are some of the common missteps, and what you can do about them.

How Automated Are You

Traditional automated testing tools require significant expertise to operate and maintain. This, coupled with a general lack of precise industry standards for what constitutes test coverage, makes it extraordinarily common for companies to overestimate their automated test coverage. “We have two hundred end-to-end tests automated” may sound impressive, but it is generally lacking on two dimensions:

First, the term “end-to-end” should imply tracking an entire business process from inception to completion. Unfortunately, 80% of the automated testing tools on the market today lack the ability to test a broad-enough set of applications and interfaces to accomplish this. What they really mean is “we test the part that’s in SAP,” or worse “we test the part that’s in SAP GUI.” If you cannot test the full complement of the technologies your company uses for its business processes, your tests are missing critical hand-offs between systems where defects are often concentrated. The best automated testing products test not only the full complement of SAP, but also browser-based technologies (e.g., .NET and Silverlight), associated non-SAP systems, custom code, and even mainframe and mobile technologies. Unfortunately many legacy test automation technologies do not support the entire enterprise landscape.

Second, when using the number of automated tests as a proxy for test coverage, it is important not to double count. A well-designed end-to-end test covers a full business process and its variants. If your test automation was done by simply automating your manual tests on a 1-to-1 basis then your 200 automated tests may actually cover very few of your overall business processes. How can you tell? As a rule of thumb, if more than a dozen tests are concentrated in any particular area (e.g., Order to Cash), you need to take a close look at how your tests are designed and what your overall coverage really is.

The Time Machine

Going back to the Tortoise and the Hare, what’s really being illustrated is the relationship between time and quality. The “Tortoise” camp gains quality by spending extended time and resources on the problem, while the”Hares” go very fast but at the expense of quality. It is critical to avoid defects in production, but with increased demands for faster cycle times and agility, are you stuck in a no-win situation?

The answer is to look differently at one of the components: Time. Traditionally, most of the testing effort in a project occurs in a testing phase toward the end of the project. This is called the “big bang” approach from the concentration of effort in one big phase ⁵.

The problem with the big bang approach to testing is that defects are not found until the end of the project. This is the most costly and risky time to fix defects. Some complex defects may even be impossible to fix.

Unfortunately, script-based legacy automation products don’t work well (or at all) until the testing environment is stable, meaning the time period to correct and run automated scripts is nearly as compressed as a traditional manual environment. Newer test automation products like Worksoft Certify^® allow tests to be created much earlier in the development cycle, eliminating the “crunch period” traditionally associated with both manual and legacy automated testing. By moving the beginning of the testing cycle much earlier in the project, more time can be spent testing for better post-deployment results, and without lengthening the overall cycle time of the project.

How To Look At ROI

When performing a business justification for implementing automation, the core of the calculation is comparing the initial investment in setting up automation with savings generated in each subsequent test run where manual testing is avoided.

The subsequent cost savings for each box on the right in the figure above is calculated as the cost to run the manual tests minus the cost to run the same test in an automated fashion. What’s often missed is that the largest component of the automated test run is not the execution of the automated test itself — which may take only minutes and can often be run unattended — it is the maintenance of that test and its associated data prior to the test being run. This is determined by ⁶:

• Level of change in the underlying application
• Automated test complexity
• Level of automated test reusability
• Effort required to process test data before (and clean up after) an automated test run

This is certainly the Achilles heel of legacy test automation products. The editing process for traditional automation scripts is so complex that any change in the underlying application requires extensive coding to repair, and in many cases results in the automated test being scrapped and re-recorded from scratch.

The result of this lack of reusability is the savings per run for traditional legacy test automation products is much lower than often projected. Not only does that result in a longer payback period for the overall automation project, but it’s also very difficult to make progress on test coverage metrics, since automation experts spend their time maintaining or re-recording rather than building new tests.

The good news is that there are test automation products on the market that have With this functionality in place, ROIs for automated testing routinely return millions or even tens of millions of dollars with payback in a single project. Each situation is unique, but good providers can work with you to perform an ROI calculation that takes into account your specific needs and goals.

Conclusions

There is no question that rigorous functional testing is critical to successful management of any major ERP system. The challenge for your business teams, QA teams and management alike is how to speed up testing processes and increase accuracy and completeness — without breaking your already tight budget.

Whether you are a slow but steady Tortoise or a fast and fix-it-later Hare, you face significant challenges. The pressure for faster cycle time is growing, yet the tolerance for defects in production is low. The answer lies in intelligent automation, which can bring speed, flexibility and accuracy to your testing efforts while maximizing your resources.

Automation using legacy tools presents some drawbacks, but a keen eye and a focus on newer, more nimble technologies can help you achieve fantastic ROI:

• True end-to-end testing requires tools that can test a wide variety of technologies and platforms. The boundaries between platforms are collection points for defects. Look for testing technologies that can handle not only SAP, but the entire “ecosystem” of software that surrounds it at your organization.
• Look closely at existing automation to understand what your true coverage is. Are your tests concentrated in a single area with dozens of variants? If so, your coverage is probably lower than you think.
• Change the way you think about time. With newer technologies you can start your testing earlier, providing you with more time to test without adversely impacting cycle time.

Returns for test automation products routinely have payback periods of only months. Don’t wait until a crisis is looming before you begin your project.The pain, the frustration and the urgency CAN be avoided.

Don’t wait for failure to find you.