“Manual testing is simply not an option going forward.”
We recently caught up with a senior quality manager at one of the country’s leading life sciences companies – who also happens to be a Worksoft customer. “In 2015, if you’re not doing automation, you’re going to fall behind,” he explained. Well, there’s no risk of that on his watch!
This multi-billion dollar life sciences company has a global presence and a long track record of success with strong performance and continuous innovation, growing to nearly 10,000 employees today. That growth has come organically as well as thru acquisitions – where they have tended to leave acquired software systems and processes in place. And because it’s in life sciences, it’s a regulated industry with significant compliance and reporting requirements.
Challenge #1: The Cost of Manual Testing
“We don’t have a dedicated testing team. We pull business analysts out of the business to test. We had over 400 people signed up as testers!”
During their last major SAP enhancement project, the company had more than 100 people testing on any given day, full time for 20 weeks – performing thousands of repetitive manual tests. The estimate was that the company was spending about $500,000 a week on testing with a total cost in the multi-millions of dollars.
“Very, very expensive. And at the end of the day what do you have? You have a bunch of executed tests – and that money is gone. If you want to repeat a test, you have to do it all again – and that’s a really expensive proposition. Pure cost, and no asset.” Of course, you can’t afford to have an outage that shuts down the business, so you have certainty of that. But the company wanted much more – speed, staff efficiency, and higher quality.
Challenge #2: The Pace of Change
“The other issue we face as an IT organization is the drive to do things faster. To be successful, we have to enable the business to move faster. But we also have to protect the business – keep the lights on, keep the ‘trains running on time.’ Yet that has traditionally meant going slower. So there is that dichotomy.”
“If you want to go faster without changing the way you do business, the only way to do that is to reduce quality. And that risk is unacceptable. So we had to do something different.”
He mentioned Amazon, a company that has been able to push technology changes to production every 11 seconds all while reducing outages by 75% since 2006. How had they done it? “They’re not doing manual testing. They don’t have an army of people clicking on keyboards. They do automation.”
How to “Sell” Test Automation Internally
The company had several objectives in mind when they turned to automation:
- Improve the quality of their systems: Eliminating the risk of glitches and business process disruption when applications fail
- Reduce test cycle time: Accelerating projects
- More efficient use of resources: Giving business analysts their time back
- Improve reporting on “system health:” Answering the question “Is every system ready for business?”
To get started on the path to automation, gaining organizational support is an essential first step. “You have to sell the idea internally — and you need data to sell.” In other words, developing compelling proof points that explain the need for a new approach. He suggested three steps that worked well in his organization:
- Estimate the cost of manual testing. Estimate how many people, how many hours, figure an average hourly rate, and put a dollar value on the total cost.
- Estimate the cost of fixing bugs in production. Figure out how effective your testing is by counting the percentage of bugs found in production over all bugs found. Estimate the cost of fixing bugs. Take into account the cost of business disruption if you can.
- Grab the attention of your decision makers with something provocative. For example, the team built a hypothetical dashboard showing a long list of automated test runs and business process tests to paint a picture of what’s possible with test automation. He suggests sending the dashboard to your VP and ask “What if we could run all our tests overnight?” It will open up a whole range of discussion on the possibilities for boosting quality and staff efficiencies.
“Most senior managers don’t really understand what’s possible. You’ve really got to grab their attention. When we started sending out the dashboard, the whole attitude about test automation changed.” Once people understood the true cost of manual testing and what’s possible with automation, then organizational support follows. But be sure to have an answer ready for “How much money will we save?”
The Result: Daily business process testing
The company has already achieved phenomenal results with automation, with over 100 automated business process tests in place, including a mix of end-to-end scenarios and detailed unit tests. These range from very long end-to-end processes – from its e-commerce system, generating an order in SAP, shipping, delivery, invoice generation and updating the e-commerce site – to unit testing as well.
Test velocity has dramatically improved. “We run all tests every night.” And they report improvements in business process quality as well. “We are finding issues that would otherwise go to production. You’re going to reduce the number of defects. Testing is much more in-depth with automation.”
Automation also allows the team to be much more thorough and complete in its business process testing. For example, the company’s purchase order form contains a ton of complex logic. “We have 38 ‘mini scenarios’ in a single PO form unit test (e.g. Direct, Indirect, Drop ship) where we verify the layout of the PO form and make sure it’s using correct addresses and more.” Automation lets you test every scenario and all combinations.
“We send out a daily dashboard to the project team and management that shows a running 5-day pass/fail history for each test and the all-time pass percentage. Tests with a low (<80%) pass percentage indicate areas of system fragility.” Those processes become the focus of special attention by the IT and business teams.
The benefits have been substantial, and the elimination of manual effort will save millions down the road. “If we can do it, other companies can do it.”
Connected to SAP: Salesforce.com, the cloud and web applications
“The bulk of our tests have been SAP and SAP GUI, but we’ve expanded into other areas. We have tests now running on Salesforce.com – that’s a web application. This is very valuable because Salesforce is in the cloud and we get updates from Salesforce very quarter – whether we like it or not. We have to take those updates – and we have to prove that those customizations work every quarter. Now we’ve automated those tests for Salesforce, and we can run them overnight as well.”
“When you pair SAP with a web property, you end up with a web property with an SAP component. Your SAP team has to move at the speed of the web team, not the other way around,” he explained. Automation allows his team to do just that.