Here is a look at the top five questions I consistently get when talking to clients about leveraging Worksoft to support Agile-Plus-DevOps environments:
- Does Worksoft support Agile-Plus-DevOps environments?
- Can Worksoft Certify tests be run from Jenkins and other continuous integration tools? (parts A and B)
- Can build numbers be tracked in Certify results?
- How does Worksoft support continuous testing?
- Which ALM and defect tracking systems does Worksoft integrate with?
We’ve also included a number of additional resources we’ve recently created to help clients on their journeys.
#1. Does Worksoft Support Agile-Plus-DevOps Environments?
Big YES on this one! We have a number of customers who have adopted Agile and DevOps and are using Worksoft every day to create and execute tests across custom, mobile and packaged applications. SAP's internal IT team wanted to apply Agile testing techniques with a “test early and often” approach and is using Worksoft to help do this. At a high level, customers leverage stand-alone capture to enable business users to create documentation and automation, and they use Worksoft Certify to harden the automation. Then, they execute tests via their CI tool chain or by using Worksoft Certify Execution Manager to execute continuous testing cycles.
We recently hosted a webcast delivered in conjunction with Forrester analyst Diego Lo Giudice in which we go over the “why” Agile and DevOps for packaged apps in the first part of the presentation, and then we cover the “how” with Worksoft in the second half.
#2. Can Worksoft Certify Tests Be Run from Jenkins and Other Continuous Integration Tools?
Part A: Yes! Worksoft offers command-line support from Certify for executing a test and an out-of-the-box integration for a RESTFUL API that can be called from Jenkins, Microsoft Visual Studio TFS, Micro Focus ALM, SAP Solution Manager and others. Users can run Certify tests from CI servers (Jenkins, TFS, etc.) and pass build numbers for reporting via command line whether the CI server and test lab are the same. This approach works in environments in which security is not an issue and the agent machines used for executing Jenkins can be left logged in with a console up and running all the time.
The Worksoft best practice is to use Certify Execution Manager to execute tests, which solves a number of the environmental problems of location, user screens and varying devices.
In most large enterprises, the build servers are not in the same place as the “test labs.” The option of spinning up virtual machines typically doesn’t work for large packaged app environments. Usually, dedicated hardware, databases and applications are staged in pre-prod to run tests on. Certify Execution Manager allows Jenkins to make requests over the network to run automated, on-demand tests in parallel across different remote test labs. Jenkins makes the test request, and Execution Manager distributes and runs the tests across the resources in various labs.
Part B: Why use Execution Manager with or instead of Jenkins? Why should you not just run Certify tests from the command line in Jenkins?
Yes, it is true that Certify is super-flexible and you can run tests from the command line, but there are challenges to doing this in large enterprises:
- Access - When you run a UI test on a Windows device, there must be a user logged in to the machine and the screen can’t be locked so you get all your screenshots in your test and business process procedure reports. Execution Manager handles access and auto-logging in to machines.
- Security - Try asking your security team to give you a Windows desktop that is always logged in, has the screen saver turned off, and will never lock because of inactivity. Execution Manager not only handles logging in to the machine but also logging out of the machine after the test has been run.
- Orchestration - Most customers want to run their tests on a wide variety of devices that mirror their production environments. This includes a wide mix of OS versions, application versions, and user security and roles. Execution Manager can request that tests be run on different machines so you can verify that the differences in OS or application patch versions don’t break the end-to-end process.
- Parallel Test Execution - Unattended parallel test execution is enabled when the CI server and test lab are the same, or via REST calls to Execution Manager to schedule and run tests in remote or cloud-based test labs.
- ALM Updates - Bidirectional integrations with ALM systems can be used to execute Certify test processes, view the results and leverage all the ALM functionality for test management. Certify test results can be viewed from within an ALM, showing both the test execution summary as well as details for each test step.
#3. Can Build Numbers Be Tracked in Certify Results?
Yes, and we do this internally when we test our own products. We like to frame this as “drinking our own Champagne.” When a Certify process is run, a user can specify a result attribute, which can be set when running from the UI, command line or Execution Manager. When we run our regression, our internal tests for Worksoft Analyze, Execution Manager runs them in multiple test labs and with multiple browsers. Our result attributes have the Lab (Dev, QA1, QA2, production) and the build number assigned by Jenkins. Our reports then show the pass/fail information by environment and build number.
#4. How Does Worksoft Support Continuous Testing?
I am a big fan of continuous testing and have written a number of blogs on the topic, including “Is Continuous Integration and Continuous Testing a One Size Fits All? No … and Here’s Why.” Leveraging Certify for continuous testing makes a ton of sense. Continuous testing means that you are running your regression tests more frequently so you can ensure that system updates are not going to break anything before the change goes into production.
Worksoft makes it easy to harden and variabilize tests so they don’t break. A Certify test never needs to be rerecorded when the application or process changes. You can easily open an existing Certify process to add, modify or delete steps when required. If an application object changes, an update can be performed once, and all related tests will update automatically. Improve test reuse even more with process compare. It’s easier than ever to consolidate and reduce test maintenance by quickly evaluating two processes to determine whether an existing test provides identical coverage.
Typically, testing done within an Agile-focused sprint focuses on a specific story and small piece of functionality. Testing at the feature level is more extensive and looks at the larger workflow. The challenge, then, is syncing these two types of tests. New features tests run as part of CI cycles also need to be passed back to run as part of CI cycles so that a larger net can be cast to ensure stories don’t conflict with one another or cause problems within the larger epic. Worksoft enables both CI and CT and optimizes ROI gained from test automation efforts.
Running Tests Daily - In addition, our services teams have come up with a set of best practices for enabling continuous testing with Worksoft, including:
- Setting Up Test Data - Certify processes can be used to set up data and then to run the tests against the data. For example, to do a delivery and create an invoice, don’t think that there is data magically in the system or that it will magically show up. Users can have the tests create their orders so they can then deliver and invoice.
- Verifying Workflows - Certify processes can be used to verify real user workflows and end-to-end workflows versus simple atomic units of work. There is huge value in running tests the same way as the end users who use the systems to run their businesses.
- The third best practice is to schedule tests via Execution Manager to run daily or weekly, based on how many changes happened that can’t be correlated to a change list item. If you have only 10 changes a week, run them on Fridays; if you have 25 changes a day, run them daily.
#5. Which ALM and Defect Tracking Systems Does Worksoft Integrate With?
I speak to customers daily who are looking at their existing ALM and defect tracking products and deciding whether they can or want to use them as they transition to Agile and DevOps. Worksoft has had the ability to link to and run tests from HP (Micro Focus) ALM, SAP Solution Manager and IBM Rational for years, along with the ability to put defects into JIRA, Microsoft Visual Studio TFS and ServiceNow.
Diving into the conversation a bit more, the follow-up question I get is, “How do I keep track of my requirement to test when all people talk about is epics and stories,” which is then followed by, “I think we’re going to use Jira for this.” A common misconception is that Jira can be used to replace an existing ALM. Why not? Jira is cloud-based and less expensive — easy decision, right? Unfortunately, it is not that easy. The workflows for managing requirement and code flows are very different from those for managing test plans.
The Jira green plus icon says add epic, story, task or bug. There are no test plans, test cases or test results in Jira. To get this functionality, a plugin must be used. In addition, multiple other plugins will need to be added to match the functionality of a Microfocus (HP) ALM. In the end, you have a plethora of different tools that have to be integrated and managed, which may end up costing more and increasing complexity.
The good news is the classic ALM platforms now have Agile Scrum workflows along with just Waterfall (i.e., HP ALM Suite and IBM). Teams can continue to leverage their existing ALM systems as the system of record, and for audits as they adopt Agile. A more modern Agile planning tool, such as CA Agile Central (formerly Rally), provides not only organization of epics, features, stories and tasks, but it also can map the supporting test plans, tests, cases and sets. This is great for project managers and enables teams to keep all the project resources together.
Looking to learn more on how to use Worksoft? Check out Worksoft University, which features a number of free, self-paced online classes.