When to automate your testing (and when not to)

This page provides an overview of the key point of the presentation When to Automate Your Testing (and When Not To) by Joe Fernandes and Alex Di Fonzo

Three myths and realities of automated testing

	Myth	Reality
1.	Automated testing always results in improved software quality	Automated testing requires a higher initial investment but can yield a higher ROI
2.	Every application development project or testing team can use automated testing tools	Skills and training are required to be successful with any automated testing tool
3.	Automated testing is an all or nothing proposition	Every company doing automated testing still does some amount of manual testing

Why manual testing happens

• Time Testing teams may not have time to investigate alternatives to manual testing, learn how to use tools, and/or build & maintain scripts
• Application Complexity Certain applications may be too complex and not suitable for automated testing
• Skillset Some testers (business analysts, etc.) may lack skills needed to be successful with test automation tools
• Cost Organizations may not own automated testing tools and may lack the budget to invest in tools
• Job Security Testers / QA organizations may be comfortable and experienced with manual testing and feel threatened by automation
• Awareness Organizations may lack the awareness that viable automated testing alternatives exist

When manual testing is a good idea

• Subjective Validation For application functions that must be validated subjectively by humans such as usability or look-and-feel, manual testing may be the only option
• New/Changing Functionality For new application functions that are still being developed and evolving / changing frequently, creating automated scripts may be a waste of time
• Strategic Development For strategic application functions that you want testers to pay specific attention to, hands-on manual testing may be a better alternative
• Complex Functionality For application functions that are extremely complex, test automation may be a major challenge (time & cost investment outweighs the benefit)

When automated testing is a good idea

• Regression Testing: For re-testing preexisting application functions that are being carried forward to new versions (usually the majority, unless app is brand new)
• Smoke Testing: For getting a quick high-level assessment on the quality of a build and making go / no-go decision on deeper testing
• Static & Repetitive Tests: For automating testing tasks that are repetitive and relatively unchanging from one test cycle to the next
• Data Driven Testing For testing application functions where the same functions needs to be validated with lots of different inputs & large data sets (i.e. login, search)
• Load & Performance Testing: No viable manual alternative exists

When to evaluate automation vs. manual testing

• Reviewing Requirements

  • Can this function be automated?
  • What, if anything, does development need to do?

• When writing test cases

  • Can this function be automated?
  • If yes, ensure test case is written to ease scripting (step by step)

• While Testing

  • Ensure the test case is clear and precise
  • Are the results predictable?
  • Does the test have to be run many time to get the desired result?

What to think about

• Positives

  • Will productivity increase?
  • Will test coverage increase?
  • Will test accuracy increase?
  • Is this a large data input?
  • Is this GUI intensive?

• Negatives

  • Will this require human intervention?
  • Will this require 3rd party systems?
  • Will tests have unpredictable results?
  • How often will this function change?

Automation: What works and what doesn’t

• Lessons learned

  • Automate functions that are 80% stable and unchanging
  • Work with development on unique names for controls and fields
  • Don’t overlook bulk data loads for the support of regression testing
  • Remember to include script maintenance in estimates
  • Make scripts as generic as possible
  • Use control file for URL, User IDs, and Passwords

• Upper management will always feel that regression should be 100% automated. This perception must be managed while setting proper expectations of what can be accomplished.

Summary

• Do

  • Use this as a guideline and change it to fit your process
  • Work to control expectations of automation
  • Use other peoples experiences (online documentation, blog posts and QA sites) for Tips, Trick and Information

• Don’t

  • Try and automate without support from development
  • Over-estimate what can be automated
  • Let someone else set the expectations of automation

---

Credit

Adapted from When to Automate Your Testing (and When Not To) (Copyright 2011 Oracle)

By Joe Fernandes and Alex Di Fonzo