Automated Mobile Testing: The Pros and Cons

All tests serve to ask, “Is the system working as intended?” Automated testing questions further, “Can testing be more efficient, accurate and cost-effective?” Your answer comes by evaluating your operation’s test requirements and numbers.  Automated testing requires an upfront investment to reduce long-term costs of manual testing.  But, are automated tests worth it?

It depends. Automated testing software can work in conjunction with software development, software iteration, production and manufacturing systems, and processes.  Regardless of the environment, the cost of manual testing considers:

1. How long does each test take?
2. The number of systems (or software versions) to be tested?
3. How frequently are the tests conducted?
4. What’s the capitalized hourly wage of those who conduct the tests?

Thus, the more testing required, the greater your likely long-term ROI.  This return typically comes from a combination of cost-avoidance and increased productivity.

Advantages of Automated Mobile Testing

While every test may have its own special advantages, three advantages are common to nearly all:

Can be cost-effective – The cost-effectiveness of all automation usually runs proportional to the combined time, frequency, volume and labor cost.

Reduces human error and technical risks – Automated tests make it easier and less painful to add new or change developers. New developers need to worry less about possible damage from changes.  Automated tests double as documentation helping developers learn the project more easily.  Plus, it allows developers to execute code refactoring calmly as the tests can define whether the system was impacted or not.

More powerful and less time-consuming – Performance or load testing typically requires the assistance of automated tests or special tools. Additionally, transferring repetitive tasks to scripts gives your team more time to focus on more interesting and challenging work.

What are good candidates for automated tests?

As detailed below, there are many types of tests, some of which overlap.  Certain tests inherently warrant automation.

Regression testing  – These tests verify that the software/system continues to work after software patches, changes in system configuration, or other software enhancements.  These tests can be performed repeatedly as needed.  They make sure changes do not result in a regression or cause previously functional components to fail.

Functional Testing –  Function tests are a QA process to verify a system is doing what it is intended to do.  If you want your app to function well, you need to test every single operation it is designed to perform.

GUI Functional Testing – Whether your GUI is attractive is best left to human evaluation, but this evaluates whether GUI objects function as intended and meet your specifications.

Smoke testing – This technique encourages testing the basic functionality of each software build. If this test fails, the entire build is declared unstable.  Testing of that build only continues after it passes the smoke test. This approach saves you tons of time and effort by uncovering obvious early bugs.

Acceptance testing – Acceptance testing commonly used in Behaviour Driven Development (BDD) verifies that the set requirement (or story) is complete. These kinds of tests help identify omissions in the previous unit or integration tests.  Acceptance tests provide an excellent overview of how “complete” your project is.

Load and performance testing – The multiple load and performance tests you can run simultaneously from different devices are a huge advantage, here. You can emulate as many users as needed.  As a result, you ensure the stability of the final product and that it can handle whatever is coming for it!  Load testing typically cannot be performed manually.

Regression Tests

It bears emphasizing that with automated testing, regression tests are performed automatically for every build. As projects develop and mature, new features are regularly added – which can dramatically increase time and labor requirements for manual smoke and regression tests. So, that adds considerable overhead for long-term projects where manual regression testing of each build can potentially take a week or more. That cascades into the release schedule, adding a week to every release.

And again, adding more devices also increases manual testing time and labor requirements. Automated testing reduces your development team’s workload so they can spend more time creating and improving features instead of testing them.

What are bad candidates for Automated Tests?

Despite all the advantages of automated testing some tests are impractical to automate.  Some tests are simply cheaper and faster to perform manually, or involve:

Subjective Evaluation – We probably are not quite ready for software to tell us whether something is attractive (look-n-feel) or sounds good (voice calls and messages).  These are best left to subjective human evaluation and so should remain manual, for now at least.

User Patterns and Strategic Development – How will the end-user try to use a system and what happens when they do?  When you want to see how actual humans will use the system, you can expect to see some unpredictable results.  The same applies when it comes to getting feedback, what users like and don’t like.

Systems pending upgrades –  Changes in a system require changes in the tests.  So, you don’t want to start developing automated tests until after scheduled upgrades of software and hardware are complete.

Complex Functions –  Developing automated tests for very complex systems may not be cost-effective.

What do we use at Reinvently for automated testing?

Our API, UI tests are written in Java, Python, Ruby. The Appium framework is used to execute UI tests on simulators and real devices. Detailed test reports are provided with the help of Allure.

Jenkins is our go-to tool for continuous integration where projects are built after each new commit or at scheduled time.  Then tests are executed. In the case of a successful result, the build can be deployed to the mobile app distribution system.  Cucumber is our choice for creating acceptance (BDD) tests.

Unit Testing assumes the testing of a unit – the smallest part of the code that can be logically isolated within a system.  Tests can be created using the TDD approach (Test-Driven Development, a technique in which unit tests are written before the code) or after the code has been written.

Making the decision to automate your tests

The decision to automate testing starts with identifying how much manual testing is costing you.  If you don’t know, talk to your technicians and engineers.  They can quantify the time they spend conducting tests.  Your technicians should also have a good idea on what could be automated.  Most will appreciate automated testing especially as their skills could likely be put to better use for your business.

Fundamentally, testing is overhead. It doesn’t improve your bottom line, only helps prevent problems from hurting it.  So, less time testing equates to more time producing – developing, production, repairs or other improvements to help your bottom line.