Miscellaneou$

Automated Mobile Testing tools and services

Recently I did a research on the tools and services that let you test your apps (iOS, Android, Hybrid/Web) and here’s what I found.

TL;DR

In case you don’t have time to read through this and you’re looking for a quick answer on what to choose; I would go with AWS Device Farm.

It was the only service where the docs actually had clear instructions and the only service where the automated tests actually worked. If for nothing else, I would suggest using their Remote Access option, as it’s really great because it gives you the ability to test your app on the particular device with the particular OS version in real time. This is definitely useful for when a user submits a bug, and you don’t have the actual device/OS combination on which you could test.

A notable mention is Appium, which is perfect if you just want to test on your machine and your simulator. It’s open source and it even has support in AWS Device Farm.

Longer version

Below are my notes about the tools/services that I’ve tried.

AWS Device Farm

As you’ve read in the TL;DR, my choice would be AWS Device Farm, so I’ll take a bit more space here and explain why I liked it the most, and I’ll showcase the simplest usage.

The official website is here and the official documentation is here.

There are two main ways to use AWS Device Farm:

  • Automated app testing using a variety of available testing frameworks (they even have a few built-in tests called Fuzz tests which create some screenshots and a usage video as well)
    • it supports hybrid apps for both Android and iOS
    • tests run in paralel on multiple devices
    • test reports contain:
      • high-level results
      • low-level logs
      • pixel-to-pixel screenshots
      • performance data
  • Remote access to devices on which you can load, run, and interact with apps in real time
    • allows you to swipe, gesture, and interact with a device through your web browser in real time
    • details about actions that take place as you interact with the devices
    • logs and video capture of the session are produced at the end of the session for your review

Demo

The docs for setting this up are really great and straight to the point.

Once you create your first project, you have to Create a new run:

Then you have to select the Android/iOS icon and upload your .apk/.ipa file:

In the next step, you have to choose the test type that you would like to use. There are a lot of options, and as a starter (in case you don’t have your tests ready – we’ll cover these below in the Appium section) you can choose the Built-in Fuzz test.

In the next step you can select the various devices:

set some additional options and even pre-install some other apps:

After this, just review the settings and start the test. Tests will run in the background and in parallel on all the devices that you’ve set. Once this is done you will get an overview like this:

You can view the details of each run on the particular device:

You can view the screenshots, performance:

And you can even see the video of how the built-in test used your application:

Sure, these tests are not something on which we should rely on, and that’s why it’s great that we can write our own. In the next section, we’ll shortly cover Appium and why it’s so great for writing tests.

Appium

Appium is a tool that makes automation possible for iOS and Android devices. It’s architected around four key principles:

  • You shouldn’t modify your app in order to test it, as you don’t want test libraries to affect the operation of your app
  • You should be able to write your tests in any programming language, using any test runner and framework.
  • There is already a very successful automation standard, so we should reuse and extend that rather than create an entirely new model.
  • Mobile automation is for everyone. The best mobile automation tool will be open source, and not just in terms of having its code available for view. It should be governed using open source project management practices, and eagerly welcome new users and contributors.

Appium satisfies all these requirements in the following ways:

  • Appium allows you to automate your Android and iOS apps without code modification because it relies on underlying automation APIs supported by the mobile OS vendors (Apple and Google) themselves. These APIs are integrated into the development process and thus don’t require any 3rd-party libraries to be embedded in test versions of your apps.
  • Appium is built around a client/server architecture, which means Appium automation sessions are HTTP conversations (just as when you use any kind of REST API). This means clients can be written in any language. Appium has a number of clients already written, for Ruby, Java, JavaScript, Python, C#, PHP, Objective-C, and even Perl. Additionally, this means that it doesn’t matter whether an Appium server is hosted on the same machine as the tests.
  • Selenium WebDriver is without a doubt the most widely-known framework for automating web browsers from a user’s perspective. Its API is well-documented, well-understood, and is already a W3C Working Draft. Appium uses this specification and extends it cleanly with additional, mobile-specific automation behaviors.
  • Appium is open source. It should be clear from the volume of issues and pull requests that we have a very active community (who also engage with one another on our discussion forum).

Few additional learning resources:

Demo

Once Appium is installed and running, click the proper OS icon (Droid/Apple) and select the path to your compressed .ipa file (it is required that you zip the .ipa file):

One thing I had to set here as well was the BundleID, as it didn’t work without it (BundleID can be found in your config.xml file as the id property of the widget tag).

Once this is set, you can simply click the Launch button. Once the launching is done you’ll see something like the following in the logs:

[Appium] Appium REST http interface listener started on 0.0.0.0:4723

Then you can click the Magnifier icon:

which will run your app in the simulator:

If you’re testing Hybrid apps, be sure to set the proper Context:

The best feature of Appium is that it lets you Record the tasks. To enable it, just click the Record button, navigate through the tree like structure and select the actions that you want to perform on the selected tags (Tap, Swipe, Shake, text inserting, etc.), and it generates the test code (in few different languages):

Xamarin test cloud

  • Website
  • The installation was surprisingly long
  • It turned out that once I uploaded my .ipa file it threw an error: The .ipa file does not seem to be linked with Calabash framework.
  • The main drawback is that you need to integrate the SDK into the app itself. Something which you don’t have to do on the AWS Device farm (through Appium [take a look at their four key principles])

Calabash

  • Later I figured out that Xamarin test cloud uses Calabash
  • Callabash was developed and open sourced by Xamarin and it is a set of libraries that enable you to write and execute automated acceptance tests which consist of end-user actions like Gestures, Assertions, and Screenshots
  • Calabash supports (and prefers) Cucumber

Cucumber

  • Cucumber lets you express the behavior of your app using natural language that can be understood by business experts and non-technical QA staff.

Browserstack

Browserstack doesn’t support mobile app testing currently. Here’s the official thread:

  • Can I test native apps or install apps on BrowserStack’s physical devices?

At this time, the mobile devices are available exclusively for cross-browser testing. Mobile app testing is on our roadmap for a later date.

Sauce labs

I’ve spent some time trying to get their solution to work as they popped up on most of the searches that I’ve done. However, I can’t say this was a pleasant experience as the docs seemed to be more like ‘Hey, we know how to do this, but if you want we can offer you our consulting support :/’.

But, didn’t like their support as I’m still waiting for the answer to my question I’ve sent them. When I got the automated sales email, I followed up with the guy but again he said he couldn’t help me as he’s ‘only’ sales.

They allow you to upload your executable to their service, but only via terminal. They don’t have an interface for this on the site as AWS does, nor do they have an overview of the uploaded files.

You run all your tests from a script, and you can then see the screenshots and video of how the tests are interacting with your app:

However, the bottom line here is that I wasn’t too impressed with them.

Other services

I have also briefly looked at these services, but they didn’t seem intriguing enough to pursue any of them further.

Firebase test lab

eggPlant Mobile

Soasta

  • Website
  • Nothing special. TBH, it seemed like it’s an older version that’s not actively developed.

Squish

  • Website
  • Seems like a quite stable solution, however only locally and it needs to be implemented in the Xcode/Android project to work

Testmunk

  • Website
  • Although very nice interface and the flow is basically as on AWS, it wasn’t able to run my app (just kept popping errors). I emailed the owners but no reply to date, so…

Keynote

  • Website
  • Asked for Credit Card – lol, are we in like 2009?

iOS only testing

Windows only apps

Miscellaneous

Conclusion

So, yeah, as stated in the TL;DR AWS seems like the most viable solution to me right now. In case you have experience with some other solution, or you disagree with some of the conclusions that I’ve made, please let me know in the comments.

Also, if someone has extensive experience with writing and running tests on AWS Device Farm via Appium tests, I would love to get your feedback.

Written by Nikola Brežnjak