Mobile test automation using AWS Device Farm

Somewhere along our mobile test automation journey most of us reach a point where we consider moving to a cloud platform. The advantages sound promising: a large variety of devices, no maintenance effort and flexible scaling. The reality is slightly more complicated, and we must always understand our exact requirements before purchasing a plan with any of these vendors. In some cases, building a device lab on-premise (using either physical devices or simulators or a mix of both) can be a more effective solution. That said, cloud providers are an exciting and useful tool to take any mobile test automation to the next level.

In this article, we describe how to utilise AWS Devicefarm’s public cloud in a way that we can easily integrate into a CI pipeline. As the tech-stack for this example, we use Java, Cucumber and Appium but the concepts apply to other technologies as well.

What Is AWS Device Farm?

Device Farm is an app testing service that you can use to test and interact with your Android, iOS, and web apps on real, physical phones and tablets that are hosted by Amazon Web Services (AWS).

There are two main ways to use Device Farm:

  • Automated testing of apps using a variety of testing frameworks.

  • Remote access of devices onto which you can load, run, and interact with apps in real time.

Automated App Testing

Device Farm allows you to upload your own tests or use built-in, script-free compatibility tests. Because testing is performed in parallel, tests on multiple devices begin in minutes.

As tests are completed, a test report that contains high-level results, low-level logs, pixel-to-pixel screenshots, and performance data is updated.

Device Farm supports testing of native and hybrid Android, iOS, and Fire OS apps, including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks. It supports remote access of Android and iOS apps for interactive testing. For more information about supported test types, see Working with Test Types in AWS Device Farm.

Automated Testing

Test your app in parallel against a massive collection of physical devices in the AWS Cloud. Use one of our built-in frameworks, to test your applications without having to write or maintain test scripts, or use one of our supported automation testing frameworks.


Remote Access

Gesture, swipe, and interact with devices in real time, directly from your web browser.


Server-side execution model

Unlike other vendors, AWS uses a different approach to executing tests, which I call server-side execution. This means that the test code runs entirely on their infrastructure. The alternative to this is what providers like Browserstack or Saucelabs use where the test code runs on the user’s machine (client-side execution) and interacts with the cloud service via a web service, for example, using a RemoteWebDriver. In general, client-side execution is easier to implement. If you already know how to execute tests locally, the configuration changes you need to make are minimal.

AWS’s approach is a bit trickier to work with as it requires a few extra steps, namely packaging and uploading everything necessary to run your tests to their servers. This comes with some challenges and pitfalls which I like to share in this write-up.