Monday, April 1, 2013

An Experience Report Building a Mobile Automated Test Lab With SOASTA TouchTest

Prelude

I came to develop a PhoneGap application on IOS and Android tablets.  Agile/lean engineering practice led to test-first development, but when I began to build out continuous integration, I found the offerings for mobile automated functional testing and deployment lacked maturity.

I discovered SOASTA when I attended a webinar on mobile functional testing that culminated with a photo of a mobile lab, to which a continuous integration process was deploying and running an automated suite of functional tests to a variety of IOS and Android targets:



No further inspiration required.  

Such an outcome would provide my organization with a high degree of code coverage for our mobile offerings and protection against defect regression -- to build the product right -- and establish an infrastructure to deploy feature changes continuously -- to build the right product.  The doors to lean mobile platform development are now wide open for us.

SOASTA and CloudBees

The webinar was hosted by SOASTA in partnership with CloudBees, a cloud-based provider of Platform-as-a-Service for developing web-based and mobile applications.  We followed the practices recommended by SOASTA and CloudBees closely, and built out our Jenkins deployment pipeline  comprised of the following jobs:
  1. build the javascript assets for deployment to the IOS PhoneGap project;
  2. build and deploy the IOS app capable of running SOASTA functional tests to devices tethered to the mobile lab;
  3. kick off SOASTA functional tests;
  4. build and deploy the production-ready IOS app to our enterprise Appaloosa store and notify QA that a new release is ready for a spot check.
We also built out an analogous pipeline to deploy and test our Android PhoneGap project designed to run concurrently with the IOS pipeline.

Features of Our Mobile Lab

(1) We run over 350 automated unit- and integration-tests and achieve greater than 80% code coverage:



(2) Using SOASTA command-line tooling and the Jenkins XCode plugin, we build a mobile-functional-test-ready IOS target and deploy it to the devices tethered to the mobile lab:


(3) Again, we use SOASTA command-line tooling to run a pre-recorded mobile functional test on a SOASTA-hosted instance of their TouchTest environment.



(4) Finally, provided each job succeeds, we build and deploy a production-ready release to our Appaloosa private enterprise store:




(5) There is a step that I purged, which uploaded to a QA site for approval.  This meant that the build would not pass until QA exercised their exploratory test phase.  I did not feel right about the way in which this step would prevent continuous deployment.  But I felt that it was right to purge this step after discussing the topic with Joshua Kerievsky.  He asked me a simple question:  can the automated test do exactly what QA would do?  Because of the accurate recording I experienced with SOASTA TouchTest, my answer was:  yes.

Quick Starts and Support

There exists a ton of practice on both SOASTA and CloudBees for getting a mobile lab up and running.  SOASTA's quick starts and knowledge base are comprehensive, and I found the forums and personalized sales support via email extra-ordinarily responsive.

The same is true of CloudBees.  Their partnerdemo site and blog provided concrete configuration and practice for many of the scenarios we required.  And Mark Prichard of CloudBees himself generously devoted a considerable amount of his time to walk thru the fine points of my Jenkins configuration.


Summary

The other offerings I discovered that support mobile functional testing had obvious shortcomings in one-way-or-another.  SOASTA is the only one, to my knowledge, that can support development at enterprise scale.  

My favorite feature is to be able to deploy from the command line to IOS hardware targets in a continuous deployment scenario without manual intervention.  

At the time of posting this, I learned that SOASTA has shipped my most-anticipated forthcoming feature:  the one that wakes up the sleeping device tethered to the mobile lab when the automated build deploys.

Our experience with SOASTA tooling for automated mobile functional testing was quite satisfying.