Super excited.  PNSCQ is always a great conference and it shows what a great Software Quality community we have here!

Hope to see you there!! http://www.pnsqc.org/

—————————————————————————————————————————

Title: Mobile Continuous Delivery – Automating the iPhone App build pipeline

Abstract:
Releasing high quality software quickly is always a challenge.  Add mobile devices and Apple’s iPhone infrastructure and now we have an almost impossible task.

Configuration changes, static code analysis, code compilation, unit testing, 3rd party & API testing, security testing, functional testing, beta testing & production release.  In the past this task would take a large team multiple weeks if not months to complete.  Today we do this in minutes and on every code change.  The mobile world moves fast and we have to move with it.

This paper describes the design and process I went through implementing hubbub Health’s continuous delivery pipeline for their native iPhone application.  It will cover tool selection, test automation frameworks, test strategy, cloud resources, mobile challenges & techniques for dealing with Apple’s challenging iOS environment.

Slides – Native iPhone App Test Automation with Appium

Thanks everyone for coming to the meetup!  It was great fun and great questions/discussion/comments!!

I have attached a copy of my presentation.

Hubbub Health iPhone Test Automation

Find a pre-prepackaged Selenium Webdriver app and iPhone simulator.  Also in the readme on github you will find easy command line instructions on how to start and stop the simulator.
https://github.com/jameseisenhauer/iPhoneWebDriverSim

Selenium iPhone Driver

http://code.google.com/p/selenium/wiki/IPhoneDriver

Performance Testing in the Cloud.  No real servers required.  How do the solutions stack up?

The plan is to test all three four (added JMeter & BlazeMeter to the mix!) of the above tools and see what they can do!

The Scenario:  The slash dot effect….or whatever they call it now…hacker news’ed?
How about a contest!

Imagine you are going to run a Super Bowl commercial and instruct people to go to a website and submit as many entries as possible for a chance to win 1 million dollars!
You just might end up with the Faster Faster Pussycat Contest!
But can your web site handle the load?  Lets see!

Test Case: 250 concurrent users viewing & submitting entries.
Yeah yeah I know sounds more like pop warner super bowl…250 users…
If the tool will allow we will gradually ramp up the users from 1-250 and run for 1 minutes.

Test Script Steps:

1. Go to Contest site
2. Go to contest entry page
3. Fill out and submit entry form
4. Go back to main Contest site

The review criteria:

1. Cost
2. Ease of setup
3. Ease of scripting
4. Ability to handle complex test cases / Flexibility
5. Reporting
6. Integration with other tools/environments

The Test Environment:
Heroku Cedar – 2 Web Dyno / 0 Worker Dyno
Rails 3.2
Ruby 1.9.2
Unicorn
Memcache 5MB
Heroku 5mb Shared Database
New Relic monitoring

After each test reset the database to Zero records:

soasta.com

Well not a big fan of SOASTA.  Would have to give kudo’s to their marketing department.  The reason it made my list to try is because “I have been hearing” it’s a leader in the cloud performance testing area.  The tool available for download on their site is called CloudTest Light….I looked …I couldn’t find anything “cloud” about this tool.

blitz.io

I really like blitz.io.  I think it can become a really great tool.  It still feels kinda “beta”.  The documentation is sparse, major API changes so old support topics don’t make any sense, all interfaces don’t behave the same, support seems kinda slow (many support requests are started with “sorry it took so long…”), but it has some really cool things check them out!  This tool really shines in giving teams a general idea on how their site will perform without doing heavy scripting.  If you have a very quick release cycle and are constantly concerned with performance this tool would work well.

Test Case Result:

With 2 Web dynos 855 entries were able to be submitted.  To get an idea of scalability here are the numbers with different number of web dyno’s running.

1 dyno – 660 entries
2 dyno – 855 entries
3 dyno – 1100 entries

This is the results ran from the command line for our “official” test case.  You can see the site started failing around ~180 concurrent users.

loadui.org

LoadUI looks and feels slick.  It kinda passes the run a simple load test on a single URL test.  It was REALLY easy, BUT there is no Cloud yet…and am thinking setting up EC2 servers gonna take away all the easy bonus points .

After I ran the simple GET load test I found the tool doesn’t have out of the box simple http POST capability, just GET.  I saw some solutions where you could write it in groovy scripts, custom plugin or use SOAPUI blah blah…got board/annoyed and uninstalled it.  Not sure if their “PRO” version would solve these issues, but it costs $9900.00!!! for a single user license per YEAR!  So wasn’t even a thought to give the trial a chance.

I told you it looks slick!

jmeter.apache.org

JMeter seems to be the de facto standard when it comes to performance testing.  Obviously there is no cloud, but one would think you could set it up to run on the Amazon EC2 cloud.  I found a few people talking about it and it doesn’t seem like a trivial task, but it sounds like an interesting one.  But for the faint of heart somebody already did this for you and they call it Blazemeter!

To see how a non-cloud single instance of JMeter (running on my 8gig mac mini via ~35Mbps/~35Mbps Fios network) would fair running our “official” test case.

Well 250 entries and didn’t even begin to tax the single dyno instance of the fasterfaster site.  Need MOAR Power!

Screen Shot of the JMeter GUI

blazemeter.com

Lets put JMeter into the Cloud!  A really really really nice implementation of JMeter running on EC2.  You can just upload your JMeter test files and run them!  It is that easy!

I didn’t spend the time doing math to determine what I think of the cost, but at first glance it seems ~ reasonable~

They limit their test accounts to only 50 virtual users so we were not able to run our “Official” test case of 250 users.  BUT they let us run the test for 1 hour!  So we decided to run a “SOAK” test!! (or whatever you want to call them ).

We ramped up from 0-50 users over the first 5 minutes and then let 50 concurrent users hit the test case for the next 55 minutes.  We ended up with 6593 entries!!  About ~2 entries a second, compared to the blitz test where we averaged ~14 entries per second.

Blazmeter has some really nice reporting features.  You can see response times start to increase as we fill the database with more and more data.  Also I noticed that MemCache was having a fit because the data grew larger then it allowed, which I am sure added to the increase response times and errors we see in the next chart.

I am a strong believer that your server monitoring tools should be the same in your test environments as they are in your production ones.  AND New Relic does an awesome job at it!  If you haven’t seen them check out the link below.

New Relic

New Relic Snap shot showing the “Overview” view during the Blazemeter soak test.

If you actually made it this far amazing!!! What a giant blog article.  If you have any comments questions hit me up on Twitter  @jameseisenhauer

Really just in the early stages, but already have some cool stuff.

Architecture and design were highly inspired by Mikeal Rogers and this talk - http://www.youtube.com/watch?v=fWuwcSm_pUo

My designs continue to be inspired by the work he and Max Ogden are doing.

Once you put all your test results into one non-relational data store like  Apache CouchDB the possibilities become endless.

First thing I created was a simple yearly test frequency chart.  If you follow my blog you might notice that it looks very familiar to the Tweet History via CouchDB & JQuery Sparklines! project I did.  That’s because it is!  The tweet history project was the proof of concept learning phase for the technology I used to build the test result site.

The entire application is built on:

• HTML 5
• CSS 3
• JQuery
• JavaScript
• CouchDB & JSON

I loved this technology stack!  Very clean and simple.  It felt like the early days of web development before we tried to make web development EXACTLY like “normal” software development.  Well let me show you some of the other features I have implemented so far.

Viewing the build status was the priority Feature.

First page lists all the builds that were tested in the past week and gives high level status.

You can then drill down deeper into a build and see exactly what tests passed, failed or error-ed (is that a word?)  out.

Drilling even further down we can see detailed information about the test.  The exact steps that were taken, screen shots and verification points.

Whats next?!? Who knows, but I am thinking with all this data I am collecting something like this might be possible!

I will be showing off my Selenium Android Farmville Tweeter bot at the Portland Oregon Selenium User Group tomorrow.

What better way to learn Selenium and Android automation then to farm some crops on Zynga’s most famous game!

I will walk you through the steps I took to create my Selenium Farmville Tweeter Bot

First step is to install Eclipse

I recommend the Eclipse IDE for Java Developers version

Next step is to install the Android SDK

Once the SDK is installed launch eclipse and install the Android Eclipse Plugin

Follow the instructions on the Android Eclipse plugin page its easy as pie.

Once you get the plugin installed, it will ask you to restart eclipse.

Restart eclipse and the android sdk manager should pop up.  If it doesn’t you can find it in the menus.

Select install the latest version.

Once you finish the installation it is time to set up the emulator for testing with selenium.

I have an iphone so we will only be talking about the emulator .  Look for a future post of Selenium on the iphone.

The Selenium team has a great page on the Android Driver start here and move down to the “Setup the Emulator” section.  They give you instructions on how to set up the emulator via the command line.  You can also launch the Android Virtual Device Manager GUI within Eclipse

So I have done this on OSX, Linux and now windows.  When I went to create the emulator in windows I got this error.

“Unable to find a ‘userdata.img’ file for ABI armeabi to copy into the AVD folder”

Some googling and I find out I am missing something from the SDK.

So I open the SDK manager and like it is a mind reader it highlights the 2 packages I need….hmmmm

The ARM EABI v7a system Image & the Google USB Driver package

I installed those packages and was able to create my emulator without error.

Now that the emulator is working time to install the Webdriver APK.

Open up a command prompt/shell and follow the instructions here

First we need to start the emulator.  Browse to the tools directory where you put the android-sdk

emulator -avd my_Android

Ran into some errors trying to start the emulator on windows.  emulator: ERROR: unknown skin name ‘WVGA800′

I didn’t have any issues on the other operating systems.  Did some windows troubleshooting AKA rebooting.  I found the issue.  The SDK windows installer installed to C:\program files\android where the eclipse android plugin put it in my home directory.  I am running on XP but this looks like a mod for windows 7.  I modified the plugin to look for the sdk in the program files directory reinstalled some of the libraries and wala!  Ok lets forget this happened and dream of having a new mac book air.

Lets create the emulator in the eclipse gui and start it via command line:

emulator -avd my_Android

Ok so we have it running now where were we….ooh yeah install Webdriver APK.

Open up a command prompt/shell and follow the instructions here

• get the device serial ID
• install the apk
• start webdriver
• port forward

Now our Android environment is set up and running time to write some selenium scripts!

Lets test it out and see if it works.  I like to use Maven.  So that is what we are going to do.  Create a new project and select other-maven-maven project.  Maven is a build tool and makes it easy to do actions like download/install selenium and running tests among a million other things.

• Select the simple project check box.
• Create a junit test under the src/test/java package.  Make sure to create a folder under the default otherwise eclipse yells at you.

Almost forgot we need to add selenium!

Go to Maven – Add Dependency and search for selenium

Select org.seleniumhq.selenium selenium-java hit ok – click maven update dependencies

Now selenium is installed!! Lets go back to our junit test.

Replace the fail(“Not yet implemented”); with the sample code here

The code is a little old so I just take the code out of the testGoogle class and paste it into the test class.

Let eclipse do its work and auto-correct all the error, which imports the stuff we need!

Lets run it!  If you did everything correctly it runs.

So what should we do with all this new found power?  I know lets create a bot that harvests and replants my crops in Farmville and then tweets how awesome I am!

If you go to m.farmville.com it asks you to login via facebook.  Clicking on the link opens a new window for facebook.

In my script I just short-circuit this by opening m.facebook.com and logging in first before heading to m.farmville.com.

It maintains my session and auto logs me into the mobile farmville site.

Once we are in I do a check to see if my crops are ready to harvest!  I check it by doing an assert on the status message and if they are not ready I fail the test.

If they are ready it clicks into the screen for harvesting and continues to harvest and harvest and harvest…like over 200 times it clicks that button!  Glad I have Selenium on my side!

When it is done harvesting it got a little tricky.  Only because the behavior of the application is not consistent.  Sometimes it just grays out the harvest button while other times it completely disappears!  I put the check for “x 0″ crops in a try catch loop that will stop clicking if there is no more to harvest or if the button disappears it will throw an stale element error and handle it nicely.

Once harvesting is done its time to tweet our bumper crop!! (that’s what it’s called right??)  I grab the text elements for how much money I have and what level I am, set those to variables and head on over to m.twitter.com

Once I log into twitter and get session I forward directly to the tweet url instead of looking for a button to push.

Using the values in my $ & level vars I tweet out my Farmville awesomeness!

Here you can see it in all it’s twitter glory!  Mos6yCanTest is my test twitter account.  If you don’t know who John Mosby was you can google him….he is one of my heros.

I did have a lot of problems with stale errors being thrown.  I handle it with try catch and loops.  Not sure if this is due to the large amount of javascript in these mobile sites or if my emulator is lagging to much or what.  Anybody else have these issues with mobile web apps?

Well thats it the famous Farmville Tweeter App!

You can find my selenium scripts out on my github account I think I need to work on my gitignore files a little .

This month some co-workers and I started the brand new Portland Oregon Selenium Users group!

We met a bunch of people and learned a lot at the Google Test Automation Conference so we decided to continue with the momentum and start a user group here in portland.

If you are interested in test automation come join us!  http://www.meetup.com/pdx-se/

Just got back from GTAC. Wow what a great conference!   It was a invite only conference and google picked up the entire bill.  It was a complete who’s who in software test automation.  Made a lot of great contacts, learned a lot and have a bunch of ideas.  Next years is in Paris.  Better start working on my boss right away .

Check out the YouTube Play list of all the sessions

Me and my coworkers were featured in a IBM Rational Case Study.  We had a lot of success with the Rational tools, but we have moved on to Selenium for added features, community and multi-browser capability.

Regence BlueCross BlueShield of Oregon halves testing time

Testing automation with IBM Rational tools dramatically improves efficiency and effectiveness

http://www-01.ibm.com/software/success/cssdb.nsf/cs/CPAR-8M5MTF?OpenDocument&Site=rational&cty=en_us