Getting started with Sikuli as a Testing Tool

I've been working with Sikuli for a couple of years now. I first ran across it via Slashdot and thought it might make a good tool for creating automated tests for the application I was a developer on at the time. Sadly I never got to explore it much due to a lack of time, but I did get to start playing with it and experimenting. Over the last year I've been (with some extra help from a full time assistant) developed a fairly robust testing framework for my current job using it. I thought others might be interested in some of the things we discovered and how we work around various problems we've encountered while trying to scale Sikuli up as a useful part of our testing systems. This is the first post in a series.

So if you've never heard of Sikuli here's a quick description. It's basically an automation tool that you can use to drive GUI applications using screenshots. Scripts are composed in a combination of Python (via Jython) and screenshots. The Sikuli IDE provides you with a convenient interface for composing scripts (though there are some bugs/limitations/rough edges). If you want to see some examples you should start by checking out the Sikuli Blog for an extensive collection of examples. There are some handy tidbits hidden in there as well that I'll probably discuss in a later post.

Using Sikuli as a testing tool seems pretty obvious but there have been a bunch of things that have come up that took a more creative approach to solve than I realized going in. For this post we'll be going over some of the major issues and discussing how to work around them.

Too much work to update many tests... use a library

One of the major problems we ran into was that as our developers changed the interface to the application we were having to update our 100 test cases with the various UI changes. This is a major amount of effort. To try and mitigate this I developed a mechanism which we could put at the top of each of our test scripts to import the necessary bits from a central script. So what we ended up with is something like this: 

import sys
import os.path
importPath = os.path.join("/".join(sys.argv[0].split("/")[0:-3]),"meta")
print importPath
if not importPath in sys.path:
    sys.path.append(importPath)
    print sys.path
if "metasikuli" in sys.modules:
    del sys.modules["metasikuli"]  exec("from metasikuli import *")

This assumes a particular structure for our scripts in directories. We have a structure that looks like this: 

- sikuli
- tests
    - test001.sikuli
    - ...
    - testNNN.sikuli
- meta
    - metasikuli.sikuli
testrunner.sh

What this allows you to do is to manage a whole collection of test fragments for your applications common operations in a central place and this then greatly simplifies the actual tests making them much easier to update and maintain. This way when the designers of your application change the color of one of the buttons you can update it in one place*.

*We haven't been 100% consistent with how we manage the individual images in the metasikuli script so there are sometimes multiple places where an image will need to be updated, but I have found that by using variables for images it tends to make the scripts less readable... so that's not a good way to make it so things can be updated in one place only and apply across all the scripts. There are also mechanisms for managing all of the images in a single giant pool, but I have not experimented with these capabilities yet since our current process has been working okay.

Keep everything in source control

Where I work we use git... but any source control system (provided it's not too hard to work with binary files... is this even an issue anymore?) should do. Mainly this is a best practices thing. At the end of the day sikuli tests are essentially source code and so you want to manage them the same way. Since everything is managed in an source control system it made integrating the testing process with Jenkins a breeze. We just have Jenkins download the latest sources and then run the testrunner script. The system is automatic from that point.

Get a dedicated machine (if you can)

Something else we found out early on is that it's pretty handy to have a dedicated machine for running your Sikuli tests. Basically Sikuli takes over your machine while it is running and you can't use it. Trying to have other tasks (unless they don't involve a GUI at all) can interfere with your tests... so you want to avoid this as much as you can. Unfortunately Sikuli doesn't work so well with remote sessions (VNC/RDP) (though this might no longer be the case...). So if you can get a dedicated machine... I highly recommend it. Most of the testing we're doing currently is for iOS using the iOS Simulator... so we just got a cheap Mac Mini and that handles the work. There's a whole collection of issues we've run into related to the fact that we're doing iOS testing which deserves its own blog post.

Avoid the Unit Test capability

It's buggy and doesn't work as expected all the time. We ended up ignoring it it. What we do instead is log the output when running the tests from the command line (will discuss further in a post about the testrunner). We then parse those results looking for signs of test failure and then extract the errors using some shell scripting*.

*I should also mention... a lot of how I make use of Sikuli is informed by my long term use of Unix/Linux. I suspect that some of the techniques I use may be somewhat unorthodox... but when it comes to these sorts of tasks I found that the shell was much better able to help with breaking down problems into manageable chunks.

Conclusion

This is starting to get a little bit long so we'll end this here for now. Future posts in this series will cover some of the details. Here's a list of topics I plan to cover in future posts (will update with links when they get written):

If you've got a specific question about something feel free to post a comment and maybe I can write a post about it.

Other Resources