Monday, July 11, 2011

Javascript Unit Tests with QUnit, Ant, and PhantomJS, Take 1

Recently I have been finding bugs in Javascript slip by a lot more easily than bugs in Scala, Java, or various other languages for which we write unit tests. jQuery seems to use QUnit (http://docs.jquery.com/QUnit) but QUnit appears to expect a web page to be setup to host it. This is better than nothing but really I want to run my js unit tests in an automated build, in my case using Ant.

The problem seemed remarkably likely to be solved already so I took to the Google and discovered some blogposts (http://twoguysarguing.wordpress.com/2010/11/26/qunit-cli-running-qunit-with-rhino/, http://twoguysarguing.wordpress.com/2010/11/06/qunit-and-the-command-line-one-step-closer/) where the author was attempting to achieve a command line unit test runner using Rhino and QUnit. Apparently John Resig tweeted some time ago (http://twitter.com/#!/jeresig/status/4477641447) to indicate QUnit should be operable in this manner so things seemed promising.

The twoguysarguing (great title) blog posts I found seemed to require modifications to QUnit source, plus Rhino not being a full browser apparently caused some issues as well. I really didn't want a custom version of QUnit, but the general approach seemed promising. In the comments for the second post someone suggested use of PhantomJS (http://twoguysarguing.wordpress.com/2010/11/06/qunit-and-the-command-line-one-step-closer/#comment-599), a headless WebKit browser. I decided to give this a try as it sounded remarkably reasonable.

My first step was to verify PhantomJS worked for me at all. It ran my first test without any issue:
//try1.js
console.log('Hello, World');
phantom.exit();
Executed similar to phantomjs try1.js this prints Hello, World just as one might hope.

The next question seemed to be whether or not PhantomJS could actually load QUnit using injectJs (ref http://code.google.com/p/phantomjs/wiki/Interface). I git cloned QUnit and attempted to invoke inject.Js on it:
//try2.js
if (window.QUnit == undefined)
 console.log('no QUnit yet!');
else
 console.log('somehow we already haz QUnit !!');
phantom.injectJs('D:\\Code\\3.7-scalaide\\JavaScriptUnitTests\\QUnit\\qunit.js');
if (window.QUnit != undefined)
 console.log('goodnes; injectJs seems to have worked');

phantom.exit();
This prints:
no QUnit yet!
goodnes; injectJs seems to have worked
So far so good!!

So, that means we should be able to setup and run a test, right? Something like this:
test("This test should fail", function() {
  console.log('the test is running!');
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});

test("This test should pass", function() {
  console.log('the test is running!');
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});
Well ... sadly this part didn't "just work". QUnit tries to execute the test queue on timers and despite PhantomJS supporting timers they just never seemed to execute. Furthermore, QUnit default feedback is via DOM modifications that are rather unhelpful to the PhantomJS runner. My first draft was to modify QUnit by adding a function that directly executed the test queue, inline, without using timers. This worked, but it required modifying QUnit source, which I specifically wish to avoid.

Luckily something similar to the changes to add a function to run QUnits tests directly works just fine outside QUnit as well. The key is that we will:
  1. Track test pass/fail count via the QUnit.testDone callback (http://docs.jquery.com/Qunit#Integration_into_Browser_Automation_Tools)
    1. We need our own pass/fail counters as QUnit tells us how many assertions passed/failed rather than how many tests passed/failed.
  2. Track whether or not the test run is done overall via the QUnit.done callback (http://docs.jquery.com/Qunit#Integration_into_Browser_Automation_Tools)
  3. Directly execute the QUnit test queue from our own code
    1. hack but the point here is to see if we can make this work at all
  4. Split tests into their own file
    1. This facilitates using an Ant task to run a bunch of different test sets; eg using apply to pickup on all .js test files by naming convention or location convention
  5. Return the count of failures as our PhantomJS exit code
    1. This facilitates setting an Ant task to failonerror to detect unit test failures
So, without further ado, error handling, namespaces/packages, or any other cleanup here is a version that works in a manner very near to the desired final result:
try4.js
function importJs(scriptName) {
 console.log('Importing ' + scriptName);
 phantom.injectJs(scriptName);
}

console.log('starting...');

//Arg1 should be QUnit
importJs(phantom.args[0]);

//Arg2 should be user tests
var usrTestScript = phantom.args[1];
importJs(usrTestScript);

//Run QUnit
var testsPassed = 0;
var testsFailed = 0;

//extend copied from QUnit.js
function extend(a, b) {
 for ( var prop in b ) {
  if ( b[prop] === undefined ) {
   delete a[prop];
  } else {
   a[prop] = b[prop];
  }
 }

 return a;
}

QUnit.begin({});

// Initialize the config, saving the execution queue
var oldconfig = extend({}, QUnit.config);
QUnit.init();
extend(QUnit.config, oldconfig);

QUnit.testDone = function(t) {
 if (0 === t.failed) 
  testsPassed++;
 else
  testsFailed++;
  
 console.log(t.name + ' completed: ' + (0 === t.failed ? 'pass' : 'FAIL'))
}

var running = true;
QUnit.done = function(i) {
 console.log(testsPassed + ' of ' + (testsPassed + testsFailed) + ' tests successful');
 console.log('TEST RUN COMPLETED (' + usrTestScript + '): ' + (0 === testsFailed ? 'SUCCESS' : 'FAIL')); 
 running = false;
}

//Instead of QUnit.start(); just directly exec; the timer stuff seems to invariably screw us up and we don't need it
QUnit.config.semaphore = 0;
while( QUnit.config.queue.length )
 QUnit.config.queue.shift()();

//wait for completion
var ct = 0;
while ( running ) {
 if (ct++ % 1000000 == 0) {
  console.log('queue is at ' + QUnit.config.queue.length);
 }
 if (!QUnit.config.queue.length) {
  QUnit.done();
 }
}

//exit code is # of failed tests; this facilitates Ant failonerror. Alternately, 1 if testsFailed > 0.
phantom.exit(testsFailed);

try4-tests.js
test("This test should fail", function() {
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});

test("This test should pass", function() {
  equals( "hello", "hello", "We expect value to be hello" );
});

This runs as follows:
>phantomjs.exe try4.js qunit.js try4-tests.js
starting...
Importing qunit\qunit.js
Importing javascript\try4-tests.js
This test should fail completed: FAIL
This test should pass completed: pass
queue is at 0
1 of 2 tests successful
TEST RUN COMPLETED (try4-tests.js): FAIL

Note that we have not modified qunit.js, and we have split our tests into their own file. This allows us to easily set the whole thing up to run from Ant:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<project default="js-tests4"> 
 <target name="js-tests4">
  <property name="phantomjs.exe.file" value="phantomjs.exe" />
  <property name="qunit.js.file" location="path/to/qunit.js" />

  <apply executable="${phantomjs.exe.file}" failonerror="true">
   <arg value="path/to/try4.js"/>
   <arg value="${qunit.js.file}" />
   <srcfile/>
   
   <fileset dir="path/to/tests">
    <include name="try4-tests.js" />
   </fileset>
  </apply>
 </target>
</project>

It even works run this way:
js-tests4:
    [apply] starting...
    [apply] Importing path\to\qunit.js
    [apply] Importing path\to\try4-tests.js
    [apply] This test should fail completed: FAIL
    [apply] This test should pass completed: pass
    [apply] queue is at 0
    [apply] 1 of 2 tests successful
    [apply] TEST RUN COMPLETED (try4-tests.js): FAIL

BUILD FAILED

Note that Ant has detected that a js unit test failed and failed the build, just as we intended.

This leaves us with a proof of concept implementation that seems to prove that using PhantomJS to run QUnit based Javascript tests to run from a command line build is fundamentally possible. Doubtless if/when we try to use it "for real" additional problems will emerge ;)

10 comments:

angel26 said...

Great info about the javascript unit tests with the Qunit, Ant.It will be very interesting working with.
web design company

Michael said...

Awesome, thanks! We're using a modified version of your try4.js as a wrapper, and would love to contribute improvements back (JUnit-compatible test result XML for Jenkins, among others). Is this hosted anywhere besides this blog? If so, let us know where, otherwise we can throw up our own git repo.

jackiebolinsky said...

This time you posted a very useful info about Javascript. I was unaware of it. Well, IMO QUnit or JUnit are some unit testing frameworks specifically for javascript that you can take a look at. Thanks dude.

JavaScript Countdown Timer

Kenny Chua said...

Hi,
If anyone is interested, I've written a Maven plugin to do this in a mavenised environment

Plugin at :
http://code.google.com/p/phantomjs-qunit-runner/

Example use at :
http://kennychua.net/blog/running-qunit-tests-in-a-maven-continuous-integration-build-with-phantomjs

ibjhb said...

Also, check out
http://code.google.com/p/phantomjs/wiki/TestFrameworkIntegration
for QUnit integration.

gaffleck said...

I'm getting an error when trying to run this "can't find variable: exports". Do you know why this might be?

Matthew Bridgeman said...

@gaffleck Updating my qunit.js library fixed this issue

la Alice said...

Amazing post with beautiful examples. Very good for inspiration.Thanks for sharing !blog Wordpress themes

David Borjas said...

for some reason when I am running this command: >phantomjs.exe try4.js qunit.js try4-tests.js
I only get this on the cmd:
starting...
Importing qunit-1.17.1.js
Importing try4-test.js


But when I comment out the QUnit.begin({}); Then it works fine, but I do not run any test.

Naviya Nair said...

I have read your blog its very attractive and impressive. I like it your blog.

Java Online Training Java EE Online Training Java EE Online Training Java 8 online training Core Java 8 online training

Java Online Training from India Java Online Training from India Core Java Training Online Core Java Training Online Java Training InstitutesJava Training Institutes

Post a Comment