Javascript Unit Tests with QUnit, Ant, and PhantomJS, Take 1

Recently I have been finding bugs in Javascript slip by a lot more easily than bugs in Scala, Java, or various other languages for which we write unit tests. jQuery seems to use QUnit (http://docs.jquery.com/QUnit) but QUnit appears to expect a web page to be setup to host it. This is better than nothing but really I want to run my js unit tests in an automated build, in my case using Ant.

The problem seemed remarkably likely to be solved already so I took to the Google and discovered some blogposts (http://twoguysarguing.wordpress.com/2010/11/26/qunit-cli-running-qunit-with-rhino/, http://twoguysarguing.wordpress.com/2010/11/06/qunit-and-the-command-line-one-step-closer/) where the author was attempting to achieve a command line unit test runner using Rhino and QUnit. Apparently John Resig tweeted some time ago (http://twitter.com/#!/jeresig/status/4477641447) to indicate QUnit should be operable in this manner so things seemed promising.

The twoguysarguing (great title) blog posts I found seemed to require modifications to QUnit source, plus Rhino not being a full browser apparently caused some issues as well. I really didn't want a custom version of QUnit, but the general approach seemed promising. In the comments for the second post someone suggested use of PhantomJS (http://twoguysarguing.wordpress.com/2010/11/06/qunit-and-the-command-line-one-step-closer/#comment-599), a headless WebKit browser. I decided to give this a try as it sounded remarkably reasonable.

My first step was to verify PhantomJS worked for me at all. It ran my first test without any issue:

//try1.js
console.log('Hello, World');
phantom.exit();

Executed similar to phantomjs try1.js this prints Hello, World just as one might hope.

The next question seemed to be whether or not PhantomJS could actually load QUnit using injectJs (ref http://code.google.com/p/phantomjs/wiki/Interface). I git cloned QUnit and attempted to invoke inject.Js on it:

//try2.js
if (window.QUnit == undefined)
 console.log('no QUnit yet!');
else
 console.log('somehow we already haz QUnit !!');
phantom.injectJs('D:\\Code\\3.7-scalaide\\JavaScriptUnitTests\\QUnit\\qunit.js');
if (window.QUnit != undefined)
 console.log('goodnes; injectJs seems to have worked');

phantom.exit();

This prints:

no QUnit yet!
goodnes; injectJs seems to have worked

So far so good!!

So, that means we should be able to setup and run a test, right? Something like this:

test("This test should fail", function() {
  console.log('the test is running!');
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});

test("This test should pass", function() {
  console.log('the test is running!');
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});

Well ... sadly this part didn't "just work". QUnit tries to execute the test queue on timers and despite PhantomJS supporting timers they just never seemed to execute. Furthermore, QUnit default feedback is via DOM modifications that are rather unhelpful to the PhantomJS runner. My first draft was to modify QUnit by adding a function that directly executed the test queue, inline, without using timers. This worked, but it required modifying QUnit source, which I specifically wish to avoid.

Luckily something similar to the changes to add a function to run QUnits tests directly works just fine outside QUnit as well. The key is that we will:

1. Track test pass/fail count via the QUnit.testDone callback (http://docs.jquery.com/Qunit#Integration_into_Browser_Automation_Tools)
1. We need our own pass/fail counters as QUnit tells us how many assertions passed/failed rather than how many tests passed/failed.
2. Track whether or not the test run is done overall via the QUnit.done callback (http://docs.jquery.com/Qunit#Integration_into_Browser_Automation_Tools)
3. Directly execute the QUnit test queue from our own code
1. hack but the point here is to see if we can make this work at all
4. Split tests into their own file
1. This facilitates using an Ant task to run a bunch of different test sets; eg using apply to pickup on all .js test files by naming convention or location convention
5. Return the count of failures as our PhantomJS exit code
1. This facilitates setting an Ant task to failonerror to detect unit test failures

So, without further ado, error handling, namespaces/packages, or any other cleanup here is a version that works in a manner very near to the desired final result:
try4.js

function importJs(scriptName) {
 console.log('Importing ' + scriptName);
 phantom.injectJs(scriptName);
}

console.log('starting...');

//Arg1 should be QUnit
importJs(phantom.args[0]);

//Arg2 should be user tests
var usrTestScript = phantom.args[1];
importJs(usrTestScript);

//Run QUnit
var testsPassed = 0;
var testsFailed = 0;

//extend copied from QUnit.js
function extend(a, b) {
 for ( var prop in b ) {
  if ( b[prop] === undefined ) {
   delete a[prop];
  } else {
   a[prop] = b[prop];
  }
 }

 return a;
}

QUnit.begin({});

// Initialize the config, saving the execution queue
var oldconfig = extend({}, QUnit.config);
QUnit.init();
extend(QUnit.config, oldconfig);

QUnit.testDone = function(t) {
 if (0 === t.failed) 
  testsPassed++;
 else
  testsFailed++;
  
 console.log(t.name + ' completed: ' + (0 === t.failed ? 'pass' : 'FAIL'))
}

var running = true;
QUnit.done = function(i) {
 console.log(testsPassed + ' of ' + (testsPassed + testsFailed) + ' tests successful');
 console.log('TEST RUN COMPLETED (' + usrTestScript + '): ' + (0 === testsFailed ? 'SUCCESS' : 'FAIL')); 
 running = false;
}

//Instead of QUnit.start(); just directly exec; the timer stuff seems to invariably screw us up and we don't need it
QUnit.config.semaphore = 0;
while( QUnit.config.queue.length )
 QUnit.config.queue.shift()();

//wait for completion
var ct = 0;
while ( running ) {
 if (ct++ % 1000000 == 0) {
  console.log('queue is at ' + QUnit.config.queue.length);
 }
 if (!QUnit.config.queue.length) {
  QUnit.done();
 }
}

//exit code is # of failed tests; this facilitates Ant failonerror. Alternately, 1 if testsFailed > 0.
phantom.exit(testsFailed);

try4-tests.js

test("This test should fail", function() {
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});

test("This test should pass", function() {
  equals( "hello", "hello", "We expect value to be hello" );
});

This runs as follows:

>phantomjs.exe try4.js qunit.js try4-tests.js
starting...
Importing qunit\qunit.js
Importing javascript\try4-tests.js
This test should fail completed: FAIL
This test should pass completed: pass
queue is at 0
1 of 2 tests successful
TEST RUN COMPLETED (try4-tests.js): FAIL

Note that we have not modified qunit.js, and we have split our tests into their own file. This allows us to easily set the whole thing up to run from Ant:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<project default="js-tests4"> 
 <target name="js-tests4">
  <property name="phantomjs.exe.file" value="phantomjs.exe" />
  <property name="qunit.js.file" location="path/to/qunit.js" />

  <apply executable="${phantomjs.exe.file}" failonerror="true">
   <arg value="path/to/try4.js"/>
   <arg value="${qunit.js.file}" />
   <srcfile/>
   
   <fileset dir="path/to/tests">
    <include name="try4-tests.js" />
   </fileset>
  </apply>
 </target>
</project>

It even works run this way:

js-tests4:
    [apply] starting...
    [apply] Importing path\to\qunit.js
    [apply] Importing path\to\try4-tests.js
    [apply] This test should fail completed: FAIL
    [apply] This test should pass completed: pass
    [apply] queue is at 0
    [apply] 1 of 2 tests successful
    [apply] TEST RUN COMPLETED (try4-tests.js): FAIL

BUILD FAILED

Note that Ant has detected that a js unit test failed and failed the build, just as we intended.

This leaves us with a proof of concept implementation that seems to prove that using PhantomJS to run QUnit based Javascript tests to run from a command line build is fundamentally possible. Doubtless if/when we try to use it "for real" additional problems will emerge ;)