Monday, July 11, 2011

Javascript Unit Tests with QUnit, Ant, and PhantomJS, Take 1

Recently I have been finding bugs in Javascript slip by a lot more easily than bugs in Scala, Java, or various other languages for which we write unit tests. jQuery seems to use QUnit (http://docs.jquery.com/QUnit) but QUnit appears to expect a web page to be setup to host it. This is better than nothing but really I want to run my js unit tests in an automated build, in my case using Ant.

The problem seemed remarkably likely to be solved already so I took to the Google and discovered some blogposts (http://twoguysarguing.wordpress.com/2010/11/26/qunit-cli-running-qunit-with-rhino/, http://twoguysarguing.wordpress.com/2010/11/06/qunit-and-the-command-line-one-step-closer/) where the author was attempting to achieve a command line unit test runner using Rhino and QUnit. Apparently John Resig tweeted some time ago (http://twitter.com/#!/jeresig/status/4477641447) to indicate QUnit should be operable in this manner so things seemed promising.

The twoguysarguing (great title) blog posts I found seemed to require modifications to QUnit source, plus Rhino not being a full browser apparently caused some issues as well. I really didn't want a custom version of QUnit, but the general approach seemed promising. In the comments for the second post someone suggested use of PhantomJS (http://twoguysarguing.wordpress.com/2010/11/06/qunit-and-the-command-line-one-step-closer/#comment-599), a headless WebKit browser. I decided to give this a try as it sounded remarkably reasonable.

My first step was to verify PhantomJS worked for me at all. It ran my first test without any issue:
//try1.js
console.log('Hello, World');
phantom.exit();
Executed similar to phantomjs try1.js this prints Hello, World just as one might hope.

The next question seemed to be whether or not PhantomJS could actually load QUnit using injectJs (ref http://code.google.com/p/phantomjs/wiki/Interface). I git cloned QUnit and attempted to invoke inject.Js on it:
//try2.js
if (window.QUnit == undefined)
 console.log('no QUnit yet!');
else
 console.log('somehow we already haz QUnit !!');
phantom.injectJs('D:\\Code\\3.7-scalaide\\JavaScriptUnitTests\\QUnit\\qunit.js');
if (window.QUnit != undefined)
 console.log('goodnes; injectJs seems to have worked');

phantom.exit();
This prints:
no QUnit yet!
goodnes; injectJs seems to have worked
So far so good!!

So, that means we should be able to setup and run a test, right? Something like this:
test("This test should fail", function() {
  console.log('the test is running!');
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});

test("This test should pass", function() {
  console.log('the test is running!');
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});
Well ... sadly this part didn't "just work". QUnit tries to execute the test queue on timers and despite PhantomJS supporting timers they just never seemed to execute. Furthermore, QUnit default feedback is via DOM modifications that are rather unhelpful to the PhantomJS runner. My first draft was to modify QUnit by adding a function that directly executed the test queue, inline, without using timers. This worked, but it required modifying QUnit source, which I specifically wish to avoid.

Luckily something similar to the changes to add a function to run QUnits tests directly works just fine outside QUnit as well. The key is that we will:
  1. Track test pass/fail count via the QUnit.testDone callback (http://docs.jquery.com/Qunit#Integration_into_Browser_Automation_Tools)
    1. We need our own pass/fail counters as QUnit tells us how many assertions passed/failed rather than how many tests passed/failed.
  2. Track whether or not the test run is done overall via the QUnit.done callback (http://docs.jquery.com/Qunit#Integration_into_Browser_Automation_Tools)
  3. Directly execute the QUnit test queue from our own code
    1. hack but the point here is to see if we can make this work at all
  4. Split tests into their own file
    1. This facilitates using an Ant task to run a bunch of different test sets; eg using apply to pickup on all .js test files by naming convention or location convention
  5. Return the count of failures as our PhantomJS exit code
    1. This facilitates setting an Ant task to failonerror to detect unit test failures
So, without further ado, error handling, namespaces/packages, or any other cleanup here is a version that works in a manner very near to the desired final result:
try4.js
function importJs(scriptName) {
 console.log('Importing ' + scriptName);
 phantom.injectJs(scriptName);
}

console.log('starting...');

//Arg1 should be QUnit
importJs(phantom.args[0]);

//Arg2 should be user tests
var usrTestScript = phantom.args[1];
importJs(usrTestScript);

//Run QUnit
var testsPassed = 0;
var testsFailed = 0;

//extend copied from QUnit.js
function extend(a, b) {
 for ( var prop in b ) {
  if ( b[prop] === undefined ) {
   delete a[prop];
  } else {
   a[prop] = b[prop];
  }
 }

 return a;
}

QUnit.begin({});

// Initialize the config, saving the execution queue
var oldconfig = extend({}, QUnit.config);
QUnit.init();
extend(QUnit.config, oldconfig);

QUnit.testDone = function(t) {
 if (0 === t.failed) 
  testsPassed++;
 else
  testsFailed++;
  
 console.log(t.name + ' completed: ' + (0 === t.failed ? 'pass' : 'FAIL'))
}

var running = true;
QUnit.done = function(i) {
 console.log(testsPassed + ' of ' + (testsPassed + testsFailed) + ' tests successful');
 console.log('TEST RUN COMPLETED (' + usrTestScript + '): ' + (0 === testsFailed ? 'SUCCESS' : 'FAIL')); 
 running = false;
}

//Instead of QUnit.start(); just directly exec; the timer stuff seems to invariably screw us up and we don't need it
QUnit.config.semaphore = 0;
while( QUnit.config.queue.length )
 QUnit.config.queue.shift()();

//wait for completion
var ct = 0;
while ( running ) {
 if (ct++ % 1000000 == 0) {
  console.log('queue is at ' + QUnit.config.queue.length);
 }
 if (!QUnit.config.queue.length) {
  QUnit.done();
 }
}

//exit code is # of failed tests; this facilitates Ant failonerror. Alternately, 1 if testsFailed > 0.
phantom.exit(testsFailed);

try4-tests.js
test("This test should fail", function() {
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});

test("This test should pass", function() {
  equals( "hello", "hello", "We expect value to be hello" );
});

This runs as follows:
>phantomjs.exe try4.js qunit.js try4-tests.js
starting...
Importing qunit\qunit.js
Importing javascript\try4-tests.js
This test should fail completed: FAIL
This test should pass completed: pass
queue is at 0
1 of 2 tests successful
TEST RUN COMPLETED (try4-tests.js): FAIL

Note that we have not modified qunit.js, and we have split our tests into their own file. This allows us to easily set the whole thing up to run from Ant:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<project default="js-tests4"> 
 <target name="js-tests4">
  <property name="phantomjs.exe.file" value="phantomjs.exe" />
  <property name="qunit.js.file" location="path/to/qunit.js" />

  <apply executable="${phantomjs.exe.file}" failonerror="true">
   <arg value="path/to/try4.js"/>
   <arg value="${qunit.js.file}" />
   <srcfile/>
   
   <fileset dir="path/to/tests">
    <include name="try4-tests.js" />
   </fileset>
  </apply>
 </target>
</project>

It even works run this way:
js-tests4:
    [apply] starting...
    [apply] Importing path\to\qunit.js
    [apply] Importing path\to\try4-tests.js
    [apply] This test should fail completed: FAIL
    [apply] This test should pass completed: pass
    [apply] queue is at 0
    [apply] 1 of 2 tests successful
    [apply] TEST RUN COMPLETED (try4-tests.js): FAIL

BUILD FAILED

Note that Ant has detected that a js unit test failed and failed the build, just as we intended.

This leaves us with a proof of concept implementation that seems to prove that using PhantomJS to run QUnit based Javascript tests to run from a command line build is fundamentally possible. Doubtless if/when we try to use it "for real" additional problems will emerge ;)

23 comments:

Anonymous said...

Great info about the javascript unit tests with the Qunit, Ant.It will be very interesting working with.
web design company

Michael said...

Awesome, thanks! We're using a modified version of your try4.js as a wrapper, and would love to contribute improvements back (JUnit-compatible test result XML for Jenkins, among others). Is this hosted anywhere besides this blog? If so, let us know where, otherwise we can throw up our own git repo.

Anonymous said...

This time you posted a very useful info about Javascript. I was unaware of it. Well, IMO QUnit or JUnit are some unit testing frameworks specifically for javascript that you can take a look at. Thanks dude.

JavaScript Countdown Timer

Kenny Chua said...

Hi,
If anyone is interested, I've written a Maven plugin to do this in a mavenised environment

Plugin at :
http://code.google.com/p/phantomjs-qunit-runner/

Example use at :
http://kennychua.net/blog/running-qunit-tests-in-a-maven-continuous-integration-build-with-phantomjs

Anonymous said...

Also, check out
http://code.google.com/p/phantomjs/wiki/TestFrameworkIntegration
for QUnit integration.

gaffleck said...

I'm getting an error when trying to run this "can't find variable: exports". Do you know why this might be?

Matthew Bridgeman said...

@gaffleck Updating my qunit.js library fixed this issue

la Alice said...

Amazing post with beautiful examples. Very good for inspiration.Thanks for sharing !blog Wordpress themes

David Borjas said...

for some reason when I am running this command: >phantomjs.exe try4.js qunit.js try4-tests.js
I only get this on the cmd:
starting...
Importing qunit-1.17.1.js
Importing try4-test.js


But when I comment out the QUnit.begin({}); Then it works fine, but I do not run any test.

Naviya Nair said...

I have read your blog its very attractive and impressive. I like it your blog.

Java Online Training Java EE Online Training Java EE Online Training Java 8 online training Core Java 8 online training

Java Online Training from India Java Online Training from India Core Java Training Online Core Java Training Online Java Training InstitutesJava Training Institutes

Sugantha Raja said...

I got more ideas from this blog, Once again thanks for sharing the good and valuable information.

Java Training in Chennai | Java Training Institute in Chennai

sathya shri said...
This comment has been removed by the author.
ananya mca said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.

rpa training in velachery| rpa training in tambaram |rpa training in sholinganallur | rpa training in annanagar| rpa training in kalyannagar

Ram priya said...

Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic.
Data Science Training in Chennai | Data Science course in anna nagar
Data Science course in chennai | Data science course in Bangalore
Data Science course in marathahalli | Data Science course in btm

john jersy said...

This is good site and nice point of view.I learnt lots of useful information.
python training in velachery | python training institute in chennai

Richa T said...

Wonderful bloggers like yourself who would positively reply encouraged me to be more open and engaging in commenting.So know it's helpful.
java training in chennai | java training in bangalore


java training in tambaram | java training in velachery

shethal said...

Thank you for allowing me to read it, welcome to the next in a recent article. And thanks for sharing the nice article, keep posting or updating news article.

advanced excel training in bangalore

vijay antony said...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

rpa interview questions and answers
automation anywhere interview questions and answers
blueprism interview questions and answers
uipath interview questions and answers
rpa training in chennai

sumathi s said...

Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.occupational health and safety course in chennai

Swetha Gauri said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…
safety course in chennai

bala D said...

Does your blog have a contact page? I’m having problems locating it but, I’d like to shoot you an email. I’ve got some recommendations for your blog you might be interested in hearing.
Best Amazon Web Services Training in Pune | Advanced AWS Training in Pune
Advanced AWS Training in Chennai | No.1 Amazon Web Services Training in Chennai
Best Amazon Web Services Training in Chennai |Advancced AWS Training in Chennai
Best Amazon Web Services Online Training | Advanced Online Amazon Web Services Certification Course Training
Best Amazon Web Services Training in Pune | Advanced AWS Training in Pune

amsa leka said...

Hey, Wow all the posts are very informative for the people who visit this site. Good work! We also have a Website. Please feel free to visit our site. Thank you for sharing. Well written article Thank You for Sharing with Us pmp training centers in chennai| pmp training in velachery | project management courses in chennai | project management training in chennai | project management certification online | project management course online

Technical Support said...

Excellent Wok, Thanks for sharing with us this valuable blog. I get the solution to my problem. Visit for
Web Design Companies - How to Choose The Best One For Your Website

Post a Comment