Friday, November 9, 2012

Unit Testing with an In-Memory Database for a Spring/Hibernate Project


The joy of Hibernate


For tasks not on a critical load/performance path I often find Hibernate handy to whip together some simple data access, particularly for simple CRUD. However, being a bungling buffoon I usually mess up my Hibernate mapping, setup queries that refer to non-existent properties, and so on. Naturally I aspire to resolve this by having automated tests of my DB code.

Testing DB code is often painful with any significant number of people. Often you either need every developer to have their own database, with it's own uniquely inconsistent data and ways of failing, or to use a shared database in which case people often manage to produce tests that don't work if more than one person runs the tests concurrently. Also testing on a "real" DB is slow and subject to failure for reasons not related to your code. To give a few of many examples, Joe removes columns at random, Wendy has a gift for locking tables, Tim has a test replication setup that doesn't work and stops his tests from running, Alice writes tests that only work if run in the correct order (Test B presumes data from test A still exists), and the continuous integration system doesn't have a DB of it's very own. Also the DBA randomly shuts down the shared DB and causes everyone's tests to fail.

So, let's try to set ourselves up so our tests can't share data, run fast, yield consistent results, and are not subject to external changes (like a common database changing) causing them to break. Our goal is that we spin up a new, clean, in-memory database for each test, run the test, and then dump the in-memory db.

In this article I walk through a complete test setup using Spring 3.1, Hibernate 4, and HSQLDB in a Maven project using Eclipse (with Maven Integration for Eclipse) as an editor. I built a fairly similar setup (same idea but built with Ant and Ivy) for tests on some projects I work with at my day job with considerable success so perhaps this will be helpful to someone else.

A significant portion of the post is devoted to setting up the project with dependencies, Hibernate configuration, Spring configuration, and all the other nonsense accompanying a modern Java project. Feel free to skip to the final section if you just want to glance over how the in-memory testing is wired in.

Basic Project Setup

Create a new Java Project (force of habit; I never remember the 'new Maven project option until too late, including when blogging):

Setup Maven style folders:
  1. Delete the 'src' folder
  2. Create four new source folders
    1. src/main/java
    2. src/main/resources
    3. src/test/java
    4. src/test/resources
Your project should look something like this:


Now right-click the project and choose Configure>Convert to Maven Project. Use the default values for Maven setup:

Now you have a pom.xml in the root of your project where you can setup dependencies and compilation options. The first change we want to make is to tell our project to build with Java 1.6 so we can use the spiffy @Resource annotation in our tests:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>InMemoryDbTests</groupId>
  <artifactId>InMemoryDbTests</artifactId>
  <version>0.0.1-SNAPSHOT</version>
    
  <build>
   <plugins>
    <!-- Please be a Java 1.6 application -->
  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>2.0.2</version>
      <configuration>
          <source>1.6</source>
          <target>1.6</target>
      </configuration>
  </plugin>  
   </plugins>  
  </build>
</project>

Now we're ready for dependencies.

Dependencies

We "simply" need to get an obscure set of jars in compatible versions and all will be ready to go! Trying to get all required dependencies with everything compatible is possibly one of the least fun jobs in all of Java. But I digress. Add dependencies to pom.xml, shown below with comments indicating what each library is for:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>InMemoryDbTests</groupId>
  <artifactId>InMemoryDbTests</artifactId>
  <version>0.0.1-SNAPSHOT</version>
    
  <dependencies>
 
 <!-- hibernate is a core part of the exercise --> 
 <dependency>
  <groupId>org.hibernate</groupId>
  <artifactId>hibernate-core</artifactId>
  <version>4.1.7.Final</version>
 </dependency>
 
 <!-- connection pooling -->
 <dependency>
  <groupId>commons-dbcp</groupId>
  <artifactId>commons-dbcp</artifactId>
  <version>1.4</version>
 </dependency>          
 
 <!-- for the Spring org.springframework.orm.hibernate4.LocalSessionFactoryBean -->
 <dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-orm</artifactId>
  <version>3.1.2.RELEASE</version>
 </dependency>            
               
 <!-- we want Spring -->              
 <dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-core</artifactId>
  <version>3.1.2.RELEASE</version>
 </dependency>
 <!-- we want Spring to manage our transactions -->
 <dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-tx</artifactId>
  <version>3.1.2.RELEASE</version>
 </dependency>              
 
 <!-- Spring annotations work via proxies which use CGLIB -->
 <dependency>
  <groupId>cglib</groupId>
  <artifactId>cglib-nodep</artifactId>
  <version>2.2.2</version>
 </dependency>
              
 
 <!-- TEST DEPENDENCIES -->
 <!-- provides the Spring JUnit test runner & @ContextConfiguration -->
 <dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-test</artifactId>
  <version>3.1.2.RELEASE</version>
  <scope>test</scope>
 </dependency>            
 <!-- to run our tests with -->
 <dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>4.10</version>
  <scope>test</scope>
 </dependency>        
 <!-- our in-memory database provider -->     
 <dependency>
  <groupId>org.hsqldb</groupId>
  <artifactId>hsqldb</artifactId>
  <version>2.2.8</version>
  <scope>test</scope>
 </dependency>
                                       
  </dependencies>  
    
  <build>
   <plugins>
    <!-- Please be a Java 1.6 application -->
  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>2.0.2</version>
      <configuration>
          <source>1.6</source>
          <target>1.6</target>
      </configuration>
  </plugin>  
   </plugins>  
  </build>
</project>
If you think this is not strictly simple nor particularly obvious I would say you are right. http://mvnrepository.com/ is your friend if you know you want a dependency (say Spring test) but don't know the exact artifact, version, or what have you for it.

Data Access Configuration

OK, so we've got all our dependencies. Now we're ready to setup for our tests. Let us hypothesize that:
  • in src/main/resources we have persistence.xml, a Spring file that defines our key data access beans
    • A data source bean named "dataSource"
    • A session factory named "sessionFactory"
    • A transaction manager, in this case the Spring Hibernate 4 Transaction manager
      • Applied automatically via Spring's annotation-driven transaction management
    • The key properties of all beans are set via properties
  • in src/main/resources we have db.properties
    • This specifies the DB driver, URL, password, and so on
For test purposes we could override the data source and/or session factory completely, but because we factored out most of the key settings into properties we can just override a few key values from db.properties at test time.

On to the gory details. Create a persistence.xml file in src/main/resources with the following content:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:tx="http://www.springframework.org/schema/tx"
    xsi:schemaLocation="
            http://www.springframework.org/schema/beans      
            http://www.springframework.org/schema/beans/spring-beans-3.1.xsd 
            http://www.springframework.org/schema/tx  
            http://www.springframework.org/schema/tx/spring-tx-3.1.xsd">

  <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
    <property name="driverClassName" value="${jdbc.driverClassName}"/>
    <property name="url" value="${jdbc.url}"/>
    <property name="username" value="${jdbc.username}"/>
    <property name="password" value="${jdbc.pwd}"/>
    <property name="defaultAutoCommit" value="${jdbc.defaultAutoCommit}"/>
    <property name="accessToUnderlyingConnectionAllowed" value="${jdbc.accessToUnderlyingConnectionAllowed}"/>
    <property name="initialSize" value="${jdbc.initialSize}"/>
    <property name="maxActive" value="${jdbc.maxActive}"/>
    <property name="maxIdle" value="${jdbc.maxIdle}"/>
    <property name="minIdle" value="${jdbc.minIdle}"/>
    <property name="maxWait" value="${jdbc.maxWait}"/>
    <property name="validationQuery" value="${jdbc.validationQuery}"/>
    <property name="testOnBorrow" value="${jdbc.testOnBorrow}"/>
    <property name="testOnReturn" value="${jdbc.testOnReturn}"/>
    <property name="testWhileIdle" value="${jdbc.testWhileIdle}"/>
    <property name="timeBetweenEvictionRunsMillis" value="${jdbc.timeBetweenEvictionRunsMillis}"/>
    <property name="numTestsPerEvictionRun" value="${jdbc.numTestsPerEvictionRun}"/>
    <property name="minEvictableIdleTimeMillis" value="${jdbc.minEvictableIdleTimeMillis}"/>
  </bean>
  
  <bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
   <property name="dataSource" ref="dataSource" />
   <property name="mappingResources">
     <list>
       <value>hibernate-mapping.xml</value>
     </list>
   </property>
   <property name="hibernateProperties">
     <props>
       <prop key="hibernate.dialect">${hibernate.dialect}</prop>
       <prop key="hibernate.hbm2ddl.auto">none</prop>      
       <prop key="hibernate.connection.release_mode">auto</prop>
       <prop key="hibernate.show_sql">${hibernate.showSql}</prop>
       <prop key="hibernate.format_sql">true</prop>
       <prop key="hibernate.use_sql_comments">true</prop>
       <prop key="hibernate.generate_statistics">true</prop>
       <prop key="hibernate.jdbc.use_scrollable_resultset">true</prop>
       <prop key="hibernate.jdbc.use_streams_for_binary">true</prop>
       <prop key="hibernate.jdbc.batch_size">${hibernate.batchSize}</prop>
       <prop key="hibernate.cache.region_prefix">hibernate.cache</prop>
       <prop key="hibernate.cache.use_query_cache">false</prop>
       <prop key="hibernate.cache.use_second_level_cache">false</prop>
     </props>
   </property>
  </bean>
  
  <bean id="transactionManager" class="org.springframework.orm.hibernate4.HibernateTransactionManager">
    <property name="sessionFactory" ref="sessionFactory" />
  </bean>
  <tx:annotation-driven transaction-manager="transactionManager" /> 
</beans>

Next, create a db.properties file in src/main/resources with the following (obviously dummy; we don't care about the "real" db connection setup for this example) values:
jdbc.driverClassName=DB DRIVER CLASS, SAY SQL OR ORACLE
jdbc.url=SOME DB URL
jdbc.username=USERNAME
jdbc.pwd=PWD
jdbc.defaultAutoCommit=false
jdbc.accessToUnderlyingConnectionAllowed=false
jdbc.initialSize=5
jdbc.maxActive=100
jdbc.maxIdle=-1
jdbc.minIdle=${jdbc.initialSize}
jdbc.maxWait=15000
jdbc.validationQuery=SELECT DATE OR SOMETHING
jdbc.testOnBorrow=true
jdbc.testOnReturn=false
jdbc.testWhileIdle=false
jdbc.timeBetweenEvictionRunsMillis=-1
jdbc.numTestsPerEvictionRun=3
jdbc.minEvictableIdleTimeMillis=1800000

hibernate.dialect=A DIALECT
hibernate.showSql=false
#0 = off
hibernate.batchSize=0
In reality some of these would vary by environment (DB URL, user/pwd for sure) and might want to be further externalized rather than bundled in here.

Hibernate Configuration

We need some sort of Hibernate entity we can use to create a sample in-memory test. We'll setup a simplified person class and map it. In src/main/resources create hibernate-mapping.xml with the following mapping:
<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
        "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
        "http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">

<hibernate-mapping package="com.blogspot.whileonefork.domain">
  <class name="com.blogspot.whileonefork.domain.Person" table="people">
 <id name="id" type="long" column="id" unsaved-value="0">
        <generator class="identity"/>
 </id>    
    <property name="firstName" column="first_name" />
    <property name="lastName" column="last_name" />   
  </class>  
</hibernate-mapping>

In src/main/java in the com.blogspot.whileonefork.domain create the Person class:
package com.blogspot.whileonefork.domain;

public class Person {
 private Long id;
 private String firstName;
 private String lastName;
 
 public Long getId() {
  return id;
 }
 public void setId(Long id) {
  this.id = id;
 }
 public String getFirstName() {
  return firstName;
 }
 public void setFirstName(String firstName) {
  this.firstName = firstName;
 }
 public String getLastName() {
  return lastName;
 }
 public void setLastName(String lastName) {
  this.lastName = lastName;
 }
}
Finally we'll make ourselves a simple helper class to manage transactions and encapsulate simple CRUD operations. In src/main/java in the com.blogspot.whileonefork.db package create DaoAid.java with the following content:
package com.blogspot.whileonefork.db;

import java.util.List;

import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.criterion.Criterion;
import org.springframework.transaction.annotation.Isolation;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;

public class DaoAid {
 private static final int MAX_RESULT_SIZE = 128;
 
 private SessionFactory sessionFactory;
 
 public SessionFactory getSessionFactory() {
  return sessionFactory;
 }

 public void setSessionFactory(SessionFactory sessionFactory) {
  this.sessionFactory = sessionFactory;
 }
 
 private Session getSession() {
  return sessionFactory.getCurrentSession();
 }

 @Transactional(isolation = Isolation.READ_UNCOMMITTED, propagation = Propagation.REQUIRED)
 public <T> List<T> list(Class<T> classOfT,
   List<? extends Criterion> criteriaList, int maxResults) {
  if (null == classOfT) {
   throw new IllegalArgumentException("classOfT");
  }
  if (maxResults < 1) {
   throw new IllegalArgumentException("maxResults must be >= 1");
  }
  Session session = getSession();

  org.hibernate.Criteria criteria = session.createCriteria(classOfT);
  criteria.setMaxResults(Math.min(maxResults, MAX_RESULT_SIZE));

  if (null != criteriaList) {
   for (Criterion criterion : criteriaList) {
    criteria.add(criterion);
   }
  }

  @SuppressWarnings("unchecked")
  List<T> list = criteria.list();

  return list;
 }

 @Transactional(isolation = Isolation.READ_UNCOMMITTED, propagation = Propagation.REQUIRED)
 public <T> T findById(Class<T> classOfT, long id) {
  @SuppressWarnings("unchecked")
  T result = (T) getSession().get(classOfT, id);
  return result;
 }

 @Transactional(isolation = Isolation.READ_COMMITTED, propagation = Propagation.REQUIRED)
 public <T> T write(T instance) {
  @SuppressWarnings("unchecked")
  T savedInstance = (T) getSession().merge(instance);
  return savedInstance;
 }

 @Transactional(isolation = Isolation.READ_COMMITTED, propagation = Propagation.REQUIRED)
 public <T> void delete(T instance) {
  getSession().delete(instance);  
 }
}

In Memory Testing

At long last we find our way to the point, doing some Hibernate operations against an in-memory database. We want to verify we can interact with a clean database for each test. We'll create a base class that sets up and tears down our in-memory database before/after each test, then write a simple test to do some data access to show it all works.

First of all, our base class for in-memory tests. In src/test/java, in the com.blogspot.whileonefork.db package, create InMemoryDbTestBase.java with the following content:
package com.blogspot.whileonefork.db;

import java.sql.Connection;

import javax.annotation.Resource;
import javax.sql.DataSource;

import org.hibernate.cfg.Configuration;
import org.hibernate.tool.hbm2ddl.SchemaExport;
import org.junit.After;
import org.junit.Before;
import org.springframework.orm.hibernate4.LocalSessionFactoryBean;
import org.springframework.test.context.ContextConfiguration;

@ContextConfiguration
public class InMemoryDbTestBase {
 //&: get the actual factory, not the object it produced 
 @Resource(name="&sessionFactory")
 private LocalSessionFactoryBean sf;
 
 @Resource
 private DataSource dataSource;
 
 private Configuration cfg;
 private Connection conn; 
 
 @Resource
 private DaoAid daoAid; 
 
 @Before
 public void setup() throws Exception {
  if (null == cfg) {
   if (null == sf) {
    throw new IllegalStateException("No LocalSessionFactoryBean; perhaps you didn't setup @RunWith and @ContextConfiguration on your test?");
   }
   cfg = sf.getConfiguration();     
  }
  
  conn = dataSource.getConnection();
  SchemaExport exporter = new SchemaExport(cfg, conn);
  exporter.execute(true, true, false, true);
  
  if (null != exporter.getExceptions() && exporter.getExceptions().size() > 0) {
   throw new IllegalStateException("Unable to setup schema; export failed");
  }  
 } 
 
 @After
 public void teardown() throws Exception {
  //see http://hsqldb.org/doc/guide/running-chapt.html#rgc_closing_db
  if (null != conn) {
   conn.createStatement().execute("SHUTDOWN");
   conn.close();
   conn = null;
  }
 }
 
 
 public DaoAid getDaoAid() {
  return daoAid;
 }

 public void setDaoAid(DaoAid daoAid) {
  this.daoAid = daoAid;
 } 
 
 
}

Creating a new connection to the HSQLDB will spin up a new, blank, database. The SchemaExport class will then create schema to match our hibernate mapping. Finally in teardown, run after each test, we shutdown the database which will release the in-memory db.

We have slapped @ContextConfiguration onto InMemoryDbTestBase so we need a Spring file to setup the beans for the test. Create src/test/resources/com/blogspot/whileonefork/db/InMemoryDbTestBase-context.xml with the following content:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:tx="http://www.springframework.org/schema/tx"
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="
            http://www.springframework.org/schema/beans      
            http://www.springframework.org/schema/beans/spring-beans-3.1.xsd 
            http://www.springframework.org/schema/tx  
            http://www.springframework.org/schema/tx/spring-tx-3.1.xsd
            http://www.springframework.org/schema/context
            http://www.springframework.org/schema/context/spring-context-3.1.xsd">

 <!-- for an in-memory test we need to load the db properties, then the test properties, then the persistence bits -->
 <context:property-placeholder location="classpath:db.properties, classpath:db.inmemory-overrides.properties"/> 
 <import resource="classpath:/persistence.xml" />
</beans>
Note that we load db.properties before db.inmemory-overrides.properties. This allows db.inmemory-overrides.properties to overwrite any properties of the same name in db.properties. We then import persistence.xml, which will load using the in memory db driver and connection URL. We only need to replace a few properties. Create src/test/resources/db.inmemory-overrides.properties with the following content:
jdbc.driverClassName=org.hsqldb.jdbcDriver
jdbc.url=jdbc:hsqldb:mem:unittestdb
jdbc.username=sa
jdbc.pwd=

jdbc.validationQuery=values(current_timestamp)

hibernate.dialect=org.hibernate.dialect.HSQLDialect
hibernate.showSql=true
So, we should now be able to author a test that does some data access and verifies that it operates on a clean database. Create PersonListOfOneTest.java in src/test/java, in the com.blogspot.whileonefork.domain package with the following content:
package com.blogspot.whileonefork.domain;

import static org.junit.Assert.assertEquals;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import com.blogspot.whileonefork.db.InMemoryDbTestBase;

/**
 * Save an item and list Person's with no criteria, twice.
 * Should both see 1 item because DB is scrubbed each time.
 *
 */
@RunWith(SpringJUnit4ClassRunner.class)
public class PersonListOfOneTest extends InMemoryDbTestBase {  
 @Test
 public void list1() {
  assertEquals(0, getDaoAid().list(Person.class, null, 8).size());
  getDaoAid().write(new Person());
  assertEquals(1, getDaoAid().list(Person.class, null, 8).size());
 }
 
 @Test
 public void list2() {
  assertEquals(0, getDaoAid().list(Person.class, null, 8).size());
  getDaoAid().write(new Person());
  assertEquals(1, getDaoAid().list(Person.class, null, 8).size());
 } 
}

Execute the test; it should pass. This means our code successfully:

  1. Created a connection to a new HSQLDB in-memory instance for each test
  2. Ran the Hibernate SchemaExport to build the desired table structure to allow data access
  3. Wrote a record into the in-memory database and loaded it back
If the in-memory database wasn't getting cleaned up between tests then the second test to execute would return two records from the call to list(). Mission accomplished! 

Your final project should look like this:

In practice I have used a similar approach to test both simple and complex data access across a large project with an extensive test suite executing fast and with reliable results and near elimination of problems involving data access. 

Friday, February 17, 2012

Moving from Google Translate API to Microsoft Translate API in Scala

Google Translate used to be gods gift to developers who want to verify their internationalization works. It made localizing to some random locale for test purposes downright trivial. And then the bastards went and deprecated it (http://googlecode.blogspot.com/2011/05/spring-cleaning-for-some-of-our-apis.html), ultimately making it it pay to play.

The rates for Google Translate are so low it is unlikely we can justify switching to Microsoft Translate based on cost but we love "free" so we'll spend a few expensive hours of developer time on it anyway. M$ translate is free for your first 2M characters and you can pay for more.

The first thing that one notices upon trying to call a Microsoft web service is that it isn't as easy as you'd like. From reading the http://api.microsofttranslator.com/V2/Http.svc/Translate API to getting a successful translation call through from code took WAY longer than Google Translate (or other APIs) and had more bumps in the road. I'm sure others have differing experiences but for me I had programmatic access to Google Translate working maybe 30-60 minutes (it was a while ago) after I decided to do it versus 2-3 hours to get Microsoft Translate working.

Trying to get Microsoft Translate working I ran into the following complications:
  1. A multi-step registration process yielding numerous constants you send in to their API’s
    1. https://code.google.com/apis/console beats the hell out of https://datamarket.azure.com/account (the Microsoft equivalent as far as I can tell)
  2. An extra http call to get a token that you have to modify before you send it back to them (addRequestHeader("Authorization", "Bearer " + tok.access_token))
    1. Bonus points for inconsistent description of how this worked, although I believe this is now fixed
  3. Inaccurate documentation of arguments
    1. I believe this is now fixed
  4. Unhelpful error messages
  5. Inconsistent documentation of how to send in authorization data 
    1. I believe this is now fixed
  6. ISO-639 2-char language codes for all languages except Chinese.
    1. Chinese requires use of zh-CHS or zh-CHT to distinguish traditional vs simplified. Apparently having "zh" default to one or the other (probably simplified) is less trouble than having this be the exception case to how everything else works.
In case this is useful to someone else, here is the code (Scala 2.9.1.final, . First up, the API our clients will call into:

class Translator(val client: HttpClient) extends Log {
  //We kept Google around just in case we decide to pay for the service one day
  private var translationServices = List(new Google(client), new Microsoft(client))

  def this() = this(new HttpClient())

  def apply(text: String, fromLang: String, toLang: String): String = {
    if (fromLang != toLang && StringUtils.isNotBlank(text))
      translate(text, fromLang, toLang)
    else
      text
  }

  private def translate(text: String, fromLang: String, toLang: String): String = {  
    for (svc <- translationServices) {
      try {
        val tr = svc(text, fromLang, toLang)
        if (StringUtils.isNotBlank(tr)) {
          return tr
        }
      } catch {
        case e: Exception =>
          logger.warn("Translation failed using " + svc.getClass().getSimpleName() + ": " + e.getMessage() + ", moving on...")
      }
    }
    
    return ""
  }
}

Consumers will call into the Translator using code similar to:

  //translate Hello, World from English to Chinese
  var tr = new Translator()
  tr("Hello, World", "en", "zh") 

The interesting part is of course the actual Microsoft implementation:

/**
 * The parent TranslationService just defines def apply(text: String, fromLang: String, toLang: String): String
 */
class Microsoft(client: HttpClient) extends TranslationService(client) with Log {

  private val tokenUri = "https://datamarket.accesscontrol.windows.net/v2/OAuth2-13"
  private val translateUri = "http://api.microsofttranslator.com/V2/Http.svc/Translate"
  private val encoding = "ASCII"

  private val appKey = Map("client_id" -> "THE NAME OF YOUR APP", "client_secret" -> "YOUR CLIENT SECRET")

  private var token = new MsAccessToken

  def this() = this(new HttpClient())

  /**
   * Ref http://msdn.microsoft.com/en-us/library/ff512421.aspx
   */
  override def apply(text: String, fromLang: String, toLang: String): String = {
    
    /**
     * Always try to re-use an existing token
     */
    val firstTry:Option[String] = try {
      Some(callTranslate(token, text, fromLang, toLang))
    } catch {
      case e: Exception =>
        logger.info("Failed to re-use token, will retry with a new one. " + e.getMessage())
        None
    }
    
    /**
     * If we didn't get it using our old token try try again.
     * 99% of the time we do a bunch in a row and it works first time; occasionally we end up
     * needing a new key.
     * Code in block won't run unless firstTry is None.
     */
    val response = firstTry getOrElse {  
      this.token = requestAccessToken()
      callTranslate(token, text, fromLang, toLang)
    }
    

    //response is similar to: <string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">Hallo Welt</string>
    val translation = StringUtils.substringAfter(StringUtils.substringBeforeLast(response, "</string>"), ">")
    translation
  }

  private def callTranslate(tok: MsAccessToken, text: String, fromLang: String, toLang: String) = {
    val get = new GetMethod(translateUri)

    //Thanks MSFT, it's awesome that the language codes are *almost* ISO 639...
    //We need to specify our type of Chinese, http://www.emreakkas.com/internationalization/microsoft-translator-api-languages-list-language-codes-and-names
    val adjustedToLang = if (toLang.equalsIgnoreCase("zh")) "zh-CHS" else toLang

    val queryPairs = Array(
      new NameValuePair("appId", ""),
      new NameValuePair("text", text),
      new NameValuePair("from", fromLang),
      new NameValuePair("to", adjustedToLang))
    get.setQueryString(queryPairs)

    /**
     * http://msdn.microsoft.com/en-us/library/hh454950.aspx
     */
    get.addRequestHeader("Authorization", "Bearer " + tok.access_token)

    val rawResponse = try {
      val sc = client executeMethod get
      val response = get getResponseBodyAsString ()
      if (sc != HttpStatus.SC_OK) {
        throw new IllegalArgumentException("Error translating; Microsoft translate request '"
          + translateUri + "?" + get.getQueryString()
          + "' failed with unexpected code " + sc + ", response: " + response)
      }
      response
    }
    rawResponse
  }

  /**
   * Ref http://msdn.microsoft.com/en-us/library/hh454950.aspx
   */
  def requestAccessToken(): MsAccessToken = {

    val post = new PostMethod(tokenUri)
    post.setParameter("grant_type", "client_credentials")
    post.setParameter("client_id", appKey("client_id"))
    post.setParameter("client_secret", appKey("client_secret"))
    post.setParameter("scope", "http://api.microsofttranslator.com")

    val rawResponse = try {
      val sc = client executeMethod post
      val response = post getResponseBodyAsString ()
      if (sc != HttpStatus.SC_OK) {
        throw new IllegalArgumentException("Error translating; Microsoft access token request failed with unexpected code " + sc + ", response: " + response)
      }
      response
    } finally {
      post releaseConnection
    }

    val tok = Json.fromJson[MsAccessToken](rawResponse, classOf[MsAccessToken])

    tok
  }
}

The MsAccessToken is a rich and exciting class:
  /**
   * Ref http://msdn.microsoft.com/en-us/library/hh454950.aspx
   */
  class MsAccessToken(var access_token: String, var token_type: String, var expires_in: String, var scope: String) {
    def this() = this(null, null, null, null)
  }


A few third party libraries are in play here. For Http we are using the Apache HttpClient. For JSON we are using Google's excellent Gson library, with a simple implementation of 'using' to make working with java.io cleaner:

object Json {
 def writeJson(something: Any): String = new Gson().toJson(something) 
 
 def writeJson(something: Any, os: OutputStream):Unit 
  = using(new OutputStreamWriter(os)) { osw => new Gson().toJson(something, osw) }
 
 def fromJson[T](json: String, graphRoot: Type):T 
  = new Gson().fromJson(json, graphRoot) 
}

The using() function is in another class; it looks like this (ref http://whileonefork.blogspot.com/2011/03/c-using-is-loan-pattern-in-scala.html):
 def using[T <: {def close(): Unit}, R](c: T)(action: T => R): R = {
  try {
   action(c)
  } finally {
   if (null != c) c.close
  }
 }

And with that our Scala calling of M$ Translate is complete!

Monday, November 14, 2011

What is in my Core Data managed store?

Suppose you have an iOS application. Suppose it uses the Core Data API to persist data. And let us further suppose that you are having a few problems; it seems like the data in your store isn't exactly as you expect.

It might prove useful to be able to get some basic information about what the heck is happening with your data at runtime. For example, let us suppose you have an API in your code that lets you get a reference to the NSManagedObjectModel. We can then print out what entities are in your database and how many of them. Create a simple ViewController with a multi-line text view on it and setup an outlet named 'output'. In the viewDidAppear method gather up some basic diagnostic information and spit it out to the output text view:
//NOTE: sample assumes ARC, hence no calls to release
//in the .m file for your diagnostic output

- (void)viewDidAppear:(BOOL)animated
{
    [super viewDidAppear:animated];
    
    NSString *diagnostics = @"";
    
    @try {
        diagnostics = [NSString stringWithFormat:@"%@%@", diagnostics, @"Data Diagnostics\n"];
        NSManagedObjectModel *mom = //CALL WHATEVER YOUR APP USES TO GET NSManagedObjectModel
        if (mom) {
            for (NSEntityDescription *entity in mom.entities) {
                NSUInteger countOf = [self countOfEntity:entity];
                diagnostics = [NSString stringWithFormat:@"%@  entity:%@ (%d records)\n", diagnostics, entity.name, countOf];
            }
        } else {
            diagnostics = [NSString stringWithFormat:@"%@%@", diagnostics, @"  objectModel is nil :(\n"];
        }
    }
    @catch (NSException *exception) {
        diagnostics = [NSString stringWithFormat:@"%@%@", diagnostics, @"  ERROR getting data diagnostics\n"];
    }
    
    output.text = diagnostics;
}

//note that this could return "NSNotFound if an error occurs"; this typically shouldn't be the case since we got the entity description directly from our object model
- (NSUInteger) countOfEntity :(NSEntityDescription*)entity
{
    NSFetchRequest *fr = [[NSFetchRequest alloc] init];
    fr.entity = entity;
    [fr setIncludesSubentities:NO];
    
    NSError *err;
 NSManagedObjectContext *moc = //CALL WHATEVER YOUR APP USES TO GET NSManagedObjectContext
    NSUInteger count = [moc countForFetchRequest:fr error:&err];
    return count;
}
If you print this to a text view your output should end up something like this:
For some types of app this is surprisingly useful.

The main things I personally find this useful for are:

  1. Identifying glaring errors, such as "your model is completely missing an entity type; something is very wrong"
  2. Identifying errors where data isn't added or removed as expected, such as "when you do X in the UI there should be +1 of those (or -2 of those or whatever) if I quickly peek at diagnostics"
    1. Hmm ... perhaps it would be useful if the diagnostic view actually told you the delta for each type since you last looked?
There are many ways to wire this in; it doesn't have to be via it's own view. Another approach I often find useful to to have a debug/logged mode enabled only in development where this type of information prints to the console when key events occur in the app.


Sunday, October 16, 2011

How to wait while an NSTimer runs in a unit test

Suppose you were a lowly noob to iOS development. Further suppose you had a Cool Idea (tm) which involved timers. The internets might rapidly guide you to NSTimer and you might decide to try to get it to log to the console in a unit test. The most obvious approach seems to be to setup a timer to tick frequently, lets say every 0.1 seconds, and setup a timer callback that logs something, then make a test that sleeps for a couple of seconds. Presumably during the sleep period we'll see a bunch of timer output. The code might look like this (inside an XCode 4.2 test implementation class):

- (void)onTimerTick:(NSTimer*)timer
{
    NSLog(@"MY TIMER TICKED");
}

- (void)testTimerBasics
{
    NSLog(@"timer time");
    
    [NSTimer scheduledTimerWithTimeInterval:0.1
                   target:self
                   selector:@selector(onTimerTick:)
                   userInfo:nil
                   repeats:YES];
    
    [timer fire]; //manually calling fire DOES log 'MY TIMER TICKED'
    
    NSLog(@"about to wait");    
    [NSThread sleepForTimeInterval:2.0]; //absolutely no logs of 'MY TIMER TICKED' occur; somehow the time doesn't fire during a thread sleep :(
    NSLog(@"wait time is over");    
}

Sadly absolutely no log messages are printed during our two second sleep ([NSThread sleepForTimeInterval:2.0]) ; WTF?!

After much Google and literally in the midst of typing a Stack Overflow question I came across a question involving waiting for something else that mentioned NSRunLoop in passing. The very existence of a run loop class suggests an answer: our tests run on the same thread as the run loop. This means if we put the run loop to sleep nothing gets processed. Instead of sleep we need some sort of "run the run loop for a while" approach. Luckily it turns out that NSRunLoop provides a runUntilDate API so we can re-write the test above as follows:

- (void)onTimerTick:(NSTimer*)timer
{
    NSLog(@"MY TIMER TICKED");
}

- (void)testTimerBasics
{
    NSLog(@"timer time");
    
    NSTimer *timer = [NSTimer scheduledTimerWithTimeInterval:0.1
                              target:self
                              selector:@selector(onTimerTick:)
                              userInfo:nil
                              repeats:YES];
    
    //[timer fire];
    
    NSDate *runUntil = [NSDate dateWithTimeIntervalSinceNow: 3.0 ];
    
    NSLog(@"about to wait");    
    [[NSRunLoop currentRunLoop] runUntilDate:runUntil];
    NSLog(@"wait time is over");    
}
We've found the right magic incantation! Knuth would be proud.

Speaking of magic incantations, I am using the SyntaxHighlighter libraries hosted @ http://syntaxhighlighter.googlecode.com/svn/trunk/Scripts/. However, there is no Objectionable-C brush there so I took the one posted @ http://www.undermyhat.org/blog/wp-content/uploads/2009/09/shBrushObjectiveC.js and updated it the casing and namespace names to the newer highlighter standard. The updated brush looks like this:


dp.sh.Brushes.ObjC = function()
{
 var datatypes = 'char bool BOOL double float int long short id void';
 
 var keywords = 'IBAction IBOutlet SEL YES NO readwrite readonly nonatomic nil NULL ';
 keywords += 'super self copy ';
 keywords += 'break case catch class const copy __finally __exception __try ';
 keywords += 'const_cast continue private public protected __declspec ';
 keywords += 'default delete deprecated dllexport dllimport do dynamic_cast ';
 keywords += 'else enum explicit extern if for friend goto inline ';
 keywords += 'mutable naked namespace new noinline noreturn nothrow ';
 keywords += 'register reinterpret_cast return selectany ';
 keywords += 'sizeof static static_cast struct switch template this ';
 keywords += 'thread throw true false try typedef typeid typename union ';
 keywords += 'using uuid virtual volatile whcar_t while';
 // keywords += '@property @selector @interface @end @implementation @synthesize ';
 
  
 this.regexList = [
  { regex: dp.sh.RegexLib.SingleLineCComments,  css: 'comments' },  // one line comments
  { regex: dp.sh.RegexLib.MultiLineCComments,  css: 'comments' },  // multiline comments
  { regex: dp.sh.RegexLib.DoubleQuotedString,  css: 'string' },   // double quoted strings
  { regex: dp.sh.RegexLib.SingleQuotedString,  css: 'string' },   // single quoted strings
  { regex: new RegExp('^ *#.*', 'gm'),      css: 'preprocessor' },  // preprocessor
  { regex: new RegExp(this.GetKeywords(datatypes), 'gm'),  css: 'datatypes' },  // datatypes
  { regex: new RegExp(this.GetKeywords(keywords), 'gm'),  css: 'keyword' },   // keyword
  { regex: new RegExp('\\bNS\\w+\\b', 'g'),     css: 'keyword' },   // keyword
  { regex: new RegExp('@\\w+\\b', 'g'),      css: 'keyword' },   // keyword
  ];
 this.CssClass = 'dp-objc';
 this.Style = '.dp-objc .datatypes { color: #2E8B57; font-weight: bold; }'; 
}
dp.sh.Brushes.ObjC.prototype = new dp.sh.Highlighter();
dp.sh.Brushes.ObjC.Aliases  = ['objc'];

Friday, October 14, 2011

Integrating Javascript tests into a CLI build

Wherein we walk through a basic setup for running Javascript unit tests on the command line. After some initial investigation (here) I didn't find time to get back to Javascript unit testing until recently. I have now managed to get Javascript unit tests running fairly gracefully in a command line build at work; here is an outline of how, simplified from the "real" implementation to highlight the basics. Fortunately a great deal of the work is done for us, always nice when it turns out that way.

We are going to run everything off the filesystem to avoid having our tests impacted by external influences.

Part 1: Basic test setup
  1. Create a directory to house Javascript unit test files; we will refer to this as \jsunit henceforth when giving paths. 
  2. Download QUnit.js and QUnit.css into \jsunit
  3. Download run-qunit.js into \jsunit
  4. Create a file testme.js in \jsunit with the following content
    /**
     * var-args; adds up all arguments and returns sum
     */
    function add() {
    }
    
  5. Create a file testme.test.htm in \jsunit with the following content
    • Note we are using local filesystem paths to load all content; we have no external dependencies
    • <!DOCTYPE html>
      <html>
      <head>
      	<!-- we need QUnit as a test runner -->
          <link rel="stylesheet" href="qunit.css" type="text/css" media="screen" />
          <script src="qunit.js"></script>
      	
      	<!-- we'd like to have the file we're going to test -->
          <script src="testme.js"></script>
      	
      	<!-- and finally lets write some tests -->
      	<script>
      		console.log("test time baby");
      
      		test("add is defined", function() {
      			equals(typeof window.add, "function", "add isn't a function :(");
      		});
      	</script>
          
      </head>
      <body>
      	 <h1 id="qunit-header">QUnit Tests</h1>
      	 <h2 id="qunit-banner"></h2>
      	 <div id="qunit-testrunner-toolbar"></div>
      	 <h2 id="qunit-userAgent"></h2>
      	 <ol id="qunit-tests"></ol>
      	 <div id="qunit-fixture"></div>    
      </body>
      </html>
      
  6. Download PhantomJS (1.3.0 at time of writing)
    • For example PhantomJS commands I will assume it is on PATH (eg phantomjs args); use the qualified path if not (eg C:\where\phantom\is\phantomjs args)
  7. Open testme.test.htm in a browser; it should look like this:

  8. Open a command prompt, navigate to \jsunit and run phantomjs run-qunit.js testme.test.htm
    • Output should be similar to:
      test time baby
      'waitFor()' finished in 211ms.
      Tests completed in 57 milliseconds.
      1 tests of 1 passed, 0 failed.
      
    • Note we don't see any "test blah pass" or "test two fail" style output
Part 2: CLI build integration prep
So far so good, now we need to get setup to run in a CLI build. There a couple of things we'd like here, most of which are already implemented in run-qunit.js:
  1. Output each test pass/fail
  2. Output log messages from tests to the console
    1. This "just works" courtesy of run-qunit.js, yay!
  3. Exit with non-zero error code if tests fail
    1. This makes it easy for build to detect failure and do something in response; for example an Ant build could simply set failonerror
    2. This "just works" courtesy of run-qunit.js, yay!
We just have to setup output of test pass/fail information to the console. We'll add a test that fails to show what that looks like. Proceed as follows:
  1. Create a file test-support.js in \jsunit with the following content:
    //create a scope so we don't pollute global
    (function() {  
       var testName;
       
       //arg: { name }
    	QUnit.testStart = function(t) {
    	    testName = t.name;
    	};
    	
    	//arg: { name, failed, passed, total }
    	QUnit.testDone = function(t) {
    	    console.log('Test "' + t.name + '" completed: ' + (0 === t.failed ? 'pass' : 'FAIL'))
    	};
    	
    	//{ result, actual, expected, message }
    	QUnit.log = function(t) {
    	    if (!t.result) {
    	        console.log('Test "' + testName + '" assertion failed. Expected <' + t.expected + '> Actual <' + t.actual + '>' + (t.message ? ': \'' + t.message + '\'' : ''));
    	    }
    	};
    }());
    
  2. Edit testme.test.htm to pull in test-support.js and add a test that will currently fail
    • <!DOCTYPE html>
      <html>
      <head>
      	<!-- we need QUnit as a test runner -->
          <link rel="stylesheet" href="qunit.css" type="text/css" media="screen" />
          <script src="qunit.js"></script>
      	
      	<!-- where would our tests be without support! -->
      	<script src="test-support.js"></script>
      	
      	<!-- we'd like to have the file we're going to test -->
          <script src="testme.js"></script>
      	
      	<!-- and finally lets write some tests -->
      	<script>
      	
      		test("add is defined", function() {
      			equals(typeof window.add, "function", "add isn't a function :(");
      		});
      		
      		test("add 1+1", function() {
      			equals(add(1, 1), 2);
      		});		
      	</script>
          
      </head>
      <body>
      	 <h1 id="qunit-header">QUnit Tests</h1>
      	 <h2 id="qunit-banner"></h2>
      	 <div id="qunit-testrunner-toolbar"></div>
      	 <h2 id="qunit-userAgent"></h2>
      	 <ol id="qunit-tests"></ol>
      	 <div id="qunit-fixture"></div>    
      </body>
      </html>
      
  3. Open a command prompt, navigate to \jsunit and run phantomjs run-qunit.js testme.test.htm
    • Output should be similar to:
      Test "add is defined" completed: pass
      Test "add 1+1" assertion failed. Expected <2> Actual <undefined>
      Test "add 1+1" completed: FAIL
      'waitFor()' finished in 209ms.
      Tests completed in 70 milliseconds.
      1 tests of 2 passed, 1 failed.
    • If you print the exit code (echo %ERRORLEVEL% in Windoze) you should get a 1, indicating we have fulfilled the 'exit with non-zero exit code on failure' requirement :)
Part 3: Ant integration
At long last we are ready to integrate this mess into a build. For this example I will use Ant and will assume Ant is on PATH. At time of writing I am using Ant 1.8.2.

  1. Create a phantomjs.bat file in \jsunit with the following content
    @echo off
    C:\where\you\put\phantom\phantomjs.exe %*
    
    • Alternately create phantomjs.sh with equivalent functionality if on *nix
  2. Create a build.xml file in \jsunit with the following content
    • <?xml version="1.0" encoding="UTF-8"?>
      <project name="jsunittests" basedir="." default="main">
      	<property name="builddir" location="${basedir}/target"/>
      	
      	<condition property="phantom.filename" value="phantomjs.bat"><os family="windows"/></condition>
      	<condition property="phantom.filename" value="phantomjs.sh"><os family="unix"/></condition>   
      	
      	<target name="clean">
      		<delete dir="${builddir}"/>
      	</target>
      	
      	<target name="prep">
      		<mkdir dir="${builddir}"/>
      	</target>
      	
      	<target name="jstest">
            <!--Run all tests w/phantom, fail if tests fail. Execute all files w/extension .test.htm. -->
            <apply executable="${phantom.filename}" failonerror="true" dir="${basedir}" relative="true">
               <arg value="run-qunit.js"/>
               <fileset dir="${basedir}">
                  <include name="**/*.test.htm" />
               </fileset>
            </apply>			
      	</target>
      	
      	<target name="main" depends="clean, prep, jstest">
      	</target>
      </project>
      
  3. Run 'ant'; you should get output similar to the following (yes, it's supposed to fail, remember we have a test that fails setup on purpose)
    • Buildfile: build.xml
      
      clean:
         [delete] Deleting directory C:\Code\jsunit-trial\target
      
      prep:
          [mkdir] Created dir: C:\Code\jsunit-trial\target
      
      jstest:
          [apply] Test "add is defined" completed: pass
          [apply] Test "add 1+1" assertion failed. Expected <2> Actual 
          [apply] Test "add 1+1" completed: FAIL
          [apply] 'waitFor()' finished in 218ms.
          [apply] Tests completed in 58 milliseconds.
          [apply] 1 tests of 2 passed, 1 failed.
      
      BUILD FAILED
      C:\Code\jsunit-trial\build.xml:18: apply returned: 1
      
      Total time: 0 seconds
      
  4. Edit testme.js just enough to fix the test
    • /**
       * var-args; adds up all arguments and returns sum
       */
      function add() {
      	var sum =0;
      	for (var i=0; i<arguments.length; i++)
      		sum += arguments[i];
      	return sum;
      }
  5. Run 'ant'; you should get output similar to the following
    • Buildfile: build.xml
      
      clean:
         [delete] Deleting directory C:\Code\jsunit-trial\target
      
      prep:
          [mkdir] Created dir: C:\Code\jsunit-trial\target
      
      jstest:
          [apply] Test "add is defined" completed: pass
          [apply] Test "add 1+1" completed: pass
          [apply] 'waitFor()' finished in 214ms.
          [apply] Tests completed in 59 milliseconds.
          [apply] 2 tests of 2 passed, 0 failed.
      
      main:
      
      BUILD SUCCESSFUL
      Total time: 0 seconds
Pretty sweet, we've got Javascript tests running in an Ant build as a first-class citizen. Now if you break my Javascript my Continuous Integration server will let me know!

Part 4: Code coverage
Finally we are ready to get some code coverage. We are going to get code coverage by instrumenting our js files using JSCoverage, running our QUnit tests such that the relative paths resolve to the instrumented copies, and then using the PhantomJS file system APIs to create a colorized copy of the original js file to visually display coverage. We'll do a quick and dirty percentage coverage output to the console as well.


  1. Download JSCoverage 0.5.1
  2. Create a jscoverage.bat file in \jsunit with the following content
    @echo off
    C:\where\you\put\jscoverage\jscoverage.exe %*
    
  3. Create a template file for coverage information named coverageBase.htm in \jsunit
    • <!DOCTYPE html>
      <html>
      <head>
          <style>
              .code {
                  white-space: pre;
                  font-family: courier new;
                  width: 100%;            
              }
              
              .miss {
                  background-color: #FF0000;
              }
              
              .hit {
                  background-color: #94FF7C;
              }
              
              .undef {
                  background-color: #AFFF9E;
              }        
          </style>
      </head>
      <body>
      
      COLORIZED_LINE_HTML
      
      </body>
      </html>
      
  4. Update build.xml to perform a few new steps
    1. Create a \target\testjs\js directory and copy our js files into it
    2. Index our js files for code coverage, putting the indexed version into \target\testjs\jsinstrumented
    3. Copy *.test.htm into \target\testhtm
    4. Copy base resources to run tests (run-qunit.js, qunit.js, qunit.css) into \target\testhtm
    5. Copy the instrumented js files into \target\testhtm
      1. Note that because we used relative paths to our test js files the *.test.htm QUnit html files will now resolve js to the instrumented version when we run the files out of \target\testhtm
    6. Run PhantomJS on *.test.htm in \target\testhtm
    7. The updated build.xml looks like this:
    8. <?xml version="1.0" encoding="UTF-8"?>
      <project name="jsunittests" basedir="." default="main">
      	<property name="builddir" location="${basedir}/target"/>
      	<property name="jstestdir" location="${builddir}/testjs"/>
      	<property name="jsdir" location="${jstestdir}/js"/>
      	<property name="jsinstrumenteddir" location="${jstestdir}/jsinstrumented"/>
      	<property name="testhtmdir" location="${builddir}/testhtm"/>
      	
      	<condition property="phantom.filename" value="phantomjs.bat"><os family="windows"/></condition>
      	<condition property="phantom.filename" value="phantomjs.sh"><os family="unix"/></condition>   
      	
      	<property name="jscoverage.filename" value="jscoverage.bat" />
      	
      	<target name="clean">
      		<delete dir="${builddir}"/>
      	</target>
      	
      	<target name="prep">
      		<mkdir dir="${jsdir}"/>
      		<mkdir dir="${jsinstrumenteddir}"/>		
      		<mkdir dir="${testhtmdir}"/>
      		
      		<!-- copy non-test js files to target so we can mess with 'em. how we select which files may vary; for this 
      			 example just pick the one file we are testing.-->
      		<copy todir="${jsdir}">
      			<fileset dir="${basedir}">
      				<include name="testme.js" />
      			</fileset>
      		</copy>
      				
      		<!-- run jscoverage to produce a version of the file instrumented for code coverage -->
      		<exec executable="${jscoverage.filename}" failonerror="true">
      			<arg value="${jsdir}"/>
      			<arg value="${jsinstrumenteddir}"/>
      		</exec>   		
      		
      		<!-- copy our test htm files and modify them to point to the coverage indexed version of the test file. -->
      		<copy todir="${testhtmdir}">
      			<fileset dir="${basedir}">
      				<include name="**/*.test.htm" />
      			</fileset>
      		</copy>		
      		
      		<!-- copy core resources to testhtmdir so we can load them with same paths as when executing test htm files directly -->
      		<copy todir="${testhtmdir}">
      			<fileset dir="${jsinstrumenteddir}">
      				<include name="**/*.js" />
      				<exclude name="jscoverage.js"/>
      			</fileset>
      		</copy>				
      		<copy todir="${testhtmdir}">
      			<fileset dir="${basedir}">
      				<include name="test-support.js" />
      				<include name="run-qunit.js" />
      				<include name="qunit.css" />
      				<include name="qunit.js" />
      			</fileset>
      		</copy>				
      	</target>
      	
      	<target name="jstest">
            <!--Run all tests w/phantom, fail if tests fail. Execute all files w/extension .test.htm. -->
            <apply executable="${basedir}/${phantom.filename}" failonerror="true" dir="${testhtmdir}" relative="false">
               <arg value="run-qunit.js"/>
      		 <srcfile/>
      		 <arg value="${basedir}"/>
               <fileset dir="${testhtmdir}">
                  <include name="**/*.test.htm" />
               </fileset>
            </apply>			
      	</target>
      	
      	<target name="main" depends="clean, prep, jstest">
      	</target>
      </project>
      
  5. Modify our test-support.js to look for jscoverage data and output a rough count of lines hit, missed, and irrelevant (non-executable). Also expose a function a caller outside of page context can use to access coverage information. The new version should look like this:
    • //create a scope so we don't pollute global
      (function() {  
         var testName;
         
         //arg: { name }
      	QUnit.testStart = function(t) {
      	    testName = t.name;
      	};
      	
      	//arg: { name, failed, passed, total }
      	QUnit.testDone = function(t) {
      	    console.log('Test "' + t.name + '" completed: ' + (0 === t.failed ? 'pass' : 'FAIL'))
      	};
      	
      	//{ result, actual, expected, message }
      	QUnit.log = function(t) {
      	    if (!t.result) {
      	        console.log('Test "' + testName + '" assertion failed. Expected <' + t.expected + '> Actual <' + t.actual + '>' + (t.message ? ': \'' + t.message + '\'' : ''));
      	    }
      	};
      	
      	//we want this at global scope so outside callers can find it. In a more realistic implementation we
      	//should probably put it in a namespace.
      	window.getCoverageByLine = function() {
      		var key = null;
              var lines = null;
              //look for code coverage data    
              if (typeof _$jscoverage === 'object') {
      			for (key in _$jscoverage) {}
      			lines = _$jscoverage[key];
              } 
      
      		if (!lines) {
                 console.log('code coverage data is NOT available');
              } 
              		
              return { 'key': key, 'lines': lines };
         };
      
         QUnit.done = function(t) {
              var cvgInfo = getCoverageByLine();
              if (!!cvgInfo.lines) {
                  var testableLines = 0;
                  var testedLines = 0;
      			var untestableLines = 0;
                  for (lineIdx in cvgInfo.lines) {
      				var cvg = cvgInfo.lines[lineIdx];
      				if (typeof cvg === 'number') {
      					testableLines += 1;
      					if (cvg > 0) {
      						testedLines += 1;
      					}					
      				} else {
      					untestableLines += 1;
      				}
                  }     
                  var coverage = '' + Math.floor(100 * testedLines / testableLines) + '%';
                  
      			var result = document.getElementById('qunit-testresult');
      			if (result != null) {
      				result.innerHTML = result.innerHTML + ' ' + coverage + ' test coverage of ' + cvgInfo.key;
      			} else {
      				console.log('can\'t find test-result element to update');
      			}			
              }
         };  	
      }());
      
  6. Finally, modify run-qunit.js to load the original js file and produce a colorized version based on the coverage data we get by running the test against the version of the js file indexed for coverage. The new version should look like this:
    • /**
       * Wait until the test condition is true or a timeout occurs. Useful for waiting
       * on a server response or for a ui change (fadeIn, etc.) to occur.
       *
       * @param testFx javascript condition that evaluates to a boolean,
       * it can be passed in as a string (e.g.: "1 == 1" or "$('#bar').is(':visible')" or
       * as a callback function.
       * @param onReady what to do when testFx condition is fulfilled,
       * it can be passed in as a string (e.g.: "1 == 1" or "$('#bar').is(':visible')" or
       * as a callback function.
       * @param timeOutMillis the max amount of time to wait. If not specified, 3 sec is used.
       */
      function waitFor(testFx, onReady, timeOutMillis) {
          var maxtimeOutMillis = timeOutMillis ? timeOutMillis : 3001, //< Default Max Timout is 3s
              start = new Date().getTime(),
              condition = false,
              interval = setInterval(function() {
                  if ( (new Date().getTime() - start < maxtimeOutMillis) && !condition ) {
                      // If not time-out yet and condition not yet fulfilled
                      condition = (typeof(testFx) === "string" ? eval(testFx) : testFx()); //< defensive code
                  } else {
                      if(!condition) {
                          // If condition still not fulfilled (timeout but condition is 'false')
                          console.log("'waitFor()' timeout");
                          phantom.exit(1);
                      } else {
                          // Condition fulfilled (timeout and/or condition is 'true')
                          console.log("'waitFor()' finished in " + (new Date().getTime() - start) + "ms.");
                          typeof(onReady) === "string" ? eval(onReady) : onReady(); //< Do what it's supposed to do once the condition is fulfilled
                          clearInterval(interval); //< Stop this interval
                      }
                  }
              }, 100); //< repeat check every 250ms
      };
      
      
      if (phantom.args.length === 0 || phantom.args.length > 3) {
          console.log('Usage: run-qunit.js URL basedir');
          phantom.exit(1);
      }
      
      var fs = require('fs');
      var page = require('webpage').create();
      
      // Route "console.log()" calls from within the Page context to the main Phantom context (i.e. current "this")
      page.onConsoleMessage = function(msg) {
          console.log(msg);
      };
      
      var openPath = phantom.args[0].replace(/^.*(\\|\/)/, '');
      var basedir = phantom.args[1];
      var coverageBase = fs.read(basedir + fs.separator + 'coverageBase.htm');
      
      page.open(openPath, function(status){
          if (status !== "success") {
              console.log("Unable to access network");
              phantom.exit(1);
          } else {
              waitFor(function(){
                  return page.evaluate(function(){
                      var el = document.getElementById('qunit-testresult');
                      if (el && el.innerText.match('completed')) {
                          return true;
                      }
                      return false;
                  });
              }, function(){
      			//BEGIN MODIFIED: output colorized code coverage
      			//reach into page context and pull out coverage info. stringify to pass context boundaries.
      			var coverageInfo = JSON.parse(page.evaluate(function() { return JSON.stringify(getCoverageByLine()); }));
      			var lineCoverage = coverageInfo.lines;
      			var originalFile = basedir + fs.separator + coverageInfo.key;
      			var fileLines = readFileLines(originalFile);
      			
                  var colorized = '';
                  
      			console.log('lines=' + JSON.stringify(lineCoverage));
                  for (var idx=0; idx < lineCoverage.length; idx++) { 
                      //+1: coverage lines count from 1.
                      var cvg = lineCoverage[idx + 1];
                      var hitmiss = '';
                      if (typeof cvg === 'number') {
                          hitmiss = ' ' + (cvg>0 ? 'hit' : 'miss');
                      } else {
                          hitmiss = ' ' + 'undef';
                      }
                      var htmlLine = fileLines[idx].replace('<', '&lt;').replace('>', '&gt;');
                      colorized += '<div class="code' + hitmiss + '">' + htmlLine + '</div>\n';
                  };        
                  colorized = coverageBase.replace('COLORIZED_LINE_HTML', colorized);
                  
                  var coverageOutputFile = phantom.args[0].replace('.test.htm', '.coverage.htm');
                  fs.write(coverageOutputFile, colorized, 'w');
                  
                  console.log('Coverage for ' + coverageInfo.key + ' in ' + coverageOutputFile);			
      			//END MODIFIED
      		
                  var failedNum = page.evaluate(function(){
                      var el = document.getElementById('qunit-testresult');
                      console.log(el.innerText);
                      try {
                          return el.getElementsByClassName('failed')[0].innerHTML;
                      } catch (e) { }
                      return 10000;
                  });
                  phantom.exit((parseInt(failedNum, 10) > 0) ? 1 : 0);
              });
          }
      });
      
      //MODIFIED: add new fn
      function readFileLines(filename) {
          var stream = fs.open(filename, 'r');
          var lines = [];
          var line;
          while (!stream.atEnd()) {
              lines.push(stream.readLine());
          }
          stream.close();
          
          return lines;
      }  
      
      
  7. Run 'ant'; you should see output similar to:

  8. Open \jsunit\target\testhtm\testme.test.htm in a browser; you should see something similar to this (note coverage % appears):

  9. Open \jsunit\target\testhtm\testme.coverage.htm in a browser; you should see something similar to this (red for untested, green for tested, light green for non-executable lines):

So where does that leave us?
We have clearly displayed we can accomplish some important things:


  • Write unit tests for Javascript
  • Run unit tests for Javascript in a command line build
  • Index Javascript files for code coverage
  • Output coverage percentage to the test runner (QUnit html file)
  • Render a colorized version of the Javascript under test clearly indicating which lines are/aren't being tested
I think this is awesome! Bear in mind in a real version we would of course make numerous refinements to this rather basic implementation; what we have is a proof of concept not by any stretch of the imagination an implementation ready for a team to consume.

Friday, September 23, 2011

Tracking a running standard deviation

It is fairly common to want to track statistics on your software as it runs. Sometimes it makes sense to scrape logs and aggregate a massive set of samples. Other times you really want "live" numbers; eg running statistics. Most commonly, arithmetic mean and standard deviation. Tracking arithmetic mean alone is relatively easy but it can be very misleading. I frequently see claims of "good performance" based on mean alone which completely break down when it turns out that if we consider standard deviation a significant percentage of our population is actually experiencing something much, much worse than the mean.

So, how do we track a standard deviation from a series of samples? We could keep all samples in memory but that sounds bad; ideally we'd just like a few variables used to keep an running count of the stat. Luckily Knuth solved this problem for us, and even more luckily a statistician has written sample code in C# for us at http://www.johndcook.com/standard_deviation.html.

In Java this winds up looking remarkably similar:
//class
public class RunningStat {
	private int m_n;
	private double m_oldM;
	private double m_newM;
	private double m_oldS;
	private double m_newS;
	
	public RunningStat() {
		m_n = 0;
	}
	
	public void clear() { m_n = 0; }
	
	public void addSample(double sample) {
        m_n++;

        // See Knuth TAOCP vol 2, 3rd edition, page 232
        if (m_n == 1)
        {
            m_oldM = m_newM = sample;
            m_oldS = 0.0;
        }
        else
        {
            m_newM = m_oldM + (sample - m_oldM)/m_n;
            m_newS = m_oldS + (sample - m_oldM)*(sample - m_newM);

            // set up for next iteration
            m_oldM = m_newM; 
            m_oldS = m_newS;
        }
	}
	
	public int getNumSamples() { return m_n; }
	public double getMean() { return (m_n > 0) ? m_newM : 0.0; }
	public double getVariance() { return ( (m_n > 1) ? m_newS/(m_n - 1) : 0.0 ); }
	public double getStdDev() { return Math.sqrt(getVariance()); }	
}

//usage
RunningStat timeStat = new RunningStat();		
...
long time = System.nanoTime();			
//do the thing we are tracking...						
time = (System.nanoTime() - time) / (1000*1000); //ms
			
timeStat.addSample(time); 

This does have some limitations for a typical Java web application, in particular it isn't threadsafe. We can simply slap synchronized on it, using a private lock to avoid unwanted publicity of our synchronization primatives:
public class RunningStat {
	private int m_n;
	private double m_oldM;
	private double m_newM;
	private double m_oldS;
	private double m_newS;
	
	private Object m_lock = new Object();
	
	public RunningStat() {
		m_n = 0;
	}
	
	public void clear() {
		synchronized(m_lock) {
			m_n = 0;
		}
	}
	
	public void addSample(double sample) {
		synchronized(m_lock) {
	        m_n++;
	
	        // See Knuth TAOCP vol 2, 3rd edition, page 232
	        if (m_n == 1)
	        {
	            m_oldM = m_newM = sample;
	            m_oldS = 0.0;
	        }
	        else
	        {
	            m_newM = m_oldM + (sample - m_oldM)/m_n;
	            m_newS = m_oldS + (sample - m_oldM)*(sample - m_newM);
	
	            // set up for next iteration
	            m_oldM = m_newM; 
	            m_oldS = m_newS;
	        }
		}
	}
	
	public int getNumSamples() { synchronized(m_lock) { return m_n; } }
	public double getMean() { synchronized(m_lock) { return (m_n > 0) ? m_newM : 0.0; } }
	public double getVariance() { synchronized(m_lock) { return ( (m_n > 1) ? m_newS/(m_n - 1) : 0.0 ); } }
	public double getStdDev() { synchronized(m_lock) { return Math.sqrt(getVariance()); } }	
}
This leaves open the question of how we actually distribute references to the stat. For example, it could be that anytime a given handler executes we want to update a stat, and we want to grab that stat to use in a periodic stat logger, and we want to grab it for our view statistics page. All these locations need to get a reference to the same instance of RunningStat. If we use something like Spring we can trivially inject the same reference. If not we may wish to manage a stats registry ourselves. A simple implementation using ConcurrentHashMap might look something like this:
import java.util.Collection;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;

public class StatisticsRegistry {
	private static final StatisticsRegistry instance = new StatisticsRegistry();				
	
	private final ConcurrentMap statById = new ConcurrentHashMap(); 
	private final Object writeLock = new Object();
	
	public static StatisticsRegistry getInstance() { return instance; }
	
	public RunningStat getNamedStatistic(String name) {
		/**
		 * Usually the stat will exist; avoid extra sync ops on write lock when possible
		 */
		RunningStat stat = statById.get(name);
		
		if (null == stat) {
			synchronized(writeLock) {
				//someone else could have just inserted it; if so putIfAbsent will return the value they put in
				RunningStat existing = statById.putIfAbsent(name, new RunningStat());
				if (existing != null) {
					stat = existing;
				}
			}
		}
		
		return stat;
	}
		
	public Collection getAllStatistics() {
		return statById.values();
	}
}
Clients can request a stat by name and clients like a stat logger or viewer can grab either all stats or specific ones as needed. As the same instance is always returned it is safe for clients to write code to retain RunningStat references.

Monday, July 11, 2011

Javascript Unit Tests with QUnit, Ant, and PhantomJS, Take 1

Recently I have been finding bugs in Javascript slip by a lot more easily than bugs in Scala, Java, or various other languages for which we write unit tests. jQuery seems to use QUnit (http://docs.jquery.com/QUnit) but QUnit appears to expect a web page to be setup to host it. This is better than nothing but really I want to run my js unit tests in an automated build, in my case using Ant.

The problem seemed remarkably likely to be solved already so I took to the Google and discovered some blogposts (http://twoguysarguing.wordpress.com/2010/11/26/qunit-cli-running-qunit-with-rhino/, http://twoguysarguing.wordpress.com/2010/11/06/qunit-and-the-command-line-one-step-closer/) where the author was attempting to achieve a command line unit test runner using Rhino and QUnit. Apparently John Resig tweeted some time ago (http://twitter.com/#!/jeresig/status/4477641447) to indicate QUnit should be operable in this manner so things seemed promising.

The twoguysarguing (great title) blog posts I found seemed to require modifications to QUnit source, plus Rhino not being a full browser apparently caused some issues as well. I really didn't want a custom version of QUnit, but the general approach seemed promising. In the comments for the second post someone suggested use of PhantomJS (http://twoguysarguing.wordpress.com/2010/11/06/qunit-and-the-command-line-one-step-closer/#comment-599), a headless WebKit browser. I decided to give this a try as it sounded remarkably reasonable.

My first step was to verify PhantomJS worked for me at all. It ran my first test without any issue:
//try1.js
console.log('Hello, World');
phantom.exit();
Executed similar to phantomjs try1.js this prints Hello, World just as one might hope.

The next question seemed to be whether or not PhantomJS could actually load QUnit using injectJs (ref http://code.google.com/p/phantomjs/wiki/Interface). I git cloned QUnit and attempted to invoke inject.Js on it:
//try2.js
if (window.QUnit == undefined)
 console.log('no QUnit yet!');
else
 console.log('somehow we already haz QUnit !!');
phantom.injectJs('D:\\Code\\3.7-scalaide\\JavaScriptUnitTests\\QUnit\\qunit.js');
if (window.QUnit != undefined)
 console.log('goodnes; injectJs seems to have worked');

phantom.exit();
This prints:
no QUnit yet!
goodnes; injectJs seems to have worked
So far so good!!

So, that means we should be able to setup and run a test, right? Something like this:
test("This test should fail", function() {
  console.log('the test is running!');
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});

test("This test should pass", function() {
  console.log('the test is running!');
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});
Well ... sadly this part didn't "just work". QUnit tries to execute the test queue on timers and despite PhantomJS supporting timers they just never seemed to execute. Furthermore, QUnit default feedback is via DOM modifications that are rather unhelpful to the PhantomJS runner. My first draft was to modify QUnit by adding a function that directly executed the test queue, inline, without using timers. This worked, but it required modifying QUnit source, which I specifically wish to avoid.

Luckily something similar to the changes to add a function to run QUnits tests directly works just fine outside QUnit as well. The key is that we will:
  1. Track test pass/fail count via the QUnit.testDone callback (http://docs.jquery.com/Qunit#Integration_into_Browser_Automation_Tools)
    1. We need our own pass/fail counters as QUnit tells us how many assertions passed/failed rather than how many tests passed/failed.
  2. Track whether or not the test run is done overall via the QUnit.done callback (http://docs.jquery.com/Qunit#Integration_into_Browser_Automation_Tools)
  3. Directly execute the QUnit test queue from our own code
    1. hack but the point here is to see if we can make this work at all
  4. Split tests into their own file
    1. This facilitates using an Ant task to run a bunch of different test sets; eg using apply to pickup on all .js test files by naming convention or location convention
  5. Return the count of failures as our PhantomJS exit code
    1. This facilitates setting an Ant task to failonerror to detect unit test failures
So, without further ado, error handling, namespaces/packages, or any other cleanup here is a version that works in a manner very near to the desired final result:
try4.js
function importJs(scriptName) {
 console.log('Importing ' + scriptName);
 phantom.injectJs(scriptName);
}

console.log('starting...');

//Arg1 should be QUnit
importJs(phantom.args[0]);

//Arg2 should be user tests
var usrTestScript = phantom.args[1];
importJs(usrTestScript);

//Run QUnit
var testsPassed = 0;
var testsFailed = 0;

//extend copied from QUnit.js
function extend(a, b) {
 for ( var prop in b ) {
  if ( b[prop] === undefined ) {
   delete a[prop];
  } else {
   a[prop] = b[prop];
  }
 }

 return a;
}

QUnit.begin({});

// Initialize the config, saving the execution queue
var oldconfig = extend({}, QUnit.config);
QUnit.init();
extend(QUnit.config, oldconfig);

QUnit.testDone = function(t) {
 if (0 === t.failed) 
  testsPassed++;
 else
  testsFailed++;
  
 console.log(t.name + ' completed: ' + (0 === t.failed ? 'pass' : 'FAIL'))
}

var running = true;
QUnit.done = function(i) {
 console.log(testsPassed + ' of ' + (testsPassed + testsFailed) + ' tests successful');
 console.log('TEST RUN COMPLETED (' + usrTestScript + '): ' + (0 === testsFailed ? 'SUCCESS' : 'FAIL')); 
 running = false;
}

//Instead of QUnit.start(); just directly exec; the timer stuff seems to invariably screw us up and we don't need it
QUnit.config.semaphore = 0;
while( QUnit.config.queue.length )
 QUnit.config.queue.shift()();

//wait for completion
var ct = 0;
while ( running ) {
 if (ct++ % 1000000 == 0) {
  console.log('queue is at ' + QUnit.config.queue.length);
 }
 if (!QUnit.config.queue.length) {
  QUnit.done();
 }
}

//exit code is # of failed tests; this facilitates Ant failonerror. Alternately, 1 if testsFailed > 0.
phantom.exit(testsFailed);

try4-tests.js
test("This test should fail", function() {
  ok( true, "this test is fine" );
  var value = "hello";
  equals( "hello", value, "We expect value to be hello" );
  equals( "duck", value, "We expect value to be duck" );
});

test("This test should pass", function() {
  equals( "hello", "hello", "We expect value to be hello" );
});

This runs as follows:
>phantomjs.exe try4.js qunit.js try4-tests.js
starting...
Importing qunit\qunit.js
Importing javascript\try4-tests.js
This test should fail completed: FAIL
This test should pass completed: pass
queue is at 0
1 of 2 tests successful
TEST RUN COMPLETED (try4-tests.js): FAIL

Note that we have not modified qunit.js, and we have split our tests into their own file. This allows us to easily set the whole thing up to run from Ant:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<project default="js-tests4"> 
 <target name="js-tests4">
  <property name="phantomjs.exe.file" value="phantomjs.exe" />
  <property name="qunit.js.file" location="path/to/qunit.js" />

  <apply executable="${phantomjs.exe.file}" failonerror="true">
   <arg value="path/to/try4.js"/>
   <arg value="${qunit.js.file}" />
   <srcfile/>
   
   <fileset dir="path/to/tests">
    <include name="try4-tests.js" />
   </fileset>
  </apply>
 </target>
</project>

It even works run this way:
js-tests4:
    [apply] starting...
    [apply] Importing path\to\qunit.js
    [apply] Importing path\to\try4-tests.js
    [apply] This test should fail completed: FAIL
    [apply] This test should pass completed: pass
    [apply] queue is at 0
    [apply] 1 of 2 tests successful
    [apply] TEST RUN COMPLETED (try4-tests.js): FAIL

BUILD FAILED

Note that Ant has detected that a js unit test failed and failed the build, just as we intended.

This leaves us with a proof of concept implementation that seems to prove that using PhantomJS to run QUnit based Javascript tests to run from a command line build is fundamentally possible. Doubtless if/when we try to use it "for real" additional problems will emerge ;)