Thursday, December 2, 2010

Dynamic Spring Beans using BeanFactoryPostProcessor

Recently we came on an interesting problem with our Spring XML files: in some situations we always have to create a bean in an entirely predictable manner. For example, to accomplish X you add a file Y to directory Z, then setup a bean thusly. A new chap coming onto the project pointed out that this felt like boilerplate code and as boilerplate code is not fun I decided to try to eliminate it.

A little investigation turned up a variety of alternatives, ranging from rewriting the XML files on app start prior to Spring initialization to writing some code that did a little introspection and created the beans on the fly. These beans aren't entirely dynamic, they are implied by other aspects of the project (existence of certain files in our case). These "implied beans" need to be referenced by manually configured beans so they have to be created during Spring bean loading process somehow. That is, we must be able to do this:
<bean id="myManualBean" class="...">
  <property name="myProperty" ref="anImpliedBean"/>
</bean>
<!-- 'anImpliedBean' will never be explicitly declared!! -->
This was outside what I'd done with Spring before but I figured a custom BeanFactory might do it? Well ... probably, but it looked like it would be fairly painful! However, Spring also has the perfect interface for this job: BeanFactoryPostProcessor. When Spring encounters a BeanFactoryPostProcessor it will call into it and allow it to modify the beans in postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory). In practice this works out to something like this:

In the Spring beans xml file:
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:p="http://www.springframework.org/schema/p"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
    
    <!-- Our BeanFactoryPostProcessor; it will register 'anImpliedBean' -->
    <bean class="our.package.for.ImplicitBeanCreator">
        <!-- can set pptys of the BeanFactoryPostProcessor if desired -->
    </bean>

    <bean id="myManualBean" class="...">
        <!-- note the ref to a bean not declared manually! It's like magic! -->
        <property name="myPpty" ref="anImpliedBean" />
    </bean>

    <!-- many lines of implied beans deleted == less xml hell! -->
</beans>
In Java:
public class ImplicitBeanCreator implements BeanFactoryPostProcessor {

 @Override
 public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory)
   throws BeansException {
  //examine our environment (in our case the filesystem) to work out what beans are 'implied'
  //create them, such as by creating an instance and calling beanFactory.registerSingleton(name, object)
 }
}
Obviously this isn't something you need everyday but when it lets you avoid hand-coding swaths of completely boilerplate beans it's a wonderful thing!

Examples are all based on Spring 2.5.6.SEC01.

Wednesday, December 1, 2010

Bad timing using System.currentTimeMillis() instead of System.nanoTime()

Sometimes people report performance concerns to me complete with actual times. While this makes me happy (as compared to the also popular report that "its slow"), I always have to ask "how did you time it". In Java the answer is very often something similar to:
long timeMillis = System.currentTimeMillis();
//do something to be timed
timeMillis = System.currentTimeMillis() - timeMillis; 
System.out.println("millis: " + timeMillis);
Sometimes it's in a loop to trap averages, standard deviation, has pre-runs to try to eliminate timing JIT steps, and so on. However, the API choice means this is not going to work terribly well. The issue is that currentTimeMillis() "Returns the current time in milliseconds. Note that while the unit of time of the return value is a millisecond, the granularity of the value depends on the underlying operating system and may be larger. For example, many operating systems measure time in units of tens of milliseconds." (see docs). This issue is theoretically common knowledge, yet it keeps on coming up, particularly from coders who did most of their learning on Java 1.4.2.

In Java 1.5 an API allowing access to the high resolution timer was finally added, System.nanoTime() (see docs).

Knowing that times around tens of milliseconds are the probably a good point to see problems we can whip up a quick and dirty little program to demonstrate the problem with using System.currentTimeMillis():
public class TimeTest {
 public static void main(String...argv) throws Exception {
  
  int spins = 99;
  int sleepMillis = 12;
  
  for (int i=0; i<spins; i++) {
   long timeMillis = System.currentTimeMillis();
   Thread.sleep(sleepMillis);
   timeMillis = System.currentTimeMillis() - timeMillis; 
   System.out.println("millis: " + timeMillis);
  }
  
  for (int i=0; i<spins; i++) {
   long timeNanos = System.nanoTime();
   Thread.sleep(sleepMillis);
   timeNanos = System.nanoTime() - timeNanos;
   long timeMillis = timeNanos / (1000 * 1000);
   System.out.println("millis from nanos: " + timeMillis + ", nanos: " + timeNanos);
  }  
 }
}

The results on a Windows 2k3 Java 1.6.0_21-b07 system tend to 15 or 16 with fairly regular 0's; nanoTime is bang on:


millis: 16
millis: 0
millis: 15
millis: 16
millis from nanos: 12, nanos: 12557445
millis from nanos: 12, nanos: 12593718
millis from nanos: 12, nanos: 12423730
millis from nanos: 12, nanos: 12576676



On a Solaris 5.10 VM the results are somewhat better:


millis: 12
millis: 12
millis: 12
millis: 12
millis from nanos: 12, nanos: 12051729
millis from nanos: 12, nanos: 12113813
millis from nanos: 12, nanos: 12089188
millis from nanos: 12, nanos: 12013906


Mac OS X also appears to have a more accurate currentTimeMillis(), with Java 1.6.0_22:


millis: 12
millis: 12
millis: 12
millis: 12
millis from nanos: 12, nanos: 12111000
millis from nanos: 12, nanos: 12087000
millis from nanos: 12, nanos: 12096000
millis from nanos: 12, nanos: 12083000


Moral of story: if you read no other manual, at least read the one for your timing API! Consider reading Robust Java Benchmarking parts 1 and 2 as well.

Monday, November 29, 2010

Interrogating the Java Model in an Eclipse JDT Plugin to find a method

Previously I noted how to setup an Eclipse Plugin that can access JDT, noting some nomenclature and setup steps that are non-intuitive to an Eclipse noob. In trying to wire up an in-Eclipse version of the JSP checker (see post), one of the first problems encountered is that finding methods using java.lang.reflect doesn't work; we need to use the JDT model to ask questions about the users Java code instead.

As with most noob problems this proves fairly simple once done, but initially the Eclipse APIs provide a nice baffling wall. Our code is still running in a org.eclipse.core.resources.IncrementalProjectBuilder derived class and thus has a reference to the IResource for the file we are interested in (in my specific example, for a JSP file that in turn has a qualified class name in it). To implement the JSP check we'll need to then search for a user class by qualified type name, then check if the class has a specific method.

Our first step is to check if the type exists. Initially we only want to look in the current project, so we'll need a reference to the IJavaElement for the project. It turns out we can get a reference to this from JavaCore by passing in the IProject, which we can get from our IResource for the file. This means checking if a type exists can be done similar to the following (given that we have IResource resource as the resource our builder is working on and String qualifiedTypeName containing the type name of interest):
...
IJavaProject javaProject = JavaCore.create(resource.getProject());
IType type;
//normally findType returns null if there is no such type but theoretically it can throw JavaModelException also
try {
  type = javaProject.findType(qualifiedTypeName);
} catch (JavaModelException jme) {
  throw new RuntimeException("Error trying to find type \"" + typeName + "\"");
}
boolean typeExists = null != type;
Wonderous! Now to see if we can look for a method. The docs seem to suggest we can use the SearchEngine to do look for stuff in Java code, passing a suitable SearchPattern in. We also have to provide a SearchRequestor implementation to trap our results, though in our case all we want to do is count them. For the time being we'll just look for 0 or >0 though eventually we'll have to determine whats going on if there are 2+ matches. We'll wrap all this into a couple of classes to help us simplify coding against the Java model as most of our code really only cares if something is a valid type and whether a JavaBeans-style property exists, not how we figure that out. Here is draft 1 of the Eclipse class to aid us in figuring out if typenames and properties are valid:
package com.example.validator;

import org.eclipse.core.resources.IProject;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.jdt.core.IJavaElement;
import org.eclipse.jdt.core.IJavaProject;
import org.eclipse.jdt.core.IType;
import org.eclipse.jdt.core.JavaCore;
import org.eclipse.jdt.core.JavaModelException;
import org.eclipse.jdt.core.search.IJavaSearchConstants;
import org.eclipse.jdt.core.search.IJavaSearchScope;
import org.eclipse.jdt.core.search.SearchEngine;
import org.eclipse.jdt.core.search.SearchParticipant;
import org.eclipse.jdt.core.search.SearchPattern;

import com.active.commons.StringUtils;

public class JavaClassInfo {
 private IJavaProject javaProject;
 private IType type;
 
 public JavaClassInfo(IProject project, String typeName) {
  javaProject = JavaCore.create(project);  
  try {
   type = javaProject.findType(typeName);
  } catch (JavaModelException jme) {
   throw new RuntimeException("Error trying to find type \"" + typeName + "\"");
  }    
 }

 public boolean hasProperty(String propertyName) {
  return hasMethod("get" + StringUtils.capitalize(propertyName))
   || hasMethod("is" + StringUtils.capitalize(propertyName));
 }
 
 private boolean hasMethod(String methodName) {
  //Guard: must be a valid type to have methods
  if (!isValidTypename()) {
   return false;
  }
  
  SearchPattern pattern = SearchPattern.createPattern(
    methodName, 
    IJavaSearchConstants.METHOD, 
    IJavaSearchConstants.DECLARATIONS, 
    SearchPattern.R_EXACT_MATCH);
  
  IJavaSearchScope scope = SearchEngine.createJavaSearchScope(new IJavaElement[] { type });
  
  CountingSearchRequestor matchCounter = new CountingSearchRequestor();
  
  SearchEngine search = new SearchEngine();
  try {
   search.search(pattern, 
    new SearchParticipant[] { SearchEngine.getDefaultSearchParticipant() }, 
    scope, 
    matchCounter, 
    null);
  } catch (CoreException ce) {
   System.out.println("Couldn't find type: " + ce.getMessage());
   ce.printStackTrace();
  } 
  
  if (matchCounter.getNumMatch() > 1) {
   System.out.println("Bit weird; " + matchCounter.getNumMatch() + " matches for " + methodName);
  }
  
  return matchCounter.getNumMatch() > 0;
 }

 public boolean isValidTypename() {
  return type != null;  
 }
}
Note that the following implementation of SearchRequestor is used to count results:
package com.example.validator;

import org.eclipse.core.runtime.CoreException;
import org.eclipse.jdt.core.search.SearchMatch;
import org.eclipse.jdt.core.search.SearchRequestor;

public class CountingSearchRequestor extends SearchRequestor {
 private int numMatch;
 
 @Override
 public void acceptSearchMatch(SearchMatch match) throws CoreException {
  numMatch++;
 }

 public int getNumMatch() {
  return numMatch;
 }
}

This will allow us to write code similar to the following to check on type/method existance:
JavaClassInfo classInfo = new JavaClassInfo(resource.getProject(), qualifiedTypeName);
...
if (!classInfo.hasProperty(propertyName)) {
   //add a Marker discussing how dumb you are to have used a property that doesn't exist!
}
One step closer to having our in-Eclipse JSP checker.

Monday, November 22, 2010

How to setup an Eclipse Plugin that can access JDT

In aid of remembering for next time, a few notes on setting up an Eclipse project that can play with the Java model of a project follow.

The first step is figuring out which version of Eclipse to use for plugin development. It was not obvious to me that "Eclipse for RCP and RAP Developers" was what I wanted on the Eclipse download page. After some Googling I puzzled it out but a name that clearly indicated to a noob it was a bundle suitable for Eclipse extension or plugin development would have helped!

Once you have Eclipse for RCP/RAP creating a plugin project is gloriously simple. Just click through some wizard pages:


This gives you a superb little project, all setup for debug and everything! However, said project doesn't have the JDT libraries on classpath so when you try to do anything interesting using the JDT it doesn't compile. You can clearly see that the "Plug-in Dependencies" don't include any of the JDT stuff ... but how to add it?

To add the JDT bits so we can do cruel and unusual things to Java projects double-click on META-INF/MANIFEST.MF in your plugin project, choose the MANIFEST.MF tab at the bottom of the editor, and add the org.eclipse.jdt, org.eclipse.jdt.core to the Require-Bundle block. On save this will add jars to your "Plug-in Dependencies", as shown below:

All easy enough once you know; however as an Eclipse noob this was rather non-obvious. Perhaps next we'll find time to make the JSP property access checker publish to the Eclipse problems view "live".

For using the JDT the documentation is pretty good, as is the JDT_fundamentals.ppt someone put together for eclipsecon 2008 (thus perhaps dated info but still helpful).

Scanning For Invalid Property Access in JSP EL Expressions V1

In applications that use JSP we often find ourselves using an object instance and accessing properties in EL expressions. For example:
<input value="${address.line1}" ... />
The problem is that if address doesn't have a getLine1() method this fails at runtime. If someone decides to rename a method our JSP may break without us knowing until someone runs the app and sees an error, particularly if the renaming was done in a library project. More commonly, someone renames something and doesn't quite find all the places it was referenced in our gloriously unchecked JSPs. Accordingly it would be really nice if we could find a way to be automatically notified of property file misuse.

A fancier version will ultimately be built but our for our first proof-of-concept version we want to crank out a quick and dirty JUnit test that will fail if we use a non-existent property. Our first problem to overcome is that even if we scan for EL expressions that look like object field accesses we can't tell what type is being used. Our super-simple solution will be to simply have this explicitly expressed in our JSPs: we'll use a server-side comment that provides this information to our scanner.
<%-- TYPECHECK(myVar,com.example.package.MyType) --%>
The first draft of the scanner will simply search our source tree for TYPECHECK blocks using a regular expression. To find the TYPECHECK blocks we can use an expression similar to:
private static final Pattern checkDecl = Pattern.compile("(?m)^\\s*<%--\\s+TYPECHECK\\s*\\(\\s*(\\w+)\\s*,([\\w.]+)\\s*\\)\\s*--%>\\s*$");
For the record, Java-escaped regular expressions make babies weep and programmers wish for a verbatim-string mechanism, ala C#. Note that we establish capturing groups for varname (group(1)) and type (group(2)) to help pull values out of our Matcher.

In the source tree we want to scan we happen to know that we can find our source dir offset from the / resource. We can then scan *.jsp for TYPECHECK declarations and scan the files using TYPECHECK for accesses to varname that reference non-existent properties. The following listing does just that:

package com.example.validate.jsp;

import static org.junit.Assert.assertEquals;

import java.io.File;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.StringUtils;
import org.junit.BeforeClass;
import org.junit.Test;

import ...internal package....ReflectionHelper;

public class JspTypeCheckTestCase {
 private static final Pattern checkDecl = Pattern.compile("(?m)^\\s*<%--\\s+TYPECHECK\\s*\\(\\s*(\\w+)\\s*,([\\w.]+)\\s*\\)\\s*--%>\\s*$");
 private static final Pattern elExpr = Pattern.compile("\\$\\{([^}])+\\}");
 private static final String useExprTemplate = "%1$s\\.(\\w+)";
 private static File srcDir;
 
 @BeforeClass
 public static void beforeClass() throws Exception {
  srcDir = new File(new File(JspTypeCheckTestCase.class.getResource("/").toURI()), "../../src/main/webapp").getCanonicalFile();
 }
 
 /**
  * Look for JSP files declaring <%-- TYPECHECK(varname,qualifiedJavaType) --%>; for all found validate that for all 
  * expressions of the form varname.something something exists on qualifiedJavaType as a JavaBean's style accessor 
  * (eg a public T isSomething() or public T getSomething() method exists) 
  * @throws Exception
  */
 @Test
 public void doTypeCheck() throws Exception {
  int numFailures = 0;
  //examine src/main/webapp/**/*.jsp files
  @SuppressWarnings("unchecked")
  Collection<File> filesToExamine = (Collection<File>) FileUtils.listFiles(srcDir, new String[] { "jsp" }, true);
  for (File fileToExamine : filesToExamine) {
   String content = FileUtils.readFileToString(fileToExamine);
   Matcher matcher = checkDecl.matcher(content);
   while (matcher.find()) {
    String fullTypeCheck = matcher.group();    
    String varName = matcher.group(1);
    String typeName = matcher.group(2);
    
    Class<?> type = null;
    try {
     type = Class.forName(typeName);
    } catch (Exception e) {
     System.out.println("Bad typecheck in " + fileToExamine + ": " + fullTypeCheck
       + " (" + e.getClass().getName() + " - " + e.getMessage() + ")");
     numFailures++;
    }
    
    List<String> expressions = new ArrayList<String>();
    Matcher exprMatcher = elExpr.matcher(content);
    while (exprMatcher.find()) {
     expressions.add(exprMatcher.group(0));
    }
    
    if (null != type) {
     for (String expression : expressions) {
      numFailures += checkUseOfType(fileToExamine, type, varName, expression);
     }
    }
   }
  }
  assertEquals(numFailures + " typecheck failures in JSP files", 0, numFailures);  
 }

 private int checkUseOfType(File file, Class<?> type, String varName, String expression) {
  int numFailures = 0;
  
  Pattern usagePattern = Pattern.compile(String.format(useExprTemplate, varName));
  Matcher m = usagePattern.matcher(expression);
  while (m.find()) {
   String pptyName = m.group(1);
   String methodName = StringUtils.capitalize(pptyName);
   if (null == ReflectionHelper.findMethod(type, "^(is|get)" + methodName + "$", new Class<?>[] {}, null)) {
    numFailures ++;
    System.out.println(file.getName() + ": " + type.getName() + " has no accessor for " + methodName
     + "; bad expression: " + expression);
   }
  }
  
  return numFailures;
 }
}


A couple of notes about the source:

  • It's very much a quick/dirty/non-optimized test to see if this idea is viable and more importantly if we find having such a scan helpful day to day. It sounded good on paper but lots of things meet that criteria ;)
  • ReflectionHelper.findMethod is an internal API that looks for a method with a given name, arguments, and return type. In our case we are simply scanning for an instance method named getBlah or isBlah if we find use of varname.blah in our jsp
  • If the test fails it will print the offending expressions to stdout similar to: myfile.jsp: com.example.package.MyType has no accessor for A; bad expression ${myvar.A or myvar.B}
  • The test will fail if there are > 0 bad accesses with an assertion error similar to java.lang.AssertionError: 2 typecheck failures in JSP files expected:<0> but was:<2>
  • The test is typically run on Java 1.6, with commons-lang 2.4 and commons-io 1.4 on classpath.

    If the scan proves helpful we speculate that it may evolve to be run directly in Eclipse using a custom builder. This would give us "live" reports of bad accesses in JSPs in the problems view. Heck, maybe we could even run the EL parser on our expressions and see if our EL is valid too...?

    Saturday, November 13, 2010

    Erlang: so awesome it makes my brain bleed

    I have read several times Erlang is a pretty cool functional language. I even bought the book. The book has a lot of cool examples and I enjoyed skimming the first chapters, being blown away by some of the way-easier-than-sockets-in-C examples and so on. I felt like I was getting all this list stuff even though I'm neither a LISP programmer nor really a pure functional programmer. Unfortunately the moment I decided to try puzzling out exactly how some of the early samples worked it turned out I didn't understand as well as I thought. Luckily I think I figured it out, as detailed below.

    First step, setting up to compile & run code. I downloaded Erlang R14B (from here), ran it, entered some expressions into the interactive shell, and all seemed well until I put some code into a .erl file in a directory, booted werl.exe (I'm on Windows for this) from that directory, and asked it to compile:

    53> c(myTestModule).
    ./myTestModule.erl:none: no such file or directory
    error
    

    Awesome. It turns out that looking in the current directory isn't on by default. A little poking through the part of the book devoted to basic setup revealed that I needed to create a .erlang file in my user home directory that added current directory to the list of places to look. On Win7 this means C:\Users\MyUserName\.erlang with content as follows:

    io:format("Running Erlang~n").
    code:add_patha(".").
    

    With that set I get a spiffy printout ('Running Erlang') when I boot the shell and now it looks for files in my current directory.

    I also found using http://erlide.sourceforge.net/ for syntax highlighting and live reporting of broken code as I typed it was helpful. Unfortunately running and debugging from Eclipse operated without error and without any apparent output so I couldn't actually use Erlide to run programs and view output :(

    Anyway, so far so good, next to try a simple program. List comprehensions were something very new given my background but the definition the book gave seemed clear enough:

    "The most general form of a list comprehension is an expression of the following form:
    [X || Qualifier1, Qualifier2, ...]
    X is an arbitrary expression, and each qualifier is either a generator or a filter.
    • Generators are written as Pattern <- ListExpr where ListExpr must be an expression that evaluates to a list of terms.
    • Filters are either predicates (functions that return true or false) or boolean expressions."
    (Programming Erlang, "3.6 List Comprehensions")

    I can totally handle that! Not quite sure how to actually use a filter but the generator part seems dead simple. So on to the first little program, a for-each function:
    -export([each/1]).
    %% the version of each/1 to run for an empty list
    each([]) ->
     io:format("each([])~n");
    %% the version of each/1 to run for a list made up of 
    %% head H (first item in list) and tail T (rest of list). 
    %% Print the arguments then run each again on the tail. 
    %% If we did something with H it'd be largely the same as
    %%    for (Type h : someList { /* do stuff with h here */ }
    each([H|T]) ->
     io:format("each(H=~w, T=~w])~n", [H, T]),
     each(T).
    
    In the shell:
    54> c(myModule), myModule:each([a, b, c]). 
    each(H=a, T=[b,c]])
    each(H=b, T=[c]])
    each(H=c, T=[]])
    each([])
    ok
    
    The last value (ok) is the return from the function. This is simply the L-value of the last statement. In this case we're getting ok as the io:format return value.

    Our each function works as expected; this is all seeming simple enough! And then we hit the books function to generate all permutations of the items in list. And somehow it didn't just click for me.
    -export([perms/1]).
    perms([]) -> [[]];
    perms(L) -> [[H|T] || H <- L, T <- perms(L--[H])].
    
    In the shell:
    57> c(myModule), myModule:perms([a, b, c]).
    [[a,b,c],[a,c,b],[b,a,c],[b,c,a],[c,a,b],[c,b,a]]
    
    It works. And I have to admit to almost hoping it didn't! But ... how did it work? It initially looked to me like it was saying take each item from the list (H<-L) and build a list of that plus the permutations of the list with the head taken off (T <- perms(L--[H])). I'm picturing equivalent pseudo-Java'ish code similar to:
    for each item in input list
      add to output that item plus the permutations of the list from the next item onwards
    
    This wouldn't work at all! Item b would only pair it with c; we'd never generate combinations like [b, a, c]. Except clearly we do. OK, so what the heck happens when you have multiple generators in a list comprehension? It seems like it must be doing "for each item in each generator run the expression X". Or to put it another way, if we had two generators that each pulled items from a list our expression would run for every possible pair of list items. The second generator seems to be able to use the output of the first so basically we're talking about nested loops.

    That seems ... kind of reasonable actually. Perhaps there is a way we could confirm that is what happens? First we'll need a way to print what is happening in the expression part of a list comprehension and still accumulate the result:
    %% print B using format A and return C. 
    %% This means we can display arbitrary output 
    %% and return something in the X part of a list comprehension.
    printABReturnC(A, B, C) ->
     io:format(A, B),
     C.
    
    We should be able to use this to simply print each step in running through a list comprehension of a simple list, and luckily it turns out it works as expected:
    -module(myModule).
    -export([lc2/1]).
    
    lc2([]) ->
     io:format("lc2([])~n");
    lc2(L) ->
     [printABReturnC("lc2(L=~w) H=~w~n", [L, H], H) || H <- L].
    
    printABReturnC(A, B, C) ->
     io:format(A, B),
     C.
    
    In the shell:
    59> c(myModule), myModule:lc2([a, b, c]).
    lc2(L=[a,b,c]) H=a
    lc2(L=[a,b,c]) H=b
    lc2(L=[a,b,c]) H=c
    [a,b,c]
    
    Perfect!

    Next we should be able to run a list comprehension that has two generators taking items from the same list. We'd expect that for [a,b,c] we'd get each possible pairing, [[a,a],[a,b],[a,c],[b,a],[b,b],[b,c],[c,a],[c,b],[c,c]].
    -module(myModule).
    -export([lc3/1]).
    
    lc3([]) ->
     io:format("lc3([])~n");
    lc3(L) ->
     [printABReturnC("lc3 H=~w, K=~w~n", [H, K], [H,K]) || H <- L, K <- L].
    
    printABReturnC(A, B, C) ->
     io:format(A, B),
     C.
    
    In the shell:
    60> c(myModule), myModule:lc3([a, b, c]).
    lc3 H=a, K=a
    lc3 H=a, K=b
    lc3 H=a, K=c
    lc3 H=b, K=a
    lc3 H=b, K=b
    lc3 H=b, K=c
    lc3 H=c, K=a
    lc3 H=c, K=b
    lc3 H=c, K=c
    [[a,a],[a,b],[a,c],[b,a],[b,b],[b,c],[c,a],[c,b],[c,c]]
    
    This is all slowly starting to make sense! So if we wanted to produce all unique pairs from a list we could do something like this:
    -module(myModule).
    
    -export([lc4/1]).
    
    lc4([]) ->
     io:format("lc3([])~n");
    lc4(L) ->
     [printABReturnC("lc3 H=~w, K=~w~n", [H, K], [H,K]) || H <- L, K <- L--[H]].
    
    printABReturnC(A, B, C) ->
     io:format(A, B),
     C.
    
    In the shell:
    61> c(myModule), myModule:lc4([a, b, c]).
    lc3 H=a, K=b
    lc3 H=a, K=c
    lc3 H=b, K=a
    lc3 H=b, K=c
    lc3 H=c, K=a
    lc3 H=c, K=b
    [[a,b],[a,c],[b,a],[b,c],[c,a],[c,b]]
    
    Hooray!

    If we go back to our original permutations example the ugly bit is perms(L) -> [[H|T] || H <- L, T <- perms(L--[H])]. We can break this down to see we have:

    • One generator that simply takes each item from the list: H <- L
    • One generator that requests all permutations of the original input list except for the element HT <- perms(L--[H])
      • That is, pair item A with all permutations of [B, C]. Pair item B with all permutations of [A, C].
    • Our first generator produces a series of simple values. Our second produces a series of lists. Our expression [H|T] takes each item from each generator and pairs them up as a list
      • That is, for each item in the input list, make a list where that item is the head and the tail is all permutations of the list without that item

    That seems reasonable. We should be able to use our spriffy print function to get output clearly showing it in action though:
    -module(myModule).
    
    -export([lc5/0]).
    -export([lc5/1]).
    
    lc5() ->
     lc5([]).
    lc5([]) ->
     io:format("lc5([])~n"),
     [[]];
    lc5(L) ->
     io:format("lc5(~w)~n", [L]),
     [printABReturnC("H=~w, T=~w~n", [H,T], [H|T]) || H <- L, T <- lc5(L--[H])].
    
    printABReturnC(A, B, C) ->
     io:format(A, B),
     C.
    
    And finally we can clearly see in the shell that what we thought was happening is indeed happening:
    62> c(myModule), myModule:lc5([a, b, c]).
    lc5([a,b,c])
    lc5([b,c])
    lc5([c])
    lc5([])
    H=c, T=[]
    H=b, T=[c]
    lc5([b])
    lc5([])
    H=b, T=[]
    H=c, T=[b]
    H=a, T=[b,c]
    H=a, T=[c,b]
    lc5([a,c])
    lc5([c])
    lc5([])
    H=c, T=[]
    H=a, T=[c]
    lc5([a])
    lc5([])
    H=a, T=[]
    H=c, T=[a]
    H=b, T=[a,c]
    H=b, T=[c,a]
    lc5([a,b])
    lc5([b])
    lc5([])
    H=b, T=[]
    H=a, T=[b]
    lc5([a])
    lc5([])
    H=a, T=[]
    H=b, T=[a]
    H=c, T=[a,b]
    H=c, T=[b,a]
    [[a,b,c],[a,c,b],[b,a,c],[b,c,a],[c,a,b],[c,b,a]]
    
    In Java it seems that perms(L) -> [[H|T] || H <- L, T <- perms(L--[H])]. would be equivalent to:
    import java.util.ArrayList;
    import java.util.Arrays;
    import java.util.List;
    
    
    public class Perms {
     public static void main(String[] argv) {
      System.out.println(perms(Arrays.asList("A", "B", "C")));
     }
     
     public static List<List<String>> perms(List<String> input) {
      List<List<String>> output = new ArrayList<List<String>>();
      for (String H : input) {
       //L--[H]
       List<String> LminusH = new ArrayList<String>(input);
       LminusH.remove(H);   
       
       if (LminusH.isEmpty()) {
        //[H|T] when T is empty
        output.add(new ArrayList<String>(Arrays.asList(H)));
       } else {
        for (List<String> T : perms(LminusH)) {    
         //a list made up of [H|T]
         List<String> HT = new ArrayList<String>();
         HT.add(H);
         HT.addAll(T);
         output.add(HT);
        }
       }
      }
      return output;
     }
    }
    
    Note that the second generator can reference the output of the first, else the L--[H] wouldn't work, hence the equivalence to nested loops seems correct. Also note this isn't meant to be the best way to accomplish this in Java, rather it is the way most similar to what I think perms(L) -> [[H|T] || H <- L, T <- perms(L--[H])]. does in Erlang.

    That was kind of a lot harder that Hello, World was!!

    Oh and for the record I'm using the Erlang brush for SyntaxHighlighter from http://stevegilham.blogspot.com/2009/08/improved-syntax-highlight-brush-for.html.

    Monday, November 8, 2010

    CXF backwards compatibility: adding an attribute on the server causes an UnmarshalException on the client

    When updating CXF (and therefore JAXB) from cxf-bundle 2.2.3 to 2.2.10 we ran into a problem where our client and server became more coupled than we had in mind. Specifically, adding an attribute to a server entity would cause an exception if the object was returned to a client that wasn't expecting that attribute (eg was using the previous definition of the server interface):

    javax.xml.ws.soap.SOAPFaultException: Unmarshalling Error: unexpected element (uri:"", local:"myNewProperty"). Expected elements are <...list of existing properties...> 
     at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:146)
      ...excitingly lengthy callstack...
    Caused by: javax.xml.bind.UnmarshalException
     - with linked exception:
    [javax.xml.bind.UnmarshalException: unexpected element (uri:"", local:"myNewProperty"). Expected elements are <...list of existing properties...>]
     at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.handleStreamException(UnmarshallerImpl.java:425)
     at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:362)
     at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:339)
     at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:755)
     at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:624)
    ...excitingly lengthy callstack...
    

    This makes it a bit hard to update our server without updating the client in lockstep. We specifically wanted to be able to add attributes to entities without breaking our clients to avoid tying teams together too closely. Luckily it turns out that this problem can be ignored by providing a custom ValidationEventHandler as of CXF 2.2.4 using new properties added for CXF-2455. Unfortunately the event is not very distinct so we have to either follow worst-practices and examine the message to decide if this is the specific error we want to suppress or simply suppress all errors:
    import javax.xml.bind.ValidationEvent;
    import javax.xml.bind.ValidationEventHandler;
    
    
    public class IgnoreUnexpectedElementsHandler implements ValidationEventHandler {
     @Override
     public boolean handleEvent(ValidationEvent event) {
      //true: keep going. In this case we only want to continue for the error we're trying to hide.
      return event.getMessage().startsWith("unexpected element (");
     }
    }
    
    Note that for this example we're assuming our handler goes into the default package which isn't a particularly good idea in reality. Once we have created our validation event handler we need to tell our client to use it. In our case in the Spring configuration of the client:
    <jaxws:client address="${a.property.for.my.services.url}" id="myServiceClient" serviceclass="com.something.myService">
      <jaxws:properties>
       <entry key="jaxb-validation-event-handler" value-ref="ignoreUnexpectedElementsHandler">
      </entry></jaxws:properties>
     </jaxws:client>
    
    Kind of lame to have to do really.

    Thursday, October 21, 2010

    jQuery exactly 1

    jQuery rules. However, it is very very common to see people write $('myselector').something() assuming they match exactly 1 thing. jQuery operates on sets and is just as happy with 0 as 1 as many so our attempt to manipulate that one specific div may go awry unnoticed. A toggle() may be fairly noticable but $('#blah').text(v) may be changing 0 elements for ages before anyone notices.

    To avoid having this type of error fly under the radar I find it very helpful to define $1 as a jQuery selection that raises an error if it doesn't match exactly 1 thing. According to Firebug the $ (or jQuery) function is defined as (jQuery 1.4.2):
    function (a, b) {
        return new (c.fn.init)(a, b);
    }
    
    So we can simply define $1 as delegating to $:
    $1 = function(a, b) {
     var result = $(a, b);
     if (1 != result.length) {
      throw "expected to match exactly one element";
     }
     return result;
    }
    
    Voila! Now we can write $1('someselector') and get a warning there was a js error if our selector doesn't match exactly one thing.

    Google AppEngine JUnit tests that use JDO persistence

    Google AppEngine has a very clear article on setting up local unit tests. It even says how to setup Datastore tests. What it doesn't say explicitly is that this means JDO will auto-magically work too. Naive idiots (self) therefore assume there is some magic incantation that makes it work. Or that it just doesn't work, but that doesn't really seem likely. To further confuse us, the internets abound with complex solutions for getting a PersistenceManager at test-time. Ignore all the noise; it is as simple as setting up the Datastore and just using JDO as normal. In more detail:
    1. As per Google instructions, make sure appengine-testing.jar, appengine-api-labs.jar, and appengine-api-stubs.jar are on your classpath
    2. Setup a datastore for testing, again just as in Googles instructions
    3. Run your test using a PersistenceManager just as you would normally (assuming you have a PMF setup as per Google's example)
      import com.google.appengine.tools.development.testing.LocalDatastoreServiceTestConfig;
      import com.google.appengine.tools.development.testing.LocalServiceTestHelper;
      public class MyTestClass {
          private final LocalServiceTestHelper helper =
              new LocalServiceTestHelper(new LocalDatastoreServiceTestConfig());
      
          @Before
          public void setUp() {
              helper.setUp();        
          }
      
          @After
          public void tearDown() {
              helper.tearDown();
          }
      
       @Test
       public void simpleJdo() {
        MyObject t = new MyObject("test");
        
        PersistenceManager pm;
        
        //prove MyObject doesn't span tests
        pm = PMF.getPm();
        boolean notFound = false;
        try {
         pm.getObjectById(MyObject.class, "test");
         fail("should have raised not found");
        } catch (JDOObjectNotFoundException e) {
         notFound = true;
        } finally {
         pm.close();
        }
        assertTrue(notFound);
        
        pm = PMF.getPm();
        try {
         pm.makePersistent(t);
        } finally {
         pm.close();
        }
        
        pm = PMF.getPm();
        try {
         t = pm.getObjectById(MyObject.class, "test");
        } finally {
         pm.close();
        }
        
        assertNotNull(pm);
        assertEquals("test", t.getName());
       }
      }
      
    4. Yay!

    Tuesday, October 19, 2010

    jQuery & JSON to draw single-elimination tournament bracket

    Often I see sites that present tournament brackets as an image (even on fairly technical sites, eg http://us.battle.net/sc2/en/blog/936927#blog). Purely out of curiosity, I decided to see what would be involved in merely providing the data from the server and letting an HTML UI build up on the fly using jQuery. To avoid needing to produce a server for this I simply hard-coded in JSON that might have been returned from a server with (very) basic information about the tournament:
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-us">
    <head>
      <title>MyTournamentName</title>
     <script src='http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js' type='text/javascript'>
     </script>
     <script src='http://ajax.googleapis.com/ajax/libs/jqueryui/1.7.2/jquery-ui.min.js' type='text/javascript'>
     </script>  
      <script type="text/javascript">  
        var matchInfo = {
          "rounds" : [
            { "name": "Round1",
              "matches" : [
                { "id" : 1, "p1" : "mTwDeMuslim", "p2" : "Luffy" },
                { "id" : 2, "p1" : "SeleCT", "p2" : "NEXGenius" },
                { "id" : 3, "p1" : "Fenix", "p2" : "SoftBall" },
                { "id" : 4, "p1" : "White-Ra", "p2" : "Ice" },
                { "id" : 5, "p1" : "HuK", "p2" : "RedArchon" },
                { "id" : 6, "p1" : "Capoch", "p2" : "Loner" },
                { "id" : 7, "p1" : "mTwDIMAGA", "p2" : "MakaPrime" },
                { "id" : 8, "p1" : "TLAF-Liquid`TLO", "p2" : "SEN" }
              ]
            },
            { "name": "Round2",
              "matches" : [
                { "id" : 9, "p1" : null, "p2" : null },
                { "id" : 10, "p1" : null, "p2" : null },
                { "id" : 11, "p1" : null, "p2" : null },
                { "id" : 12, "p1" : null, "p2" : null }
              ]
            },
            { "name": "Round3",
              "matches" : [
                { "id" : 13, "p1" : null, "p2" : null },
                { "id" : 14, "p1" : null, "p2" : null },
              ]
            },
            { "name": "Round4",
              "matches" : [
                { "id" : 15, "p1" : null, "p2" : null },
              ]
            }                
          ]
        };
      </script>
    </head>
    <body>
      <div>blah blah blah</div>
      <div id="writeHere" class="tournament"></div>
      <div>blah blah blah</div>
    </body>
    </html>
    
    Next we need to write some jQuery code to fill in the div with id="writeHere" with our purely html-based tournament bracket. Easy enough to do (note that some rudimentary css has been slapped in to show us where which bits are):
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-us">
    <head>
      <title>MyTournamentName</title>
     <script src='http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js' type='text/javascript'>
     </script>
     <script src='http://ajax.googleapis.com/ajax/libs/jqueryui/1.7.2/jquery-ui.min.js' type='text/javascript'>
     </script>  
      <style type="text/css">
      .tournament {    
        background-color: #F0F0F0;
        border: dashed 1px solid;
        overflow: auto;
      }
      .tournament .bracket {
        background-color: #DFDFDF;
        min-width: 100px;    
        vertical-align: top;
        float: left;
      }
      
      .tournament .bracket .match {
        background-color: #D0D0D0;
        border-top: 1px solid;
        border-right: 1px solid;
        border-bottom: 1px solid;  
      }
      .tournament .bracket .match .p1 {    
        height: 20px;
      }
      .tournament .bracket .match .p2 {
        height: 20px;
      }    
      .tournament .bracket .match .spacer {
        background-color: #DFDFDF;
        height: 38px;
      }
      .tournament .bracket .spacer {
        height: 80px;
      }
      .tournament .bracket .half-spacer {
        height: 40px;
      }
      .tournament .bracket .small-spacer {
        height: 10px;
        background-color: #F1F1F1;
      }
      
      .left-line {
        border-left: 1px solid;
      }
      
      .tournament .cell {
        min-width: 100px;
        height: 20px;
        float: left;
        background-color: #DFDFDF;    
      }   
      .tournament .l2 {
        background-color: #D0D0D0;
      }     
      .tournament .lmax {
        width: 0px;
        clear: both;
      }    
      </style>
      <script type="text/javascript">
      
        var matchInfo = {
          "rounds" : [
            { "name": "Round1",
              "matches" : [
                { "id" : 1, "p1" : "mTwDeMuslim", "p2" : "Luffy" },
                { "id" : 2, "p1" : "SeleCT", "p2" : "NEXGenius" },
                { "id" : 3, "p1" : "Fenix", "p2" : "SoftBall" },
                { "id" : 4, "p1" : "White-Ra", "p2" : "Ice" },
                { "id" : 5, "p1" : "HuK", "p2" : "RedArchon" },
                { "id" : 6, "p1" : "Capoch", "p2" : "Loner" },
                { "id" : 7, "p1" : "mTwDIMAGA", "p2" : "MakaPrime" },
                { "id" : 8, "p1" : "TLAF-Liquid`TLO", "p2" : "SEN" }
              ]
            },
            { "name": "Round2",
              "matches" : [
                { "id" : 9, "p1" : null, "p2" : null },
                { "id" : 10, "p1" : null, "p2" : null },
                { "id" : 11, "p1" : null, "p2" : null },
                { "id" : 12, "p1" : null, "p2" : null }
              ]
            },
            { "name": "Round3",
              "matches" : [
                { "id" : 13, "p1" : null, "p2" : null },
                { "id" : 14, "p1" : null, "p2" : null },
              ]
            },
            { "name": "Round4",
              "matches" : [
                { "id" : 15, "p1" : null, "p2" : null },
              ]
            }                
          ]
        };
      
        $(document).ready(function($) {       
          var base = $('#writeHere');
          var numTeams = 16;
          var matchesByRound = setupMatchboxes(numTeams);
          
          for (var lvl=0; lvl<matchesByRound.length; lvl++) {                
            var matchBoxes = matchesByRound[lvl];        
            var bracket = checkedAppend('<div class="bracket"></div>', base);
            
            for (var i=0; i<matchBoxes.length; i++) {                     
              var match = matchInfo.rounds[lvl].matches[i];
              var matchHtml = '<div class="match" id="match' + match.id + '">'
                + '<div class="p1">' + fmtName(match.p1) + '</div>'
                + '<div class="spacer"></div>'
                + '<div class="p2">' + fmtName(match.p2) + '</div>';
              checkedAppend(matchHtml, bracket);  
            }
          }      
        });
        
        function fmtName(name) {
          return null != name ? name : '?';
        }
        
        function setupMatchboxes(numTeams) {
          var numLevels = Math.log(numTeams)/Math.LN2;
          var numMatchesForLevel = numTeams / 2;
          var matchBoxes = [];
          
          do {
            var matchesForLevel = [];        
            matchBoxes.push(matchesForLevel);
            
            for (var match=0; match<numMatchesForLevel; match++) {
              matchesForLevel.push(match);
            }
            
            numMatchesForLevel = numMatchesForLevel / 2;
          } while(numMatchesForLevel >= 1);
          return matchBoxes;
        }
        
        function checkedAppend(rawHtml, appendTo) {
          var html = $(rawHtml);
          if (0 == html.length) {
            throw "Built ourselves bad html : " + rawHtml;
          }
          html.appendTo(appendTo);      
          return html;
        }
      </script>
    </head>
    <body>
      <div>blah blah blah</div>
      <div id="writeHere" class="tournament"></div>
      <div>blah blah blah</div>
    </body>
    </html>
    
    However, this doesn't line things up quite as nicely as one might hope (to say the least):

    We have a couple of clear problems:

    1. We probably want a small vertical space between the first row of matches.
    2. For rows 2..N, a match needs to line up such that its top is at the center of one of the matches on the previous row and its bottom is at the center of another. The specific offset helpfully changes from row to row. It turns out to be a bit of a pain to write css for this so instead we'll just write jQuery code to manually size elements for our first pass. Eg we want something like this (note inconsistent sizing and positioning row to row):
    Luckily jQuery provides convenient accessors for height and position so we can write code that literally says "make a vertical spacing div that is half the size of that div and make my div tall enough to stretch from there to there". The main thing that will need an update is that we'll need to keep references to the divs as we go along row by row. This will let us easily set things relative to other things similar to:
    var newH = stretchTo.position().top + stretchTo.height()/2 - matchDiv.position().top;
    
    This will ultimately yield the following javascript gibberish:
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-us">
    <head>
      <title>MyTournamentName</title>
     <script src='http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js' type='text/javascript'>
     </script>
     <script src='http://ajax.googleapis.com/ajax/libs/jqueryui/1.7.2/jquery-ui.min.js' type='text/javascript'>
     </script>  
      <style type="text/css">
      .tournament {    
        background-color: #F0F0F0;
        border: dashed 1px solid;
        overflow: auto;
      }
      .tournament .bracket {
        background-color: #DFDFDF;
        min-width: 100px;    
        vertical-align: top;
        float: left;
      }
      
      .tournament .bracket .match {
        background-color: #D0D0D0;
        border-top: 1px solid;
        border-right: 1px solid;
        border-bottom: 1px solid;  
      }
      .tournament .bracket .match .p1 {    
        height: 20px;
      }
      .tournament .bracket .match .p2 {
        height: 20px;
      }    
      .tournament .bracket .match .spacer {
        background-color: #DFDFDF;
        height: 38px;
      }
      .tournament .bracket .spacer {
        height: 80px;
      }
      .tournament .bracket .half-spacer {
        height: 40px;
      }
      .tournament .bracket .small-spacer {
        height: 10px;
        background-color: #F1F1F1;
      }
      
      .left-line {
        border-left: 1px solid;
      }
      
      .tournament .cell {
        min-width: 100px;
        height: 20px;
        float: left;
        background-color: #DFDFDF;    
      }   
      .tournament .l2 {
        background-color: #D0D0D0;
      }     
      .tournament .lmax {
        width: 0px;
        clear: both;
      }    
      </style>
      <script type="text/javascript">
      
        var matchInfo = {
          "rounds" : [
            { "name": "Round1",
              "matches" : [
                { "id" : 1, "p1" : "mTwDeMuslim", "p2" : "Luffy" },
                { "id" : 2, "p1" : "SeleCT", "p2" : "NEXGenius" },
                { "id" : 3, "p1" : "Fenix", "p2" : "SoftBall" },
                { "id" : 4, "p1" : "White-Ra", "p2" : "Ice" },
                { "id" : 5, "p1" : "HuK", "p2" : "RedArchon" },
                { "id" : 6, "p1" : "Capoch", "p2" : "Loner" },
                { "id" : 7, "p1" : "mTwDIMAGA", "p2" : "MakaPrime" },
                { "id" : 8, "p1" : "TLAF-Liquid`TLO", "p2" : "SEN" }
              ]
            },
            { "name": "Round2",
              "matches" : [
                { "id" : 9, "p1" : null, "p2" : null },
                { "id" : 10, "p1" : null, "p2" : null },
                { "id" : 11, "p1" : null, "p2" : null },
                { "id" : 12, "p1" : null, "p2" : null }
              ]
            },
            { "name": "Round3",
              "matches" : [
                { "id" : 13, "p1" : null, "p2" : null },
                { "id" : 14, "p1" : null, "p2" : null },
              ]
            },
            { "name": "Round4",
              "matches" : [
                { "id" : 15, "p1" : null, "p2" : null },
              ]
            }                
          ]
        };
      
        $(document).ready(function($) {       
          var base = $('#writeHere');
          var numTeams = 16;
          var matchesByRound = setupMatchboxes(numTeams);
          var matchDivsByRound = [];
          
          for (var lvl=0; lvl<matchesByRound.length; lvl++) {                
            var matchBoxes = matchesByRound[lvl];        
            var bracket = checkedAppend('<div class="bracket"></div>', base);
            var matchDivs = [];
            matchDivsByRound.push(matchDivs);
            
            for (var i=0; i<matchBoxes.length; i++) {                     
              var vOffset = checkedAppend('<div></div>', bracket);
            
              var match = matchInfo.rounds[lvl].matches[i];
              var matchHtml = '<div class="match" id="match' + match.id + '">'
                + '<div class="p1">' + fmtName(match.p1) + '</div>'
                + '<div class="spacer"></div>'
                + '<div class="p2">' + fmtName(match.p2) + '</div>';
              matchDiv = checkedAppend(matchHtml, bracket);
              matchDivs.push(matchDiv);
              
              if (lvl > 0) {
                //row 2+; line up with previous row
                var alignTo = matchDivsByRound[lvl-1][i*2];
                //offset to line up tops
                var desiredOffset = alignTo.position().top - matchDiv.position().top;
                
                //offset by half the previous match-height
                desiredOffset += alignTo.height() / 2;
                vOffset.height(desiredOffset);            
              } else {
                checkedAppend('<div class="small-spacer"></div>', bracket);
              }
              
              if (lvl > 0) {
                //tweak our size so we stretch to the middle of the appropriate element
                var stretchTo = matchDivsByRound[lvl-1][i*2+1];
                var newH = stretchTo.position().top + stretchTo.height()/2 - matchDiv.position().top;            
                var deltaH = newH - matchDiv.height();
                matchDiv.height(newH);
                var spacer = matchDiv.find('.spacer');
                spacer.height(spacer.height() + deltaH);
              }          
            }
          }      
        });
        
        function fmtName(name) {
          return null != name ? name : '?';
        }
        
        function setupMatchboxes(numTeams) {
          var numLevels = Math.log(numTeams)/Math.LN2;
          var numMatchesForLevel = numTeams / 2;
          var matchBoxes = [];
          
          do {
            var matchesForLevel = [];        
            matchBoxes.push(matchesForLevel);
            
            for (var match=0; match<numMatchesForLevel; match++) {
              matchesForLevel.push(match);
            }
            
            numMatchesForLevel = numMatchesForLevel / 2;
          } while(numMatchesForLevel >= 1);
          return matchBoxes;
        }
        
        function checkedAppend(rawHtml, appendTo) {
          var html = $(rawHtml);
          if (0 == html.length) {
            throw "Built ourselves bad html : " + rawHtml;
          }
          html.appendTo(appendTo);      
          return html;
        }
      </script>
    </head>
    <body>
      <div>blah blah blah</div>
      <div id="writeHere" class="tournament"></div>
      <div>blah blah blah</div>
    </body>
    </html>
    
    On nice modern browsers this yields something like this:

    Last of all lets clean up our javascript slightly, in particular making our code a little more directly based on the JSON and a little less on hard-coded test variables like numTeams. And lets add a spot for the final victor:
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-us">
    <head>
      <title>MyTournamentName</title>
     <script src='http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js' type='text/javascript'>
     </script>
     <script src='http://ajax.googleapis.com/ajax/libs/jqueryui/1.7.2/jquery-ui.min.js' type='text/javascript'>
     </script>  
      <style type="text/css">
      .tournament {    
        background-color: #F0F0F0;
        border: dashed 1px solid;
        overflow: auto;
      }
      .tournament .bracket {
        background-color: #DFDFDF;
        min-width: 100px;    
        vertical-align: top;
        float: left;
      }
      
      .tournament .bracket .match {
        background-color: #D0D0D0;
        border-top: 1px solid;
        border-right: 1px solid;
        border-bottom: 1px solid;  
      }
      .tournament .bracket .match .p1 {    
        height: 20px;
      }
      .tournament .bracket .match .p2 {
        height: 20px;
      }    
      .tournament .bracket .match .spacer {
        background-color: #DFDFDF;
        height: 38px;
      }
      .tournament .bracket .spacer {
        height: 80px;
      }
      .tournament .bracket .half-spacer {
        height: 40px;
      }
      .tournament .bracket .small-spacer {
        height: 10px;
        background-color: #F1F1F1;
      }
      .tournament .bracket .winner {
        border-bottom: 1px solid;
      }
      
      .left-line {
        border-left: 1px solid;
      }
      
      .tournament .cell {
        min-width: 100px;
        height: 20px;
        float: left;
        background-color: #DFDFDF;    
      }   
      .tournament .l2 {
        background-color: #D0D0D0;
      }     
      .tournament .lmax {
        width: 0px;
        clear: both;
      }    
      </style>
      <script type="text/javascript">
      
        var matchInfo = {
          "rounds" : [
            { "name": "Round1",
              "matches" : [
                { "id" : 1, "p1" : "mTwDeMuslim", "p2" : "Luffy" },
                { "id" : 2, "p1" : "SeleCT", "p2" : "NEXGenius" },
                { "id" : 3, "p1" : "Fenix", "p2" : "SoftBall" },
                { "id" : 4, "p1" : "White-Ra", "p2" : "Ice" },
                { "id" : 5, "p1" : "HuK", "p2" : "RedArchon" },
                { "id" : 6, "p1" : "Capoch", "p2" : "Loner" },
                { "id" : 7, "p1" : "mTwDIMAGA", "p2" : "MakaPrime" },
                { "id" : 8, "p1" : "TLAF-Liquid`TLO", "p2" : "SEN" }
              ]
            },
            { "name": "Round2",
              "matches" : [
                { "id" : 9, "p1" : null, "p2" : null },
                { "id" : 10, "p1" : null, "p2" : null },
                { "id" : 11, "p1" : null, "p2" : null },
                { "id" : 12, "p1" : null, "p2" : null }
              ]
            },
            { "name": "Round3",
              "matches" : [
                { "id" : 13, "p1" : null, "p2" : null },
                { "id" : 14, "p1" : null, "p2" : null },
              ]
            },
            { "name": "Round4",
              "matches" : [
                { "id" : 15, "p1" : null, "p2" : null },
              ]
            } 
          ]
        };
      
        $(document).ready(function($) {       
          var base = $('#writeHere');
          var matchDivsByRound = [];
          
          for (var roundIndex=0; roundIndex<matchInfo.rounds.length; roundIndex++) {    
            var round = matchInfo.rounds[roundIndex];
            var bracket = checkedAppend('<div class="bracket"></div>', base);
            var matchDivs = [];
            matchDivsByRound.push(matchDivs);
            
            //setup the match boxes round by round
            for (var i=0; i<round.matches.length; i++) {                     
              var vOffset = checkedAppend('<div></div>', bracket);
            
              var match = round.matches[i];
              var matchHtml = '<div class="match" id="match' + match.id + '">'
                + '<div class="p1">' + fmtName(match.p1) + '</div>'
                + '<div class="spacer"></div>'
                + '<div class="p2">' + fmtName(match.p2) + '</div>';
              matchDiv = checkedAppend(matchHtml, bracket);
              matchDivs.push(matchDiv);
              
              if (roundIndex > 0) {
                //row 2+; line up with previous row
                var alignTo = matchDivsByRound[roundIndex-1][i*2];
                //offset to line up tops
                var desiredOffset = alignTo.position().top - matchDiv.position().top;
                
                //offset by half the previous match-height
                desiredOffset += alignTo.height() / 2;
                vOffset.height(desiredOffset);            
              } else {
                checkedAppend('<div class="small-spacer"></div>', bracket);
              }
              
              if (roundIndex > 0) {
                //tweak our size so we stretch to the middle of the appropriate element
                var stretchTo = matchDivsByRound[roundIndex-1][i*2+1];
                var newH = stretchTo.position().top + stretchTo.height()/2 - matchDiv.position().top;            
                var deltaH = newH - matchDiv.height();
                matchDiv.height(newH);
                var spacer = matchDiv.find('.spacer');
                spacer.height(spacer.height() + deltaH);
              }          
            }                
          }
          //setup the final winners box; just a space for a name whose bottom is centrally aligned with the last match
          bracket = checkedAppend('<div class="bracket"></div>', base);
          var vOffset = checkedAppend('<div></div>', bracket);
          var alignTo = matchDivsByRound[matchInfo.rounds.length - 1][0]; //only 1 match in the last round
          var html = '<div class="winner">?</div>';
          var winnerDiv = checkedAppend(html, bracket);      
          vOffset.height(alignTo.position().top - winnerDiv.position().top + alignTo.height() / 2 - winnerDiv.height());
        });
        
        function fmtName(name) {
          return null != name ? name : '?';
        }
        
        function checkedAppend(rawHtml, appendTo) {
          var html = $(rawHtml);
          if (0 == html.length) {
            throw "Built ourselves bad html : " + rawHtml;
          }
          html.appendTo(appendTo);      
          return html;
        }
      </script>
    </head>
    <body>
      <div>blah blah blah</div>
      <div id="writeHere" class="tournament"></div>
      <div>blah blah blah</div>
    </body>
    </html>
    

    Ugly, but sized and positioned the way we want, ready to actually talk to a server and/or get some dynamic elements (eg the ability to designate a winner and have them promote through the tournament).

    Ultimately this will hopefully get rolled up into a practicum project involving tournament management.

    Saturday, October 9, 2010

    Windows 7 has directory symlinks! And they are a great way to work around Dropbox limitations!

    Dropbox is moderately awesome. One of the remarkably rare pieces of software that "just works" (so far). Except they haven't implemented the sync any folder feature (vote here!) despite receiving 60k votes for it over an 11 month period :( This causes horribly mundane and annoying problems like:
    • I want to use P:\Dropbox as my sync folder; ok, works great!
    • I have a shared folder for Starcraft 2 replays so I get replays under P:\Dropbox\My Dropbox\SC2-Replay. 
    • Starcraft 2 doesn't look there; in order to have these replays show up for selection in the replays list I need to sync only the SC2-Replay directory into a subdirectory of the directory Starcraft 2 looks at 
      • That is, I want C:\Users\UserName\Documents\StarCraft II\Accounts\########\1-S2-1-######\Replays to sync with the same thing as P:\Dropbox\My Dropbox\SC2-Replay
    I could just put the entire Dropbox folder under Starcraft 2 but that really isn't ideal. In XP we'd have a problem; NTFS only had junction points (ref) and it wasn't as easy as you might hope to create them. Luckily in Windows 7 (and Vista) we finally get directory symlinks (ref)! Always nice to learn something useful while trying to accomplish something frivolous like making watching game replays easier.

    So, having found out directory symlinks are now available I figured I'd find out how to create one. The first way I found was perhaps not as easy as I'd hoped:

    BOOLEAN WINAPI CreateSymbolicLink(
      __in  LPTSTR lpSymlinkFileName,
      __in  LPTSTR lpTargetFileName,
      __in  DWORD dwFlags
    );
    

    Luckily it turns out that Windows 7 also provide mklink.exe (Vista has it too; although new to me mklink.exe is old news to the world) to handle all your command prompt NTFS linking needs. You can even create directory junctions (ref) with it if so inclined! Plus Windows Explorer now handles deletion of a directory junction graciously - the files don't get deleted if you delete the junction directory expecting the original/other access point in the file system to stick around.

    The mklink.exe arguments are nice and simple:

    >mklink /?
    Creates a symbolic link.
    
    MKLINK [[/D] | [/H] | [/J]] Link Target
    
            /D      Creates a directory symbolic link.  Default is a file
                    symbolic link.
            /H      Creates a hard link instead of a symbolic link.
            /J      Creates a Directory Junction.
            Link    specifies the new symbolic link name.
            Target  specifies the path (relative or absolute) that the new link
                    refers to.
    

    Somewhat curiously (at least to me) creating a directory junction to link up my Dropbox replay to the Starcraft 2 directory tree did not require elevated permissions but creating a directory symlink required me to run cmd.exe as admin. Apparently this is by design but it seems a bit odd; I would have thought a junction was a bigger deal than a symlink. Anyway, once running as admin the link creates as easily as one could hope:

    C:\Users\Imaginary\Documents\StarCraft II\Accounts\########\1-S2-1-######\Replays\Multiplayer>mklink /D DropboxSymLink "P:\Dropbox\My Dropbox\SC2-Replay"
    symbolic link created for DropboxSymLink <<===>> P:\Dropbox\My Dropbox\SC2-Replay
    

    And now I can see the replays people drop into our shared Dropbox folder from within Starcraft 2; yay!

    It may be a bit manual but until that Dropbox gets that external folders feature this is going to be very handy; I can have different parts of my Dropbox show up scattered all over my filesystem now!

    Tuesday, October 5, 2010

    Why did nobody tell me Windows 7 has copy path built in on a hidden menu?!

    For years I've been installing Ninotech Path Copy on my Windows boxes. Recently I upgraded from XP to Win7 and after using it for a few months I finally learned from the internets that Path Copy is not for Win7 ... but it already has the most important feature - copying a full path to a file or folder - built in!

    Just shift-rightclick:
    Amazing. Worth the upgrade for that alone! Well ... almost. Now if only I could figure out how to tell HKEY_CLASSES_ROOT\Allfilesystemobjects I want that on the regular right-click menu instead of only showing up if I shift-right-click.

    Monday, October 4, 2010

    NoteToSelf: JVisualVM rules. And -XX:+UseConcMarkSweepGC messes it up.

    JVisualVM (under your JDK /bin directory, on Windows something like C:\Program Files\Java\jdk1.6.0_07\bin\jvisualvm.exe), released with JDK version 6, update 7, provides an awesome view into your JVM processes.

    Local processes are automagically shown, which is very helpful:

    The overview will give you JVM arguments and system properties. This can be very helpful when someone claims they started with one set of arguments and you believe otherwise! The Monitor tab has a nice UI for grabbing a quick heap dump and viewing it, as well as some graphs that can often immediately point out that you have a memory leak (particularly if you don't see a nice healthy saw-tooth for the heap). The Threads tab shockingly gives you information about threads and the ability to perform thread dumps. Although it has never actually been helpful for me yet the visibility into running/waiting is very cool:

    Even better, JVisualVM makes monitoring remote processes very easy. Just start the remote JVM with arguments similar to java -Dcom.sun.management.jmxremote.port=7890 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false ...whatever... and then on the client start JVisualVM and choose File>Add JMX Connection and enter hostname:7890 (or whatever number you set for com.sun.management.jmxremote.port) and to see into a remote VM. You can also use jstad but in many cases we don't start this so being able to easily boot arbitrary processes as monitorable is a major plus.

    Unfortunately there is a gotcha. If the JVM arguments to the process you want to watch included -XX+UseConcMarkSweepGC you will see a little red warning in the bottom right informing you there was a NullPointerException at com.sun.tools.visualvm.jvm.MonitoredDataImpl.:

    This will result in the Monitor tab being completely useless; none of the graphs will populate. Apparently this is due to a bug (see https://visualvm.dev.java.net/issues/show_bug.cgi?id=128) that is fixed. Sadly the fix seems not to be deployed into my environments yet :(

    Thursday, September 23, 2010

    Not the timezones you are looking for

    Probably purely by coincidence I have recently been asked repeatedly about Java Date values changing when in fact the only difference is how the value was formatted for display.

    For example, why did the date change here?

    2010-09-23 17:27:54,565 DEBUG [SomeClass] start date Thu Sep 09 00:00:00 UTC 2010 
    ...
    2010-09-23 10:44:24,728 DEBUG [SomeClass] start date Wed Sep 08 17:00:00 PDT 2010
    

    At a glance it looks like the date isn't the same: it isn't the same day, hour, or timezone!

    The answer is the first line of the Date class javadoc:
    "The class Date represents a specific instant in time, with millisecond precision." (see here)
    Note that it does NOT have a timezone. A timezone is applied only when formatting the date. We can even prove the date hasn't changed:

    import static org.junit.Assert.assertEquals;
    
    import java.text.DateFormat;
    import java.text.ParseException;
    import java.text.SimpleDateFormat;
    import java.util.Date;
    
    import org.junit.Test;
    
    
    public class DateTzTest {
     @Test
     public void dateCompare() throws ParseException {
      DateFormat df = new SimpleDateFormat("EEE MMM dd hh:mm:ss z yyyy");
      Date d1 =  df.parse("Thu Sep 09 00:00:00 UTC 2010");
      Date d2 =  df.parse("Wed Sep 08 17:00:00 PDT 2010");
      
      assertEquals(d1.getTime(), d2.getTime());
     }
    }
    

    Wednesday, September 15, 2010

    Make it CamelCase: removing sequences of capitals leveraging regex lookahead/lookbehind

    I ran into a novel (in that it isn't something I need to do very often) problem today: the need to convert names that might have sequences of capitals to CamelCase with no consecutive capitals in Java. For example:

    BEFORE => AFTER
    MyGoodName => MyGoodName
    MYGoodName => MyGoodName
    MyGOODName => MyGoodName
    MyGoodNAME => MyGoodName
    EndOfIT => EndOfIt
    EndOfItALL => EndOfItAll
    UUIDIsACOOLType => UuidIsAcoolType

    It's easy enough to find such sequences using a regular expression but how to correctly replace them was slightly less clear. So, starting with the easy part, to find such sequences we can use a pattern that says "a capital, followed by some more capitals, ending with either another capital or the end of the string":

    //1+ CAPS preceeded by a CAP and ending in either another CAP or end-of-string. 
    String expr = "[A-Z][A-Z]+([A-Z]|$)";
    

    The middle set of capitals - the [A-Z]+ - are the ones we'd want to change to lowercase. So ... how to match and replace? Probably we'll want to put that into a group so we can easily use it in a replace:

    //1+ CAPS preceeded by a CAP and ending in either another CAP or end-of-string. 
    String expr = "[A-Z][A-Z]+([A-Z]|$)";
    

    The normal replaceAll style APIs on Matcher do not appear to easily allow you to swap in a modified version of a group. Luckily the appendReplacement and appendTail APIs do allow this. There is a great example in the javadoc.

    So ... we should be able to do something like this:

    import java.util.regex.Matcher;
    import java.util.regex.Pattern;
    
    public class RegexTest {
     public static void main(String[] argv) {  //
      //1+ CAPS preceeded by a CAP and ending in either another CAP or end-of-string. 
      String expr = "[A-Z][A-Z]+[A-Z]|$";
      String[] testStrings = { "MyGoodName", "MYGoodName", "MyGOODName", "MyGoodNAME", "EndOfIT", "EndOfItALL", "UUIDIsACOOLType" };
      
      Pattern pattern = Pattern.compile(expr);
      for (String testString : testStrings) {
       Matcher matcher = pattern.matcher(testString);
       
       StringBuffer sb = new StringBuffer();
       while (matcher.find()) {
        matcher.appendReplacement(sb, matcher.group().toLowerCase());    
       }
       matcher.appendTail(sb);
       System.out.printf("%1$s => %2$s\n", testString, sb.toString());
      }
     }    
    }
    

    Unfortunately this doesn't work because our match is the entire string so we end up with the following output:

    MyGoodName => MyGoodName
    MYGoodName => mygoodName
    MyGOODName => Mygoodname
    MyGoodNAME => MyGoodname
    EndOfIT => EndOfIT
    EndOfItALL => EndOfItall
    UUIDIsACOOLType => uuidisacooltype
    

    We could monkey around with trying to replace only a specific group based on it's start/end indices or some such nonsense but it would really be much nicer to match only the consecutive caps, with the leading cap and trailing cap or end-of-string not actually being considered a part of the match. Luckily java supports lookaround in regular expressions. We can revise our program to use lookahead/behind for the start/end and have only the desired bit be the match:

    import java.util.regex.Matcher;
    import java.util.regex.Pattern;
    
    public class RegexTest {
     public static void main(String[] argv) {  //
      //1+ CAPS preceeded by a CAP and ending in either another CAP or end-of-string. 
      String expr = "(?<=[A-Z])[A-Z]+(?=[A-Z]|$)";
      String[] testStrings = { "MyGoodName", "MYGoodName", "MyGOODName", "MyGoodNAME", "EndOfIT", "EndOfItALL", "UUIDIsACOOLType" };
      
      Pattern pattern = Pattern.compile(expr);
      for (String testString : testStrings) {
       Matcher matcher = pattern.matcher(testString);
       
       StringBuffer sb = new StringBuffer();
       while (matcher.find()) {
        matcher.appendReplacement(sb, matcher.group().toLowerCase());    
       }
       matcher.appendTail(sb);
       System.out.printf("%1$s => %2$s\n", testString, sb.toString());
      }
     }    
    }
    

    This will finally produce the desired output:
    MyGoodName => MyGoodName
    MYGoodName => MyGoodName
    MyGOODName => MyGoodName
    MyGoodNAME => MyGoodName
    EndOfIT => EndOfIt
    EndOfItALL => EndOfItAll
    UUIDIsACOOLType => UuidIsAcoolType
    

    Both the lookahead/behind and the appendReplacement/Tail are very handy but relatively rarely used in my experience.

    Friday, September 10, 2010

    The leaking of the JAXB

    A novel memory leak was discovered in some of our software today. Apparently JAXBContext will create new dynamic classes each time newInstance is called. This means if you naively implemented a JAXB-based XML translation, similar to the example below, you would create new classes endlessly and run out of memory:

    StringWriter sw = new StringWriter();
    JAXBContext jaxbContext = JAXBContext.newInstance(myObject.getClass());
    Marshaller marshal = jaxbContext.createMarshaller();
    marshal.marshal(myObject, sw);
    

    The unfortunate thing is that this is exactly what one would come up with after skimming the examples in the javadoc.

    The recommended solution on the internets is to cache the JAXBContext, including re-using it on multiple threads. This is a bit novel as the documentation for the JAXBContext doesn't indicate this will happen and it also doesn't indicate the JAXBContext is thread-safe. To discover that you have to go into specific implementation information, such as https://jaxb.dev.java.net/guide/Performance_and_thread_safety.html.

    These details vary by implementation so one might legitimately argue the javase documentation is correct. It's really just aggravating to discover that the implementation is built in a way so likely to cause clients to leak. This strikes me as a great example of a very bad implementation of an API, especially given that the implementation could easily maintain a lookup and avoid re-creating the dynamic classes and creating the leak for us.

    Thursday, June 24, 2010

    One-liner for errors per hour in a rolling log4j log

    Because I keep forgetting valid syntax and having an example is very very handy:



    for h in $(seq 0 23); do hh=$(printf "%02d" $h);  echo $hh $(grep "^2010-06-24 $hh.*general exception" some-dir?/ALogFile?.log* | wc -l); done;

    Monday, June 7, 2010

    So you smashed your macbook screen and want to use an external monitor as primary

    After destroying the screen on my macbook I got to thinking the machine should work fine with an external monitor. The system would boot fine and display an exciting kaleidoscope of colors but aside from the monitor problem it appeared to be working. On connecting an external monitor I found that it does indeed start using that monitor. Now the only problem was that everything likes to start on the main monitor and when you can't see that monitor it is challenging to move things onto the secondary display.

    Searches for keyboard shortcuts to move things between screens or ways to disable the primary monitor on startup didn't turn up much. Simply searching for the actual problem "macbook monitor broken" turned up a thread titled "Macbook screen broken, help w/ external monitor" (here). This proposed use of "clamshell mode" to get around the problem. This sounded perfect. Unfortunately, after getting a keyboard and mouse from IT the Macbook just went to sleep every time I closed it. Imperfect clamshell.

    More googling revealed (eventually) that the Macbook only works in clamshell mode when connected to power; no battery powered clamshells, as described on both Forever For Now (here) and an Apple support page regarding clamshell (here).

    Luckily next time I smash a Macbook screen I will remember that it happened to me before and I wrote down how to deal with it!

    Wednesday, April 7, 2010

    Watir 102: Javascripting XPCOM to remove session cookies in a FireWatir script

    Yesterday (see previous entry) I was playing with FireWatir and found that I needed to remove session cookies to ensure a clean starting point for my script. XPCOM allowed me to read back cookies and it seemed very likely that it would also facilitate deletion.

    The Mozilla Developer Center cookies examples were very close to what I needed. nsICookieManager.remove seemed perfect. In a FireWatir script it looked like what I needed to read/write cookies was something like this:

    def removeCookie(brwsr, host, name, path)
       cmd = "Components.classes['@mozilla.org/cookiemanager;1']"
       cmd += ".getService(Components.interfaces.nsICookieManager)"
       cmd += ".remove(\"" + host + "\", \"" + name + "\", \"" + path + "\", false)";
       puts "Removing cookie " + name + " from " + host + " for path " + path
       result = brwsr.execute_script "#{cmd}\n"   
    end
    
    def readCookies(brwsr, site)
       cmd = "var uri = Components.classes['@mozilla.org/network/io-service;1'].getService(Components.interfaces.nsIIOService)"
       cmd += ".newURI(\"" + site + "\", null, null);"
       cmd += "Components.classes['@mozilla.org/cookieService;1']"
       cmd += ".getService(Components.interfaces.nsICookieService)"
       cmd += ".getCookieString(uri, null);"
       result = brwsr.execute_script "#{cmd}\n"   
    end
    

    In my initial attempts to use this I passed in "http://myhost" and although the method succeeded subsequent logging of the updated cookie string showed no change at all. After some fumbling around I discovered a detailed example of manipulating cookies through javascript in an extension. The main difference in their example was no protocol:// in front of the host/domain name. After that it worked perfectly. A simple example follows:

    require "watir"
    
    protocol = "http";
    host = "mytesthost";
    initial_path = "/somethingfuntotest"
    
    site = protocol + "://" + host
    
    def url(brwsr)
       return brwsr.js_eval("document.location.toString();")
    end
    
    def readCookies(brwsr, site)
       cmd = "var uri = Components.classes['@mozilla.org/network/io-service;1'].getService(Components.interfaces.nsIIOService)"
       cmd += ".newURI(\"" + site + "\", null, null);"
       cmd += "Components.classes['@mozilla.org/cookieService;1']"
       cmd += ".getService(Components.interfaces.nsICookieService)"
       cmd += ".getCookieString(uri, null);"
       result = brwsr.execute_script "#{cmd}\n"   
    end
    
    #Ref https://developer.mozilla.org/en/nsICookieManager
    #Ref http://www.springenwerk.com/2007/07/using-cookies-with-firefox-extensions.html
    #Usage removeCookie(brwsr, "myhostname", "SessionIdCookieName", "/")
    #Note that passing http://myhostname for host does not appear to work
    def removeCookie(brwsr, host, name, path)
       cmd = "Components.classes['@mozilla.org/cookiemanager;1']"
       cmd += ".getService(Components.interfaces.nsICookieManager)"
       cmd += ".remove(\"" + host + "\", \"" + name + "\", \"" + path + "\", false)";
       puts "Removing cookie " + name + " from " + host + " for path " + path
       result = brwsr.execute_script "#{cmd}\n"   
    end
    
    Watir::Browser.default = "firefox"
    brwsr = Watir::Browser.new
    
    #May have a session id
    puts "Cookies for " + site + ": " + readCookies(brwsr, site)
    removeCookie(brwsr, host, "JSESSIONID", "/")
    #Session id is now gone ... yay!
    puts "Cookies for " + site + ": " + readCookies(brwsr, site)
    

    Tuesday, April 6, 2010

    Watir 101: Javascripting XPCOM to read cookies in a FireWatir script

    Today I ran into the need to automate a web UI built primarily of html and javascript. HtmlUnit based solutions seemed nice but Watir seemed like an ideal solution as it would necessarily produce the exact same results a real user would experience.

    First step was to get setup, following the instructions at http://watir.com/installation/#win. Despite my complete lack of experience with both Ruby and Watir things went pretty smoothly. My setup sequence was:

    1. Install Ruby (http://rubyforge.org/frs/download.php/29263/ruby186-26.exe)
    2. At new cmdprompt:
      1. We need a new command prompt because one open from before the Ruby install will not notice the updated environment variables and thus will not have ruby on PATH
      2. Note that the second command is slow as hell (at least it was for me)
        gem update --system
        gem install watir
        
    3. Install FF JSSH plugin; ref http://watir.com/installation/#win

    4. Poke around a couple of helpful documents to try to learn the basics

    5. Run scripts similar to ruby my_script.rb

    Scripting the basic interaction went pretty smoothly. Obtaining a Firefox browser and jumping around is every bit as straight forward as you might hope. Obtaining the current url is slightly more problemmatic than one might help but luckily others have solved this problem already. Here is an example of a simple script:
    require "watir"
    
    site = "http://my.test.host"
    initial_path = "/blah"
    
    def url(brwsr)
       return brwsr.js_eval("document.location.toString();")
    end
    
    Watir::Browser.default = "firefox"
    
    puts "Requesting a new " + Watir::Browser.default
    brwsr = Watir::Browser.new
    
    puts "Loading " + site + initial_path
    brwsr.goto site + initial_path
    puts "At " + url(brwsr)
    
    puts "Selecting and navigating forward"
    brwsr.radio(:name, "inputRadio").set
    brwsr.button(:name, "Continue").click
    puts "At " + url(brwsr)
    

    The only real problem with basic scripting functionality I ran into (as yet unresolved) is that if something takes longer than Watir likes to wait (say the browser is slow to pop or a page is slow to respond) the script gets ahead of the browser and gets severely confused.

    So now for the first real problem I ran into: Cookies. The application I want to test is session-aware so ultimately I will want to nuke cookies at the beginning of the test. First step, lets see if we can get cookie information back. After a bit of searching it turns out manipulating cookies from Watir is less simple than one might hope. For IE people seem to simply use Ruby to delete the files out of the filesystem (see FAQ). For Firefox we can alter cookies via XPCOM, Mozilla even has an example.

    This is all well and good but how can we execute javascript against XPCOM objects from Ruby code running FireWatir?? Luckily I found a great series of articles that were almost what I wanted:

    The only problem was that the example called for rebuilding some of the code ("I opened up the Firewatir::Firefox class and added...") for FireWatir. This would definitely be stretching my fledgling Ruby/Watir skills so I went looking for a simpler approach.

    The browser object for Firefox (FireWatir::Firefox) exposes the wonderful execute_script method. The description "Executes the given JavaScript string" sounds pretty promising. A quick check of the source firefox.rb file showed that the method is a thin wrapper around the same JSSH interaction sequence that is used in the http://sticklebackplastic.com articles. This all seemed almost too good to be true so I decided to see if it actually worked the way it appeared. Luckily it works just as you would hope; the following example shows Ruby code reading cookies for whatever domain it happens to be interested in:

    require "watir"
    
    site = "http://my.test.host"
    initial_path = "/blah"
    
    def prepare(brwsr, site)
       puts "Preparing..."
       cmd = "var uri = Components.classes['@mozilla.org/network/io-service;1'].getService(Components.interfaces.nsIIOService)"
       cmd += ".newURI(\"" + site + "\", null, null);"
       cmd += "Components.classes['@mozilla.org/cookieService;1']"
       cmd += ".getService(Components.interfaces.nsICookieService)"
       cmd += ".getCookieString(uri, null);"
       result = brwsr.execute_script "#{cmd}\n"
       
       puts "Cookies for " + site + ": " + result
    end
    
    Watir::Browser.default = "firefox"
    
    puts "Requesting a new " + Watir::Browser.default
    brwsr = Watir::Browser.new
    
    showcookie(brwsr, site)
    showcookie(brwsr, "http://www.bing.com")
    
    Hopefully a similar mechanism will allow me to alter cookies but that's as far as I've made it so far. Seems quite promising! And there are lots of cool names (Watir, Ruby, JSSH, XPCOM, ...) involved, always a plus!

    I was very impressed that starting with no knowledge of Ruby or Watir I was able to get everything installed and create a functional proof of concept script (albeit one very low on validation or error checks) for a semi-complex sequence (add item to cart and checkout) in a couple of hours! If I can get cookie removal working to ensure a clean start state this could grow to be a very valuable verification tool for our developers.