Thursday, September 23, 2010

Not the timezones you are looking for

Probably purely by coincidence I have recently been asked repeatedly about Java Date values changing when in fact the only difference is how the value was formatted for display.

For example, why did the date change here?

2010-09-23 17:27:54,565 DEBUG [SomeClass] start date Thu Sep 09 00:00:00 UTC 2010 
...
2010-09-23 10:44:24,728 DEBUG [SomeClass] start date Wed Sep 08 17:00:00 PDT 2010

At a glance it looks like the date isn't the same: it isn't the same day, hour, or timezone!

The answer is the first line of the Date class javadoc:
"The class Date represents a specific instant in time, with millisecond precision." (see here)
Note that it does NOT have a timezone. A timezone is applied only when formatting the date. We can even prove the date hasn't changed:

import static org.junit.Assert.assertEquals;

import java.text.DateFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;

import org.junit.Test;


public class DateTzTest {
 @Test
 public void dateCompare() throws ParseException {
  DateFormat df = new SimpleDateFormat("EEE MMM dd hh:mm:ss z yyyy");
  Date d1 =  df.parse("Thu Sep 09 00:00:00 UTC 2010");
  Date d2 =  df.parse("Wed Sep 08 17:00:00 PDT 2010");
  
  assertEquals(d1.getTime(), d2.getTime());
 }
}

Wednesday, September 15, 2010

Make it CamelCase: removing sequences of capitals leveraging regex lookahead/lookbehind

I ran into a novel (in that it isn't something I need to do very often) problem today: the need to convert names that might have sequences of capitals to CamelCase with no consecutive capitals in Java. For example:

BEFORE => AFTER
MyGoodName => MyGoodName
MYGoodName => MyGoodName
MyGOODName => MyGoodName
MyGoodNAME => MyGoodName
EndOfIT => EndOfIt
EndOfItALL => EndOfItAll
UUIDIsACOOLType => UuidIsAcoolType

It's easy enough to find such sequences using a regular expression but how to correctly replace them was slightly less clear. So, starting with the easy part, to find such sequences we can use a pattern that says "a capital, followed by some more capitals, ending with either another capital or the end of the string":

//1+ CAPS preceeded by a CAP and ending in either another CAP or end-of-string. 
String expr = "[A-Z][A-Z]+([A-Z]|$)";

The middle set of capitals - the [A-Z]+ - are the ones we'd want to change to lowercase. So ... how to match and replace? Probably we'll want to put that into a group so we can easily use it in a replace:

//1+ CAPS preceeded by a CAP and ending in either another CAP or end-of-string. 
String expr = "[A-Z][A-Z]+([A-Z]|$)";

The normal replaceAll style APIs on Matcher do not appear to easily allow you to swap in a modified version of a group. Luckily the appendReplacement and appendTail APIs do allow this. There is a great example in the javadoc.

So ... we should be able to do something like this:

import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class RegexTest {
 public static void main(String[] argv) {  //
  //1+ CAPS preceeded by a CAP and ending in either another CAP or end-of-string. 
  String expr = "[A-Z][A-Z]+[A-Z]|$";
  String[] testStrings = { "MyGoodName", "MYGoodName", "MyGOODName", "MyGoodNAME", "EndOfIT", "EndOfItALL", "UUIDIsACOOLType" };
  
  Pattern pattern = Pattern.compile(expr);
  for (String testString : testStrings) {
   Matcher matcher = pattern.matcher(testString);
   
   StringBuffer sb = new StringBuffer();
   while (matcher.find()) {
    matcher.appendReplacement(sb, matcher.group().toLowerCase());    
   }
   matcher.appendTail(sb);
   System.out.printf("%1$s => %2$s\n", testString, sb.toString());
  }
 }    
}

Unfortunately this doesn't work because our match is the entire string so we end up with the following output:

MyGoodName => MyGoodName
MYGoodName => mygoodName
MyGOODName => Mygoodname
MyGoodNAME => MyGoodname
EndOfIT => EndOfIT
EndOfItALL => EndOfItall
UUIDIsACOOLType => uuidisacooltype

We could monkey around with trying to replace only a specific group based on it's start/end indices or some such nonsense but it would really be much nicer to match only the consecutive caps, with the leading cap and trailing cap or end-of-string not actually being considered a part of the match. Luckily java supports lookaround in regular expressions. We can revise our program to use lookahead/behind for the start/end and have only the desired bit be the match:

import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class RegexTest {
 public static void main(String[] argv) {  //
  //1+ CAPS preceeded by a CAP and ending in either another CAP or end-of-string. 
  String expr = "(?<=[A-Z])[A-Z]+(?=[A-Z]|$)";
  String[] testStrings = { "MyGoodName", "MYGoodName", "MyGOODName", "MyGoodNAME", "EndOfIT", "EndOfItALL", "UUIDIsACOOLType" };
  
  Pattern pattern = Pattern.compile(expr);
  for (String testString : testStrings) {
   Matcher matcher = pattern.matcher(testString);
   
   StringBuffer sb = new StringBuffer();
   while (matcher.find()) {
    matcher.appendReplacement(sb, matcher.group().toLowerCase());    
   }
   matcher.appendTail(sb);
   System.out.printf("%1$s => %2$s\n", testString, sb.toString());
  }
 }    
}

This will finally produce the desired output:
MyGoodName => MyGoodName
MYGoodName => MyGoodName
MyGOODName => MyGoodName
MyGoodNAME => MyGoodName
EndOfIT => EndOfIt
EndOfItALL => EndOfItAll
UUIDIsACOOLType => UuidIsAcoolType

Both the lookahead/behind and the appendReplacement/Tail are very handy but relatively rarely used in my experience.

Friday, September 10, 2010

The leaking of the JAXB

A novel memory leak was discovered in some of our software today. Apparently JAXBContext will create new dynamic classes each time newInstance is called. This means if you naively implemented a JAXB-based XML translation, similar to the example below, you would create new classes endlessly and run out of memory:

StringWriter sw = new StringWriter();
JAXBContext jaxbContext = JAXBContext.newInstance(myObject.getClass());
Marshaller marshal = jaxbContext.createMarshaller();
marshal.marshal(myObject, sw);

The unfortunate thing is that this is exactly what one would come up with after skimming the examples in the javadoc.

The recommended solution on the internets is to cache the JAXBContext, including re-using it on multiple threads. This is a bit novel as the documentation for the JAXBContext doesn't indicate this will happen and it also doesn't indicate the JAXBContext is thread-safe. To discover that you have to go into specific implementation information, such as https://jaxb.dev.java.net/guide/Performance_and_thread_safety.html.

These details vary by implementation so one might legitimately argue the javase documentation is correct. It's really just aggravating to discover that the implementation is built in a way so likely to cause clients to leak. This strikes me as a great example of a very bad implementation of an API, especially given that the implementation could easily maintain a lookup and avoid re-creating the dynamic classes and creating the leak for us.

Thursday, June 24, 2010

One-liner for errors per hour in a rolling log4j log

Because I keep forgetting valid syntax and having an example is very very handy:



for h in $(seq 0 23); do hh=$(printf "%02d" $h);  echo $hh $(grep "^2010-06-24 $hh.*general exception" some-dir?/ALogFile?.log* | wc -l); done;

Monday, June 7, 2010

So you smashed your macbook screen and want to use an external monitor as primary

After destroying the screen on my macbook I got to thinking the machine should work fine with an external monitor. The system would boot fine and display an exciting kaleidoscope of colors but aside from the monitor problem it appeared to be working. On connecting an external monitor I found that it does indeed start using that monitor. Now the only problem was that everything likes to start on the main monitor and when you can't see that monitor it is challenging to move things onto the secondary display.

Searches for keyboard shortcuts to move things between screens or ways to disable the primary monitor on startup didn't turn up much. Simply searching for the actual problem "macbook monitor broken" turned up a thread titled "Macbook screen broken, help w/ external monitor" (here). This proposed use of "clamshell mode" to get around the problem. This sounded perfect. Unfortunately, after getting a keyboard and mouse from IT the Macbook just went to sleep every time I closed it. Imperfect clamshell.

More googling revealed (eventually) that the Macbook only works in clamshell mode when connected to power; no battery powered clamshells, as described on both Forever For Now (here) and an Apple support page regarding clamshell (here).

Luckily next time I smash a Macbook screen I will remember that it happened to me before and I wrote down how to deal with it!

Wednesday, April 7, 2010

Watir 102: Javascripting XPCOM to remove session cookies in a FireWatir script

Yesterday (see previous entry) I was playing with FireWatir and found that I needed to remove session cookies to ensure a clean starting point for my script. XPCOM allowed me to read back cookies and it seemed very likely that it would also facilitate deletion.

The Mozilla Developer Center cookies examples were very close to what I needed. nsICookieManager.remove seemed perfect. In a FireWatir script it looked like what I needed to read/write cookies was something like this:

def removeCookie(brwsr, host, name, path)
   cmd = "Components.classes['@mozilla.org/cookiemanager;1']"
   cmd += ".getService(Components.interfaces.nsICookieManager)"
   cmd += ".remove(\"" + host + "\", \"" + name + "\", \"" + path + "\", false)";
   puts "Removing cookie " + name + " from " + host + " for path " + path
   result = brwsr.execute_script "#{cmd}\n"   
end

def readCookies(brwsr, site)
   cmd = "var uri = Components.classes['@mozilla.org/network/io-service;1'].getService(Components.interfaces.nsIIOService)"
   cmd += ".newURI(\"" + site + "\", null, null);"
   cmd += "Components.classes['@mozilla.org/cookieService;1']"
   cmd += ".getService(Components.interfaces.nsICookieService)"
   cmd += ".getCookieString(uri, null);"
   result = brwsr.execute_script "#{cmd}\n"   
end

In my initial attempts to use this I passed in "http://myhost" and although the method succeeded subsequent logging of the updated cookie string showed no change at all. After some fumbling around I discovered a detailed example of manipulating cookies through javascript in an extension. The main difference in their example was no protocol:// in front of the host/domain name. After that it worked perfectly. A simple example follows:

require "watir"

protocol = "http";
host = "mytesthost";
initial_path = "/somethingfuntotest"

site = protocol + "://" + host

def url(brwsr)
   return brwsr.js_eval("document.location.toString();")
end

def readCookies(brwsr, site)
   cmd = "var uri = Components.classes['@mozilla.org/network/io-service;1'].getService(Components.interfaces.nsIIOService)"
   cmd += ".newURI(\"" + site + "\", null, null);"
   cmd += "Components.classes['@mozilla.org/cookieService;1']"
   cmd += ".getService(Components.interfaces.nsICookieService)"
   cmd += ".getCookieString(uri, null);"
   result = brwsr.execute_script "#{cmd}\n"   
end

#Ref https://developer.mozilla.org/en/nsICookieManager
#Ref http://www.springenwerk.com/2007/07/using-cookies-with-firefox-extensions.html
#Usage removeCookie(brwsr, "myhostname", "SessionIdCookieName", "/")
#Note that passing http://myhostname for host does not appear to work
def removeCookie(brwsr, host, name, path)
   cmd = "Components.classes['@mozilla.org/cookiemanager;1']"
   cmd += ".getService(Components.interfaces.nsICookieManager)"
   cmd += ".remove(\"" + host + "\", \"" + name + "\", \"" + path + "\", false)";
   puts "Removing cookie " + name + " from " + host + " for path " + path
   result = brwsr.execute_script "#{cmd}\n"   
end

Watir::Browser.default = "firefox"
brwsr = Watir::Browser.new

#May have a session id
puts "Cookies for " + site + ": " + readCookies(brwsr, site)
removeCookie(brwsr, host, "JSESSIONID", "/")
#Session id is now gone ... yay!
puts "Cookies for " + site + ": " + readCookies(brwsr, site)

Tuesday, April 6, 2010

Watir 101: Javascripting XPCOM to read cookies in a FireWatir script

Today I ran into the need to automate a web UI built primarily of html and javascript. HtmlUnit based solutions seemed nice but Watir seemed like an ideal solution as it would necessarily produce the exact same results a real user would experience.

First step was to get setup, following the instructions at http://watir.com/installation/#win. Despite my complete lack of experience with both Ruby and Watir things went pretty smoothly. My setup sequence was:

  1. Install Ruby (http://rubyforge.org/frs/download.php/29263/ruby186-26.exe)
  2. At new cmdprompt:
    1. We need a new command prompt because one open from before the Ruby install will not notice the updated environment variables and thus will not have ruby on PATH
    2. Note that the second command is slow as hell (at least it was for me)
      gem update --system
      gem install watir
      
  3. Install FF JSSH plugin; ref http://watir.com/installation/#win

  4. Poke around a couple of helpful documents to try to learn the basics

  5. Run scripts similar to ruby my_script.rb

Scripting the basic interaction went pretty smoothly. Obtaining a Firefox browser and jumping around is every bit as straight forward as you might hope. Obtaining the current url is slightly more problemmatic than one might help but luckily others have solved this problem already. Here is an example of a simple script:
require "watir"

site = "http://my.test.host"
initial_path = "/blah"

def url(brwsr)
   return brwsr.js_eval("document.location.toString();")
end

Watir::Browser.default = "firefox"

puts "Requesting a new " + Watir::Browser.default
brwsr = Watir::Browser.new

puts "Loading " + site + initial_path
brwsr.goto site + initial_path
puts "At " + url(brwsr)

puts "Selecting and navigating forward"
brwsr.radio(:name, "inputRadio").set
brwsr.button(:name, "Continue").click
puts "At " + url(brwsr)

The only real problem with basic scripting functionality I ran into (as yet unresolved) is that if something takes longer than Watir likes to wait (say the browser is slow to pop or a page is slow to respond) the script gets ahead of the browser and gets severely confused.

So now for the first real problem I ran into: Cookies. The application I want to test is session-aware so ultimately I will want to nuke cookies at the beginning of the test. First step, lets see if we can get cookie information back. After a bit of searching it turns out manipulating cookies from Watir is less simple than one might hope. For IE people seem to simply use Ruby to delete the files out of the filesystem (see FAQ). For Firefox we can alter cookies via XPCOM, Mozilla even has an example.

This is all well and good but how can we execute javascript against XPCOM objects from Ruby code running FireWatir?? Luckily I found a great series of articles that were almost what I wanted:

The only problem was that the example called for rebuilding some of the code ("I opened up the Firewatir::Firefox class and added...") for FireWatir. This would definitely be stretching my fledgling Ruby/Watir skills so I went looking for a simpler approach.

The browser object for Firefox (FireWatir::Firefox) exposes the wonderful execute_script method. The description "Executes the given JavaScript string" sounds pretty promising. A quick check of the source firefox.rb file showed that the method is a thin wrapper around the same JSSH interaction sequence that is used in the http://sticklebackplastic.com articles. This all seemed almost too good to be true so I decided to see if it actually worked the way it appeared. Luckily it works just as you would hope; the following example shows Ruby code reading cookies for whatever domain it happens to be interested in:

require "watir"

site = "http://my.test.host"
initial_path = "/blah"

def prepare(brwsr, site)
   puts "Preparing..."
   cmd = "var uri = Components.classes['@mozilla.org/network/io-service;1'].getService(Components.interfaces.nsIIOService)"
   cmd += ".newURI(\"" + site + "\", null, null);"
   cmd += "Components.classes['@mozilla.org/cookieService;1']"
   cmd += ".getService(Components.interfaces.nsICookieService)"
   cmd += ".getCookieString(uri, null);"
   result = brwsr.execute_script "#{cmd}\n"
   
   puts "Cookies for " + site + ": " + result
end

Watir::Browser.default = "firefox"

puts "Requesting a new " + Watir::Browser.default
brwsr = Watir::Browser.new

showcookie(brwsr, site)
showcookie(brwsr, "http://www.bing.com")
Hopefully a similar mechanism will allow me to alter cookies but that's as far as I've made it so far. Seems quite promising! And there are lots of cool names (Watir, Ruby, JSSH, XPCOM, ...) involved, always a plus!

I was very impressed that starting with no knowledge of Ruby or Watir I was able to get everything installed and create a functional proof of concept script (albeit one very low on validation or error checks) for a semi-complex sequence (add item to cart and checkout) in a couple of hours! If I can get cookie removal working to ensure a clean start state this could grow to be a very valuable verification tool for our developers.