bad.robot

Java source on Mac

2011-10-29T10:11:00.000Z

Mostly as a reminder to myself, getting the Java source on your Mac involves the following.

Go to the Apple Developer Connection downloads page, search for Java for Mac OS X 10.x Developer Package where 10.x matches your version of OS X. The developer bundle includes the source whereas the regular software update version does not.
Download and install. Running /Applications/Utilities/Java Preferences.app should now show "Java SE 6 (System)" in the list.
Open a Terminal.app window
cd $JAVA_HOME (aka /System/Library/Frameworks/JavaVM.framework/Home)
Setup a symlink to the source archive with sudo ln -s /Library/Java/JavaVirtualMachines/1.6.0_26-b03-383.jdk/Contents/Home/src.jar
And for the JavaDoc, sudo ln -s /Library/Java/JavaVirtualMachines/1.6.0_24-b07-334.jdk/Contents/Home/docs.jar
Now point your IDE of choice to the new source folder symlink.

Any update to Java will set things up to point to Maven 3, so if you use Maven 2, it'll break things with
java.lang.NoClassDefFoundError: org/codehaus/plexus/classworlds/launcher/Launcher Reset things by;

cd /usr/share
sudo mv maven maven.new (a symlink which should incorrectly be pointing to java/maven-3.0.3)
sudo ln -s /maven2/install/folder maven
run maven -version to check its back up.
Have a cup of tea.

Reflecting on Interviewing Mistakes

2011-08-29T12:30:00.000Z

Recruiting for the next guy on your team is hard. At first glance it doesn't seem to be, we've developed techniques like pair tests but as I start to look at it more closely, I've started to notice that even the more progressive techniques don't preclude us from making the same mistakes as the traditional interview.

Let's take an example from two teams...

Team A's process starts off by favoring buzz word heavy CVs and CVs that meet a minimum number of years of experience. A unattended pen and paper test, characterised by very closed questioning against specialist areas of the programming language. This might include questions around language syntax semantics (keywords and modifiers, object equality etc etc). Things like bubble sorts algorithms are requested. Scores out of 100 are tallied. Things are fairly black and white.

Team B's process favors mention of agile experience in the CV. Follow up questions prompt genuine conversation but often an implied hurdle that the candidate must jump is "has he reached the same conclusion as me on topic X?". The unattended coding exercise is not a test, at least it should be more of an exercise to explore the way a candidate approaches things. The team might require the presence of unit tests and evidence of TDD but should actively not persecute style or syntax. Something that's harder in practice to do than in theory.

Hopefully, its clear that Team A's selection process is heavily biased towards developers with good memories. It's probably unfairly prejudice against candidates that haven't had specific exposure to specific scenarios / solutions. I experienced this when I was asked to write a algorithm to calculate prime numbers with pen and paper. I fumbled through and handed over my scrawl. I explained that I'd prefer write tests, experiment with the code and improve the design; basically to learn as I went along. The response from the interviewer, looking down at my scribbling, was "that's not really what we're looking for... have you heard of the Sieve of Eratosthenes?". Obviously, I hadn't.

Rather than assess my approach, the interviewer was looking for a specific piece of knowledge but what for? If I got the job I'm pretty sure my first task wouldn't be to write something to work out prime numbers. Would that fix some production problem? Would it introduce a new feature that had no other solution? No.

A huge part of what we do is learn, or at least it should be. Failure is what makes us better and in environments where failure is embraced and we write code that we can (fairly) easily rework, we get better systems (as we refine our understanding). We never now what the real problems are going to be when we start a story. The interviewer above simply brushed over this, it seemed he wanted me to reach the same conclusion he had without explaining the steps I took to get there. Without any advocacy on my part, how would he know I could do it again with a different problem?

"Right or wrong answers don't really have a place because there's never a right or wrong answer in what we do."

Having said all that, I'm sure we'd all favour a process like Team Bs but I'm starting to see that Team B are making at least some of the same mistakes just in a more subtle way...

For the CV selection, Team A look for "spring", "hibernate" and other technology buzzwords. Team B look for "refactoring", "TDD", "XP" and other development buzz words, the reason usually cited as being because the technologies aren't as import. Team B are favouring the why over the how, they're assuming given the right approach and smart people, specifics around technologies can be learnt. Both teams are trying to expose characteristics of the candidates that mirror their own.

Team B asks candidates to complete a short programming exercise off-line. Implement a library, a DVD store, a robot explorer, whatever. It should only take an hour or so and demonstrates the candidates style. I've certainly seen it as an effective tool to eliminate people that really can't code for toffee but I've also seen people fall into the same old trap and eliminate people who missed something specific hidden there. A trivial example might be "oh! they didn't use dependency injection. Fail!".

Team B's pair test should be a great way to understand how a candidate operates in front of an IDE and if you'll actually be able to work with him. A bit like the unattended test, it's a good way to eliminate extreme cases. If the candidate behaves completely anti-socially, wont listen and codes like mad man, you can probably reject him with confidence. It's easy to let bad interview habits creep in though; to focus more on some obscure gotcha in the code than how the candidate is actually pairing.

"I think the problem with both these techniques (unattended exercise and the pair test) is when too much specificity comes in at the start. When you are looking for something specific, you'll often be disappointed."

I've certainly heard myself say "oh, he didn't spot that there was a precision issue with double there...". In all honesty, I'd miss that kind of bug as often as I'd spot it but I'd hire me! The upshot there, especially when we doing a couple of pair tests a week, is to stay focused on why you're doing the pair test and not on the test itself. Are we doing this to see if the candidate can spot all the traps and pitfalls that we spent so long putting in or do we want to see how they pair? In my view, if they get the "right" answer is almost irrelevant, it's how they explore the problem.

I guess what I'm reflecting on here is how as a peer group, we pretty much realise that closed questioning limits our choices and that open ended questions lead to real conversations that are more relevant to the types of conversations we have day to day. Right or wrong answers don't really have a place because there's never a right or wrong answer in what we do. If I implement a prime number finder without the Colander of Eratosthenes, am I wrong? The tests still pass so I must be right? Is Eratosthenes more right? Despite this realisation though, we can easily fall into a more subtle way of behaving where we mentally start ticking off specifics for a candidate.

I guess we have to keep reminding ourselves what's important and what we're looking for in a candidate. I guess I'm mellowing in the way I assess candidates and probably rejected a fair few unfairly in the past. Sorry.

Catching Multiple Exceptions (and rethrowing them all!)

2011-08-29T12:09:00.001Z

Sometimes, we may want to catch an exception, temporarily ignoring it to continue work before rethrowing it when its more appropriate to do so. I recently saw a slight variation of this whereby the developer wanted to (potentially) catch multiple exceptions, perform some processing then throw. However, it left the question that if more than one was caught, which exception should we actually rethrow. We certainly don't want to loose any information and should really allow the client to catch the exception in a standard way.

This got me thinking about how we should deal with this kind of thing. In the end, I came up with the idea of a collection class to capture the Exceptions and a sub-class of Exception to represent an exception containing other, embedded exceptions. When you're done collecting exceptions, you can just check and rethrow as a new exception type.

For example, the domain cleaning class below can throw an exception during the deleteAll method. Rather than abandon the cleanup of subsequent objects, we can employ this tactic to continue the cleanup and throw an exception containing the underlying problems when we're done.

public class DomainCleaner {

 

    public static void clean(Domain domain) throws CompositeException {

        Exceptions exceptions = new Exceptions();

        clean(domain.customers(), exceptions);

        clean(domain.suppliers(), exceptions);

        clean(domain.invoices(), exceptions);

        exceptions.checkAndThrow();

    }

 

    private static void clean(Repository repository, Exceptions exceptions) {

        try {

            ((TestRepository) repository).deleteAll();

        } catch (RepositoryException e) {

            exceptions.add(e);

        }

    }

}

We simply add to the exception collection class (exceptions.add(e)) and then when we're done, we can check it and throw a composite exception if needed with exceptions.checkAndThrow().

So far, we've only been interested in the fact that multiple exception can be handled and so haven't needed to programmatically query for specific exception types. For example, we've only needed this up until now.

try {

   // ... something that calls checkAndThrow()

} catch (CompositeException e) {

   // ... this is enough for now

}

The details of the classes are below.

public class Exceptions implements java.lang.Iterable<Exception> {



    private final List<Exception> exceptions = new ArrayList<Exception>();



    public void add(Exception exception) {

        exceptions.add(exception);

    }



    @Override

    public Iterator<Exception> iterator() {

        return exceptions.iterator();

    }



    public void checkAndThrow() throws CompositeException {

        if (!exceptions.isEmpty())

            throw new CompositeException(this);

    }

}

The toString() implementation below outputs the embedded exceptions in a way that is consistent with how you'd expect to see regular exceptions.

public class CompositeException extends Exception {



    private final Exceptions exceptions;



    public CompositeException(Exceptions exceptions) {

        super("composite exception was thrown with embedded exceptions (see details)");

        this.exceptions = exceptions;

    }



    @Override

    public String toString() {

        StringBuilder builder = new StringBuilder();

        for (Exception exception : exceptions)

            builder.append('\t').append(new ExceptionToString(exception).toString()).append('\n');

        return String.format("%s\n{composite exceptions=\n%s}\n%s", this.getClass().getName(), builder.toString(), super.toString());

    }

}

Loggin is still evil but...

2011-06-22T20:14:00.001Z

In a previous post, I was going on about how evil logging is. How it's often confused as a requirement and often badly misused. The upshot of the post was that if you're going to log stuff, in our case using Log4J, lets be honest about it and test it. We should be able to say upfront what's important to log, in what situations and at what log level. Sounds like a straight forward case of test first.

Mocking Log4J however can be a real pain. I've managed it in the past using Apache's logging abstraction and configuring it to use Log4J under the covers but in my previous post, I demonstrated a slightly easier way. A helper class called Log4J that we can use to represent the logging system and that we can make assertions against. Pretty cool so far.

There was one caveat, I wasn't entirely happy with the fact that the class would rely on your external Log4J configuration. To assert that a log message appeared at the level INFO for example, you'd have to make sure that the test environment sets up the appropriate class to log at that level. It made for a kind of integration / environmental test which in some cases might be a sensible test but for the most part, I kept seeing test failures down to configuration on different environments. Yuk.

So I updated the helper class to include a log level override which will ignore what the actual configuration says. This means you can write less brittle tests to say things like "ensure my log message is output at debug level regardless of the runtime configuration".

The updated class looks like this.

public class Log4J {
 
    private final StringWriter writer = new StringWriter();
    private final Logger logger;
    private final String uuid = UUID.randomUUID().toString();
 
    public static Log4J appendTo(Logger logger) {
        return new Log4J(logger, ALL);
    }
 
    public static Log4J appendTo(Logger logger, Level level) {
        return new Log4J(logger, level);
    }
 
    private Log4J(Logger logger, Level level) {
        this.logger = logger;
        WriterAppender appender = new WriterAppender(new SimpleLayout(), writer);
        appender.setName(uuid);
        logger.addAppender(appender);
        logger.setLevel(level);
    }
 
    public void clean() {
        logger.removeAppender(uuid);
    }
 
    public void assertThat(Matcher<String> matcher) {
        org.junit.Assert.assertThat(writer.toString(), matcher);
    }
}

Which means you can setup to expect a log level at say the ERROR level like this.

private final Log4J logger = Log4J.appendTo(Logger.getLogger(Post.class), LogLevel.ERROR);

The make assertions like this (which would fail if the matcher fails or because its not logged at the expected level.

logger.assertThat(containsString(EXCEPTION_MESSAGE));

I still think logging is evil and try really hard not to use a single log statement but if you have to, I hope the helper class helps keep you honest in your tests ;) Have a look at the previous post for more details and extended examples.

JDK7 Artcile in JavaTech Journal

2011-06-10T15:52:00.003Z

My article "Java the language vs Java the platform" (about the new release of JDK7) has been published in this months Java Tech Journal. You can download the full issue directly below.

I'd love to get some feedback, so please feel free to comment here.

tempus-fugit 1.1 released

2011-04-13T09:36:00.001Z

Yesterday, I released the 1.1 version of my micro-library tempus-fugit. From the project's website

The tempus-fugit library is a small collection of classes and interfaces capturing common abstractions useful when writing concurrent and time sensitive code.

It's now available from the Maven Central repository having had a bad experience with java.net since their migration (and no longer being able to publish, see this post and another if you're interested).

See the change list for what's included in this release.

JDK7 Previwed

2011-03-04T15:44:00.011Z

Oracle put out the preview release of JDK7 last month. I guess they felt they had to. So, it's not what was once heralded (will 8 see lamdas?) but still has one or two interesting language features. The ones that caught my eye include;

Type Inference on Generic Object Creation

Which allows a little brevity to the garrulity of the language, at least against generic object instantionation where the type can be infered. For example,

private Map<Size, List<Shoe>> stock = new HashMap<Size, List<Shoe>>();

can be reduced to

private Map<Size, List<Shoe>> stock = new HashMap<>();

where the diamond operator can be filled in or inferred from the declaration. It's subtly different than leaving out the generic completely which will reduce your type to being of Object Things don't get much better than this.

Actually, it does. Just a little. Constructor generics always used to be fun and that hasn't really changed, although with JDK7 you can do a little more. For example,

public class Bob<Y> {

    public <T> Bob(T t) {

    }



    public void example() {

        Bob<Integer> bob = new Bob<>("yum");

    }



    public void anotherExample() {

        Bob<Integer> bob = new <String> Bob<>("yum");

    }

}

The examples are the same as the one Oracle gives, they both work with JDK7 only and show the Integer type inferred as the class generic (Y) in combination with the diamond operator. The second example shows new syntax to explicitly set type of the method generic and give some additional compile time checks.

try-with-resource and `AutoClosable`

Another bugbear with the verbosity of Java has always been the try-catch-finally syntax. The new language feature try-with-resource allows you to chop this down some what in combination with auto-closable resources. Here, rather than the familiar, try-finally to close a resource, you can "open" the resource within the parenthesis of the try statement (as long as the object implements AutoCloassable and the resource will always close itself in a finally like way.

For example,

private String example() throws IOException {

    BufferedReader reader = new BufferedReader(...);

    try {

        return reader.readLine();

    } finally {

        reader.close();

    }

}

gets replaced with

private String example() throws IOException {

    try(BufferedReader reader = new BufferedReader(...) {

         return reader.readLine();

    }

}

Dr Heinz combined this technique with a way to automatically unlock locked resources in a recent news letter.

There may be a little gotcha using this where exceptions can be suppressed and have to be retrieved using Throwable.getSuppressed(). This seems like it could be nasty.

Catching Multiple Exceptions

This one allows you to catch multiple exceptions using a pipe to separate the exception types. This looks like another work around for the general grips with Java but removes the duplicated code you often get catching several exceptions and treating them in the same way. For example,

catch (IOException | SQLException e) {

    logger.log(e);

    throw ex;

}

I usually end up pushing the code to execute as a Callable and dealing with the exception in a lamda-like piece of code, or decorating the fragment to deal with exceptions (logging or wrapping) or even trying really hard to throw around runtime exceptions, so this one is at odds with my general approach. Given the example from Oracle above, I suspect this will just facilitate ugly, jammed in code. It seems to say "it's ok to deal with a bunch of exceptions in the same way. in fact, we'll make it easier for you" without any warning about if you actually should be doing this type of thing. The fact the example above (Oracle's example, by the way) logs then re-throws is a smell in it's self. Perhaps I'm being premenstrual, but I'm not a fan of this one.

Have a look here for on the new features and download from here (unfortunately, not for the Mac).

UPDATE: An extended version of this post has been published in May edition of the JavaTech Journal.

Logging is evil but...

2010-10-18T19:51:00.005Z

Logging is a nightmare. I don't mean here that conveying information about exceptional circumstances is a nightmare, I mean the combination of over eager developers and [insert your current logging framework here] is a recipe for disaster. We've all seen too much of

Logger log = Logger.getLogger(ThisSucks.class);

    ...

    try {

        somethingRisky();

    } catch (SomethingVeryBadException e) {

       log.error(e);

       throw e;

    }

which is just one example where the exception handling policy for the system (it's a system-wide concern remember) is muddled at best. Nothing is saying that the same exception isn't logged elsewhere or that the exception is even handled correctly or the right people notified. It's not ok to just log and rethrow and every single time we go to declare a new logger, we should think twice.

We've taken this very literally in my current project and everyone is actively discouraged from instantiating a logger. I'd rather be explicit that some exception event has occurred and fire an event that some interested party can listen for. This makes perfect sense when you think about the huge log files that someone has to trawl through, armed only with for some vague clue as to what went wrong, a grep manual and the futile hope that developers actually log something useful. All without the context of the code to actually guide them. Good luck.

The disseminated log problem is exacerbated if there is no clear audit trail tying pieces of information together. In a system with thousands of request per second, how do you tie the logged request inputs to some stack trace embedded in the middle of another thousand requests? What should have been a clear set of requirements from the business (in this case, presumably the support team) can easily get lost in the technical translation.

Logging is evil, but if I really *have* to log, be honest about it...

Asking the business "what information do you want to see in the event of x happening" rather than assuming they want to see some stack trace in a huge log can make a lot of sense. We're often not logging for ourselves (we have debuggers for that), we're often logging for our customers. If we start to think about this stuff early, in terms of exception events and their audience, we can build systems that tell the outside world something meaningful in flexible ways. We start to define a system wide exception handling policy rather than relying of the default exception handler (System.out is rarely the right choice!).

So back to my current project... people are regularly beaten with a chair leg for creating loggers but I'll admit that on occasion, I've actually logged stuff and didn't resort to some Opus Dei style self-flagellation. Logging is evil, but if I really *have* to log, my saving grace is to be explicit about it. I'll hunt down a customer and I'll write a test to advertise the fact the log contains what they asked for.

Most of the common logging frameworks make it troublesome to inject a logger instance, and I'm reluctant to subvert behaviour just because some logging framework wants me to. Logging (or preferably, firing an event) should be orthogonal to the classes core behaviour, why should I compromise? My preferred approach is the canonical example of using Aspects, or less esoterically, using decorators.

For example, I created a interface to handle HTTP POST requests, imaginatively called Post. Why should I add logging to implementations and open the door to ad-hoc, erratic logging? I shouldn't, but when my implementation CustomerPost requires logging of the request and response, I can decorate with a LoggingPost

public class LoggingPost implements Post {



    private static final Logger LOG = Logger.getLogger(Post.class);    

    ...



    public LoggingPost(Post delegate) {

        this.delegate = delegate;

    }



    @Override

    public Response post(Body body) {

       try {

           return delegate.post(body);

       } catch (IOException e) {

           LOG.error(e.getMessage(), e);

           throw e;

       }

    }

}

You might be concerned that the try/catch above looks very similar to the original negative example. The good thing about our decorated example above is that by being explicit about this classes responsibility, declaring the usage in the correct context, we can actually define the system wide policy for logging the Post calls in one place, without affecting the contract of the interface. We'd do this for example, on the system boundary, for example where the RESTful API is implemented.

@Resource

public class CustomerServlet {



    public void doPost(Request chuck, Response up) {

        ...

        customer = new LoggingPost(new CustomerPost(...));

        customer.post(...)

        ...

    }

}

In our LoggingPost above, we haven't even tried to inject a logger in to make the testing easier. Instead, mostly because I was being lazy, I used the helper class below. This is intended to represent Log4J in the context of a test and give access to the logger for assertion purposes.

public class Log4J {

 

    private final StringWriter writer = new StringWriter();

    private final Logger logger;

    private final String uuid = UUID.randomUUID().toString();

 

    public static Log4J appendTo(Logger logger) {

        return new Log4J(logger);

    }

 

    private Log4J(Logger logger) {

        this.logger = logger;

        WriterAppender appender = new WriterAppender(new SimpleLayout(), writer);

        appender.setName(uuid);

        logger.addAppender(appender);

    }

 

    public void clean() {

        logger.removeAppender(uuid);

    }

 

    public void assertThat(Matcher<String> matcher) {

        org.junit.Assert.assertThat(writer.toString(), matcher);

    }

}

Using it in the test for LoggingPost is shown below

@RunWith(JMock.class)

public class LoggingPostTest {    

    

    private final Mockery context = new Mockery();

    private final Post mock = context.mock(Post.class);

    private final Log4J logger = Log4J.appendTo(Logger.getLogger(Post.class));

 

    private static final String EXCEPTION_MESSAGE = "bar bar black sheep...";

 

    @Test

    public void shouldDelegate() throws Exception {

        context.checking(new Expectations() {{

            one(mock).post(...);

        }});

        new LoggingPost(mock).post(...);

    }

 

    @Test

    public void shouldLogWhenExceptionIsThrown() {

        try {

            postWill(throwException(new IOException(EXCEPTION_MESSAGE)));

            new LoggingPost(mock).post(...);

            fail();

        } catch (IOException e) {

            logger.assertThat(allOf(containsString("ERROR"), containsString(EXCEPTION_MESSAGE)));

        }

    }

 

    @After

    public void cleanupLogger() {

        logger.clean();

    }

 

    private void postWill(final Action action) {

        context.checking(new Expectations(){{

            allowing(mock); will(action);

        }});

    }

}

It relies on nasty statics to dynamically add a logger to Log4J's list of loggers and thereby appending any generated logs to something that the Log4J test helper can assert on. I can't decide if I like this or not. It gives you an extra test that your class under test is using a logger with the name that you expect ("Post.class" in the example above), testing your logger configuration as a by-product.

What I found interesting about this though was that it was always seemed a lot of effort making some logging framework play nicely with mocks, or writing and configuring a custom in memory appender and asserting on it. With the above example, I very quickly added confirmation to existing Log4J infrastructure. It seemed almost too easy... so I'd love to hear your comments and how you write tests for logging.

PS. Logging is evil.

Pairing Honestly

2010-08-15T10:39:00.006Z

Recently we had particularly good retrospective where the team were able to admit that each of us has had difference experiences pairing. We were honest in saying that despite having "done pairing" we'd all done different amounts of pairing and that, at times, we weren't even sure what we were supposed to get out of it. There can be a fair amount of peer pressure to pair but if the pair don't know what they can get out of it, it's unlikely to succeed. We should be honest about that. What makes a good pair (see a previous post) and how do we know that we're getting something out of it?

Once we honest in our experiences and expectations around pairing, we were able to be explicit in what we want to achieve. We had the conversation along the lines of "do we even believe we could get something out of pairing? Do we want to try?" and when the team bought into that idea we could be explicit about starting from scratch and putting tools in place to help us.

Some techniques I favour include

when sitting down to start a session, stating explicitly what each individual hopes to get out of the session and what isn't helpful. In short, what they think a pairing session should be like.
each pair can set up one or two "rules", written on cards and placed prominently to remind each other about something they feel is important to the session. For example, I often have a rule "ask don't tell" to remind me to be asked before I go rambling on about something in the code.
the silent partner; saying up front that when not driving, the silent partner should hold their tongue and make notes / record a task stack. Interruption is interruption, so be considerate about when to make a comment. This one should be agreed up front, it may or may not be appropriate for your pair.
most important one of all; have a mini pair-retrospective at the end of the session. It's a great chance to ask the question "how was that for you?" and a great way to let the other person know if some aspect didn't go well.

Another useful tool can be the pair stairs. It's very easy to avoid pairing rather than face it's challenges so pair stairs can be a good way to keep you honest. Probably the most helpful thing though is having people on the team that have lots of experience pairing, know what the team can get out of it and are able to help guide pairing sessions for the less experienced.

Pairing is hard. It's probably one of the hardest things we do as developers but it can also be one of the most rewarding, personally and for the team as a whole. Collaboration has given the world some of its greatest achievements but you have to be honest that in this case, you might have to work for it.

Lamdas vs. Closures

2010-07-13T22:21:00.001Z

When writing Java in a functional style, apart from the verbosity of it all, it always bugged me about the terminology we use. I tend to talk about closure-like arguments but revisiting some old University materials when clearing out the loft, I've adjusted my vocabulary somewhat. Taking the WaitFor class from tempus-fugit as an example, passing an anonymous class instance as a parameter to a method that will later call the instance is a kind of functional programming. I say kind-of because its not really functional programming, Java isn't a functional language but we can bend it into a style that's similar. For example,

...
server.start();
waitOrTimeout(new Condition() {
   @Override
   public boolean isSatisfied() {
      return server.isRunning();
   }
}, timeout(seconds(5)));

The annonymoose class implementing Condition is evaluated by the method waitOrTimeout (it's that which will call the isSatisfied) method (an annonymoose class is a type of Moose that keeps himself to himself).

The recent shift to this functional style has lead to eager anticipation of JDK7 and the promise of closures. More accurately however, it's the inclusion of lamdas that we're waiting for, not closures. Closures have in fact have been available in Java since 1.1, so what's the difference?

Lambs to the Slaughter

So, we want to be able to define anonymous functions on the fly, the result of the function is purely dependent on it's arguments and this is called a lamda. Those functions that depend on external values (not just it's arguments) are when closures come into it. The act of binding those external values to the anonymous function is referred to as closure. After closure, when all variables have been captured and bound to the function, the term is closed.

For example, the code snippet above will return a new Condition instance on each invocation. Because it will bind the variable server to the anonymous function, it will return a closure. To put it another way, we'll extract the anonymous part to a method to explicitly create a new instance, such

private static Condition isRunning(final Server server) {
   return new Condition() {
      @Override
      public boolean isSatisfied() {
         return server.isRunning();
      }
   };
}

This should make it more obvious that the variable outside the scope of the anonymous Condition is required (the server variable), each call to the isRunning method will return a closure over the argument, the instance of which captures the value of server. Java implements the closure by passing a reference to the outer scoped (lets say Foo.class) to the anonymous class (Foo$1.class). The access$000 call accesses the appropriate private field in the outer class directly as bytecode

class Foo$1 implements Condition {

    final Foo this$0;

    Foo$1() {
        this$0 = Foo.this;
        super();
    }

    public boolean isSatisfied() {
        return Foo.access$000(Foo.this).isRunning();
    }
}

So, if, we have update the example again, this time removing the out of scope variable, we're left with something like this;

private static Condition isRunning() {
   return new Condition() {
      @Override
      public boolean isSatisfied() {
         return true; // optimistic!
      };
   }
}

Then no out of scope variables are required, the term doesn't need to be closed. The annonymouse function that is left is effectively a lamda (not a shy Mouse).

What JDK7 will (finger's crossed) bring is more explicit, concise way of expressing the same ideas. It will support lamdas as a language feature although I can't quite figure out what the example would look like in those terms. See the straw man proposal and see if you can figure it out!

Growing Team Skills

2010-07-11T16:15:00.000Z

I love the idea of growing teams, the lure of the gardening analogy is just too strong... they have to be nurtured to flourish blar blar blar. What is obvious though is that team dynamics are really important and we have to be conscious of how we work together if we want to be able to reflect on how well we, the team, are doing. It's useful to understand the strengths and weaknesses of the team, together and individually so that we can a) improve and b) match people to tasks, and teams to projects. It all contributes to a happy working environment.

Often, new or junior developers worry about the barrage of new and unfamiliar technologies. Technologies are such a small part of what we do in my opinion but it can be useful to be explicit about individual competencies to re-assure the bedazzled new developer. In this post, I present a tool I came up with to help, a kind of competency depth distribution chart (that makes a nice palindrome so it's a good a name as any).

It works like this;

As a team, define the technologies that are important / exotic / of interest. Rank them by importance. Try to limit to around 5-10. Start the distribution chart, along the Y axis you'll list the technologies, ranked by importance and on the X, the Dreyfus Scale of Competency. The team must agree on the ranking of technologies and agree on their understanding of the Dreyfus scale.

Next, each team member grades themselves on the scale for each technology. It's important to be honest. The team can then agree and plot the desired competency in each area on a separate sheet (the black line in the example below). Each individual can then see immediately their delta and use it to focus their personal development.

If the team choose to come together and share their individual charts, they'll be able to see how the team compares to their minimum requirements. It might be that some people are strong in some area and others less so. In this case, there's an obvious knowledge sharing opportunity and it might be that the urgency to close the delta changes. For example, if there is already an expert in some technology on the team, the others may not need to stress. Hopefully, you can start to look at the needs of the team and not the individual. Working through this as a team can also help build trust which is vital to healthy plants, sorry, teams.

If the Dreyfus scale seems a little too harsh, you could always try a scale to reflect general confidence in the area. This tool doesn't have to be about technologies, it can be about any competencies that you want analyse.

Changing Test Gears

2010-07-09T17:55:00.002Z

Good poker players know when to change gears. They know when to alter their playing style from cautious to aggressive as the game changes and players drop out. They look at how the odds change as the game progresses and react appropriately. It's the same with testing, you gotta know when to change gears.

To put it development terms, good developers know when to change gears. They know when to change their testing style from cautious to aggressive as the code evolves.
Lets pretend there is just three types of testing; unit, integration and acceptance. In the interest of stereotyping, we'll define them simplistically as

Unit - single object tests, no collaborations (strict I know, but bear with me)
Integration - testing object collaborations, for the purposes of this article, lets assume end-to-end testing slot into this bracket
Acceptance - leaning towards end-to-end but key here is that they are customer authored. As such, to convince the customer these will likely be relatively coarse grained and start outside the system boundary

Starting in a low gear with unit tests

People are probably most comfortable with this type of testing. The term unit testing and the technology JUnit have become so intertwined in the Java world, that people often confuse tests written with JUnit as unit tests. They may be, but they may not be. So where's the value in defining the term unit testing and how does knowing what type of test you've just written in JUnit help with changing gears?

Knowing what gear you're in and knowing the terrain that’s coming up is essential for you to select the right gear. Writing a non-unit test in JUnit has value, of course it does (assuming it actually tests something) so why should I care if it’s a unit test or a chazzwazzer? Knowing where you are and where you want to be is useful because you can defer some things and avoid duplication. So for me, unit testing is good for testing the edge cases (Right BICEP) and exploring the class you're writing. It can be especially useful when you test drive towards something or explore relationships and your understanding of the classes roles and responsibilities. In this sense, testing becomes a design activity or an analysis activity, a chance to phrase your thinking and understanding in code. The regression element can quickly lose value here. For example, writing a test with mocks to explore the interaction between two collaborators, A and B. Then writing a separate test to explore the same for classes B and C. Then a test for A, B and C. There was value in each test individually, but is there still value in all three when there is obvious cross-over? When might I consciously choose not to write unit tests?

Changing up to integration tests

If I know the context that a set of objects are going to work together in, I'm going to want to be confident that they work together as expected. I can't test these in isolation, so I'm going to need to test them in cooperation. At this level, my confidence is fairly high around the composites so I'm already up and running, I'm more concerned here with a broader brush approach. I'm certainly not interested in re-testing all the lower level object tests, just how they operate together. So I change up a gear and as a developer, convince myself that these object work together in context. I'm most likely still using JUnit but I understand what gear I'm in.

Cruising with acceptance tests

So how about the value to the business? The unit tests in particular don't advertise value to the business, they're a developer tool and its all too easy to write individual classes well with good test coverage and yet combine them into something that doesn't work. Demonstrating to business that their specifications have been meet is the ultimate gear, and to change up to that gear and have overall confidence in the system means going through the previous gears to get there. Knowing that you'll be changing up whilst in lower gears can help you decide what to do in those lower gears.

For example, lets assume the business want to see the affect of a configuration file in the system. When developing the code to load and decode the contents of the file, you started by proving the component works at the unit level so why would you test that the component is wired up correctly as a unit test? You'd be forced to mock that component in some higher level component and test it calls it. But what does this give you? You can still wire the higher level component up incorrectly. The acceptance test is going to have to test this to demonstrate the affect, so there's an argument to say you can leave it to the acceptance test to verify. This might have the added bonus of avoiding "breaking up" the higher level component albeit a minor point.

The missing gear?

When we build software in a componentised way, we're often left with objects that work in isolation with their dependencies passed in. We push up and up the assembly of the objects and their ultimate iterations and are left with parts of the system that are responsible for this assembly. Hopefully this is done declaratively in code but it might be some dependency injection framework is involved. Either way, it feels like these 'assemblers' have a clear role and responsibility and so shouldn't they be tested? What gear do we test them in? Personally, I'm comfortable in limiting the assembly options and so reduce the combinations of testing required. With a single "builder" or "spring context", I'm comfortable with testing these through acceptance tests.

In this article, I suppose I'm using "gears" as an analogy for pragmatism and certainly not pace. I'm not saying that good developers know when to rush, compromising quality but they do know how to optimise their testing strategies. I think its important not to get bogged down in exhaustive unit-style testing if its not of value and so understanding what gear you're in and what gear you'll soon be in can help focus your attentions. What to test and when is the question I find myself asking again and again.

Generate Concordion Overviews

2010-07-07T21:01:00.000Z

Creating customer authored acceptance tests is awesome! I love it. Getting your users to tell you exactly what they want (and don't want) in a form that they can (ghost) write can make for a world that even the Care Bears are jealous of.

I can't comment on some of the BDD targeting frameworks like JBehave, EasyB or Cucumber but I do like using Concordion. We try and use it in such as way to fit in with a BDD approach, its flexible enough to use in almost any approach in fact. That's probably why I like it so much. It's frequently cited closest comparator is probably Fit but that's a little unfair. As I say, Concordion tries really hard not to tie you into a particular approach, so to compare it against Fit (which leans towards the inflexible in my view), doesn't serve as a fair comparison.

Anyway, the point to this post isn't really to comment on Concordion but to advertise a little Ant task I wrote to help auto-generate Concordion-friendly summary pages for your existing Concordion tests.

It's purpose is to collect your Concordion tests as an "overview page" which itself is a Concordion specification. You'd run just this specification and it would in turn run all your tests, giving you a nice red / green overview. You can fold it into your continuous itegration process (Ant or Maven) and publish the overview and related specifications straight to some HTTP server for your customers to review, per-build, 24/7. Nice.

Check out the user manual, written as Concordion specifications and download the binaries from Google Code.

As a closing note, as I mentioned Concordion, its only fair to mention Xcordion (and more recently Xcordion2) which is essentially a fork of Concordion. The main difference being a philosophical one. If Concordion constrains some activities within the specification (mostly to encourage certain principles), Xcordion is more of a free for all. With great power comes great responsibility and all that. Whilst Xcordion2 is being worked on, be prepared to build from source. You'll notice a couple of features missing, at least in the short term.

Objectives

2010-06-17T07:13:00.023Z

It's objective setting time for the guys at work. Aligning what's really good for you with what's good for the company shouldn't be difficult, right? Your generally heading in the same direction and have similar interests at heart. Why then do we end up with generic meaningless objectives like "attend a XXX course"? I think you can get the most out of the objective setting exercise by thinking about what you want personally out of your career, don't settle for the bland, phrase your objectives so there is real value in them. So, if the company think of them in terms of

S.M.A.R.T

I like the think of them as

S.M.A.R.T.Y

Where the Y is all about YOU. Make it personal.

For example, we all want to be better developers but how do you phrase that as an acceptable business objective that management can neatly label it, put in a box and pull out in twelve months time to see if it's flourished? Defining "better" in terms of SMART isn't going to be easy. How about rephrasing it and thinking about it as reflecting on your coding practice? Keep a code journal of good and bad coding choices, yours and your teams. Reflect on it regularly and update entries. Was choosing a ThreadsafeDooDar here the right decision three months on, did it ever get used in a threaded context? How about that choice of using setter over constructor injection? How's that working out for you? The responsibility to reflect is yours but you can get outside perspectives from your peers.

The key personal objective here is to keep learning, the company objective is to become a better developer. It can be specific because your wording will reflect something along the lines of "demonstrate a self-assessment of personal and the team coding standard and style". It's hard to avoid the word "quality" but if you do mention it, you should define it (not easy). The point is there's something the company can buy into but there's something more subtle that you can really buy into and get value from. It's all about YOU after all.

It's measurable (at least from the companies perspective, and so can get signed off on) because you have your journal as evidence along with dates and conclusions. You can even emphasize a few key learning points / techniques that you've changed your opinion on. Change is good and demonstrates learning.

You might be able to extend this by talking in terms of coaching. I poo poo'ed a XXX training course earlier because I feel corporate training camps are the worst place to learn anything (well, there are probably worse places but there's certainly plenty of better environments). Anyway, asking for professional and formal coaching is a great way to develop and the journal idea can be used as evidence. The coach can co-author to give it that "official stamp".

An example journal template might include date, initial description (the approach you took), pros/cons of the initial decision (why you took the approach), updated thoughts (including what caused you to reflect on this specific entry), updated pros/cons, conclusion, peer comments. Try and keep an entry specific to a single decision you made in the code (coarse or fine grained). Getting a feel about what to journal and what not to journal should be pretty natural, any "niggles" you get might prompt you or spotting that you're all talking about this part of the code a lot might be a clue.

The real benefit though is what you can take away from it, if you talk over your journal points with the team and keep an open mind, I'd hope it gets filled with lots of nuggets you can add to your toolbox.

Un/Marshalling

2010-05-01T08:35:00.000Z

This post should probably be titled "Why I don't like Unmarhsalling frameworks". They just piss me off. I mean really, who wants to have to use the Java objects that they make you use? It's not that they don't represent as objects the data that the frameworks unmarshall, they do. It's more that the underlying data may or may not match your systems view of the domain. I don't want to have to run some process to generate Foo.java only to convert it to MyFoo.java, I'd rather go straight to MyFoo.

I was using Castor to do the former some time ago. It sounded like a good idea, and I got going really quickly. However, I quickly didn't like the objects its produced so had to modify the mapping.xml to "tailor" the marshalling. Still fine, this was ok for a while but soon enough I descended into Castor mapping hell. I seriously spent days trying to tweak things and had to compromise the model in the end.

After reflection, I thought I'd try and dump Castor. I replaced the marshalling with merging to a template and produced cleaner XML in under an hour. The unmarshalling, replaced with manually parsing the XML and inserting elements directly into objects. I couldn't believe how much less pain I was feeling!

For me, the overhead of maintaining these more "manual" approaches is far less than working around any framework mismatches. If you're lucky enough to find an marhsalling framework that can grow with you, fair play, but I'm heavily biased towards a more manual approach these days. Down with [insert your framework here].

Setter vs Constructor Injection

2010-05-01T08:18:00.000Z

So what is the argument for / against? It can be a tough one to describe as I recently discovered.

You can say that constructor injection forces an object to have its dependencies set explicitly and setter injection is open to forgetfulness but how powerful an argument is that really? Surely the tests would catch it if you miss a set call? Constructor injection does say upfront "this is what I need", so there's no "first call this, then this and don't forget this" - its explicit and you don't need to know the internals that setters try and expose. That's useful. But on the small, how complex do the combination of set calls get?

I don't think that a large number of constructor arguments justifies defecting to the setter camp as the real smell here is often that the class is doing too much and/or has too many dependencies. It's often cited that the reason there are lots of setters is because of particular dependency injection framework leads you in that direction but why compromise (I should probably say, comply) because you're told to?

How about the way the system would grow using constructors vs setters? I think this is where the real argument lies. You can move forward perfectly happily with either approach, content with the fact that dependencies are isolated. Testing becomes simpler, more focused on the objects under test and the earth continues to orbit the sun. Its only much later when the system is all but grown up that you can look back and reflect. Have setters contributed to creating a system that is assembled in a complex, disheveled way? Are you leaning on external (ahem, xml) configuration to manage this? Alternatively, has a constructor centric approach left you with a system with a more concise declarative assembly strategy? Which has more noise? You tell me...

Wrapping Exceptions zzzZZ

2010-04-25T17:19:00.004Z

I'm totally bored of wrapping exceptions in Java,

try {
   // do something
} catch (BoredomException e) {
   // do something else
}

It's verbose, ugly and has nothing to do with what you're really trying to convey. It's just noise (more or less). For example, when using the just dreadful Google Data API to access my calendar, I wrapped a couple of underlying Google services to be able to mock. Each service wanted to throw a bunch of Google specific exceptions which I, of course, wanted to rethrow as application specific. Boring...

public O call() throws CalendarException {
    try {
        return service.call();  // the call to the google service
    } catch (MalformedURLException e) {
        throw new CalendarException(BAD_URL_MESSAGE, e); 
    } catch (IOException e) {
        throw new CalendarException(IO_EXCEPTION_MESSAGE, e);
    } catch (AuthenticationException e) {
        throw new CalendarException(AUTHENTICATION_MESSAGE, e);
    } catch (ServiceException e) {
        throw new CalendarException(SERVICE_EXCEPTION_MESSAGE, e);
    }
}

If I didn't delegate like this, every service call would have to wrap and handle the google exceptions. As I've done this a few times, I decided to add it to tempus-fugit as a ExceptionWrapper class. Using this class, you can wrap a Callable to rethrow any caught exception as some other (including the underlying exception as the cause).

So, in a similar way to the above, the client can ignore any declared exceptions and just rethrow them in-line. For example,

wrapAnyException(new Callable<Object>() {     
    @Override     
    public Object call() throws ServiceException {
         // nasty code throwing a bunch of exceptions     
    } 
}, with(CalendarException.class));

when this is in-lined further, it hopefully becomes more succinct.

wrapAnyException(serviceCall(), with(CalendarException.class));

This will wrap any exception and rethrow as a new CalendarException to include as the cause any underlying exception. It uses reflection to create the new exception, and forces the syntactically sugary with by taking a WithException as the second parameter.

It's a candidate feature for tempus-fugit 1.1 for now, and will go in for sure if it gets some millage. Let me know if you find it useful. Source is here with the test here.

Nibbles the Cat

2010-03-19T21:39:00.002Z

What has Nibbles the Cat got to do with deadlocks? I'm glad you asked. It all started when I introduced a deadlock in some performance monitoring. I inadvertently prevented a statistic collection daemon I wrote from shutting down thanks to some unlucky timing and bad synchronisation policy. Because the synchronisation that was involved was distributed across a couple of classes (including some external classes) it wasn't obvious where they clashed. It got me thinking more about deadlocks and how many times we actually see them in real systems and gave rise to me creating the DeadlockDetector class in tempus-fugit.

I'm talking here about Java level deadlocks really and to illustrate the point, poor old Nibbles got himself in quite a pickle. This situation is like this; Nibbles has been abducted and the Kidnapper and Negotiator have started a dialogue...

public void potentialDeadlock() {

     new Kidnapper().start();

     new Negotiator().start();

}

However, in the process of negotiation it becomes apparent that the Kidnapper is unwilling to release poor Nibbles until he has received the Cash and the Negotiator is unwilling to part with the Cash until he has poor Nibbles back in his arms.

By synchronising on nibbles below, the Kidnapper is holding onto him (more specifically his monitor) until the end of the synchronised block. However, within this block the Kidnapper is trying to take the cash. The access to this method is itself synchronised on the cash, meaning that no one else can access the cash whilst the Kidnapper is grabbing it. Meanwhile, the Negotiator is synchronising on the cash, holding onto it (or again, more specifically, it's monitor) until the end of the synchronised block then within that block, it requires nibbles. We can start to see the potential for deadlock.

public class Kidnapper extends Thread {

   public void run() {

      synchronized (nibbles) {

         synchronized (cash) {

            take(cash);

         }

      }

   }

}



public class Negotiator extends Thread {

   public void run() {

      synchronized (cash) {

         synchronized (nibbles) {

            take(nibbles);

         }

      }

   }

}

The deadlock detector displays this woeful situation as follows.

Deadlock detected

=================



"Negotiator-Thread-1":

  waiting to lock Monitor of com.google.code.tempusfugit.concurrency.DeadlockDetectorTest$Cat@ce4a8a

  which is held by "Kidnapper-Thread-0"



"Kidnapper-Thread-0":

  waiting to lock Monitor of com.google.code.tempusfugit.concurrency.DeadlockDetectorTest$Cash@7fc8b2

  which is held by "Negotiator-Thread-1"

If you fire up the example and point jconsole at it, you'll get similar results from the Thread tab. You can see how tempus-fugit tests the DeadlockDetector class here and to find out more about see the project documentation.

oh and don't worry, Nibbles was released and the Kidnapper arrested in the end.

Type Safe Annotation

2010-01-04T20:51:00.011Z

A new year and another Java gripe! This time its annotations and the lack of anything useful by way of parameters. Implementing the Goetz annotations from Concurrency In Practice, I wanted to include an enum as a parameter type. Kind of like this

public @interface GuardedBy {
   Type value();

   public enum Type { FIELD, CLASS; }
}

So far so good. I then wanted to somehow parameterise the enum constants themselves to give extra information.

public @interface GuardedBy {
   Type value();

   public enum Type {
      CLASS, FIELD;

      public static Type FIELD(String field) {
         return FIELD;
      }

      public static Type CLASS(String type) {
         return CLASS;
      }
   }
}

Now, here's where the trouble began. Using the static constructor method is fine when I want to create an instance of a Type but not when I want to annotate some method. For example,

@GuardedBy(GuardedBy.Type.CLASS("more info")) // javac cries 
public void foo() {
   GuardedBy.Type type = GuardedBy.Type.CLASS("more info"); // fine
}

The compiler very quickly complains that the attribute value must be constant. Specifically,

an enum annotation value must be an enum constant

To get round things, you can just create several attributes for the annotation. Rather than have a nice CLASS type which can optionally have a description, I was forced to have one attribute of type and another to capture the additional information. grrrr.

public @interface GuardedBy {
   Type value();
   String details() default "";

   public enum Type { CLASS, FIELD; }
}

Shame on you Java for leading me on a merry dance! I'd love to know more about why things are like this, so if you've got any more details, feel free to post a comment.

Running JUnit tests in parallel

2009-12-29T19:03:00.004Z

I've been playing with running tests in their own threads for a while now (in particular with reference to GUI testing) and am settling on a warm fuzzy feeling towards my general approach. I'm trying to capture this warm feeling more explicitly in the tempus-fugit project and today I was working on running tests in parallel with JUnit.

Looking more into recent versions of JUnit, there seems to be lots and lots of integration points for you to play with. I've been playing with Rules, Statements and Runners mostly and when creating your own BlockJUnit4ClassRunner, I spotted you can override the scheduler which, er, schedules the actual test methods to be run.

After experimenting with much less straight forward integrations, overriding the scheduler gave the following.


public class ConcurrentTestRunner extends BlockJUnit4ClassRunner {

public ConcurrentTestRunner(Class type) throws Exception {
    super(type);
    setScheduler(new ConcurrentScheduler());
}

private static class ConcurrentScheduler implements RunnerScheduler {

    private ExecutorService executor;

    public ConcurrentScheduler() {
        executor = newCachedThreadPool(new ThreadFactory() { ... });
    }

    public void schedule(Runnable childStatement) {
        executor.submit(childStatement);
    }

    public void finished() {
        shutdown(executor).waitingForCompletion(seconds(10));
    }
}
}

This results in all the test methods within a given class running in parallel. I'm excited about speeding up the execution time of my tests, next step would be run all tests across classes in parallel! hmmm.

This is all available to use via the tempus-fugit project by the way. See the tempus-fugit documentation for more details on the runner.

Flickering tests and a JUnit Rule

2009-12-24T15:55:00.004Z

Occasionally I'll see pesky flickering tests. Sometimes they're green, sometimes they're red and this can happen without any code changes! Something is afoul for sure. What bugs me the most though is that when trying to fix the problem, I can never be sure that I haven't just been lucky and the green I'm seeing isn't really a false positive. I'll have to manually run the test several times before my confidence grows.

In an attempt to ease this heavy burden and make light work of this drudgery, I created an @Intermittent annotation with a corresponding JUnit Rule and Runner. Now, I can mark up a suspect test and get someone else to do the repetition. Joy.


@Test
@Intermittent
public void flickering() {
   // ...
}

I can then use the IntermittentRule to run the test method repeatedly.


public class FlickeringTest {

 @Rule public IntermittentRule rule = new IntermittentRule();

 @Test
 @Intermittent
 public void flickering() {
     // ...
 }
}

Or use the @RunWith annotation to run the test using the IntermittentTestRunner.


@RunWith(IntermittentTestRunner.class)
public class FlickeringTest {

 @Rule public IntermittentRule rule = new IntermittentRule();

 @Test
 @Intermittent
 public void flickering() {
     // ...
 }
}

What's interesting here is the way in which the Rule and Runner interact with JUnit. Newer versions of JUnit have introduced the idea of Rules and Statements. Using a Rule allows access to the underlying Statement which in our case is the action to run the test method. So we're able to run the underlying statement again and again. Nice.

The Runner however, can hook into JUnit's framework a different way. It can access more than one statement and so position itself slightly differently. What this means for us is that using the Rule above, any @Before or @After methods will only be run once but the test method will run multiple times. Using the Runner above however, will run any @Before or @After methods once for each test repetition.

The code is available as part of the tempus-fugit library so feel free to browse and feedback your comments.

Deadlock detection in Java

2009-12-24T15:41:00.007Z

I recently added a basic deadlock detection mechanism to tempus-fugit. The DeadlockDetector class allows you to programmatically detect basic deadlocks in your Java code. You can output deadlocks using the following code (note that printing a thread dump using the ThreadDump class will automatically attempt to find any deadlocks).


DeadlockDetector.printDeadlocks(System.out);

The DeadlockDecector class can only spot Java monitor cyclic locking problems. It's implementation is basically the same as that used by jconsole and jstack although Java 1.6 versions can additionally detect Lock based cyclic problems. The types of deadlock it can detect can be illustrated in the example below.

@Test
public void potentialDeadlock() {
  new Kidnapper().start();
  new Negotiator().start();
}

public class Kidnapper extends Thread {
  public void run() {
     synchronized (nibbles) {
        synchronized (cash) {
            take(cash);
        }
     }
  }
}

public class Negotiator extends Thread {
  public void run() {
     synchronized (cash) {
        synchronized (nibbles) {
            take(nibbles);
        }
     }
  }
}

Here, the Kidnapper is unwilling to release poor Nibbles the Cat until he has the Cash but our Negotiator is unwilling to part with the Cash until he has poor Nibbles back in his arms. The deadlock detector displays this woeful situation as follows.


 Deadlock detected
 =================

 "Negotiator-Thread-1":
    waiting to lock Monitor of ...DeadlockDetectorTest$Cat@ce4a8a
    which is held by "Kidnapper-Thread-0"

 "Kidnapper-Thread-0":
    waiting to lock Monitor of ...DeadlockDetectorTest$Cash@7fc8b2
    which is held by "Negotiator-Thread-1"

I don't imagine you'll need to print out information about deadlocks or thread dumps from within your code very often, but on the off chance you'll need to during debugging, they are both available for you to use in the tempus-fugit project.

Atomiticy of the Thread class

2009-12-10T09:18:00.009Z

I had an interesting time getting a couple of tests running for tempus-fugit recently. It threw up a couple of interesting aspects about using threads that I hadn't come across before.

In one particular test, I wanted to show that interrupt is called on a thread and so followed what's becoming a common pattern for me.

@Test
public void sleepInterrupted() throws Exception {
   Thread thread = threadSleepsForever();
   thread.start();
   waitForStartup(thread);
   thread.interrupt();
   waitForShutdown(thread);
   assertThat(thread.isInterrupted(), is(true));
}

The alternative patten is to wait for the assertion and avoid the wait for shutdown method above. Either way, the test above is testing the the thread created has been interrupted by checking the interrupt flag.

The test was failing for me even though I was able to show that the interrupt is called and the interrupt flag is set immediately after the sleeping thread is woken. However, when the test reached the assert, it was false. Weird.

I setup another thread to just poll the sleeping thread for its status and it showed the following.

thread.isInterrupted() = false, thread.getState() = RUNNABLE
thread.isInterrupted() = false, thread.getState() = RUNNABLE
thread.isInterrupted() = false, thread.getState() = RUNNABLE
thread.isInterrupted() = false, thread.getState() = TIMED_WAITING
thread.isInterrupted() = true, thread.getState() = RUNNABLE
thread.isInterrupted() = false, thread.getState() TERMINATED

So my thread was interrupted! It seems to say that when a thread terminates, it will reset the interrupt status flag. Looking at the source, it looks like Java maintains the flag at the native level and not as a member of the Thread class.

private native void interrupt0();

Looking at another run, I got the following

thread.isInterrupted() = false, thread.getState() = RUNNABLE
thread.isInterrupted() = false, thread.getState() = RUNNABLE
thread.isInterrupted() = false, thread.getState() = RUNNABLE
thread.isInterrupted() = false, thread.getState() = TIMED_WAITING
thread.isInterrupted() = true, thread.getState() = TIMED_WAITING
thread.isInterrupted() = false, thread.getState() = TERMINATED

This is even more interesting as it would suggest that an interrupt doesn't update the thread's state and the interrupt flag atomically.

It would seem thread termination will reset the interrupt status flag and that an interrupt doesn't update the interrupt status flag and state atomically.

As an couple of caveats to the type of test above where I want to check the interrupt status flag, its a good idea to avoid side affects by using the terribly named method Thread.interrupted rather than isIntrrupted. I also had to use a stubbed thread to reliably tell if the interrupt was called.

See the code here.

Time Flies

2009-11-11T22:01:00.005Z

I've just published tempus-fugit 1.0-SNAPSHOT to java.net's public maven repository with a view to synchronising it with Apache's uber-repo. Download here, or point your pom at it to enjoy all the fanciful goodness inside.

oh, I should mention... tempus-fugit represents two things to me

A focal point to a lot of thinking done by a lot of very smart people I've been fortunate to work with (along with my own special kind of magic) themed around concurrency and time.
An attempt to consolidate this thinking into a single place that can be easily reused before finally being discarded with mild disdain*.

What it actually is, is

a dozen or so classes that capture key abstractions we've found are frequently shared across domains.
a even smaller number of wrapper or helper classes that we've found can help using concurrency in Java that little bit nicer.
all with an underlying theme of testing and improving testability.

* more specifically, in collaborative projects, it represents a move away from collaboration into a more socially introvert culture whereby collective code ownership becomes uncooperative code ownership. Eat my JAR and don't dare dream of refactoring it. Bad, robot.

Thawte claim I'm not to be trusted...

2009-11-11T19:48:00.005Z

...well, not quite, but they have recently announced that they'd be discontinuing the Personal E-mail Certificates and Web of Trust. These were free services and it comes as bit of a blow as I've been using them to sign my JARs for WebStart deployment. As Thawte are the root authority, it meant no annoying prompts from WebStart claiming I was a would-be terrorist. Looks like I'm back to hiding in shadows.

A couple of useful links below.

http://ciardhubh.de/node/4
http://www.dallaway.com/acad/webstart/

Performance Monitoring (Part 1)

2009-10-31T20:15:00.034Z

Keeping an eye on the performance of your applications is something that it easy to neglect. We all know that we should be regularly recording key performance indicators and regularly leaning on the profiler to spot dangers early, but how often is the dust actually blown off?

I've been doing a fair bit of this lately (blowing off that is) and thought it might be a good idea to capture a few thoughts in a mini-series of posts that will be rivaled in drama only by the likes of Dynasty and Dallas. Enjoy...

Part 1. The Beginning

It's probably a shrewd move whatever your application to setup performance measurement early. I'm usually interested in two areas, the real-time performance and the resource usage.

Performance

What you define as "performance" is completely down to you and your application. It may be in the request / response world, the time taken to process a request and return a response. It may reflect the ability to handle thousands of requests within a SLA agreed time or it may be around reliability and up time, scalability or accuracy. Whatever. Some things that are usually constant though are an emphasis on time, correctness and robustness of system under load. This is all about measurement and optimisation.

Response times - of your critical path
Throughput - typically the number of requests / transactions / operations per second
Load - the number of request / transactions / operations being observed.
Success / failure counts - it's no good being quick if you're always wrong!
Garbage collection profile - you'll probably be more interested in this if your system is already highly performant. In these cases you can potentially see dramatic differences in critical path processing times when excessive or inefficient GC is taking place. Tuning is a specialist skill, good luck. Google Sun's G1 for renewed hope.

Resources

The section on resources is more open ended, I mean hear both hardware resources and software resources (including the resources available to the JVM). Of course, taking into account the resource usage of interrelated external systems may also be of interest to the happy tuner and its all about avoiding resources starvation and spotting bottlenecks.

CPU
Memory - The JVM can theoretically allocate only 2GB to the heap on 32 bit machines and 16GB on 64 bit machines. In practice, hitting this upper bound with -Xmx can be tricky. Either way, you'll want to understand the memory characteristics of your application and how it behaves if you start to reach the upper bound of the heap.
IO - I mean here things like handles, database connections, physical network connections and other tasty bites. Without these, all is lost and managing these to avoid resource starvation can be of primary importance.
Threads - there is a relationship between the number of threads your software requires and hardware resources available to you, threads are not unbounded in your application so the prudent coder will seek insight into the affect creating lots of threads will have. See a previous post.

Statistic Gathering

Once we have an idea of what we want to measure, the next step is to think about how to capture and represent the raw data. We need to capture the raw data and turn it into information that's useful. It turns out that turning this data into information will influence how we capture it...

You could capture all the data about every single request but it's unlikely that this is the information you'll need without doing some form of manipulation. Instead, I prefer to capture information, manipulating and aggregating the data from each request as we go.

For example, typically you might be interested in the means, samples or percentile based information.

Mean values are probably the simplest to get started with, recording figures like mean response time are useful but not always the full picture. Average figures can be skewed by extreme results or extreme results may be lost in the mean. Supplementing with unmodified samples can help give a fuller picture.

Percentile based reporting gives more insight than a smoothed average figure. For example, if most of the requests are sub-second, it can be hard to spot the small number of slow requests. By reviewing at a particular percentile you can gain visibility of these and help determine what's an acceptable level. For example, the mean response time at the 95th percentile will likely be noticeably different than the mean.

Another option is to implement a sliding view on the data. Here, you record information for, say, an hour and as time progresses beyond that hour, you start to loose past data and include new data. At most, an hour of data is available. This can be useful as rather than cumulative counts, the sliding window is visually more responsive to changes (in particular, drops). For example, once charted, a drop in a cumulative chart will show as a slow in accent of the series where with a sliding window chart, it will show as a clear drop. A subsequent post in the series will talk in more detail about this sliding window approach.

Implementing

There's plenty of ways you can as statistics to your application. Instrumenting the code at key points through decoration is a good option. Using something like AspectJ is another option and both free your code from any clutter that would otherwise pollute and distract from your classes real responsibilities. Logging is a good example of instrumentation polluting things. Typically, the act of logging has nothing to do with the role of the class doing the logging.

The instrumenting code will likely be modifying objects in memory that collect and manipulate the raw data. You'll almost certainly want to make sure this memory object is thread safe and performant, it should be light weight and non-obtrusive.

What happens to the memory object is up to you, but you'll need to output it for analysis at some point. I prefer to publish the objects via JMX, that way you can setup a separate 'collector' to periodically read the values and have it's wicked way with them. With this approach, its even more important to keep the object lightweight - the MBean will always be in memory (the MBean server will see to this) and so if it contains rogue references to big objects, they'll never get garbage collected. The other tip is to create one of these such objects per request type, don't create a new one per request for example.

In the next exciting post in the series, I'll talk about interpreting the results you get. Hope that wasn't too dull.

Java source for Mac OSX

2009-08-01T11:53:00.002Z

Grrrr arrrrh Apple, grrrr...

See here for a more balanced view.

Abstracting ReentrantReadWriteLock

2009-08-01T09:47:00.015Z

All locks in Java are reentrant, they have to be in case the owner of the monitor ends up calling a method that needs that monitor. So, if a thread requests a lock that it already holds, it'll be given it. Without this, a subclass couldn't override a snynchronised method and then call the superclass method without deadlocking.

Java's ReenstrantReadWriteLock is about acquiring seperate read and write locks for efficiency. For example, in the case where you may have infrequent writes but frequent reads, it may be more efficient to not synchronise all access with just one lock. Instead, ReenstrantReadWriteLock can allow all read access to only block when a write is taking place. You'll end up with multiple simultaneous reads but synchronised writes and all the reads will have guaranteed visibility of the writes.

How to know when to actually use the ReenstrantReadWriteLock is going to have to be the subject of a followup post, but as you'd expect, profiling is going help you make decisions against your specific situation.

With all Lock implementations, you specifically acquire the lock and are politely asked to release the lock within a finally block. Makes sense but it always bugs me that we're forced into Java's verbosity trap, yet again.

So, vanilla Java would have you;

Lock l = ...;
l.lock();
try {
    // access the resource protected by this lock
} finally {
    l.unlock();
}

So why not wrap the boiler plate code up in a micro-DSL and pass a closure-like thing to execute? Any implementation must call both lock and unlock and re-throw any exceptions. The following test shows this to be true.

@RunWith(JMock.class)
public class ExecuteUsingLockTest {

    private final Mockery context = new JUnit4Mockery() {{
        setImposteriser(ClassImposteriser.INSTANCE);
    }};

    ReadLock readLock = context.mock(ReadLock.class);
    WriteLock writeLock = context.mock(WriteLock.class);

    @Test
    public void readLock() {
        setExpectationsOn(readLock);
        execute(something()).using(readLock);
    }

    @Test
    public void writeLock() {
        setExpectationsOn(writeLock);
        execute(something()).using(writeLock);
    }

    @Test (expected = Exception.class)
    public void readLockThrowingException() throws Exception {
        setExpectationsOn(readLock);
        execute(somethingThatThrowsException()).using(readLock);
    }

    @Test(expected = Exception.class)
    public void writeLockThrowingException() throws Exception {
        setExpectationsOn(writeLock);
        execute(somethingThatThrowsException()).using(writeLock);
    }

    private Callable<Void, RuntimeException> something() {
        return new Callable<Void, RuntimeException>() {
            public Void call() throws RuntimeException {
                return null;
            }
        };
    }

    private Callable<Void, Exception> somethingThatThrowsException() {
        return new Callable<Void, Exception>() {
            public Void call() throws Exception {
                throw new RuntimeException("bad robot");
            }
        };
    }

    private void setExpectationsOn(final Lock lock) {
        context.checking(new Expectations() {{
            one(lock).lock();
            one(lock).unlock();
        }});
    }
}

The implementation is fairly straight forward with a couple of interesting points to note around generics.

public class ExecuteUsingLock<T, E extends Exception> {

    private final Callable<T, E> callable;

    private ExecuteUsingLock(Callable<T, E> callable) {
        this.callable = callable;
    }

    public static <T, E extends Exception> ExecuteUsingLock<T, E> execute(Callable<T, E> callable) {
        return new ExecuteUsingLock<T, E>(callable);
    }

    public T using(ReentrantReadWriteLock.WriteLock write) throws E {
        try {
            write.lock();
            return callable.call();
        } finally {
            write.unlock();
        }
    }

    public T using(ReentrantReadWriteLock.ReadLock read) throws E {
        try {
            read.lock();
            return callable.call();
        } finally {
            read.unlock();
        }
    }  
}

Having the micro-DSL pass in the generic Callable on the static constructor meant that I couldn't make just the using method generic and instead had to link the types by making the class definition generic. You might also notice that the Callable used isn't Java's Callable, as Sun saw fit not to have the Exception as a generic type. By creating the new Callable interface below, I was able to neaten up the micro-DSL so that we're not forced to throw Exception from a method that uses the ExecuteUsingLock class. Instead, you define your closure-like function to throw RuntimeException.

A real world example might be something that updates a status probe where the variable lock below is an instance of ReentrantReadWriteLock.

public void setStatus(final Status status) {
    execute(settingStatus(status)).using(lock.writeLock());
}

public String getStatus() {
    return execute(gettingStatus()).using(lock.readLock());
}

private Callable<Void, RuntimeException> settingStatus(final Status status) {
    return new Callable<Void, RuntimeException>() {
        public Void call() {
            EnclosingClass.this.status = status;
            return null;
        }
    };
}

private Callable<Status, RuntimeException> gettingStatus() {
    return new Callable<Status, RuntimeException>() {
        public Status call() {
            return EnclosingClass.this.status.toString();
        }
    };
}

All this to avoid the boiler plate code! Fresh.

Update: Since writing this entry, I created the tempus-fugit project to capture these kinds of ideas.

is(not(instanceOf(smell)));

2009-07-29T18:17:00.008Z

For some reason, common perception is that using instanceof is a bit of smell. I think its fallen in with a bad crowd and isn't really as bad as its cracked up to be.

For example, given the following, what's clearer in the following exception handling code?


try {
 ...
} catch (InvocationTargetException e) {
 if (RuntimeException.class.isAssignableFrom(e.getCause().getClass())) {
    throw new ApplicationException(...);
 }
 throw e.getCause();
}


try {
 ...
} catch (InvocationTargetException e) {
 if (e.getCause() instanceof RuntimeException) {
    throw new ApplicationException(...)
 }
 throw e.getCause();
}

There seems very little difference in the semantics of the two calls, both handle class hierarchies and both handle interfaces. I think the legitimate use for Class.isAssignableFrom is around scenarios where reflective-type checks are more suited (for example, where the class to check against is only known or may change at runtime). Using Class.isInstance is a similar alternative. The other point that the API documentation points out is that isAssignableFrom works with primitives whereas instanceof can not.

I wonder if the bad reputation instanceof suffers from is more due to often being seen in the company of strange conditional logic than anything else. The kind of stuff that seems to go against your Mumma's good advice. It may just have fallen in with a bad crowd but is itself a good kid at heart...

I suspect instanceof, being around since forever, is seen as a bit antiquated but given the example above using Class.isAssignableFrom seems to just obfuscate things. The example needs to find out why a dynamic proxy throws an exception and because the InvocationTargetException wraps the underlying cause, it kind of breaks checked exception handling and it feels like we need to do some conditional check. If there is a smell here, that's it. The above is trying to hide the symptom of the smell (instanceof) but the smell is still there, its just harder to see. I often catch similar whiffs when catching ExecutionException.

More conventional cases where instanceof is a genuine smell may be around conditional logic making decisions based on class type where using polymrphism would be the obvious solution. It's this kind of thing that your Mumma warned you about and I suspect its this is the real smell that's rubbed off on poor downtrodden instanceof.

Convert a Callable to a Runnable

2009-05-29T20:37:00.009Z

Everytime I come accross it, it bugs me.

The Executors class has helper methods to convert from a Runnable to a Callable, presumably so you can submit an old school Runnable task to an exectuor, but I can never find the counterpart helper. Something to convert a Callable to a Runnable.

I seem to keep finding myself needing one and you might think me silly for doing so but it's because I'm into the habit of coding as Callables now. I tend to think of Runnable as simply something that can run, not in terms of a Thread or concurrent task. I like the way I think about Runnable.

Callable is like the new cool kid, with its funky return value and the ability to throw an exception. Wow, why did I ever bother with Runnable in the first place? Callable, like Runnable, however is still just something that can be called, it just so happens to fit in nicely with the executor framework and kind of looks all bling.

I digress... back to jusitifying why I think I'm always needing to convert from Callable to Runnable. As I say, if both interfaces represent something that can be called, I favour Callable becuase its more powerful. I can return useful things and catch useful exceptions. I'm in the habit of using Callable.

Often then, I'll have utility classes, useful nuggets of functionality dressed up as a Callable, and if I want to schedule them with an executor with a fixed delay or fixed rate, I can't. The interface wants a Runnable and only a Runnable. Most likely becuase it doesn't really make much sense to schedule a fixed rate execution of a task that returns something when it would take quite some thinking to actually do something with the return value.

None the less, I'd like to schedule something at a fixed rate (ignoring the result) that I also schedule elsewhere and actually do something with the result.

Starting with the tests, any helper must delegate to the Callable and handle any exceptions.


@RunWith(JMock.class)
public class CallableAdapterTest {
   private final Mockery context = new Mockery();
   private final Callable callable = context.mock(Callable.class);

   private static final Object RESULT = new Object();

   @Test
   public void delegates() throws Exception {
      callableWill(returnValue(RESULT));
      runnable(callable).run();
   }  

   @Test(expected=RuntimeException.class)
   public void exceptionBubblesUp() throws Exception {
      callableWill(throwException(new Exception()));
      runnable(callable).run();
   }
 
   private void callableWill(final Action action) throws Exception {
      context.checking(new Expectations() {{
         one(callable).call(); will(action);
      }});
   }
}

To get a green light, the implementation is fairly trivial.


public class CallableAdapter {
 
   public static Runnable runnable(final Callable callable) {
      return new Runnable() {
         @Override
         public void run()
            try
               callable.call();
            } catch (Exception e) {
               throw new RuntimeException(e);
            }
         }
      };
   }
}

Let me know if you've got any comments, great ideas on the subject or can improve on the implementation above.

Update: Since writing this entry, I created the tempus-fugit project to capture these kinds of ideas.

SWTBot vs Window Licker

2009-03-15T11:30:00.009Z

The title is a bit misleading because these guys arn't really squaring off like Gozilla and Gigan, the projects aren't competing and this blog isn't really a full or fair compairison. For a start Window Licker doesn't yet support SWT and as its name suggests SWTBot, er, does.

My comments are really based on having a go at implementing some basic SWT support in Window Licker and adding some features to SWTBot so it shouldn't be taken as authoritative. This entry is more of an experience report after trying to contribute to both projects. I'll try to follow up with an API usage report when I've used both more with my applications.

The Code

Window Licker supports Swing and Html (including Ajax) very elegantly and uses the Java java.awt.robot package to take control of the mouse and keyboard and generate native events. SWTBot simulates events using manual notification. That is to say it calls the SWT widget notifyListeners method directly where as Window Licker fires native GUI events. This wouldn't necessarily be much of a distinction but it does mean SWTBot doesn't always behave exactly as expected (for example, it doesn't handle text caret movement or clearing text fields). An application under test from Window Licker behaves much more like an application driven by a user whereas SWTBot doesn't always have to wait for visual ques to move along.

Window Licker feels like a more mature framework, its nicely componentised and test driven where as SWTBot seems more evolutionary and although there are plenty of nice abstractions in common with Window Licker, it is harder to test due to the lack interfaces/mocks. SWTBot's code base is sprawling and covers a lot of ground, Windows Licker feels more focused on the problem it tries to solve. Having said that, its much more straight forward contributing to SWTBot, I was able to contribute (and understand) more of the SWTBot code base upfront.

SWTBot's project hosting presents itself as a confused, capricious old boy, shuffling papers and smelling mildly of pee

The other slight gripe I have with SWTBot is the project structure, its more suited to Eclipse than IntelliJ as its dependencies are provided via plugins rather than raw Jars. Window Licker comes with Eclipse and IntelliJ project files which is always nice. SWTBot also has several projects that make up the software, the build process isn't as obvious as others and it uses some generated code that you'll probably won't spot but still need to know about.

The Project Hosting

SWTBot has recently moved to its new home on Eclipse.org but the team lead hasn't deprecated the old site and resources. Consequentially, it can be a bit confusing for a new user to follow when there are two subversion repositories, two sets of bug tracking instances, lots of mailings lists and so on. Window Licker is much more low key, at the time of writing there is no pre-packaged release to download, it has a minimal site on Google Code that invites you to just download the source and get stuck in! In lots of ways I prefer this approach.

SWTBot has lots of broken links on the site and presents itself as a confused, capricious old boy, shuffling papers and smelling mildly of pee. Window Licker on the other hand, portrays itself confidently, most likely wears a monocle and speaks in a concise and authoritative tone. Even though some of what it has too say seems to go over my head.

Conclusion

For me, SWTBot really lets itself down is how it's being run. We all realise that open source software often has to take a back seat to your day job but if you're going to release something open source and encourage participation (say by hosting it on Eclipse.org) you should really be able to take the helm and support your community. As such, I had a bit of a bad experience with SWTBot trying to get a feature committed. In short, I did a whole bunch of work after explicitly being asked to do so in order to get it committed, then when it was done, the team lead ignored it and committed a previous version I'd submitted! The previous version works fine, it did the job but i was asked to re-jig things to better fit in with the project's standards, yet it got dumped. See the Bugzilla trail less I waffle even more.

It's a shame because experiences like that just lower confidence in a project, I'm unlikely to want to push any changes back now. I guess though in summary, SWTBot's code was easier to contribute to (although I was working in one small area of a large project structure). Window Licker is probably the superior technically and in terms of code quality but was harder (for me) to work with. I was working on essentially cloning some of the core Licker functionality to work with SWT and found that I didn't fully understand all the abstractions and mechanisms. I suspect that in a pairing environment, it wouldn't be a problem but contributing via email and patches is unlikely to get the most out of collaboration.

Less is More

2009-02-26T19:20:00.012Z

Dirty and Jim were trying to convince me the other day that less is more when it comes to maximum heap size in Java. After I finally got the point, I was aghast! Shock, horror... creating threads in Java takes up non-VM managed memory!

After running a few tests I confirmed that the lower the max heap size is, the more threads I could create. I can easily get a OutOfMemoryError when allocating memory for threads way before any real heap space is used.

The helpful message you'll get is.

java.lang.OutOfMemoryError: unable to create new native thread

I ran a simple loop that would create dozy threads (they sleep a lot) and watched how many threads could be created before the out of memory error above. I repeated this after setting the maximum heap size. Check out the results below, you can see that no real Java heap is used, still I get an out of memory error and am limited on the number of threads I can create. As a caveat, these results should in no way taken as representative on any other environment than my laptop running Java 1.6. Your mileage may vary.

I'd always considered OS native threads or light weight processes to be effectively unbounded with the restrictions being on the resources they are associated with. I don't think this is really the case though as clearly, the act of creating a thread that does nothing and has no real resources associated with it (other than any native voodoo) requires OS memory allocation in my tests.

I thought this was really interesting as it demonstrates how small decisions can have a big impact on the run-time characteristics of a system. Being able to tune your application to suit your production requirements is always tricky. Large memory models and data manipulation can often require large heap sizes but if in addition, you then want to create large numbers of threads, you might get into the murky world I've touched on here. Of course, thread pools are generally the way to go if you want to manage your resources properly, but that's another story all together.

I'm sure Google can tell you more, but see this random blog for another description.

Infering the Type in a Micro DSL

2009-02-23T19:34:00.008Z

In a recent post, I was talking about a micro DSL to create a simple "find x in a list" service. The key thing here is that it defines how to look for x in the list. So the list can be a list of anything, not just a list of x's.

Just to recap then, to find something in a list, the original client code (using a static import) looks like this.


find(needle).in(haystack)

The class (in this case NeedleFinder) implements the DSL and specifically decides in the in method how to compare a Needle object to whatever is in the haystack list. I wanted to create a more generic class so started to implement the ListFinder to use generics and a couple of interesting things came out.

The generified class looks like this.


public final class ListFinder<T, L> {
 private final T target;
 private List<L> list = new ArrayList<L>();

 private ListFinder(T target) {
    this.target = target;
 }

 public static <T, L> ListFinder<T, L> find(T target) {
    return new ListFinder<T, L>(target);
 }

 public ListFinder<T, L> in(List<L> list) {
    this.list = new ArrayList<L>(list);
    return this;
 }

 public L using(Comparator<T, L> comparator) {
    for (L item : list) {
       if (comparator.equals(target, item)) {
          return item;
       }
    }
    return null;
 }

 public interface Comparator<T, L> {
    boolean equals(T target, L item);
 }

}

With the following test case showing its usage (the Needle and Bale class aren't show for brevity).


public class ListFinderTest {

 private final Needle needle = new Needle("Bob");
 private final Needle missing = new Needle("Billy");
 private final Bale bale1 = new Bale("Christian");
 private final Bale bale2 = new Bale("Bob in disguise");
 private final Bale bale3 = new Bale("Kelly");
 private final List<Bale> haystack = asList(bale1, bale2, bale3);

 private final ListFinder.Comparator<Needle, Bale> comparator = new ListFinder.Comparator<Needle, Bale>() {
    @Override
    public boolean equals(Needle needle, Bale bale) {
       return bale.name.contains(needle.name);
    }
 };

 @Test
 public void needleFoundInHaystack() throws Exception {
    assertThat(find(needle).in(haystack).using(comparator), is(bale));
 }

 @Test
 public void needleNotFoundInHaystack() throws Exception {
    assertThat(find(missing).in(haystack).using(comparator), is(nullValue()));
 }

 private static ListFinder<Needle, Bale> find(Needle value) {
    return ListFinder.find(value);
 }
}

Here we're defining the equality of a Needle in a list of Bale objects to be when the name of a Needle is contained in the name of the Bale. A silly example I know but it illustrates that we redefine what we mean by equality for the list finder by implementing the ListFinder.Comparator. The concrete example that spawned the idea was when searching for a Race object inside a list of Event objects; two completely different entities.

Anyway, what I thought was interesting about this example was the type inference going on in the static find method. I originally wanted to just use ListFinder.find method directly as in the following.


ListFinder.find(needle).in(haystack).using(comparator)

Where ListFinder is statically imported. Usually, I'd rely on type inference here to work out that needle means T and therefore T is of type Needle. However, in the case above, the compiler will complain as the haystack parameter is not of type Object. The trick is that the generic method find in ListFinder needs to infer two types (T and L) but only has enough information for T. So it defaults L to type Object. grrr damn Java indisputable logic.

The alternative is to use the full notation as follows.


ListFinder.<Needle, Bale>find(needle).in(haystack).using(comparator)

Or (as I've done in the test) use an internal method who's return type gives the compiler enough information to infer both types. I prefer this approach as it makes the DSL expression to find a needle much more readable.

So, Java can't chain methods to infer the types. I didn't really expect it to be able to so, its a bit much to ask for. Although it would be pretty sweet if it could.

One last thing, Papa Ray was showing me a Jedi alternative to the finder. If we're lucky, he might blog about it. It seems Jedi offers some measure of insurgency against the proliferation of anonymous inner classes in lieu of closures but in all honestly, I just wanted to get in some big words in before signing off. TTFN.

More on Micro DSLs

2009-02-16T20:06:00.015Z

I was recently talking about what I call micro DSLs and I thought I'd follow up with another example.

So, another example of a micro DSL I found myself writing is one of finding some object within a collection of differently typed objects. In my example, I want to find a Race object inside a bunch of calendar events objects, the finder micro DSL looks like this.

final class RaceFinder {

private final Race race;

   private RaceFinder(Race race) {
      this.race = race;
   }

   static RaceFinder find(Race race) {
      return new RaceFinder(race);
   }

   CalendarEventEntry in(List<CalendarEventEntry> events) {
      for (CalendarEventEntry event : events) {
         if (race.getName().equals(event.getTitle().getPlainText())) {
            return event;
         }
      }
      return null;
   }
}

The way the class is built, it forces the client to use it such.

CalendarEventEntry event = find(race).in(events);

Where find is statically imported and race and events have been pre-populated.

The original version had a method in the class to do the find, in the context of this class it was harder to test the find. I had a bunch of mocks and I was testing the find function amongst other behaviours of the class. The search line looked something like this.

CalendarEventEntry event = searchForRaceIn(events);

Why I term this a micro DSL is that it has a very specific usage (to find a race within a calendar) and it has a private constructor and public static construction method to ensure strict usage. The real thing though is the method names don't really do what they imply they're going to do. The 'in' method actually does the search, but the 'find' method doesn't do any such thing, it creates an object.

How do you feel about this? When its used in a tiny domain like this and when it expresses the conversation more concisely than the alternatives, I feel pretty good about it. It makes me realise its not the words (method names) that are important, its the sentences they put together. So I'm compromising on realistic method names for more expressive sentences. The fact that the class prevents you from constructing invalid sentences just makes the warm feeling grow.

Micro DSLs are recognisable by their tiny-domain, the way in which construction is done to ensure they can only be called in an order that makes sense to the domain and badly named methods.

Micro DSLs are starting to spread in my code like jam on toast but I know I have to check myself and really justify their use before I go mad. I also had fun trying to turn the finder above into a generified class which I'll talk about in my next post. I bet you can't wait, you lucky people.

I blog, therefore, I am

2009-02-12T16:22:00.004Z

Just because I blog, it doesn't mean I'm important. Just because I use commas, it doesn't mean I can grammatically correct others. Just because I use the word 'grammatically', it doesn't mean I can spell. Just becuase I can spell, it doesn't mean I can grammatically correct others spellings.

Also, Twitter is shit and exists solely for the purpose of fulfilling its own wet dream.

Inheritance vs Composition

2009-01-24T12:12:00.014Z

When interviewing, I often like to ask a candidate to discuss inheritance vs composition using Stack as an example.

Example 1.


public class EvilStack<T> extends Vector<T> {
  public T pop() {
     // ...
  }
  public void push(T item) {
     // ...
  }
}

Example 2


public class Stack<T> {
  private Vector<T> items = new Vector<T>();
  public T pop() {
     // ...
  }
  public void push(T item) {
     // ...
  }
}

Extending Vector as in example 1 weakens the encapsulation of the class. Suddenly, methods to get and insert elements at specific positions are available to clients of the stack. We move from trying to create a well behaved LIFO stack to creating a socially irrisponsible monster: an EvilStack.

So I was surprised when looking at the Java implementation to see that java.util.Stack extends Vector! Bad robot.

Deprecated Annotation

2009-01-22T15:25:00.006Z

Why didn't Sun add a value property to the @Deprecated annotation? Instead of

@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface Deprecated {
}

which kind of implies we should do the following.

@Deprecated
/** @deprecated use {@link Foo} instead */
public class GoneOff {
   // ...
}

why can't we have

@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface MyDeprecated{
   public abstract String value() default "";
}

which means we can use

@MyDeprecated("use Foo instead")
public class GoneOff {
   // ...
}

Be more Expressive with Builders

2009-01-06T17:07:00.023Z

I came up with a neat little pattern that's helped me be more expressive in some (fairly specific) situations. Here I'll give you a feel for it using the CountDownLatch class as an example.

CountDownLatch.await()

Using an instance of a CountDownLatch, we can wait for the latch to count down to zero, blocking the calling thread before continuing. When using a timeout, the method returns true if the count reached zero or false if the timeout expires. To make the timeout more explicit in my application code, I started with the following.


public void waitForStartup() throws InterruptedException, TimeoutException {
  if (!startup.await(5, SECONDS))
     throw new TimeoutException();
}

public void waitForShutdown() throws InterruptedException, TimeoutException {
  if (!shutdown.await(5, SECONDS))
     throw new TimeoutException();
}

Using a Builder with Static Constructor

To be more concise, I wanted to wrap the logic above in some kind of helper class. Thanks to lovely static imports, a private constructor and a static factory method called await(), I was able to express the same thing with the following.


public void waitForStartup() throws InterruptedException, TimeoutException {
  await(startup).with(TIMEOUT);
}

public void waitForShutdown() throws InterruptedException, TimeoutException {
  await(shutdown).with(TIMEOUT);
}

The with(...) method is the thing that actually wraps the call to the await() method on the latch. I created a Duration class in the above case to capture the timeout constant.

This can be a neat little pattern which when used sparingly can lead to little nuggets of really readable code. However, the helper class that employs the static factory can feel unnatural as the method names don't really represent their function but rather their context within the DSL that the class defines. Using a private constructor and verifying the internal state is essential to ensure that the DSL can't be used incorrectly. Basically, if there are very few methods and you make sure they can only be called in a sensible order, I'd suggest its a good pattern to follow. The helper class is shown below in full.


public class CountDownLatchWithTimeout {

  private final CountDownLatch latch;

  private CountDownLatchWithTimeout(CountDownLatch latch) {
     this.latch = latch;
  }

  public static CountDownLatchWithTimeout await(CountDownLatch latch) {
     return new CountDownLatchWithTimeout(latch);
  }

  public void with(Duration timeout) throws InterruptedException, TimeoutException {
     if (!latch.await(timeout.inMillis(), MILLISECONDS))
        throw new TimeoutException();
  }
}

What Makes a Good Pair?

2008-12-31T09:15:00.006Z

Communication plays such a big part in what we do, we're always striving to be more expressive in our code, conveying our intent as clearly as possible. It's the same when pairing. We need to be expressive, communicate our intent, engage, listen and feedback. When we don't do these things, pairing just doesn't work. When we don't collaborate, we're not getting as much out of pairing as we could be. Perhaps it's only after working in a good pair that you realise what you're missing working in a bad pairing.

Be Explicit with the UI Thread

2008-12-30T12:00:00.007Z

Following up from the post SWT Support in Window Licker.

The thing I found most interesting about looking at this was having to be more explicit about the UI thread. When developing SWT applications, we're all aware that accessing almost anything graphical in SWT from any other thread than the UI thread spells "invalid thread access", but it was fun to be more explicit about the "UI thread".

When I started to look at how I go about writing SWT applications, I noticed that

I often run the application from a main somewhere.
I often discover invalid thread access problems at run time.
I often plug the problem by wrapping the offender in a Display.a/sync(...) call.
I often have very few long running processes that need to spawn a thread and update the UI.

It wasn't until I started looking at being able to run the application from the context of a JUnit test that I started to think in more detail about these.

Running the application from a main method

This seems simple but being more explicit about the UI thread means that this is worth a closer look. I always used to start an SWT application from the main, in variations of the code below.

The default constructor would do or delegate the work to setup the various shells and widgets and finally start the event loop. When using JFace's ApplicationWindow, I'd do pretty much the same thing. Calling window.setBlockOnOpen(true) just shifts the responsibility of starting the event loop to the ApplicationWindow class. If you call window.setBlockOnOpen(false) for example, you have to manually start an event loop.

However you do it, getting into that event loop blocks the client and its the thread that executes this event loop that is called the UI thread. It just so happens that most SWT applications will hit that loop from the main thread as you can see from the following partial thread dump.

Name: main
State: RUNNABLE
Total blocked: 0 Total waited: 0

Stack trace:
org.eclipse.swt.internal.win32.OS.WaitMessage(Native Method)
org.eclipse.swt.widgets.Display.sleep(Unknown Source)
com.objogate...SwtCalculator.startEventLoop(SwtCalculator.java:104)
com.objogate...SwtCalculator.main(SwtCalculator.java:113)

At this point, the SWT event loop is doing its thing, waiting for UI events (mouse clicks, keyboard input etc) and dispatching them to the appropriate listeners. We've also defined what we mean by the UI thread;

The UI thread or event dispatching thread is the thread that executes the standard SWT event loop.

Running the tests and GUI in different threads

So, if the event loop was called from within say a @Before annotated method in a test, the test would block until the event loop finished, the display would be disposed and any subsequent tests against the GUI elements would quickly discover that they no longer exist.

It should be pretty clear then that in order to test GUI elements the event loop has to be started in a different thread than the tests run in. The gotcha is that the test thread will likely want to interact with this UI thread in order to push buttons and make assertions and that's when we get into invalid thread access territory with SWT.

The way I implemented this was to use a class extending Thread to represent the UI thread and to start the event loop in its run() method. The tests can interact with the UI thread by either searching for the display with Display.findDisplay(thread) or by cooperating and ensuring that only the default display is used (retrieving it using Display.getDefault()).

A minor change to the application's main method is required to optionally not call the event loop when calling main. For example;

The tests can then explicitly create a UI thread delegating shell setup to the SwtCalculator class before starting the event loop and allowing the test thread to continue.

The UISetupClosure allows the setup code (in this case the main method) to run inside the UI thread. This uses the strategy pattern but an alternative design could just as easily sub-class the UIThread and use the template pattern in a similar way.

The interrupt allows the event loop on the UI thread to be interrupted and stop gracefully.

The following partial thread dump shows it in action. As you can see, a explicit thread has started the event loop (the Display.sleep and WaitMessage are the hints).

Name: SWT-Event-Dispatcher-Thread-1
State: RUNNABLE
Total blocked: 0 Total waited: 0

Stack trace:
org.eclipse.swt.internal.win32.OS.WaitMessage(Native Method)
org.eclipse.swt.widgets.Display.sleep(Unknown Source)
com.objogate.wl.swt.UIThread.startEventLoop(UIThread.java:90)
com.objogate.wl.swt.UIThread.run(UIThread.java:68)
- locked java.lang.Class@1d7fbfb

It was a good exercise exploring the UI thread and I encourage you to take a look at Window Licker (and my SWT patch). Have a play and see if you agree with the approach I took above.

SWT Support for Window Licker

2008-12-29T14:05:00.006Z

Window Licker is a framework designed to help you define your own fluent API for GUI testing. It provides a driver style API that your Swing or Ajax applications plug into before you go about writing GUI tests in a language natural to your application.

We've been doing some driver based GUI testing for our web apps using HtmlUnit so I was naturally interested in Window Licker with regard to rich client based testing. I've been trying to test my SWT based apps at the unit level as much as possible, but it's not always straight forward separating the behaviour of GUI classes from GUI elements. How do you test that all the validation behaviour is plumbed in correctly and fires at the right time in a text box? So, Window Licker seemed like a great candidate to getting some real user style testing. Great!

The bad news is that Window Licker doesn't yet support SWT. Boo! The good news is it gave me a great chance to have a go at implementing basic SWT support in Window Licker. Yey!

Check out the patch I wrote, or the thread where we discuss it.

SWT Applications on Mac OS X

2008-12-29T12:03:00.004Z

I've had a couple of problems running SWT applications on Mac, in particular, getting balloon tool tips to appear and incorrect invalid thread access errors. It seems that the Mac specific VM option -XstartOnFirstThread is the cure to what ails ya! It even addresses the bug I reported here!

It seems that by default, Java will not start on the main thread. Cocoa/Carbon/[insert mac vodoo here] starts it all and as a consequence, the SWT UI thread might get associated with it and not your main application thread. Therefore, whenever the application tries to access it (even though it looks like it owns it in your code), it may not and invalid thread access hilarity will ensue.

The flip side to this is that AWT/Swing apps and plugins might have problems.

Interfaces vs Class Imposterisers

2008-12-24T11:36:00.008Z

I've been mocking with JMock2 for a while now and fully buy into using mocks and driving out behaviour using interfaces. However, I'm not sure I get the resistance when people want to use class imposterisers.

I've been doing a lot with SWT lately and always want to unit test the GUI elements. One way I've managed to do this is to mock parts of the GUI framework using imposterisers. Unless I want a full on GUI brought up, there just isn't another way. The GUI API doesn't offer me any convienent interfaces and I can't even create sub-classed mocks myself as the framework prevents you sub-classing with run time checks.

I understand that by slowly teasing out behaviour during the mocking / TDD process you can clearly express the collaborations a class may have but I feel that you can still be clear about a classes behaviour when imposterising. It's just code right? I'm not sure I care that collaborations are documented as interfaces or not, my tests express the relationships and I've gone through the same thought process to understand clearly how the classes under test interact. So why is an imposter a bad guy?

In the case of legacy code or third part code, you're kind of stuck with imposterisers right? I mean you could build an application layer between your code and the third party stuff but I'd want better motivation for doing this than to avoid using imposterisers.

In the traditional description of an interface, we're taught to use interfaces when we want to share behaviour across multiple implementations. When driven to creating interfaces for mocking purposes we often only ever have one implementation in production which would seem at odds with this traditional definition. We do usually end up with a well designed applications which are nicely componentised and plugable, but we rarely plug anything different in. This in itself often leads us to testing the production configuration in larger end-to-end style tests.

So I feel that if you're clear about what's driving you to expose a public method on a class, your test exercises the collaboration clearly with mocks and your class under test is small and discrete then imposterising isn't really the devils work. You've considered what you're doing and why and at the end of the day, you've created a unit test that does what its supposed to.

If working naturally defining interfaces, then like me, mock away but at least consider not creating an interface if the only reason you're doing this is for mocking. If you do end up creating an interface just for mocking, don't do it just for mocking but take it further and be explicit in why you're formalising the collaboration. Think about the behaviour and let it drive your design.

JUnit and Threaded Tests

2008-12-17T18:52:00.011Z

I recently noticed a bit of a problem when playing with threads within JUnit. Take the following snippet as an example;

Here, the string "bye" is never shown, it would seem that when the test method finishes it somehow messes with the threads...

Looking into things, it seems in my case that naughty Eclipse is muddying the water! The RemoteTestRunner class which Eclipse uses to run the tests ends up calling System.exit(0) after it's run all the tests to kill the parent Java process. Even if we changed the daemon status of the thread, you'd get the same behaviour, ie, no "bye" being shown. Bad robot.

So, in some situations it is possible to interrupt a thread in a JUnit test bypassing the interrupt mechanism when running a test from Eclipse. IntelliJ actually seems to have the same behaviour but I can't review to source to confirm.

XPDay 2008

2008-12-11T18:55:00.006Z

Bit disappointing really. Things didn't get started properly will around 12:00 on day 1. I got to attend two really interesting talks of ~30 minutes each but then got trapped in a couple of nightmarish 'Open Spaces' which ended up being a rather self-indulgent discussions about nothing in particular.

New Blog

2006-03-14T09:19:00.002Z

Moved to a new blog today at blogspot.com. How exciting does life get?