New features and extensions in Mustache.java


New features and extensions in Mustache.java

Almost a year and a half ago I talked about my implementation of mustache, which I call mustache.java. Since then, we used it to develop Bagcheck and it has also been adopted for a variety of use cases by Netflix, Twitter and others. In my post I mention that for all intents and purposes, the template engine was ‘finished’ and I would say that for the most part, that was accurate. However, there have been a couple of things that have been done since then around optimization and also some interesting extensions that I would like to share with people to get feedback.

First though, I will shortly recap the biggest differentiating feature of this implementation over other implementations:

  • Concurrent, streaming template evaluation

Mustache.java exectutes the template in such a way that partials, loops and even replacement (if you use futures) do not block the further evaluation of the template. This allows you to quickly launch all your I/O (or CPU) bound work for the whole template in context during a quick evaluation and then the engine can do things like flush to the underlying writer incrementally as work completes giving you great client-side and server-side concurrency. You can also use this to defer sections of the page to write them out later, a la Big Pipe. This has been invaluable to easily maintaining Bagcheck’s low-latency design. The downside to this is that you need to write your Java code in a functional style, I suggest that you use Google collections to do it right. The design of this has remained unchanged since the initial release though recently I added the ability for the template backing code writer to take control of concurrency in order to reduce context switches and garbage creation. This can get you a 5x improvement in template evaluation performance at the cost of developer time and expertise without losing the ability to do concurrent evaluation when necessary.

Since the first release, the handlebar server has been included to allow you to easily view mustache templates fully rendered in the browser with only JSON data as your backing code. This lets you quickly iterate on design without having to have the full stack running with enough information for someone to figure what needs to be implemented on the backend when new things are added. Sometimes though it can be painful to get realistic data for new features using production data. In those cases, I had always imagined a method to take a template and rendered content and give you back data would generate that same content in combination with the template. That functionality is now in the master branch (since 0.6-SNAPSHOT). Given any Mustache instance you can now call unexecute(content) on that instance and if it isn’t ambiguous or in some other way irreversible you will get back a Scope object that contains the data. Using toJSON() on that object will provide mock data that can be used with handlebar. Here is an example:

[gist id=1274327]
[gist id=1274330]

We then take the template and the output and unexecute them using the new method on Mustache objects:

[gist id=1274334]

This produces the following JSON (after pretty printing):

[gist id=1274337]

What it really does is turn the Mustache language into a look ahead grammar that can be used to read data from content, though that content has to be stringently authored. Another use case that seems reasonable to me is for people to check that their complicated backend code is combining in the right way with their template. Something like an integration test for template processing without having to hard code the non-template interpolated text into your test. Would love to get feedback on this feature as it is still unclear whether it will be useful to a wide audience.

The most recent feature added is support for Twitter-style functions (_ vs just #) and i18n support. Long ago I had added support for # functions: you can return a Google Guava Function from the value callback and after the engine executes the enclosed template it will call your function to transform the value. This works for simple i18n and caching but fails for more complicated i18n applications. In those cases, you may want to also change the template itself which means you will need a callback before the underlying template is executed. If you need that behavior, you instead return an instance of TemplateFunction which gets called back before evaluation and then compiles the template text that you return from the function invocation. This is most often used to change the order of words or phrases that include things like user names. Here is a simple example from the tests:

[gist id=1274365]

The biggest change since the release though is the addition of the MustacheBuilder. In 0.6 I will be deprecating MustacheCompiler as not only does the Builder not require the use of the Java compiler, but it also compiles almost instantly and performs better than the Java compiled versions of the templates. The Builder also gives us a lot more runtime flexibility that I am going to take advantage of over time for a variety of optimizations that are in the queue. I may return to investigate using ASM for compilation though in order to take advantage of JDK 7 and invokedynamic as that becomes the standard JVM in deployment.

You may not be familiar with some of the things that you can enabled while it is running to make debugging and profiling easier. There are 3 system properties that can be turned on with little documentation:

  • mustache.debug: output a warning when a callback is absent or null
  • mustache.trace: create a multithreaded trace, use MustacheTrace.setUniqueId() in each thread, output with MustacheTrace.toASCII()
  • mustache.profile: time all callbacks, output the top 10 total + average with Scope.report() and then reset the profiler

Now is the time for input. Here are some things that I have on my TODO list:

  • Full JDK7 invokedynamic support with ASM generated classes
  • Extensible templates a la Django templates
  • Native Scala support based on Twitter’s util-core
  • Generate scaffolding in a variety of languages for backing code with verification

Would love to hear from users and prospects. You can either comment here or join the dicsussions on the Mustache.java group.