Apple needs to take better care of their Java developers

Building applications on the Apple platform would be a lot better if they would put Java on the same or better footing than their Objective-C runtime.

You might expect this entry to be a rant about how Apple hasn’t released a beta or even an alpha of Java 1.5 for Mac OS X. Actually, that is a perfectly good topic, but in fact, this entry is not about that. This is about the less than stellar support Apple has for writing native Mac OS X applications using Java. You might say, “But Java is supposed to be cross-platform! Just use Java APIs for everything!”. That’s fine for most things, but if you want an embedded HTML view or need an application to look exactly like a native application, that isn’t an option. Additionally, on the Mac you have an application called Interface Builder , that is quite possibly the best GUI building tool in the universe.

So, in fact, the title was just a ploy. I’m actually going to complain about a wide-variety of Apple development shortcomings. This whole story started a few days ago when I decided I don’t much like my RSS reader and wouldn’t it just be grand if I wrote my own. A friend of mine had been discussing Sherlock 3 and I had been playing around with its XQuery engine and thought that it might be the perfect development platform for such a thing. If you are unfamiliar, Sherlock 3 is a thick client application that downloads Mac native user interfaces and some scripting code that in combination create a “Channel”. These channels can then be used locally but typically perform some web service but with a better UI than you can get from your browser. You might respond, “I can make anything in DHTML, how dare you!”. Well, I can’t, and since you can use Interface Builder to make Sherlock channels I thought that would be an easy way to go.

Sherlock 3 Ebay Channel:

And I was right, for the user interface part. That took about 5 minutes of UI building to get a decent 3 pane setup with split-panes, a subscription field, a subscribe button, and a progress indicator. Then you give them all names so Sherlock can talk to them. Things like “RSSReader.SubscribeButton”. There is a webview at the top for ads (little experiment with google adsense) and another webview below the entry table for entry content. The web view panes are actually embedded Safari pages, so they should look good.

RSS reader user interface:

On the Sherlock scripting side you basically have two different languages that you can use to implement callbacks. One is JavaScript, the other is XQuery. This is where I got my first bit of bad news about Sherlock. It turns out after a few stumbles trying to use the things that I found in the JavaScript user guide I found on the net that Sherlock has JavaScript 1.0 embedded in it. Yes, you read that right, 1.0. Netscape 2.0’s implementation. It doesn’t even support Array properly. Then I found a hopeful message on usenet where an Apple employee was announcing the checkin of the JavaScript engine used in Safari to OpenDarwin with the P.S. that it would be in Sherlock real-soon-now. Too bad that message was posted in June 2002. This is when I decided that Sherlock 3 is not getting proper care and feeding and I should give up. I did however take the time to play around with the XQuery engine that is there. Contrary to the JavaScript experience it appears to be a faithful implementation of XQuery 1.0 which is relatively up-to-date and quite useful. Here is an example using their XPath Finder against the Yahoo Finance site. Notice that it is showing the current XPath along the bottom and the content that it matches in the display:

The only oddity, is when I was using their XQuery evalutation developer channel. When I put in XQueries that would resolve down to text entries, like say $html//a/@href, it would properly return the href attribute of the nodes. However, when I tried to put in queries that would return actual nodes, it would come up with this curious result:

Since I had already decided that I wasn’t going to use it due to the JavaScript problems — wake up Apple, this is not a real language for Sherlock, this was just icing on the cake, and I said goodbye to Sherlock. However, our friend the mysterious NSCFType will come back for a surprise visit a little later. So what was my next option? Well, I knew that I could get a good XQuery engine for Java and that I should be able to use the same Interface Builder interface with it under what Apple calls the Cocoa-Java bridge. This is a set of classes that are mirrors of the Cocoa classes that exist in the Objective-C based, NextStep derived APIs. Here is a quick intro on how they structured their APIs:

Level 1: Core Foundation C-based API
Level 2: Cocoa Foundation and AppKit Objective-C API with “Toll-free bridging” of some CF types
Level 3: Cocoa-Java Bridge to some (as I found) of the Cocoa APIs

So if you are using some base type like an com.apple.cocoa.foundation.NSDictionary it is actually a JNI wrapper calling an Objective-C wrapper calling a C function. This is less than ideal, but it gets the job done for the most part. To continue building the application in Java, I created a new Cocoa-Java Application in XCode and pulled in the NIB (Next Interface Builder) bundle that I had already built for the user interface. To that interface I added a controller, made a bunch of outlets (data) and actions (events) for the interface elements, and hooked it all together. Then interface builder will generate either Obj-C code for the controller class or Java code. I picked Java, it automatically added it to the project, and then tested it to ensure it was hooked up correctly. As soon as I tried to build my heart filled with dread. I got the error “Cannot resolve class ‘WebView’”. A couple of the outlets had been declared to hold the WebView components in the UI. This error was it subtly telling me that it couldn’t find a Java version of the Cocoa WebView class. Google to the rescue… alas the Safari WebView SDK has only been released for Objective-C. ARGH!

That quickly ended my foray into writing the RSS reader in Java since I definitely wanted to use a real browser to display the content of the entries when I selected one. This was starting to get a bit frustrating. So I breathed deep and thought about some possible solutions to my problem. It became clear that what I wanted to do had to be done in Cocoa, so I was going to have to learn some Obj-C and that I was going to need an XQuery engine. There doesn’t seem to be an easy way to mix Java and Cocoa code so I couldn’t use the Java version. Then it hit me… Sherlock is written in Cocoa and so must its XQuery engine. So I scanned the public Apple frameworks and didn’t come up with anything. Then I noticed this PrivateFrameworks directory that looked pretty suspcious, looked in it and found the XQuery framework. This just might be too easy! So I fired up a new project, this time a Cocoa Application, added the NIB file as before, this time generating an Objective-C stub for my controller. I then added the newly found XQuery framework to my project.

If you are unfamiliar with Frameworks , they are an Apple creation that is quite marvelous. They are bundles of files that include version information, libraries, and headers for functionality that you might use in an application. When you add one to a project, the headers get automatically added, and the libraries are automatically linked in at build time. In some ways they are equivalent to Java .jar files except that they can have sub-frameworks and a lot more meaningful meta-data. Some examples include the Apple JavaVM, the Core Foundation libraries, Quicktime libraries, etc. The base system for Mac OS X 10.3.3 has almost 200 frameworks for using various APIs in the system. There was only one problem with the XQuery.framework (aside from it not being public), it included no header files. Maybe I am a hacker at heart, or maybe I am just crazed, but I ran the “nm” program against the dynamic library in the framework and low-and-behold there were a bunch of things that almost looked like .h files sitting there in the output:

…..
7afe18ec t +[XQuery xqueryWithSource:]
7afdea04 t +[XQuery xqueryWithSource:andFlags:]
7afd6e08 t -[XQuery baseURI]
7afd390c t -[XQuery evaluateNotifyingClient:]
7afe19b0 t -[XQuery evaluate]
7afdca4c t -[XQuery flags]
7afd71c0 t -[XQuery globals]
7afe1a58 t -[XQuery initWithSource:]
7afdb28c t -[XQuery initWithSource:andFlags:]
7afdf284 t -[XQuery input]
7afe1b88 t -[XQuery xqueryEvaluateComplete:withResult:]
…..

After reading all about how great this Obj-C language was at dynamic dispatch I thought that this might be a good time to test it. I wrote a quick XQuery.h file:

@interface XQuery : NSObject { } @end

Notice I didn’t put any methods on it. They are all discovered at runtime anyway by the Obj-C runtime, much like using reflection in Java to call all your methods. The only problem with a .h file like that one is that you will get compiler warnings telling you that the object might not accept the message that you are trying to send to it. If I later wanted to get rid of the warnings I can just add the declarations then. So instead of playing around with this API from my RSS reader UI I quickly whipped up a UI with an input box, an output box, and an execute button to try and use XQuery like they were doing in Sherlock. After some missteps I eventually get it to exactly the same stuff as the Sherlock channel. Unfortunately, that includes returning NSCFType. The NSCFType is completely useless to me. After googling for a second I find that it basically means that Apple is lazy and they didn’t bother to wrap up the type that you are getting back from the Core Foundation classes with a Cocoa version so you can’t use this thing except from the C-API. This is similar to the problem we were seeing when I tried to access the WebView from Java, except…. there is a solution. These clever code monkeys actually went and wrote wrappers for the classes that I needed. As it turns out, the XQuery API was returning CFXMLTreeRef objects. The CF XML parser and the CFTree collection are a couple of the things that Apple doesn’t ship wrappers for. Maybe if they did, their Sherlock channel would work. Lazy, lazy, lazy!

This whole experience affected me in a deep and meaningful way, so I thought I would work off some of the frustration by making a Framework that people can use to access the XQuery framework built into the system that Apple doesn’t give us easy access to. I decided to this by writing a Groovy script to extract all the symbols from the “nm” output and write .h files for me. Here is the script and the framework:
processxquery.groovy NSQuery.tar.gz

You can use this in your Objective-C code quite easily by placing the framework in your ~/Library/Frameworks directory and then adding it to your project. Then you use it in code like this:

#import <NSXQuery/NSXQuery.h> ..... NSString* result = [NSXQuery query:@"...your xquery here...."];

All the juice is there for you to use the results as Nodes if you wish, but I was only interested in text so thats the only higher-level API that I wrote. Now that I have my desk cleared off, the 5 books I used to figure all this out put away, and the 10 bookmarks for reference I added filed away, its now time to use all these tools to build an RSS reader. For some reason, with all the challenges out of the way, I sort of feel like NetNewsWire is good enough now. :)

Hopefully someone at Apple reads this and begins to understand that people want to program in Java on their platform and are being held back by its second-tier status. Basically I think it comes down to them ignoring the fundamental rule of integrating systems: provide high-level custom APIs targetting the most used parts of the underlying system and additionally offer a generic, automatic way to access all of the other functionality. Who knows, maybe someone will send me an email showing me how they accessed the WebView from Java.