BookSmarts!

There are a few ways to invoke Java code from within Javascript, and when I made Screengrab, I used the easiest:

var list = new Packages.java.util.ArrayList();

Basically, to use any standard Java packages from within Javascript, you just use the fully qualified package name and append “Packages” to the front of it. This is kind of cumbersome, but you can ‘include’ classes by just assigning them to a local variable:

var ArrayList = Packages.java.util.ArrayList;

var list = new ArrayList();

Now, this is all fine and it will get you through most Java work in Javascript (these concepts have worked fine for Screengrab!), but what if you need to implement a Java interface from within Javascript?
Well, I’ve tried. When I was trying to get clipboard support for the Java-based grabber in Screengrab I looked for answers and tried a few things, but I just couldn’t do it. I eventually gave up and currently Screengrab will only copy images to the clipboard using the Canvas-based grabber.

The only way to do this is to somehow create a Java class externally and then load it from within Firefox. This is possible, because Firefox lets you create components in other languages and link them in using some magic. The problem with components is that there is a massive amount of crap to write to get one to work.

Here is a tutorial on creating a Java component for extension development that I found. This is the Java class it is trying to get access to. And this monolithic file is the Javascript code that you have to write to get Firefox to understand it. I haven’t read it. I stopped about 2 pages in and thought “there has got to be a way to do this that doesn’t suck”.

And there is…

From this page at Mozillazine we find “the easy way” (and actually, as it turns out, the easy way comes from the same guy that did the hard way.

As you can see from the tutorial, it boils down to using a ClassLoader. Essentially, since we can access most standard Java from inside Javascript, it stands to reason that we can create ClassLoaders and use reflection. And that’s what this code does. What it doesn’t do is tell you how to do this with a local url, pointing to a jar within your extension. Here’s how…

var id = "booksmarts@getbooksmarts.org";
var ext = Components.classes["@mozilla.org/extensions/manager;1"]
    .getService(Components.interfaces.nsIExtensionManager)
    .getInstallLocation(id)
    .getItemLocation(id);

var cl = new Packages.java.net.URLClassLoader(
    [new Packages.java.net.URL(
        new Packages.java.io.File(ext.path + "/components/tiny.jar").toURL())]);

var aClass = Packages.java.lang.Class.forName("Tiny", true, cl);
var instance = aClass.newInstance();

ftst.logger.debug(instance.getMessage());

I’m pointing it to the components directory in my extension folder. When the extension is installed, the xpi is unzipped, but the chrome.jar isn’t, so if you want to load the your classes, you’ll need to put them somewhere that gets extracted. “components” seems like a good place for that, since it contains libs and the like.

Now obviously this is a bit ugly, but it could easily be wrapped in some code that does most of the work for you. To the point where you should be able to write code that looks like the following (or much much better):

var tiny =
    jarLoader.using("tiny.jar")
        .createNew("org.getbooksmarts.Tiny", arg1, arg2, arg3);

One of the good parts about including things with scope is that you don’t have to worry about changing them into a more extension-friendly namespace format, which is nice.

So I didn’t have to adapt log4js to get it to play nicely in chrome at all. However, I did have to change it to make it a little more Andy friendly.

Basically, I wanted the log output to hit the console (using dump()) and to look like…

startTime logLevel [fileName:lineNum] message\n
eg.

23/03/07 22:10:11:342 DEBUG [overlay.js:14] A debug message
23/03/07 22:10:12:589 DEBUG [overlay.js:21] Another message

Log4js can’t log to a command prompt using dump(). That was an easy change, we just needed to add another appender, as below.

/**
 * Appender writes the logs to the Shell of Firefox when it is launched with
 * firefox -console (and has the necessary properties enabled)
 * PLEASE NOTE - Only works in Firefox
 * @constructor
 * @extends Appender
 * @param logger log4js instance this appender is attached to
 * @author Andrew Mutton
 */
function FirefoxConsoleAppender(logger) {
...
	this.layout = new MozStackLayout(true);
}

FirefoxConsoleAppender.superclass = Appender.prototype;
FirefoxConsoleAppender.prototype = {
	/**
	 * @see Appender#doAppend
	 */
	doAppend: function(loggingEvent) {
		dump(this.layout.format(loggingEvent));
	},
...

So now we can dump to the console. Notice that there’s a new Layout there too? You probably didn’t, since not many people use log4js I imagine.

The new layout is there to take advantage of the information we can get from Components.stack - notably, line numbers and source files.

/**
 * MozStackLayout is a layout that takes advantage of stack data from Mozilla.
 *
 * startTime logLevel [fileName:lineNum] messagen
 *
 *
 * @constructor
 * @extends Layout
 * @author Andrew Mutton
 */
function MozStackLayout(showDate) {
	this.LINE_SEP  = "n";
	this.showDate = showDate;
}
MozStackLayout.prototype = {
	format: function(loggingEvent) {
	    if (this.showDate && Log4js.simpledate) {
    	    var d = new Log4js.simpledate.SimpleDate(loggingEvent.startTime.getTime());
          	return d.toFormattedString("~d/~k/~y ~H:~m:~s:~S") + " " + loggingEvent.level.toString() + " [" +  loggingEvent.getStackFileName() + ":" + loggingEvent.getStackLineNum() + "] " + loggingEvent.message + this.LINE_SEP;
	    } else {
	        return loggingEvent.level.toString() + " [" +  loggingEvent.getStackFileName() + ":" + loggingEvent.getStackLineNum() + "] " + loggingEvent.message + this.LINE_SEP;
	    }
	},
...

this is the bit that uses simpledate: to format the date in a less ugly way than you usually get for free from Javascript. I honestly have no idea why there isn’t a nice date formatter built into the language. It is ridiculous.

There’s a few more changes here as well. I had to change the log() method so that it attempts to get stack data (if it’s available).

log: function(message, logLevel) {
    if (Components && Components.stack) {
        var stack = Components.stack.caller;
        if (stack.toString().indexOf("log4js.js") != -1) {
            stack = stack.caller;
        }
        var loggingEvent = new Log4js.LoggingEvent(this.category, logLevel, message, this, stack.toString());
    } else {
        var loggingEvent = new Log4js.LoggingEvent(this.category, logLevel, message, this);
    }
...

and also LoggingEvent, so that it can give the stack information

getStackFileName : function() {
    if (this.stackInfo != null) {
        var split = this.stackInfo.split(" :: ");
        var fileName = split[1];
        if (fileName.lastIndexOf("/") != -1) {
            var pathSplit = fileName.split("/");
            fileName = pathSplit[pathSplit.length - 1];
        }
        return fileName
    }
    return null;
},

getStackLineNum : function() {
    if (this.stackInfo != null) {
        var split = this.stackInfo.split(" :: ");
        var lineNum = split[3].split(" ")[1];
        return lineNum;
    }
    return null;
}

So there you have it. Not very interesting today, but this should be the last infrastructure post. I think we can begin doing things properly now.

I’m very interested in getting log4js to be configurable via a properties files (ala log.properties in log4j). This doesn’t seem too hard, so I may have to check it out. An alternative to log.properties would be to write the configuration in JS. Something like…

logger("booksmarts.log").atLevel("DEBUG")
    .withAppender("FirefoxConsoleAppender")
    .andLayout("MozStackLayout");

this would be too simple though. Maybe something more elaborate. Who can really say eh?

I’ve finished (tenatively) my work on includes and it’s working really well.

There’s a reasonable example of its use in the code that sets it all up, include.js. In particular, you should take a look at the ftst.configureLogger() method and the ftst.include(ftst.getMain()) call at the bottom.

Basically, a call to ftst.include will include the specified file in that place. A call to ftst.includeInScope will include the file within a confined scope and return an object that holds that scope.

Why do I care about scoping so much? Because all extension development in Firefox is in a shared scope. Your code is in everyone else’s way and theirs is in yours. That’s why it’s so important to compartmentalise your code. This isn’t such a problem in normal web-development. In a webpage all of your JS shares the same scope, but you control it all. In extension dev there can be all sorts of stuff running around, which is a nightmare. Because of this, it’s best to limit your footprint as much as possible.

In include.js, you can see that I’m using Log4js to do logging, and I’m including it in a hidden scope, so that it doesn’t get into the global namespace. I’ve changed it (I’ll explain why in my next post) and now it also uses this date formatting code, which I configure into it after it’s initialised (from another hidden scope). Why don’t I just include it directly? Well, because there are…

Problems with mozIJSSubScriptLoader and scope
Remember that this is the code that sets up the loader.

var jsLoader = Components.classes["@mozilla.org/moz/jssubscript-loader;1"]
    .getService(Components.interfaces.mozIJSSubScriptLoader)

then we call (1)jsLoader.loadSubscript(filename) to load a JS file into the global scope, or (2)jsLoader.loadSubscript(filename, localScope) to load a JS file into the variable localScope.

The complication happens when we try to include one JS from within another. You can call (1) from within (1), (1) from within (2) and (2) from within (1). What you absolutely can’t do is call (2) from within (2). You can’t include something in a localScope when you are a script that is being included in a localScope no visible error occurs, but your script will halt and nothing will happen.

That would be alright, if it weren’t for this problem: A script included in a global scope from a script that is being included in a localScope will be globally available outside the local scope. If you include (1) from (2), (2) is confined to its localScope, but (1) is completely global. This seems wrong to me. It could be argued both ways, but I think that a script included from (2) should be confined to (2).

Anyway, what this means is that for Log4js to use simpledate, simpledate must either be included globally (by include.js or Log4js) or it can be included locally by include.js and then set into Log4js after it is loaded. I care a lot about scope (as you can tell) so I’m doing the latter.

The very worst thing about Firefox extension development is the shared JS context that every extension shares.

PS
I know that dojo does something similar to this, but from reading, it doesn’t seem to do it right (particularly for chrome) and also does a lot more than I want.

Doing any sort of development requires logging and dumping stuff to a console. Firefox extension development is no exception to this (Venkmann is useful, but he sure can be a pain).

There’s two ways that I know of to do this… The first outputs to a shell and the second uses the Error Console.

Using dump()
Firefox has a function called dump() that will just dump whatever you give it to a standard console. To use it you have to…

• Start Firefox in console mode - from a command prompt type “/path_to_firefox_app/firefox -console”
• Set “browser.dom.window.dump” to true in about:config - the excellent Extension Developer’s extension can do this for you.
• Use dump("My message\n"); to dump to the console.

It isn’t pretty, but it works.

The Console Service
The other way is to use the console service, which will put debug statements into Firefox’s Error Console.

var consoleService = Components.classes["@mozilla.org/consoleservice;1"]
        .getService(Components.interfaces.nsIConsoleService);
    consoleService.logStringMessage("My Message");

It’s easier, and requires less config, but it isn’t working so well for me right now. Sigh…

What we really need for logging in Firefox is a log4j style logging framework. That would be super sweet!

I forgot that you have to set the MIME type for XPIs right, otherwise you get that annoying problem where someone who clicks a link for it just ends up seeing a page of bytes that starts with PKZIP.

The MIME type for XPIs is “application/x-xpinstall”.

(this is mostly just a mental note for me - I always forget to configure this on my servers)

Having spend the last couple of hours trying, I’d almost concluded that this was impossible. And it was! The way I was trying to do it.

The dynamic injection of script was stupid. It won’t work in XUL (but it will work in HTML).

The secret is found in two things.

1. An XPCOM interface (isn’t it always?): mozIJSSubScriptLoader
2. A JS hook into an XPCOM interface: http://www.mozilla.org/scriptable/components_object.html#_stack

The first one lets you load Javascript dynamically within XUL apps. You call it as follows…

var jsLoader = Components.classes["@mozilla.org/moz/jssubscript-loader;1"]
    .getService(Components.interfaces.mozIJSSubScriptLoader);
jsLoader.loadSubScript("chrome://booksmarts/content/overlay.js");

The cool thing is that you can also pass it in a scope object, so all of the members of the JS file you load will be attached to it. If overlay had a function called jon(), then the code above would’ve loaded it up into the global scope and I could just call jon() to access it.

But if I’d called loadSubScript(chrome://booksmarts/content/overlay.js, anObject) then I would call anObject.jon() to access it. Brilliant!

So we can load Javascript. But I said I wanted to do it with one file. Now this is where it gets hacky. In HTML, my one-file plan involved navigating the DOM to find my script element and then getting the src value from it. So the script tag would look like <script src="include.js?main=another.js">.

Well it turns out that doesn’t work in XUL. As far as I can tell, the script tags are destroyed after they’re loaded in an overlay (document.getElementById can’t find it, so it must be lost). So instead, I discovered (accidentally while I was trying to find out how to load scripts) that you can get a Javascript stack dump from XPConnect by calling Components.stack. This yields something like:

JS frame :: chrome://booksmarts/content/include.js?main=chrome://booksmarts/content/overlay.js :: anonymous :: line 56

Notice that Firefox loaded the script perfectly, despite the URL params AND it preserved them into the stack trace. AWESOME!

If you assign that to a string and split it with split(" :: ") you get an array, the second value of which is your url chrome://booksmarts/content/include.js?main=chrome://booksmarts/content/overlay.js.

From that, a handy bit of URL parsing courtesy of a helpful person on the internet and you get your parameter. Brilliant.

That’s basically all of the pieces you need to do dynamic imports in Javascript. Which I’ll continue next time…

Today, I made the project files for BookSmarts.

You used to have to do lots of thinking and reading to make an extension; now a handy chap called Ted has written an extension wizard.

Ted’s wizard makes a perfectly structured base for starting extensions, all bundled up into a neat zip archive. He even includes shell scripts for packaging it all up into an extension file.

If you’re very new to Firefox extension development, then you would do well to read the Mozilla docs on the files in the extension package. They give a good overview and I’m much more interested in writing code than explaining things that have been explained over and over by people much better at it than me.

This isn’t a single page tutorial, it’s more of a log of the pain of extension development, a guide as to how “I” do it (it’s my way, not THE way) and, hopefully, a source of answers for common problems that tutorials won’t cover so well.

Anyway, I used Ted’s tool as a starting point for BookSmarts. I like to put my chrome in a chrome sub-directory (which helps with my Ant replace script later) so I made some changes to the chrome.manifest (I added “chrome/” in a couple of places).

If you download the folder and run ant, it should build an xpi for you to try in Firefox. It should all work.

Today is a lazy Saturday and I’ve decided it’s a perfect day to start writing this extension. Luckily I don’t actually have to do that much to start. It’s all getting the file structure right and configuring the dev environment. Admin stuff really (the exciting stuff was getting the server to work for this website - you missed it).

Dev environment
I’m a Java developer by day (now working at an investment bank in London), so I have ways I like to do things that come from my Java life. You’ll find this in everything I do. It’s just how I am. I’d like to get into other languages (Ruby) but scripting languages don’t have the amazing refactoring tools that I love from Java. When they do, I’m there!

1. I use Eclipse. It’s a choice and it’s probably overkill for extension development, but it’s what I use.
2. I use the JSEclipse plugin to get semi-decent JavaScript development going from within Eclipse.
3. I use Subversion to hold all of my stuff in source-control (normally on my local machine, but for this I’ll put it up on the website).
4. I build my extensions using Ant. I have a script that works and I use it.

The SVN connection url is http://getbooksmarts.org/svn/booksmarts/. Anonymouse access is enabled. If anyone wants to contribute get in touch and we’ll see what we can do. You’d have to write about how you’re doing stuff on here too.

If you ever checkout the SVN repository, you’ll have my source, project and Ant scripts. It’ll all work without Eclipse, but it’s all integrated in there so I like it.

JSEclipse was recently bought by Adobe, so I don’t know what they’re going to do with it. Hopefully it means money will be there and they can make it REALLY good. Refactoring support for JavaScript would be really hard and REALLY awesome.

Firefox (post 1.5) lets you work on extensions in place, and tell it where to find the source using a pointer file in your profile. This is probably the most useful thing they added for extension developers. The only thing it requires is a special testing format chrome.manifest file. I use the testing chrome.manifest and then produce the package format (using Ant substitutions) when building a deployable. You shouldn’t have to think about it if you use my script.

There’s always a better way to do stuff. I just do what seems best and stick with it. Tell me if I’m being dumb.