Looking on the web for xml/xslt benchmarks, I ran into the Sarvega XSLTBench benchmark. In the test are a bunch of stylesheets that are pretty “real-life” – ones you might run into in a real situation. They run from medium to very complex when you look at the transformation complexity, from medium to huge in stylesheet size, and overall they are a good model of real life cases.

There’s only one problem with the benchmark: it is single threaded.

So, I wrote some code to start a large number of threads, and had them each work over the same transformation repeatedly. Wrap that in a thread that runs it for about half an hour, and presto, results.

I tried this on the T2000, of course, and on two other computers. First, the Pentium IV at 3.2 GHz I have been talking about. That, unfortunately, didn’t work at all. The machine is running FreeBSD, so to get Java on it, I had to compile from source (with gcc), with a linux compatibility layer, and this simply doesn’t perform. The results were utterly useless. If anybody has a fast intel box that can be tested, if possible with solaris 10, or otherwise with linux, please let me know. Next, I tried it on my Mac. A dual-core G4, with 1 Ghz processors. And oldy, but it has served me well, and OS X has a decent Java on it. Tomorrow I can test with two sun V440′s, so that’ll be interesting as well.

First, the sources. If you already have the xalan and saxon libs, download this file, if you don’t, or you want to make sure you’re testing with the same version as I am, download this file, but before you do, make sure you understand the licenses that come with xalan and saxon at their respective web sites.

Next, some results. First, let’s look at the fastest transformation the systems were able to do. This was, amonst the thousands of transformations done during the test, the fastest single one. This sort of tells us what the maximum speed was the system could muster for one transformation (time is in milliseconds, lower is better/faster):

There’s two things that stand out: on most transformations, my Mac isn’t that bad. It’s 1 GHz G4 chip can keep up with a 1 GHz core from the T2000, except for the very complex chess transformation (which basically takes the moves from a chess game and draws a chess diagram for each move – about as complex as you can get when it comes to transformations). Next, it’s clear saxon is faster. But before you toss you xalan libraries out the window: test, test, test. The chess transformation didn’t work with the latest saxon (version 8 ), and I’ve been seeing weird “out of file descriptors” error messages with it – it may be leaking file handles. I’ll probably file a bug report later on.

So, given the same amount of time to do transformations, which of the two systems managed to do the most work? The T2000, of course, since it can bring 32 CPU’s to the table where my old Mac only has two. Here are the numbers (counted transformations, higher is better):

Number of transformations done:

	T2000/saxon	T2000/xalan	Mac/saxon	Mac/xalan
chess	29310	10100	756	1052
game	968610	865870	31807	27629
nitf	693270	580570	19491	19260
recipes	363270	247490	13124	14296
table	905800	558570	23317	22697
topic	335350	230930	12093	13311
wai	770380	110780	8715	9261

As you can see, the T2000 blew my Mac just completely out of the water, the difference is much more than you’d expect by simply multiplying the Mac results with 16. It shows you how bogged down a system can become when it has to juggle many threads. The T2000 really, really shines with this kind of work. My guess is that a single-CPU pentium will suffer the wrath of a T2000 with this test as well, but I’m counting on my readers here…

And here’s the average time it took the systems to do a transformation (time is in milliseconds, lower is better/faster):

	T2000/saxon	T2000/xalan	Mac/saxon	Mac/xalan
chess	334	988	13207	9557
game	9	11	313	361
nitf	13	17	510	520
recipes	25	40	760	697
table	10	17	430	445
topic	27	43	825	747
wai	12	90	1140	1070

Again, the difference is huge. The individual processors in a T2000 may not be the fastest in the market, but it does keep going long, long, long after other systems are bogged down with load. Note how close “fastest” and “average” are with the T2000, compared to what my Mac could do.

If all goes well, I’ll have some results from the V440 systems from Sun tomorrow. If you have some systems that can be added to these tables, please let me know!