GNU Classpath


It’s been a while since I last blogged and I’ve since made quite a bit of progress. Today, I committed the last patch that gives JikesRVM support for three of the management beans. The remaining ones focus on memory and threading. For the former, I’ll need to take a more detailed look at MMTk over the next few days, while the threading bean will require me to merge my branch back with HEAD, as Ian’s made some structural changes in that area.

In the long run, it would be nice to not just implement the bean interfaces but also to provide additional information. Sun already does this; their OS bean for example contains more data than the management interface specifies. Really, anything about the VM that people want to monitor should be available through these beans.

Peter Donald has also kindly committed my patch since I last wrote. This means that the Classpath examples can be run with HEAD as well as on my local branch. Prior to this patch, the gnu.classpath namespace was blacklisted, stopping gnu.classpath.examples.* being used.

It would be nice to see another Classpath release soon, so I’m going to try and look into helping Mark out a bit this next week by running Mauve and looking for any regressions. There has been a lot less coding, but there are some important patches that should really be released. I’d also like the next release to support building JikesRVM, which means making sure my patches get in. I still have no clue how to implement hex string parsing though, and no-one else has come forward ;(

OpenJDK and CACAO

I’ve also been following twisti’s instructions on installing OpenJDK with CACAO this week. The advantage of this for me is that it means I can have OpenJDK on ppc64 :) In the process, we uncovered problems with the Math trigonometry functions. sin(0.7) seems to wrongly return 0.7 on ppc and ppc64, and some negative value on alpha. According to the reply we got on the core-libs mailing list, this is because the functions are implemented intrinsically to the VM for Math and only via fdlibm for StrictMath. Of course, gcj gives the right answer :)

twisti gave me a patch for solving a problem with ioctl on 64-bit archs over the weekend, and so I’ll be trying again with that and b17 tomorrow. Hopefully, we may then get a successful build.

Finally seen some success this morning. Firstly, JikesRVM now boots. It won’t load the classes I want it to from a zip but it does boot. I found the solution following some advice from Ian Rogers last night. I copied java.lang.Runtime (the failing class) into the library interface and added a few VM.sysWriteln calls to get some debugging info. As I guessed from looking at the code, it seems the Runtime isn’t null and so an error is thrown. JikesRVM then fails to throw the error because not enough of it is there yet.

The solution is actually quite simple: I extended the test to also check that VM.fullyBooted is true. If it isn’t, it still goes ahead and initialises Runtime and this gets things going to a point where JikesRVM goes ‘oi, specify some classes!’ (well nicer than that, but you get the jist).

Unfortunately, actually specifying some doesn’t seem to work:

Boot sequence completed; finishing boot thread
java.lang.NoClassDefFoundError: Could not find the class gnu.classpath.examples.management.TestClassLoading:
gnu.classpath.examples.management.TestClassLoading

Iced Success

To top it all, I just finally got a build of IcedTea through!

IcedTea is served: openjdk/control/build/linux-amd64

This is on Debian stable with a copy of gcj svn lurking around (from my earlier attempts). Biggest problem was getting it happy with a compiler; I had to use ecj 3.2 in the end from the JikesRVM build as it didn’t like my version or javac. Other funny issues include messing up the generated files because my copy of gcj has some of my debug output in it and linking against a rogue libjvm that was still floating around. It also didn’t like fastjar or any of my Sun tool wrappers, preferring the ones from Classpath.

I also had to drop back to the copy of OpenJDK I originally downloaded on the release. b15 won’t work, either built using IcedTea or from the binary drop. The binary drop of b13 does, so something went wrong since. It also fails for twisti, but he’s doing some weird build stuff with cacao so it could be that… ;)

Comments on IcedTea: seems to work well, although I wonder why it rebuilds rt.jar and jce.jar all the time, and just gets completely confused if I specify an openjdk source dir but no zip file (guess I didn’t understand that option properly)

Weather’s bad ‘ere too

This is for Andrew Cowie — it’s not only Australia seeing abnormal weather at the moment..

Sheffield Floods

Still not much further in getting JikesRVM to build on a GNU Classpath platform. It looks like I may have to give in and use Sun’s code for the time being (hopefully IcedTea, so at least it’s still Free).

Following on from last time, I got twisti to patch the hole in cacao only to fall over another large hole in the annotation support. So that’s cacao out until the student working on annotations completes the getAnnotations method.

I was assured by Andrew Haley that gcj had such support so I went and built the latest svn. And it got past that bit. Then we hit more fun, more very confusing fun.

What JikesRVM is doing at this stage is building a boot image that can then be jumped into by a small bootstrapping piece of C code. This image contains copies of the classes needed to get the VM to be self-hosting. Trouble is, in using a Classpath-based VM, there are more collisions between the new RVM classes and those in the host VM.

This bit of building the boot image involves quite a few Class.forName calls. These get confused over which classes we actually want to get, it seems (still not sure if that is meant to be the case, or whether the classloading is a bit messed up). Presumably with Sun’s VM, it only conflicts on the main API classes, and the code has special cases to deal with these.

The first problem with GCJ I found was that it can reach some odd cases where it loads a class from the RVM jar that is package-private and has a superclass which is loaded by GCJ (giving them different boot loaders, hence the compliant of a IncompatibleClassChangeError). These are anonymous inner classes that are getting named differently by GCJ. My guess is that this is because it’s using an older version of ecj (Jikes is using 3.2).

Suffice to say, JikesRVM has java.util.Collections$CheckedMap$CheckedEntrySet$1 while GCJ has java.util.Collections$6. As the former has a superclass of java.util.Collections$CheckedIterator which is loaded by the bootclassloader, an error is thrown. This isn’t the only case either; a number of inner classes in Collections are prone to this.
I’m still wondering if this is solvable by getting GCJ to build with a newer ecj. This requires adding GCCMain to a newer version. Not sure if this is even possible. I’ve got around it so far on the JikesRVM side by getting the boot image writer to ignore the error, as it does with ClassNotFound and Security exceptions (which suggests that loading the class isn’t required).

The current problem is a NullPointerException:

[echo] Exception in thread “main” java.lang.ExceptionInInitializerError
[echo] at java.lang.Class.initializeClass(libgcj.so.9)
[echo] at java.lang.Class.forName(libgcj.so.9)
[echo] at BootImageWriter.getJdkType(BootImageWriter.java:2473)
[echo] at BootImageWriter.createBootImageObjects(BootImageWriter.java:1150)
[echo] at BootImageWriter.main(BootImageWriter.java:614)
[echo] Caused by: java.lang.NullPointerException
[echo] at java.lang.VMClassLoader.getSystemClassLoaderInternal(libgcj.so.9)
[echo] at java.lang.VMClassLoader.getSystemClassLoader(libgcj.so.9)
[echo] at java.lang.ClassLoader$StaticData.(ClassLoader.java:154)
[echo] at java.lang.Class.initializeClass(libgcj.so.9)

This occurs because JikesRVM is trying to call the static initializer for ClassLoader$StaticData which in turn calls VMClassLoader.getSystemClassLoader. Note that GCJ’s copy doesn’t do this (or even contain this class); it is the Jikes class (from Classpath) that is doing this, so again it’s a case of interaction between the two class libraries.

I’m still not sure exactly where the NPE occurs (it’s somewhere in this native code). Any help would be much appreciated. The offending line is:

_Jv_CopyClassesToSystemLoader (gnu::gcj::runtime::ExtensionClassLoader::system_instance);

which calls code in natClassLoader.cc. This error occurs with a prototype build, but the same NPE occurs with a development build except this time it’s somewhere in the compilation performed by the OPT compiler.

Update

Tried to add GCCMain (from rhug’s gcj-ecj it turns out) to ecj 3.2, and it failed:

org/eclipse/jdt/internal/compiler/batch/GCCMain.java:231: cannot find symbol
symbol : method handleWarningToken(java.lang.String,boolean,boolean)
location: class org.eclipse.jdt.internal.compiler.batch.Main
super.handleWarningToken(token, isEnabling, useEnableJavadoc);
^
org/eclipse/jdt/internal/compiler/batch/GCCMain.java:440: cannot find symbol
symbol : variable destinationPaths
location: class org.eclipse.jdt.internal.compiler.batch.GCCMain
this.destinationPaths = new String[this.filenames.length];
^
org/eclipse/jdt/internal/compiler/batch/GCCMain.java:442: cannot find symbol
symbol : variable destinationPaths
location: class org.eclipse.jdt.internal.compiler.batch.GCCMain
this.destinationPaths[i] = this.destinationPath;
^
org/eclipse/jdt/internal/compiler/batch/GCCMain.java:460: cannot find symbol
symbol : method disableWarnings()
location: class org.eclipse.jdt.internal.compiler.batch.GCCMain
disableWarnings();

Trying JamVM eventually proved successful. I patched the same VMSupportsCS8 hole myself (just returning FALSE), and seem to have managed to work around a bug in VMDouble.toString to do with NaN detection to get a successful build (but not a running RVM…yet…)

Yesterday, I succeeded in getting the furthest I have so far in building JikesRVM. I’m guessing I’m nearly at the end and just falling at the final hurdle, as it gets to writing the boot image:

build-bootimage:
[echo] Building bootimage. Output redirected to : /home/gandalf/projects/java/classpath/jikesrvm/target/development_x86_64-linux/BootImageWriterOutput.txt
[java] Java Result: 1
[echo] BootImageWriter: compiler arg: O2
[echo] VM_BootImageCompiler: init (opt compiler)
[echo] Exception in thread “main” java.lang.UnsatisfiedLinkError: VMSupportsCS8
[echo] at BootImageWriter.createBootImageObjects(BootImageWriter.java:1109)
[echo] at BootImageWriter.main(BootImageWriter.java:614)

What’s proved such a problem is not so much JikesRVM itself as ant. It seems to assume a Sun-esque JDK and tries to run everything from tools.jar by default (presumably for efficiency, as this avoids spawning another VM). I’ve got round this by creating a tools.jar using OpenJDK and a little build system I hacked together. I’ve just announced this on the OpenJDK distro packaging mailing list, and hope that it can be made public in the same place as IcedTea (which can now build a Free OpenJDK).

Because this little bit of hackery gives me a tools.jar containing javac and javah, and an apt binary, the JikesRVM build process is satisfied. Of course, javac and javah have always been around on my path; it just doesn’t seem to want to use them. Next step is to see if the GNU Classpath tools will work as well, with the addition of a little wrapper for javah (so com.sun.tools.javah.Main exists, simply calling the GNU equivalent).

Any help on this final build problem would be much appreciated too. I’m eager to get hacking and actually contributing code for SoC!

Update

I’ve narrowed down the problem above which is nothing to do with Jikes RVM, but instead is due to a hole in CACAO’s implementation of the java.util.concurrent framework. VMSupportsCS8 is a native method which returns true if the underlying JVM supports lockless CompareAndSet for longs. I’m now looking around for VMs that do provide this…

So I’ve been playing around with the OpenJDK source code drop over the last week, in addition to still trying to build JikesRVM and doing odd bits of gjdoc maintainer stuff (Michael Koch is packaging the newest version for Debian). Unfortunately, I’ve not yet managed to build it without the binary blobs, which I don’t want to even have to download (in the belief that if I do that, I may as well just download a proprietary JDK). The build system relies heavily on an existing JDK and a lot of environment variables… oddly makes me miss the autotools!

My discoveries so far have lead me to find that the full build (hotspot+the j2se classes) won’t build without an existing tools.jar, of which Classpath does not have a full implementation (it wants com.sun.jdi, whatever that is). Of course, this stuff is in the tree as well (but the hotspot build doesn’t seem to know that). Hacking things to make hotspot use these classes lead to the revelation that it actually wants an older version; it tries to do a 1.4 compile, while the versions in the tree use 1.5 features such as generics.

Giving up on Hotspot and trying a j2se_only build gave more interesting results (seeing the build pull out bits of Classpath, believing them to be from the Sun JDK was fun), but again stalled because it wanted a libjvm.so from Hotspot. Assuming that it would be more difficult to remove this dependency than those in the full build (as I guess the classes are closely tied to Hotspot as their VM), I went back to that and tried to pull out the tools so they can be built separately. This has other uses, in so far as it would, for one, give me an apt I could use to build JikesRVM :) Javadoc and appletviewer also seem to be in there. I’ve since been plodding through a dependency hell which sees me slowly pulling more and more of the JDK into my newly created ‘tools’ directory. javax.script, javax.xml.bind and some management code are an example of the larger stuff it needs that Classpath doesn’t yet have.

This also gave me a patch for Classpath (fixing the RMI classes to use generics so rmic could build) and I have had apt at least give me its help output. More troubling is that I can’t find some stuff — ‘Version’ and ‘JDWP’ classes seem to be proving elusive. If anyone can shed any light on that, it would be much appreciated. In the meantime, hopefully the CP teams ‘iced tea’ plans will come to fruition and we can work on the code while the legal guys sort out a wonderful shared community between OpenJDK and Classpath.

_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_

As expected, JavaOne was the venue for yesterday’s announcement that the JDK class library source code is now available. Unfortunately, it is still heavily encumbered. About 60 megabytes of binary blobs are emitting a rather nasty smell. Seems like the next task for us freedom fighters is to at least try and build it without these dependencies and with a Free environment (not the proprietary JDK). It was nice to hear of people building it early and getting ebuilds out (well done Betelgeuse), but for me doing so in a Free environment will be the real result. I can safely assume that Sun has already built it themselves with the JDK (including itself presumably).

The hard work is now to come as Sun has to forge a community around this. A good start was made with the governance board, which includes our own Kaffe maestro, Dalibor Topic (congrats to him). The thing is Sun will now have to change their usual attitude towards the JDK in order to best work with the community. I doubt long turnarounds on patches which have to be approved internally by Sun will find much favour with the majority of FOSS hackers. The best thing about working on GNU Classpath has been the vibrant, happy and fun community that has been built around it, formed from a diverse group of folks with their one goal of a full Free Java implementation. Of course, this has still not been achieved, there is still work to be done and I hope that the new OpenJDK project can harness some of the enthusiasm and vigour that has been the epitome of our community.

So I finally got some time to hack some Classpath code again this bank holiday weekend, and ended up feeling like I got nowhere, running into bugs and annoying problems at each turn.

Me and JikesRVM are still fighting; I haven’t had a successful build as yet. After working round the apt problem (by moving the offending files out the way and commenting out the build rule, hehe), I proceeded to watch it download ecj (after having ignored my native version earlier). This ran into problems, which I later found were down to my own stupidity. I hadn’t installed the optional ant tasks as required. That fixed, I ran into a fun issue with its Classpath download. It decided to try and ungzip it twice, so tar complained and stopped the build. Something is being clever along the way and decompressing the gzip before it reaches tar I guess.

That fixed (by again hacking the build files), I realised that the file Ian kindly gave me (the one generated by the apt tool) had got mangled in downloading. Leastways, it didn’t look like any Java file I’d ever seen before. To cap it all, I then updated and that broke the build even earlier. Hopefully, that will be fixed soon.

Returning to Classpath itself, I noticed that the Mauve runs were coming up with a ridiculous number of errors, so decided to investigate. Updating, I found I couldn’t even build. The new gtkpeer.c file wonderfully assumes that the system is 32-bit. Good job I have -Werror on or I would be experiencing some nice random segfaults next time I use the GUI stuff. A couple of #ifdefs later and it was fixed.

I then proceeding to uncover the root of the test problems — there had to be some common problem, given that it was a fairly mixed bunch of tests. Turns out the build was no longer including the locale information. I’m quite amazed no-one had spotted this earlier. It was soon fixed by correcting the new find invocation in lib/Makefile.am.

So all in all, didn’t get very far but at least thinks seem to work again…

We now have two FOSS 1.5-capable compilers: ecj and the openjdk version of Sun’s javac. Both clearly have parsers for this version of the language, which got me thinking as to whether we should simply depend on one or the other (maybe even both) to provide parsing for gjdoc. I’ve looked at the OpenJDK code so far, and this seems fairly feasible. I’ve yet to look at ecj. In the meantime, I’d be interested to hear people’s opinions on what they think best. Clearly using one of these is a dependency we have to introduce, but, if we switch to 1.5 source code, this will be required anyway. It also removes the dependency on the lexer/parser framework we use (name escapes me right now…). As I see it, there are four options:

  • Retain the current parser code and update it for 1.5. Given the state of this code, it’s likely to be something of a headache, and likely to remain fairly unmaintainable in the future.
  • Replace the parser with our own generated using the tool we currently just use for expression parsing. Seems a little redundant as there are already capable parsers available.
  • Depend on either ecj or OpenJDK javac for parsing.
  • Use either ecj (if feasible) or OpenJDK for parsing, depending on which is also used to compile.

Any feedback is much welcomed.

I’ve also started to look at JikesRVM for SoC and run into some issues just trying to build it. Seems that not all is Free in the JikesRVM world and I can’t even build on a Free Java stack. My first trauma was trying to get ant (now used for the build system) to find a Java compiler. I couldn’t seem to get it to talk to ecj, so I pointed it at the JAR file for OpenJDK as an apparent ‘tools.jar’. I then ran into problems a few steps in, where it tries to use apt (the annotation processing tool, not the Debian package manager — which could have been interesting…) to preprocess some code. We don’t have an equivalent in Classpath, and it appears this is deprecated anyway in 1.6, in favour of javax.annotation.processing and capabilities in javac. Changing this is something we’re just beginning to discuss on the mailing lists, but it seems to be accepted that Sun or IBM is needed to build… ick. It seems there was some Kaffe support earlier, but it has gone the way of the dodo (along with its reference in the user manual). Ian Rogers has kindly supplied me with the file that is generated by this apt stage, so maybe I can at least see what hurdle comes next… Makes me think I should have made the project ‘Liberating JikesRVM’…

_
_
_
_
_
_
_
_
_
_
_
_
_

So, last week was quite eventful. Firstly, I got accepted for Google’s Summer of Code, working on the Jikes RVM project. I’ll be continuing in much the same vein as last year’s GSoC in implementing the java.lang.management extensions. It gives me a good excuse to finally play around with Jikes RVM again. The last time I tried was several years back when I first started hacking on Classpath, and I fell a victim to its build system, never even getting a running VM out of it. The build system has since been seriously re-jigged, so this time I intend to have more success. At least this time round, I know who to talk to if I don’t…. ;) Implementing the extensions on a VM is a good way to get to know the ins and outs, as it requires some fairly deep hooks. Hopefully, I won’t cause too many ruptions in the codebase…

Other than that, I’ve been playing around with GJDoc, trying valiently to get it to eat the latest Classpath CVS source. I finally succeeded in getting a run through last night, and the results are available on my website. File your bugs in the usual place; believe me, there are plenty!

Still haven’t got round to looking at the stuff I captured at FOSDEM, and surprisingly it’s already over a week ago — how time flies. There are a few pictures, but I left most of this to Mario and Steph as they were much more well prepared and equipped than I. What I do have is video of the Runtime Rumble and a couple of the Sun talks (IIRC). I need to work out where I can dump multigigabytes worth of DV before anything else; I took two tapes full in all.

What’s been occupying my thoughts so far this week is the imminent start of Google’s Summer of Code for 2007. It starts quite a bit earlier this year (student deadline of March 24th), and with all the Sun kerfuffle, it’s a little more difficult to choose a project than I expected at the end of last year’s SoC. I don’t want to put in a proposal for something that might get shipwrecked by any imminent arrivals of Free Sun code…

I raised the matter on #classpath today, and got some good feedback. There are plenty of tasks on underlying stuff (rather than APIs) that I could tackle. Also, Ian Rogers from Jikes RVM and Tom Marble (Sun’s OpenJDK ambassador) mentioned that their respective projects may be acting as mentoring organisations, so there seems to be plenty of choice. Unfortunately, I can’t really make any firm decisions until the organisations are decided, which is the same time as student proposals open. Just have to be prepared I guess…

« Previous PageNext Page »