I wouldn’t normally choose the JVM as a platform for numerical computing. One rather tricky area is in working with arrays of primitives: ints and floating point data with compact and efficient representations, with hardware support for doing math. The JVM usually provides a contiguous chunk of memory to store this data in situ (even though it’s not guaranteed). The JVM can sometimes eliminate bounds checking, and avoid boxing and unboxing array elements during operations, so one might expect relatively close-to-native performance of array operations. A simple tight loop summing over an array on the JVM seems to perform 2x slower than native (C) code in my tests. This feels like a reasonable price to pay for the obvious benefits of the JVM.

In practice, it has proven difficult to avoid boxing and un-boxing and still write expressive, idiomatic, generic Scala code. One promising feature in Scala comes in the form of the @specialization type annotation, which directs the compiler to generate additional code specialized for primitive types. But some extremely subtle issues arise which make this a lot trickier than it ought to be.

For instance, in the following code, the Iterator-based sum is about 10x slower than the while loop. What happens is the non-specialized ‘sum’ function of TraversableOnce is called, which itself invokes the generic ‘foldLeft’, and onward into the guts of the collection library. Since neither sum nor foldLeft is specialized, their invocations force boxing and un-boxing while interacting with our specialized code.

The obvious answer is that we might provide our own ‘sum’ function. And following the design in the TraversableOnce trait, we ought to do so via a specialized ‘foldLeft’. However, working alongside with my “Scalleague” Chris Lewis, I’ve found the compiler not to play along nicely. There are tons of specialization pitfalls that can cripple your fast-looking code. Details in a future post.

It took me some fiddling to fix the following message:

java.lang.UnsatisfiedLinkError: no jhdf5 in java.library.path

Hopefully this post will save someone else some time.

After you get the pre-built binaries, un-tar the archive to the /lib sub-directory of your sbt project. The /lib directory will look like this:

If you run sbt -> console, and look at your java.library.path, you’ll probably see something like this:

scala> println(System.getProperty("java.library.path"))
.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java

So I added some symlinks to /usr/lib/java:

 

Where dir is replaced with the path to your sbt project’s directory.

After that, it worked.

© 2014 Adam Klein's Blog Suffusion theme by Sayontan Sinha, modified by Adam :)