BLOG.THE-GENI.US

Building build files

A long time ago, I wrote a tool that generated ant build scripts for Eclipse projects. The tool, which I named “Smartbuild”, built a build script for a project by examining its “.project” and “.classpath” files, finding all of its dependent projects and libraries (and their dependent projects and libraries, recursively). Armed with this information, it generated a bare bones build script that contained the “usual suspects”: targets for “clean”, “init”, “compile” and “dist”. Smartbuild provided some nice functionality like merging manifests, setting up “Main” classes (for executable JAR’s), etc..

The nice things about Smartbuild were that it greatly simplified the grunt work of writing multi-project build scripts, but, more importantly, ensured that build scripts would stay in sync with my Eclipse projects. Each of my eclipse projects had a boilerplate build script that essentially was just about five lines long: all that the build script did was invoke smartbuild, have it generate the “real” build script, and run the generated script.

The “ugly” part about Smartbuild is that, like many projects that programmers create for themselves, it suffered from “organic” growth to the point that it became almost impossible to maintain. Also, since different parts of the code base reflected my then-current Java and OO skills, they became increasingly embarrassing to look at and debug as I became more skilled at my trade. Things got to the point where I decided that I needed to rewrite Smartbuild to continue using it. I will blog about my exercise of rewriting Smartbuild as I make progress.

Build Script generators and contributors

Build scripts are generated by generators with the help of one or more contributors. Generators are responsible for the overall organization of build tasks and how they depend on each other, while contributors are responsible for generating specific tasks.

Typically, users choose a generator depending upon the type of eclipse project that must be built, configure it with individual contributors to fine-tune the generated build script and finally, invoke the generator to obtain a build script. Example contributors are those that package source code (the “SourceCodeContributor”) and generate test builds (“JUnit contribitors”). Generators implement the IBuildScriptGenerator interface shown below:

package com.subhajit.tools.build;

import java.io.File;
import java.io.IOException;

import org.jdom.JDOMException;

import com.subhajit.eclipse.entities.EclipseMetafile;
import com.subhajit.eclipse.entities.EclipseProject;

public interface IBuildScriptGenerator {
    /**
     * Generates the build script for the selected Eclipse project.
     *
     * @return
     * @throws IOException
     * @throws JDOMException
     */
    String generateBuildScript() throws IOException, JDOMException;

    /**
     * Gets a list of all dependents of the selected Eclipse project, including
     * other projects on which the project depends on, third-party JAR files,
     * etc.
     *
     * @return
     */
    EclipseMetafile[] getDependents();

    /**
     * Returns the Eclipse project for which the build file is generated.
     *
     * @return
     */
    EclipseProject getEclipseProject();

    /**
     * Returns the build output directory.
     *
     * @return
     */
    File getBuildOutputDirectory();

    /**
     * Returns the distribution output directory.
     *
     * @return
     */
    File getDistOutputDirectory();

BuildConfiguration getBuildConfiguration();
}

The stand alone build script generator

I provide a default implementation of the IBuildScriptGenerator that generates build-scripts to create stand-alone applications from Eclipse projects. I also provide a class named “ScriptGenerator” that contains a main method encapsulating the stand alone build script generator. This standalone program lets the user specify various inputs such as the project directory (for which the build script must be generated), the name of a “main” class, and a set of directories containing Eclipse project files. The last parameter must be specified if the eclipse project depends upon other eclipse projects which lie under a different directory; this option need not be specified if the project and all the projects it depends on happen to lie under the same directory.

Invoking the program without specifying any inputs shows its usage:

The following sample invocation shows how to generate a build file for the “tools” project, which contains the source code for the build generator:

The generated build file is:

Using build generators programmatically

The following code snippet shows how the ScriptGenerator application uses the StandAloneAppBuildGenerator programmatically:

IBuildScriptGenerator genBuild = new StandAloneAppBuildGenerator(
        project);
genBuild.getBuildConfiguration().setSourceContributor(
        new SrcZipContributor(genBuild));
genBuild.getBuildConfiguration().setDistContributor(
        new ReferencedDependenciesContributor(genBuild));
genBuild.getBuildConfiguration().setExclusionFilter(
        new ExcludeTestCodeAndResourcesFilter());
ITestContributor testContributor = new JUnitContributor(genBuild);
testContributor.setDependsTaskName("build-test";
genBuild.getBuildConfiguration()
        .setTestContributor(testContributor);
genBuild.getBuildConfiguration().setSourceLevel(
        SourceLevel.ONE_FIVE);
genBuild.getBuildConfiguration().setTargetLevel(
        SourceLevel.ONE_FIVE);
if (mainClassName != null) {
    genBuild.getBuildConfiguration().setMainClass(mainClassName);
}

File buildOutput = new File(projectDirectory,
"build.xml";
FileUtils.saveTextFile(buildOutput, genBuild
.generateBuildScript());

The StandAloneAppBuildGenerator uses the following assumptions:

The eclipse project being built depends upon zero or more other eclipse projects.
The eclipse project may contain two types of source directories, of which one contains functionality and the other contains tests.
There are two possible ways to generate distributions for an eclipse project: one that contains functionality only, and another that contains both functionality and tests. It is assumed that distributions intended for release only contain functionality, while distributions intended for development include both functionality and code.

It generates a build script with the following features:

There is a “clean” target that cleans, or removes, all build-output directories.
There is a default “dist” target that produces a distribution suitable for release.
There is a “test” target that depends upon a “build-test” target. The “build-test” target produces a development-time distribution. The “test” target effectively generates a development time distribution, followed by invoking tests.

Both release- and development-time distributions result in the generation of one or more JAR files. Of these, one JAR file (the “main” JAR) contains classes and resources contributed by the project being built and all of its dependent projects, recursively. The remaining JAR files are third-party libraries upon which the code in the main JAR depends (the “library” JARs). The main JAR’s manifest (the “META-INF/MANIFEST.MF” entry) is setup as follows:

It contains an optional “Main-Class” attribute if the user specifies a “main” class.
Its “Class-Path” attribute of its manifest points to the library JAR’s.

Next steps

I intend to pursue the following next steps:

Create implementations of the IBuildScriptGenerator to generate build scripts for web applications.
Create an eclipse plug-in to allow end users to generate build scripts (and, optionally, to invoke them) interactively within Eclipse.

Posted by Idle musings at 2/14/2010 12:57 PM | Add Comment

A lightweight remoting framework

Introduction

I have long been intrigued with lightweight Java frameworks for configurable “processing pipelines”. A processing pipeline is an object that accepts input messages, processes each message and returns a response message representing the results of the request message. Along the way, processing pipelines use “interceptors” to intercept incoming and outgoing messages, and “transformers” to transform them. As you can see, processing pipelines are general purpose message (or event) processors.

What makes processing pipelines interesting is that they can be joined to each other (so that one pipeline processes the output messages from another), and that different “links” in the chain can be located remotely with respect to each other. For example, an order processing pipeline could have one pipeline that accepts orders, which feeds another that processes payment information. The payment-processing pipeline then feeds either an order-fulfillment pipeline (if the payment is processed successfully) or a decline-notification pipeline that notifies the purchaser of the problem with processing the payment information. The order-fulfillment pipeline finally feeds an order-completion pipeline that processes the payment information, generates an order for a delivery service (to pick up and deliver the order) and sends an email to the user notifying them that the order has been processed and providing them with tracking-information.

Processing pipelines are either “message processing” pipelines or “invocation pipelines”. The former simply delivers messages to some user-provided functionality that processes the message and generates an output message. The latter expects the input message to represent an action to be performed, the arguments for the action and some additional meta-data, locates an appropriate public method on a user-specified POJO, invokes this method with the given arguments, packages the returned value into a “response” message, and, finally, sends this response message on its way. The response message either encapsulates an “exception” (if the method threw an exception during method invocation) or a result.

Pipelines are built using “components”, which are logical “building blocks” consisting of an “input connector”, a “processing unit” and an “output connector”. A component reads messages from its input connectors, processes them through its processing unit, and sends processed messages through its output connector. Incoming messages read by a component are “request messages”, while outgoing, processed messages are “response messages”.

Processing pipelines are built by “chaining” one or more components in such a manner that messages are processed and passed from component to component, finally coming out from the processing pipeline from the output connector of the last component.

As an example, consider a processing pipeline that consists of three components (named A, B and C) that are joined to each other in the sequence A->B->C.

In other words, the output connector of A is connected to the input connector of B, and the output connector of B is connected to the input connector of C. A and C process only one type of message called “increment” that accepts an integer and adds one to it. B processes only one type of message named “addTwo” that accepts an integer and adds two to it.

Now, if a message requesting an “increment” action is written to A’s input connector with an input value of 1, A adds one to the value (resulting in a value of 2), and writes a message requesting the “addTwo” action, containing the value 2, to its output connector. This causes B to read the message, extract its value (which is 2), add two to this value, and write a message requesting an “increment” action, containing the value 4, to its output connector. This causes C to read the message, extract the value (which is now 4), and add one to this value, resulting in a new value of 5. Finally, C writes this value to its output connector, and the user reads the message (which now contains a value of 5) from the processing pipeline’s output connector. Note that A’s output message has its “action” identifier changed from “increment” to “addTwo” in this example. The mechanism which accomplishes this transformation is described later.

Messages

It is apparent that the concept of a “message” is central to components and processing pipelines. A message is modeled using the IMessage interface (for simplicity, package names are omitted):

public interface IMessage extends Serializable {
    /**
     * Returns the payload of this message.
     *
     * @return
     */
    Serializable getPayload();

    /**
     * Returns a {@link Map} containing all the properties associated with this
     * message.
     *
     * @return
     */
    Map<String, Serializable> getProperties();

    /**
     * Sets the given property, identified by its name and value.
     *
     * @param name
     * @param value
     */
    void setProperty(String name, Serializable value);
}

Thus, a message is modeled as a “payload” and (a logical map of) properties. The payload contains a representation of the “action” to be performed and any required parameters. The properties contain additional meta-data required to process the message (such as “quality of service” type information such as security-related information, etc.).

Messages are further modeled as “request” and “response” messages:

public interface IRequestMessage extends IMessage {
    /**
     * Returns a method-id (obtained via
     * {@link MethodUtil#toString(java.lang.reflect.Method)} indicating the
     * method to invoke.
     *
     * @return
     */
    String getMethod();

    /**
     * Returns an array of objects representing the arguments for the method to
     * be invoked (see {@link #getMethod()}.
     *
     * @return
     */
    Object[] getArguments();
}

and

public interface IResponseMessage extends IMessage {
    /**
     * Represents the value returned by the invoked method (see
     * {@link IRequestMessage#getMethod()}).
     *
     * @return
     */
    Object getResult();

    /**
     * Represents the remotely thrown exception (if any) by the invoked method
     * (see {@link IRequestMessage#getMethod()}).
     *
     * @return
     */
    Throwable getRemoteException();
}

where “request” messages represent a request to perform an action (such as would be provided to a component via its input connector) and “response” messages represent the result of performing the action on the request message, such as would be written out by a component via its output connector.

Components

As we have seen above, a component consists of three constituent parts, viz. an input connector, a message processor and an output connector.

Input connectors

Input connectors read request messages from specific “media”, such as TCP sockets, HTTP endpoints, JMS message queues, memory-based message queues, etc.. The exact type of media (or “transport”) that an input connector reads messages from are specified by the component that owns the input connector.

Output connectors

Similarly, output connectors write response messages to specific “media” (or “transports”) such as TCP sockets, HTTP endpoints, JMS or memory-based message queues, files or the console. Again, the exact type of transport is specified by the component which owns the output connector.

Message processing logic

Besides specifying the transports used by its input and output connectors, a component also specifies its message processing logic. Message processing logic consists of one or more business interfaces, a class implementing these interfaces, interceptors that intercept method invocations on instances of the implementation class and transformers that transform messages. Components specify message processing logic using the following items of information:

Business Services (Interfaces and implementations)

Business services consist of a list of Java interfaces (each of which represents a business service) and a Java class implementing these interfaces. In the following, the term POJI is used to describe the interfaces and POJO is used to describe the implementation class, respectively. Methods are invoked on instances of the POJO by setting up request messages with a) an identifier for the method to invoke and b) required arguments.

Interceptors

A list of “interceptors” that “intercept” message just before and just after invoking the requested method. Interceptors are represented by the following interface:

public interface IInterceptor {
    /**
     * Returns the name of this interceptor.
     *
     * @return
     */
    String getName();

    /**
     * Returns a {@link IRequestMessage} by modifying the original
     * {@link IRequestMessage}.
     *
     * <p>
     * The modified {@link IRequestMessage} is applied to the next
     * {@link IInterceptor} in the chain, or dispatched if this is the last
     * {@link IInterceptor} in the chain.
     * </p>
     *
     * @param request
     * @return
     */
    IMessage before(IMessage request);

    /**
     * Returns a {@link IResponseMessage} from the {@link IResponseMessage}
     * originally returned by dispatching a method.
     *
     * <p>
     * The modified {@link IInterceptor} is passed to the
     * {@link IInterceptor#after(IResponseMessage)} message of the next
     * {@link IInterceptor} in the chain, or returned to the caller if this is
     * the last {@link IInterceptor} in the chain.
     * </p>
     *
     * @param response
     * @return
     */
    IMessage after(IMessage response);

final static List<IInterceptor> EMPTY_INTERCEPTORS_LIST = Collections
.unmodifiableList(new ArrayList<IInterceptor>());
}

If more than one interceptor is specified, the interceptors are logically “chained” as follows. Assuming that two interceptors, X and Y are specified, the “before” method of X is called, then the “before” method of Y, and then, after the method invocation, the “after” method of Y is called, followed by the “after” method of X.

Transformers

Components can specify input and output transformers, where each transformer is modeled by the following interface:

public interface ITransformer {
    /**
     * Transforms the given <tt>message</tt> and returns the resulting message.
     *
     * @param message
     * @return
     */
    IMessage transform(IMessage message);

static final List<ITransformer> EMPTY_TRANSFORMERS_LIST = Collections
.unmodifiableList(new ArrayList<ITransformer>());
}

Input and output transformers are logically chained when more than one is specified: the “transform” method of each is called in sequence.

An example transformer is one that transforms the response message read from one component into a request message for another component.

Posted by Idle musings at 12/29/2009 8:26 AM | Add Comment

My new Core-i7 development laptop

My new Core-i7 development laptop (an HP dv6t with 8GB of RAM and a 500GB high-speed drive) arrived on the day before yesterday. I have just completed setting it up the way I like, and am putting it through its paces.

I am really excited about putting much of my multi-threaded code (especially the code analysis code-base) through its paces. This laptop does have eight logical cores (organized as four physical cores, each of which is split into two logical cores). So far, I have been using a Core2 Duo based development laptop, which meant that I could not perform real tests on scalability, and the ability to utilize additional cores, in my multi-threaded code. With 8 cores, I think i will be able to tune my multi-threaded code with an eye to making it as finely parallel as possible.

Watch this space.

Posted by Idle musings at 12/16/2009 7:16 AM | Add Comment

Continuations with Apache Javaflow

I recently ran across Apache Javaflow, a continuations framework for Java, and was sufficiently intrigued to try it out. It turns out that downloading, installing and (successfully) using Javaflow require a few tricks. These tricks are, in reality, simple steps which just happen to be undocumented. I am posting them here hoping that they are of some use to myself (and others).

Downloading Javaflow

You need subversion (svn) to be installed and correctly configured on your development machine before you can download Javaflow. If you are on Ubuntu (or have access to an Ubuntu machine), you can get subversion by running: sudo apt get svn. On Windows, you can download and install svn from here.

Once svn is installed, create a temporary directory, and run

svn co http://svn.apache.org/repos/asf/commons/sandbox/javaflow

This checks out the Javaflow development tree under the temporary directory.

Building Javaflow

You need maven to build Javaflow. If you are on Ubuntu, you can get maven by running sudo apt get maven2. On Windows, you can download and install maven by following the instructions on the maven download page. Once you have installed maven, change back to the temporary directory where you downloaded Javaflow and run “mvn install”. (This might take a while the first time you run maven, since it has to download and install many of its plugins).

Once the maven command completes, you will find the javaflow output JAR file in your “target” directory.

Using Javaflow

You need several libraries to use Javaflow. Of course, you need the Javaflow library itself, which was built using the steps described above. Additionally, you need the following JAR files:

ant-launcher.jar, ant.jar, asm-3.1.jar, asm-analysis-3.1.jar, asm-commons-3.1.jar, asm-tree-3.1.jar, asm-util-3.1.jar, asm-xml-3.1.jar, bcel-5.2.jar, commons-jci-core-1.0.jar, commons-logging.jar.

The following program shows how to use Javaflow within eclipse (note how a new copy of the MyRunnable class is loaded in the context of the ContinuationClassLoader instance):

public class ContinuationsTest {
    public static void main(String[] args) {
        int status = 0;
        try {
            Runnable runnable = (Runnable) new ContinuationClassLoader(
                    new URL[] {}, Thread.currentThread()
                            .getContextClassLoader()).forceLoadClass(
                    MyRunnable.class.getName()).newInstance();
            assert runnable.getClass().getClassLoader() != MyRunnable.class
                    .getClassLoader();
            Continuation c = Continuation.startWith(runnable);
            while (c != null) {
                c = Continuation.continueWith(c);
            }
        } catch (Throwable exc) {
            status = 1;
            exc.printStackTrace();
        } finally {
            System.exit(status);
        }
    }
}

The code snippet shown above uses a class named “MyRunnable”, which is shown below:

public class MyRunnable implements Runnable {
    public void run() {
        for ( int i=0; i<10; i++){
            System.out.println(i);
            Continuation.suspend();
        }
    }
}

Running ContinuationTest in Eclipse gives:

24569170 AMonitored locals 0
19272103 AMonitored locals 0
24569170 Monitored locals 0
19272103 Monitored locals 0
0
1
2
3
4
5
6
7
8
9

Posted by Idle musings at 10/17/2009 9:14 AM | Add Comment

Stripper revisited: the final version

This is the final update (for now) to the Stripper project about which I have posted lately. In this update, I not only present updated code, but also a detailed description of how Stripper works.

The Stripper is an application that produces minimal and self-contained distributions of java applications. Java applications are typically deployed in the form of one or more JAR files. Of course, they might be distributed in the form of a single ZIP-file, or via Java Web Start, but they usually end up as a set of JAR files deployed in some directory-structure after they are installed.

The fact of the matter is that the constituent JAR files of the application are usually not minimal: more likely than not, they contain unneeded classes that are never used. Application JAR’s are generally produced as the outcome of a build process, and most of these focus on producing self-contained, not minimal distributions. As a result, application JAR’s tend to get bloated with all manner of “baggage” files that serve no purpose (except, of course, to fatten their distribution files).

The Stripper helps to slim down application JAR files after they have been generated by a build process. It does so by statically analyzing a given main class to transitively find all classes that are (or may be) used by this class, and including all such classes (and only such classes) in a “stripped” JAR file. The stripped JAR file is equivalent to the input JAR files, the only difference being that it contains only classes that can ever be used by the main class. (As a bonus, the stripped JAR file is an executable JAR file: running “java –jar stripped_jar_file.jar” launches the main class).

Designing the Stripper

The stripper performs two functions:

Identify all classes that can ever be loaded by the main class.
Package these classes into a given output JAR file.

Let us look at each of these in turn.

Identifying reachable classes

The problem of identifying all classes that can be directly or indirectly reached by a given main class is impossible to solve using static-analysis alone. Static analysis enables us to examine a class file to find out all classes it uses (and further examine each used class, transitively, to find out all classes they use). However, it is unable to address the following:

Classes sometimes dynamically instantiate other classes (such as via Class.forName(“otherClass”).
Classes sometimes use abstract classes and interfaces, using factories to create actual instances of these at runtime (such as through Spring XML configuration). Static analysis can identify the abstract classes and interfaces, but not the actual instance classes.

So, how do we identify these classes? We use the following heuristics to address the possibility of factory classes (see 2. above). For each class obtained by static analysis:

Find all descendent classes (those that extend the class or a sub-class of the class). If this is an interface, find all classes implementing the interface.
Find all super-classes, and their descendent classes.
Repeat the above process iteratively for all classes found.

We address dynamic instantiation by allowing the user to specify a list of package names of classes that must be included in their entirety. All classes belonging to any such package (or sub-packages thereof) are included in the output. For example, specifying “org.apache.log4j” causes all classes belonging to any package beginning with “org.apache.log4j” to be included in the output.

Finally, all non-class resources are included in the output.

Packaging class and non-class resources

Now that we have a list of resources to include in the output, let us see how to create a JAR file containing these resources. The “easy” way to create a JAR file is to simply use code like this:

ZipOutputStream out = null;
try {
    out = new ZipOutputStream(new BufferedOutputStream(
            new FileOutputStream(jarFile)));
    for (String entryName : entryNames) {
        ZipEntry zipEntry = new ZipEntry(entryName);
        out.putNextEntry(zipEntry);
        out.write(contents.get(entryName));
        out.closeEntry();
    }
    out.finish();
    out.flush();
} finally {
    if (out != null) {
        out.close();
    }
}

The problem is that while the above code produces a JAR file containing all the entries, the Java launcher cannot use it to load the main class. (Why it fails to do is a mystery to me at the present time). What does work is the following pre-processing code that first sorts the entries by name, then inserts directory entries each time the directory of a file entry changes:

private List<String> updateEntryNamesWithDirectoryEntries(
        List<String> entryNames) {
    // If there is a manifest entry, move it to the beginning of entryNames.
    if (entryNames.contains("META-INF/MANIFEST.MF") {
        entryNames.remove("META-INF/MANIFEST.MF";
        entryNames.add(0, "META-INF/MANIFEST.MF";
    }

List<String> ret = new ArrayList<String>();

    StringBuilder str = new StringBuilder();
    String lastDir = null;
    for (String entryName : entryNames) {
        if (!entryName.contains("/") {
            if (!ret.contains(entryName)) {
                ret.add(entryName);
            }
        } else if (!entryName.endsWith("/") {
            String dir = entryName.substring(0, entryName.lastIndexOf("/");
            if (!dir.equals(lastDir)) {
                lastDir = dir;

                // Prepare to insert this as a directory-entry.
                String directoryEntryName = dir;
                String[] tokens = StrUtils.parse(directoryEntryName, "/";
                str.setLength(0);
                for (String token : tokens) {
                    if (str.length() != 0) {
                        str.append("/";
                    }
                    str.append(token);
                    if (!ret.contains(str.toString() + "/") {
                        ret.add(str.toString() + "/";
                    }
                }
            }
            if (!ret.contains(entryName)) {
                ret.add(entryName);
            }
        }
    }
    Collections.sort(ret);
    return ret;
}

The following code then writes the processed entries (which now contain directory-entries interspersed with file-entries) to the output JAR file:

ZipOutputStream out = null;
try {
    out = new ZipOutputStream(new BufferedOutputStream(
            new FileOutputStream(jarFile)));
    for (String entryName : entryNames) {
        ZipEntry zipEntry = new ZipEntry(entryName);
        out.putNextEntry(zipEntry);
        if (entryName.endsWith("/") {
            // Directory entry
        } else {
            // File entry
            out.write(contents.get(entryName));
        }
        out.closeEntry();
    }
    out.finish();
    out.flush();
} finally {
    if (out != null) {
        out.close();
    }
}

Using the Stripper programmatically

While the Stripper is useful as a build-tool or a command-line application, it may also be used programmatically. See the “com.subhajit.codeanalysis.distribution.DistributionManager” class in the provided source code, and the “StripperTest” file shown below to see how to do this:

package com.subhajit.stripper.test;

import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ExecutionException;

import org.junit.Test;

import com.subhajit.codeanalysis.distribution.DistributionManager;
import com.subhajit.common.classloaders.ParentLastURLClassLoaderX;
import com.subhajit.common.classloaders.URLClassLoaderX;
import com.subhajit.common.util.StrUtils;
import com.subhajit.common.util.streams.FileUtils;

public class StripperTest {
    private URLClassLoaderX getJavaClassPathBasedClassLoader()
            throws MalformedURLException, IOException, ClassNotFoundException {
        String[] classpathElements = StrUtils.parse(System
                .getProperty("java.class.path", File.pathSeparator);
        List<URL> urls = new ArrayList<URL>();
        for (String classpathElement : classpathElements) {
            File classpathFile = new File(classpathElement);
            // This is a hack. Maven includes the current-directory in the
            // classpath, leading to badly named classes being found by the code
            // analysis utils.
            if (!classpathFile.getAbsolutePath().equals(
                    System.getProperty("user.dir")) {
                urls.add(classpathFile.toURI().toURL());
            }
        }
        URLClassLoaderX classLoader = new URLClassLoaderX(urls
                .toArray(new URL[0]));
        return classLoader;
    }

    @Test
    public void testDistributionManager3() throws MalformedURLException,
            IOException, ClassNotFoundException, InterruptedException,
            ExecutionException {
        URLClassLoader classLoader1 = getJavaClassPathBasedClassLoader();
        final File tempFile1 = File.createTempFile("tf1", ".jar";
        try {
            final File tempFile2 = File.createTempFile("tf2", ".jar";
            try {
                // Create a distribution of the Stripper in a temporary file.
                Set<String> classes1 = DistributionManager
                        .createDistributionWithAdditionalPackages(classLoader1,
                                tempFile1, null,
                                "com.subhajit.stripper.Stripper";
                assert classes1
                        .contains("com.subhajit.common.listingprovider.ResourceManagerImpl";
                assert classes1
                        .contains("com.subhajit.common.listingprovider.ListingProvider";
                assert classes1
                        .contains("com.subhajit.common.classloaders.ByteMapListingProvider";
                assert classes1
                        .contains("com.subhajit.common.contenturl.MemoryContent";

                // Now, use tempFile1 as the input and generate tempFile2 by
                // stripping tempFile1.
                ParentLastURLClassLoaderX classLoader2 = new ParentLastURLClassLoaderX(
                        new URL[] { tempFile1.toURI().toURL() });
                Set<String> classes2 = DistributionManager
                        .createDistributionWithAdditionalPackages(classLoader2,
                                tempFile1, null,
                                "com.subhajit.stripper.Stripper";
                assert classes2
                        .contains("com.subhajit.common.listingprovider.ResourceManagerImpl";
                assert classes2
                        .contains("com.subhajit.common.listingprovider.ListingProvider";
                assert classes2
                        .contains("com.subhajit.common.classloaders.ByteMapListingProvider";
                assert classes2
                        .contains("com.subhajit.common.contenturl.MemoryContent";

                assert classes1.equals(classes2);
            } finally {
                if (tempFile2.exists()) {
                    FileUtils.deleteFile(tempFile2);
                }
            }
        } finally {
            if (tempFile1.exists()) {
                FileUtils.deleteFile(tempFile1);
            }
        }
    }
}

Download the Stripper

I am providing the following files for download:

stripper-all.jar	The maven build-output of the stripper project. This is an”unstripped” JAR in the context of this blog post. You can get it here.
stripper.jar	The stripped output of the stripper, obtained by running the stripper on “stripper-all.jar”. You can get it here.
src.zip	Zip file containing the source-code for all classes in the ‘com.subhajit” packages in stripper.jar. You can get it here.

Posted by Idle musings at 9/18/2009 1:33 PM | Add Comment

Following up on the Stripper

In a recent blog post., I wrote about “Stripper”, an application that produces minimal and self-contained distributions of Java applications. I quickly found some issues with the Stripper code while preparing a subsequent blog-post, which I am happy to say I have fixed. The updated code may be downloaded from here. The source code can be downloaded from here.

This version of the Stripper removes an option from the previous version (the “i” option to include sub-classes of all included classes) and adds a new option, namely, a “p” option that accepts a comma-separated list of package-names from which all resources (class-files and non-class resources) are included in the output. The “i” option is not really necessary as far as I can see. The “p” option provides greater control on generating the output, accounting for third-party code that dynamically instantiates classes.

Here is a screen-shot of the new Stripper in action stripping the build-output of this project. Note that I choose to include all classes and resources from the following packages by specifying the “-p org.jdom,org.jaxen,org.saxpath,com.werken” option:

com.werken
org.jdom
org.jaxen
org.saxpath

Posted by Idle musings at 9/14/2009 4:59 PM | Add Comment

Embedding FOSS: ActiveMQ, Tomcat and Derby

Ever written a test-suite and wanted an ad-hoc app-server deployment (such as tomcat) that you could create (and use) on the fly, only to get rid of it once the test suite completed? Ever written applications and wanted to create an embedded database instance (such as derby) or message-queue (such as your private ActiveMQ instance) for internal use?

In this post, I present an approach to creating self-contained instances of some popular, free and open-source products. I also provide working code for a product that facilitates the dynamic creation of ActiveMQ, Tomcat and Derby instances for embedded and/or ad-hoc use.

Let me jump directly into the code.

The InstallableProduct interface

First, I define an interface named “InstallableProduct” that defines the life-cycle of such embedded and/or ad-hoc application instances:

public interface InstallableProduct {
    /**
     * Sets up (installs) the product in the given
     * <tt>installationDirectory</tt>.
     *
     * @param installationDirectory
     * @throws IOException
     *             if there is a problem setting up the product.
     */
    void setup() throws IOException;

    /**
     * Configures the product after it has been setup.
     *
     * @throws IOException
     *             if there is a problem configuring the product.
     */
    void configure() throws IOException;

    /**
     * Starts the product.
     *
     * @throws IOException
     * @throws InterruptedException
     */
    void startup(long millis) throws IOException, InterruptedException;

    /**
     * Stops the product.
     *
     * @throws IOException
     * @throws InterruptedException
     */
    void shutdown(long millis) throws IOException, InterruptedException;

    /**
     * Checks if the product is running.
     * @return
     * @throws IOException
     */
    boolean isRunning() throws IOException;

    /**
     * Uninstalls the product.
     * @throws IOException
     */
    void uninstall() throws IOException;
}

The following diagram shows the life-cycle of an InstalledProduct:

The first step in the life-cycle of an InstallableProduct is setup, which creates, in a user-specified directory, a set of files embodying the product. The created files represent a ready-to-run image of the product, and thus include all the startup and shutdown scripts, JAR files, configuration files, etc., that a stand-alone installation of the product needs.

The next step in the life-cycle of an InstallableProduct is configuration, when product-specific configuration parameters, such as port numbers, database names, etc. are written into the installation.

Next, the InstallableProduct instance can be started or stopped as many times as desired.

Finally, the InstallableProduct instance can be uninstalled, which simply removes the installation directory of the instance.

PortBasedInstallableProduct implementation

Since many InstalledProducts take the form of daemon-processes that provide services via one or more ports, I created an abstract subclass of InstallableProduct named PortBasedInstallableProduct:

/**
* Abstract class models products that run as daemon processes listening on a
* known port to provide services.
*
* @author sdasgupta
*
*/
public abstract class PortBasedInstallableProduct implements InstallableProduct {
    private static final Logger sLog = Logger
            .getLogger(PortBasedInstallableProduct.class);
    static {
        // If there are no appenders, use the basic configurator to add one.
        if (!sLog.getAllAppenders().hasMoreElements()) {
            BasicConfigurator.configure();
        }
    }
    private final int listeningPort;

    public PortBasedInstallableProduct(int listeningPort) {
        super();
        this.listeningPort = listeningPort;
    }

    public boolean isRunning() throws IOException {
        return NetUtils.isListening(
                getLocalHost(), listeningPort);
    }

    public final void shutdown(long millis) throws IOException,
            InterruptedException {
        sLog.info("Shutting down product.";
        doShutdown();
        sLog
                .info("Shutdown signal sent : waiting for port to stop listening - "
                        + listeningPort);
        NetUtils.waitToDisconnect(getLocalHost(),
                listeningPort, millis);
        sLog.info("Product has shut down.";
    }

protected abstract void doShutdown() throws IOException,
InterruptedException;

    public final void startup(long millis) throws IOException,
            InterruptedException {
        sLog.info("Starting product.";
        if (isRunning()) {
            throw new IllegalStateException(
                    "Already running: listening on port: " + listeningPort);
        }
        doStartup();
        sLog
                .info("Product has been started up : waiting for port to begin listening - "
                        + listeningPort);
        NetUtils.waitToConnect(getLocalHost(),
                listeningPort, millis);
        sLog.info("Product startup complete.";
    }

    private String getLocalHost() throws UnknownHostException {
        return "localhost";
    }

protected abstract void doStartup() throws IOException,
InterruptedException;

    protected int getListeningPort() {
        return listeningPort;
    }
}

Note that PortBasedInstallableProduct does not implement “configure”, and requires its concrete sub-classes to imp;lement two methods, namely, “doStart” and “doStop”.

Next, I present source code for some concrete InstallableProducts.

Embeddable Tomcat

The EmbeddableTomcat class creates a tomcat instance in a given installation directory, and providing HTTP service over a given port. Note how the “configure” method sets up port numbers in the “conf/server.xml” file of the instance:

public void configure() throws IOException {
    try {
        File serverXml = new File(installationDirectory, "conf/server.xml";
        Document doc = XmlUtils.loadDocument(serverXml);

        int index = 1;
        int shutdownPort = getListeningPort() + index++;
        int redirectPort = getListeningPort() + index++;
        int ajpPort = getListeningPort() + index++;

        // Replace the shutdown port
        Element serverElement = (Element) XPath.selectSingleNode(doc
                .getRootElement(), "/Server[@shutdown='SHUTDOWN']";
        serverElement.setAttribute("port", "" + shutdownPort);

        // Replace the listening and redirect ports.
        Element connectorElement = (Element) XPath
                .selectSingleNode(doc.getRootElement(),
                        "/Server/Service[@name='Catalina']//Connector[@executor='tomcatThreadPool']";
        connectorElement.setAttribute("port", "" + getListeningPort());
        connectorElement.setAttribute("redirectPort", "" + redirectPort);

        // Replace the AJP port.
        Element ajpElement = (Element) XPath
                .selectSingleNode(doc.getRootElement(),
                        "/Server/Service[@name='Catalina']//Connector[@protocol='AJP/1.3']";
        ajpElement.setAttribute("port", "" + ajpPort);
        ajpElement.setAttribute("redirectPort", "" + redirectPort);

        // Save the updated serverXml.
        FileUtils.saveTextFile(serverXml, XmlUtils.toString(doc));
    } catch (JDOMException exc) {
        throw new IOException(StrUtils.toString(exc));
    }
}

and how the “doStart” and “doStop” methods execute “bin/catalina start” and “bin/shutdown”, respectively.

Embeddable Derby

Creates an embeddable Derby instance in a specified installation directory, listening on a given port and creating a given database. I do not reproduce snippets of the source code here since the “setup” and “configure” methods are somewhat involved (and, I am afraid, over-complicated).

Embeddable ActiveMQ

Creates an embeddable ActiveMQ instance in a specified installation directory and providing service via a specified port.

Download

You can download a JAR file containing the InstallableProduct class and all its dependencies from here, and a zip file containing the source code from here.. Look at the com.subhajit.embeddable.packaging.Main class once you download the source files to see how to use these classes:

public class Main {
    public static void main(String[] args) {
        int status = 0;
        try {
            System.out.println("Contents: "
                    + EmbeddableTomcat.class.getSimpleName() + IConstants.COMMA
                    + EmbeddedActiveMQ.class.getName() + IConstants.COMMA
                    + EmbeddedDerby.class.getName());
            File tmpDir = new File("tmp";
            try {
                FileUtils.mkdirs(tmpDir);
                final File installationDir = new File(tmpDir, "temp";
                EmbeddableTomcat tomcat = new EmbeddableTomcat(installationDir,
                        20000);
                tomcat.setup();
                tomcat.configure();
                tomcat.startup(10000);
                tomcat.shutdown(10000);
                tomcat.uninstall();

                EmbeddedDerby derby = new EmbeddedDerby(installationDir,
                        "test", 20000);
                derby.setup();
                derby.configure();
                derby.startup(10000);
                derby.shutdown(10000);
                derby.uninstall();

                EmbeddedActiveMQ activeMQ = new EmbeddedActiveMQ(
                        installationDir, 20000);
                activeMQ.setup();
                activeMQ.configure();
                activeMQ.startup(10000);
                activeMQ.shutdown(10000);
                activeMQ.uninstall();
            } finally {
                if (tmpDir != null && tmpDir.exists()) {
                    FileUtils.deleteDirectoryAndContents(tmpDir);
                }
            }
        } catch (Throwable exc) {
            status = 1;
            exc.printStackTrace();
        } finally {
            System.exit(status);
        }
    }
}

Posted by Idle musings at 9/14/2009 12:23 PM | Add Comment

Stripper: A Self-contained and minimal CODE-DISTRIBUTION generator

I have been thinking about an interesting problem for some time: how to generate code-distributions that are both self-contained and minimal.

By code-distribution I mean a Java application distributed in the form of one or more JAR files, of which one class is a designated “main class” (that is, it contains a “public static void main(String[])” method).
By self-contained I mean that the code-distribution contains all the classes and resources needed by the main class to run.
By minimal I mean that the code-distribution does not contain any classes that are not used, directly or indirectly, by the main class.

Minimal and self-contained code-distributions such as these (which I will call “MSD’s” in this posting) are generally useful for application-deployments since they minimize storage requirements, load-times and, where applicable, download-times.They are perfect for “sample code” provided by bloggers who do not want their readers to be bogged down in a “morass” of source-code that they do not need to look at for the task at hand.

MSD’s may not always be feasible, especially if third-party licenses prevent classes (present in third-party JAR’s) from being extracted and re-packaged from their originally distributed form. For example, a certain third-party product’s license might require that they be distributed in the form they were originally provided.

Having looked at MSD’s, I present an MSD-generator named “Stripper” (including complete source-code) to creating them.

Stripper requires the following inputs:

A list of JAR files representing a self-contained distribution for a “main” class: Typically, this is the end-result of your build process (such as after running an ant or maven build script).
The name of the main-class.
Optionally, a list of source-code locations (directories or JAR files containing “.java” files for some or all of the classes in the distribution.

and produces the following outputs:

An executable JAR that is the MSD, with the “Main-Class” attribute in its manifest set to the name of the designated main class.
If requested, a separate JAR file containing the source-code for your classes in the MSD.

Download details, such as a ready-built executable JAR file, a source JAR and instructions for use are provided below.

Approach

Stripper uses the following approach to generate MSD’s:

Starting with the main class, use static analysis to find all classes it uses, followed by transitively finding all classes those classes use. In other words, Stripper first finds all classes used by the main class, followed by finding the classes used by all of these classes, and continuing this process transitively for all classes.
For all interfaces found above, find their super-interfaces and sub-interfaces. For each of these, find all implementation classes, and the classes used by those classes, recursively.
For all classes found above, include their super classes and sub-classes, and the classes they use, recursively.

Of these steps, the first one (static analysis) seems reasonable on its face, since we wish to generate a self-contained distribution. The second and third steps are needed because classes sometimes use classes via external configuration (such as properties or XML configuration files) dynamically; static analysis (step 1) alone does not recognize these dynamically named classes.

For example, the following source code shows an InputStream being created dynamically from a custom class named “my.custom.InputStream”:

InputStream in = ( InputStream) Class.forName(“my.custom.InputStream”).newInstance(bytes);

Static analysis of this class shows that it uses InputStream, not “my.custom.InputStream”. We still need to include “my.custom.InputStream” in the output MSD, however, since not doing so would result in a “NoClassDefFoundError” at runtime.

To address this form of dynamic instantiation, we must include all subclasses of used classes (and implementations of used interfaces) in the generated MSD. Unfortunately, the MSD then potentially ends up containing classes that are never used.

FINDING USED CLASSES

Stripper finds all classes a given class uses uses the following steps:

Include all interfaces implemented by the class and its super-classes.
Include all super-classes of the class.
Include the types (and their super-types) of all declared fields of the class and its super-classes.
Include the return types (and their super-types), exception types (and their super-types) and parameter types of all methods declared by the class and its super-classes.
Finally, scan the class’s constant pool, including all string entries that “look like” class names and for which “class” resources are found in the input distribution.

MANIFESTS

Stripper computes the manifest (the “META-INF/MANIFEST.MF” entry) of the generated MSD by first computing an “effective manifest”, then updating its “Main-Class” attribute (if any) with the given main class. The “effective manifest” is computed as follows. If the user specifies a custom manifest, then that becomes the effective manifest. Otherwise, all the manifests found in the input JAR files are merged to create the effective manifest. The updated effective manifest is then saved into the generated MSD JAR file.

USING Stripper

The MSD generator presented here, called “Stripper”, is a command-line application that accepts the following options:

d	A comma-separated list of directories containing JAR files comprising the input distribution.
f	A comma-separated list of JAR file names comprising the input distribution.
o	The output file (name of the generated MSD), which must be a JAR file.
m	The name of the main class.
s	Comma-separated list of directories or ZIP/JAR files containing source code.
i	Optional parameter which, if “true”, causes the stripper to include sub-classes of all classes in the output. Any value other than “true” causes sub-classes not to be included.

I used the Stripper to “strip” itself, by first generating the build-output of my project (using maven “jar-with-dependencies”), copying the resulting JAR to another location, and then running the build output on that JAR:

Note that I did not specify the “-s” option (list of source-code locations), but the Stripper generated a source JAR (“src.zip”). The Stripper implicitly uses all input distribution JAR’s as source-code locations, picking up source entries for output-classes it finds there. In my case, I choose my maven project to emit source-code into my build outputs, so I do not need to specify any explicit source-code locations.

SOURCE CODE

The stripper code (in executable JAR form) can be downloaded from here. The MSD generated by the stripper (the “stripped” stripper) can be downloaded here.

Posted by Idle musings at 9/12/2009 7:33 AM | Add Comment

Imported entries from my SUN blog

I work at Sun Microsystems. My technical blog (http://blogs.sun.com/adventures) is hosted by Sun, right besides other blogs created by Sun employees.

Recently, Sun offered its employees the chance to enter into a cross-licensing agreement with them that allows blog syndication, certain perpetual rights, etc (see Blog License FAQ).One nice thing about this agreement is that it allows us to export (and re-import) the contents of our blogs. I decided to try out that feature. The entries that you see here that pre-date 9/1/2009 are the result of this exercise.

Everything did not work perfectly right out of the box, neither did I have that expectation. You might find broken download links, missing images and some out-of-format text. I plan to go over the postings in this blog in the next few days and make updates as appropriate. In the meantime, please use http://blogs.sun.com/adventures for those entries.

Once everything is said and done, I plan to post entries to both blogs. Future postings in http://blogs.sun.com/adventures will point back to this blog. Once things pan out, I plan to change all these “old” postings so that they refer back to http://blogs.sun.com/adventures (their original home).

Posted by Idle musings at 9/3/2009 6:42 AM | Add Comment

Windows Live Writer

This is a test blog posting using Windows Live Writer. I just changed the template image through QuickCast, but Windows Live Writer did not pick up the change. Let me check if deleting and re-adding this account helps…ok, that worked.

Next, let me check how images work. Here is one:

Hmm..that worked quite well.

Posted by Idle musings at 9/2/2009 10:28 AM | Add Comment

Quick Search

Syndicate

Monthly Archives

Category Archives

Recent Entries

Recent Comments

Subscribe

Tag Cloud