The Kaptain on ... stuff » Development

Five Cool Things You Can Do With Groovy Scripts

TheKaptain — Sun, 04 Dec 2011 21:51:34 +0000

1. Ensure all of your Jenkins builds are building the correct branch from source control

I manage a large number of builds at work, spread across several build servers. When we release a new version all of the builds need to be updated to point to new working branches. This script takes advantage of the fact that our branches all end in the version number to quickly check that all of the last builds were on the expected version.
Using the Jenkins API is very easy and the new capabilities of Groovy introduced by JSONSlurper make it easier than ever to consume.

2. Look up the artifacts from the last successful Jenkins build

This is very handy if you want to automate deployment of software from your build system and can replace a lot of error prone manual updating of scripts. In this particular example you can link up quickly with the latest, greatest build of JRuby from their public CI server.

Here’s the sample output at time of writing:

[relativePath:dist/jruby-bin-1.7.0.dev.tar.gz, fileName:jruby-bin-1.7.0.dev.tar.gz, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.tar.gz.md5, fileName:jruby-bin-1.7.0.dev.tar.gz.md5, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.tar.gz.sha1, fileName:jruby-bin-1.7.0.dev.tar.gz.sha1, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.zip, fileName:jruby-bin-1.7.0.dev.zip, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.zip.md5, fileName:jruby-bin-1.7.0.dev.zip.md5, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.zip.sha1, fileName:jruby-bin-1.7.0.dev.zip.sha1, displayPath:null]
[relativePath:dist/jruby-complete-1.7.0.dev.jar, fileName:jruby-complete-1.7.0.dev.jar, displayPath:null]
[relativePath:dist/jruby-complete-1.7.0.dev.jar.md5, fileName:jruby-complete-1.7.0.dev.jar.md5, displayPath:null]
[relativePath:dist/jruby-complete-1.7.0.dev.jar.sha1, fileName:jruby-complete-1.7.0.dev.jar.sha1, displayPath:null]
[relativePath:dist/jruby-jars-1.7.0.dev.gem, fileName:jruby-jars-1.7.0.dev.gem, displayPath:null]
[relativePath:dist/jruby-jars-1.7.0.dev.gem.md5, fileName:jruby-jars-1.7.0.dev.gem.md5, displayPath:null]
[relativePath:dist/jruby-jars-1.7.0.dev.gem.sha1, fileName:jruby-jars-1.7.0.dev.gem.sha1, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.tar.gz, fileName:jruby-src-1.7.0.dev.tar.gz, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.tar.gz.md5, fileName:jruby-src-1.7.0.dev.tar.gz.md5, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.tar.gz.sha1, fileName:jruby-src-1.7.0.dev.tar.gz.sha1, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.zip, fileName:jruby-src-1.7.0.dev.zip, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.zip.md5, fileName:jruby-src-1.7.0.dev.zip.md5, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.zip.sha1, fileName:jruby-src-1.7.0.dev.zip.sha1, displayPath:null]

3. Read in a CSV file, filter it, and write out the result

Groovy Grapes and one of my favorite libraries, OpenCSV, make it trivial to script ‘one off’ solutions for removing work that would normally take place manually in Excel. For me it comes up most often with data import/export from systems I’m working on and the need to create particular test environments using those import/export mechanisms. For example, using a scripted solution like this you can quickly:

Export users from one test system into a CSV file, along with local phone number contact data
Replace the existing phone numbers with a mix of local and international numbers
Import the new data into a different test system
Verify that the system properly deals with international calling concerns

This example simply trims out all of the rows where the second column of input contains values less than 100. It’s also nice to note that OpenCSV is smart enough to let you ignore one or more ‘header rows’, a common attribute of CSV files.

4. Download, install and run a stand-alone webserver

This example comes courtesy of a recent article by Andrew Glover about Gretty, a lightweight http server. I don’t have a concrete use-case for this right now, but it’s easy to note that in this case you can code a large part of the server behaviour directly in the script, unlike, for instance, if you were to use a script to stand up a Jetty server. This should let you publish a script once in a central location and stand up as many Gretty servers providing the declared services as you like. This particular example simply echos back ‘Hello’ plus whatever path you append to localhost:8080.

5. Remotely execute any of these scripts by url

As of Groovy 1.8.3/1.9-beta-4 you can refer to a Groovy script hosted at a remote url and execute it locally. If you work on different machines a lot(as I do) this can really help keep your toolbox of scripts handy, regardless of where you are. Since these scripts are publicly hosted on GitHub(thanks guys!), you can simply copy the url from the ‘view raw’ link at the bottom and go to your console to execute them like this:

$ groovy https://gist.github.com/raw/1430856/cea0186a862217015fb4fc2b63eb0ad3575c06fc/jenkinsBuildArtifacts.groovy
{
    "actions": [
        {
            "causes": [
                {
                    "shortDescription": "Started by timer"
                }
            ]
        },
        {
            "buildsByBranchName": {
                "origin/master": {
                    "buildNumber": 684,
                    "buildResult": null,
                    "revision": {
                        "SHA1": "3419fee4bb9436d3222ff3df8dd7fcf2308a6919",
                        "branch": [
                            {
                                "SHA1": "3419fee4bb9436d3222ff3df8dd7fcf2308a6919",
                                "name": "origin/master"
                            }
                        ]
                    }
                }
            },
... and so on

Final note

Being able to share scripts across a variety of machines has definitely made my professional life a lot easier and I’m very glad to see the ability to execute code from a URL in Groovy. But what really turned me onto the idea was its usage in Gradle, which supports loading remote Groovy/Gradle scripts using the “apply from: {url or file}” syntax. I find that more and more I reference particular tasks remotely in a build, especially for static analysis tools that don’t necessarily need to be run on a regular basis.
A great example of this is the recently released Gradle Templates Plugin, which is basically the Gradle answer to the Maven Archetype plugin. Create a build.gradle with the following content:

apply from: 'http://launchpad.net/gradle-templates/trunk/1.2/+download/apply.groovy'

And we have the following tasks available to create new Gradle projects and objects with conventional structure:

Template tasks
--------------
createGradlePlugin - Creates a new Gradle Plugin project in a new directory named after your project.
createGroovyClass - Creates a new Groovy class in the current project.
createGroovyProject - Creates a new Gradle Groovy project in a new directory named after your project.
createJavaClass - Creates a new Java class in the current project.
createJavaProject - Creates a new Gradle Java project in a new directory named after your project.
createScalaClass - Creates a new Scala class in the current project.
createScalaObject - Creates a new Scala object in the current project.
createScalaProject - Creates a new Gradle Scala project in a new directory named after your project.
createWebappProject - Creates a new Gradle Webapp project in a new directory named after your project.
exportAllTemplates - Exports all the default template files into the current directory.
exportGroovyTemplates - Exports the default groovy template files into the current directory.
exportJavaTemplates - Exports the default java template files into the current directory.
exportPluginTemplates - Exports the default plugin template files into the current directory.
exportScalaTemplates - Exports the default scala template files into the current directory.
exportWebappTemplates - Exports the default webapp template files into the current directory.
initGradlePlugin - Initializes a new Gradle Plugin project in the current directory.
initGroovyProject - Initializes a new Gradle Groovy project in the current directory.
initJavaProject - Initializes a new Gradle Java project in the current directory.
initScalaProject - Initializes a new Gradle Scala project in the current directory.
initWebappProject - Initializes a new Gradle Webapp project in the current directory.

Using Gradle to Bootstrap your Legacy Ant Builds

TheKaptain — Sun, 18 Sep 2011 20:44:11 +0000

Gradle provides several different ways to leverage your existing investment in Ant, both in terms of accumulated knowledge and the time you’ve already put into build files. This can greatly facilitate the process of porting Ant built projects over to Gradle, and can give you a path for incrementally doing so. The Gradle documentation does a good job of describing how you can use Ant in your Gradle build script, but here’s a quick overview and some particulars I’ve run into myself.

Gradle AntBuilder

Every Gradle Project includes an AntBuilder instance, making any and all of the facilities of Ant available within your build files. Gradle provides a simple extension to the existing Groovy AntBuilder which adds a simple yet powerful way to interface with existing Ant build files: the importBuild(Object antBuildFile) method. Internally this method utilizes an Ant ProjectHelper to parse the specified Ant build file and then wraps all of the targets in Gradle tasks making them available in the Gradle build. The following is a simple Ant build file used for illustration which contains some properties and a couple of dependent targets.



    Building ${ant.file}

    
    

    
        
        dist.dir=${dist.dir}, foo=${foo}

Importing this build file using Gradle is a one-liner.

ant.importBuild('src/main/resources/build.xml')

And the output of gradle tasks –all on the command line shows that the targets have been added to the build tasks.

$ gradle tasks --all
...
Other tasks
-----------
all - Build everything
    dist - Build the distribution
...

Properties used in the Ant build file can be specified in the Gradle build or on the command line and, unlike the usual Ant property behaviour, properties set by Ant or on the command line may be overwritten by Gradle. Given a simple build.properties file with foo=bar as the single entry, here’s a few combinations to demonstrate the override behaviour.

Command line invocation	Gradle Build Config	Effect	Result
gradle dist	ant.importBuild('src/main/resources/build.xml')	build.properties value loaded from ant build is used	foo=bar
gradle dist -Dfoo=NotBar	ant.importBuild('src/main/resources/build.xml')	command line property is used	foo=NotBar
gradle dist -Dfoo=NotBar	ant.foo='NotBarFromGradle' ant.importBuild('src/main/resources/build.xml')	Gradle build property is used	foo=NotBarFromGradle
gradle dist -Dfoo=NotBar	ant.foo='NotBarFromGradle' ant.importBuild('src/main/resources/build.xml') ant.foo='NotBarFromGradleAgain'	Gradle build property override is used	foo=NotBarFromGradleAgain

How to deal with task name clashes

Since Gradle insists on uniqueness of task names attempting to import an Ant build that contains a target with the same name as an existing Gradle task will fail. The most common clash I’ve encountered is with the clean task provided by the Gradle BasePlugin. With the help of a little bit of indirection we can still import and use any clashing targets by utilizing the GradleBuild task to bootstrap an Ant build import in an isolated Gradle project. Let’s add a new task to the mix in the Ant build imported and another dependency on the ant clean target to the all task.


    
        Called clean task in ant build with foo = ${foo}

And a simple Gradle build file which will handle the import.

ant.importBuild('src/main/resources/buildWithClean.xml')

Finally, in our main gradle build file we add a task to run the targets we want.

task importTaskWithExistingName(type: GradleBuild) { GradleBuild antBuild ->
    antBuild.buildFile ='buildWithClean.gradle'
    antBuild.tasks = ['all']
}

This works, but unfortunately suffers from one small problem. When Gradle is importing these tasks it doesn’t properly respect the declared order of the dependencies. Instead it executes the dependent ant targets in alphabetical order. In this particular case Ant expects to execute the dist target before clean and Gradle executes them in the reverse order. This can be worked around by explicitly stating the task order, definitely not ideal, but workable. This Gradle task will execute the underlying Ant targets in the way we need.

task importTasksRunInOrder(type: GradleBuild) { GradleBuild antBuild ->
    antBuild.buildFile ='buildWithClean.gradle'
    antBuild.tasks = ['dist', 'clean']
}

Gradle Rules for the rest

Finally, you can use a Gradle Rule to allow for calling any arbitrary target in a GradleBuild bootstrapped import.

tasks.addRule("Pattern: a- will execute a single  in the ant build") { String taskName ->
    if (taskName.startsWith("a-")) {
        task(taskName, type: GradleBuild) {
            buildFile = 'buildWithClean.gradle'
            tasks = [taskName - 'a-']
        }
    }
}

In this particular example, this can allow you to string together calls as well, but be warned that they execute in completely segregated environments.

$ gradle a-dist a-clean

Source code

All of code referenced in this article is available on github if you’d like to take a closer look.

A Groovy/Gradle JSLint Plugin

TheKaptain — Mon, 04 Apr 2011 01:07:31 +0000

This article originally appeared in the January 2011 issue of GroovyMag.

Gradle is a build system in which builds are described using a declarative and concise DSL written in the Groovy language. This article describes how you can wrap proven Apache Ant Tasks in a Gradle Plugin to make using them as effortless as possible. We’ll also go over some of the tools Gradle provides for building and testing robust Plugin functionality following some easy patterns.

Creating new custom Plugins for Gradle is a relatively straightforward and easy process. Within a Plugin it’s possible to configure a Gradle Project with new properties, dependencies, Tasks – pretty much anything that you can configure in a build.gradle file can be encapsulated into a Plugin for abstraction, portability and reuse. One of the easier ways to add functionality through a Plugin is to encapsulate an existing Ant Task and enhance it by providing the ease-of-use and configuration that Gradle users have come to expect. Recently, I’ve been writing a lot more JavaScript and was looking for static analysis tools to help guide me away from ‘bad habits’. The popular choice for static analysis of JavaScript code seems to be JSLint, so here’s an example of providing that functionality for a Gradle build by wrapping an existing JSLint task and making it easier to work with.

Anatomy of a Gradle Plugin

Gradle plugins can most easily be built using Gradle itself. There is a conveniently available gradleApi() method you can call to include the required framework classes, demonstrated in the dependencies section of a build.gradle file shown in Listing 1. For this example we’re also using the Groovy Plugin and JUnit for testing, so we will include those dependencies as well.

dependencies {
    compile gradleApi()
    groovy group: 'org.codehaus.groovy', name: 'groovy', version: '1.7.6'
    testCompile group: 'junit', name: 'junit', version: '4.8.2
}

Listing 1: The dependencies portion of a build.gradle file for building our Plugin

Creating a new Gradle Plugin is a simple matter of implementing the Gradle interface and its single required method, the skeleton of which is shown in Listing 2. Within the apply method, the Plugin can configure the Project to add Tasks or properties as needed.

class JSLintPlugin implements Plugin
{
	void apply(Project project)
	{
		//configure the Project here
	}
}

Listing 2: Skeleton of the Plugin implementation

Integrating with Ant

I’ve never been overly fond of Ant, mostly due to the extremely verbose and repetitive nature of the xml declaration. But the fact remains that Ant has been a primary and well-used build tool for years, and the Tasks written for it have been tried and tested by many developers. The Groovy AntBuilder, in combination with the facilities Gradle provides for dependency resolution and classpath management, makes it very easy to incorporate existing Ant functionality into a build and abstract most of the details away from the end user. For this plugin, we add the library containing the Ant Task to a custom configuration so that we can have it automatically downloaded and easily resolve the classpath.
Listing 3 shows how the configuration for the JSLint4Java Task could appear in an Ant build.xml file. Note that you’re on your own here to provide the required library for the classpath.

Listing 3: Configuring the Ant target in a build.xml file

Gradle makes it easy to separate the configuration and execution phases of the build, allowing for a Plugin to add an Ant Task to the Gradle Project and expose the (optional) configuration in a build script. In addition, Gradle encourages a pattern of providing a ‘convention’ object alongside of a Plugin to clearly separate the concerns.
Listing 4 demonstrates some code from the Plugin implementation that adds a ‘jslint’ Task to a Gradle Project, setting the specific options based on a convention object. Note how we extract the classpath using the notation project.configurations.jslint.asPath.

private static final String TASK_NAME = 'jslint'
private Project project
private JSLintPluginConvention jsLintpluginConvention
...
// some of the code in the apply method
this.jsLintpluginConvention = new JSLintPluginConvention(project)
project.convention.plugins.jslint = jsLintpluginConvention
project.task(TASK_NAME) << {
    project.file(project.reportsDir).mkdirs()
    logger.info("Running jslint on project ${project.name}")
    ant.taskdef(name: TASK_NAME, classname: jsLintpluginConvention.taskName,
        classpath: project.configurations.jslint.asPath)
    ant."$TASK_NAME"(jsLintpluginConvention.mapTaskProperties()) {
        formatter(type: jsLintpluginConvention.decideFormat(),
                destfile:  jsLintpluginConvention.createOutputFileName())
        jsLintpluginConvention.inputDirs.each { dirName ->
            fileset(dir: dirName,
			 includes: jsLintpluginConvention.includes,
			 excludes: jsLintpluginConvention.excludes)
        }
    }
}

Listing 4: Adding a jslint Task to a Gradle Project

If you’re not already familiar with the Gradle syntax for creating new Tasks inline, the project.task(String taskName) method is called to instantiate a new Task, and the ‘< <' syntax is used to to push the Task activities into the 'doLast' Task lifecycle phase.
Allowing for configuration of the Task in a build script is as simple as exposing a method named the same as the Task that takes in a Closure parameter and applies that Closure to set properties on the convention object, as shown in Listing 5.

/**
 * Perform custom configuration of the plugin using the provided closure.
 * @param closure
 */
def jslint(Closure closure)
{
    closure.delegate = this
    closure()
}

Listing 5: A method to allow for configuration of the jslint Task from a build script

The simple use-case

As per usual when working with Gradle, using this plugin in the most basic case requires only these things:

Declare a dependency on the Plugin source. This can be either a released jar to be downloaded from a repository by Gradle or you can download a jar manually from pretty much anywhere and add it to the classpath directly.
Apply the Plugin to a Gradle build.
Call the jslint Task as part of the build.

The entire configuration and usage looks something like Listing 6, assuming that the gradle-jslint-plugin jar is found in /usr/home/gradlelibs.

/* In a build.gradle file */
buildscript {
	dependencies {
		classpath fileTree(dir: '/usr/home/gradlelibs', include: '*.jar')
	}
}
apply plugin: org.kar.jslint.gradle.plugin.JSLintPlugin

/* and on the command line... */
gradle jslint

Listing 6: Configuring the jslint Plugin in a build script and calling it from the command line

By default, this is enough to scan for all .js files under the directory where the build script is located and create a JSLint text report using the basic settings.

The not-so-simple case

Of course in the real world the defaults aren't always what we need, so being able to easily configure the Task is essential. Fortunately, Gradle makes extending a custom Plugin to allow for configuration by a simple Closure, so we can exercise the code from Listing 5 in a build script with the Closure definition in Listing 7.

jslint {
	haltOnFailure = false
	excludes = '**/metadata/'
	options = 'rhino'
	formatterType = 'html'
}

Listing 7: Departing from the default jslint Task configuration

Extending Ant Task capabilities

The Ant Task as is can produce either a plain text document or an xml report, but transforming the results into a more consumable html format is easy to do using an Ant xslt Task. Having Gradle wrap the Task definition allows for simply adding a new formatter type to the configuration and abstracting away the details from the end user. A copy of an xsl file available online is easy to incorporate with the plugin and can be used to transform the xml output into a nicely formatted web page. Being able to program around Ant Tasks like this is a great way to enhance their value in your build. An example of simple output from the test cases included in the Plugin is shown in Figure 1.

Figure 1: Example html formatted output from jslint

Testing using ProjectBuilder

In order to facilitate testing custom Tasks and Plugins, the Gradle framework provides the ProjectBuilder implementation to handle most of the heavy lifting. This gives you a mocked out instance of a Gradle Project that builds into a temporary directory; very handy for testing how Tasks behave under real working conditions. Having tools like this available directly from the framework removes a lot of potential barriers that might otherwise discourage testing of custom components. The source code that accompanies this article uses the ProjectBuilder to achieve 100% code coverage of the project and is available on github at https://github.com/kellyrob99/gradle-jslint-plugin if you'd like to look closer for some ideas on how to test your own Gradle Plugins. An already built version of the jar is also available if you'd like to try without having to build it yourself: https://github.com/kellyrob99/gradle-jslint-plugin/blob/master/releases/gradle-jslint-plugin-0.1-SNAPSHOT.jar.

How could we improve this Plugin?

This implementation represents the 'brute-force' method of wrapping an Ant Task, and there are several ways to enhance its function, if we wanted to spend some additional time and effort on the Plugin. It would actually be far more flexible if we extended the Gradle DefaultTask to provide the actual functionality; this would allow for the possibility of executing the Task separately against different sets of JavaScript in the same project with different configurations of JSLint. In the case of an application with both client and server-side JavaScript, for instance, you might want to apply different rules. In that event you'd probably also want to have the capability to aggregate multiple result sets, which would be a relatively easy feature to add.
Having a separate Task implementation defined would also make it easier to clearly define the inputs and outputs of the process using one of the @Input or @Output Gradle annotations, allowing for incremental builds including JSLint execution. The full set of annotations available from the org.gradle.api.tasks package allow for combining File and/or simple property types to make your Tasks smarter regarding whether or not running again would produce a different result than the last execution.
This article was created using the recently released 0.9 version of Gradle and version 1.4.4 of jslint4java.

Learn more

If you'd like to find out some background or more about Gradle and how to create your own custom Plugins and Tasks, you can find some good information on these sites:

The Gradle documentation at http://gradle.org/documentation.html
In particular, the Project API is a great starting point for getting to know Gradle: http://gradle.org/latest/docs/javadoc/org/gradle/api/Project.html
The excellent examples of Mr. Haki at http://mrhaki.blogspot.com/search/label/Gradle%3AGoodness
Published plugins on github can be found with this search: https://github.com/search?type=Repositories&language=&q=gradle+plugin

Hooking into the Jenkins(Hudson) API

TheKaptain — Sun, 27 Mar 2011 17:24:09 +0000

Which one – Hudson or Jenkins?

Both. I started working on this little project a couple of months back using Hudson v1.395 and returned to it after the great divide happened. I took it as an opportunity to see whether there would be any significant problems should I choose to move permanently to Jenkins in the future. There were a couple of hiccups- most notably that the new CLI jar didn’t work right out of the box- but overall v1.401 of Jenkins worked as expected after the switch. The good news is the old version of the CLI jar still works, so this example is actually using a mix of code to get things done. Anyway, the software is great and there’s more than enough credit to go around.

The API

Jenkins/Hudson has a handy remote API packed with information about your builds and supports a rich set of functionality to control them, and the server in general, remotely. It is possible to trigger builds, copy jobs, stop the server and even install plugins remotely. You have your choice of XML, JSON or Python when interacting with the APIs of the server. And, as the build in documentation says, you can find the functionality you need on a relative path from the build server url at:

“/…/api/ where ‘…’ portion is the object for which you’d like to access”.

This will show a brief documentation page if you navigate to it in a browser, and will return a result if you add the desired format as the last part of the path. For instance, to load information about the computer running a locally hosted Jenkins server, a get request on this url would return the result in JSON format: http://localhost:8080/computer/api/json.

{
  "busyExecutors": 0,
  "displayName": "nodes",
  "computer": [
    {
      "idle": true,
      "executors": [
        {
        },
        {
        }
      ],
      "actions": [

      ],
      "temporarilyOffline": false,
      "loadStatistics": {
      },
      "displayName": "master",
      "oneOffExecutors": [

      ],
      "manualLaunchAllowed": true,
      "offline": false,
      "launchSupported": true,
      "icon": "computer.png",
      "monitorData": {
        "hudson.node_monitors.ResponseTimeMonitor": {
          "average": 111
        },
        "hudson.node_monitors.ClockMonitor": {
          "diff": 0
        },
        "hudson.node_monitors.TemporarySpaceMonitor": {
          "size": 58392846336
        },
        "hudson.node_monitors.SwapSpaceMonitor": null,
        "hudson.node_monitors.DiskSpaceMonitor": {
          "size": 58392846336
        },
        "hudson.node_monitors.ArchitectureMonitor": "Mac OS X (x86_64)"
      },
      "offlineCause": null,
      "numExecutors": 2,
      "jnlpAgent": false
    }
  ],
  "totalExecutors": 2
}

Here’s the same tree rendered using GraphViz.

This functionality extends out in a tree from the root of the server, and you can gate how much of the tree you load from any particular branch by supplying a ‘depth’ parameter on your urls. Be careful how high you specify this variable. Testing with a load depth of four against a populous, long-running build server (dozens of builds with thousands of job executions) managed to regularly timeout for me. To give you an idea, here’s a very rough visualization of the domain at depth three from the root of the api.

Getting data out of the server is very simple, but the ability to remotely trigger activity on the server is more interesting. In order to trigger a build of a job named ‘test’, a POST on http://localhost:8080/job/test/build does the job. Using the available facilities, it’s pretty easy to do things like:

load a job’s configuration file, modify it and create a new job by POSTing the new config.xml file
move a job from one build machine to another
build up an overview of scheduled builds

The CLI Jar

There’s another way to remotely drive build servers in the CLI jar distributed along with the server. This jar provides simple facilities for executing certain commands remotely on the build server. Of note, this enables installing plugins remotely and executing a remote Groovy shell. I incorporated this functionality with a very thin wrapper around the main class exposed by the CLI jar as shown in the next code sample.

/**
 * Drive the CLI with multiple arguments to execute.
 * Optionally accepts streams for input, output and err, all of which
 * are set by default to System unless otherwise specified.
 * @param rootUrl
 * @param args
 * @param input
 * @param output
 * @param err
 * @return
 */
def runCliCommand(String rootUrl, List args, InputStream input = System.in,
        OutputStream output = System.out, OutputStream err = System.err)
{
    def CLI cli = new CLI(rootUrl.toURI().toURL())
    cli.execute(args, input, output, err)
    cli.close()
}

And here’s a simple test showing how you can execute a Groovy script to load information about jobs, similar to what you can do from the built-in Groovy script console on the server, which can be found for a locally installed deployment at http://localhost:8080/script.

def "should be able to query hudson object through a groovy script"()
{
    final ByteArrayOutputStream output = new ByteArrayOutputStream()
    when:
    api.runCliCommand(rootUrl, ['groovysh', 'for(item in hudson.model.Hudson.instance.items) { println("job $item.name")}'],
            System.in, output, System.err)

    then:
    println output.toString()
    output.toString().split('\n')[0].startsWith('job')
}

Here are some links to articles about the CLI, if you want to learn more :

HTTPBuilder

HTTPBuilder is my tool of choice when programming against an HTTP API nowadays. The usage is very straightforward and I was able to get away with only two methods to support reaching the entire API: one for GET and one for POST. Here’s the GET method, sufficient for executing the request, parsing the JSON response, and complete with (albeit naive) error handling.

/**
 * Load info from a particular rootUrl+path, optionally specifying a 'depth' query
 * parameter(default depth = 0)
 *
 * @param rootUrl the base url to access
 * @param path  the api path to append to the rootUrl
 * @param depth the depth query parameter to send to the api, defaults to 0
 * @return parsed json(as a map) or xml(as GPathResult)
 */
def get(String rootUrl, String path, int depth = 0)
{
    def status
    HTTPBuilder http = new HTTPBuilder(rootUrl)
    http.handler.failure = { resp ->
        println "Unexpected failure on $rootUrl$path: ${resp.statusLine} ${resp.status}"
        status = resp.status
    }

    def info
    http.get(path: path, query: [depth: depth]) { resp, json ->
        info = json
        status = resp.status
    }
    info ?: status
}

Calling this to fetch data is a one liner, as the only real difference is the ‘path’ variable used when calling the API.

private final GetRequestSupport requestSupport = new GetRequestSupport()
    ...
/**
 * Display the job api for a particular Hudson job.
 * @param rootUrl the url for a particular build
 * @return job info in json format
 */
def inspectJob(String rootUrl, int depth = 0)
{
    requestSupport.get(rootUrl, API_JSON, depth)
}

Technically, there’s nothing here that limits this to JSON only. One of the great things about HTTPBuilder is that it will happily just try to do the right thing with the response. If the data returned is in JSON format, as these examples are, it gets parsed into a JSONObject. If on the other hand, the data is XML, it gets parsed into a Groovy GPathResult. Both of these are very easily navigable, although the syntax for navigating their object graphs is different.

What can you do with it?

My primary motivation for exploring the API of Hudson/Jenkins was to see how I could make managing multiple servers easier. At present I work daily with four build servers and another handful of slave machines, and support a variety of different version branches. This includes a mix of unit and functional test suites, as well as a continuous deployment job that regularly pushes changes to test machines matching our supported platform matrix, so unfortunately things are not quite as simple as copying a single job when branching. Creating the build infrastructure for new feature branches in an automatic, or at least semi-automatic, fashion is attractive indeed, especially since plans are in the works to expand build automation. For a recent 555 day project, I utilized the API layer to build a Grails app functioning as both a cross-server build radiator and a central facility for server management. This proof of concept is capable of connecting to multiple build servers and visualizing job data as well as specific system configuration, triggering builds, and direct linking to each of the connected servers to allow for drilling down further. Here’s a couple of mock-ups that pretty much show the picture.

Just a pretty cool app for installing Jenkins

This is only very indirectly related, but I came across this very nice and simple Griffon app, called the Jenkins-Assembler which simplifies preparing your build server. It presents you with a list of plugins, letting you pick and choose, and then downloads and composes them into a single deployable war.

Enough talking – where’s the code???

Source code related to this article is available on github. The tests are more of an exploration of the live API than an actual test of the code in this project. They run against a local server launched using the Gradle Jetty plugin. Finally, here’s some pretty pictures for you.

[Show as slideshow]

[View with PicLens]

Why do I Like Gradle?

TheKaptain — Sat, 13 Nov 2010 23:53:47 +0000

Gradle, if you don’t already know it, is rapidly gaining traction as a strong leader in the next generation of build systems. It builds heavily upon excellent aspects of the Maven and Ant frameworks, yet is pitched as not suffering from the same “Frameworkitis“. And I’ve gotta say – the results are pretty spectacular. Among the major selling features, at least as I see it, are:

Groovy syntax and a very terse and descriptive dsl that makes build scripts easily comprehensible
flexibility of layout, configuration, organizing build logic – pretty much everything
incremental builds, based on an easily implementable pattern
convention over configuration paradigm, thank you very much Maven
clear separation of build configuration from execution
extensibility at every level

The most basic Java build

apply plugin: 'java'

That’s it. One line of Groovy in a file called ‘build.gradle’ and you can build a Java project with a Maven-standardized project layout.

├── build.gradle
└── src
    ├── main
    │   ├── java
    │   └── resources
    └── test
        ├── java
        └── resources

Included with the Java plugin are tasks to compile, package, test and javadoc your code. You also get configuration objects to describe the artifacts that your build depends on and those that it produces. Of course, with this most basic setup these configurations don’t yet have anything in them, but in a complex build they’re very handy for isolating the responsibilities of each task. Because you can both configure the existing configurations and add custom ones yourself, it’s very easy to accommodate a project that follows a different structure, whether that is just differently named source directories or multiple directories that need specific processing. This is VERY handy for legacy builds.

Incremental builds

Gradle provides a very easy way to create tasks that are able to execute only if their declared input and/or output artifacts have changed. This makes it trivial to incorporate your custom build behaviour into an incremental build. As an example I’d like to expand upon something I read by Etienne Studer in this month’s JAXmag. It’s a great example of developing an incremental task. First here’s the task implementation almost verbatim from the article. I’ve updated it slightly to make the output more readable using FileUtils, and you can examine the file it spits out from the build/reports/size directory that will be automatically created when it executes.

class Size extends DefaultTask
{
    @InputFiles FileTree inputDir
    @OutputFile File outputFile

    @TaskAction
    void generate()
    {
        def totalSize = inputDir.files.inject(0) { def size, File file -> size + file.size()
        }
        outputFile.text = FileUtils.byteCountToDisplaySize(totalSize)
    }
}

If you simply include this class definition in a build file, it will be automatically compiled and available for use elsewhere in the script. It could just as easily be defined in a separate file(local or remote), in a jar on the classpath or in the buildSrc directory of your Gradle project. This flexibility enables developing and evolving tasks in a very agile fashion, encouraging you to publish the results for reuse instead of re-implementing the logic in other projects.
In order to execute this task as part of a build, we first need to configure it. In this case, I’m configuring it to work on all declared configurations and all source, both code and associated resources. In order to do so, I’m using a couple of Gradle internal classes, FileTree and SourceSet, which actually sound pretty self-explanatory to me. The key part is the assignment of the ‘inputDir’ and ‘outputFile’ properties on the task.

task size(type:Size){
    def filetree = sourceSets.inject(new UnionFileTree()) { FileTree total, SourceSet sourceSet ->
        total += sourceSet.allSource
        total
    }
    inputDir = filetree
    outputFile = file("$reportsDir/size/size.txt")
}

Just to prove that it’s working incrementally, here’s the output from successive invocations. Note that ‘UP-TO-DATE’ in the output that indicates the task was skipped the second time around, because none of the inputs changed and the output file hasn’t been deleted.

gradle-intro$ gradle size
:size

BUILD SUCCESSFUL

Total time: 2.963 secs

gradle-intro$ gradle size
:size UP-TO-DATE

BUILD SUCCESSFUL

Total time: 2.912 secs

This task does actually have a concrete dependency on the Java plugin, since without that convention applied neither the ‘sourceSets’ or ‘reportsDir’ objects would be present. Here’s the complete version of the build file that’s been built so far, 28 lines including imports and spacing.

import org.apache.commons.io.FileUtils
import org.gradle.api.internal.file.UnionFileTree
import org.gradle.api.tasks.SourceSet

apply plugin: 'java'           

task size(type:Size){
    def filetree = sourceSets.inject(new UnionFileTree()) { FileTree total, SourceSet sourceSet ->
        total += sourceSet.allSource
        total
    }
    inputDir = filetree
    outputFile = file("$reportsDir/size/size.txt")
}

class Size extends DefaultTask
{
    @InputFiles FileTree inputDir
    @OutputFile File outputFile

    @TaskAction
    void generate()
    {
        def totalSize = inputDir.files.inject(0) { def size, File file -> size + file.size()
        }
        outputFile.text = FileUtils.byteCountToDisplaySize(totalSize)
    }
}

Sharing your new Task

The simplest way to share build logic is to ‘apply’ it. This simple shorthand covers everything from plugins to local files to remotely hosted resources. Here’s how easy it is to incorporate this particular task implementation and configuration into a build using a relative file path.

apply from : '../gradle-intro/build.gradle'

And because only a simple http connection is required to share it to a broader base, here’s how you can reference a copy on this site. Sorry about the .txt extension, but I didn’t feel like editing php to allow a new filetype today

apply from : 'http://www.kellyrob99.com/blog/wp-content/uploads/downloads/2010/11/gradleSizeTask.txt'

Some other reading

If you’re still not sold on Gradle, here’s some articles I’ve seen recently that at the very least will give you a better perspective.
Hibernate – Gradle why?
Maven VS Gradle VS Ant
Maven to Gradle: Part 1, Part 2, Part 3
A comparison of build script length
Ant/Gradle/Maven comparison
DZone article
OpenMRS Mailing list on Maven VS Gradle

Groovy inspect()/Eval for Externalizing Data

TheKaptain — Sun, 26 Sep 2010 20:32:13 +0000

One of the things I love about Groovy is how easy it makes reading and writing text files. I’ve written Groovy scripts for everything from parsing log files for extracting timing information to finding (and replacing with selectors) in-line css blocks. Often there’s a piece of information extracted from a file that I want to keep for further examination, or just for future reference. The Groovy inspect() method provides a nice easy way to take simple results that are stored in variables and write them to a file. Then the Groovy Eval class provides convenience methods to parse that information back from the file in a single line. Here’s examples of test code that creates Lists and Maps, writes them out to a file and then asserts that evaluating the stored Groovy code results in the same data structures.

    private static final Closure RANDOM_STRING = RandomStringUtils.&randomAlphanumeric
    private static final String TMP_DIR = System.getProperty('java.io.tmpdir')

    @Test
    public void testSerializeListToFile()
    {
        List accumulator = []
        100.times { int i ->
            accumulator << RANDOM_STRING(i + 1)
        }
        File file = new File("$TMP_DIR/inspectListTest.groovy")
        file.deleteOnExit()
        file << accumulator.inspect()
        assertThat(accumulator, equalTo(Eval.me(file.text)))
    }

    @Test
    public void testSerializeMapToFile()
    {
        Map accumulator = [:]
        100.times { int i ->
            accumulator[RANDOM_STRING(i + 1)] = RANDOM_STRING(i + 1)
        }
        File file = new File("$TMP_DIR/inspectMapTest.groovy")
        file.deleteOnExit()
        file << accumulator.inspect()
        assertThat(accumulator, equalTo(Eval.me(file.text)))
    }

Two little tidbits of goodness embedded in this example are the ability to capture a method as a closure, in this case RandomStringUtils.randomAlphanumeric(), and the File.deleteOnExit() method – which is only cool because I never noticed it in the API before and it turns out to be a great way to clean up after tests.

A particular usage of this technique I’ve been using lately has been sparked by a shift of tools at work. I’ve worked with the Atlassian stack of web applications for years now, and have always enjoyed the rich feedback Bamboo gives for build history. But now I’m using Hudson as the primary continuous integration tool and one of the things I’ve been sorely missing is the ‘Top 10 Longest Running Tests’. If you’re not familiar with this view in Bamboo, stroll on over to the Groovy build results page and click on the tab.
One of my present priorities at work is to speed up build times and rooting out slow or inefficient tests is one way to do this. It’s possible to elicit this information from JUnit test reports using xsl(see this link for an example) but I came up with a nice way to incorporate similar functionality into a Gradle build. Following the Gradle project-report conventions, this task simply uses the inspect() method to write out a results file in the ‘reportsDir’. I’ve captured this output(manually) over time to track the progress of speeding up test times, and the format makes it easy to read back in results and do deeper analysis and aggregation – such as the creation of csv reports and simple graphs for instance.

tolerance = 2.0
task findLongRunningTests << {
    description= "find all tests that take over $tolerance seconds to run"
    String testDir = "${project.reportsDir}/tests".toString()
    file(testDir).mkdirs()
    File file = file("$testDir/longRunningTests.txt")
    file.createNewFile()
    BufferedWriter writer = file.newWriter()
    writer << parseTestResults('**/TESTS-TestSuites.xml', tolerance)
    writer.close()
}

/**
 * Read in xml files in the junit format and extract test names and times
 * @includePattern ant pattern describing junit xml reports to inspect, recursively gathered from the rootDir
 * @tolerance the number of seconds over which tests should be included in the report
 */
private String parseTestResults(includePattern, float tolerance)
{
    def resultMap = [:]
    fileTree(dir: rootDir, include: includePattern).files.each {
        def testResult = new XmlSlurper().parse(it)
        testResult.depthFirst().findAll {it.name() == 'testcase'}.each { testcase ->
            def key = [testcase.@classname.text(), testcase.@name.text()].join(':')
            def value = testcase.@time.text() as float
            resultMap[(key)] = value
        }
    }
    return resultMap.findAll {it.value > tolerance}.sort {-it.value}.inspect()
}

Note that this works recursively from the ‘rootDir’ of a Project, so it is effective for multi-project Gradle builds. Source code for this simple example is available on github if you want to check it out yourself.

Here’s a sample result drawn from examining some test output in the Gradle build itself. Only one of the tests executed in WrapperTest takes more than 2 seconds to execute.

["org.gradle.api.tasks.wrapper.WrapperTest:testWrapper":2.37]

Groovy and CSV: How to Get Your Data Out?

TheKaptain — Thu, 01 Jul 2010 21:07:35 +0000

I don’t know exactly how many CSV files I’ve read/written to date, but I’m willing to bet it’s a lot. These kind of files are a simple and common way to exchange data and are interoperable with spreadsheet programs as well, making them more easily accessible to non-programmer types. There is some excellent support out there for reading and writing CSV files and so here, from simplest to most complex, are a few different ways to load and query the data trapped inside your CSV files. But first off…

CSV is Not XML

Or any other suitably constrained data format for that matter. There are some pretty clear ‘rules’ out there for transporting data in this format, but of course hardly anybody listens to them. I found this and this in the first page of a quick Google search.
However, while looking for available sample material on the web to demonstrate with, I found many great examples of how to screw it up: not quoting empty strings, not escaping embedded line breaks, commas embedded in unquoted text, etc. All of these are things that prevent automation from properly interacting with the data. There are also, of course, the CSV ‘formats’ that insist on embedding comments and descriptions, trying to present more than one matrix of information or any of the other crazy things you can do when writing to a format with no real metadata layer.

< \rant>

Unfortunately, there’s nothing as convenient as an xml schema available for describing what information is encoded in the document and, in general, the only potentially useful metadata in an individual file is the header line. The good news is that all of these deficiencies can be overcome if you know they exist. And there’s some pretty nifty tools you can use to take out much of the guesswork. I’ll suggest a couple of ideas for how to deal with files that don’t adhere to the ‘rules’, but the main example I’ve decided on is a straightforward and well formed set representing the periodic table of elements, provided generously by Akiscode. This file is directly machine readable(has no headers or embedded data), quotes all values and just follows all the ‘rules’. Here’s the first five lines as an example.

Atomic Number	Atomic Mass	Name	Symbol
1	1.00794	Hydrogen	H
2	4.002602	Helium	He
3	6.941	Lithium	Li
4	9.012182	Beryllium	Be
5	10.811	Boron	B

The simple format of the example CSV file.

Groovy All by Itself

Groovy makes dealing with String objects pretty painless. By adding facilities like easy casting and find()/findAll() for quickly turning raw Strings into real data, you can program some fairly complex questions. In this simple example I’m depending on all values being quoted and I am not protecting against casting problems. I’m running this through Maven which really helps to keep test data organized; all you have to do is drop your test file in /src/test/resources and it’s automagically available on the test classpath.
No specific handling is done for different types here; everything is read in as a String and cast to a more specific type as needed. In a ‘real’ application you’re very likely going to encounter mixed alphanumeric data in any given column of data – ‘UNKNOWN’ instead of a number in a column labeled ‘Quantity’ for instance – and be more careful about casting.

//load and split the file
InputStream inputFile = getClass().classLoader.getResourceAsStream(TEST_FILE_NAME)
String[] lines = inputFile.text.split('\n')
List rows = lines.collect {it.split(',')}

/**
 * A little helper method to get rid of the quotes in the input
 * and cast values so they can be compared properly.
 */
private double castToDouble(string)
{
    return string.replaceAll('"', '').toDouble()
}

//OK, it's parsed - let's ask some questions
private static final int ATOMIC_MASS = 1;

def elementsOver200Mass = rows.findAll {castToDouble(it[ATOMIC_MASS]) > 200}
def elementsBetween10And20 = rows.findAll { row ->
            double mass = castToDouble(row[ATOMIC_MASS])
            mass <= 20 && mass >= 10
}

So this approach works fine for well formed input, but falls apart quickly in other cases. For instance, if any columns in the file are missing values the split() function treats them as nulls, leading to different size arrays being stored in the list. Similarly you need to deal with quoted VS non-quoted content, embedded line breaks and other issues individually. Fortunately, some nice guys have done that for us.

OpenCSV

Glen Smith’s OpenCSV library nicely abstracts away a lot of the minutiae of dealing with CSV. It properly deals with escaped characters, multiline input and a host of other issues for you. Give it pretty much anything that satisfies the Reader interface as an input and it’s good to go. The return type of the CSVReader.readAll() method also satisfies the same contract as the plain Groovy version shown above, so we can interact with the parsed results in exactly the same way as in the previous example.
All values are still typed as Strings, so we need to cast in order to compare numeric values.

import au.com.bytecode.opencsv.CSVReader

List rows = new CSVReader(
        new InputStreamReader(getClass().classLoader.getResourceAsStream(TEST_FILE_NAME)))
            .readAll()

//same finders as in the Groovy version
def elementsOver200Mass = rows.findAll {it[ATOMIC_MASS].toDouble() > 200}
def elementsBetween10And20 = rows.findAll { row ->
    double mass = castToDouble(row[ATOMIC_MASS])
    mass <= 20 && mass >= 10
}

HsqlDB

HsqlDB allows for using flat files as Text Tables, effectively turning a CSV file into a database table. Data values are cast to the types specified in your table definition, so any potential type cast failures happen right up front when loading the data. The benefit is that once the data is read in no further manual processing is necessary. Note: by default the csv file and the database file must be located in the same directory, as a security precaution (can be overridden through configuration).

In addition, leveraging sql makes querying the data extremely easy. Groovy brings in the Sql class to abstract away a lot of the normal Java boilerplate you encounter when working with a ResultSet. I didn’t do any in-depth testing to prove it out, but this is also the only one of the strategies described here that doesn’t require holding the entire data in memory in order to do arbitrary queries. That can be mitigated in the previously shown methods by processing files line by line rather than in bulk if memory usage is a concern.

//create the table definition to insert
String tableName = 'elements'
String tableDefinition = """CREATE TEXT TABLE $tableName (
    atomic_number INTEGER PRIMARY KEY,
    atomic_mass NUMERIC,
    name VARCHAR(255),
    symbol VARCHAR(3)
);"""

//create a new file database and a table corresponding to the csv file
Sql sql = Sql.newInstance("jdbc:hsqldb:file:${testdbDir.absolutePath}/testdb", 'sa', ''
    ,'org.hsqldb.jdbcDriver')
sql.execute(tableDefinition)

//set the source to the csv file
sql.execute("SET TABLE elements SOURCE '${TEST_FILE_NAME};all_quoted=true'".toString())

//querying the database that's wrapping our CSV file
def elementsOver200Mass = sql.rows("SELECT * FROM $tableName WHERE atomic_mass > ?", [200])
def elementsBetween10And20 = sql.rows(
    "SELECT * FROM $tableName WHERE atomic_mass <= ? AND atomic_mass >= ?", [20, 10])

//simple db aggregates
def count = 0
sql.eachRow("SELECT count(1) FROM $tableName WHERE atomic_mass <= ?", [20]){row->
    count = row[0]
}
def avg = 0
sql.eachRow("SELECT avg(atomic_mass) FROM $tableName".toString()){row->
    avg = row[0]
}

Which One do I Use???

Pick the one that fits best for your use case is the real answer. I didn’t know about the HsqlDB option until recently, but OpenCSV has been a personal standby for years. Then again, if you’re writing something quick and dirty at the script level, the simplicity of just applying a couple of split() operations is pretty appealing. Here’s how I generally decide.

	Plain Groovy	OpenCSV	HsqlDB
Use it	File contains well-formed input and questions to be asked involve a small subset of columns.	File contains well or badly formed input and questions to be asked involve a small subset of columns.	File contains well or badly formed input, questions to be asked involve many columns or span multiple files.
Don't use it	File contains badly formed input or questions being asked are complex.	Not able to easily leverage the OpenCSV library.	Not able to easily leverage the HsqlDB database.
Where it works	Groovy only	Groovy or Java	Groovy or Java

Some common sense guidelines for which method to apply given a problem to solve.

A Grails App Demoing the StackExchange API

TheKaptain — Thu, 03 Jun 2010 07:12:17 +0000

So I was making an attempt to catch up on my Google Reader‘ing this weekend and I came across this post on StackOverflow regarding the shiny new StackExchange API. That and a fresh 1.3.1 drop of Grails seemed like as good a reason as any to hack together a little app suitable for seeing what you can do with said API.
Turns out it’s not a whole lot, at least not yet. The minor detraction of having read-only access and limited connects without an API key are more than offset by seeing such a minimal, clean and easy to use interface. According to the grails stats script, it took about this much effort to implement calls to each and every one of the API endpoints for all of the supported domains:

    +----------------------+-------+-------+
    | Name                 | Files |  LOC  |
    +----------------------+-------+-------+
    | Controllers          |     1 |    42 |
    | Groovy Helpers       |     1 |    47 |
    +----------------------+-------+-------+
    | Totals               |     2 |    89 |
    +----------------------+-------+-------+

Please note that this is about as minimal as you could reasonably get away with(and kinda ugly to boot), but nevertheless it does manage to implement a UI and backend for exercising the entire API in a total of 2 gsp files and 1 controller action. What’s noticeably missing are some tailored views for each of the different JSON responses and the implementation of additional query parameters on the calls.

StackExchange API

This is really pretty well documented and “consistent”. Calls in some cases require an {id} in the url and that’s about it. Each of the individual calls has its own help page that describes the options, like the badges page for instance. To make things a little more helpful while exploring the API, each of the implemented endpoints in this app are also hyper-linked to the corresponding manual page on api.stackoverflow.com.

RESTful Access

Couldn’t be a whole lot easier than using HttpBuilder, which provides nice and concise ways to execute the GET request, inspect the response and deal with success/failure of the request. Aside from assigning the root domain and method using variables, this is little different from the canonical example on codehaus.

def http = new HTTPBuilder("http://api.$domain")
http.request(GET, JSON) {
    uri.path = "/${VERSION}$method"
    headers.'User-Agent' = 'Mozilla/5.0 Ubuntu/8.10 Firefox/3.0.4'
     response.success = { resp, json ->
        answer = json
    }
     response.failure = { resp ->
        answer = "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
    }
}

The syntax here could probably get even Groovier using the Grails REST client plugin wrappers, and indeed incorporating that plugin is how the HttpBuilder dependency is being provided in this app, but I hit on a working implementation first time out so we’ll leave that for another day, shall we?

What’s it Look Like?

Did I mention it was ugly? Where the API call requires a parameter, you get a text field. Each domain gets its own submit button, officially representing my first attempted usage of g:actionSubmit in Grails. Doesn’t work quite the way I expected, but it does certainly work. Click a button and you’re shown the raw JSON result, along with the call you made. In an ugly fashion, or did I say that already? Still, for a couple of hours of hacking, I’m not unhappy with the result.

StackApps

This is where the apps in the contest get shown off and, in a wonderful display of dogfooding, it’s guess what – another StackOverflow clone! The page for this application can be found here. And the source code is up on github if you’d like to take a look.

Next Steps?

Toying with the idea of porting this to appengine, which would unfortunately mean replacing HttpBuilder and the underlying Apache HttpClient in favor of something that didn’t use a threaded approach, most likely using URL fetch. That and some slightly less ugly tabular and/or tree views of the JSON responses would be kinda nice to have.

Achieving Groovy-like Fluency in Java with Google Collections

TheKaptain — Sun, 16 May 2010 05:33:19 +0000

One of the most compelling things about using Groovy is the fluent and concise syntax, as well as the productivity and readability gains that come out of it. But there’s no reason not to take advantage of some of the same techniques and some library support, in this case google-collections, to make Java code easy to write AND to read.

Creating Collections

Groovy really shines for this one, making the creation of Lists and Maps, empty or populated, an absolute breeze. Java has some help for creating basic Lists but begins to struggle when creating maps. This is an area that google-collections can help in, especially in regards to creating immutable Maps.

//Empty Lists
        List groovyList = []
        List javaList = new ArrayList()
        List googleList = Lists.newArrayList()  //can omit generics

//Populated Lists
        List groovyList = ["1", "2"]
        List javaList = Arrays.asList("1", "2")
        List googleList = Lists.newArrayList("1", "2")

//Immutable Lists
        List groovyList = ["1", "2"].asImmutable()
        List javaList = Collections.unmodifiableList(Arrays.asList("1", "2"))
        List googleList = ImmutableList.of("1", "2")

//Empty Maps
        Map groovyMap = [:]
        Map javaMap = new LinkedHashMap()
        Map googleMap = Maps.newLinkedHashMap()

//Immutable Maps
        Map groovyMap = ["a":"1", "b":"2"].asImmutable()

        Map javaMap = new LinkedHashMap()
        javaMap.put("a", "1")
        javaMap.put("b", "2")
        javaMap = Collections.unmodifiableMap(javaMap)

        //OR(works only in Java, will not compile in Groovy)
        Map javaMap = new LinkedHashMap()
        {
            {
                put("a", "1");
                put("b", "2");
            }
        };

        Map googleMap = ImmutableMap.of("a", "1", "b", "2")  //clunky syntax but it works

Filtering Collections

Groovy provides the very handy ‘findAll’ method that allows you to filter a Collection by applying a Closure. Google-collections provides similar facilities using the Predicate interface and two filter methods available statically on Collections2 and Iterables. This would also be a lot more readable if the Predicate definition were extracted but I wanted to show that it’s still possible to create them in-line quickly.

import static com.google.common.collect.Collections2.*

List toFilter = [1, 2, 3, 4, 5]
List groovyVersion = toFilter.findAll{ it < 3}
List googleVersion = filter(toFilter, new Predicate()
    {
        public boolean apply(Integer input)
        {
            return input < 3;
        }
    };)

Joining Collections into a String Representation

This one has come up pretty often over the years, and it’s not surprising that where the JDK hasn’t provided, enterprising developers have added support through libraries. The problem is: given a Collection of objects, create a String representation of that Collection suitable for view from a consumer of the system. And admit it – the first time you hand-coded the algorithm you left a trailing comma, didn’t you? I know I did.
Groovy has fantastic support for this use-case by simply including the ‘join’ method on Lists. Google-collections utilizes static calls on the Joiner class along with a simple DSL-like syntax to achieve the same effect. Throw in a static import to make it even more concise and it does a fine job. These two examples yield the same result.

import static com.google.common.base.Joiner.*
def toJoin = ['a', 'b', 'c']
def separator = ', '

//groovy version
def groovyJoin = toJoin.join(separator)

//google-collections version
def googleJoin = on(separator).join(toJoin)

And google-collections also supports join for Maps, something missing from Groovy(although not very hard to implement).

import static com.google.common.base.Joiner.*
def toJoin = [1: 'a', 2: 'b', 3: 'c']
def separator = ', '
def keyValueSeparator = ':'

//results in '1:a, 2:b, 3:c' which is essentially what is returned from Groovy map.toMapString()
def googleVersion = on(separator).withKeyValueSeparator(keyValueSeparator).join(map)

//results in '1 is a and 2 is b and 3 is c'
googleVersion = on(" and ").withKeyValueSeparator(" is ").join(map)

//doing the same in Groovy is slightly more involved, but really not that bad
def groovyVersion = toJoin.inject([]) {builder, entry ->
            builder << "${entry.key} is ${entry.value}"
            builder
        }.join(' and ')

Multimaps

Multimaps are one of the more interesting parts of google-collections, at least to me. Have you ever found yourself writing code to create a Map of keys to Lists? Well the various Multimap implementations in google-collections mean you never have to write that boilerplate kind of code again. Here’s a “first-stab” effort to simulate a fairly generic Multimap with pure Java code.

public class JavaMultimap
{
    private Map> multimap = new LinkedHashMap>();

    public boolean put(Object key, Object value)
    {
        List objects = multimap.get(key);
        objects = objects != null ? objects : new ArrayList();
        objects.add(value);
        multimap.put(key, objects);
        return true;
    }
}

And the same thing in Groovy, achieving a slightly smaller version.

class GroovyMultimap
{
    Map map = [:]

    public boolean put(Object key, Object value)
    {
        List list = map.get(key, [])
        list.add(value)
        map."$key" = list
    }
}

I did some primitive timings comparing Java, Groovy and google-collections Multimaps implementations and, as you’d pretty much expect, google clearly takes the lead. Where things really start to get interesting though is when you start using the Multimap in Groovy code. Imagine if you will that you want to iterate over a collection of Objects and map some of the properties to a List. Here’s a contrived example of what I’m talking about, but applying this same pattern to domain objects in your application or even a directory full of xml files is pretty much the same. If you look closely you’ll notice that Groovy actually makes this a one liner to extract all values for a property across a List of Objects(used in the assertion), but I imagine that Multimap is probably a better alternative for large data sets.

class GoogleCollectionsMultiMapTest
{
    private Random random = new Random()

    @Test
    public void testMultimap()
    {
        def list = []
        10.times {
            list << createObject()
        }
        List properties = ['value1', 'value2', 'value3']
        Multimap multimap = list.inject(LinkedListMultimap.create()) {Multimap map, object ->
            properties.each {
                map.put(it, object."$it")
            }
            map
        }
        properties.each {
            assertEquals (multimap.get(it), list."$it")
        }
    }

    Object createObject()
    {
        Expando expando = new Expando()
        expando.value1 =  random.nextInt(10) + 1
        expando.value2 =  random.nextInt(100) + 100
        expando.value3 =  random.nextInt(50) + 50
        return expando
    }
}

So Where Does This Get Us?

Between google-collections and the newer guava-libraries that contain it, there’s lots of help available for simplifying programming problems and making your code more readable. I haven’t even touched on the newly available support for primitives, Files, Streams and more, but if you’re interested in reducing the amount of code you write AND simultaneously making it more readable you should probably take a look. There’s a very nice overview available in part one and part two by Aleksander Stensby. And here’s a closer look at what google-collections can do for you.

Source Code

As per usual, source code is available at github as a maven project. Big thanks to the Spock team for sharing how they configure GMaven to properly utilize Groovy 1.7. Please feel free to take a look and comment here. Thanks!

Groovy and Hibernate Validator for Dynamic Constraints

TheKaptain — Sun, 18 Apr 2010 06:17:50 +0000

I was reading this article about hibernate validator today and it inspired me to apply a little Groovy to the problem of validating a bean. More specifically, finding out how hard it would be to apply different validation rules to the same classes at runtime. Turns out it’s really pretty simple.
Hibernate validator, if you didn’t already know, is the reference implementation of JSR-303 and it provides the ability to specify by xml or annotation configuration validation rules for pojos.

Where Hibernate Validator Shines

Annotations on domain classes allow for easily validating object state at the time of persistence. Excellent integration with frameworks like JBoss Seam allow this same ability to be utilized for validating web forms on the client-side with little more than an tag. Seam practically hides the entire interaction with validation components from the developer. Since validation rules are defined directly in the domain class, you can (almost) guarantee that no objects with inconsistent state will ever end up being saved in your database. There are certain validations that aren’t possible to verify without actually looking in the database, unique constraints for example, but generally in my experience hibernate validator is extremely easy to configure and work with. Implementing CRUD functionality is pretty trivial, and UIs can achieve consistency since all validations are applied equally.
Alternatively, if you can’t or don’t want to use annotations for some reason, you can specify your validation rules in an xml file. Usually a singleton validation.xml file is made available on the classpath and picked up automagically when a ValidationFactory is created. A simple xml configuration looks like this:


    org.kar.test

and is meant to be applied to this simple class:

class ValidateTestableClass
{
    int id
    String name
    String description
    boolean enabled
}

Comparing with Grails Validation

Grails automatically provides validation capabilities for domain classes and command objects, and enables adding the same behavior to any pogo through a combination of the @Validateable annotation and a static constraints closure. Adding validation support to arbitrary classes also requires specifying which packages to scan for the annotation.
Plugin support from projects like bean-fields simplifies the handling of client-side validation and rendering error markers in the UI, an ability which the Grails framework provides natively by adding an ‘errors’ field directly onto the domain or command object class instances bound to a web form.

Dynamic Constraints

Both the hibernate validator and the Grails strategies for applying validation described here have the same limitation: both are universally applied to all instances of a class. There’s no easy apparent way to override those constraints at runtime, although I suspect that some fancy MOP’ing or configuration could probably be used to accomplish overrides at runtime.
Hibernate validator also supports creation of ad hoc validators by seeding with one or more xml documents. Or if you’re like me and hate hand editing xml, you can leverage Groovy to take a bit of the pain away. Here’s the same xml snippet from above in a Groovy Closure, generated simply by turning the DomToGroovy class loose on the raw xml:

looseConstraint = {
        mkp.declareNamespace(xsi: 'http://www.w3.org/2001/XMLSchema-instance')
        'constraint-mappings'(xmlns: 'http://jboss.org/xml/ns/javax/validation/mapping',
                'xsi:schemaLocation': 'http://jboss.org/xml/ns/javax/validation/mapping validation-mapping-1.0.xsd') {
            'default-package'('org.kar.test.objects')
            bean('class': 'ValidateTestableClass') {
                field(name: 'name') {
                    constraint(annotation: 'javax.validation.constraints.NotNull')
                }
            }
        }
    }

Losing all the angle brackets is a good start, but we really haven’t saved a lot of typing. Until you start taking advantage of the ability to define more complicated structures. Note the use of a list structure here to apply NotNull constraints to multiple fields.

strictConstraint = {
        mkp.declareNamespace(xsi: 'http://www.w3.org/2001/XMLSchema-instance')
        'constraint-mappings'(xmlns: 'http://jboss.org/xml/ns/javax/validation/mapping',
                'xsi:schemaLocation': 'http://jboss.org/xml/ns/javax/validation/mapping validation-mapping-1.0.xsd') {
            'default-package'('org.kar.test.objects')
            bean('class': 'ValidateTestableClass') {
                ['name', 'description'].each {
                    field(name: it) {
                        constraint(annotation: 'javax.validation.constraints.NotNull')
                    }
                }
                field(name: 'id') {
                    constraint(annotation: 'javax.validation.constraints.DecimalMin') {
                        element(name: 'value', '2')
                    }
                }
                field(name: 'enabled') {
                    constraint(annotation: 'javax.validation.constraints.AssertTrue')
                }
            }
        }
    }

Applying Dynamic Constraints

Applying constraints is as simple as converting Closures to xml and mapping them to a Configuration object, which then supplies a Validator to use. StreamingMarkupBuilder is utilized to create the xml behind the scenes.

    /**
     * Create a configuration object passing closures as validation mapping documents.
     * @param closures closures to render into validation mapping documents
     * @return config
     */
    public Configuration createConfig(Closure... closures)
    {
        Configuration config = Validation.byDefaultProvider().configure()
        closures.each {
            config.addMapping(new ByteArrayInputStream(GroovyXmlConversionUtil.convertToXml(it).bytes))
        }
        config
    }

I haven’t tested the use of multiple mappings extensively, but minimally each class you’re configuring must be confined to a single mapping – you can’t extend the validations by layering configurations on top of one another. You should however be able to map constraints for different classes in separate Closures.

Crying out for a Builder!

Going from Closures to xml is a quick and dirty way to test out this functionality, but what would really be nice is a Builder that could create an appropriate validation environment more directly. At the least it would allow for removing the namespace declarations and explicit package naming that make up the bulk of the content.

So what do you get?

1. Ability to declare validations against any existing Java or Groovy class without changing the source code
2. Programmatic ability to create the configuration of validations
3. A choice of which validations to apply at runtime
4. Consistency with the behavior of domain class validation

And what’d I get? A fun bit of quick coding on a Saturday afternoon. Nice! Source code is available on git-hub if you want to check it out.

Gource Visualizations of the Groovy/Grails/Griffon Projects

TheKaptain — Mon, 22 Mar 2010 01:58:41 +0000

That’s a whole lot of Gr8ness in the title, don’t you think? Ok, ok – I apologize for the pun. I whipped these up using the open source Gource visualization software and with HD content free on Youtube now, I just had to try that out too.

Gource

Gource is a software version control visualization program with support for Git, Mercurial, and with a tiny bit of extra effort, CVS and Subversion. It’s available as a macport install, as is the ffmpeg program used to stitch together a series of screenshots into the final video. All of these visualizations were created with Git or Git-svn. When the Gource program is running it brings up an interactive window so you can interact with it to a degree, affecting the camera, zoom, etc. The default resolution is 1024×768 but I tried to bump it up to 1920×1080. I’m assuming that stretched the limits of the rendering, as there’s quite a bit of pixellation in the final product; either that or the ffmpeg params need to be tweaked. The three visualizations on this page were all created using this one-liner command line invocation (broken down here over several lines for readability).

gource -1920x1080 --highlight-all-users --bloom-intensity .3 -a 1 -s .01
     --hide-dirnames --hide-filenames --stop-at-end --output-ppm-stream -
     | ffmpeg -y -b 3000K -r 60 -f image2pipe -vcodec ppm -i -
     -vcodec libx264 -vpre default  gource.mp4

Groovy

From the initial commit on August 28, 2003, Groovy has come a long way. I tried the Gource provided python script to translate an svn log file into the required format the first time, but cloning the repository with git-svn seemed to produce better results. It’s interesting to see that the overall structure of the Groovy project doesn’t look like is has changed very much over the years. I recognized a lot of the names you see on the screen. Be sure to watch the 1080p version if you’re having trouble reading the text and see if you can’t find some recognizable committers.

Click here to view the embedded video.

Grails

Starting on Sunday March 14, 2004 here’s how it all unfolded, at least from the perspective of the public Git repository. Grails looks like it grew a little more spontaneously than Groovy; there are more frequent restructurings and some transitions that seem to cover a whole lot of ground in very little time. In particular watch out for the burst of activity in the summer of 2009.

Click here to view the embedded video.

Griffon

September 17, 2008 marks the first commit for Griffon and this short but sweet video explodes right from the start. Benefiting greatly from the Grails model this project shows a lot of progress in a very short time indeed. Most, if not all, of the names you see here you probably caught in the previous two videos as well.

Click here to view the embedded video.

What’s the Point?

Watching these gives me (and maybe you) some appreciation for how much effort over time it takes to bring a new language/framework/technology to maturity and all of the individual contributions that go into it. Gource provides a lot more detail into the actual structure if you want to dig in. For purposes of keeping these images fairly uncluttered I removed file and directory names and just focused on the committers and the general structure. I think I gained a git of insight into how a codebase can evolve, and properly appreciate that organizing the increasing complexity of an ever-growing system over time is not a simple job. Plus they look cool. Enjoy!

Breaking Weak CAPTCHA in… slightly more than 26 Lines of Groovy Code

TheKaptain — Sun, 14 Mar 2010 23:44:59 +0000

I read an interesting article recently about using python and open source software to defeat a particular captcha implementation and I set out to see how hard it would be to do the same in Groovy. In particular, coming from the Java side of the fence, I was impressed by how the available libraries in python made loading, mutating and saving images so easy. Admittedly I have limited experience working with image data, but when I have it has always seemed like a complex(and easy to get wrong) process. Maybe there’s a Java library out there that provides a simple ‘image_resize’ method, but it’s certainly not in the BufferedImage API. Still, when porting the 26 lines of code over to Groovy, I was able to get it considerably less verbose than the Java equivalent.

The Pretty Pictures

Here are the three images to test against. In order to put them in a suitable format for the open source tesseract-ocr program to process we need to make them bigger, remove the background noise and transform them into a ‘tif’ format. The python program we’re porting utilizes the PIL library for image handling and the pytesseract library for wrapping tesseract; I didn’t look very hard for java equivalents and just coded the required functions directly.

Reading in the Image

The python code for this is three lines, one to load the image and a couple more to convert it into a format suitable for directly manipulating pixel color through RGB values. Groovy takes a bit more to do the same, but being able to use a ‘with’ block makes interacting with the Graphics object a lot cleaner than the same Java code

//python
from PIL import Image
img = Image.open('input.gif')
img = img.convert("RGBA")
pixdata = img.load()

//Groovy
BufferedImage image = ImageIO.read(new File(fileName))
BufferedImage dimg = new BufferedImage(image.width, image.height, BufferedImage.TYPE_INT_ARGB)
dimg.createGraphics().with {
    setComposite(AlphaComposite.Src)
    drawImage(image, null, 0, 0)
    dispose()
}

Removing the Background Noise

In both cases we’re doing essentially the same thing: finding all non-black pixels and setting them to white. This leaves only the actual embedded text to stand out. Being able to utilize the Java Color constants makes the Groovy version a little more readable, IMO, but otherwise the two pieces of code are generally equivalent.

//python
for y in xrange(img.size[1]):
    for x in xrange(img.size[0]):
        if pixdata[x, y] != (0, 0, 0, 255):
            pixdata[x, y] = (255, 255, 255, 255)

//Groovy
(0..
    (0..
        if (dimg.getRGB(j, i) != Color.BLACK.RGB)
        {
            dimg.setRGB(j, i, Color.WHITE.RGB)
        }
    }
}

Resizing the Image

Python‘s library usage really shines here, making this a one line call. Not quite the same in Java-land, although again there’s probably a better way to do this(I just don’t know it offhand).

//python
big = im_orig.resize((116, 56), Image.NEAREST)

//Groovy
dimg = resizeImage(dimg, 116, 56)
...
def resizeImage = {BufferedImage image, int w, int h ->
    BufferedImage dimg = new BufferedImage(w, h, image.type)
    dimg.createGraphics().with {
        setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BILINEAR)
        drawImage(image, 0, 0, w, h, 0, 0, image.width, image.height, null)
        dispose()
    }
    return dimg
}

By this point the original images now look like this, and are almost ready for OCR.

Converting to a tif File

This one turns out to be a bit of a PITA in Java and particularly on a Mac, and represents the bulk of the Groovy code. Unfortunately it is also the only format that tesseract appears to accept ‘out of the box’. After googling the fun that is JAI and working with the .tif(f) format with it on a Mac, I ended up taking the code kindly provided in this blog post and Groovified it a bit to make a working transformation. Thanks very much to Allan Tan for that. One more time, there’s likely a better/easier way to do this, but honestly it’s more effort than I’m willing to put in on a weekend afternoon just to satisfy my curiosity.

//python
ext = ".tif"
big.save("input-NEAREST" + ext)

//Groovy
void convertToTiff(String inputFile, String outputFile)
{
    OutputStream ios
    try
    {
        ios = new BufferedOutputStream(new FileOutputStream(new File(outputFile)))
        ImageEncoder enc = ImageCodec.createImageEncoder("tiff", ios, new TIFFEncodeParam(compression: TIFFEncodeParam.COMPRESSION_NONE, littleEndian: false))
        RenderedOp src = JAI.create("fileload", inputFile)

        //Apply the color filter and return the result.
        ColorConvertOp filterObj = new ColorConvertOp(ColorSpace.getInstance(ColorSpace.CS_sRGB), null)
        BufferedImage dst = new BufferedImage(src.width, src.height, BufferedImage.TYPE_3BYTE_BGR)
        filterObj.filter(src.getAsBufferedImage(), dst)

        // save the output file
        enc.encode(dst)
    }
    catch (Exception e)
    {
        println e
    }
    finally
    {
        ios.close()
    }
}

OCR with Tesseract-OCR

Finally we need to pass the processed image to tesseract so it can ‘read’ it for us. Again, the python library makes this a breeze, but calling out to a command line program with Groovy is so simple that it ends up being about the same. Tesseract itself is available as a macport, as well in downloadable unix binaries and a windows executable so installing the software is a breeze.

//python
from pytesser import *
image = Image.open('input-NEAREST.tif')
print image_to_string(image)

//Groovy
def tesseract = ['/opt/local/bin/tesseract', tmpTif, tmpTesseract].execute()
tesseract.waitFor()
return new File("${tmpTesseract}.txt").readLines()[0]

Testing it out

To test it out I implemented the code in a maven project, iterate over the images and write out intermediate results to a temp directory. And it only works on two out of three of the cases. For some reason tesseract insists on consistently seeing ‘e4ya’ as ‘e4ga’. I tried to see if I could get it working by tweaking the image manipulation parameters and the order of operations(resizing before removing the background noise for instance) but that just caused the other cases to fail as well. Since in the final image the ‘y’ seems pretty clear, it’s more likely that tweaking tesseract configuration might yield better results.

public void testPrintImage()
{
    def breaker = new CaptchaBreaker()
    /* tesseract interprets "e4ya" as "e4ga" unfortunately */
    ['9koO', 'jxt9'/*,'e4ya'*/].each {String imageName ->
        def fileName = "src/test/resources/${imageName}.gif"
        assertEquals("Testing $imageName",imageName, breaker.imageToString(fileName))
    }
}

C’est Finis

I had some fun playing with areas of Java that I don’t usually interact with, and gained some appreciation for the diversity and ease-of-use exposed by just a couple of python libraries. It’s comforting to note that I was able to implement all of the required functionality from those libraries in < 90 lines of Groovy. With a little more effort I think the final product could be tweaked to avoid the intermediate file system reads/writes as well, but that’s for another day.
Source code is available on github if you’d care to take a look, and thanks for stopping by!

Developing Faster with the Atlassian IntelliJ Connector

TheKaptain — Sun, 21 Feb 2010 19:53:38 +0000

Once upon a time I used Eclipse as a development environment. It had a lot of things going for it: free(as in beer), rich community involvement, a plethora of plugins and probably my favorite feature: Mylyn. The problem was that it seemed everytime I wanted to upgrade to a newer version, inevitably half of the integrations broke. Please don’t get me wrong, Eclipse is AMAZING software and I do still use it occasionally for specific tasks – but nevertheless I now spend most of my day in IntelliJ. In particular it had better support for Groovy/Grails development and Maven integration – both of which were essential to my everyday work. Throw in default included support for JBoss Seam, JSF/Facelets, html and css and I didn’t really need a lot of plugins anymore. One of the ones I have been using, and that I’ve watched mature over the course of the last year, is the Atlassian IntelliJ Connector. Between it and the greatly improved changeset functionality I finally feel like I have a solid replacement for Mylyn’s excellent task management facilities.

This plugin integrates the IDE with one or more components of the Atlassian application suite. Multiple instances of Jira, Fisheye, Bamboo and Crucible can all be configured and used to streamline the development workflow.

Jira

This is perhaps the most essential piece of the puzzle, and inevitably the part a developer is going to interact with the most – the issue tracker. From within the IDE Jira master view you can load filters(basically stored searches for issues), do ad hoc searches and start work on a particular issue. If you drill down to a particular issue you can comment on it, assign it to yourself or another user, log work against it and generally manage it in most of the ways you can from the Jira web interface. Granted it’s not quite as pretty as the web interface, but the essential information and interaction is all there, and if you’re missing something an action is provided to open any issue in a web browser.

What really works for me is how the interaction supports my general workflow so closely:

Look in Jira for an issue to work on
Assign it to myself(if it’s not already)
Start progress on the issue, which starts a timer and creates a corresponding changeset
Do whatever development work that is required to satisfy the issue, pausing and resuming as necessary for
the duration of the task
Commit the changeset, optionally logging time against the issue and creating a Crucible review for later
Rinse and repeat

Granted, I’m not the best at remembering to pause the timer, but being confronted with the time when I commit forces me to honestly evaluate how much time it took to complete the task when it’s clearest in my mind. And incidentally, while we’re here, the option to automatically organize imports in the commit dialog, a built in IDE function, has saved my butt from maven dependency-analyze any number of times.

It’s also convenient that you can see all comments and attachments for issues – viewing a screenshot describing a UI issue is pretty much essential after all, don’t you think?

Bamboo

So once your code is committed, a build is kicked off on Bamboo. Hopefully all goes well, but if any build you’re listening to fails the IDE will give you a message to that effect. You also have access to changes, tests and any associated build logs. And whether or not your build does fail, stacktraces from the log are immediately available and clickable in the IDE. In addition, you can manually trigger builds and label or comment them.

Fisheye

Integration with Fisheye is bi-directional between the IDE and the Fisheye webview. Context menus are available on right clicks in the IDE that open a file in Fisheye. And in the Fisheye web app clicking an IntelliJ icon will open a file in the IDE. It should be noted that this feature only appears to be available with Fisheye 2. I know because I’ve been missing it in Jira Studio, which still uses the 1.6 version of Fisheye.

Crucible

This is the one integration I’ve used the least so far, primarily because I don’t have Crucible installed locally for testing and again it seems a lot of the power of the integration is only available with version 2. The documentation certainly seems to suggest more rich functionality than I’ve found available anyhow. Fisheye and Crucible are actually bundled together for installation, so the Jira Studio version appears also restricted to the 1.6 version – for the time being at least. Mostly the Crucible integration is convenient because it provides messaging when reviews are assigned or commented.

Documentation

To be perfectly honest, I didn’t even look for these until writing this blog post. Configuring and using the plugin is very straightforward, provided you’re familiar with using these Atlassian tools at least. Nevertheless, I did discover a few additional bonuses and as usual the docs are both complete and up to date.

Overall

Really the point of using this plugin is to significantly reduce context switching; as much as possible your work is concentrated in one interface, and for the vast majority of cases you only need one piece of software running to get the job done. Where context switching is inevitable, this software tries to make it as “one-click” as possible. The end result is to put the power of your Atlassian products front and center in the IDE, where us developer types spend most of our working lives. Now, if only Jira Studio gets updated to the latest available software versions, because I’m dying to try out side-by-side diffs for Crucible reviews in IntelliJ!

Anyhow, if you use Atlassian tools the Connector is well worth checking out. And you Eclipse users aren’t left in the dark either. I can’t vouch for it’s quality, but there is an equivalent plugin for Eclipse available as well.

[Show as slideshow]

[View with PicLens]

A One Day Griffon Application/Presentation

TheKaptain — Fri, 12 Feb 2010 05:15:57 +0000

I took the opportunity this past weekend to test drive the latest beta version of Griffon and along with it the as-of-yet unreleased slideware plugin. If you’re not already aware, Griffon is a Grails inspired framework for creating Java Swing applications. The project lead, Andres Almiray, has given several presentations using this plugin and it provides an excellent platform for showcasing both the power of Swing and the capabilities of Griffon to make it all look so easy.

The slideware plugin provides a framework for creating presentations with a little twist – you can execute code live from the presentation software. If you’ve ever given a presentation about programming you probably are fully aware of the transition from presentation software to your preferred environment for demonstrating code samples. Well, now you can stay entirely within the same application, editing and running code live.

The plugin itself isn’t available from the Griffon repository but source code and a built 0.2 version can be found on github. Considering both the youth of the plugin and the beta status of the framework, it worked impressively well AND had a rich feature set. All of the expected “powerpoint” features are there: themes, layout control, styling, slide transitions and export are all pretty easy to incorporate and configure. The code editor view also works very well. A great variety of additional plugins are harnessed to put all the pieces together and as a result building one of these applications is a great way to tour the platform.

Theming

Applying a theme to the presentation is simply selecting a Java Look and Feel to apply in Initialize.groovy. The Substance jar is included with the plugin so I test drove a few of the nice setups in there and finally settled on the SubstanceMagmaLookAndFeel. There is definitely a wide variety of L&F’s to choose from in that bundle alone and, although I haven’t done it myself, they seem pretty tweakable as well. Plus any old L&F should plug in nicely, I would imagine.

//Initialize.groovy
SwingBuilder.lookAndFeel('org.jvnet.substance.skin.SubstanceMagmaLookAndFeel',
       'mac', 'nimbus', 'gtk', ['metal', [boldFonts: false]])

Layout

Controlling the page composition is a standard Swing Layout or, in the case of the default slide you get with the included “create-slide” script, a MigLayout. Framing the standard variety of slides is very simple. Bulleted pages, title slides, code slides and custom layouts are very easy to accomplish. I don’t have a lot of experience using this particular layout but the presentations Andres has made available on github have a good diversity of examples of how they look in practice.

//the default create-slide generated template
import net.miginfocom.swing.MigLayout

slide(id: "slide0", layout: new MigLayout("fill","[center]","[center]")) {
    label("Insert your text here")
}

Styling

Styling is supplied by the css plugin, on which slideware has a dependency. The default style.css file sets out just some reasonable defaults for the fonts used in different parts of the app, and I didn’t see any real need to fiddle with it. Especially happy to see the nice monospace code font. On a totally related note I recently installed the Inconsolata monospace font to try for development and I’ve been very happy seeing it in my editor, but it’s still nice to see this kind of polish applied to the presentation. The code editor view even includes syntax highlighting! More on that coming right up…

Slide Transitions

Moving between slides with style is the responsibility of the transitions plugin. You can see see all of the animations this plugin enables over here at this page describing Transitions and Transition2Ds. Pretty slick stuff and defined as simply as a parameter to each “slide” node in a script.

slide(id: "slide3", layout: new MigLayout("fill","3%[center]3%","3%[center]3%"),
        title: "Junit3",
        transition: new FlurryTransition2D(Transition2D.OUT)) {
    scrollPane(constraints: "grow") {
        widget(createEditor(text: script))
    }
}

Code Editor

Code slides embed an editable widget and allow for composing and executing Groovy scripts or classes of arbitrary complexity. You can execute the code with a keyboard shortcut and a window will open displaying the console output. The code itself is executed through a GroovyShell, enabling pretty much anything you might want to do. If you have an internet connection and the required repositories configured, Grapes simplifies packaging the dependencies for the application as your code samples can directly Grab the jars they need. Basically, the first time you execute a code slide you’ll have to put up with a small pause while Ivy downloads to your local repository, unless of course the required dependencies are already there. In my case I changed strategies from including jars in the application lib directory to a Grapes approach and I think it’s a better way to go.

I was surprised to find that the editor even included undo functionality. It’s definitely not close to a full blown IDE, and there’s absolutely no reason that it should be. For the task of demoing simple code examples it’s more than up to the task, even to the point of maintaining your edits between slide transitions, allowing you to move back and forth through the slide deck without any state problems.

Export

The application includes a “Print” feature which iterates through the entire slide deck and renders it to a pdf. Distribution of one of these presentations is really very easy, including the ability to create installers for all major platforms simply by adding the packaging plugin.

Test Application

The test app I built while looking into this is pretty simple. It’s got a title slide, a bullet slide and 4 code slides. The code simply demonstrates how you can create test classes and make assertions for Junit 3(with GroovyTestCase), Junit 4 and TestNG. GroovyShell recognizes all 3 of these test files by interface or annotation and executes them appropriately. In each case the console output of the test framework is the result, including the new Spock inspired ascii art assert failure renderings. The Test result files are also written to disk, and that TestNG html output is what I’m used to looking at anyhow. Show me the Green!

The last code slide is just a slightly updated version of an example script on the Grapes page, using the current version of Google Collections and intentionally introducing a failure- mostly just to show off that new assert rendering I mentioned a moment ago. VERY helpful at highlighting the exact nature of a failure. It also encourages me to pay more attention to how I name variables, something I’m sure every developer that has ever worked with me will cheer at. Man, I suck at naming things.

I also developed a brute force test that loads each slide and executes scripts if finds embedded there. Failures are hard to detect since the direct output is simply a text block, but some fairly simple regex’s applied to the output make me at least moderately confident that the code won’t fail at show time.

Overall Impression

It took a couple of afternoons(six hours or so total) to download the source code from github, explore it, create a simple presentation and document the experience. I won’t begin to suggest that I’m fully aware of all the details happening behind the scenes, but the end user experience is pretty fluid: create a slide, tailor the layout, add content and then repeat. The included examples were more than enough documentation on how to hit the ground running. The plugin code itself is a great example of the MVC nature of Griffon, not a whole lot of code, but a great deal of power and expressability. There were a couple of glitches happening in the background, mostly just logging to the console with no visible effect to the application, but overall it functioned as well as (not) advertised. For publicly unreleased software it was an absolute pleasure to work with and I plan on continuing with the development of this particular presentation.

Everything you need to build and run this stuff yourself is publicly available. In my case, Griffon generally has a recent Macport available for both the released(griffon @0.2.1) and development versions(griffon-devel @0.3-BETA-2). Switching versions is relatively painless and, for applications this simple, testing out upgrades is basically just going through the presentation once in a functional test. Versions for other platforms can be downloaded from the Griffon download page.

Deliverables

Source code for the sample application is available on github here. Please just leave a comment on this page if you have any problems running it.

Here’s the pdf produced by the application ‘Print’ feature: Groovy Testing Presentation (252)

And if you don’t feel like downloading anything, here’s how it all looks in pretty pictures.

[Show as slideshow]

[View with PicLens]

Thanks for the shirt Atlassian!

TheKaptain — Thu, 28 Jan 2010 04:12:43 +0000

So my t-shirt arrived today in the mail compliments of Atlassian. Bright green and well earned when I slayed the dragon, I’ll wear it with pride. So far I’ve been getting the most mileage out of Bamboo and Jira, but I’ve also been tweaking Confluence to make it just right. Following right along with Atlassian’s lead, both TaskDock and Gliffy are offering $10 donation licenses for less than ten user systems, so I’ve pretty much got to at least try them out right?

Actually I’ve used Gliffy in Confluence before, and it’s come in handy more than once or twice. But between Balsamiq and OmniGraffle I’m already well equipped to cover up the fact that my artistic abilities lie somewhere below drawing stick men. TaskDock has however been installed and set up to nag me via email when I am getting behind schedule. Next will be Customware’s scaffolding and reporting plugins. The scaffolding capabilities make dealing with tables in Confluence a breeze – almost(but not quite) enough to make me abandon the wiki editor completely. Why take my word for it when you can just watch the video?

Anyhow, so far I’m very happy with the software and how it’s already helped me organize projects at home better. The t-shirt is just a nice bonus.