and turn it into a series of graphs that look like this:

Recently I was looking for an opportunity to get some practice with JFreeChart and ended up looking at a dataset released by the Canadian government as part of their ‘Open Data’ initiative.

The particular set of data is entitled ‘Number of Seedlings Planted by Ownership, Species’ and is delivered as an Excel spreadsheet, hence the need for the Apache POI library in order to read the data in. As is fairly usual, at least in my experience, the Excel spreadsheet is designed primarily for human consumption which adds a degree of complexity to the parsing. Fortunately the spreadsheet does follow a repetitive pattern that can be accounted for fairly easily, so this is not insurmountable. Still, we want to get the data out of Excel to make it more approachable for machine consumption so the first step is to convert it to a JSON representation. Once it is in this much more transportable form we can readily convert the data into graph visualizations using JFreeChart.

The spreadsheet format

Excel as a workplace tool is very well established, can increase individual productivity and is definitely a boon to your average office worker. The problem is that once the data is there it’s often trapped there. Data tends to be laid out based on human aesthetics and not on parsability, meaning that unless you want to use Excel itself to do further analysis, there’s not a lot of options. Exports to more neutral formats like csv suffer from the same problems- namely that there’s no way to read in the data coherently without designing a custom parser. In this particular case, parsing the spreadsheet has to take into account the following:

• Merged cells where one column is meant to represent a fixed value for a number of sequential rows.
• Column headers that do not represent all of the actual columns. Here we have a ‘notes’ column for each province that immediately follows its’ data column. As the header cells are merged across both of these columns, they cannot be used directly to parse the data.
• Data is broken down into several domains that lead to repetitions in the format.
• The data contains a mix of numbers where results are available and text where they are not. The meanings of the text entries are described in a table at the end of the spreadsheet.
• Section titles and headers are repeated throughout the document, apparently trying to match some print layout, or perhaps just trying to provide some assistance to those scrolling through the long document.

Data in the spreadsheet is first divided into reporting by Provincial crown land, private land, Federal land, and finally a total for all of them.

Within each of these sections, data is reported for each tree species on a yearly basis across all Provinces and Territories along with aggregate totals of these figures across Canada.

Each of these species data-tables has an identical row/column structure which allows us to create a single parsing structure sufficient for reading in data from each of them separately.

Converting the spreadsheet to JSON

For parsing the Excel document, I’m using the Apache POI library and a Groovy wrapper class to assist in processing. The wrapper class is very simple but allows us to abstract most of the mechanics of dealing with the Excel document away. The full source is available on this blog post from author Goran Ehrsson. The key benefit is the ability to specify a window of the file to process based on ‘offset’ and ‘max’ parameters provided in a simple map. Here’s an example for reading data for the text symbols table at the end of the spreadsheet.

We define a Map which states which sheet to read from, which line to start on(offset) and how many lines to process. The ExcelBuilder class(which isn’t really a builder at all) takes in the path to a File object and under the hood reads that into a POI HSSFWorkbook which is then referenced by the call to the eachLine method.

public static final Map SYMBOLS = [sheet: SHEET1, offset: 910, max: 8]
...
    final ExcelBuilder excelReader = new ExcelBuilder(data.absolutePath)
    Map<String, String> symbolTable = [:]
    excelReader.eachLine(SYMBOLS) { HSSFRow row ->
        symbolTable[row.getCell(0).stringCellValue] = row.getCell(1).stringCellValue
    }

Eventually when we turn this into JSON, it will look like this:

    "Symbols": {
        "...": "Figures not appropriate or not applicable",
        "..": "Figures not available",
        "--": "Amount too small to be expressed",
        "-": "Nil or zero",
        "p": "Preliminary figures",
        "r": "Revised figures",
        "e": "Estimated by provincial or territorial forestry agency",
        "E": "Estimated by the Canadian Forest Service or by Statistics Canada"
    }

Now processing the other data blocks gets a little bit trickier. The first column consists of 2 merged cells, and all but one of the other headers actually represents two columns of information: a count and an optional notation. The merged column is handled by a simple EMPTY placeholder and the extra columns by processing the list of headers;.

public static final List<String> HEADERS = ['Species', 'EMPTY', 'Year', 'NL', 'PE', 'NS', 'NB', 'QC', 'ON', 'MB', 'SK', 'AB',
    'BC', 'YT', 'NT *a', 'NU', 'CA']
/**
* For each header add a second following header for a 'notes' column
* @param strings
* @return expanded list of headers
*/
private List<String> expandHeaders(List<String> strings)
{
    strings.collect {[it, "${it}_notes"]}.flatten()
}

Each data block corresponds to a particular species of tree, broken down by year and Province or Territory. Each species is represented by a map which defines where in the document that information is contained so we can iterate over a collection of these maps and aggregate data quite easily. This set of constants and code is sufficient for parsing all of the data in the document.

        public static final int HEADER_OFFSET = 3
        public static final int YEARS = 21
        public static final Map PINE = [sheet: SHEET1, offset: 6, max: YEARS, species: 'Pine']
        public static final Map SPRUCE = [sheet: SHEET1, offset: 29, max: YEARS, species: 'Spruce']
        public static final Map FIR = [sheet: SHEET1, offset: 61, max: YEARS, species: 'Fir']
        public static final Map DOUGLAS_FIR = [sheet: SHEET1, offset: 84, max: YEARS, species: 'Douglas-fir']
        public static final Map MISCELLANEOUS_SOFTWOODS = [sheet: SHEET1, offset: 116, max: YEARS, species: 'Miscellaneous softwoods']
        public static final Map MISCELLANEOUS_HARDWOODS = [sheet: SHEET1, offset: 139, max: YEARS, species: 'Miscellaneous hardwoods']
        public static final Map UNSPECIFIED = [sheet: SHEET1, offset: 171, max: YEARS, species: 'Unspecified']
        public static final Map TOTAL_PLANTING = [sheet: SHEET1, offset: 194, max: YEARS, species: 'Total planting']
        public static final List<Map> PROVINCIAL = [PINE, SPRUCE, FIR, DOUGLAS_FIR, MISCELLANEOUS_SOFTWOODS, MISCELLANEOUS_HARDWOODS, UNSPECIFIED, TOTAL_PLANTING]
        public static final List<String> AREAS = HEADERS[HEADER_OFFSET..-1]

        ...

        final Closure collector = { Map species ->
            Map speciesMap = [name: species.species]
            excelReader.eachLine(species) {HSSFRow row ->
                //ensure that we are reading from the correct place in the file
                if (row.rowNum == species.offset)
                {
                    assert row.getCell(0).stringCellValue == species.species
                }
                //process rows
                if (row.rowNum > species.offset)
                {
                    final int year = row.getCell(HEADERS.indexOf('Year')).stringCellValue as int
                    Map yearMap = [:]
                    expandHeaders(AREAS).eachWithIndex {String header, int index ->
                        final HSSFCell cell = row.getCell(index + HEADER_OFFSET)
                        yearMap[header] = cell.cellType == HSSFCell.CELL_TYPE_STRING ? cell.stringCellValue : cell.numericCellValue
                    }
                    speciesMap[year] = yearMap.asImmutable()
                }
            }
            speciesMap.asImmutable()
        }

The defined collector Closure returns a map of all species data for one of the four groupings(Provincial, private land, Federal and totals). The only thing that differentiates these groups is their offset in the file so we can define maps for the structure of each simply by updating the offsets of the first.

    public static final List<Map> PROVINCIAL = [PINE, SPRUCE, FIR, DOUGLAS_FIR, MISCELLANEOUS_SOFTWOODS, MISCELLANEOUS_HARDWOODS, UNSPECIFIED, TOTAL_PLANTING]
    public static final List<Map> PRIVATE_LAND = offset(PROVINCIAL, 220)
    public static final List<Map> FEDERAL = offset(PROVINCIAL, 441)
    public static final List<Map> TOTAL = offset(PROVINCIAL, 662)

    private static List<Map> offset(List<Map> maps, int offset)
    {
        maps.collect { Map map ->
            Map offsetMap = new LinkedHashMap(map)
            offsetMap.offset = offsetMap.offset + offset
            offsetMap
        }
    }

Finally, we can iterate over these simple map structures applying the collector Closure and we end up with a single map representing all of the data.

        def parsedSpreadsheet = [PROVINCIAL, PRIVATE_LAND, FEDERAL, TOTAL].collect {
            it.collect(collector)
        }
        Map resultsMap = [:]
        GROUPINGS.eachWithIndex {String groupName, int index ->
            resultsMap[groupName] = parsedSpreadsheet[index]
        }
        resultsMap['Symbols'] = symbolTable

And the JsonBuilder class provides an easy way to convert any map to a JSON document ready to write out the results.

        Map map = new NaturalResourcesCanadaExcelParser().convertToMap(data)
        new File('src/test/resources/NaturalResourcesCanadaNewSeedlings.json').withWriter {Writer writer ->
            writer << new JsonBuilder(map).toPrettyString()
        }

Parsing JSON into JFreeChart line charts

All right, so now that we’ve turned the data into a slightly more consumable format, it’s time to visualize it. For this case I’m using a combination of the JFreeChart library and the GroovyChart project which provides a nice DSL syntax for working with the JFreeChart API. It doesn’t look to be under development presently, but aside from the fact that the jar isn’t published to an available repository it was totally up to this task.

We’re going to create four charts for each of the fourteen areas represented for a total of 56 graphs overall. All of these graphs contain plotlines for each of the eight tree species tracked. This means that overall we need to create 448 distinct time series. I didn’t do any formal timings of how long this takes, but in general it came in somewhere under ten seconds to generate all of these. Just for fun, I added GPars to the mix to parallelize creation of the charts, but since writing the images to disk is going to be the most expensive part of this process, I don’t imagine it’s speeding things up terribly much.

First, reading in the JSON data from a file is simple with JsonSlurper.

        def data
        new File(jsonFilename).withReader {Reader reader ->
            data = new JsonSlurper().parse(reader)
        }
        assert data

Here’s a sample of what the JSON data looks like for one species over a single year, broken down first by one of the four major groups, then by tree species, then by year and finally by Province or Territory.

{
    "Provincial": [
        {
            "name": "Pine",
            "1990": {
                "NL": 583.0,
                "NL_notes": "",
                "PE": 52.0,
                "PE_notes": "",
                "NS": 4.0,
                "NS_notes": "",
                "NB": 4715.0,
                "NB_notes": "",
                "QC": 33422.0,
                "QC_notes": "",
                "ON": 51062.0,
                "ON_notes": "",
                "MB": 2985.0,
                "MB_notes": "",
                "SK": 4671.0,
                "SK_notes": "",
                "AB": 8130.0,
                "AB_notes": "",
                "BC": 89167.0,
                "BC_notes": "e",
                "YT": "-",
                "YT_notes": "",
                "NT *a": 15.0,
                "NT *a_notes": "",
                "NU": "..",
                "NU_notes": "",
                "CA": 194806.0,
                "CA_notes": "e"
            },
    ...

Building the charts is a simple matter of iterating over the resulting map of parsed data. In this case we're ignoring the 'notes' data but have included it in the dataset in case we want to use it later. We're also just ignoring any non-numeric values.

GROUPINGS.each { group ->
            withPool {
                AREAS.eachParallel { area ->
                    ChartBuilder builder = new ChartBuilder();
                    String title = sanitizeName("$group-$area")
                    TimeseriesChart chart = builder.timeserieschart(title: group,
                            timeAxisLabel: 'Year',
                            valueAxisLabel: 'Number of Seedlings(1000s)',
                            legend: true,
                            tooltips: false,
                            urls: false
                    ) {
                        timeSeriesCollection {
                            data."$group".each { species ->
                                Set years = (species.keySet() - 'name').collect {it as int}
                                timeSeries(name: species.name, timePeriodClass: 'org.jfree.data.time.Year') {
                                    years.sort().each { year ->
                                        final value = species."$year"."$area"
                                        //check that it's a numeric value
                                        if (!(value instanceof String))
                                        {
                                            add(period: new Year(year), value: value)
                                        }
                                    }
                                }
                            }
                        }
                    }
...
}

Then we apply some additional formatting to the JFreeChart to enhance the output styling, insert an image into the background, and fix the plot color schemes.

                    JFreeChart innerChart = chart.chart
                    String longName = PROVINCE_SHORT_FORM_MAPPINGS.find {it.value == area}.key
                    innerChart.addSubtitle(new TextTitle(longName))
                    innerChart.setBackgroundPaint(Color.white)
                    innerChart.plot.setBackgroundPaint(Color.lightGray.brighter())
                    innerChart.plot.setBackgroundImageAlignment(Align.TOP_RIGHT)
                    innerChart.plot.setBackgroundImage(logo)
                    [Color.BLUE, Color.GREEN, Color.ORANGE, Color.CYAN, Color.MAGENTA, Color.BLACK, Color.PINK, Color.RED].eachWithIndex { color, int index ->
                        innerChart.XYPlot.renderer.setSeriesPaint(index, color)
                    }

And we write out each of the charts to a formulaically named png file.

                    def fileTitle = "$FILE_PREFIX-${title}.png"
                    File outputDir = new File(outputDirectory)
                    if (!outputDir.exists())
                    {
                        outputDir.mkdirs()
                    }
                    File file = new File(outputDir, fileTitle)
                    if (file.exists())
                    {
                        file.delete()
                    }
                    ChartUtilities.saveChartAsPNG(file, innerChart, 550, 300)

To tie it all together, an html page is created using MarkupBuilder to showcase all of the results, organized by Province or Territory.

    def buildHtml(inputDirectory)
    {
        File inputDir = new File(inputDirectory)
        assert inputDir.exists()
        Writer writer = new StringWriter()
        MarkupBuilder builder = new MarkupBuilder(writer)
        builder.html {
            head {
                title('Number of Seedlings Planted by Ownership, Species')
                style(type: "text/css") {
                    mkp.yield(CSS)
                }
            }
            body {
                ul {
                    AREAS.each { area ->
                        String areaName = sanitizeName(area)
                        div(class: 'area rounded-corners', id: areaName) {
                            h2(PROVINCE_SHORT_FORM_MAPPINGS.find {it.value == area}.key)
                            inputDir.eachFileMatch(~/.*$areaName\.png/) {
                                img(src: it.name)
                            }
                        }
                    }
                }
                script(type: 'text/javascript', src: 'https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js', '')
                script(type: 'text/javascript') {
                    mkp.yield(JQUERY_FUNCTION)
                }
            }
        }
        writer.toString()
    }

The generated html page assumes that all images are co-located in the same folder, presents four images per Province/Territory and, just for fun, uses JQuery to attach a click handler to each of the headers. Click on a header and the images in that div will animate into the background. I’m sure the actual JQuery being used could be improved upon, but it serves its purpose. Here’s a sample of the html output:

<ul>
      <div class='area rounded-corners' id='NL'>
        <h2>Newfoundland and Labrador</h2>
        <img src='naturalResourcesCanadaNewSeedlings-Federal-NL.png' />
        <img src='naturalResourcesCanadaNewSeedlings-PrivateLand-NL.png' />
        <img src='naturalResourcesCanadaNewSeedlings-Provincial-NL.png' />
        <img src='naturalResourcesCanadaNewSeedlings-Total-NL.png' />
      </div>
    ...

The resulting page looks like this in Firefox.

Source code and Links

The source code is available on GitHub. So is the final resulting html page. The entire source required to go from Excel to charts embedded in an html page comes in at slightly under 300 lines of code and I don’t think the results are too bad for the couple of hours effort involved. Finally, the JSON results are also hosted on the GitHub pages for the project for anyone else who might want to delve into the data.

Some reading related to this topic:

• Groovy loves POI and POI loves Groovy
• Writing batch import scripts with Grails, GSQL and GPars
• GSheets – A Groovy Builder based on Apache POI
• Groovy for the Office

[Show as slideshow]

[View with PicLens]

Related posts:

1. Groovy inspect()/Eval for Externalizing Data
2. Groovy reverse map sort done easy
3. Five Cool Things You Can Do With Groovy Scripts

The project structure

The code is hosted on Github, and provides a Gradle build which downloads and launches a Jenkins(or Hudson) server locally to execute tests. The server is set to use the Gradle build directory as its working directory, so it can be deleted simply by executing gradle clean. I tried it using both the Jenkins and the Hudson versions of the required libraries and, aside from some quirks between the two CLI implementations, they continue to function very much the same. If you want to try it with Hudson instead of Jenkins, pass in the command flag -Pswitch and the appropriate war and libraries will be used. The project is meant to be run with Gradle 1.0-milestone-8, and comes with a Gradle wrapper for that version. Most of the code remains the same since the original article, but there are some enhancements and changes to deal with the newer versions of Jenkins and Hudson.
The library produced by this project is published as a Maven artifact, and later on I’ll describe exactly how to get at it. There are also some samples included that demonstrate using that library in Gradle or Maven projects, and in Groovy scripts with Grapes. We’re using Groovy 1.8.6, Gradle 1.0-milestone-8 and Maven 3.0.3 to build everything.

Getting more out of the CLI

As an alternative to the api, the CLI jar is a very capable way of interacting with the build server. In addition to a variety of built-in commands, Groovy scripts can be executed remotely, and with a little effort we can easily serialize responses in order to work with data extracted on the server. As an execution environment, the server provides a Groovysh shell and stocks it with imports for the hudson.model package. Also passed into the Binding is the instance of the Jenkins/Hudson singleton object in that package. In these examples I’m using the backwards-compatible Hudson version, since the code is intended to be runnable on either flavor of the server.

The available commands

There’s a rich variety of built-in commands, all of which are implemented in the hudson.cli package. Here are the ones that are listed on the CLI page of the running application:

• build: Builds a job, and optionally waits until its completion.
• cancel-quiet-down: Cancel the effect of the “quiet-down” command.
• clear-queue: Clears the build queue
• connect-node: Reconnect to a node
• copy-job: Copies a job.
• create-job: Creates a new job by reading stdin as a configuration XML file.
• delete-builds: Deletes build record(s).
• delete-job: Deletes a job
• delete-node: Deletes a node
• disable-job: Disables a job
• disconnect-node: Disconnects from a node
• enable-job: Enables a job
• get-job: Dumps the job definition XML to stdout
• groovy: Executes the specified Groovy script.
• groovysh: Runs an interactive groovy shell.
• help: Lists all the available commands.
• install-plugin: Installs a plugin either from a file, an URL, or from update center.
• install-tool: Performs automatic tool installation, and print its location to stdout. Can be only called from
  inside a build.
• keep-build: Mark the build to keep the build forever.
• list-changes: Dumps the changelog for the specified build(s).
• login: Saves the current credential to allow future commands to run without explicit credential information.
• logout: Deletes the credential stored with the login command.
• mail: Reads stdin and sends that out as an e-mail.
• offline-node: Stop using a node for performing builds temporarily, until the next “online-node” command.
• online-node: Resume using a node for performing builds, to cancel out the earlier “offline-node” command.
• quiet-down: Quiet down Jenkins, in preparation for a restart. Don’t start any builds.
• reload-configuration: Discard all the loaded data in memory and reload everything from file system. Useful when
  you modified config files directly on disk.
• restart: Restart Jenkins
• safe-restart: Safely restart Jenkins
• safe-shutdown: Puts Jenkins into the quiet mode, wait for existing builds to be completed, and then shut down
  Jenkins.
• set-build-description: Sets the description of a build.
• set-build-display-name: Sets the displayName of a build
• set-build-result: Sets the result of the current build. Works only if invoked from within a build.
• shutdown: Immediately shuts down Jenkins server
• update-job: Updates the job definition XML from stdin. The opposite of the get-job command
• version: Outputs the current version.
• wait-node-offline: Wait for a node to become offline
• wait-node-online: Wait for a node to become online
• who-am-i: Reports your credential and permissions

It’s not immediately apparent what arguments are required for each, but they almost universally follow a CLI pattern of printing usage details when called with no arguments. For instance, when you call the build command with no arguments, here’s what you get back in the error stream:

Argument “JOB” is required
java -jar jenkins-cli.jar build args…
Starts a build, and optionally waits for a completion.
Aside from general scripting use, this command can be
used to invoke another job from within a build of one job.
With the -s option, this command changes the exit code based on
the outcome of the build (exit code 0 indicates a success.)
With the -c option, a build will only run if there has been
an SCM change
JOB : Name of the job to build
-c : Check for SCM changes before starting the build, and if there’s no
change, exit without doing a build
-p : Specify the build parameters in the key=value format.
-s : Wait until the completion/abortion of the command

Getting data out of the system

All of the interaction with the remote system is handled by streams and it’s pretty easy to craft scripts that will return data in an easily parseable String format using built-in Groovy facilities. In theory, you should be able to marshal more complex objects as well, but let’s keep it simple for now. Here’s a Groovy script that just extracts all of the job names into a List, calling the Groovy inspect method to quote all values.

@GrabResolver(name = 'glassfish', root = 'http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name = "github", root = "http://kellyrob99.github.com/Jenkins-api-tour/repository")
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')
import org.kar.hudson.api.cli.HudsonCliApi

String rootUrl = 'http://localhost:8080'
HudsonCliApi cliApi = new HudsonCliApi()
OutputStream out = new ByteArrayOutputStream()
cliApi.runCliCommand(rootUrl, ['groovysh', 'hudson.jobNames.inspect()'], System.in, out, System.err)
List allJobs = Eval.me(cliApi.parseResponse(out.toString()))
println allJobs

Some useful examples

Finding plugins with updates and and updating all of them

Here’s an example of using a Groovy script to find all of the plugins that have updates available, returning that result to the caller, and then calling the CLI ‘install-plugin’ command on all of them. Conveniently, this command will either install a plugin if it’s not already there or update it to the latest version if already installed.

def findPluginsWithUpdates = '''
Hudson.instance.pluginManager.plugins.inject([]) { List toUpdate, plugin ->
    if(plugin.hasUpdate())
    {
        toUpdate << plugin.shortName
    }
    toUpdate
}.inspect()
'''
OutputStream updateablePlugins = new ByteArrayOutputStream()
cliApi.runCliCommand(rootUrl, ['groovysh', findPluginsWithUpdates], System.in, updateablePlugins, System.err)

def listOfPlugins = Eval.me(parseOutput(updateablePlugins.toString()))
listOfPlugins.each{ plugin ->
    cliApi.runCliCommand(rootUrl, ['install-plugin', plugin])
}

Install or upgrade a suite of Plugins all at once

This definitely beats using the ‘Manage Plugins’ UI and is idempotent so running it more than once can only result in possibly upgrading already installed Plugins. This set of plugins might be overkill, but these are some plugins I recently surveyed for possible use.

@GrabResolver(name='glassfish', root='http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name="github", root="http://kellyrob99.github.com/Jenkins-api-tour/repository")
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')
import static java.net.HttpURLConnection.*
import org.kar.hudson.api.*
import org.kar.hudson.api.cli.HudsonCliApi

String rootUrl = 'http://localhost:8080'
HudsonCliApi cliApi = new HudsonCliApi()

['groovy', 'gradle', 'chucknorris', 'greenballs', 'github', 'analysis-core', 'analysis-collector', 'cobertura',
        'project-stats-plugin','audit-trail', 'view-job-filters', 'disk-usage', 'global-build-stats',
        'radiatorviewplugin', 'violations', 'build-pipeline-plugin', 'monitoring', 'dashboard-view',
        'iphoneview', 'jenkinswalldisplay'].each{ plugin ->
    cliApi.runCliCommand(rootUrl, ['install-plugin', plugin])
}

//  Restart a node, required for newly installed plugins to be made available.
cliApi.runCliCommand(rootUrl, 'safe-restart')

Finding all failed builds and triggering them

It’s not all that uncommon that a network problem or infrastructure event can cause a host of builds to fail all at once. Once the problem is solved this script can be useful for verifying that the builds are all in working order.

@GrabResolver(name = 'glassfish', root = 'http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name = "github", root = "http://kellyrob99.github.com/Jenkins-api-tour/repository")
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')
import org.kar.hudson.api.cli.HudsonCliApi

String rootUrl = 'http://localhost:8080'
HudsonCliApi cliApi = new HudsonCliApi()
OutputStream out = new ByteArrayOutputStream()
def script = '''hudson.items.findAll{ job ->
    job.isBuildable() && job.lastBuild && job.lastBuild.result == Result.FAILURE
}.collect{it.name}.inspect()
'''
cliApi.runCliCommand(rootUrl, ['groovysh', script], System.in, out, System.err)
List failedJobs = Eval.me(cliApi.parseResponse(out.toString()))
failedJobs.each{ job ->
    cliApi.runCliCommand(rootUrl, ['build', job])
}

Open an interactive Groovy shell

If you really want to poke at the server you can launch an interactive shell to inspect state and execute commands. The System.in stream is bound and responses from the server are immediately echoed back.

@GrabResolver(name = 'glassfish', root = 'http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name = "github", root = "http://kellyrob99.github.com/Jenkins-api-tour/repository")
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')
import org.kar.hudson.api.cli.HudsonCliApi
/**
 * Open an interactive Groovy shell that imports the hudson.model.* classes and exposes
 * a 'hudson' and/or 'jenkins' object in the Binding which is an instance of hudson.model.Hudson
 */
HudsonCliApi cliApi = new HudsonCliApi()
String rootUrl = args ? args[0] :'http://localhost:8080'
cliApi.runCliCommand(rootUrl, 'groovysh')

Updates to the project

A lot has happened in the last year and all of the project dependencies needed an update. In particular, there have been some very nice improvements to Groovy, Gradle and Spock. Most notably, Gradle has come a VERY long way since version 0.9.2. The JSON support added in Groovy 1.8 comes in handy as well. Spock required a small tweak for rendering dynamic content in test reports when using @Unroll, but that’s a small price to pay for features like the ‘old’ method and Chained Stubbing. Essentially, in response to changes in Groovy 1.8+, a Spock @Unroll annotation needs to change from:

@Unroll("querying of #rootUrl should match #xmlResponse")

to a Closure encapsulated GString expression:

@Unroll({"querying of $rootUrl should match $xmlResponse"})

It sounds like the syntax is still in flux and I’m glad I found this discussion of the problem online.

Hosting a Maven repository on Github

Perhaps you noticed from the previous script examples, we’re referencing a published library to get at the HudsonCliApi class. I read an interesting article last week which describes how to use the built-in Github Pages for publishing a Maven repository. While this isn’t nearly as capable as a repository like Nexus or Artifactory, it’s totally sufficient for making some binaries available to most common build tools in a standard fashion. Simply publish the binaries along with associated poms in the standard Maven repo layout and you’re off to the races! Each dependency management system has its quirks(I’m looking at you Ivy!) but they’re pretty easy to work around, so here’s examples for Gradle, Maven and Groovy Grapes to use the library produced by this project code. Note that some of the required dependencies for Jenkins/Hudson aren’t available in the Maven central repository, so we’re getting them from the Glassfish repo.

Gradle

Pretty straight forward, this works with the latest version of Gradle and assumes that you are using the Groovy plugin.

repositories {
    mavenCentral()
    maven {
        url 'http://maven.glassfish.org/content/groups/public/'
    }
    maven {
        url 'http://kellyrob99.github.com/Jenkins-api-tour/repository'
    }
}
dependencies {
    groovy "org.codehaus.groovy:groovy-all:${versions.groovy}"
    compile 'org.kar:hudson-api:0.2-SNAPSHOT'
}

Maven

Essentially the same content in xml and in this case it’s assumed that you’re using the GMaven plugin

<repositories>
    <repository>
        <id>glassfish</id>
        <name>glassfish</name>
        <url>http://maven.glassfish.org/content/groups/public/</url>
    </repository>
    <repository>
        <id>github</id>
        <name>Jenkins-api-tour maven repo on github</name>
        <url>http://kellyrob99.github.com/Jenkins-api-tour/repository</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.codehaus.groovy</groupId>
        <artifactId>groovy-all</artifactId>
        <version>${groovy.version}</version>
    </dependency>
    <dependency>
        <groupId>org.kar</groupId>
        <artifactId>hudson-api</artifactId>
        <version>0.2-SNAPSHOT</version>
    </dependency>
</dependencies>

Grapes

In this case there seems to be a problem resolving some transitive dependency for an older version of Groovy which is why there’s an explicit exclude for it.

@GrabResolver(name='glassfish', root='http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name="github", root="http://kellyrob99.github.com/Jenkins-api-tour/repository")
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')

Links

The Github Jenkins-api-tour project page
Maven repositories on Github
Scriptler example Groovy scripts
Jenkins CLI documentation

Related posts:

1. Hooking into the Jenkins(Hudson) API
2. Five Cool Things You Can Do With Groovy Scripts
3. A Grails App Demoing the StackExchange API

I manage a large number of builds at work, spread across several build servers. When we release a new version all of the builds need to be updated to point to new working branches. This script takes advantage of the fact that our branches all end in the version number to quickly check that all of the last builds were on the expected version.
Using the Jenkins API is very easy and the new capabilities of Groovy introduced by JSONSlurper make it easier than ever to consume.

import groovy.json.JsonSlurper

assert args && args.size() == 2
def urls = args[0].split(',')
def expectedVersion = args[1]
urls.each { url ->
    println "examining url $url"
    def json = new JsonSlurper().parseText(url.toURL().text)

    json.jobs.each {
        try
        {
            def job = new JsonSlurper().parseText("${it.url}/api/json?depth=1".toURL().text)
            if (!job.builds[0].changeSet.revisions[0].module.endsWith(expectedVersion))
            {
                println "$it.url fails!"
            }
            else
            {
                println "$it.url passes!"
            }
        }
        catch (e)
        {
            println "Exception thrown processing $it.url : $e"
        }
    }
}

2. Look up the artifacts from the last successful Jenkins build

This is very handy if you want to automate deployment of software from your build system and can replace a lot of error prone manual updating of scripts. In this particular example you can link up quickly with the latest, greatest build of JRuby from their public CI server.

import groovy.json.*
def HOSTNAME = 'http://ci.jruby.org'
def JOBNAME = 'jruby-dist'
def JOB_URL = "$HOSTNAME/job/$JOBNAME/lastSuccessfulBuild"
def text = "$JOB_URL/api/json".toURL().text
println JsonOutput.prettyPrint(text)
def json = new JsonSlurper().parseText(text)
json.artifacts.each{
    println it
}

[relativePath:dist/jruby-bin-1.7.0.dev.tar.gz, fileName:jruby-bin-1.7.0.dev.tar.gz, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.tar.gz.md5, fileName:jruby-bin-1.7.0.dev.tar.gz.md5, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.tar.gz.sha1, fileName:jruby-bin-1.7.0.dev.tar.gz.sha1, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.zip, fileName:jruby-bin-1.7.0.dev.zip, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.zip.md5, fileName:jruby-bin-1.7.0.dev.zip.md5, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.zip.sha1, fileName:jruby-bin-1.7.0.dev.zip.sha1, displayPath:null]
[relativePath:dist/jruby-complete-1.7.0.dev.jar, fileName:jruby-complete-1.7.0.dev.jar, displayPath:null]
[relativePath:dist/jruby-complete-1.7.0.dev.jar.md5, fileName:jruby-complete-1.7.0.dev.jar.md5, displayPath:null]
[relativePath:dist/jruby-complete-1.7.0.dev.jar.sha1, fileName:jruby-complete-1.7.0.dev.jar.sha1, displayPath:null]
[relativePath:dist/jruby-jars-1.7.0.dev.gem, fileName:jruby-jars-1.7.0.dev.gem, displayPath:null]
[relativePath:dist/jruby-jars-1.7.0.dev.gem.md5, fileName:jruby-jars-1.7.0.dev.gem.md5, displayPath:null]
[relativePath:dist/jruby-jars-1.7.0.dev.gem.sha1, fileName:jruby-jars-1.7.0.dev.gem.sha1, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.tar.gz, fileName:jruby-src-1.7.0.dev.tar.gz, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.tar.gz.md5, fileName:jruby-src-1.7.0.dev.tar.gz.md5, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.tar.gz.sha1, fileName:jruby-src-1.7.0.dev.tar.gz.sha1, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.zip, fileName:jruby-src-1.7.0.dev.zip, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.zip.md5, fileName:jruby-src-1.7.0.dev.zip.md5, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.zip.sha1, fileName:jruby-src-1.7.0.dev.zip.sha1, displayPath:null]

3. Read in a CSV file, filter it, and write out the result

Groovy Grapes and one of my favorite libraries, OpenCSV, make it trivial to script ‘one off’ solutions for removing work that would normally take place manually in Excel. For me it comes up most often with data import/export from systems I’m working on and the need to create particular test environments using those import/export mechanisms. For example, using a scripted solution like this you can quickly:

• Export users from one test system into a CSV file, along with local phone number contact data
• Replace the existing phone numbers with a mix of local and international numbers
• Import the new data into a different test system
• Verify that the system properly deals with international calling concerns

This example simply trims out all of the rows where the second column of input contains values less than 100. It’s also nice to note that OpenCSV is smart enough to let you ignore one or more ‘header rows’, a common attribute of CSV files.

@Grab(group = 'net.sf.opencsv', module = 'opencsv', version = '2.3')
import au.com.bytecode.opencsv.CSVReader
import au.com.bytecode.opencsv.CSVParser
import au.com.bytecode.opencsv.CSVWriter

def TEST_FILE_NAME = 'test.csv'
def TEST_OUTPUT_FILE_NAME = 'testOut.csv'

List<String[]> rows = new CSVReader(new FileReader(new File(TEST_FILE_NAME)), CSVParser.DEFAULT_SEPARATOR, CSVParser.DEFAULT_ESCAPE_CHARACTER, CSVParser.DEFAULT_QUOTE_CHARACTER, 1).readAll()
def rowsOver100 = rows.findAll {it[1].toInteger() > 100}

File output = new File(TEST_OUTPUT_FILE_NAME)
if (output.exists())
{
    output.delete()
}
output.createNewFile()
output.withWriter { writer ->
    new CSVWriter(writer).writeAll(rowsOver100)

}

//println rows

4. Download, install and run a stand-alone webserver

This example comes courtesy of a recent article by Andrew Glover about Gretty, a lightweight http server. I don’t have a concrete use-case for this right now, but it’s easy to note that in this case you can code a large part of the server behaviour directly in the script, unlike, for instance, if you were to use a script to stand up a Jetty server. This should let you publish a script once in a central location and stand up as many Gretty servers providing the declared services as you like. This particular example simply echos back ‘Hello’ plus whatever path you append to localhost:8080.

import org.mbte.gretty.httpserver.* 

@GrabResolver(name='gretty', 
  root='http://groovypp.artifactoryonline.com/groovypp/libs-releases-local')
@Grab('org.mbte.groovypp:gretty:0.4.279') 

GrettyServer server = [] 
server.groovy = [ 
    localAddress: new InetSocketAddress("localhost", 8080), 
    defaultHandler: { 
        response.redirect "/" 
    }, 
    "/:name": {
        get {
            response.text = "Hello ${request.parameters['name']}"
        } 
    } 
] 
server.start()

5. Remotely execute any of these scripts by url

As of Groovy 1.8.3/1.9-beta-4 you can refer to a Groovy script hosted at a remote url and execute it locally. If you work on different machines a lot(as I do) this can really help keep your toolbox of scripts handy, regardless of where you are. Since these scripts are publicly hosted on GitHub(thanks guys!), you can simply copy the url from the ‘view raw’ link at the bottom and go to your console to execute them like this:

$ groovy https://gist.github.com/raw/1430856/cea0186a862217015fb4fc2b63eb0ad3575c06fc/jenkinsBuildArtifacts.groovy
{
    "actions": [
        {
            "causes": [
                {
                    "shortDescription": "Started by timer"
                }
            ]
        },
        {
            "buildsByBranchName": {
                "origin/master": {
                    "buildNumber": 684,
                    "buildResult": null,
                    "revision": {
                        "SHA1": "3419fee4bb9436d3222ff3df8dd7fcf2308a6919",
                        "branch": [
                            {
                                "SHA1": "3419fee4bb9436d3222ff3df8dd7fcf2308a6919",
                                "name": "origin/master"
                            }
                        ]
                    }
                }
            },
... and so on

Final note

Being able to share scripts across a variety of machines has definitely made my professional life a lot easier and I’m very glad to see the ability to execute code from a URL in Groovy. But what really turned me onto the idea was its usage in Gradle, which supports loading remote Groovy/Gradle scripts using the “apply from: {url or file}” syntax. I find that more and more I reference particular tasks remotely in a build, especially for static analysis tools that don’t necessarily need to be run on a regular basis.
A great example of this is the recently released Gradle Templates Plugin, which is basically the Gradle answer to the Maven Archetype plugin. Create a build.gradle with the following content:

apply from: 'http://launchpad.net/gradle-templates/trunk/1.2/+download/apply.groovy'

And we have the following tasks available to create new Gradle projects and objects with conventional structure:

Template tasks
--------------
createGradlePlugin - Creates a new Gradle Plugin project in a new directory named after your project.
createGroovyClass - Creates a new Groovy class in the current project.
createGroovyProject - Creates a new Gradle Groovy project in a new directory named after your project.
createJavaClass - Creates a new Java class in the current project.
createJavaProject - Creates a new Gradle Java project in a new directory named after your project.
createScalaClass - Creates a new Scala class in the current project.
createScalaObject - Creates a new Scala object in the current project.
createScalaProject - Creates a new Gradle Scala project in a new directory named after your project.
createWebappProject - Creates a new Gradle Webapp project in a new directory named after your project.
exportAllTemplates - Exports all the default template files into the current directory.
exportGroovyTemplates - Exports the default groovy template files into the current directory.
exportJavaTemplates - Exports the default java template files into the current directory.
exportPluginTemplates - Exports the default plugin template files into the current directory.
exportScalaTemplates - Exports the default scala template files into the current directory.
exportWebappTemplates - Exports the default webapp template files into the current directory.
initGradlePlugin - Initializes a new Gradle Plugin project in the current directory.
initGroovyProject - Initializes a new Gradle Groovy project in the current directory.
initJavaProject - Initializes a new Gradle Java project in the current directory.
initScalaProject - Initializes a new Gradle Scala project in the current directory.
initWebappProject - Initializes a new Gradle Webapp project in the current directory.

Related posts:

1. Using Gradle to Bootstrap your Legacy Ant Builds
2. Hooking into the Jenkins(Hudson) API, Part 2
3. Hooking into the Jenkins(Hudson) API

Gradle AntBuilder

Every Gradle Project includes an AntBuilder instance, making any and all of the facilities of Ant available within your build files. Gradle provides a simple extension to the existing Groovy AntBuilder which adds a simple yet powerful way to interface with existing Ant build files: the importBuild(Object antBuildFile) method. Internally this method utilizes an Ant ProjectHelper to parse the specified Ant build file and then wraps all of the targets in Gradle tasks making them available in the Gradle build. The following is a simple Ant build file used for illustration which contains some properties and a couple of dependent targets.

<?xml version="1.0"?>
<project name="build" default="all">
    <echo>Building ${ant.file}</echo>

    <property file="build.properties"/>
    <property name="root.dir" location="."/>

    <target name="dist" description="Build the distribution">
        <property name="dist.dir" location="dist"/>
        <echo>dist.dir=${dist.dir}, foo=${foo}</echo>
    </target>

    <target name="all" description="Build everything" depends="dist"/>
</project>

Importing this build file using Gradle is a one-liner.

ant.importBuild('src/main/resources/build.xml')

And the output of gradle tasks –all on the command line shows that the targets have been added to the build tasks.

$ gradle tasks --all
...
Other tasks
-----------
all - Build everything
    dist - Build the distribution
...

Properties used in the Ant build file can be specified in the Gradle build or on the command line and, unlike the usual Ant property behaviour, properties set by Ant or on the command line may be overwritten by Gradle. Given a simple build.properties file with foo=bar as the single entry, here’s a few combinations to demonstrate the override behaviour.

Command line invocation	Gradle Build Config	Effect	Result
gradle dist	ant.importBuild('src/main/resources/build.xml')	build.properties value loaded from ant build is used	foo=bar
gradle dist -Dfoo=NotBar	ant.importBuild('src/main/resources/build.xml')	command line property is used	foo=NotBar
gradle dist -Dfoo=NotBar	ant.foo='NotBarFromGradle' ant.importBuild('src/main/resources/build.xml')	Gradle build property is used	foo=NotBarFromGradle
gradle dist -Dfoo=NotBar	ant.foo='NotBarFromGradle' ant.importBuild('src/main/resources/build.xml') ant.foo='NotBarFromGradleAgain'	Gradle build property override is used	foo=NotBarFromGradleAgain

How to deal with task name clashes

Since Gradle insists on uniqueness of task names attempting to import an Ant build that contains a target with the same name as an existing Gradle task will fail. The most common clash I’ve encountered is with the clean task provided by the Gradle BasePlugin. With the help of a little bit of indirection we can still import and use any clashing targets by utilizing the GradleBuild task to bootstrap an Ant build import in an isolated Gradle project. Let’s add a new task to the mix in the Ant build imported and another dependency on the ant clean target to the all task.

<!-- excerpt from buildWithClean.xml Ant build file -->
    <target name="clean" description="clean up">
        <echo>Called clean task in ant build with foo = ${foo}</echo>
    </target>
    <target name="all" description="Build everything" depends="dist,clean"/>

And a simple Gradle build file which will handle the import.

ant.importBuild('src/main/resources/buildWithClean.xml')

Finally, in our main gradle build file we add a task to run the targets we want.

task importTaskWithExistingName(type: GradleBuild) { GradleBuild antBuild ->
    antBuild.buildFile ='buildWithClean.gradle'
    antBuild.tasks = ['all']
}

This works, but unfortunately suffers from one small problem. When Gradle is importing these tasks it doesn’t properly respect the declared order of the dependencies. Instead it executes the dependent ant targets in alphabetical order. In this particular case Ant expects to execute the dist target before clean and Gradle executes them in the reverse order. This can be worked around by explicitly stating the task order, definitely not ideal, but workable. This Gradle task will execute the underlying Ant targets in the way we need.

task importTasksRunInOrder(type: GradleBuild) { GradleBuild antBuild ->
    antBuild.buildFile ='buildWithClean.gradle'
    antBuild.tasks = ['dist', 'clean']
}

Gradle Rules for the rest

Finally, you can use a Gradle Rule to allow for calling any arbitrary target in a GradleBuild bootstrapped import.

tasks.addRule("Pattern: a-<target> will execute a single <target> in the ant build") { String taskName ->
    if (taskName.startsWith("a-")) {
        task(taskName, type: GradleBuild) {
            buildFile = 'buildWithClean.gradle'
            tasks = [taskName - 'a-']
        }
    }
}

In this particular example, this can allow you to string together calls as well, but be warned that they execute in completely segregated environments.

$ gradle a-dist a-clean

Source code

All of code referenced in this article is available on github if you’d like to take a closer look.

Related posts:

1. Why do I Like Gradle?
2. A Groovy/Gradle JSLint Plugin
3. Five Cool Things You Can Do With Groovy Scripts

Gradle is a build system in which builds are described using a declarative and concise DSL written in the Groovy language. This article describes how you can wrap proven Apache Ant Tasks in a Gradle Plugin to make using them as effortless as possible. We’ll also go over some of the tools Gradle provides for building and testing robust Plugin functionality following some easy patterns.

Creating new custom Plugins for Gradle is a relatively straightforward and easy process. Within a Plugin it’s possible to configure a Gradle Project with new properties, dependencies, Tasks – pretty much anything that you can configure in a build.gradle file can be encapsulated into a Plugin for abstraction, portability and reuse. One of the easier ways to add functionality through a Plugin is to encapsulate an existing Ant Task and enhance it by providing the ease-of-use and configuration that Gradle users have come to expect. Recently, I’ve been writing a lot more JavaScript and was looking for static analysis tools to help guide me away from ‘bad habits’. The popular choice for static analysis of JavaScript code seems to be JSLint, so here’s an example of providing that functionality for a Gradle build by wrapping an existing JSLint task and making it easier to work with.

Anatomy of a Gradle Plugin

Gradle plugins can most easily be built using Gradle itself. There is a conveniently available gradleApi() method you can call to include the required framework classes, demonstrated in the dependencies section of a build.gradle file shown in Listing 1. For this example we’re also using the Groovy Plugin and JUnit for testing, so we will include those dependencies as well.

dependencies {
    compile gradleApi()
    groovy group: 'org.codehaus.groovy', name: 'groovy', version: '1.7.6'
    testCompile group: 'junit', name: 'junit', version: '4.8.2
}

Listing 1: The dependencies portion of a build.gradle file for building our Plugin

Creating a new Gradle Plugin is a simple matter of implementing the Gradle interface and its single required method, the skeleton of which is shown in Listing 2. Within the apply method, the Plugin can configure the Project to add Tasks or properties as needed.

class JSLintPlugin implements Plugin<Project>
{
	void apply(Project project)
	{
		//configure the Project here
	}
}

Listing 2: Skeleton of the Plugin implementation

Integrating with Ant

I’ve never been overly fond of Ant, mostly due to the extremely verbose and repetitive nature of the xml declaration. But the fact remains that Ant has been a primary and well-used build tool for years, and the Tasks written for it have been tried and tested by many developers. The Groovy AntBuilder, in combination with the facilities Gradle provides for dependency resolution and classpath management, makes it very easy to incorporate existing Ant functionality into a build and abstract most of the details away from the end user. For this plugin, we add the library containing the Ant Task to a custom configuration so that we can have it automatically downloaded and easily resolve the classpath.
Listing 3 shows how the configuration for the JSLint4Java Task could appear in an Ant build.xml file. Note that you’re on your own here to provide the required library for the classpath.

<taskdef name="jslint"
         classname="com.googlecode.jslint4java.ant.JSLintTask"
         classpath="/path/to/jslint4java-1.4.jar"/>
<target name="jslint">
    <jslint options="undef,white" haltOnFailure="false">
        <formatter type="xml" destfile="${build.dir}/reports"/>
        <fileset dir="." includes="**/*.js" excludes="**/server/"/>
    </jslint>
</target>

Listing 3: Configuring the Ant target in a build.xml file

Gradle makes it easy to separate the configuration and execution phases of the build, allowing for a Plugin to add an Ant Task to the Gradle Project and expose the (optional) configuration in a build script. In addition, Gradle encourages a pattern of providing a ‘convention’ object alongside of a Plugin to clearly separate the concerns.
Listing 4 demonstrates some code from the Plugin implementation that adds a ‘jslint’ Task to a Gradle Project, setting the specific options based on a convention object. Note how we extract the classpath using the notation project.configurations.jslint.asPath.

private static final String TASK_NAME = 'jslint'
private Project project
private JSLintPluginConvention jsLintpluginConvention
...
// some of the code in the apply method
this.jsLintpluginConvention = new JSLintPluginConvention(project)
project.convention.plugins.jslint = jsLintpluginConvention
project.task(TASK_NAME) << {
    project.file(project.reportsDir).mkdirs()
    logger.info("Running jslint on project ${project.name}")
    ant.taskdef(name: TASK_NAME, classname: jsLintpluginConvention.taskName,
        classpath: project.configurations.jslint.asPath)
    ant."$TASK_NAME"(jsLintpluginConvention.mapTaskProperties()) {
        formatter(type: jsLintpluginConvention.decideFormat(),
                destfile:  jsLintpluginConvention.createOutputFileName())
        jsLintpluginConvention.inputDirs.each { dirName ->
            fileset(dir: dirName,
			 includes: jsLintpluginConvention.includes,
			 excludes: jsLintpluginConvention.excludes)
        }
    }
}

Listing 4: Adding a jslint Task to a Gradle Project

If you’re not already familiar with the Gradle syntax for creating new Tasks inline, the project.task(String taskName) method is called to instantiate a new Task, and the ‘< <' syntax is used to to push the Task activities into the 'doLast' Task lifecycle phase.
Allowing for configuration of the Task in a build script is as simple as exposing a method named the same as the Task that takes in a Closure parameter and applies that Closure to set properties on the convention object, as shown in Listing 5.

/**
 * Perform custom configuration of the plugin using the provided closure.
 * @param closure
 */
def jslint(Closure closure)
{
    closure.delegate = this
    closure()
}

Listing 5: A method to allow for configuration of the jslint Task from a build script

The simple use-case

As per usual when working with Gradle, using this plugin in the most basic case requires only these things:

• Declare a dependency on the Plugin source. This can be either a released jar to be downloaded from a repository by Gradle or you can download a jar manually from pretty much anywhere and add it to the classpath directly.
• Apply the Plugin to a Gradle build.
• Call the jslint Task as part of the build.

The entire configuration and usage looks something like Listing 6, assuming that the gradle-jslint-plugin jar is found in /usr/home/gradlelibs.

/* In a build.gradle file */
buildscript {
	dependencies {
		classpath fileTree(dir: '/usr/home/gradlelibs', include: '*.jar')
	}
}
apply plugin: org.kar.jslint.gradle.plugin.JSLintPlugin

/* and on the command line... */
gradle jslint

Listing 6: Configuring the jslint Plugin in a build script and calling it from the command line

By default, this is enough to scan for all .js files under the directory where the build script is located and create a JSLint text report using the basic settings.

The not-so-simple case

Of course in the real world the defaults aren't always what we need, so being able to easily configure the Task is essential. Fortunately, Gradle makes extending a custom Plugin to allow for configuration by a simple Closure, so we can exercise the code from Listing 5 in a build script with the Closure definition in Listing 7.

jslint {
	haltOnFailure = false
	excludes = '**/metadata/'
	options = 'rhino'
	formatterType = 'html'
}

Listing 7: Departing from the default jslint Task configuration

Extending Ant Task capabilities

The Ant Task as is can produce either a plain text document or an xml report, but transforming the results into a more consumable html format is easy to do using an Ant xslt Task. Having Gradle wrap the Task definition allows for simply adding a new formatter type to the configuration and abstracting away the details from the end user. A copy of an xsl file available online is easy to incorporate with the plugin and can be used to transform the xml output into a nicely formatted web page. Being able to program around Ant Tasks like this is a great way to enhance their value in your build. An example of simple output from the test cases included in the Plugin is shown in Figure 1.

Figure 1: Example html formatted output from jslint

Testing using ProjectBuilder

In order to facilitate testing custom Tasks and Plugins, the Gradle framework provides the ProjectBuilder implementation to handle most of the heavy lifting. This gives you a mocked out instance of a Gradle Project that builds into a temporary directory; very handy for testing how Tasks behave under real working conditions. Having tools like this available directly from the framework removes a lot of potential barriers that might otherwise discourage testing of custom components. The source code that accompanies this article uses the ProjectBuilder to achieve 100% code coverage of the project and is available on github at https://github.com/kellyrob99/gradle-jslint-plugin if you'd like to look closer for some ideas on how to test your own Gradle Plugins. An already built version of the jar is also available if you'd like to try without having to build it yourself: https://github.com/kellyrob99/gradle-jslint-plugin/blob/master/releases/gradle-jslint-plugin-0.1-SNAPSHOT.jar.

How could we improve this Plugin?

This implementation represents the 'brute-force' method of wrapping an Ant Task, and there are several ways to enhance its function, if we wanted to spend some additional time and effort on the Plugin. It would actually be far more flexible if we extended the Gradle DefaultTask to provide the actual functionality; this would allow for the possibility of executing the Task separately against different sets of JavaScript in the same project with different configurations of JSLint. In the case of an application with both client and server-side JavaScript, for instance, you might want to apply different rules. In that event you'd probably also want to have the capability to aggregate multiple result sets, which would be a relatively easy feature to add.
Having a separate Task implementation defined would also make it easier to clearly define the inputs and outputs of the process using one of the @Input or @Output Gradle annotations, allowing for incremental builds including JSLint execution. The full set of annotations available from the org.gradle.api.tasks package allow for combining File and/or simple property types to make your Tasks smarter regarding whether or not running again would produce a different result than the last execution.
This article was created using the recently released 0.9 version of Gradle and version 1.4.4 of jslint4java.

Learn more

If you'd like to find out some background or more about Gradle and how to create your own custom Plugins and Tasks, you can find some good information on these sites:

• The Gradle documentation at http://gradle.org/documentation.html
• In particular, the Project API is a great starting point for getting to know Gradle: http://gradle.org/latest/docs/javadoc/org/gradle/api/Project.html
• The excellent examples of Mr. Haki at http://mrhaki.blogspot.com/search/label/Gradle%3AGoodness
• Published plugins on github can be found with this search: https://github.com/search?type=Repositories&language=&q=gradle+plugin

Related posts:

1. Using Gradle to Bootstrap your Legacy Ant Builds
2. Why do I Like Gradle?
3. Groovy inspect()/Eval for Externalizing Data

Both. I started working on this little project a couple of months back using Hudson v1.395 and returned to it after the great divide happened. I took it as an opportunity to see whether there would be any significant problems should I choose to move permanently to Jenkins in the future. There were a couple of hiccups- most notably that the new CLI jar didn’t work right out of the box- but overall v1.401 of Jenkins worked as expected after the switch. The good news is the old version of the CLI jar still works, so this example is actually using a mix of code to get things done. Anyway, the software is great and there’s more than enough credit to go around.

The API

Jenkins/Hudson has a handy remote API packed with information about your builds and supports a rich set of functionality to control them, and the server in general, remotely. It is possible to trigger builds, copy jobs, stop the server and even install plugins remotely. You have your choice of XML, JSON or Python when interacting with the APIs of the server. And, as the build in documentation says, you can find the functionality you need on a relative path from the build server url at:

“/…/api/ where ‘…’ portion is the object for which you’d like to access”.

This will show a brief documentation page if you navigate to it in a browser, and will return a result if you add the desired format as the last part of the path. For instance, to load information about the computer running a locally hosted Jenkins server, a get request on this url would return the result in JSON format: http://localhost:8080/computer/api/json.

{
  "busyExecutors": 0,
  "displayName": "nodes",
  "computer": [
    {
      "idle": true,
      "executors": [
        {
        },
        {
        }
      ],
      "actions": [

      ],
      "temporarilyOffline": false,
      "loadStatistics": {
      },
      "displayName": "master",
      "oneOffExecutors": [

      ],
      "manualLaunchAllowed": true,
      "offline": false,
      "launchSupported": true,
      "icon": "computer.png",
      "monitorData": {
        "hudson.node_monitors.ResponseTimeMonitor": {
          "average": 111
        },
        "hudson.node_monitors.ClockMonitor": {
          "diff": 0
        },
        "hudson.node_monitors.TemporarySpaceMonitor": {
          "size": 58392846336
        },
        "hudson.node_monitors.SwapSpaceMonitor": null,
        "hudson.node_monitors.DiskSpaceMonitor": {
          "size": 58392846336
        },
        "hudson.node_monitors.ArchitectureMonitor": "Mac OS X (x86_64)"
      },
      "offlineCause": null,
      "numExecutors": 2,
      "jnlpAgent": false
    }
  ],
  "totalExecutors": 2
}

Here’s the same tree rendered using GraphViz.

This functionality extends out in a tree from the root of the server, and you can gate how much of the tree you load from any particular branch by supplying a ‘depth’ parameter on your urls. Be careful how high you specify this variable. Testing with a load depth of four against a populous, long-running build server (dozens of builds with thousands of job executions) managed to regularly timeout for me. To give you an idea, here’s a very rough visualization of the domain at depth three from the root of the api.

Getting data out of the server is very simple, but the ability to remotely trigger activity on the server is more interesting. In order to trigger a build of a job named ‘test’, a POST on http://localhost:8080/job/test/build does the job. Using the available facilities, it’s pretty easy to do things like:

• load a job’s configuration file, modify it and create a new job by POSTing the new config.xml file
• move a job from one build machine to another
• build up an overview of scheduled builds

The CLI Jar

There’s another way to remotely drive build servers in the CLI jar distributed along with the server. This jar provides simple facilities for executing certain commands remotely on the build server. Of note, this enables installing plugins remotely and executing a remote Groovy shell. I incorporated this functionality with a very thin wrapper around the main class exposed by the CLI jar as shown in the next code sample.

/**
 * Drive the CLI with multiple arguments to execute.
 * Optionally accepts streams for input, output and err, all of which
 * are set by default to System unless otherwise specified.
 * @param rootUrl
 * @param args
 * @param input
 * @param output
 * @param err
 * @return
 */
def runCliCommand(String rootUrl, List<String> args, InputStream input = System.in,
        OutputStream output = System.out, OutputStream err = System.err)
{
    def CLI cli = new CLI(rootUrl.toURI().toURL())
    cli.execute(args, input, output, err)
    cli.close()
}

And here’s a simple test showing how you can execute a Groovy script to load information about jobs, similar to what you can do from the built-in Groovy script console on the server, which can be found for a locally installed deployment at http://localhost:8080/script.

def "should be able to query hudson object through a groovy script"()
{
    final ByteArrayOutputStream output = new ByteArrayOutputStream()
    when:
    api.runCliCommand(rootUrl, ['groovysh', 'for(item in hudson.model.Hudson.instance.items) { println("job $item.name")}'],
            System.in, output, System.err)

    then:
    println output.toString()
    output.toString().split('\n')[0].startsWith('job')
}

Here are some links to articles about the CLI, if you want to learn more :

• Hudson CLI wikidoc
• Jenkins CLI wikidoc
• A template for PHP jobs on Jenkins
• An article from Kohsuke Kawaguchi
• A nice tutorial

HTTPBuilder

HTTPBuilder is my tool of choice when programming against an HTTP API nowadays. The usage is very straightforward and I was able to get away with only two methods to support reaching the entire API: one for GET and one for POST. Here’s the GET method, sufficient for executing the request, parsing the JSON response, and complete with (albeit naive) error handling.

/**
 * Load info from a particular rootUrl+path, optionally specifying a 'depth' query
 * parameter(default depth = 0)
 *
 * @param rootUrl the base url to access
 * @param path  the api path to append to the rootUrl
 * @param depth the depth query parameter to send to the api, defaults to 0
 * @return parsed json(as a map) or xml(as GPathResult)
 */
def get(String rootUrl, String path, int depth = 0)
{
    def status
    HTTPBuilder http = new HTTPBuilder(rootUrl)
    http.handler.failure = { resp ->
        println "Unexpected failure on $rootUrl$path: ${resp.statusLine} ${resp.status}"
        status = resp.status
    }

    def info
    http.get(path: path, query: [depth: depth]) { resp, json ->
        info = json
        status = resp.status
    }
    info ?: status
}

Calling this to fetch data is a one liner, as the only real difference is the ‘path’ variable used when calling the API.

private final GetRequestSupport requestSupport = new GetRequestSupport()
    ...
/**
 * Display the job api for a particular Hudson job.
 * @param rootUrl the url for a particular build
 * @return job info in json format
 */
def inspectJob(String rootUrl, int depth = 0)
{
    requestSupport.get(rootUrl, API_JSON, depth)
}

Technically, there’s nothing here that limits this to JSON only. One of the great things about HTTPBuilder is that it will happily just try to do the right thing with the response. If the data returned is in JSON format, as these examples are, it gets parsed into a JSONObject. If on the other hand, the data is XML, it gets parsed into a Groovy GPathResult. Both of these are very easily navigable, although the syntax for navigating their object graphs is different.

What can you do with it?

My primary motivation for exploring the API of Hudson/Jenkins was to see how I could make managing multiple servers easier. At present I work daily with four build servers and another handful of slave machines, and support a variety of different version branches. This includes a mix of unit and functional test suites, as well as a continuous deployment job that regularly pushes changes to test machines matching our supported platform matrix, so unfortunately things are not quite as simple as copying a single job when branching. Creating the build infrastructure for new feature branches in an automatic, or at least semi-automatic, fashion is attractive indeed, especially since plans are in the works to expand build automation. For a recent 555 day project, I utilized the API layer to build a Grails app functioning as both a cross-server build radiator and a central facility for server management. This proof of concept is capable of connecting to multiple build servers and visualizing job data as well as specific system configuration, triggering builds, and direct linking to each of the connected servers to allow for drilling down further. Here’s a couple of mock-ups that pretty much show the picture.

Just a pretty cool app for installing Jenkins

This is only very indirectly related, but I came across this very nice and simple Griffon app, called the Jenkins-Assembler which simplifies preparing your build server. It presents you with a list of plugins, letting you pick and choose, and then downloads and composes them into a single deployable war.

Enough talking – where’s the code???

Source code related to this article is available on github. The tests are more of an exploration of the live API than an actual test of the code in this project. They run against a local server launched using the Gradle Jetty plugin. Finally, here’s some pretty pictures for you.

[Show as slideshow]

[View with PicLens]

Related posts:

1. Hooking into the Jenkins(Hudson) API, Part 2
2. A Grails App Demoing the StackExchange API
3. Five Cool Things You Can Do With Groovy Scripts

• Groovy syntax and a very terse and descriptive dsl that makes build scripts easily comprehensible
• flexibility of layout, configuration, organizing build logic – pretty much everything
• incremental builds, based on an easily implementable pattern
• convention over configuration paradigm, thank you very much Maven
• clear separation of build configuration from execution
• extensibility at every level

The most basic Java build

apply plugin: 'java'

That’s it. One line of Groovy in a file called ‘build.gradle’ and you can build a Java project with a Maven-standardized project layout.

├── build.gradle
└── src
    ├── main
    │   ├── java
    │   └── resources
    └── test
        ├── java
        └── resources

Included with the Java plugin are tasks to compile, package, test and javadoc your code. You also get configuration objects to describe the artifacts that your build depends on and those that it produces. Of course, with this most basic setup these configurations don’t yet have anything in them, but in a complex build they’re very handy for isolating the responsibilities of each task. Because you can both configure the existing configurations and add custom ones yourself, it’s very easy to accommodate a project that follows a different structure, whether that is just differently named source directories or multiple directories that need specific processing. This is VERY handy for legacy builds.

Incremental builds

Gradle provides a very easy way to create tasks that are able to execute only if their declared input and/or output artifacts have changed. This makes it trivial to incorporate your custom build behaviour into an incremental build. As an example I’d like to expand upon something I read by Etienne Studer in this month’s JAXmag. It’s a great example of developing an incremental task. First here’s the task implementation almost verbatim from the article. I’ve updated it slightly to make the output more readable using FileUtils, and you can examine the file it spits out from the build/reports/size directory that will be automatically created when it executes.

class Size extends DefaultTask
{
    @InputFiles FileTree inputDir
    @OutputFile File outputFile

    @TaskAction
    void generate()
    {
        def totalSize = inputDir.files.inject(0) { def size, File file -> size + file.size()
        }
        outputFile.text = FileUtils.byteCountToDisplaySize(totalSize)
    }
}

If you simply include this class definition in a build file, it will be automatically compiled and available for use elsewhere in the script. It could just as easily be defined in a separate file(local or remote), in a jar on the classpath or in the buildSrc directory of your Gradle project. This flexibility enables developing and evolving tasks in a very agile fashion, encouraging you to publish the results for reuse instead of re-implementing the logic in other projects.
In order to execute this task as part of a build, we first need to configure it. In this case, I’m configuring it to work on all declared configurations and all source, both code and associated resources. In order to do so, I’m using a couple of Gradle internal classes, FileTree and SourceSet, which actually sound pretty self-explanatory to me. The key part is the assignment of the ‘inputDir’ and ‘outputFile’ properties on the task.

task size(type:Size){
    def filetree = sourceSets.inject(new UnionFileTree()) { FileTree total, SourceSet sourceSet ->
        total += sourceSet.allSource
        total
    }
    inputDir = filetree
    outputFile = file("$reportsDir/size/size.txt")
}

Just to prove that it’s working incrementally, here’s the output from successive invocations. Note that ‘UP-TO-DATE’ in the output that indicates the task was skipped the second time around, because none of the inputs changed and the output file hasn’t been deleted.

gradle-intro$ gradle size
:size

BUILD SUCCESSFUL

Total time: 2.963 secs

gradle-intro$ gradle size
:size UP-TO-DATE

BUILD SUCCESSFUL

Total time: 2.912 secs

This task does actually have a concrete dependency on the Java plugin, since without that convention applied neither the ‘sourceSets’ or ‘reportsDir’ objects would be present. Here’s the complete version of the build file that’s been built so far, 28 lines including imports and spacing.

import org.apache.commons.io.FileUtils
import org.gradle.api.internal.file.UnionFileTree
import org.gradle.api.tasks.SourceSet

apply plugin: 'java'           

task size(type:Size){
    def filetree = sourceSets.inject(new UnionFileTree()) { FileTree total, SourceSet sourceSet ->
        total += sourceSet.allSource
        total
    }
    inputDir = filetree
    outputFile = file("$reportsDir/size/size.txt")
}

class Size extends DefaultTask
{
    @InputFiles FileTree inputDir
    @OutputFile File outputFile

    @TaskAction
    void generate()
    {
        def totalSize = inputDir.files.inject(0) { def size, File file -> size + file.size()
        }
        outputFile.text = FileUtils.byteCountToDisplaySize(totalSize)
    }
}

Sharing your new Task

The simplest way to share build logic is to ‘apply’ it. This simple shorthand covers everything from plugins to local files to remotely hosted resources. Here’s how easy it is to incorporate this particular task implementation and configuration into a build using a relative file path.

apply from : '../gradle-intro/build.gradle'

And because only a simple http connection is required to share it to a broader base, here’s how you can reference a copy on this site. Sorry about the .txt extension, but I didn’t feel like editing php to allow a new filetype today

apply from : 'http://www.kellyrob99.com/blog/wp-content/uploads/downloads/2010/11/gradleSizeTask.txt'

Some other reading

If you’re still not sold on Gradle, here’s some articles I’ve seen recently that at the very least will give you a better perspective.
Hibernate – Gradle why?
Maven VS Gradle VS Ant
Maven to Gradle: Part 1, Part 2, Part 3
A comparison of build script length
Ant/Gradle/Maven comparison
DZone article
OpenMRS Mailing list on Maven VS Gradle

Related posts:

1. Using Gradle to Bootstrap your Legacy Ant Builds
2. A Groovy/Gradle JSLint Plugin
3. Groovy inspect()/Eval for Externalizing Data

    private static final Closure RANDOM_STRING = RandomStringUtils.&randomAlphanumeric
    private static final String TMP_DIR = System.getProperty('java.io.tmpdir')

    @Test
    public void testSerializeListToFile()
    {
        List<String> accumulator = []
        100.times { int i ->
            accumulator << RANDOM_STRING(i + 1)
        }
        File file = new File("$TMP_DIR/inspectListTest.groovy")
        file.deleteOnExit()
        file << accumulator.inspect()
        assertThat(accumulator, equalTo(Eval.me(file.text)))
    }

    @Test
    public void testSerializeMapToFile()
    {
        Map<String, String> accumulator = [:]
        100.times { int i ->
            accumulator[RANDOM_STRING(i + 1)] = RANDOM_STRING(i + 1)
        }
        File file = new File("$TMP_DIR/inspectMapTest.groovy")
        file.deleteOnExit()
        file << accumulator.inspect()
        assertThat(accumulator, equalTo(Eval.me(file.text)))
    }

Two little tidbits of goodness embedded in this example are the ability to capture a method as a closure, in this case RandomStringUtils.randomAlphanumeric(), and the File.deleteOnExit() method – which is only cool because I never noticed it in the API before and it turns out to be a great way to clean up after tests.

A particular usage of this technique I’ve been using lately has been sparked by a shift of tools at work. I’ve worked with the Atlassian stack of web applications for years now, and have always enjoyed the rich feedback Bamboo gives for build history. But now I’m using Hudson as the primary continuous integration tool and one of the things I’ve been sorely missing is the ‘Top 10 Longest Running Tests’. If you’re not familiar with this view in Bamboo, stroll on over to the Groovy build results page and click on the tab.
One of my present priorities at work is to speed up build times and rooting out slow or inefficient tests is one way to do this. It’s possible to elicit this information from JUnit test reports using xsl(see this link for an example) but I came up with a nice way to incorporate similar functionality into a Gradle build. Following the Gradle project-report conventions, this task simply uses the inspect() method to write out a results file in the ‘reportsDir’. I’ve captured this output(manually) over time to track the progress of speeding up test times, and the format makes it easy to read back in results and do deeper analysis and aggregation – such as the creation of csv reports and simple graphs for instance.

tolerance = 2.0
task findLongRunningTests << {
    description= "find all tests that take over $tolerance seconds to run"
    String testDir = "${project.reportsDir}/tests".toString()
    file(testDir).mkdirs()
    File file = file("$testDir/longRunningTests.txt")
    file.createNewFile()
    BufferedWriter writer = file.newWriter()
    writer << parseTestResults('**/TESTS-TestSuites.xml', tolerance)
    writer.close()
}

/**
 * Read in xml files in the junit format and extract test names and times
 * @includePattern ant pattern describing junit xml reports to inspect, recursively gathered from the rootDir
 * @tolerance the number of seconds over which tests should be included in the report
 */
private String parseTestResults(includePattern, float tolerance)
{
    def resultMap = [:]
    fileTree(dir: rootDir, include: includePattern).files.each {
        def testResult = new XmlSlurper().parse(it)
        testResult.depthFirst().findAll {it.name() == 'testcase'}.each { testcase ->
            def key = [testcase.@classname.text(), testcase.@name.text()].join(':')
            def value = testcase.@time.text() as float
            resultMap[(key)] = value
        }
    }
    return resultMap.findAll {it.value > tolerance}.sort {-it.value}.inspect()
}

Note that this works recursively from the ‘rootDir’ of a Project, so it is effective for multi-project Gradle builds. Source code for this simple example is available on github if you want to check it out yourself.

Here’s a sample result drawn from examining some test output in the Gradle build itself. Only one of the tests executed in WrapperTest takes more than 2 seconds to execute.

["org.gradle.api.tasks.wrapper.WrapperTest:testWrapper":2.37]

Related posts:

1. Groovy and CSV: How to Get Your Data Out?
2. Five Cool Things You Can Do With Groovy Scripts
3. Hooking into the Jenkins(Hudson) API, Part 2

CSV is Not XML

Or any other suitably constrained data format for that matter. There are some pretty clear ‘rules’ out there for transporting data in this format, but of course hardly anybody listens to them. I found this and this in the first page of a quick Google search.
However, while looking for available sample material on the web to demonstrate with, I found many great examples of how to screw it up: not quoting empty strings, not escaping embedded line breaks, commas embedded in unquoted text, etc. All of these are things that prevent automation from properly interacting with the data. There are also, of course, the CSV ‘formats’ that insist on embedding comments and descriptions, trying to present more than one matrix of information or any of the other crazy things you can do when writing to a format with no real metadata layer.

< \rant>

Unfortunately, there’s nothing as convenient as an xml schema available for describing what information is encoded in the document and, in general, the only potentially useful metadata in an individual file is the header line. The good news is that all of these deficiencies can be overcome if you know they exist. And there’s some pretty nifty tools you can use to take out much of the guesswork. I’ll suggest a couple of ideas for how to deal with files that don’t adhere to the ‘rules’, but the main example I’ve decided on is a straightforward and well formed set representing the periodic table of elements, provided generously by Akiscode. This file is directly machine readable(has no headers or embedded data), quotes all values and just follows all the ‘rules’. Here’s the first five lines as an example.

Atomic Number	Atomic Mass	Name	Symbol
1	1.00794	Hydrogen	H
2	4.002602	Helium	He
3	6.941	Lithium	Li
4	9.012182	Beryllium	Be
5	10.811	Boron	B

     The simple format of the example CSV file.

Groovy All by Itself

Groovy makes dealing with String objects pretty painless. By adding facilities like easy casting and find()/findAll() for quickly turning raw Strings into real data, you can program some fairly complex questions. In this simple example I’m depending on all values being quoted and I am not protecting against casting problems. I’m running this through Maven which really helps to keep test data organized; all you have to do is drop your test file in /src/test/resources and it’s automagically available on the test classpath.
No specific handling is done for different types here; everything is read in as a String and cast to a more specific type as needed. In a ‘real’ application you’re very likely going to encounter mixed alphanumeric data in any given column of data – ‘UNKNOWN’ instead of a number in a column labeled ‘Quantity’ for instance – and be more careful about casting.

//load and split the file
InputStream inputFile = getClass().classLoader.getResourceAsStream(TEST_FILE_NAME)
String[] lines = inputFile.text.split('\n')
List<String[]> rows = lines.collect {it.split(',')}

/**
 * A little helper method to get rid of the quotes in the input
 * and cast values so they can be compared properly.
 */
private double castToDouble(string)
{
    return string.replaceAll('"', '').toDouble()
}

//OK, it's parsed - let's ask some questions
private static final int ATOMIC_MASS = 1;

def elementsOver200Mass = rows.findAll {castToDouble(it[ATOMIC_MASS]) > 200}
def elementsBetween10And20 = rows.findAll { row ->
            double mass = castToDouble(row[ATOMIC_MASS])
            mass <= 20 && mass >= 10
}

So this approach works fine for well formed input, but falls apart quickly in other cases. For instance, if any columns in the file are missing values the split() function treats them as nulls, leading to different size arrays being stored in the list. Similarly you need to deal with quoted VS non-quoted content, embedded line breaks and other issues individually. Fortunately, some nice guys have done that for us.

OpenCSV

Glen Smith’s OpenCSV library nicely abstracts away a lot of the minutiae of dealing with CSV. It properly deals with escaped characters, multiline input and a host of other issues for you. Give it pretty much anything that satisfies the Reader interface as an input and it’s good to go. The return type of the CSVReader.readAll() method also satisfies the same contract as the plain Groovy version shown above, so we can interact with the parsed results in exactly the same way as in the previous example.
All values are still typed as Strings, so we need to cast in order to compare numeric values.

import au.com.bytecode.opencsv.CSVReader

List<String[]> rows = new CSVReader(
        new InputStreamReader(getClass().classLoader.getResourceAsStream(TEST_FILE_NAME)))
            .readAll()

//same finders as in the Groovy version
def elementsOver200Mass = rows.findAll {it[ATOMIC_MASS].toDouble() > 200}
def elementsBetween10And20 = rows.findAll { row ->
    double mass = castToDouble(row[ATOMIC_MASS])
    mass <= 20 && mass >= 10
}

HsqlDB

HsqlDB allows for using flat files as Text Tables, effectively turning a CSV file into a database table. Data values are cast to the types specified in your table definition, so any potential type cast failures happen right up front when loading the data. The benefit is that once the data is read in no further manual processing is necessary. Note: by default the csv file and the database file must be located in the same directory, as a security precaution (can be overridden through configuration).

In addition, leveraging sql makes querying the data extremely easy. Groovy brings in the Sql class to abstract away a lot of the normal Java boilerplate you encounter when working with a ResultSet. I didn’t do any in-depth testing to prove it out, but this is also the only one of the strategies described here that doesn’t require holding the entire data in memory in order to do arbitrary queries. That can be mitigated in the previously shown methods by processing files line by line rather than in bulk if memory usage is a concern.

//create the table definition to insert
String tableName = 'elements'
String tableDefinition = """CREATE TEXT TABLE $tableName (
    atomic_number INTEGER PRIMARY KEY,
    atomic_mass NUMERIC,
    name VARCHAR(255),
    symbol VARCHAR(3)
);"""

//create a new file database and a table corresponding to the csv file
Sql sql = Sql.newInstance("jdbc:hsqldb:file:${testdbDir.absolutePath}/testdb", 'sa', ''
    ,'org.hsqldb.jdbcDriver')
sql.execute(tableDefinition)

//set the source to the csv file
sql.execute("SET TABLE elements SOURCE '${TEST_FILE_NAME};all_quoted=true'".toString())

//querying the database that's wrapping our CSV file
def elementsOver200Mass = sql.rows("SELECT * FROM $tableName WHERE atomic_mass > ?", [200])
def elementsBetween10And20 = sql.rows(
    "SELECT * FROM $tableName WHERE atomic_mass <= ? AND atomic_mass >= ?", [20, 10])

//simple db aggregates
def count = 0
sql.eachRow("SELECT count(1) FROM $tableName WHERE atomic_mass <= ?", [20]){row->
    count = row[0]
}
def avg = 0
sql.eachRow("SELECT avg(atomic_mass) FROM $tableName".toString()){row->
    avg = row[0]
}

Which One do I Use???

Pick the one that fits best for your use case is the real answer. I didn’t know about the HsqlDB option until recently, but OpenCSV has been a personal standby for years. Then again, if you’re writing something quick and dirty at the script level, the simplicity of just applying a couple of split() operations is pretty appealing. Here’s how I generally decide.

Related posts:

1. Groovy inspect()/Eval for Externalizing Data
2. Five Cool Things You Can Do With Groovy Scripts
3. Achieving Groovy-like Fluency in Java with Google Collections

    +----------------------+-------+-------+
    | Name                 | Files |  LOC  |
    +----------------------+-------+-------+
    | Controllers          |     1 |    42 |
    | Groovy Helpers       |     1 |    47 |
    +----------------------+-------+-------+
    | Totals               |     2 |    89 |
    +----------------------+-------+-------+

Please note that this is about as minimal as you could reasonably get away with(and kinda ugly to boot), but nevertheless it does manage to implement a UI and backend for exercising the entire API in a total of 2 gsp files and 1 controller action. What’s noticeably missing are some tailored views for each of the different JSON responses and the implementation of additional query parameters on the calls.

StackExchange API

This is really pretty well documented and “consistent”. Calls in some cases require an {id} in the url and that’s about it. Each of the individual calls has its own help page that describes the options, like the badges page for instance. To make things a little more helpful while exploring the API, each of the implemented endpoints in this app are also hyper-linked to the corresponding manual page on api.stackoverflow.com.

RESTful Access

Couldn’t be a whole lot easier than using HttpBuilder, which provides nice and concise ways to execute the GET request, inspect the response and deal with success/failure of the request. Aside from assigning the root domain and method using variables, this is little different from the canonical example on codehaus.

def http = new HTTPBuilder("http://api.$domain")
http.request(GET, JSON) {
    uri.path = "/${VERSION}$method"
    headers.'User-Agent' = 'Mozilla/5.0 Ubuntu/8.10 Firefox/3.0.4'
     response.success = { resp, json ->
        answer = json
    }
     response.failure = { resp ->
        answer = "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
    }
}

The syntax here could probably get even Groovier using the Grails REST client plugin wrappers, and indeed incorporating that plugin is how the HttpBuilder dependency is being provided in this app, but I hit on a working implementation first time out so we’ll leave that for another day, shall we?

What’s it Look Like?

Did I mention it was ugly? Where the API call requires a parameter, you get a text field. Each domain gets its own submit button, officially representing my first attempted usage of g:actionSubmit in Grails. Doesn’t work quite the way I expected, but it does certainly work. Click a button and you’re shown the raw JSON result, along with the call you made. In an ugly fashion, or did I say that already? Still, for a couple of hours of hacking, I’m not unhappy with the result.

StackApps

This is where the apps in the contest get shown off and, in a wonderful display of dogfooding, it’s guess what – another StackOverflow clone! The page for this application can be found here. And the source code is up on github if you’d like to take a look.

Next Steps?

Toying with the idea of porting this to appengine, which would unfortunately mean replacing HttpBuilder and the underlying Apache HttpClient in favor of something that didn’t use a threaded approach, most likely using URL fetch. That and some slightly less ugly tabular and/or tree views of the JSON responses would be kinda nice to have.

Related posts:

1. Hooking into the Jenkins(Hudson) API
2. Grails-UI DataTable using XML for a model
3. Hooking into the Jenkins(Hudson) API, Part 2

Creating Collections

Groovy really shines for this one, making the creation of Lists and Maps, empty or populated, an absolute breeze. Java has some help for creating basic Lists but begins to struggle when creating maps. This is an area that google-collections can help in, especially in regards to creating immutable Maps.

//Empty Lists
        List<String> groovyList = []
        List<String> javaList = new ArrayList<String>()
        List<String> googleList = Lists.newArrayList()  //can omit generics

//Populated Lists
        List<String> groovyList = ["1", "2"]
        List<String> javaList = Arrays.asList("1", "2")
        List<String> googleList = Lists.newArrayList("1", "2")

//Immutable Lists
        List<String> groovyList = ["1", "2"].asImmutable()
        List<String> javaList = Collections.unmodifiableList(Arrays.asList("1", "2"))
        List<String> googleList = ImmutableList.of("1", "2")

//Empty Maps
        Map<String, String> groovyMap = [:]
        Map<String, String> javaMap = new LinkedHashMap<String,String>()
        Map<String, String> googleMap = Maps.newLinkedHashMap()

//Immutable Maps
        Map<String, String> groovyMap = ["a":"1", "b":"2"].asImmutable()

        Map<String, String> javaMap = new LinkedHashMap<String,String>()
        javaMap.put("a", "1")
        javaMap.put("b", "2")
        javaMap = Collections.unmodifiableMap(javaMap)

        //OR(works only in Java, will not compile in Groovy)
        Map<String, String> javaMap = new LinkedHashMap<String, String>()
        {
            {
                put("a", "1");
                put("b", "2");
            }
        };

        Map<String, String> googleMap = ImmutableMap.of("a", "1", "b", "2")  //clunky syntax but it works

Filtering Collections

Groovy provides the very handy ‘findAll’ method that allows you to filter a Collection by applying a Closure. Google-collections provides similar facilities using the Predicate interface and two filter methods available statically on Collections2 and Iterables. This would also be a lot more readable if the Predicate definition were extracted but I wanted to show that it’s still possible to create them in-line quickly.

import static com.google.common.collect.Collections2.*

List<Integer> toFilter = [1, 2, 3, 4, 5]
List<Integer> groovyVersion = toFilter.findAll{ it < 3}
List<Integer> googleVersion = filter(toFilter, new Predicate<Integer>()
    {
        public boolean apply(Integer input)
        {
            return input < 3;
        }
    };)

Joining Collections into a String Representation

This one has come up pretty often over the years, and it’s not surprising that where the JDK hasn’t provided, enterprising developers have added support through libraries. The problem is: given a Collection of objects, create a String representation of that Collection suitable for view from a consumer of the system. And admit it – the first time you hand-coded the algorithm you left a trailing comma, didn’t you? I know I did.
Groovy has fantastic support for this use-case by simply including the ‘join’ method on Lists. Google-collections utilizes static calls on the Joiner class along with a simple DSL-like syntax to achieve the same effect. Throw in a static import to make it even more concise and it does a fine job. These two examples yield the same result.

import static com.google.common.base.Joiner.*
def toJoin = ['a', 'b', 'c']
def separator = ', '

//groovy version
def groovyJoin = toJoin.join(separator)

//google-collections version
def googleJoin = on(separator).join(toJoin)

And google-collections also supports join for Maps, something missing from Groovy(although not very hard to implement).

import static com.google.common.base.Joiner.*
def toJoin = [1: 'a', 2: 'b', 3: 'c']
def separator = ', '
def keyValueSeparator = ':'

//results in '1:a, 2:b, 3:c' which is essentially what is returned from Groovy map.toMapString()
def googleVersion = on(separator).withKeyValueSeparator(keyValueSeparator).join(map)

//results in '1 is a and 2 is b and 3 is c'
googleVersion = on(" and ").withKeyValueSeparator(" is ").join(map)

//doing the same in Groovy is slightly more involved, but really not that bad
def groovyVersion = toJoin.inject([]) {builder, entry ->
            builder << "${entry.key} is ${entry.value}"
            builder
        }.join(' and ')

Multimaps

Multimaps are one of the more interesting parts of google-collections, at least to me. Have you ever found yourself writing code to create a Map of keys to Lists? Well the various Multimap implementations in google-collections mean you never have to write that boilerplate kind of code again. Here’s a “first-stab” effort to simulate a fairly generic Multimap with pure Java code.

public class JavaMultimap
{
    private Map<Object, List<Object>> multimap = new LinkedHashMap<Object, List<Object>>();

    public boolean put(Object key, Object value)
    {
        List<object> objects = multimap.get(key);
        objects = objects != null ? objects : new ArrayList<object>();
        objects.add(value);
        multimap.put(key, objects);
        return true;
    }
}

And the same thing in Groovy, achieving a slightly smaller version.

class GroovyMultimap
{
    Map map = [:]

    public boolean put(Object key, Object value)
    {
        List list = map.get(key, [])
        list.add(value)
        map."$key" = list
    }
}

I did some primitive timings comparing Java, Groovy and google-collections Multimaps implementations and, as you’d pretty much expect, google clearly takes the lead. Where things really start to get interesting though is when you start using the Multimap in Groovy code. Imagine if you will that you want to iterate over a collection of Objects and map some of the properties to a List. Here’s a contrived example of what I’m talking about, but applying this same pattern to domain objects in your application or even a directory full of xml files is pretty much the same. If you look closely you’ll notice that Groovy actually makes this a one liner to extract all values for a property across a List of Objects(used in the assertion), but I imagine that Multimap is probably a better alternative for large data sets.

class GoogleCollectionsMultiMapTest
{
    private Random random = new Random()

    @Test
    public void testMultimap()
    {
        def list = []
        10.times {
            list << createObject()
        }
        List properties = ['value1', 'value2', 'value3']
        Multimap multimap = list.inject(LinkedListMultimap.create()) {Multimap map, object ->
            properties.each {
                map.put(it, object."$it")
            }
            map
        }
        properties.each {
            assertEquals (multimap.get(it), list."$it")
        }
    }

    Object createObject()
    {
        Expando expando = new Expando()
        expando.value1 =  random.nextInt(10) + 1
        expando.value2 =  random.nextInt(100) + 100
        expando.value3 =  random.nextInt(50) + 50
        return expando
    }
}

So Where Does This Get Us?

Between google-collections and the newer guava-libraries that contain it, there’s lots of help available for simplifying programming problems and making your code more readable. I haven’t even touched on the newly available support for primitives, Files, Streams and more, but if you’re interested in reducing the amount of code you write AND simultaneously making it more readable you should probably take a look. There’s a very nice overview available in part one and part two by Aleksander Stensby. And here’s a closer look at what google-collections can do for you.

Source Code

As per usual, source code is available at github as a maven project. Big thanks to the Spock team for sharing how they configure GMaven to properly utilize Groovy 1.7. Please feel free to take a look and comment here. Thanks!

Related posts:

1. Groovy and Glazed Lists with Grape
2. Different Flavors of Embedded Groovy in Java Apps or “How To Make your Java Groovier!”
3. My favorite new Groovy trick

Where Hibernate Validator Shines

Annotations on domain classes allow for easily validating object state at the time of persistence. Excellent integration with frameworks like JBoss Seam allow this same ability to be utilized for validating web forms on the client-side with little more than an <s:validateAll/> tag. Seam practically hides the entire interaction with validation components from the developer. Since validation rules are defined directly in the domain class, you can (almost) guarantee that no objects with inconsistent state will ever end up being saved in your database. There are certain validations that aren’t possible to verify without actually looking in the database, unique constraints for example, but generally in my experience hibernate validator is extremely easy to configure and work with. Implementing CRUD functionality is pretty trivial, and UIs can achieve consistency since all validations are applied equally.
Alternatively, if you can’t or don’t want to use annotations for some reason, you can specify your validation rules in an xml file. Usually a singleton validation.xml file is made available on the classpath and picked up automagically when a ValidationFactory is created. A simple xml configuration looks like this:

<constraint-mappings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://jboss.org/xml/ns/javax/validation/mapping validation-mapping-1.0.xsd" xmlns="http://jboss.org/xml/ns/javax/validation/mapping">
    <default-package>org.kar.test</default-package>
    <bean class="ValidateTestableClass">
        <field name="name">
            <constraint annotation="javax.validation.constraints.NotNull">
        </constraint>
    </field>
</bean>

and is meant to be applied to this simple class:

class ValidateTestableClass
{
    int id
    String name
    String description
    boolean enabled
}

Comparing with Grails Validation

Grails automatically provides validation capabilities for domain classes and command objects, and enables adding the same behavior to any pogo through a combination of the @Validateable annotation and a static constraints closure. Adding validation support to arbitrary classes also requires specifying which packages to scan for the annotation.
Plugin support from projects like bean-fields simplifies the handling of client-side validation and rendering error markers in the UI, an ability which the Grails framework provides natively by adding an ‘errors’ field directly onto the domain or command object class instances bound to a web form.

Dynamic Constraints

Both the hibernate validator and the Grails strategies for applying validation described here have the same limitation: both are universally applied to all instances of a class. There’s no easy apparent way to override those constraints at runtime, although I suspect that some fancy MOP’ing or configuration could probably be used to accomplish overrides at runtime.
Hibernate validator also supports creation of ad hoc validators by seeding with one or more xml documents. Or if you’re like me and hate hand editing xml, you can leverage Groovy to take a bit of the pain away. Here’s the same xml snippet from above in a Groovy Closure, generated simply by turning the DomToGroovy class loose on the raw xml:

looseConstraint = {
        mkp.declareNamespace(xsi: 'http://www.w3.org/2001/XMLSchema-instance')
        'constraint-mappings'(xmlns: 'http://jboss.org/xml/ns/javax/validation/mapping',
                'xsi:schemaLocation': 'http://jboss.org/xml/ns/javax/validation/mapping validation-mapping-1.0.xsd') {
            'default-package'('org.kar.test.objects')
            bean('class': 'ValidateTestableClass') {
                field(name: 'name') {
                    constraint(annotation: 'javax.validation.constraints.NotNull')
                }
            }
        }
    }

Losing all the angle brackets is a good start, but we really haven’t saved a lot of typing. Until you start taking advantage of the ability to define more complicated structures. Note the use of a list structure here to apply NotNull constraints to multiple fields.

strictConstraint = {
        mkp.declareNamespace(xsi: 'http://www.w3.org/2001/XMLSchema-instance')
        'constraint-mappings'(xmlns: 'http://jboss.org/xml/ns/javax/validation/mapping',
                'xsi:schemaLocation': 'http://jboss.org/xml/ns/javax/validation/mapping validation-mapping-1.0.xsd') {
            'default-package'('org.kar.test.objects')
            bean('class': 'ValidateTestableClass') {
                ['name', 'description'].each {
                    field(name: it) {
                        constraint(annotation: 'javax.validation.constraints.NotNull')
                    }
                }
                field(name: 'id') {
                    constraint(annotation: 'javax.validation.constraints.DecimalMin') {
                        element(name: 'value', '2')
                    }
                }
                field(name: 'enabled') {
                    constraint(annotation: 'javax.validation.constraints.AssertTrue')
                }
            }
        }
    }

Applying Dynamic Constraints

Applying constraints is as simple as converting Closures to xml and mapping them to a Configuration object, which then supplies a Validator to use. StreamingMarkupBuilder is utilized to create the xml behind the scenes.

    /**
     * Create a configuration object passing closures as validation mapping documents.
     * @param closures closures to render into validation mapping documents
     * @return config
     */
    public Configuration createConfig(Closure... closures)
    {
        Configuration config = Validation.byDefaultProvider().configure()
        closures.each {
            config.addMapping(new ByteArrayInputStream(GroovyXmlConversionUtil.convertToXml(it).bytes))
        }
        config
    }

I haven’t tested the use of multiple mappings extensively, but minimally each class you’re configuring must be confined to a single mapping – you can’t extend the validations by layering configurations on top of one another. You should however be able to map constraints for different classes in separate Closures.

Crying out for a Builder!

Going from Closures to xml is a quick and dirty way to test out this functionality, but what would really be nice is a Builder that could create an appropriate validation environment more directly. At the least it would allow for removing the namespace declarations and explicit package naming that make up the bulk of the content.

So what do you get?

1. 1. Ability to declare validations against any existing Java or Groovy class without changing the source code
2. 2. Programmatic ability to create the configuration of validations
3. 3. A choice of which validations to apply at runtime
4. 4. Consistency with the behavior of domain class validation

And what’d I get? A fun bit of quick coding on a Saturday afternoon. Nice! Source code is available on git-hub if you want to check it out.

Related posts:

1. Groovy and CSV: How to Get Your Data Out?
2. Jira Grails Plugin
3. Breaking Weak CAPTCHA in… slightly more than 26 Lines of Groovy Code

Fair Disclosure

We live with 3 pets. Two dogs and one cat. The dogs are black and tan. The cat is gray and white. The end result is that we have hair of every pretty much every possible color just about everywhere, all the time. If you too are a multi-pet owner you may know of the fur dust-devils that can take over your kitchen floor, the inevitable dirt by the door where the dogs come in or the “track proof” kitty litter that inevitably gets tracked. Over time, in order to properly clean up after the pets we’ve invested in a very good vacuum cleaner and a steam cleaner to go with it. Not to mention a weekly two hour visit from a professional cleaner. The Roomba has taken a lot of the work out of it, and that’s a good thing because if there’s one thing I hate it’s doing dishes by hand. Followed closely by vacuuming.

First Impressions

After fully charging the unit, we set it loose on the main traffic areas of our house – the living room and the kitchen. The first time we ran it the area was left pretty much ‘as is’ and the Roomba needed help a couple of times when it got caught up around chairs, table legs and an edge of a couch. The dirt trap was also stuffed to the max with pet hair after an hour. I blame the fact that the couch it DID fit under had probably gone way too long without us pulling it away from the wall. My Bad. At the end of an hour, the floor was very visibly and noticeably cleaner.

Week One

Let me let you in on a little secret. I’m a tech guy. In practical terms what does that boil down to? Wires. I’ve got em. Where I can, they’re routed, fixed and generally hidden, but in the shady recesses behind desks where Roombas might venture they can be found, exposed and vulnerable to the elements. Turns out it’s not that big a deal. As things the Roomba got stuck on revealed themselves, they slowly got evaluated, assessed, and in some cases removed. Getting caught under the edge of one couch? Solved by some $1.99 plastic furniture lifts. Stuck rotating around the legs of a wooden chair for 10 minutes? Chair ended up in the garage. The dirt trap was still pretty much full after every run, one per day in a different part of the house everyday, but at the end of a week the house was looking much better overall. And as a side benefit there was a whole lot less crap lying around on the floor.

Week Two

So by this time there’s a really big bonus to note about the Roomba over a conventional vacuum. Not only does it not require a lot of direct interaction to get your floors clean, but it also doesn’t make the pets go mental when you turn it on. One sure way to make the dogs and cat disperse in an immediate fashion is to turn on the vacuum cleaner. The noise drives them out of the room as far as they can get and it takes awhile for them to settle down when you finally shut it off.

The Roomba on the other hand doesn’t appear to phase them in the least. It’s (reasonably) quiet for starters. It also isn’t really all that “aggressive”. Our big dog, Aurora, a 13 year old Black Labrador/Dane cross, has a habit of falling asleep right in the middle of the room. The Roomba happily cleans right around her, so much so that she might turn her head to look at it, but rarely is it enough to make her actually stand up. Keep in mind this is when it runs directly into her. The pressure sensor on this piece of hardware is pretty impressive. Before I let it run into Aurora, I did test it out myself first of course – what do you people think I am, a monster? In this way the robot most definitely appears to obey the first law; when it runs into something it immediately stops and tries a different direction. And it stops on a dime. The cat thinks that we bought him a new toy. He chases and jumps on the Roomba like it was covered in catnip. LMAO.

After two weeks we figured out how to properly clean out the unit. Finally breaking down and RTFM helped to figure out how to clean the filter. The chassis is pretty much hollow so a lot of dirt and hair had accumulated there, something you can fix by pretty much shaking the whole thing over a garbage can, preferably outside. The carpet rollers also needed a little attention by this point, as pet hair and assorted other things(thread, plastic twist tie, etc) had been rooted out from the floors of our home and wrapped around the spindle. At this point our cleaning lady, a wonderful woman who has contributed much to keeping our house livable, not only noticed the difference but took the initiative to suggest tackling some of the less “essential” cleaning aspects and generally taking things up a notch: removing all the sink and shower knobs, cleaning in and around them; cleaning the oven(more than the once/yr average); removing all of the window screens, cleaning both them and the windows they’re covering. Did I mention she also takes the dogs out for us once a week? If you’re around #yyj and you want someone hard-working and dependable to give you a hand I can highly recommend this lady. Thank you Sandy!

The trap is still pretty much always full.

Week Three

Stepping it up a notch, the Roomba is now being run 2 or 3 times a day, as much as possible covering the 3 primary areas of the house: the bedroom/bathroom areas, the dining room, and the living room and kitchen. All told it’s about 1800 square feet, broken down in order to roughly 600, 400 and 800 sq/ft chunks. The dirt trap now is starting to not be full. The Roomba is covering more territory every day, all of the electrical cords are tucked away where they can’t be harmed. The cat is now bored with the Roomba and merely takes one tiny step back when it threatens to run into him. The dogs continue to mostly ignore it. The house actually looks better than when we moved in. Despite the two month lifetime stated for the filter, I retired the first filter because it was, quite frankly, toast. Fortunately along with the “special” price 3 more filters were included in the box with the original purchase.

Week Four

Keeping up with things now is a lot easier. The Roomba runs once or twice a day, usually either right after we get home from work(in an area of the house we’re not in), and at night when we’re going to bed. The noise is pretty minimal so running it downstairs in the high-traffic areas at night while we’re sleeping works out really well. It’s also become kind of automatic to deal with. If we’re walking past the place where we charge the Roomba and the light is “green” we pretty much grab it and take it somewhere to do it’s job. If we’re walking past it in the hallway and the light is “red” then it’s time to empty it out and take it back to the charging station. That and remembering to pick up the limited number of things left hanging around on the floor that it can get caught on is pretty much it.

Is it worth it?

Well unless you’ve been paying really bad attention so far, you already know that I think it is.
So here’s the breakdown as I see it.

The Roomba cleans for between 7 and 14 hours per week. I can’t honestly say we ever averaged that much time pushing a vacuum before. The physical effort involved I would estimate at about 5 minutes or so per run, to clear up the floor, turn it loose and then come back and empty it later. So that’s about 30 to 60 minutes a week. And that is kinda the point of a robot isn’t it? Nothing like a division of labor that is 10% planning and 90% labor.

It cleans for approximately an hour every time, and takes a couple or three hours to charge back up to full. I haven’t been explicitly timing this, but it hasn’t varied a lot from this pattern.

It occasionally gets stuck. Personally I just think it’s kinda cool to watch it edge around the top of a staircase, and usually it’s sensible enough to pull back, but every once and awhile it just goes a teeny tiny bit too far and gets stuck slanted and bleating at the edge of the stair. Third Law in full effect here.

Similarly it seemingly hates the half inch foam mat we have under a painting easel, getting caught half-on/half-off the corners of it. With the more expensive Roomba models these are the kind of problems that seem to be usually solved by virtual “walls” which mark boundaries the Roomba should not cross. For us that bought the cheap version, some pretty simple measures close the gap. For the top of the stairs? We already have a gate(s) in place for the pets and it works equally well to keep the Roomba away from places it should not go. For the for mat? Sometimes a chair, sometimes a pair of shoes – whichever one happens to be closer at hand at the time. Weight really isn’t a big factor, but height is. If a surface is high enough and the Roomba bumps into it, it just back off. If it’s too low, the Roomba tries to climb it. Long story short is if there’s anywhere you don’t want your Roomba to go, you can gate it easily with just about anything that will stand in its way and is more than an inch high. As already stated this includes, but is not limited to, pet gates, furniture, people and pets themselves. This pretty much covers the Second Law.

If it gets stuck, well… so what? So far this thing has been so careful that it’s trained us to better behavior. Less things to get stuck on means less things on the floor means less clutter and more things in a closet or on a shelf. If it gets stuck on what’s left, it beeps to tell you so. Give it a kick and set it back on its merry way. The most you lose is <1hr of cleaning time, and you get the balance back with nothing more than moving it around and hitting the button again. Honestly with one simple color/status changing button you get all the information you need about the Roomba just by happening to walk by it. Simple UI FTW!

For the truly interested out there, I tracked this month or Roomba in a Google spreadsheet. Blame my OCD if you want. And for those of you who might be reading this who aren’t total geeks, the Three Laws of Robotics are classic Asimov science fiction.

Date	Duration	Areas	#TimesEmptied	Notes
1/30/2010	~45m	Den + kitchen	1	Totally full
1/31/2010	~30m	Master bedroom + dressing room	1	Totally full
2/1/2010	60m+	Den + kitchen + storage	3	had trouble getting into storeroom past easel; stuck under couch in places; sticks in corner of couch by glass doors consistently; realized I wasn't emptying trap properly after 2nd time; trap was full even after emptying properly the last time
2/2/2010	60m+	3 bedrooms upstairs	3	trap was half to 3/4 full each time
2/3/2010	<60m	front room	2	trap was full first time(40m), almost empty the 2nd; light turned yellow after 20m, orange after 40m; tray got caught on persian rug and fell off; first time died under 1h
2/4/2010	~60m	den+kitchen+storeroom	2
2/5/2010	~60m	3 bedrooms + bathroom	1
2/6/2010	~45m	living room	1	pretty full, not totally
2/7/2010	~60 m	living room	1	full
2/7/2010	~60 m	2 bdrms & bath	1
2/8/2010	~60m	2 bdrms & bath	1
2/8/2010	~60m	Den + kitchen + storage	2
2/9/2010	~60m	front room	1
2/10/2010	unattended	front room	1	got caught under couch, weren't home to correct it
2/10/2010	~60m	den+kitchen+storeroom	3
2/11/2010	~60m	2 bdrms & both baths	1
2/11/2010	~60m	whole downstairs	3	chassis was filled with crud, need to be more thorough when emptying; trap fell off
2/12/2010	~60m	front room	2	caught under couch
2/12/2010	~60m	2 bdrms & both baths	1
2/13/2010	~60m	front room	1	trap was only about a third full!
2/14/2010	~60m	front room	1
2/14/2010	~60m	2 bdrms & both baths	1
2/14/2010	~60m	den+kitchen+storeroom	1	run when going to bed, minimal noise
2/15/2010	~60m	2 bdrms & both baths	1
2/16/2010	~60m	2 bdrms & both baths	1	full, but not compact full
2/16/2010	25 min	den, back room	1	1/2 full
2/17/2010	~60 min	den	1	1/2 full
2/17/2010	~60m	3 bdrms & both baths	1	1/2 full, finished completely underneath a bed :(
2/18/2010	~60m	front room	1	1/2 full
2/19/2010	~60m	2 bdrms & both baths	1	Got stuck once early on, 1/2 full
2/20/2010	~60m	whole downstairs	2	Full both times
2/20/2010	~60m	2 bdrms & both baths	1
2/21/2010	~60m	3 bdrms & both baths	1	full. Got stuck hanging on lip of stairs.
2/21/2010	60m+	den+kitchen+storeroom	3	full each time
2/21/2010	~60m	whole downstairs		run when going to bed, minimal noise
2/22/2010	~60m	2 bdrms & both baths	1
2/23/2010	~60m	2 bdrms & both baths	1
2/24/2010	~60m	den+kitchen+storeroom	1
2/24/2010	~60m	den+kitchen+storeroom	1	fun for the guests!
2/25/2010	~60m	den+kitchen+storeroom	1
2/26/2010	~60m	front room	1
2/27/2010	~60m	2 bdrms & both baths	1
2/28/2010	~60m	whole downstairs	1
2/28/2010	~60m	2 bdrms & both baths & closet	1
3/1/2010	~60m	front room	1	real problems with the couch and persian carpet; got stuck on both more than once
3/1/2010	~60m	den+kitchen+storeroom	1

No related posts.

Gource

Gource is a software version control visualization program with support for Git, Mercurial, and with a tiny bit of extra effort, CVS and Subversion. It’s available as a macport install, as is the ffmpeg program used to stitch together a series of screenshots into the final video.  All of these visualizations were created with Git or Git-svn. When the Gource program is running it brings up an interactive window so you can interact with it to a degree, affecting the camera, zoom, etc.  The default resolution is 1024×768 but I tried to bump it up to 1920×1080. I’m assuming that stretched the limits of the rendering, as there’s quite a bit of pixellation in the final product; either that or the ffmpeg params need to be tweaked. The three visualizations on this page were all created using this one-liner command line invocation (broken down here over several lines for readability).

gource -1920x1080 --highlight-all-users --bloom-intensity .3 -a 1 -s .01
     --hide-dirnames --hide-filenames --stop-at-end --output-ppm-stream -
     | ffmpeg -y -b 3000K -r 60 -f image2pipe -vcodec ppm -i -
     -vcodec libx264 -vpre default  gource.mp4

Groovy

From the initial commit on August 28, 2003, Groovy has come a long way. I tried the Gource provided python script to translate an svn log file into the required format the first time, but cloning the repository with git-svn seemed to produce better results. It’s interesting to see that the overall structure of the Groovy project doesn’t look like is has changed very much over the years. I recognized a lot of the names you see on the screen. Be sure to watch the 1080p version if you’re having trouble reading the text and see if you can’t find some recognizable committers.

Click here to view the embedded video.

Grails

Starting on Sunday March 14, 2004 here’s how it all unfolded, at least from the perspective of the public Git repository. Grails looks like it grew a little more spontaneously than Groovy; there are more frequent restructurings and some transitions that seem to cover a whole lot of ground in very little time. In particular watch out for the burst of activity in the summer of 2009.

Click here to view the embedded video.

Griffon

September 17, 2008 marks the first commit for Griffon and this short but sweet video explodes right from the start. Benefiting greatly from the Grails model this project shows a lot of progress in a very short time indeed. Most, if not all, of the names you see here you probably caught in the previous two videos as well.

Click here to view the embedded video.

What’s the Point?

Watching these gives me (and maybe you) some appreciation for how much effort over time it takes to bring a new language/framework/technology to maturity and all of the individual contributions that go into it. Gource provides a lot more detail into the actual structure if you want to dig in. For purposes of keeping these images fairly uncluttered I removed file and directory names and just focused on the committers and the general structure. I think I gained a git of insight into how a codebase can evolve, and properly appreciate that organizing the increasing complexity of an ever-growing system over time is not a simple job. Plus they look cool. Enjoy!

Related posts:

1. Bamboo, Grails and Git for Continuous Integration
2. A Grails App Demoing the StackExchange API
3. Grails-UI DataTable using XML for a model

The Pretty Pictures

Here are the three images to test against. In order to put them in a suitable format for the open source tesseract-ocr program to process we need to make them bigger, remove the background noise and transform them into a ‘tif’ format. The python program we’re porting utilizes the PIL library for image handling and the pytesseract library for wrapping tesseract; I didn’t look very hard for java equivalents and just coded the required functions directly.

Reading in the Image

The python code for this is three lines, one to load the image and a couple more to convert it into a format suitable for directly manipulating pixel color through RGB values. Groovy takes a bit more to do the same, but being able to use a ‘with’ block makes interacting with the Graphics object a lot cleaner than the same Java code

//python
from PIL import Image
img = Image.open('input.gif')
img = img.convert("RGBA")
pixdata = img.load()

//Groovy
BufferedImage image = ImageIO.read(new File(fileName))
BufferedImage dimg = new BufferedImage(image.width, image.height, BufferedImage.TYPE_INT_ARGB)
dimg.createGraphics().with {
    setComposite(AlphaComposite.Src)
    drawImage(image, null, 0, 0)
    dispose()
}

Removing the Background Noise

In both cases we’re doing essentially the same thing: finding all non-black pixels and setting them to white. This leaves only the actual embedded text to stand out. Being able to utilize the Java Color constants makes the Groovy version a little more readable, IMO, but otherwise the two pieces of code are generally equivalent.

//python
for y in xrange(img.size[1]):
    for x in xrange(img.size[0]):
        if pixdata[x, y] != (0, 0, 0, 255):
            pixdata[x, y] = (255, 255, 255, 255)

//Groovy
(0..<dimg.height).each {i="" -="">
    (0..<dimg.width).each {j="" -="">
        if (dimg.getRGB(j, i) != Color.BLACK.RGB)
        {
            dimg.setRGB(j, i, Color.WHITE.RGB)
        }
    }
}

Resizing the Image

Python‘s library usage really shines here, making this a one line call. Not quite the same in Java-land, although again there’s probably a better way to do this(I just don’t know it offhand).

//python
big = im_orig.resize((116, 56), Image.NEAREST)

//Groovy
dimg = resizeImage(dimg, 116, 56)
...
def resizeImage = {BufferedImage image, int w, int h ->
    BufferedImage dimg = new BufferedImage(w, h, image.type)
    dimg.createGraphics().with {
        setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BILINEAR)
        drawImage(image, 0, 0, w, h, 0, 0, image.width, image.height, null)
        dispose()
    }
    return dimg
}

By this point the original images now look like this, and are almost ready for OCR.

Converting to a tif File

This one turns out to be a bit of a PITA in Java and particularly on a Mac, and represents the bulk of the Groovy code. Unfortunately it is also the only format that tesseract appears to accept ‘out of the box’. After googling the fun that is JAI and working with the .tif(f) format with it on a Mac, I ended up taking the code kindly provided in this blog post and Groovified it a bit to make a working transformation. Thanks very much to Allan Tan for that. One more time, there’s likely a better/easier way to do this, but honestly it’s more effort than I’m willing to put in on a weekend afternoon just to satisfy my curiosity.

//python
ext = ".tif"
big.save("input-NEAREST" + ext)

//Groovy
void convertToTiff(String inputFile, String outputFile)
{
    OutputStream ios
    try
    {
        ios = new BufferedOutputStream(new FileOutputStream(new File(outputFile)))
        ImageEncoder enc = ImageCodec.createImageEncoder("tiff", ios, new TIFFEncodeParam(compression: TIFFEncodeParam.COMPRESSION_NONE, littleEndian: false))
        RenderedOp src = JAI.create("fileload", inputFile)

        //Apply the color filter and return the result.
        ColorConvertOp filterObj = new ColorConvertOp(ColorSpace.getInstance(ColorSpace.CS_sRGB), null)
        BufferedImage dst = new BufferedImage(src.width, src.height, BufferedImage.TYPE_3BYTE_BGR)
        filterObj.filter(src.getAsBufferedImage(), dst)

        // save the output file
        enc.encode(dst)
    }
    catch (Exception e)
    {
        println e
    }
    finally
    {
        ios.close()
    }
}

OCR with Tesseract-OCR

Finally we need to pass the processed image to tesseract so it can ‘read’ it for us. Again, the python library makes this a breeze, but calling out to a command line program with Groovy is so simple that it ends up being about the same. Tesseract itself is available as a macport, as well in downloadable unix binaries and a windows executable so installing the software is a breeze.

//python
from pytesser import *
image = Image.open('input-NEAREST.tif')
print image_to_string(image)

//Groovy
def tesseract = ['/opt/local/bin/tesseract', tmpTif, tmpTesseract].execute()
tesseract.waitFor()
return new File("${tmpTesseract}.txt").readLines()[0]

Testing it out

To test it out I implemented the code in a maven project, iterate over the images and write out intermediate results to a temp directory. And it only works on two out of three of the cases. For some reason tesseract insists on consistently seeing ‘e4ya’ as ‘e4ga’. I tried to see if I could get it working by tweaking the image manipulation parameters and the order of operations(resizing before removing the background noise for instance) but that just caused the other cases to fail as well. Since in the final image the ‘y’ seems pretty clear, it’s more likely that tweaking tesseract configuration might yield better results.

public void testPrintImage()
{
    def breaker = new CaptchaBreaker()
    /* tesseract interprets "e4ya" as "e4ga" unfortunately */
    ['9koO', 'jxt9'/*,'e4ya'*/].each {String imageName ->
        def fileName = "src/test/resources/${imageName}.gif"
        assertEquals("Testing $imageName",imageName, breaker.imageToString(fileName))
    }
}

C’est Finis

I had some fun playing with areas of Java that I don’t usually interact with, and gained some appreciation for the diversity and ease-of-use exposed by just a couple of python libraries. It’s comforting to note that I was able to implement all of the required functionality from those libraries in < 90 lines of Groovy. With a little more effort I think the final product could be tweaked to avoid the intermediate file system reads/writes as well, but that’s for another day.
Source code is available on github if you’d care to take a look, and thanks for stopping by!

Related posts:

1. Different Flavors of Embedded Groovy in Java Apps or “How To Make your Java Groovier!”
2. Achieving Groovy-like Fluency in Java with Google Collections
3. Groovy and Hibernate Validator for Dynamic Constraints