The Kaptain on ... stuff » JSON

Five Cool Things You Can Do With Groovy Scripts

TheKaptain — Sun, 04 Dec 2011 21:51:34 +0000

1. Ensure all of your Jenkins builds are building the correct branch from source control

I manage a large number of builds at work, spread across several build servers. When we release a new version all of the builds need to be updated to point to new working branches. This script takes advantage of the fact that our branches all end in the version number to quickly check that all of the last builds were on the expected version.
Using the Jenkins API is very easy and the new capabilities of Groovy introduced by JSONSlurper make it easier than ever to consume.

2. Look up the artifacts from the last successful Jenkins build

This is very handy if you want to automate deployment of software from your build system and can replace a lot of error prone manual updating of scripts. In this particular example you can link up quickly with the latest, greatest build of JRuby from their public CI server.

Here’s the sample output at time of writing:

[relativePath:dist/jruby-bin-1.7.0.dev.tar.gz, fileName:jruby-bin-1.7.0.dev.tar.gz, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.tar.gz.md5, fileName:jruby-bin-1.7.0.dev.tar.gz.md5, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.tar.gz.sha1, fileName:jruby-bin-1.7.0.dev.tar.gz.sha1, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.zip, fileName:jruby-bin-1.7.0.dev.zip, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.zip.md5, fileName:jruby-bin-1.7.0.dev.zip.md5, displayPath:null]
[relativePath:dist/jruby-bin-1.7.0.dev.zip.sha1, fileName:jruby-bin-1.7.0.dev.zip.sha1, displayPath:null]
[relativePath:dist/jruby-complete-1.7.0.dev.jar, fileName:jruby-complete-1.7.0.dev.jar, displayPath:null]
[relativePath:dist/jruby-complete-1.7.0.dev.jar.md5, fileName:jruby-complete-1.7.0.dev.jar.md5, displayPath:null]
[relativePath:dist/jruby-complete-1.7.0.dev.jar.sha1, fileName:jruby-complete-1.7.0.dev.jar.sha1, displayPath:null]
[relativePath:dist/jruby-jars-1.7.0.dev.gem, fileName:jruby-jars-1.7.0.dev.gem, displayPath:null]
[relativePath:dist/jruby-jars-1.7.0.dev.gem.md5, fileName:jruby-jars-1.7.0.dev.gem.md5, displayPath:null]
[relativePath:dist/jruby-jars-1.7.0.dev.gem.sha1, fileName:jruby-jars-1.7.0.dev.gem.sha1, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.tar.gz, fileName:jruby-src-1.7.0.dev.tar.gz, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.tar.gz.md5, fileName:jruby-src-1.7.0.dev.tar.gz.md5, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.tar.gz.sha1, fileName:jruby-src-1.7.0.dev.tar.gz.sha1, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.zip, fileName:jruby-src-1.7.0.dev.zip, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.zip.md5, fileName:jruby-src-1.7.0.dev.zip.md5, displayPath:null]
[relativePath:dist/jruby-src-1.7.0.dev.zip.sha1, fileName:jruby-src-1.7.0.dev.zip.sha1, displayPath:null]

3. Read in a CSV file, filter it, and write out the result

Groovy Grapes and one of my favorite libraries, OpenCSV, make it trivial to script ‘one off’ solutions for removing work that would normally take place manually in Excel. For me it comes up most often with data import/export from systems I’m working on and the need to create particular test environments using those import/export mechanisms. For example, using a scripted solution like this you can quickly:

Export users from one test system into a CSV file, along with local phone number contact data
Replace the existing phone numbers with a mix of local and international numbers
Import the new data into a different test system
Verify that the system properly deals with international calling concerns

This example simply trims out all of the rows where the second column of input contains values less than 100. It’s also nice to note that OpenCSV is smart enough to let you ignore one or more ‘header rows’, a common attribute of CSV files.

4. Download, install and run a stand-alone webserver

This example comes courtesy of a recent article by Andrew Glover about Gretty, a lightweight http server. I don’t have a concrete use-case for this right now, but it’s easy to note that in this case you can code a large part of the server behaviour directly in the script, unlike, for instance, if you were to use a script to stand up a Jetty server. This should let you publish a script once in a central location and stand up as many Gretty servers providing the declared services as you like. This particular example simply echos back ‘Hello’ plus whatever path you append to localhost:8080.

5. Remotely execute any of these scripts by url

As of Groovy 1.8.3/1.9-beta-4 you can refer to a Groovy script hosted at a remote url and execute it locally. If you work on different machines a lot(as I do) this can really help keep your toolbox of scripts handy, regardless of where you are. Since these scripts are publicly hosted on GitHub(thanks guys!), you can simply copy the url from the ‘view raw’ link at the bottom and go to your console to execute them like this:

$ groovy https://gist.github.com/raw/1430856/cea0186a862217015fb4fc2b63eb0ad3575c06fc/jenkinsBuildArtifacts.groovy
{
    "actions": [
        {
            "causes": [
                {
                    "shortDescription": "Started by timer"
                }
            ]
        },
        {
            "buildsByBranchName": {
                "origin/master": {
                    "buildNumber": 684,
                    "buildResult": null,
                    "revision": {
                        "SHA1": "3419fee4bb9436d3222ff3df8dd7fcf2308a6919",
                        "branch": [
                            {
                                "SHA1": "3419fee4bb9436d3222ff3df8dd7fcf2308a6919",
                                "name": "origin/master"
                            }
                        ]
                    }
                }
            },
... and so on

Final note

Being able to share scripts across a variety of machines has definitely made my professional life a lot easier and I’m very glad to see the ability to execute code from a URL in Groovy. But what really turned me onto the idea was its usage in Gradle, which supports loading remote Groovy/Gradle scripts using the “apply from: {url or file}” syntax. I find that more and more I reference particular tasks remotely in a build, especially for static analysis tools that don’t necessarily need to be run on a regular basis.
A great example of this is the recently released Gradle Templates Plugin, which is basically the Gradle answer to the Maven Archetype plugin. Create a build.gradle with the following content:

apply from: 'http://launchpad.net/gradle-templates/trunk/1.2/+download/apply.groovy'

And we have the following tasks available to create new Gradle projects and objects with conventional structure:

Template tasks
--------------
createGradlePlugin - Creates a new Gradle Plugin project in a new directory named after your project.
createGroovyClass - Creates a new Groovy class in the current project.
createGroovyProject - Creates a new Gradle Groovy project in a new directory named after your project.
createJavaClass - Creates a new Java class in the current project.
createJavaProject - Creates a new Gradle Java project in a new directory named after your project.
createScalaClass - Creates a new Scala class in the current project.
createScalaObject - Creates a new Scala object in the current project.
createScalaProject - Creates a new Gradle Scala project in a new directory named after your project.
createWebappProject - Creates a new Gradle Webapp project in a new directory named after your project.
exportAllTemplates - Exports all the default template files into the current directory.
exportGroovyTemplates - Exports the default groovy template files into the current directory.
exportJavaTemplates - Exports the default java template files into the current directory.
exportPluginTemplates - Exports the default plugin template files into the current directory.
exportScalaTemplates - Exports the default scala template files into the current directory.
exportWebappTemplates - Exports the default webapp template files into the current directory.
initGradlePlugin - Initializes a new Gradle Plugin project in the current directory.
initGroovyProject - Initializes a new Gradle Groovy project in the current directory.
initJavaProject - Initializes a new Gradle Java project in the current directory.
initScalaProject - Initializes a new Gradle Scala project in the current directory.
initWebappProject - Initializes a new Gradle Webapp project in the current directory.

A Grails App Demoing the StackExchange API

TheKaptain — Thu, 03 Jun 2010 07:12:17 +0000

So I was making an attempt to catch up on my Google Reader‘ing this weekend and I came across this post on StackOverflow regarding the shiny new StackExchange API. That and a fresh 1.3.1 drop of Grails seemed like as good a reason as any to hack together a little app suitable for seeing what you can do with said API.
Turns out it’s not a whole lot, at least not yet. The minor detraction of having read-only access and limited connects without an API key are more than offset by seeing such a minimal, clean and easy to use interface. According to the grails stats script, it took about this much effort to implement calls to each and every one of the API endpoints for all of the supported domains:

    +----------------------+-------+-------+
    | Name                 | Files |  LOC  |
    +----------------------+-------+-------+
    | Controllers          |     1 |    42 |
    | Groovy Helpers       |     1 |    47 |
    +----------------------+-------+-------+
    | Totals               |     2 |    89 |
    +----------------------+-------+-------+

Please note that this is about as minimal as you could reasonably get away with(and kinda ugly to boot), but nevertheless it does manage to implement a UI and backend for exercising the entire API in a total of 2 gsp files and 1 controller action. What’s noticeably missing are some tailored views for each of the different JSON responses and the implementation of additional query parameters on the calls.

StackExchange API

This is really pretty well documented and “consistent”. Calls in some cases require an {id} in the url and that’s about it. Each of the individual calls has its own help page that describes the options, like the badges page for instance. To make things a little more helpful while exploring the API, each of the implemented endpoints in this app are also hyper-linked to the corresponding manual page on api.stackoverflow.com.

RESTful Access

Couldn’t be a whole lot easier than using HttpBuilder, which provides nice and concise ways to execute the GET request, inspect the response and deal with success/failure of the request. Aside from assigning the root domain and method using variables, this is little different from the canonical example on codehaus.

def http = new HTTPBuilder("http://api.$domain")
http.request(GET, JSON) {
    uri.path = "/${VERSION}$method"
    headers.'User-Agent' = 'Mozilla/5.0 Ubuntu/8.10 Firefox/3.0.4'
     response.success = { resp, json ->
        answer = json
    }
     response.failure = { resp ->
        answer = "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
    }
}

The syntax here could probably get even Groovier using the Grails REST client plugin wrappers, and indeed incorporating that plugin is how the HttpBuilder dependency is being provided in this app, but I hit on a working implementation first time out so we’ll leave that for another day, shall we?

What’s it Look Like?

Did I mention it was ugly? Where the API call requires a parameter, you get a text field. Each domain gets its own submit button, officially representing my first attempted usage of g:actionSubmit in Grails. Doesn’t work quite the way I expected, but it does certainly work. Click a button and you’re shown the raw JSON result, along with the call you made. In an ugly fashion, or did I say that already? Still, for a couple of hours of hacking, I’m not unhappy with the result.

StackApps

This is where the apps in the contest get shown off and, in a wonderful display of dogfooding, it’s guess what – another StackOverflow clone! The page for this application can be found here. And the source code is up on github if you’d like to take a look.

Next Steps?

Toying with the idea of porting this to appengine, which would unfortunately mean replacing HttpBuilder and the underlying Apache HttpClient in favor of something that didn’t use a threaded approach, most likely using URL fetch. That and some slightly less ugly tabular and/or tree views of the JSON responses would be kinda nice to have.

Grails-UI DataTable using XML for a model

TheKaptain — Mon, 26 Oct 2009 07:06:05 +0000

I had a great chance to play with some new stuff this weekend, namely Grails 1.2-M3, the new Grails REST plugin and Groovy in general. Awhile back I wrote some Groovy code to explore the various Jira, Fisheye and Crucible REST api’s using Apache HttpClient directly, and I wanted to see what it would like abstracted behind HTTPBuilder. With the help of the Grails-UI and Google-charts plugins, I got some pretty nice views of the data as well.

First off, dealing with a RESTful call was about as simple as I’ve ever seen. This is the entire code that deals with creating the connection, authenticating, executing a GET, and parsing the returned XML into a ‘data’ property on the response.

    private withFisheye(String url, Map args) {
        withRest(uri: url) {
            auth.basic userName, password
            return get(query: args)
        }
    }

Getting XML using Andres’ plugin is extremely simple and we can achieve some pretty nice visualizations of the resulting data very easily as well. In particular I wanted to share how to use an XML response as a backing model for DataTable. The excellent examples provided by Matthew Taylor on his blog seemed to concentrate on using GORM to load and sort domain objects to utilize as data models. When using XML as an alternative way to represent the data, and using RESTful calls to populate that data, the available sorting options on a DataTable weren’t able to function properly without a little extra help.

Consider an list of XML elements like this, which represents a single review in Crucible.


    false
    
        Kelly Robinson
        TheKaptain
    
    20010101T00::00.0000000
    20010101T00::00.0000000
    
        Kelly Robinson
        TheKaptain
    
    
        Test review
    
    
        Kelly Robinson
        TheKaptain
    
    
        Test review
    
    
        CR1
    
    CR
    Closed
    
      Test review

You can leverage this list as a DataTable model by extracting specific elements from the XML, and providing a sorting mechanism based on dynamic map value lookups.

    private def createReviewDataJson(HttpServletResponse response, Map params, reviews) {
        def list = []
        response.setHeader("Cache-Control", "no-store")
       //build a List of Maps, one for each row in the DataTable
        reviews.reviewData.each {
            list << [
                    id: it.permaId.id.text(),
                    author: it.author.displayName.text(),
                    creator: it.creator.displayName.text(),
                    moderator: it.moderator.displayName.text(),
                    description: it.description.text(),
                    state: it.state.text()
            ]
        }
       //sort by the 'order' and 'sort' parameters passed in by DataTable
        switch (params.order) {
            case 'asc':
                list.sort { it."${params.sort}" } //sort by value where key == params.sort
                break
            case 'desc':
                list.sort {a, b -> b."${params.sort}" <=> a."${params.sort}" } //reverse sort by value
                break
        }
       //the entire model
        def data = [totalRecords: list.size(), results: list]
        render data as JSON
    }

This model can then be rendered using the tag. Note that this is pretty much a direct copy of a provided example, nothing fancy here.

So for very little work you get a sortable, paginated DataTable, complete with column resizing and dragging. The DataTable is also being rendered using a template, allowing it to be used arbitrarily by controller methods that populate the model with different reviews. Pretty cool if you ask me. I also used the same backing XML model to drive some PieCharts, but that’s perhaps a topic for another post.