Graham Brooks - Commit History Metrics

For the last few years I have been very interested in what we can learn about our software development processes by looking at the data generated by the tools that we use to manage and deploy our work. This work started the Kuona project (Kuona is the shona word "To see.." http://kuona.io. All the code is open source on github https://github.com/kuona

Initial work focused on creating a dashboard from gathered metrics. Building full stack meant great flexibility but a lot of work and finding time for that work challenging with busy schedules. Over the past couple of weeks I have been working on feeding data from the collectors into elasticsearch and then using kibana to visualise the collected data.

Cutting to the chase heres and example chart:

I am not close enough to the project to draw significant insight from this chart but as an outside observer I can see that the project has been continually active since 2013 in this repository.

Here’s another metric drawn from Spring Boot history:

Putting these metrics together in Kibana is really straightforward. But how do we get the data there?

Setting up the Git Kuona Collectors

Kuona and the collectors are still in development so you will need to build/run from source.

Pre-requisites

Make sure you have an up to date Java Development Kit JDK.
Install Leiningen https://leiningen.org/#install
Clone the git collector locally git clone [email protected]:kuona/git-collector.git

Before running you will need to write a configuration file for the collector to use.

properties.json

{"workspace": "<workspace directory>",
  "git": [{ "url": "[email protected]:spring-projects/spring-framework.git" },
                { "url": "[email protected]:spring-projects/spring-boot.git" }]
}

Change <workspace directory> to some temporary space. On first run the git collector clones the repositories into the workspace. Subsequent runs update the repository before scanning history.

The collector only supports history on a single branch (master). Still need to work out how to handle multiple branches well.

Install ElasticSearch. If you are using a mac brew install elasticseach works nicely
Install Kibana. Again homebrew has a recipe brew install kibana
Make sure Elastic search is running http://localhost:9200
Make sure that Kibana is running http://localhost:5601

Elasticsearch does not have any of or data so we will need to postpone Kibana configuration until we have at least one record.

Running the collector:

In the git-collector directory

Create your properties file (see above)

lein run

Sit back and relax - collecting revision history takes a while.

Once the initial clone is complete records will appear in Elasticsearch so we can configure Kibana to use the generated data:

Once set up the index should look something like this:

It would be great to get feedback on this and the other tools being developed for Kuona. Currently working on a collector for Jenkins https://github.com/kuona/jenkins-collector that works like the git collector but for Jenkins build jobs.

Check out the project https://github.com/kuona. Join the team http://kuona.io https://groups.google.com/forum/#!forum/kuona-dev or check out the slack channel https://kuonadev.slack.com/