Managing Complexity: Keeping a Large Java Project on Track

by Tom Copeland
09/10/2003

Software Projects Need Constant Attention

Managing a large software project is a challenge. Code is constantly being changed: bugs surface and are fixed, branches are created and merged — it's no small task to keep things coherent. It's even trickier if the developers are geographically distributed; if a development team keeps their code away from everyone else for a month or two, it's likely that when they check it in you'll enter integration hell, with painful merges, conflicting interfaces, and a bevy of dependency problems.

There's no silver bullet for this problem. However, one practice that may help keep things under control is continuous integration. That sounds nice, since the software has to be integrated at some point — but what does continuous integration mean practically, on a day-to-day basis?

UltraLog is a large Defense Advanced Research Projects Agency (DARPA) project. The purpose of the project is to "extend the open source COUGAAR cognitive agent architecture using a layered, integrated approach with technologies in robustness, security, stability, and scalability." More importantly for this article's purposes, the UltraLog project is written in Java, by developers from over a dozen companies distributed around the United States. We needed something to help avoid integration problems; we needed a status page. So we put together the "Dashboard."

Figure 1. The Dashboard (Click for larger view)

The Dashboard — What's Not on There?

There's lots of stuff related to the UltraLog project that, while important in its own right, doesn't make it onto the Dashboard.

• Bug reports. We store bugs in a Bugzilla installation. While it'd be possible to query the Bugzilla database or screen-scrape the reports, we haven't found it necessary to do so.
• Project charts. Neither Pert nor Gantt; much of the scheduling information is maintained in a Wiki.
• Project documentation. We keep project documentation in a separate document repository. Everyone knows where it is, but it doesn't need to be maintained on the status page.

The Dashboard — What's on There?

So, what does the dashboard show?

• Compile success/failure: every hour, all of the code in the project is recompiled. Being able to check out and compile the code means several things:
  Our source code server is up and serving requests.
  Our dependencies are known and are all available.
  Our build process is well-defined enough to automate.
  All of our code compiles and we know if we're using any deprecated methods. That doesn't mean the code is bug-free, but it's a good start.
• Lines of code, as determined by JavaNCSS.
• Recent Concurrent Versions System (CVS) activity.
• CVS charts and graphs via StatCVS.
• Coding guideline problems, as determined by PMD.
• Duplicate code occurrences, as determined by CPD.
• JUnit test results.
• JavaDoc generation.
• Links to .zip files of the source code and class files from the hourly build.

That's an overview. But if you've put together an hourly build, you know that there are a lot of details involved. Let's look at some of the technical issues we encountered while hooking things together.

Ruby and Ant

First of all, the whole process is driven by Ruby scripts and a Jakarta Ant build template. There's no room here to discuss those products, so I'll summarize by saying that Ruby is an excellent open source scripting language and that Ant is an excellent Java build tool.

The most important item on the Dashboard is compilation success/failure. If we can't compile the code, we can't do much else. So compilation status is displayed in color — a strategy that has added phrases like "[some project] is back in the green" to our lexicon. This allows someone to come to the Dashboard and see at a glance whose code is not integrating cleanly. It's an excellent motivational tool.

Technically, the Ant javac task takes care of the compilation. Once it's done and the report is written to an XML file, the Ruby script that drives the process parses that report, determines success or failure, and counts the number of deprecated methods. This allows a developer to see that while the code may have been compiled, there may be newer methods to use. It helps to reduce the integration load when new releases occur.

Ikko

Ikko is a little templating engine written in Ruby. It's fine for small projects that don't get heavy traffic. In the case of the Dashboard, the web page gets rebuilt once an hour, so template caching is not a requirement.

You can see examples of Ikko's simple operation on the Ikko home page. Here's an example of loading a file and plugging in a couple values:

#!/usr/local/bin/ruby
require 'ikko'
fm=Ikko::FragmentManager.new
fm.base_path="."
puts fm["people.html", {"name"=>"Fred", "age"=>"25"}]

Here's the HTML file for the above snippet:

<html>
<body>
<!--Fragment key="name"--> is <!--Fragment key="age"--> years old.
</body>
</html>

Specify a file name and a Ruby Hash object and you have templated HTML.

JavaNCSS

JavaNCSS provides a command-line interface and an XML output format. We used the Unix find utility to gather up a list of files:

$ find . -name *.java > files.txt

and then executed JavaNCSS with the XML flag:

$ javancss files.txt -xml > report.xml

With the help of the Ruby REXML library, the result can be parsed in a line of Ruby:

ncss = (REXML::Document.new(File.new("report.xml"))).elements["ncss"].text

and then it's plugged into an HTML template for display in the final report page.

This process is representative of how the other items are handled. The Ruby script invokes an Ant target or a command-line tool, a report is generated, and the Ruby script parses the result and plugs it into the HTML page.

Pages: 1, 2

Next Page