Untapped Potential in the Java Build Tool Experience

The Java language is known to be fast, safe, and easy, but Java build tools like Maven or Gradle don’t always live up to that reputation. This article will explore what “could be”: where current Java build tools fall short in performance, extensibility, and IDE experience, and the reasons to believe that we can do better. We will end with a demonstration of an experimental build tool “Mill” that makes use of these ideas, proving out the idea that Java build tooling has the potential to be much faster and easier to use than it is today.

Build tools like Maven and Gradle have been staples of the JVM ecosystem for decades now. Countless developers have used these tools to build projects large and small, so they clearly work for the tasks they are used for. However, just because something works doesn’t mean that there isn’t room to improve! There are three main areas that these Java build tools typically fall short:

1. IDE Experience
2. Extensibility
3. Performance

We will discuss each one in turn.

IDE Experience

Although build tools like Maven or Gradle have support in all modern IDEs, that support can be surprisingly thin and unhelpful. For example, consider the following snippet in a Maven pom.xml file:

If you already familiar with the various plugins involved and have already memorized how they are configured, this snippet makes perfect sense. However, in the real world things are not always quite so neat, and a developer may not be familiar enough with every single tool or plugin to be able to instantly recall how they are configured and used. Sometimes mistakes are made and the developer has to dig deep to investigate a bug or misbehavior. That is where IDEs come in: they assist the developer by instantly pulling up the documentation or implementation of APIs they may be unfamiliar with, so they can quickly learn what something does and how it can be used to accomplish their goals.

However, if you try to use the Intellij IDE’s jump to definition on this Maven pom.xml snippet, it brings you to this:

This is the signature of the sources configuration value: we now know that sources has the typejava.io.File[], it is required, and it is documented as Additional source directories. If that is all you need then great, but sometimes it isn’t. If I as a developer need to debug an issue in the local build or the upstream plugin, I may need to ask follow ups: how is sources used by the plugin? How do changes in sources end up influencing the behavior of the system, in this case the build tool?

In application code, you can almost always use your IDE’s jump to the definition, find usages, or other code navigation tools to explore and understand the underlying logic, but these tools are conspicuously absent when working with build tool config files. As a result you often end up re-reading the documentation for the Nth time, digging through Stackoverflow, or copy-pasting from other local examples. While this can help, it is much more time-consuming and error-prone than being able to explore the system in your IDE.

This problem is not unique to Maven and it’s pom.xml; Gradle suffers the same problem. For example, let’s say you looked at the following code and weren’t 100% sure what compilerArgs does:

Again, you could probably guess what this does: it configures the compiler. But it is still reasonable to want to learn more:

• Which compiler? Gradle supports Java, Kotlin, Scala, Swift, and other languages!
• What is the default value of this field?
• What was the value of this field before we appended the --add-exports flag to it?
• Does it apply to test code compilation? Or javadoc compilation?

But if you asked your IDE to jump-to-definition to try and learn more about this flag, you will receive a screenful of de-compiled bytecode

Not only is the de-compiled code missing all javadoc and comments and other niceties, but there is a more fundamental problem: the code is a simple getter and setter! All you know is that someone is setting this mutable variable, and someone is getting it, but these could be happening anywhere in the huge Gradle codebase, any of the third-party plugins you use, or anywhere in your own code. Although your IDE nominally lets you jump to definition in your Gradle build files, in practice it often isn’t helpful in figuring out where the values actually come from.

While the Gradle example above is in Groovy, the experience with Gradle Kotlin is similar, despite Kotlin being a language with excellent IDE support in most other scenarios.

The reason there is untapped potential here is that this is not the IDE experience that someone expects from a JVM project! If I open a random Java file in any Java project, I expect to be able to pull up the documentation and signature of any identifier in that file with a single keypress:

And with another keypress, I expect to be able to jump to definition to see where the identifier comes from, and also find usages to see where the identifier is used throughout the program:

In a typical Java setup, this seamless jump-to-definition and find usages works throughout: your own code, third-party libraries, standard libraries, and so on. You can use this to get a deep understanding of any codebase quickly and conveniently in your IDE. But somehow that experience does not translate to build tools, and it is common to find yourself Google-ing for answers, reading and re-reading online docs that are always somehow not quite enough, and digging through plugin source code on Github. Can build tools do better, to really match the seamless IDE experience that Java developers are used to?

Extensibility

All build tools are extensible to some degree: even if 99% of the time you are doing standard things like compiling Java sources into classfiles and packaging them into jars, 1% of the time you actually do need to do something custom and unusual. Let’s consider a simple requirement:

• “Count the number of lines of code in the project and save it in a line-count.txt resource file”

While this requirement may be contrived and arbitrary, it is representative of the many arbitrary things that any real-world project needs. Custom linters, custom deployment artifacts, custom BOM metadata, etc. are all things that real world build systems need to support. “counting the lines and saving it” is just the “hello world” version of these common real-world build customizations

Both Maven and Gradle can be extended to let you do this, but it is non-trivial. For example, in Maven you may come up with:

This snippet uses the exec-maven-plugin to count the lines of code using shell command find src -name '*.java' | xargs wc -l > line-count.txt. This works, but there’s a lot of subtlety and trickiness around it:

1. You have to make sure to use && rather than ;, or set -e, otherwise errors may be silently ignored
2. You have to escape the && as && in XML in order to make it parse correctly
3. find only works on Mac and Linux, so this config as-written won’t work on Windows
4. What happens if some source files live outside of src/, e.g. Generated source files in target/?

You can also implement this line-count logic in Gradle, and it looks something like the following:

This is also tricky, but for different reasons than the Maven XML version. It runs as Kotlin on the JVM, so you don’t need to worry about cross-platform support. But there are other issues:

1. You need to remember to keep sourceDirs in sync with inputs.files. In the example above, the two refer to different folders, which would result in unnecessary re-running of this tasks if e.g. src/main/resources/ changes
2. You need to remember to add the dependsOn clause in processResources, otherwise you will find generateLineCount not re-running when it should resulting in a stale line-count.txt.
3. Again, src/main/java is not the only place where sources live! What if it’s actually src/main/kotlin? What about generated sources in target/

The above code has a bug that can cause the line-count.txt to be spuriously re-computed if some files not in src/main/java are modified. Can you spot it?

In general, the bugs in the Gradle case won’t come from the line count logic: The sourceDirs.map(::file), walkTopDown, totalLines += file.readLines().size section is verbose but straightforward. Rather, bugs would come from how this piece of code integrates with the rest of Gradle: the registration of tasks, registration of task dependencies, registration of input files and output files, all of which are done manually and be easily fat-fingered or fall out of sync as the code changes over time.

What is notable about “count lines and save it to a file” requirement is that it is entirely trivial. Any first-year programming student should be able to write it, and any professional programmer should be able to bang it out in their language of choice in about 30 seconds and have it work flawlessly. But the build tools like Maven or Gradle make this trivial task decidedly non-trivial. That’s not to say it’s impossible, but it’s a lot harder than such a simple task should be!

Performance

The last area to discuss here is performance. Java is a very performant language, and the Java compiler written in Java is also very fast. But no Java build tool seems to be able to surface that speed to the developer.

For example, consider the time taken to clean compile a single module, common, in the 500kLOC Netty codebase

The exact time taken will depend on hardware, OS, and Java versions. But when I ran it above it took about ~6s to compile this common module, which contains ~30kLOC

6s to compile 30kLOC works out to be about ~5k lines per second. If we do another benchmark and try to compile the entire 500kLOC codebase, it takes about ~100s to compile on a single core:

Again, we are looking at Maven compile ~5k lines of Java code per second. These numbers are with all dependencies already downloaded, with the ~/.m2 cache already populated, and with all linters and tests disabled via -DskipTests and -Pfast. We just want to look at compilation speeds.

In isolation that number seems fine: ~6s to compile a single moderately-sized module, ~100s to compile the entire project (this drops to ~50s if we parallelize it with -T10). But it’s worth asking, how fast should Java code compile?

It turns out, Java compiles a lot faster than 5kLOC per second! If we go back to the common module we looked at earlier, and try compiling it directly with javac rather than going through 

Above we are calling javac and passing in the (rather long) classpath along with a source file glob to the javac command line tool. This does effectively the same thing that Maven does internally, but it does it without Maven, and we are seeing a ~4x speedup as a result. This suggests that 3/4 of the time Maven spends on a ./mvnw compile is actually just build tool overhead unrelated to the compilation itself.

But that’s not all: javac is a Java program, and any experienced Java developer knows that running a Java program “cold” from the command line gets you the worst possible performance out of it. Java programs should be kept warm in-memory in a long-lived process so the JVM has time to JIT-compile and optimize the bytecode. If javac run cold from the commandline can compile Java at ~20kLOC/s, we would expect javac kept warm in a long-lived process to be much faster.

To test this, we can run javac in-memory using its javax.tools.* API that is available with most recent versions of the JDK. The Bench.java file below simply runs this over and over in a while loop and prints out the time taken:

This can be run as follows:

If you do this, you will find it start of slow but gradually speed up over time. Running this for about ~30s on my laptop results in the following output near the end:

So while Maven compiles Java code at ~5kLOC/s, and javac run from the command line compiles at ~20kLOC/s, javac run in-memory and allowed to get hot compiles at almost ~120kLOC/s!

If we repeat this exercise with Gradle, using the Mockito codebase as an example, we get similar results, which I tabulated below

So Maven and Gradle actually compile Java code 15-20x slower than javac itself is able to do so! While Netty’s ~500kLOC compiles in ~100s with Maven, it should compile in 4-5s if compiled using Javac directly. Java compiles should be basically instant even for large codebases like Netty, but build tools like Maven or Gradle add enough overhead that you can really feel the slowness.

The Mill Build Tool

Mill is a fast, scalable, multi-language build tool that supports Java, Scala, Kotlin. At its core, it does many of the same things as Maven or Gradle, but tries hard to improve upon the IDE experience, extensibility, and performance issues described above. You define a build.mill file as below:

And you can use this build file to compile, test, run, and generate assemblies of your project:

Mill can build any Java application if configured with the appropriate dependencies. For example, the following build.mill configures a Spring-Boot TodoMVC application with all bells and whistles: data-jpa, thymeleaf, validation, jaxb-api, webjars for the CSS and Javascriptc, as well as unit in-memory tests using H2 and integration tests using Dockerized Postgres in TestContainers:

This can be run locally from the command line:

Or interacted with in the browser:

So far, nothing we have seen here is unusual: these are just things that any Maven or Gradle project can do. So what value does Mill provide as a build tool?

To look at how Mill improves upon things, we will consider the same three areas we looked at earlier: IDE experience, extensibility, and performance:

IDE Experience

In terms of IDE experience, Mill has support in IDEs like IntelliJ or VSCode like Maven or Gradle do, but that support turns out to be much more useful.

Earlier, we looked a Maven and Gradle build, and saw how the IDE experience of exploring the configuration was shallow: it told us what types things were, but not how these configuration values came from or how they were actually used. Mill’s experience is different. For example, consider the following custom task that creates some generated sources:

Right off the bat, you can mouse over the task and see the documentation inherited from the overridden task. If you want to learn more, you can jump to definition to see where the overridden task is defined:

From there, you can see how this generatedSources task ends up being used in allSources, allSourceFiles, and finally in compile:

The Mill build logic is no simpler than Maven or Gradle: it still needs to process the input configuration to decide how to correctly build your project. Where Mill differs is that the build logic is navigate-able in your IDE: you can see how these configuration values, files, and tasks combine together to produce the output artifacts you want. While in the example above we stopped after seeing how def generatedSources feeds into def compile, we can continue to explore the build logic arbitrarily deeply if we desired: looking at the implementation of .findSourceFiles, .compileJava, and so on.

This IDE experience is not new: it is what Java developers everywhere are experiencing every day working on their application code! But Mill is able to bring this IDE experience to your build tool in a way that other build tools like Maven or Gradle cannot, which greatly helps anyone who is trying to debug issues with their build and understand why it is acting the way it does.

Perhaps the key insight here is that Mill tasks and configuration values are just methods! The Mill build is just made of methods calling other methods to form a call-graph, which Mill uses as the foundation for tasks calling other tasks forming a build-graph. And IDEs like IntelliJ or VSCode already know how to deal with method defs, method calls with ()s, and even more advanced features like override and super. As far as your IDE is concerned your Mill build logic looks just like any application codebase made of methods calling each other, and so the IDE can provide the same deep understanding and code navigation that it provides for any Java application codebase.

Extensibility

Most build tools require extensions to be written as plugins: you end up assembling a collection of plugins off of Github of varying quality, maintenance, and fit, and trying to coerce those plugins into doing what you want. Mill is different in that it allows you to just write code and use any Java library from Maven Central to configure your build and do what you want.

In Mill, customizing a module to add a generated line-count.txt resource file looks like this:

One method def to compute the line count, another override def to replace resources, and that’s it. We see the same the business logic here as in the Gradle/Kotlin example earlier, perhaps slightly simplified (the allSourceFiles() method already handles the recursive listing of .java files for us so we don’t need to repeat it here). We can then use ./mill show foo.lineCount to see the task’s return value directly, or ./mill foo.run to run the application code that makes use of the line-count.txt resource file:

What’s notable here is what we don’t see here: doLast, dependsOn, inputs.files, outputs.files, hard-coded references to paths like "src/main" and "src/main/java". With Mill, you just write the business logic of what your build tasks need to do: def lineCount reads the lines of each file and sums them up, override def resources writes out the line-count.txt file and adds a path reference to it to the resource path. All the manual work done in Gradle to register task dependencies, register input-output files, decide when the task will run, etc. is all done automatically for you in Mill. And it is done with full IDE support, so if you’re uncertain about what methods are available or what they do, your IDE is happily to provide real-time autocomplete and documentation to assist you:

Build-time HTML Rendering

Most Java applications use more than just the standard library to do their work, and build systems are no different: you often need custom code-generators, IDL parsers, and other third-party libraries in your build. Most build systems require these libraries to be bundled as plugins, which adds extra indirection and complexity (is there a plugin on Github? Does the plugin do everything I need? Is the plugin well maintained? Do I need to fork it or write my own plugin?) on top of the existing complexity of the underlying library

Mill takes a different approach: you can directly use any third-party Java library as part of your Mill build. This removes a layer of indirection, and allows Java developers to directly use the same Java libraries they are familiar with without needing to first wrap them in a plugin or find an existing wrapper.

For example, let’s say we had a new requirement that the line-count.txt should be rendered as a HTML string. Mill as a build tool does not come with HTML templating libraries built in, but it makes it very easy to import such libraries using the import $ivy syntax:

In this snippet, we import $ivy the org.thymeleaf:thymeleaf:3.1.1.RELEASE library, which immediately makes it available in our build config. We can then directly import org.thymeleaf.TemplateEngine and org.thymeleaf.context.Context and use it just as we would use it in any Java application. In this example, we are using it to pre-render a <h1>{lineCount}</h1> html snippet in line-count.txt for our application to use at runtime. We can then use ./mill show or ./mill foo.run to see the value being generated and used at runtime:

What’s interesting about this example is that there are no plugins here:

• No mill-thymeleaf-plugin to integrate HTML rendering into your build pipeline
• No mill-linecount-plugin to count the lines of code
• No mill-generated-resources plugin to generate resource files that can be read at runtime.

Instead, Mill lets you directly write code to do exactly what you want, using the common open source Java libraries you already know how to use. This democratizes your build so that any Java developer can configure it, rather than being limited to those that hold the title of “build tool expert” or “plugin author”.

Declarative vs Imperative Configuration

The last thing worth mentioning on the topic of extensibility is the idea of “declarative” vs “imperative” builds: Maven using XML is usually thought of as “declarative”, while Gradle is considered “imperative”. But the line between the two is a lot blurrier than people realize. For example, consider again the Maven line-count implementation we saw earlier:

While nominally a “declarative” XML config, it actually contains Bash code that is as imperative as any code ever written:

This imperative Bash code is functionally equivalent to the Mill configuration we saw earlier:

And is also equivalent to part of the Gradle Kotlin code we saw:

The lesson here is that build configuration does have some intrinsic complexity. There are things you want your build tool to do, and you need to somehow tell your build tool to do it. Whether you end up writing those instructions in a Maven Bash snippet, in Mill code, or Gradle Kotlin code, that logic needs to exist somewhere. Just because you wrap your imperative Bash script in a screenful of declarative XML does not make that Bash script any less imperative, or help mitigate the fragile and non-portable nature of the Bash language!

This also calls into question why Gradle code is confusing. Common wisdom says that Gradle code is confusing because it is imperative, but if you look at the Gradle snippet above it is no more complex than the Mill snippet or the Maven bash snippet: a student or junior developer should have no problems writing any of these snippets without error. The error-prone part of the Gradle configuration isn’t the business logic of counting lines of code, but the plumbing: manually registering input files, output files, task dependencies, and so on to make it fit into the Gradle framework:

While anyone can write a line-count program without issue, I would expect most developers to have to spend significant amounts of time Googling and poring over documentation to figure out how to write the snippet above, and even more time to convince themselves they have done it correctly!

Arguably the problem with Gradle isn’t so much that the configuration is code: it is that the configuration is very low-level code that forces the user to manually perform a lot of steps that are both tedious and easy to get wrong. So while Mill’s build config is also code, the fact that it automates away all this tedious boilerplate makes the def lineCount and override def resources Mill implementation easier to read and maintain than both the Maven and Gradle versions.

Performance

Compilation Performance

Mill can compile the same Java module much faster than Maven or Gradle. For example, we can compile the Netty common module we discussed earlier via

Or compile the entire 500kLOC Netty codebase on a single thread via

Below we extend the table we saw earlier, and repeat the Mill measurements on the same Mockito codebase we used to test Gradle. we can see that although Mill only matches the performance of Javac Cold, and still has significant overhead over Javac Hot, it is able to compile the same code much faster than Maven or Gradle:

Mill compiling the entire Netty codebase in ~23s on a single thread still does not live up to the ~4-5s that extrapolating our single-module benchmarks would suggests, but is a significant ~4x improvement over Maven compiling the codebase in ~100s on a single core! Similarly, compiling Netty’s common module in 1.11s is a big improvement over Maven compiling it in 6.15s, even if not quite as fast as Javac Hot compiling it in 0.29s.

If we benchmark a variety of scenarios, we see that the speedup is pretty consistent regardless of whether you run on a single-thread or in parallel, whether you compile the entire codebase or just a single module, and whether you do clean or incremental compiles. In fact, Mill’s performance stands out even more for incremental compiles, where it is able to compile and return in ~0.2s where Gradle takes >1s and Maven several seconds!

Other Performance Features

Compilation speed is just one aspect of a build tool’s performance, and while it may be the easiest to measure, it is by no means the most important. Mill

• –watch and re-run: automatically re-run tests or services when code changes, saving time having to manually click the “run” button or tab back to your terminal to repeat a command
• Parallel Testing: this lets you use all cores on your machine to speed up testing, which makes an enormous difference in today’s multi-core world
• Selective Test Execution: speeds up CI by only running tests affected by code being changed. This can cut hour-long PR validation down to minutes by skipping unrelated tests
• Incremental Assembly Jar Creation: cuts down the creation of large assembly jars from 20s to 1s, which really speeds up workflows that use them (e.g. manual testing, or spark-submit)
• Automatic Parallel Profile Generation: mill automatically generates profile files for every command, that you can load into your chrome://tracing profiler included in any chrome browser to visualize where the time in your build is being spent

Conclusion

In this article, we have looked at the ways in which Java build tools like Maven and Gradle fall short of Java’s reputation as a performant and easy-to-use language with great IDE support. While the performance, usability, and IDE integrations with these build tools are ok, they are not great, and are far from what you would expect working within the application code of any Java codebase.

We then introduced the Mill build tool and how it is able to improve upon the shortcomings of tools like Maven or Gradle: 3-6x faster compiles, easier extensibility, and an IDE experience that allows you to navigate your build logic and as easily as any Java library. While there is still room to improve on all counts, it nevertheless shows a significant step up from tools like Maven or Gradle.

JVM build tools definitely have a lot of room to improve. While Mill takes some baby steps in that direction, there’s a lot more work to be done to give Java developers a build tool experience worthy of the platform it runs on.