SourceFS: A 2h+ Android build becomes a 15M task with a virtual filesystem

Slow Builds and Code Checkouts

Today's connected devices are powered by some of the largest codebases ever developed.

The latest Linux kernel has 40 Million lines of code while the Android AOSP has 140M+, and this is just the foundation. Real-world device codebases are far larger – add the code for hardware support, custom features, services, additional OSs and a smartphone quickly tops 200M lines of code, while an Electric Vehicle exceeds 500M. And these codebases continue to grow throughout a device’s lifetime, with each software update.

Every code checkout pulls millions of files and hundreds of GB, every build runs hundreds of thousands of steps. And because dependency graphs at this scale are imperfect, even a small change can trigger a massive rebuild or, worse, produce an incorrect result. 

The impact: hours of developer time lost every day and $ millions wasted on ever increasing CI compute requirements — or more often, configurations that are not tested due to compute constraints.

The Source File System (SourceFS)

Not another build system. Meet SourceFS – a high-performance virtual file system that delivers unparalleled speedups when checking out and building the Android codebase. It integrates seamlessly into your existing developer workflows and CI, with near-zero migration overhead, and dramatically lowers compute costs. Here’s how it works.‍

At its core, SourceFS virtualizes everything, materializes on demand, and does all of this transparently — so neither you nor the rest of the system ever notice it.

‍

SourceFS accelerates code checkouts by over 10×. It does this by creating a virtual file representation of the entire codebase at checkout and materializing files on demand, instantly, the moment you need them. These virtual files look and behave just like real ones, with the correct permissions, timestamps, and attributes, yet their content appears, as if by magic, only if required.

This eliminates the need to download hundreds of GB of untouched code, drastically reducing disk space requirements. And given that most changes only affect a small fraction of the codebase, most files remain virtual and are never materialized. All this is seamlessly integrated with Git and Repo.

SourceFS accelerates builds by up to 10×. Every build step, when first encountered, is executed in a lightweight sandbox that records all inputs, outputs, and environment. This covers over 99% of the build steps – not just compilation, but linking, packaging, generating docs, and more. As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused; any new or invalidated steps, for example from your local changes, are executed and recorded for future use.

Under the hood, SourceFS packs novel algorithms, advanced virtualization, high-performance caching and replay, efficient sandboxing, and a state-of-the-art backend with near-zero overhead, that is built to scale across your entire organization. Most of this is written in Rust and powered by decades of kernel and operating system advancements and expertise.

Fast Builds, Efficient Storage and Cost Savings

The results of checking out code and building in a SourceFS environment are truly impressive.

Fast checkouts, even when compared to the most optimised standard way of downloading the code, are over 20x faster. The SourceFS code checkout gives developers a working Git tree, and aside from running in a SourceFS backed folder, everything else is the same workflows developers are already used to. 

In addition to speed, a non-obvious superpower is the fact that with SourceFS a codebase takes a small amount of disk space. This is game-changing for device developers, who often need to switch between multiple codebases — one for each device type, and sometimes even for individual device versions.

Fixing large-scale bugs, that affect multiple device codebases, or even just checking out another codebase to see how something is implemented is seamless and similar to working on a small GitHub repository.

Fast builds are what truly makes a difference to developer productivity. With SourceFS builds complete over 9x faster on a regular developer machine. This sets a new standard as it enables developers to get their sword fighting time back and speeds-up the lengthy feedback loop on CI pipelines.

Even compared to other fast build solutions, SourceFS achieves significantly better performance, by being able to replay nearly all the build steps - across all programming languages, compilers and all the other tools used in a device codebase – think packaging, linking, documentation and a lot more. 

‍

‍Cost savings are what makes a difference for organizations. The ability to achieve more with the same compute budget or to reduce the costs where they have grown out of control is what truly matters from a financial standpoint.

To understand the impact SourceFS has on costs, it helps to look at the priorities of fast-paced device organizations, where developers can’t wait for over two hours for a build to finish. In such teams, developers use the most powerful machines available to stay productive.

While powerful machines reduce build times to a more reasonable level, they still perform significantly slower than SourceFS running on a standard machine — and at a much higher cost. Here, SourceFS not only delivers better performance but also enables cost savings of up to 14×.

Alternatives: Why They Fall Short

SourceFS builds on prior efforts. Each approach listed below has its own place, yet in our experience they all fall short when faced with the complexity of a modern device codebase.

Migrating to a new build system (Bazel, Buck2)

SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration.

Moving a device codebase to a new build system is a massive undertaking that even well-resourced teams have abandoned midway. Further still, this complexity multiplies for real device codebases, like electric vehicles, that incorporate multiple operating systems (Yocto, Android, QNX), each with its own bespoke build system.

Using a compiler wrapper (REClient, Goma)

An intermediate step, short of migrating to a new build system, is to use a compiler wrapper that enables caching and replay via a remote-execution protocol. However, wrappers cover only a subset of build actions, so they speed up only part of the build – and don’t help checkouts. They’re also brittle as they rely on parsing command-line flags and correctly inferring inputs/outputs, which can break in unexpected ways.

Next Steps

We are rapidly expanding SourceFS support to more real world device codebases, including support for other operating systems, like Yocto, in parallel with optimising performance even further.

Fast builds and checkouts are the first superpowers in our mission to transform how software for smart devices is developed and maintained – making it simpler, faster, and cost-effective. Stay tuned!