Orca Build System

Orca now has a new build system that leverages the Zig toolchain, replacing the old python-based build system. Immediate benefits include:

• “One-click” builds
• Intermediate artifact caching: unlocks faster rebuilds
• Dependency tracking of all files involved in the build: automatic rebuild of invalidated dependencies
• Single compiler for all builds: drops MSVC on Windows, no differences between Apple and Brew versions of clang
• Drops Python requirement: avoids annoying differences with different versions of Python
• Bundled support for WebAssembly: don’t have to manually install compiler runtime files for clang to compile to wasm
• Download Angle and Dawn dependencies instead of building locally: huge speedup in fresh checkout build times
• Improvements to CI: simplifies the build process, runs tests, and ensures sketches don’t regress

Overview

Orca is a project with a lot of moving parts. There are several distinct pieces of the Orca project itself:

• Platform library: exposes a platform-independent programming interface for Orca apps and other Orca libraries.
• Runtime: the executable that hosts and runs Orca apps.
• WASM SDK: the wasm side of the Orca SDK. Note that not all Orca APIs are fully in native - some of them are compiled to wasm.
• WASM libc: a (mostly) complete implementation of libc, including IO, provided for Orca apps.
• CLI Tool: allows users to bundle loose wasm files and data into a standalone package. Also allows easy in-place updates to the latest release.

Additionally, there are several dependencies:

• Angle: cross-platform graphics library exposing an OpenGL-based API. Currently the only option for low-level graphics programming in Orca apps.
• Dawn: a cross-platform graphics library exposing a WebGPU-based API. Currently only used by the platform layer for the high-level drawing API.
• Curl: Supports downloading new releases from GitHub.
• Zlib: Supports extracting compressed releases from GitHub.
• Wasm3: the WebAssembly VM used to load and run wasm apps. Soon to be replaced by a new VM that has tighter integration with the Orca runtime for seamless debugging.
• Harfbuzz (coming soon): industrial-strength text glyph shaping.

Needless to say, wrangling all these components with their individual build needs can get complicated! We needed a cross-platform build system that could handle the complexity without making it too difficult to add new pieces.

Old ’n busted

The old build system was a homegrown python script that Ben Visness initially wrote and other maintainers extended over time. It was relatively simple to read and easy to maintain, especially compared to other build systems (e.g. cmake). However, as the Orca project grew larger and added more artifacts and dependencies, it began to show limitations.

If you wanted to build and run Orca from scratch, you would first have to explicitly build two expensive dependencies, angle and dawn. Next you’d need to build the rest of the Orca project, then manually install it to a particular location that’s in your PATH.

Building angle and dawn in particular is a lengthy process, and can take over an hour. Building the rest of the Orca project takes under a minute, but since orcadev didn’t support intermediate caching, if you made a change to the low-level platform layer, you’d need to rebuild the whole thing again.

There were commands to rebuild specific parts of the Orca project - for example, running ./orcadev build-orca-libc when making changes to the Orca libc. But even that command could take a long time, since it would blindly rebuild all the source files that went into that library.

Included in the Orca repo are a set of samples that demonstrate how to write Orca apps. They show off how to use the Orca, C, and graphics APIs, build them into a wasm module using the Orca libraries, and bundle them into a standalone app. These also serve as important testbeds for ensuring the wasm-side functionality still works while the test suite gets fleshed out. However, iterating with them is a bit painful, since they each have their own individual build script. For example, when making a platform change, you need to:

1. Make the change in Orca src/
2. Build and install to the system directory: ./orcadev build && ./orcadev install
3. cd into the sample directory: cd samples/clock
4. Build the sample: ./build.sh
5. Run the sample: open Clock.app (MacOS) or ./Clock/bin/Clock.exe (Windows)

Additionally, there was no easy way of running all the tests at once - you’d have to manually build and run each test:

Similarly, there was no easy way of building and running the sketches - small, standalone programs that use the Orca platform layer to explore/exercise some feature of the API, such as the vector drawing or UI APIs. Again, you had to cd into every directory and build each one individually. While they aren’t a core part of the user-facing Orca story, it’s nice to have a little code sample that acts as a showcase for how to use some part of the API. But without CI integration, many had bitrotted over time.

Cross-platform toolchain compatibility was also a pain. Apple’s version of clang, shipped with XCode tools, doesn’t support WebAssembly, so you had to install the version shipped with Homebrew. Additionally, because clang doesn’t bundle WebAssembly compiler runtime support by default, you had to manually install the libclang_rt.builtins-wasm32 file from the wasi SDK (https://github.com/WebAssembly/wasi-sdk/releases) for all platforms.

Maintenance of this toolchain was also a pain. At one point GitHub changed the version of clang installed by default on the Windows runners, which removed support for WebAssembly, requiring us to manually install a version that did support it. Fun times!

New hotness

The Zig programming language has been making some waves with it’s compatibility story around C/C++ projects. It has first-class support for building C/C++ source, even if you never run any Zig code. The compiler and build system offer:

• First-class multiplatform and Webassembly target support
• Incremental builds
• Dependency tracking

It promised to solve a lot of the problems with the old build system, so we decided to port it and have been happy with the result.

In the new build system, to build and install from a fresh checkout, you simply run:

This will download any needed dependencies (i.e. Angle and Dawn), generate any needed source files, build all the libraries and exes, and package the SDK to the specified path, all in one line. You don’t need to install any additional toolchains or dependencies manually, other than the Zig compiler (https://ziglang.org/download).

The best part is if you make a change to any file that’s participating in the build graph, only the downstream dependencies of the file are invalidated and rebuilt. For example, editing these files:

• src/graphics/graphics_common.c: rebuilds the platform library and downstream dependencies
• src/graphics/wgsl_shaders/raster.wgsl: regenerates wgpu_renderer_shaders.h and rebuilds the platform library
• src/wasmbind/core_api.json: regenerates the core API C bindings and rebuilds the wasm SDK library
• src/tool/bundle.c: rebuilds the CLI tool

We’re also leveraging the Zig package manager to download and cache Angle and Dawn dependencies to avoid putting the burden of building those on the local user. These dependencies have their own repos and build scripts. We build them and create a release in CI that contains the artifacts for each platform. The build.zig.zon file references the specific release artifact URL hosted on GitHub and saves its’ hash to ensure newly-downloaded archives are well-formed. The result is a ~4MB one-time download to get the platform-specific dependency, instead of 1+ hour builds and additional dependency management to get these libraries to build. This shaves a huge amount of friction off the onboarding process for anyone trying out the project.

Regarding samples, tests, and sketches, these are also now one-click builds:

Below is an image that shows a high-level overview of the build graph:

Build Graph

Wrap-up

Thanks for reading! If you are interested in Orca and want to try it out, there’s been no better time to clone the repo and build it from scratch. ;) If you want to keep following along with development, feel free to subscribe to the newsletter. And finally, if you want to support the project, you can do so via Github Sponsors.

Special thanks to Julian for his great code review suggestions and proofing this post.