Understanding Apple Debug Info

Apple platforms (macOS, iOS, etc), and specifically Mach-O binaries, have a slightly different approach to debug info than ELF binaries for Linux. If you are familiar with Xcode, you might have seen a few related settings that control what is produced and wondered what the trade-offs are. The goal of this post is to help you debug cases where these differences lead to a degraded debugging experience in lldb so that you can fix them.

If you have a particularly complex build, potentially managed by Bazel¹ or another tool, especially if you are using distributed builds, you are even more likely to hit issues.

Let's dive in to how the pieces fit together.

Debug info is metadata produced by the compiler that is consumed by debuggers (like lldb), profilers, and other tools. It is used to map runtime information, like addresses, function arguments, and stack traces, back to the source that was used to produce the binary. Without this information debugging in lldb shows primarily raw instructions and addresses, which is rarely acceptable for common debugging workflows.

When building for Apple platforms debug info isn't contained in the final binary (this is the primary difference from the default Linux workflows). Instead the binary contains references to the files where lldb can find it (this is conceptually similar to if you use -gsplit-dwarf on Linux).

Let's inspect some binaries to see what this really means. First we create a small binary:

If we attempt to inspect the debug info contained in main, we find nothing:

However when we debug this binary in lldb, you will correctly see the source file and line number information:

Let's trace back how lldb discovers the debug info. First the main binary has a reference to the intermediate main.o object file. We can see the references by looking for the N_OSO entries in the binary:

Then lldb loads the debug info directly from the main.o object file. We can see the debug info it contains:

We can also see the object file contains specific __debug* sections when debug info is present that are absent in our main binary:

The same pattern holds when you depend on static libraries:

This is almost everything we need to know for where to find debug info, so now let's try to understand where this can go wrong.

1. Invalid absolute paths

So far we've been looking at an isolated example outside of Bazel. If you've debugged Bazel issues before you may have rightfully felt some alarm bells go off when you saw the absolute paths in the debug info. In Bazel, if we are not being careful, those absolute paths would point to ephemeral locations that only exist during the Bazel link action, and not exist during debug time. This can also be the case if you are using distributed builds with other tools. For example if I break apple_support today, this is what you would see:

In this case you can see the .../darwin-sandbox/4 directory is where the final binary recorded the object file existed during link time. While this was valid during the link action, these directories are wiped after the build completes, so if you try to dwarfdump this file afterwards, you will see it doesn't exist.

You could potentially workaround this issue with --sandbox_debug, which will keep those directories around, but instead of doing that we use the -oso_prefix linker argument. This argument tells the linker to strip the given prefix from the final N_OSO entries, and replace it with . so that it is relative to the current directory. With this applied, the N_OSO entries look like this:

This path now exists relative to the root of the repository, so as long as you are launching lldb from the root of the repository, it will correctly discover this object file.

Now that these paths are reproducible and valid, let's shift our attention to the paths in the object file itself (again by breaking apple_support):

Here we can see 3 different absolute paths which might affect our ability to debug, and undoubtedly affects the reproducibility of our build. In this case if I debug the binary, while it can find the debug info, it cannot find the source files to display inline when it hits a breakpoint:

Notably lldb knows that the breakpoint is set at main.c:2:3 which is pulled from the debug info, even though the source file cannot be found to be displayed. If it didn't have the debug info at all, that output would be subtly different and instead show no specific file location:

To solve this clang has a few different arguments, similar to -oso_prefix, to rewrite the absolute paths in the object files. The most modern and comprehensive version of this flag is -ffile-compilation-dir=.. This flag remaps all known paths that can be embedded in the binary (not only debug info, but also coverage info, and __FILE__ macros, which otherwise won't be covered in this post) to be relative to the current directory.

Once we apply this flag, we see the source file paths are fixed:

The DW_AT_LLVM_sysroot path is still absolute, but this likely doesn't affect debugging since lldb rediscovers the SDK if this path is invalid. In this case for reproducibility in Bazel we still rewrite this to:

By passing -fdebug-prefix-map=$DEVELOPER_DIR=/PLACEHOLDER_DEVELOPER_DIR, which works similar to -ffile-compilation-dir but instead affects a single prefix, not only relative paths.

With these 3 flags applied, our final binary is debuggable, and reproducible!

2. Missing object or source Files

Since we're relying on lldb reading the N_OSO entries in our final binary, tracing those back to the object files, and then finally tracing those back to the source files, all of those files must exist on the machine running lldb at the time you are debugging the binary.

For simple local builds, this should always be the case, but in Bazel there are a few potential times where you would break this assumption.

2.1. Pulling from the Bazel cache

If you are using a remote cache or remote execution in Bazel, along with --remote_download_toplevel (also known as BwtB or Builds Without The Bytes which is enabled by default in modern Bazel versions), then Bazel will avoid downloading as much as possible to speed up build time.

In this case if you didn't compile something locally, the intermediate files may never be downloaded to your local machine, and therefore might be missing when you go to debug. This can be pretty subtle since if you do make changes that result in a local recompile (specifically if you are only using the remote cache, or are using dynamic mode with remote execution), the intermediate object files will be present and your ability to debug will appear to be flaky.

To fix this you can instruct Bazel to always download the intermediate files with something like this in your .bazelrc:

2.2. Bazel external symlink

Another subtle case in Bazel is that while all first party code is valid with the flags above, third party code is compiled from a separate directory (since it isn't in your source tree), and therefore requires more paths to be available at debug time.

For example building a binary that depends on boringssl has valid N_OSO paths:

But if you inspect that archive's debug info, you see the source file paths are nested under a directory called external:

Unfortunately this path doesn't exist in the root of your repository like bazel-out does. The common workaround to solve this is to create a symlink in the root of your repository manually with:

Then you can validate that this path exists so that lldb can find it:

If you don't often debug third party code in your project you can choose to skip this, but it is also required for producing valid compile_commands.json files for IDEs, so you might want it for that too. You should add this symlink to your gitignore after creating it.

Currently only apple_support handles all the flags above for Apple platforms. That isn't an inherent limitation of rules_cc or toolchains_llvm (which uses the config from rules_cc), but would require some work to implement. Because of this you have to make sure that you are getting the correct CC toolchain when building for everything to work correctly.

If you think Bazel isn't choosing the correct toolchain you can pass --toolchain_resolution_debug=".*" while building and you should see a line like this:

Where the important part is that the toolchain on the right hand side is from @apple_support, if it instead looks like this:

You are getting the toolchain from @rules_cc.

When using bzlmod you must be sure to have apple_support above rules_cc in your MODULE.bazel file to ensure this happens (this won't affect non-Apple platforms):

4. Bazel producing no debug info

By default only the dbg compilation mode produces debug info. Therefore you must make sure to pass --compilation_mode=dbg in order to get debug info. Otherwise the final binary will have no N_OSO entries at all, which you can verify with dsymutil -s <binary>. Alternatively you could manually pass --copt=-g in your build.

5. Bazel stripping produced debug info

If you pass --strip=always or --strip=sometimes (the default) to Bazel, it will strip the N_OSO references from the produced binary. Therefore you must make sure you don't do this in the cases where you want debug info. Bazel also attempts to warn you in this case:

By default Bazel doesn't produce debug info for tools built in the exec configuration (because the default exec compilation mode is opt). Instead of debugging a tool built for the exec configuration, you should likely rebuild it directly (for the target configuration) and debug that instead. This issue most commonly happens if you are debugging a bazel rule and copy a command from Bazel's --subcommands output. You could also pass --host_copt=-g if building the target directly doesn't work for you.

7. Temporary files

If you use a single clang command to both compile and link a binary, the intermediate object files referenced by the N_OSO entries are temporary and deleted after the build:

In this case you should likely split these commands apart to ensure the object files are kept around. Alternatively as a quick workaround you can pass -save-temps to clang to keep the intermediate object file around.

An entirely separate option for how to produce debug info for Apple platforms is to use dSYM bundles. A dSYM is created from a linked binary and accumulates the debug info for all transitive dependencies, so you don't have to keep the other intermediate files around.

All of the same concerns stated above around absolute paths still apply, but this can potentially simplify downloading intermediates or redistributing debug info alongside a binary.

You can produce a dSYM from a binary with:

If you are missing intermediates when creating a dSYM, you will see relevant warnings (I would love a dsymutil flag to be able to make this an error):

After you have the dSYM you can delete the intermediate files and still debug as expected.

Bazel can automatically produce dSYMs for you if you pass --apple_generate_dsym --output_groups=+dsyms when building (with any --compilation_mode, but the --strip concerns above still apply). Previously this wasn't supported for cc_binary targets but that has recently been added in Bazel.

In Bazel, after producing the dSYM, the N_OSO entries in the binary are stripped, and all debug info can be inspected in the dSYM:

Since there are no N_OSO entries in the binary, lldb uses a different mechanism to discover the dSYM. This is done by looking up the UUID of the binary, and then looking for a matching dSYM. You can inspect the UUIDs to make sure you have the appropriate dSYM locally:

If lldb isn't correctly discovering the dSYM, you can manually add it with:

You can also verify that lldb is correctly discovering the dSYM by enabling the host logging by adding this to your ~/.lldbinit file:

After running lldb you will see that log should be populated with a line like:

If the dSYM isn't found, there will be no related log entry. If you are using other macOS tools like Instruments.app and it cannot find the dSYM, you should try using the symbolscache CLI to manually add it. See symbolscache --help for usage.

Generally the downside of using dSYMs is the (potentially small) amount of time it takes to produce them, especially if you don't end up debugging the binary. On the other hand if you use a crash reporting service, you likely need to produce them anyway to symbolicate crash reports, so it might be worth unifying the debug workflows.

Another benefit of using dSYMs is that you can fully strip the final binary and still debug it. This is useful for making sure to produce the smallest possible binary for release and test the same artifacts so you know that the stripped binary works correctly (primarily risky if you are using aggressive strip flags to heavily optimize binary size).

In Bazel, because producing the dSYM requires all the intermediate files from the entire transitive dependency tree of the binary, it must be done at the same time as the linking of the binary. This is done with custom logic in apple_support's CC toolchain, which doesn't exist in rules_cc. You cannot run dsymutil manually on a Bazel binary unless you have also downloaded all the intermediate files.

Launching lldb from other directories

Many of the recommendations above are about using relative paths for the files that lldb is attempting to load. This works great as long as you always launch lldb from the same directory (usually the root of your repository). In some cases you might want to launch lldb from another directory (especially in Bazel where runfiles discovery might require a different PWD). In this case you can set the processes' working directory separately from lldb's working directory by still launching lldb from the root of the repository (so that the relative debug info paths are still valid), but then passing the other directory as a setting:

Or once you've launched lldb itself:

Partially missing debug info

If you are linking pre-built artifacts, such as 3rd party static libraries, it's possible they are built without debug info. In this case if you are debugging and stop in their code, you might see no useful information. You can use the commands outlined above to try and understand if they have any information at all, but it is common that closed source libraries ship stripped of debug info. If their code is open source it's possible that you could pull it locally in order to debug it, even while using the prebuilt artifact (if you attempt this make sure to use the same sha the prebuilt artifact is built from).

lldb source maps

If you have prebuilt artifacts as outlined above which have absolute paths in their debug info that you cannot change, you can attempt to use lldb's source mapping feature to rewrite those paths at debug time to something that exists locally. In that case once you've identified the path you need to rewrite by inspecting the binary with dwarfdump, you can add a mapping in lldb like this:

You can have multiple of these paths which can be useful if you have multiple prebuilt artifacts with different debug locations:

You could also use this instead of the clang flags, or the external symlink mentioned above, but often this is harder to manage since the paths could differ between different developer's machines and requires injecting lldb settings.

Note that -oso_prefix is not affected by these mappings, and there can only be 1 of them. If this is an issue for you, you could potentially create your own symlink tree to solve this. I imagine it would also be possible to rewrite the N_OSO entries in the binary but I don't think any of the related tools do this today.

If you are debugging a binary that depends on shared libraries (dylibs), the same principles that apply to the main binary apply to every individual shared library. The shared libraries will have their own N_OSO entries that must be valid, and their own dSYMs (if you go that route). In this case the main binary won't have N_OSO entries for the shared libraries (or their transitive dependencies), but instead lldb will discover them as they are loaded by dyld.

In lldb you can see all of the loaded shared libraries with image list (the first field is the same UUID discussed above):

Symlinks versus actual paths

In the past there have been multiple bugs around how -oso_prefix and -ffile-compilation-dir handle symlinks in paths. If your build heavily uses symlinks, as Bazel does, and you're getting unexpected absolute path behavior, this is something to consider. The typical bugs arise from whether the path is remapped before or after the symlinks are resolved.

Hopefully all this information helps you the next time you need to debug something related to debug info!

If you have any questions or feedback please reach out to me on the Bazel slack or Mastodon. Feel free to cc me on any relevant Bazel issues.