Find memory errors in unsafe Rust in production with GWP-ASan and the Scudo hard

This article is a companion to my talk Rust Is Not as Safe as You Think It Is: Improving Safety and Reliability in Rust.

Rust is an increasingly popular systems programming language, partly due to the memory-safety guarantees enforced both at compile-time and at run-time. Memory safety is not enforced in unsafe Rust, but for an application developer like myself, the need to use unsafe Rust is exceedingly rare. However, if I look at the libraries I use from Rust, many of them are in unsafe C or C++, called through Rust’s Foreign Function Interface (FFI).

Rust may be memory safe, but most devs are still calling C and C++ through FFI. Even the cleanest Rust app is only as safe as the legacy it wraps.
— Dale Peterson

In an earlier article entitled Making Unsafe Rust a Little Safer: Tools for Verifying Unsafe Code, Including Libraries in C and C++, I detailed how LLVM sanitizers, like the address sanitizer and the thread sanitizer, can be used to find memory errors in both Rust, and in C and C++ libraries called from Rust. In general, sanitizers are not used in production environments because of their resource and performance overhead. For resource-constrained embedded systems, it can even be difficult to use the sanitizers in test environments.

Unlike the other LLVM sanitizers, GWP-ASan is designed to be used in production environments. It was developed at Google, and is enabled in Chrome, Android, and Google’s server infrastructure. It has detected thousands of bugs, and has since been adopted by many other organizations, including Apple and Meta.

I will describe how GWP-ASan works, and demonstrate how it can be used to find memory errors in C or C++ libraries called from Rust.

Electric Fence Malloc Debugger

GWP-ASan is based on the Electric Fence Malloc Debugger which was invented by Bruce Perens and used at Pixar in the 1980s. Electric Fence detects buffer-overrun errors in heap memory. It works by inserting a virtual memory page after each heap allocation, then making this guard page inaccessible through an mprotect system call to modify the page protection. Any access to a guard page—a read or a write—will immediately cause a segmentation fault, terminating the program at the offending instruction. The programmer can deterministically identify the memory error instead of dealing with latent data corruptions or crashes much later in program execution—notoriously difficult bugs to identify and fix.

The Electric Fence Malloc Debugger guards every memory allocation with an inaccessible virtual memory page. It will result in an immediate segmentation fault if the program reads or writes this page. It is used to detect buffer-overrun heap-memory errors.

Electric Fence can also detect use-after-free errors in heap memory. When memory is deallocated, the virtual page is made inaccessible through another call to mprotect, and the page isn’t reused for a period of time. If the page is accessed, it will result in a segmentation fault at the offending instruction.

When memory is freed, the Electric Fence Malloc Debugger makes the page inaccessible and does not reuse the page for a period of time. Any read or write of this page will result in an immediate segmentation fault, which is used to detect use-after-free heap-memory errors.

GWP-ASan

Electric Fence is extremely effective, but it is also extremely expensive. It requires a lot of memory for the guard pages, and it is slow, because of the extra memory allocations, and due to the additional system call required to modify the page protection for every guard-page allocation and memory deallocation.

GWP-ASan is Electric Fence combined with a sampling-based approach to reduce the overhead. Normally, GWP-ASan calls malloc, but every so often, dictated by the sampling rate, it calls the Electric Fence guarded malloc implemented by the LLVM Scudo Hardened Allocator.

GWP-ASan calls the guarded memory allocation probabalistically based on the samping rate.

GWP-ASan relies on broad deployment and a large sample size to identify memory errors. The sampling rate can be dialed up or down to adjust the performance and memory overhead. In fact, the sampling rate can be set to zero to completely disable GWP-ASan at run-time, which provides a lot of operational flexibility. GWP-ASan is simple, yet brilliantly inspiring engineering.

Find Memory Errors in C++

I will first demonstrate the use of GWP-ASan in C++ with an example straight from the LLVM documentation:

#include <iostream> #include <string> #include <string_view> int main() { std::string s = "Hellooooooooooooooo "; std::string_view sv = s + "World\n"; std::cout << sv; }

The std::string_view references a temporary result that goes out of scope by the time it is dereferenced by std::cout, resulting in a use-after-free error.

The program is compiled with Clang, using the Scudo Hardened Allocator to make GWP-ASan available:

clang++ -std=c++17 -fsanitize=scudo -g buggy_code.cpp -0 buggy_code

GWP-ASan is enabled using the GWP_ASAN_SampleRate parameter and the SCUDO_OPTIONS environment variable. Run the program multiple times with a reasonably frequent sampling rate:

for i in `seq 1 500`; do SCUDO_OPTIONS="GWP_ASAN_SampleRate=100" ./buggy_code > /dev/null; done

Eventually the program will terminate with a segmentation fault and GWP-ASan will report three stack traces:

*** GWP-ASan detected a memory error *** Use After Free at 0xffffa90f4fd0 (0 bytes into a 41-byte allocation at 0xffffa90f4fd0) by thread 1 here: #0 ./buggy_app(+0x1ea04) [0xaaaab7ecea04] #1 ./buggy_app(+0x1ecb4) [0xaaaab7ececb4] #2 linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffffa921d7a0] #3 /lib/aarch64-linux-gnu/libc.so.6(_IO_default_xsputn+0x7c) [0xffffa8cc735c] #4 /lib/aarch64-linux-gnu/libc.so.6(_IO_file_xsputn+0x18c) [0xffffa8cc579c] #5 /lib/aarch64-linux-gnu/libc.so.6(_IO_fwrite+0xc0) [0xffffa8cb9e70] #6 /lib/aarch64-linux-gnu/libstdc++.so.6(_ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_l+0x1a8) [0xffffa8f347b8] #7 ./buggy_app(_ZStlsIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_St17basic_string_viewIS3_S4_E+0x50) [0xaaaab7ee2a84] #8 ./buggy_app(run_cpp_code+0x8c) [0xaaaab7ee2848] #9 ./buggy_app(main+0x1c) [0xaaaab7ee28b4] #10 /lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffffa8c773fc] #11 /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffffa8c774cc] #12 ./buggy_app(_start+0x30) [0xaaaab7eb6cb0] 0xffffa90f4fd0 was deallocated by thread 1 here: #0 ./buggy_app(+0x1e8dc) [0xaaaab7ece8dc] #1 ./buggy_app(+0x1d19c) [0xaaaab7ecd19c] #2 ./buggy_app(+0x1dffc) [0xaaaab7ecdffc] #3 ./buggy_app(run_cpp_code+0x70) [0xaaaab7ee282c] #4 ./buggy_app(main+0x1c) [0xaaaab7ee28b4] #5 /lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffffa8c773fc] #6 /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffffa8c774cc] #7 ./buggy_app(_start+0x30) [0xaaaab7eb6cb0] 0xffffa90f4fd0 was allocated by thread 1 here: #0 ./buggy_app(+0x1e8dc) [0xaaaab7ece8dc] #1 ./buggy_app(+0x1d19c) [0xaaaab7ecd19c] #2 ./buggy_app(+0x1de30) [0xaaaab7ecde30] #3 ./buggy_app(+0x2e498) [0xaaaab7ede498] #4 ./buggy_app(+0x2e0fc) [0xaaaab7ede0fc] #5 ./buggy_app(_Znwm+0x1c) [0xaaaab7ee2640] #6 /lib/aarch64-linux-gnu/libstdc++.so.6(_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_mutateEmmPKcm+0x74) [0xffffa8f43474] #7 /lib/aarch64-linux-gnu/libstdc++.so.6(_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_appendEPKcm+0x78) [0xffffa8f44e38] #8 ./buggy_app(_ZStplIcSt11char_traitsIcESaIcEENSt7__cxx1112basic_stringIT_T0_T1_EERKS8_PKS5_+0x4c) [0xaaaab7ee29d8] #9 ./buggy_app(run_cpp_code+0x4c) [0xaaaab7ee2808] #10 ./buggy_app(main+0x1c) [0xaaaab7ee28b4] #11 /lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffffa8c773fc] #12 /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffffa8c774cc] #13 ./buggy_app(_start+0x30) [0xaaaab7eb6cb0] *** End GWP-ASan report ***

The first stack trace is when the use-after-free is detected, the second stack trace describes how that virtual page was previously deallocated, and the third stack trace describes how that virtual page was originally allocated. Recall, GWP-ASan is used in production to detect rare memory errors that are hard to reproduce and have escaped detection through other means like static analysis, unit testing, fuzz testing, and even the use of other sanitizers for dynamic analysis. The three stack traces may be the only information available, but it should be enough for the programmer to identify the bug.

Find Memory Errors in C and C++ Called From Rust

Now that I've demonstrated how GWP-ASan works, I will describe how to use it from Rust to detect memory errors in unsafe C or C++ called from Rust through FFI.

Start by wrapping the C++ code above in a function that can be called from Rust:

extern "C" { void run_cpp_code() { std::string s = "Hellooooooooooooooo "; std::string_view sv = s + "World\n"; std::cout << sv; } }

This C++ code must then be compiled using a build.rs file:

fn main() { cc::Build::new() .cpp(true) .compiler("clang++") .file("src/cpp_code.cpp") .flag("-std=c++17") .compile("cpp_code"); println!("cargo:rerun-if-changed=src/cpp_code.cpp"); }

Note, this is a standard compilation and there is no compile-time instrumentation, as GWP-ASan only requires link-time dependencies and run-time configuration.

Next, include the Rust crate with bindings for the Scudo Hardened Allocator and enable it as the global allocator in a Rust program that calls the unsafe C++ code:

use scudo::GlobalScudoAllocator; #[global_allocator] static SCUDO_ALLOCATOR: GlobalScudoAllocator = GlobalScudoAllocator; unsafe extern "C" { fn run_cpp_code(); } fn main() { unsafe { run_cpp_code(); } }

Using the RUSTFLAGS environment variable, Rust’s default linker must be overridden to use Clang to link with the Scudo Hardened Allocator:

RUSTFLAGS="-C linker=clang -C link-arg=-fsanitize=scudo" cargo build --release

Finally, the Rust program can be run with GWP-ASan enabled with a sampling rate of 1 so that it detects the use-after-free error on the first execution:

GWP_ASAN_OPTIONS="SampleRate=1" ./rust-scudo

As before, GWP-ASan detects the error and reports three stack traces, one for when the error was detected, one for when the page was freed, and one for when the page was originally allocated:

*** GWP-ASan detected a memory error *** Use After Free at 0xffff93642fd0 (0 bytes into a 41-byte allocation at 0xffff93642fd0) by thread 1 here: #0 ./target/release/rust-scudo(+0x5b718) [0xaaaac97eb718] #1 ./target/release/rust-scudo(+0x5b9c8) [0xaaaac97eb9c8] #2 linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffff937657a0] #3 /lib/aarch64-linux-gnu/libc.so.6(_IO_default_xsputn+0x84) [0xffff932c8a74] #4 /lib/aarch64-linux-gnu/libc.so.6(_IO_file_xsputn+0x124) [0xffff932c6ea4] #5 /lib/aarch64-linux-gnu/libc.so.6(_IO_fwrite+0xc0) [0xffff932bb7b0] #6 /lib/aarch64-linux-gnu/libstdc++.so.6(_ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_l+0x17c) [0xffff935298ac] #7 ./target/release/rust-scudo(run_cpp_code+0x138) [0xaaaac97f9650] #8 ./target/release/rust-scudo(+0x69474) [0xaaaac97f9474] #9 ./target/release/rust-scudo(+0x69278) [0xaaaac97f9278] #10 ./target/release/rust-scudo(+0x69264) [0xaaaac97f9264] #11 ./target/release/rust-scudo(_ZN3std2rt19lang_start_internal17h5e621041f01a4c14E+0x448) [0xaaaac981e354] #12 ./target/release/rust-scudo(main+0x28) [0xaaaac97f950c] #13 /lib/aarch64-linux-gnu/libc.so.6(+0x27740) [0xffff93277740] #14 /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffff93277818] #15 ./target/release/rust-scudo(_start+0x30) [0xaaaac97dfb70] 0xffff93642fd0 was deallocated by thread 1 here: #0 ./target/release/rust-scudo(+0x5b5f0) [0xaaaac97eb5f0] #1 ./target/release/rust-scudo(+0x5a198) [0xaaaac97ea198] #2 ./target/release/rust-scudo(+0x5ad50) [0xaaaac97ead50] #3 ./target/release/rust-scudo(run_cpp_code+0x124) [0xaaaac97f963c] #4 ./target/release/rust-scudo(+0x69474) [0xaaaac97f9474] #5 ./target/release/rust-scudo(+0x69278) [0xaaaac97f9278] #6 ./target/release/rust-scudo(+0x69264) [0xaaaac97f9264] #7 ./target/release/rust-scudo(_ZN3std2rt19lang_start_internal17h5e621041f01a4c14E+0x448) [0xaaaac981e354] #8 ./target/release/rust-scudo(main+0x28) [0xaaaac97f950c] #9 /lib/aarch64-linux-gnu/libc.so.6(+0x27740) [0xffff93277740] #10 /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffff93277818] #11 ./target/release/rust-scudo(_start+0x30) [0xaaaac97dfb70] 0xffff93642fd0 was allocated by thread 1 here: #0 ./target/release/rust-scudo(+0x5b5f0) [0xaaaac97eb5f0] #1 ./target/release/rust-scudo(+0x5a198) [0xaaaac97ea198] #2 ./target/release/rust-scudo(+0x5ac3c) [0xaaaac97eac3c] #3 ./target/release/rust-scudo(+0x66040) [0xaaaac97f6040] #4 ./target/release/rust-scudo(+0x65ca4) [0xaaaac97f5ca4] #5 /lib/aarch64-linux-gnu/libstdc++.so.6(_Znwm+0x1c) [0xffff934a2cac] #6 /lib/aarch64-linux-gnu/libstdc++.so.6(_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_mutateEmmPKcm+0x60) [0xffff93538430] #7 /lib/aarch64-linux-gnu/libstdc++.so.6(_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_appendEPKcm+0x78) [0xffff93539df8] #8 ./target/release/rust-scudo(run_cpp_code+0x110) [0xaaaac97f9628] #9 ./target/release/rust-scudo(+0x69474) [0xaaaac97f9474] #10 ./target/release/rust-scudo(+0x69278) [0xaaaac97f9278] #11 ./target/release/rust-scudo(+0x69264) [0xaaaac97f9264] #12 ./target/release/rust-scudo(_ZN3std2rt19lang_start_internal17h5e621041f01a4c14E+0x448) [0xaaaac981e354] #13 ./target/release/rust-scudo(main+0x28) [0xaaaac97f950c] #14 /lib/aarch64-linux-gnu/libc.so.6(+0x27740) [0xffff93277740] #15 /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffff93277818] #16 ./target/release/rust-scudo(_start+0x30) [0xaaaac97dfb70] *** End GWP-ASan report ***

Open Questions

First, GWP-ASan should be able to detect memory errors in unsafe Rust, code that never calls C or C++, as long as the memory allocations use malloc and are managed by the Scudo Hardended Allocator. But so far, I have been unable to construct a demonstrative example. GWP-ASan would be valuable for detecting rare memory errors in Rust crates, or in the Rust standard library where approximately twenty percent of functions use some unsafe Rust.

Second, the Scudo memory allocator strikes a balance between security and performance. Replacing the memory allocator for a memory-optimized application, like a database or a database query optimizer, may impact the performance characteristics, but I’m not aware of any data on this subject. I’m curious about the performance impacts of using Scudo Hardended Allocator with a library like SQLite, which is in C, or DuckDB, which is in C++.

Summary

GWP-ASan finds rare memory errors in C and C++ called from Rust by using it continuously, at scale, in production, for critical software, errors that escape detection by other means, like static analysis, testing, or other sanitizers. GWP-ASan doesn’t require changes to C or C++ code or compile-time instrumentation, only a link-time dependency on the Scudo Hardended Allocator. The performance impact and risks can be dialed up or down by adjusting the sampling rate at run-time. Give GWP-ASan a try to improve the safety of C and C++ called from Rust in your production applications.

Thanks to Kostya Serebryany for input on this article.