When More Cores Means Less Speed: Debugging PyTorch with Valgrind on ARM

If you’ve ever tried to debug a PyTorch program on an ARM64 system using Valgrind, you might have stumbled on something really odd: “Why does it take so long?”. And if you’re like us, you would probably try to run it locally, on a Raspberry pi, to see what’s going on… And the madness begins!

TL;DR, as you probably figured out from the title of this post, it’s a counter-intuitive experience: the more cores your machine has, the slower your (torch) code seems to run under Valgrind. Shouldn’t more cores mean more speed? Let’s dive into why that’s not always the case ;)

The background Link to heading

In an effort to improve our testing infrastructure for vAccel and make it more robust, we started cleaning up our examples, unifying the build & test scripts and started adding more elaborate test cases for both the library and the plugins. Valgrind provides a quite decent experience for this, especially to catch multi-arch errors, memory leaks and dangling pointers (something quite common when writing in C :D).

The issue Link to heading

While adding the Valgrind mode of execution in our tests for the vAccel plugins, we noticed something really weird in the Torch case. The test was taking forever!

Figure 1: Build & Test run on amd64

Specifically, while the equivalent amd64 was taking roughly 4 and a half minutes (Figure 1), the arm64 run was taking nearly an hour (53 minutes) – see Figure 2.

Figure 2: Why is it taking sooo long?

Debugging Link to heading

The first thing that came to mind was that there’s something wrong with our infrastructure. We run self-hosted Github runners, with custom container images that support the relevant software components we need for each plugin/case. We run those on our infra, a set of VMs running on top of diverse low-end bare-metal machines, both amd64 and arm64. The arm64 runners run on a couple of Jetson AGX Orins, with 8 cores and 32GB of RAM.

And what’s the first thing to try (especially when debugging on arm64? A Raspberry Pi of course!

So getting the runner container image on a Raspberry Pi 5, with 8GB of RAM, spinning up the container, building the library and the plugin, all took roughly 10 minutes. And we’re ready for the test:

Note: We’ll talk about the suppressions a bit later

The test took roughly 13 minutes. At this point, we were scratching our heads. Why would a high-end Jetson Orin, with way more cores and RAM, perform so much worse under Valgrind than a humble Raspberry Pi? Time to dig deeper into what’s really going on under the hood…

The Surprise Link to heading

When the results came in, the numbers were still striking: the same Valgrind-wrapped Torch test that took almost an hour on our Jetson Orin finished in just 13 minutes on the Raspberry Pi. The Pi, with far less RAM and CPU muscle, still managed to outperform the Orin by a wide margin under these specific conditions.

This result was the definition of counter-intuitive. Everything we know about hardware says the Orin should wipe the floor with the Pi. Yet, here we were, staring at the Pi’s prompt, wondering if we’d missed something obvious.

Digging Deeper: What’s Really Happening? Link to heading

So, what’s going on? Why does a high-end, multi-core ARM system get crushed by a humble Pi in this scenario? The answer lies at the intersection of Valgrind, multi-threaded workloads, and the quirks of the ARM64 ecosystem.

Thread Count: The Double-Edged Sword Link to heading

Modern CPUs, especially high-end ARM chips like the Orin, have lots of cores, and frameworks like PyTorch are eager to use them all. By default, PyTorch will spawn as many threads as it thinks your system can handle, aiming for maximum parallelism.

But Valgrind, which works by instrumenting every memory access and synchronizing thread activity to catch bugs, doesn’t scale gracefully with thread count. In fact:

• Each additional thread multiplies Valgrind’s overhead. More threads mean more context switches, more synchronization, and more internal bookkeeping.
• On platforms where Valgrind’s threading support is less mature (like aarch64), this overhead can balloon out of control.
• On the Raspberry Pi, with its modest core count, PyTorch only spawns a handful of threads. But on the Orin, with many more cores, PyTorch ramps up the thread count—and Valgrind’s overhead explodes.

The ARM64 Valgrind Quirk Link to heading

The arm64 port of Valgrind is still catching up to its amd64 sibling in terms of optimizations and robustness. Some operations, especially those involving threads and memory, are simply slower to emulate and track on arm64. This compounds the thread explosion problem, making high-core-count systems paradoxically slower under Valgrind.

Dealing with library suppressions on arm64 with Valgrind Link to heading

For instance, when running applications that rely on specific libraries under Valgrind on arm64 systems, developers frequently encounter a barrage of memory-related warnings and errors. Many of these issues are not actual bugs in your code, but rather artifacts of how these libraries manage memory internally, or limitations in Valgrind’s emulation on such architectures.

For instance, OpenSSL is known for its custom memory management strategies. It often allocates memory statically or uses platform-specific tricks, which can confuse Valgrind’s memory checker. For example, you might see reports of “still reachable” memory or even “definitely lost” memory at program exit.

In reality, much of this memory is intentionally held for the lifetime of the process—such as global tables or the state for the random number generator. These are not leaks in the conventional sense, but Valgrind will still flag them, especially if you run with strict leak checking enabled.

On arm64 platforms, the situation can be further complicated. Valgrind may not fully emulate every instruction used by the specific library. This can lead to false positives, such as uninitialized value warnings, or even more dramatic errors like SIGILL (illegal instruction) if Valgrind encounters an unsupported operation.

It’s not uncommon to see a flood of warnings that are, in practice, harmless or simply not actionable unless you’re developing for that specific library itself.

To manage this noise and focus on real issues in our application, we use Valgrind’s suppression mechanism. Suppression files allow us to tell Valgrind to ignore specific known issues, so we can zero in on genuine bugs in our own code.

Suppression entries are typically matched by library object names, so on arm64 we use patterns like /usr/lib/aarch64-linux-gnu/libssh.so* or obj:*libc10*.so*, obj:*libtorch*.so*.

An example suppression snippet (valgrind.supp) looks like the following:

It’s important to note that not all problems can be suppressed away. For example, if Valgrind encounters a truly unsupported instruction and throws a SIGILL, a suppression file won’t help; you may need to update Valgrind or avoid that code path. Still, for the majority of benign memory warnings from OpenSSL or Torch, well-crafted suppressions keeps our Valgrind output manageable and meaningful.

Debug Symbol Overhead Link to heading

Another factor: large binaries with lots of debug symbols (common in deep learning stacks) can cause Valgrind to spend an inordinate amount of time just parsing and managing symbol information. The more complex the binary and its dependencies, the longer the startup and runtime overhead. Again, amplified on arm64.

Lessons Learned (and What You Can Do) Link to heading

Limit Thread Count: When running under Valgrind, explicitly set PyTorch to use a single thread OMP_NUM_THREADS=1. This alone can make a world of difference.

Test Small: Use the smallest possible model and dataset for Valgrind runs. Save the big workloads for native or lighter-weight profiling tools.

Expect the Unexpected: Don’t assume that “bigger is better” when debugging with Valgrind – sometimes, less really is more!

Profile Performance Separately: Use Valgrind for correctness and bug-hunting, not for benchmarking or performance profiling.

And here’s the full snippet of the test, on a runner VM on the Jetson Orin, taking less than 6 minutes:

and the actual test in Figure 3, taking 8 minutes, almost 7 times faster than the original execution:

Figure 3: Fixed arm64 valgrind test

Wrapping Up Link to heading

This experience was a great reminder that debugging tools and parallel workloads don’t always play nicely, especially on less mature platforms. Sometimes, the humble Raspberry Pi will leave a high-end chip in the dust, at least when Valgrind is in the mix.

So next time you’re staring at a progress bar that refuses to budge, remember: more cores might just mean more waiting. And don’t be afraid to try your tests on the “little guy” – you might be surprised by what you find.