.png)
1 day ago
3
I built a simple web crawler using good old platform threads. It was just a multithreaded crawler, nothing fancy. But then, curiosity struck: “What happens if I use Virtual Threads instead?”
Virtual Threads are one of my favorite recent additions to the Java ecosystem. Switching from platform threads to Virtual Threads dramatically improved the URL processing rate… until the whole thing blew up with an OutOfMemoryError.
Yep. Fast became too fast.
I love exploring Java’s new features. Virtual Threads, Records, Pattern Matching, you name it. But every tool has tradeoffs. In software engineering, solving one problem can introduce another. Our job isn’t just writing code; it’s balancing performance, safety, and maintainability.
This post is about my little hacking session, where I tried to unleash the power of Virtual Threads, only to discover it takes a bit of finesse to avoid turning performance into a memory bomb. I’ll walk you through what happened, how I fixed it, and how you can too.
Before playing with Virtual Threads, I created a very basic crawler using traditional platform threads. The list of URLs to process was predetermined and inserted into an executor queue. Each platform thread would take a new task from the queue, which consisted of: getting a URL, fetching its content from a local server (to eliminate bandwidth and external rate limiting), emulating some processing, and then moving on to the next one.
public class PlatformThreadsCrawler { //the executor service used to fetch and process data
private final ExecutorService executorService = Executors.newFixedThreadPool(200);
private final HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.build();
private final AtomicLong processedCount = new AtomicLong(0);
//submit the job in the executor queue and then wait for results
public void crawlUrls(List<String> urls) {
long startTime = System.currentTimeMillis();
// Submit all download tasks
CompletableFuture<?>[] futures = new CompletableFuture[urls.size()];
int index = 0;
for (String url : urls) {
futures[index++] = CompletableFuture.runAsync(
() -> downloadAndProcess(url),
executorService
);
}
// Wait for all to complete
CompletableFuture.allOf(futures).join();
long endTime = System.currentTimeMillis();
System.out.println("\n=== Crawl Complete ===");
System.out.println("Time taken: " + (endTime - startTime) + "ms");
}
private void downloadAndProcess(String url) {
try {
//download
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.timeout(Duration.ofSeconds(10))
.build();
HttpResponse<byte[]> response = httpClient.send(request,
HttpResponse.BodyHandlers.ofByteArray());
//if ok process the data
if (response.statusCode() == 200) {
byte[] content = response.body();
// Simulate some processing work
int result = process(content);
} else {
System.err.println("HTTP " + response.statusCode() + " for: " + url);
}
} catch (Exception e) {
System.err.println("Error downloading " + url + ": " + e.getMessage());
} finally {
long count = processedCount.incrementAndGet();
if (count % 100 == 0) {
System.out.println("Processed " + count + " URLs");
}
}
}
private static int process(byte[] content) {
//emulate some processing logic (omitted here)
}
}
I submitted a list of 20K urls, to be processed by 200 platform threads. For testing, I used a local HTTP server with static routes to simulate pages. I generated the urls starting from this list, and then duplicating it. The goal was to emulate small and medium file download (like simple html and small images).
urls.addAll(List.of("http://localhost:8080/data/1kb", // 1KB
"http://localhost:8080/data/10kb", // 10KB
"http://localhost:8080/data/100kb", // 100KB
"http://localhost:8080/data/1mb" // 1MB
));
The Java heap max size was 1GB, to emulate a scenario in which memory available was not “infinite”. I emulated the file processing with a simple word count logic. Here you can see some stats captured with VisualVM:
Since a large portion of thread time is spent blocked receiving network data, Virtual Threads should theoretically allow us to handle many more concurrent operations without the overhead of platform threads. Time to make it faster!
I swapped out my
Executors.newFixedThreadPool(200)with
Executors.newVirtualThreadPerTaskExecutor()As you can see in Virtual Threads there isn’t a direct built-in mechanism to set a global limit on the total number of virtual threads that can be created (Virtual threads are designed to be lightweight and numerous).
I ran the exact same logic. Boom! pages were being fetched in what felt like milliseconds. I was thrilled!
Until the JVM gave up with an OutOfMemoryError!
Let’s break it down:
- Virtual Threads removed the I/O bottleneck.
- As a result, URLs were fetched in parallel at a much higher rate.
- But processing (e.g., parsing and consuming the response) didn’t get the same speed boost.
- Without any back-pressure, the program kept stuffing memory with pending results.
It wasn’t just a faster crawler, it became a hyperactive downloader with no brakes!
(Note that I voluntarily limited memory in both crawler versions to easily compare their memory usage. In this particular scenario Virtual Threads required more memory. This might not be a problem in a real world application depending on heap size and on the kind of business logic.)
So how do we fix this without giving up on Virtual Threads?
Limit concurrency using a semaphore
We can introduce a Semaphore to limit the number of concurrent tasks in flight:
public class ControlledVirtualThreadsCrawler {private final ExecutorService executorService =
Executors.newVirtualThreadPerTaskExecutor();
private final Semaphore concurrencyLimit = new Semaphore(500);
private void downloadAndProcess(String url) {
concurrencyLimit.acquire();
try {
// ... existing download and process logic
} finally {
concurrencyLimit.release();
}
}
}
Before launching a new task, acquire a permit. Release it after processing completes. If no permits are available then the Virtual Thread is blocked. This keeps Virtual Threads under control, preventing too many urls to be downloaded and processed at the same time.
Avoid submitting too many tasks at the same time
In our test scenario, we submitted all 10,000 URLs at once, an artificial burst that rarely happens in production. In realistic applications, work arrives continuously over time.
Implementing rate limiting or spreading the arrival of scraping requests over time might prevent overwhelming the crawler.
This experience taught me that Virtual threads aren’t just faster platform threads, they fundamentally change how we think about concurrency limits and resource management. The traditional patterns and assumptions that worked with platform threads may not apply.
Virtual threads are incredibly powerful, but they require us to be more explicit about resource management. The JVM’s natural resource constraints (like thread limits) that previously acted as implicit backpressure mechanisms are no longer there to save us from ourselves.
