Simulate Apache Spark Workloads Without a Cluster Using FauxSpark

A discrete event simulation of Apache Spark, built with SimPy.

The implementation reflects my understanding of Apache Spark internals. Contributions, feedback, and discussions are welcome!

If you're running Apache Spark at large scale, experimentation can be costly and sometimes impractical. While data analysis can offer insights, I found simulations to be more approachable in understanding system behavior. Surprisingly, they work just fine!

This simulator intends to fill that gap by allowing users to experiment and observe Apache Spark's runtime characteristics such as performance and reliability for different job schedules, cluster configurations, and failure modes.

Like any simulator, the numbers produced here are approximate & may differ from real-world behavior, and are only as accurate as the model. The plan of course is to make the model better 😀

A walkthrough demonstrating how to use this tool is provided below.

FauxSpark currently implements a simplified model of Apache Spark, which includes:

• DAG scheduling with stages, tasks, and dependencies
• Automatic retries on executor or shuffle-fetch failures
• Single-job execution with configurable cluster parameters
• Simple CLI to tweak cluster size, simulate failures, and scaling up executors

Planned enhancements:

• Speculative Task Execution
• Caching in Spark
• Support for multiple concurrent jobs & fair resource sharing
• Modeling different cluster topologies (e.g., for inter-AZ traffic and cost)
• Enhanced reporting
• Accepting RDD graphs / SparkPlans as input

Some stretch goals:

• Modeling Network & Disk IO (e.g., network bandwidth to observe changes in shuffle performance, spills)
• Adaptive Query Execution (AQE) behavior

Consider a straightforward SQL query.

which could be represented using this DAG.

This is a single stage query (no shuffle) reading an input of 1024 MB uniformly distributed across 10 partitions. Let's assume for now a single core can process at a rate of 102.4 MB/s.

Let's run the simulation:

Since the simulator is currently idealized (no network/scheduling delays etc.,) the utilization is 1.0 (100%).

A few more runs:

Two observations:

1. The execution time didn't shrink by half unlike before (5.0 ➜ 3.0)
2. The utilization dropped by ~16%

Observation #1 is not surprising. The total number of tasks wasn't divisible by the number of cores. Looking at the schedule step by step:

• First batch: 4 tasks run in 1s
• Second batch: 4 tasks run in 1s
• Final batch: 2 tasks run in 1s

To understand utilization, we first need to define it. In the simulator, utilization is defined as:

Σ task.runtime / Σ executor.uptime * executor.cores

In our example, the last batch kept 2 cores idle hence the drop in utilization.

The example above didn't really justify a simulator – but neither was the example either!

In practice, jobs:

• process non-uniformly distributed data – often skewed,
• are often quite complex,
• may encounter failures during execution.

Analyzing such situations & planning for it can quickly become challenging. For example, let's consider skews.

This will split our 1024 MB input into 10 randomly sized partitions that follow a Pareto distribution, resulting in a heavily skewed dataset. Far more realistic!

Running the sim:

It's clear that some tasks are handling more data than others. Running the sim several times, it's not clear how to interpret the numbers.

Since our simulation is now stochastic (using random inputs), the outputs will constantly vary. You can't rely on any single run to draw conclusions. In simulations, it's common to run thousands of simulations to form a statistically significant result.

Suppose we anticipate that our dataset will skew over the coming months, and your team wants to plan capacity to minimize wasted cost (1 − utilization) while maintaining a target SLA. A simple way to approach this is using a optimizer function such as this this.

This function takes two inputs

• utilization
• runtime

and performs 10k simulations for each cluster configuration and filters the ones where the p90 of 10k sim runtimes & waste (1-utilization) was below the target SLA and wasted budget. I chose p90 arbitrarily for this example.

Let's be ambitious

pretty printed markdown table from console logs

It's apparent that under skewed conditions, utilization declines quickly. We might have to sacrifice some $$ for the projected skew or simply mitigate skew altogether.

Finally! According to the simulation, a 2 core configuration offers the optimal trade-off, achieving 65% utilization (35% wasted computational power) and a runtime of 7.6s under the expected skew.

By the way, did you notice that despite all the randomness in our simulations, the percentiles consistently converged?!

Randomness is seemingly chaotic, yet inherently consistent