Show HN: Gosim Distributed Systems Testing Framework Written in Go

Gosim is a simulation testing framework that aims to make it easier to write reliable distributed systems code in go. Gosim takes standard go programs and tests and runs them with a simulated network, filesystem, multiple machines, and more.

In Gosim's simulated environment you can run an entire distributed system in a single go process, without having to set up any real servers. The simulation can also introduce all kinds of problems, such as flaky networks, restarting machines, or lost writes on a filesystem, to test that the program survive those problems before they happen in real life.

An interesting and entertaining introduction to simulation testing is the talk Testing Distributed Systems w/ Deterministic Simulation from Strangeloop 2014.

Gosim is an experimental project. Feedback, suggestions, and ideas are all very welcome. The underlying design feels quite solid, but the implemented and simulated APIs are still limited: files and directories work, but no permissions or links; TCP works, but not UDP; IP addresses work, but not hostnames. The API can and will change. Now that those warnings are out of the way, please take a look at what gosim can do.

Gosim tests standard go code and is used just like another go package. To get started with gosim, you need to import it in your module:

To test code with Gosim, write a small a test in a file simple_test.go and then run it using go test:

The test prints that Gosim is not enabled, and each time the test runs the random number is different.

To run this test with gosim instead, replace go test with go run github.com/jellevandenhooff/gosim/cmd/gosim test:

The gosim test command has flags similar to go test. The test output is more involved than a normal go test. Every log line includes the simulated machine and the goroutine that invoked the log to help debug tests running on multiple machines.

The === RUN line includes the test's seed. Each time this test runs with the same seed it will output the same random number. To test with different seeds, you can pass ranges of seeds to gosim test:

If running gosim fails with errors about missing go.sum entries, run go mod tidy to update your go.sum file.

Tests running in Gosim run inside Gosim's simulation environment. In the simulation, tests can create simulated machines that can talk to each other over a simulated network, crash and restart machine, introduce latency, and more. The following longer example shows some of those features:

This example contains a HTTP server running on its own gosim machine, and a client that makes several requests to the server. Then the client messes with the server, restarting it and adding latency. Let's see what happens this test is run:

In the log, each line is annotated with the machine and goroutine that wrote the log. When debugging code running on multiple machines, it is helpful to know where logs are coming from.

The first thing that happens is the server starting at step 1. Then the client makes a request at step 2, the server receives it, responds, and the client prints the response. When the client makes another request the server this time responds with a higher count at steps 6 and 7.

Afterwards, the client restarts the server at step 8 and the server prints that is starting again at step 9. When the client makes a request after the restart the server responds once again with a count of 1 at steps 11 and 12. How did that happen? After the server is restarted, the counter resets to zero. Each machine has its own copy of global variables which get re-initialized when a machine crashes.

Notice that the client address (after "got a request from:" in the logs) also changes after the restart, and the goroutine handling the requests on the server changes. The simulated TCP connection breaks when the machine crashes and the net/http client makes a new connection afterwards.

After the restart, the client calls a Gosim API to add latency to the simulated network at step 13. The earlier requests all got sent and received at the same timestamp, such as on steps 10, 11, and 12 all at 14:10:05. The request after adding latency is sent at 14:10:05 on step 14 and received a second later at 14:10:06 on step 15. This is the added latency in action.

Even though the test takes over 4 simulated seconds, running the tests takes less than a second. Simulated time can be sped up when Gosim knows that all goroutines are waiting for time advance. With simulated time, tests can use realistic latency and timeouts without development time becoming slow.

Gosim's simulation is deterministic, which means that running a test twice will result in the exact same behavior, from randomly generated numbers to goroutine concurrency interleavings. That is useful because it means that if we see an interesting log after running the program but do not fully understand why that log printed a certain value, we can re-run the program and see exactly what happened at the moment of printing.

The numbers at the start of each log are Gosim's step numbers. Take step 11 from the log above, from the line "got a request ...". We can debug the state of the program at the time the log was printed with gosim debug:

The gosim debug command runs the requested test inside of the Delve go debugger and stops the test at the requested step. The debugger here puts us in Gosim's log/slog handler which calls gosimruntime.Step() and includes the step number.

Now we can debug the program as a standard Go program. To see what was happening in the HTTP handler that called print, we run backtrace

and then select the HTTP handler's stack frame that called log.Printf

Now we can inspect the state of memory to see details that the log line might have missed

Gosim's simulation is at the system call level. To see why a program behaved the way it did, Gosim can output detailed logs of all system calls with the -simtrace=syscall flag:

Here you can see on step 60 the HTTP GET request being written by the client, and then on step 64 being read by the server.

Gosim can run some sizeable real-world code. In this repository there are examples for:

• examples/bolt: reading and writing a Bolt database

• examples/etcd: running a 3-node Etcd cluster and partitioning the nodes

A description of Gosim's architecture and design decisions is in docs/design.md.

Gosim's public packages are:

• github.com/jellevandenhooff/gosim: the core API for creating machines and manipulating the simulation environment

• github.com/jellevandenhooff/gosim/cmd/gosim: the CLI for running Gosim tests

• github.com/jellevandenhooff/gosim/metatesting: a package for running Gosim tests inside of normal go test.

• github.com/jellevandenhooff/gosim/nemesis: a package, still sparse, to introduce chaos into simulations

Gosim and its tools are tested using the ./test.sh script, which invokes the tests defined in ./Taskfile.yml. These test the Gosim tooling as well as the behavior of simulated code. Ideally tests verify that the simulation behaves like reality: The tests for the filesystem in github.com/jellevandenhooff/gosim/internal/tests/behavior/disk_test.go are run in both simulated and non-simulated builds.

Gosim simulates Linux, but should work on macOS, Linux, and Windows (not tested on Windows) running on either arm64 or amd64. To test that code works on Linux amd64 and arm64, there are scripts .ci/crossarch-tests/test-amd64.sh and .ci/crossarch-tests/test-arm64.sh that run the tests using Docker. A Github Action runs the tests on Linux amd64.