A Baby Chaos Monkey for Our Microservices

In complex systems built on a microservice architecture, resilience isn’t just a nice-to-have, it’s essential. One common approach to testing system robustness is Chaos Engineering, where you intentionally introduce failures to see how well your system recovers. Netflix’s Chaos Monkey is a famous example: it randomly kills services in production to ensure systems can handle failure gracefully.

While that level of chaos might be overkill for most teams (especially outside of production), the core idea, testing your system’s response to failure, is invaluable. Inspired by this, we recently added a small but effective piece of middleware to one of our services: a manually triggered route-level failure tool that acts like a “baby Chaos Monkey.”

Our infrastructure is composed of many interdependent microservices. Even small changes can have surprising ripple effects, especially when services rely on each other in subtle ways. This middleware gives us a lightweight, controlled way to simulate partial outages in development and staging, without the risk or randomness of full-blown chaos engineering.

Here’s how it works:

• The middleware can be configured via a dedicated route.
• It only runs in development and staging environments.
• It lets you specify which routes should fail, for how long, and with what status code.
• After a maximum of 5 minutes, the configuration will automatically reset.

The config route accepts the following options:

Once active, the middleware checks each request against the configured forceFailRouteMatch. If the request URL contains that string, it immediately returns the configured forceFailStatus and a response body explaining that the failure was intentional.

It’s a simple trick, but surprisingly powerful. By faking partial outages, we can test how other services respond, whether error handling is working as expected, and where we might need retries, circuit breakers, or clearer alerts.