The Journey to Gits: Building an In-Memory Graph Database in Go

1. The Humble Beginnings - "MERDE" (Memory Entity Relation Database Experiment)

So, where did this whole journey begin? Well, about a decade ago, I was dealing with the concept of a specific software architecture I wanted to build. The problem? Traditional relational databases just weren't cutting it for the kind of complex relationships I needed to model. Especially the need to build relations and create new types on the fly wasn't satisfied. It quickly became clear that a Directed Graph storage was the ideal fit for this architecture.

Back then, I dove into the world of graph databases, specifically trying out OrientDB. I managed to get a first prototype of my architecture up and running using it, which was a win in itself. But honestly, I wasn't really satisfied with the result. The API felt tedious, working with it wasn't as straightforward as I'd hoped, and the administration side of things was kinda pain. It worked, but it didn't satisfy me, you know? Plus, I had some specific ideas that I just couldn't realize using OrientDB's existing structure.

Around that same time, I decided to pick up Go. The language looked fascinating, and I just thought, "Why not? Let's try to write my own in-memory graph storage in Go." It was a bit of a wild idea at the time (bit of a big vision for a starter project ^^), but the thought of having something tailor-made for my needs was too tempting.

The very first version I created was incredibly simple. It basically offered a directed graph in-memory static global storage with just "direct storage functions". No fancy query language or anything like that. I built it as a standalone application, accessible via HTTP, and I even gave it a name: MERDE – which stood for Memory Entity Relation Database Experiment. Yeah, it was an experiment, and the name definitely reflected that early, raw stage.

2. Evolving Beyond an Experiment - SlingshotDB

After some time, that initial "experiment" started to feel a lot more stable. The core idea was proving itself, and the code was getting solid, so I decided it was time for a rebrand to something more "serious". That's when SlingshotDB was born.

SlingshotDB still operated as a standalone application, offering those direct storage access functions via an HTTP(S) API. But this time, it was all handled in a much more stable and robust manner. I even implemented an optional (and configurable) asynchronous persistence mechanism. It basically saved data in a modified JSONL like format, sort of like a logger. It was simple, and it did the job of providing some basic recovery, but I wouldn't call it a truly great, production ready solution. It was more of a stopgap.

One of SlingshotDB's most significant real-world applications was actually driving a Corona Statistics Dashboard for about 1.5 years. It served hourly statistics for every country, plus Germany's subnational units. This project really pushed the boundaries of its in-memory nature, consistently serving data until the storage eventually hit a memory size of around 20GB, at which point I decided to stop the project. The project worked flawless due its full life span and approved to me that an in-memory graph written on golang can perform the type of high speed traversal operations that i was looking for.

As my Go skills grew, I continued to extend SlingshotDB for quite a few years, adding features and making it more resilient. It was becoming a pretty capable tool for my internal projects. However, I started finding it rather annoying always having to access it via its HTTP API from my other Go applications. It introduced an unnecessary layer of communication and complexity when all I really wanted was direct access from my go projects.

3. The Refocus - Introducing GITS

Based on the experience made with SlingshotDB i decided to take on the task to return the focus to my original goal. My focus had to shift back to what truly mattered: creating an in-memory graph storage explicitly optimized for multithreading applications. This wasn't just another iteration; it was a massive reconstruction, and it became GITS.

The first step in this port was moving away from the standalone application model. GITS was written from the ground up as a library. You can simply import it directly into your Go projects, providing seamless in-process access . This drastically cut down on overhead and made GITS feel like a native part of any Go application.

Crucially, in this process, I made the decision to remove persistence entirely. This might sound counter-intuitive for a "database," but it was a deliberate move to focus on in-memory effectiveness. The goal was to maximize speed and efficiency for process relevant volatile data, and not be weighed down by disk I/O or complex persistence mechanisms. If you need disk persistence, GITS could be a extremely fast processing/caching layer, with the actual storage handled by something else.

The next big step was to move away from a pure direct storage API by developing a custom query language. One focus here was making it fully JSON compatible, which offers a ton of flexibility especially in terms of providing an interface for other languages to interact (everything 'can' json). To provide a simple and efficient way to use it right within Go, I also built a dedicated query builder, making it straightforward to construct complex queries programmatically.

When looking at how users (including myself) would integrate GITS into larger applications, i realized that it is often rather limiting to only have one static global storage. Instead, I decided to implement a factory pattern that allows for multiple instances of GITS within a single application. This means you can have separate, isolated graph datasets running concurrently, each managed by its own GITS instance, all within the same application process. This was the final big step of reconstructing SlingshotDB to GITS.

4. GITS Today - A Focused Tool for the Right Job

So, that's the story of GITS – a journey that started over a decade ago with a simple frustration and evolved through experiments, rebrands, and significant architectural shifts. What began as "MERDE" (a very fitting name for an early experiment ^^) and matured into SlingshotDB, has finally arrived at GITS in its current form.

Today, GITS stands as a highly efficient, in-memory graph database library, purpose-built for Go applications. It's the result of years of dealing with multithreading, concurrency, complex data structures an optimizing for performance.

Its core strengths are now:

• High Speed: Being purely in-memory, GITS offers high performance for graph traversals and data manipulation, crucial for time-sensitive operations.
• Flexible Graph Structure: It provides the ability to model complex data structures directly in your code on the fly.
• Powerful Querying: The custom, JSON-compatible query language, paired with the Go query builder, provides a easy way to interact with your graph data.
• Designed for Concurrency: From the ground up, GITS is optimized for multithreaded environments, ensuring that multiple goroutines can safely and efficiently interact with your graph instances.
• Native Go Integration: As a pure Go library, it easily integrates directly into your projects.
• Storage API: While Queries are a powerful tool and solve most use cases, you still can access the Storage directly if you want to optimize your software even more. After all, this is a tool for developers, so feel free to do whatever you wish

GITS an optimal tool for specific, demanding use cases. Think of it as your extremely-fast workbench for (but not only):

• High speed processing and live analytics on volatile data streams.
• Temporary data staging for complex transformations and ETL pipelines.
• High-performance graph-based caching for frequently accessed relationships.
• Concurrent data management state management on multithreading applications.

and many more.

The journey from a simple "experiment" to a robust, specialized tool like GITS has been a fascinating one. If you're building a Go application that handles complex relationships, high speed caching or concurrency safe multithreading operations - you might go ahead and give it a try.

Right now the projects is closing in on its v1.0.0 stable release (coming soon!)

Final words

This article sums a huge amount of time spend over the span of over 10 years - and i don't regret a second of it.

It's by far my favorite private project and even tho it did not attract too much attention, it was a great journey so far and i learned A LOT.

I will continue to work on GITS and try to make it a even better tool! There is always room for optimization .)!

If the article caught your interest and you want to check it out: GITS on Github

As always, I hope this was an interesting read and motivated you to work on your own very own special project.

So long and thanks for all the fish