.png)
1 day ago
2
Aurora DSQL is based on PostgreSQL. The Query Processor (QP) component is a running postgres process, although we’ve significantly modified it to work with our architecture.
If you connect to DSQL with a Postgres client (such as psql), you’re connected to one of these postgres processes. You’re connected to a QP, and you can start to interact with it as you would with any other Postgres server.
If you run a local Postgres operation, such as SELECT 1, then that query is processed entirely locally by the QP. But what happens when you query a table:
Usually, you’d expect Postgres to read from storage locally, which might mean reading from the buffer cache, or doing disk I/O. When running on Aurora Postgres (APG), cache misses would result in a load from remote storage.
Like APG, reads in DSQL also go to remote storage. In our above query, which is a scan of the entire test table, the QP is going to turn around and scan storage, and storage is going to return all the rows in the table.
But how did storage get the rows in the first place?
Vanilla Postgres would process that transaction locally, inserting into the Write-Ahead Log (WAL), updating the buffer cache, and using fsync() to persist the changes to disk. In APG, the buffer cache is also updated, but the durability of the transaction is ensured by fsync() to the remote storage in multiple Availability Zones (AZ).
Commits in DSQL have the same basic ingredients, but they’re expressed quite differently. In DSQL, data is durably persisted when it’s written to the journal1. Storage follows the journal, and keeps itself up to date.
When I first started working on DSQL (many years ago!), I didn’t really get this flow. I’d been told “writes go to the journal, reads go to storage”. I nodded, but I didn’t deeply, truly, understand that simple explanation. I’d spent too much time with traditional architectures, and my mind kept falling back on the familiar.
What helped me get it was the picture at the top of this post. Imagine somebody drawing this on the whiteboard. They draw the three boxes: QP, journal and storage. Then, they draw the Circle of Life:
There’s something about this presentation, vs. the one at the top of the article, that helped it go click for me. Removing the service interactions certainly helps. Notice how in the first picture there’s an arrow from the QP to storage, while in the Circle, the arrow is the other way round?
“Writes go to the journal, reads go to storage”, never quite did it for me. “Writes never go to storage, reads never go to the journal” also didn’t quite the message across. The Circle did for me, and I hope it does for you.
