Daniel J. Bernstein updated cdb (Constant database) to go beyond 4GB

2 weeks ago 2

cdb: Intro

cdb is a fast, reliable, simple package for creating and reading constant databases. The cdb database structure provides several features:

Fast lookups: A successful lookup in a large database normally takes just two disk accesses. An unsuccessful lookup takes only one.
Low overhead: A database uses 2048 bytes (or 4096 bytes for cdb64), plus 24 bytes per record (or 48 bytes per record for cdb64), plus the space for keys and data.
No random limits: cdb handles any database up to 4 gigabytes (or up to an exabyte for cdb64). There are no other restrictions. Records are streamed and don't have to fit into memory. Databases are stored in a machine-independent format (one format for cdb, one format for cdb64).
Fast atomic database replacement: cdbmake rewrites an entire database much more quickly than other hashing packages. Database replacement is safe against system crashes. Readers don't pause during a rewrite: they continue transparently using the old database until the new database is in place.
Fast database dumps: cdbdump prints the contents of a database in cdbmake-compatible format.

Latest release: 20251021.

Author

More packages saying that they provide tools for you to use cdb format (and in some cases cdb64 format):

cdb64 from Pierre Carrier: Tweak of cdb-0.75 to support cdb64.
cdb64-rs from Jiseok Choi: Rust implementation of cdb64, plus wrappers for C, Node.js, and Python.
CDB_File: from Tim Goodwin, Matt Sergeant, and Todd Rinaldo: Perl wrapper around cdb.
cdb-full from Kazuteru Okahashi and Rich Lane: Ruby wrapper around cdb.
CDB::TinyCDB from Alex J. G. Burzyński: Perl wrapper around TinyCDB.
go-cdb-1: from John Barham: Go implementation.
go-cdb-2 from Colin Marc: Go implementation.
go-cdb64 from Chris Lu: Go implementation of cdb64.
howerj-cdb from Richard James Howe: C implementation.
hs-cdb from Adam Smith: Haskell implementation.
libowfat: C implementation from Felix von Leitner (as part of a broader library).
luacdb from Taj Khattra: Lua wrapper around cdb.
lua-tinycdb from A. S. Bradbury: Lua wrapper around TinyCDB.
php-cdb: PHP implementation.
pure-cdb from Boris Sukholitko: Haskell implementation.
pycdb from Yusuke Shinyama: Python implementation.
python-cdb from Mike Pomraning and Alan Grow: Python wrapper around cdb.
python-pure-cdb from David Wilson: Python implementation. Also supports cdb64.
rust-cdb from Bruce Guenter: Rust implementation.
sg-cdb from Michael Alyn Miller: Java implementation.
swift-cdb: Swift wrapper around howerj-cdb.
TinyCDB from Michael Tokarev: C implementation.