Show HN: LightDB – fast embedded DB for Scala

4 months ago 19

Computationally focused database using pluggable stores

Store Type Embedded Persistence Read Perf Write Perf Concurrency Transactions Full-Text Search Notes

HaloDB	KV Store	✅	✅	✅	✅✅	🟡 (Single-threaded write)	🟡 (Basic durability)	❌	Fast, simple write-optimized store
ChronicleMap	Off-Heap Map	✅	✅ (Memory-mapped)	✅✅	✅✅	✅✅	❌	❌	Ultra low-latency, off-heap storage
LMDB	KV Store (B+Tree)	✅	✅	✅✅✅	✅	🟡 (Single write txn)	✅✅ (ACID)	❌	Read-optimized, mature B+Tree engine
MapDB	Java Collections	✅	✅	✅	✅	✅	✅	❌	Easy Java-native persistence
RocksDB	LSM KV Store	✅	✅	✅✅	✅✅✅	✅	✅	❌	High-performance LSM tree
Redis	In-Memory KV Store	🟡 (Optional)	✅ (RDB/AOF)	✅✅✅	✅✅	✅	✅	❌	Popular in-memory data structure store
Lucene	Full-Text Search	✅	✅	✅✅	✅	✅	❌	✅✅✅	Best-in-class full-text search engine
SQLite	Relational DB	✅	✅	✅	✅	🟡 (Write lock)	✅✅ (ACID)	✅ (FTS5)	Lightweight embedded SQL
H2	Relational DB	✅	✅	✅	✅	✅	✅✅ (ACID)	❌ (Basic LIKE)	Java-native SQL engine
DuckDB	Analytical SQL	✅	✅	✅✅✅	✅	✅	✅	❌	Columnar, ideal for analytics
PostgreSQL	Relational DB	❌ (Server-based)	✅	✅✅✅	✅✅	✅✅	✅✅✅ (ACID, MVCC)	✅✅ (TSVector)	Full-featured RDBMS

✅: Supported / Good
✅✅: Strong
✅✅✅: Best-in-class
🟡: Limited or trade-offs
❌: Not supported

Tantivy (https://github.com/quickwit-oss/tantivy) - Working on creating a wrapper around Rust's extremely fast alternative to Apache Lucene (See https://github.com/outr/scantivy)

To add all modules:

libraryDependencies += "com.outr" %% "lightdb-all" % "4.0.0"

For a specific implementation like Lucene:

libraryDependencies += "com.outr" %% "lightdb-lucene" % "4.0.0"

Watch this Java User Group demonstration of LightDB

This guide will walk you through setting up and using LightDB, a high-performance computational database. We'll use a sample application to explore its key features.

NOTE: This project uses Rapid (https://github.com/outr/rapid) for effects. It's somewhat similar to cats-effect, but with a focus on virtual threads and simplicity. In a normal project, you likely wouldn't be using .sync() to invoke each task, but for the purposes of this documentation, this is used to make the code execute blocking.

Ensure you have the following:

Scala installed
SBT (Scala Build Tool) installed

Add LightDB to Your Project

Add the following dependency to your build.sbt file:

libraryDependencies += "com.outr" %% "lightdb-all" % "4.0.0"

Example: Defining Models and Collections

Step 1: Define Your Models

LightDB uses Document and DocumentModel for schema definitions. Here's an example of defining a Person and City:

import lightdb._ import lightdb.id._ import lightdb.store._ import lightdb.doc._ import fabric.rw._ case class Person( name: String, age: Int, city: Option[City] = None, nicknames: Set[String] = Set.empty, friends: List[Id[Person]] = Nil, _id: Id[Person] = Person.id() ) extends Document[Person] object Person extends DocumentModel[Person] with JsonConversion[Person] { override implicit val rw: RW[Person] = RW.gen val name: I[String] = field.index("name", _.name) val age: I[Int] = field.index("age", _.age) val city: I[Option[City]] = field.index("city", _.city) val nicknames: I[Set[String]] = field.index("nicknames", _.nicknames) val friends: I[List[Id[Person]]] = field.index("friends", _.friends) }

case class City(name: String) object City { implicit val rw: RW[City] = RW.gen }

Step 2: Create the Database Class

Define the database with stores for each model:

import lightdb.sql._ import lightdb.store._ import lightdb.upgrade._ import java.nio.file.Path object db extends LightDB { override type SM = CollectionManager override val storeManager: CollectionManager = SQLiteStore lazy val directory: Option[Path] = Some(Path.of(s"docs/db/example")) lazy val people: Collection[Person, Person.type] = store(Person) override def upgrades: List[DatabaseUpgrade] = Nil }

Step 1: Initialize the Database

Initialize the database:

Add records to the database:

val adam = Person(name = "Adam", age = 21) // adam: Person = Person( // name = "Adam", // age = 21, // city = None, // nicknames = Set(), // friends = List(), // _id = StringId(value = "fKJ5L8wKjw4dLiu31LWsCfOC6LsgJ6EG") // ) db.people.transaction { implicit txn => txn.insert(adam) }.sync() // res1: Person = Person( // name = "Adam", // age = 21, // city = None, // nicknames = Set(), // friends = List(), // _id = StringId(value = "fKJ5L8wKjw4dLiu31LWsCfOC6LsgJ6EG") // )

Retrieve records using filters:

db.people.transaction { txn => txn.query.filter(_.age BETWEEN 20 -> 29).toList.map { peopleIn20s => println(s"People in their 20s: $peopleIn20s") } }.sync() // People in their 20s: List(Person(Adam,21,None,Set(),List(),StringId(bxvPuSxBZ3MJ7rihSEssM697Bq32xzFw)), Person(Adam,21,None,Set(),List(),StringId(fKJ5L8wKjw4dLiu31LWsCfOC6LsgJ6EG)))

Transactions: LightDB ensures atomic operations within transactions.
Indexes: Support for various indexes, like tokenized and field-based, ensures fast lookups.
Aggregation: Perform aggregations such as min, max, avg, and sum.
Streaming: Stream records for large-scale queries.
Backups and Restores: Backup and restore databases seamlessly.

db.people.transaction { txn => txn.query .aggregate(p => List(p.age.min, p.age.max, p.age.avg, p.age.sum)) .toList .map { results => println(s"Results: $results") } }.sync() // Results: List(MaterializedAggregate({"ageMin": 21, "ageMax": 21, "ageAvg": 21.0, "ageSum": 42},repl.MdocSession$MdocApp$Person$@5be0985d))

db.people.transaction { txn => txn.query.grouped(_.age).toList.map { grouped => println(s"Grouped: $grouped") } }.sync() // Grouped: List(Grouped(21,List(Person(Adam,21,None,Set(),List(),StringId(bxvPuSxBZ3MJ7rihSEssM697Bq32xzFw)), Person(Adam,21,None,Set(),List(),StringId(fKJ5L8wKjw4dLiu31LWsCfOC6LsgJ6EG)))))

Backup your database:

import lightdb.backup._ import java.io.File DatabaseBackup.archive(db.stores, new File("backup.zip")).sync() // res5: Int = 3

Restore from a backup:

DatabaseRestore.archive(db, new File("backup.zip")).sync() // res6: Int = 3

Dispose of the database when done:

This guide provided an overview of using LightDB. Experiment with its features to explore the full potential of this high-performance database. For advanced use cases, consult the API documentation.

Read Entire Article