Lakekeeper: Secure, fast and user-friendly Iceberg REST catalog built with Rust

4 months ago 6

This is Lakekeeper: A secure, fast, and user-friendly Apache Iceberg REST Catalog built with Rust and available under the Apache License.

Overview

Lakekeeper Architecture Overview

Our Features

Warehouse in the Front, data lake in the back

Iceberg achieves transactional consistency of data warehouses, while enabling modular horizontal scaling of compute and storage on data lakes.

Centralized Data Governance + Decentralized Compute

Manage data access policies centrally - without duplication in compute engines.

Open Data Governance

Lakekeeper connects to open external permission systems like OpenFGA and can expose permissions via Open Policy Agent (OPA). This enables best-in-class integration with query engines like trino that support external permission systems.

OpenID Provider Integration

Use your own identity provider for authentication. Lakekeeper never generates (API)-tokens itself. You already have an IdP. Let's use it!

Automatic Table Maintenance

Optimize query performance for all your data lake engines with out-of-the-box automated compaction and maintenance strategies. (Coming soon)

Easy to Use

Single binary executable for all major platforms; no JVM or Python environment required. Native Kubernetes deployments with Helm chart or k8s operator. UI and batteries included.

Storage Access Management

Lakekeeper secures access to your data for on-premise and cloud deployments using Vended-Credentials and remote signing for S3.

Change Events

Lakekeeper can emit change events to Event Queues like Nats or Kafka to keep stakeholders informed.

Normalized Database Model

Lakekeeper uses a normalized relational Database model internally. This allows us to add powerful new endpoints and statistics in the future without file-system access!

Highly Available & Horizontally Scalable

There is no local state - the catalog can be scaled horizontally easily. Autoscaling in helm included.

Extensible

See something that's missing? Build it! Lakekeeper is meant to be extended through. And because Lakekeeper is written in Rust, you can use powerful Rust traits to do so.

Ecosystem first

We are not bound to a query engine vendor. We care about the whole ecosystem and interoperability.

Rust Native

Lakekeeper is written in Rust and based on `iceberg-rust`. No unsafe Code - guaranteed!

Checkout The Code

Lakekeeper Announcements

APR 2025

Meet

The first meetup will be on April 2nd in Amsterdam at 17:00. Sign up and find more details on the event page.
MAR 2025

Trino

Join us on the trino community broadcast at on Thursday March 13th, where we demonstrate Lakekeeper's OPA integration with Trino.
FEB 2025

0.7.0

Release 0.7.0 adds support for s3a and s3n Filesystems, improves reverse-proxy support and introduces table and view statistics!
JAN 2025

0.6.0

Release 0.6.0 focuses on security, introducing Lakekeeper's OPA bridge for Trino integration, with a new "check" endpoint for simpler permission queries. It also supports Iceberg versions 1.5 to 1.7, adds automatic file cleanup for managed tables, and fixes ADLS cleanup issues.
DEC 2024

0.5.0

Release 0.5.0 is the biggest yet, featuring a new UI, detailed docs, and table-level access controls. It adds native support for Kubernetes Service Accounts and improves integration with external IdPs like EntraID and Keycloak.