Data version control over multiple heterogenous storage back ends

Today’s organizations don’t just use a single data storage solution – they operate across on-prem servers, multiple cloud providers, and hybrid environments. This distributed approach has become necessary, but it comes with significant costs: teams struggle with siloed tools, duplicated processes, and an endless cycle of environment management that diverts focus from delivering actual value. 

At lakeFS, we believe your data infrastructure should accelerate innovation, not create friction. That’s why we’re excited to introduce Multiple Storage Backends support – a new capability that unifies data management across all your storage systems.

The Problem: Distributed Data, Disconnected Management

Whether you’re operating across AWS, Azure Blob and MinIO, separating dev and prod environments, or maintaining storage-specific copies of data for compliance or performance, the reality is this:

You’re managing multiple storage systems

Until now, lakeFS instances were tightly coupled to individual storage backends. This forced users to either deploy multiple instances, or ignore valuable datasets spread across silos.

In our deep dive on distributed data management challenges, we outlined how fragmentation increases operational burden, slows delivery, and complicates collaboration. However, managing distributed data shouldn’t mean giving up visibility, consistency, or control.

The Solution: One lakeFS, Multiple Storage Backends

With Multiple Storage Backends support, lakeFS now allows you to register and manage data stored in multiple storage systems – all from a single lakeFS installation.

This means:

• A single lakeFS instance can version data across multiple object stores and control access to all of them through one interface.
  
• You can define one repository per storage backend, and manage them all through a unified API.
  
• You can now build reproducible pipelines and enforce standards across storage boundaries.

What You Gain by Managing Multiple Storage Systems with lakeFS

Example: Managing a Multi-Cloud Data Lake with lakeFS

Imagine your organization stores data across three cloud providers:

• Raw data in AWS S3
  
• Refined data in Google Cloud Storage
  
• Sensitive models in Azure Blob Storage

Traditionally, managing this kind of multi-cloud setup would require three separate lakeFS instances – each siloed, with its own configuration, governance, and policies.

With Multiple Storage Backends support, you can now manage all three from a single lakeFS instance. Each backend is registered as a separate blockstore, and each storage environment is versioned via its own lakeFS repository.

Multi-Storage Backends Support Configuration Example

Here’s how you’d connect these storage systems to a single lakeFS instance:

Once configured, each storage backend becomes a fully supported blockstore in lakeFS. You can create repositories on top of them, apply version control, and manage data access and governance – all through a single, unified lakeFS instance.

For full setup instructions and advanced options, refer to the official configuration guide.

Creating Repositories

Once lakeFS is connected to your storage systems, you can create repositories to start managing data in each one.

Here’s how to do it using the lakeFS UI:

Step 1: Select the storage backend you want to use

Step 2: Enter the repository name and details

For example, let’s create a repository for raw data ingested by a data pipeline. We’ll store this data on the AWS S3 backend we previously configured.

Note: Each lakeFS repository is tied to a single storage system. In the next section, we’ll explore strategies for tracking and versioning data across multiple repositories connected to different backends.

To continue our setup, we will also create a second repository, curated-data, which uses the GCS blockstore for refined datasets.

At this point, we’ve created two repositories:

• raw-data (on S3)
• curated-data (on GCS)

Centralized Access Control 

With our repositories in place, the next step is to manage who can do what using lakeFS Role-Based Access Control (RBAC).

lakeFS RBAC serves as a centralized access control layer, allowing you to define and enforce fine-grained permissions across all connected storage systems – without duplicating or syncing roles in each cloud provider.

Here’s how to use it:

Step 1: Define Groups

We define two groups:

• DataPlatform: responsible for ingesting and transforming data
• DataScience: focused on exploration and model development

The DataPlatform team needs full read/write access to both repositories.
The DataScience team, however, should only have read-only access to the raw data repository.

Step 2: Create Policies & Assign Permissions

We create the following lakeFS policy to restrict DataScience team’s access:

Then, we attach the policy to the DataScience group. 

The DataPlatform team is attached to the pre-configured FSFullAccess policy, which grants them full permissions across all repositories. 

Building Cross-Storage Workflows

Now that we’ve connected multiple storage systems to lakeFS, let’s see what it’s like to actually work across them. The following example uses the high-level lakeFS Python SDK to build a simple transformation workflow that:

1. Reads raw data stored in AWS S3
2. Writes refined results to GCS
3. Tracks lineage between the two using commit metadata

And does all of that without needing to worry about which storage system is being used.

This example shows that whether your raw data lives in S3 and your analytics layer is in GCS, or vice versa. lakeFS gives you a consistent interface, seamless branching and committing, and built-in lineage tracking.

You can go one step further: now that refined data is available in the curated-data repository, sensitive models stored in Azure can be read from it directly through lakeFS. This sets the stage for reproducible, cross-cloud ML pipelines – all under one control plane.

Getting Started

Want to try lakeFS Enterprise and multiple storage backend support out? Contact us to get access!