CSI Driver for Rclone

4 weeks ago 2

This is a repository for Rclone CSI driver, csi plugin name: rclone.csi.veloxpack.io. This driver enables Kubernetes pods to mount cloud storage backends as persistent volumes using rclone, supporting 50+ storage providers including S3, Google Cloud Storage, Azure Blob, Dropbox, and many more.

Container Images & Kubernetes Compatibility:

driver version supported k8s version status

master branch	1.20+	GA
v1.0.0	1.20+	GA

Install driver on a Kubernetes cluster

install Rclone CSI driver on microk8s

install via helm charts
install via kubectl

Please refer to rclone.csi.veloxpack.io driver parameters

CSI driver troubleshooting guide

Please refer to development guide

testgrid sig-storage-csi-rclone dashboard.
Driver image build pipeline: post-csi-driver-rclone-push-images

50+ Storage Providers: Supports Amazon S3, Google Cloud Storage, Azure Blob, Dropbox, SFTP, and many more
No External Dependencies: Uses rclone as a Go library directly - no rclone binary installation required
No Process Overhead: Direct library integration means no subprocess spawning or external process management
Dynamic Volume Provisioning: Create persistent volumes via StorageClass
Secret-based Configuration: Secure credential management using Kubernetes secrets
Inline Configuration: Direct configuration in StorageClass parameters
Template Variable Support: Dynamic path substitution using PVC/PV metadata
VFS Caching: High-performance caching with configurable options
No Staging Required: Direct mount without volume staging
Flexible Backend Support: Choose between minimal or full backend support for smaller images

Kubernetes 1.20 or later
CSI node driver registrar
FUSE support on nodes (for mounting)
No rclone installation required - the driver uses rclone as a Go library directly

This will install:

CSI Controller (StatefulSet)
CSI Node Driver (DaemonSet)
RBAC permissions
CSIDriver CRD
Example StorageClass

2. Configure Storage Backend

Create a secret with your storage backend configuration:

apiVersion: v1 kind: Secret metadata: name: rclone-secret namespace: default type: Opaque stringData: remote: "s3" remotePath: "my-bucket" configData: | [s3] type = s3 provider = AWS access_key_id = YOUR_ACCESS_KEY_ID secret_access_key = YOUR_SECRET_ACCESS_KEY region = us-east-1

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rclone-csi provisioner: rclone.csi.veloxpack.io parameters: remote: "s3" remotePath: "my-bucket" csi.storage.k8s.io/node-publish-secret-name: "rclone-secret" csi.storage.k8s.io/node-publish-secret-namespace: "default" reclaimPolicy: Delete volumeBindingMode: Immediate allowVolumeExpansion: true

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-rclone spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi storageClassName: rclone-csi --- apiVersion: v1 kind: Pod metadata: name: nginx-rclone spec: containers: - name: nginx image: nginx volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: pvc-rclone

Method 1: Kubernetes Secrets (Recommended)

Store sensitive credentials in Kubernetes secrets and reference them in StorageClass.

Method 2: Inline Configuration

Include configuration directly in StorageClass parameters.

Method 3: PersistentVolume Configuration

Configure directly in PersistentVolume volumeAttributes.

Priority: volumeAttributes > StorageClass parameters > Secrets

Supported Storage Backends

The driver supports all rclone backends, including:

Amazon S3 and S3-compatible storage (MinIO, DigitalOcean Spaces, etc.)
Google Cloud Storage
Azure Blob Storage
Dropbox
SFTP/SSH
Google Drive
OneDrive
Box
Backblaze B2
WebDAV
FTP
And 60+ more backends

See examples for configuration examples.

Dynamic Path Substitution

The driver supports template variables in the remotePath parameter:

Variable Description Example

${pvc.metadata.name}	PVC name	my-pvc-12345
${pvc.metadata.namespace}	PVC namespace	default
${pv.metadata.name}	PV name	pv-rclone-abc123

Example:

parameters: remote: "s3" remotePath: "my-bucket/${pvc.metadata.namespace}/${pvc.metadata.name}"

Optimized Default Configuration

The driver now includes optimized default settings for better performance:

Cache Info Age: 48h (reduced from 72h for better memory usage)
Cache Cleanup: Every 10 minutes (increased frequency)
Cache Max Age: 12h (reduced for better performance)
Directory Cache: 3s (faster directory listing)
Cache Size Limits: 1GB with 2GB free space requirement

apiVersion: v1 kind: PersistentVolume metadata: name: pv-rclone-performance spec: mountOptions: - vfs-cache-mode=writes - vfs-cache-max-size=10G - dir-cache-time=30s csi: driver: rclone.csi.veloxpack.io volumeHandle: performance-volume volumeAttributes: remote: "s3" remotePath: "my-bucket" configData: | [s3] type = s3 provider = AWS access_key_id = YOUR_ACCESS_KEY_ID secret_access_key = YOUR_SECRET_ACCESS_KEY

resources: requests: memory: "256Mi" cpu: "100m" limits: memory: "512Mi" cpu: "500m"

# Check controller pods kubectl get pods -n kube-system -l app=csi-rclone-controller # Check node pods kubectl get pods -n kube-system -l app=csi-rclone-node # Check logs kubectl logs -n kube-system -l app=csi-rclone-controller kubectl logs -n kube-system -l app=csi-rclone-node

Verify Driver Functionality

# Check if the driver is working correctly kubectl exec -n kube-system -l app=csi-rclone-node -- /rcloneplugin --help # Check driver version information (shows when driver starts) kubectl logs -n kube-system -l app=csi-rclone-node --tail=10 | grep "DRIVER INFORMATION" -A 10

Authentication failures: Verify credentials in secrets or configData
Network connectivity: Ensure nodes can reach the storage backend
Permission errors: Check that credentials have proper access rights
Configuration format: Ensure configData is valid INI format
Resource constraints: Verify sufficient memory and disk space

For detailed troubleshooting, see the debug guide.

# Clone repository git clone https://github.com/veloxpack/csi-driver-rclone.git cd csi-driver-rclone # Build binary make build # Build Docker image make container # Push to registry make push

The driver supports two backend configurations for different use cases:

Full Backend Support (Default)

Includes all 70+ rclone backends for maximum compatibility:

# Build with all backends (default) docker build -t csi-rclone:latest . # Or explicitly specify docker build --build-arg RCLONE_BACKEND_MODE=all -t csi-rclone:latest .

Includes only the most common backends for smaller image size:

# Build with minimal backends docker build --build-arg RCLONE_BACKEND_MODE=minimal -t csi-rclone:minimal .

Minimal backends include:

Amazon S3 and S3-compatible storage
Google Cloud Storage
Azure Blob Storage
Dropbox
Google Drive
OneDrive
Box
Backblaze B2
SFTP
WebDAV
FTP
Local filesystem

Benefits of minimal build:

Smaller Docker image size
Faster container startup
Reduced attack surface
Lower memory footprint

Choose the build that fits your needs - full support for maximum compatibility or minimal for production efficiency.

./bin/golangci-lint run --config .golangci.yml ./...

# Run driver locally ./bin/rcloneplugin --endpoint unix:///tmp/csi.sock --nodeid CSINode -v=5

This driver is based on the csi-driver-nfs reference implementation, following CSI specification best practices. It also draws inspiration from the original csi-rclone implementation by WunderIO.

Components:

Identity Server: Plugin metadata and health checks
Controller Server: Volume lifecycle management (create/delete)
Node Server: Volume mounting/unmounting on nodes

Key Design Decisions:

No Staging: Rclone volumes don't require staging
Direct Rclone Integration: Uses rclone's Go library directly
Remote Creation: Creates temporary remotes for each mount
VFS Caching: Leverages rclone's VFS for improved performance
Template Variable Support: Dynamic path substitution using PVC/PV metadata

Use Secrets: Store sensitive credentials in Kubernetes secrets
RBAC: Ensure proper RBAC permissions are configured
Network Policies: Consider using network policies to restrict access
Image Security: Use trusted container images
Credential Rotation: Regularly rotate storage backend credentials

The driver exposes Prometheus metrics at /metrics endpoint.

Set log level for debugging:

args: - "--v=5" # Verbose logging - "--logtostderr=true"

Community, discussion, contribution, and support

Learn how to engage with the Kubernetes community on the community page.

You can reach the maintainers of this project at:

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

This project is licensed under the MIT License. See the LICENSE file for details.

Contributions welcome! Please ensure:

All code passes golangci-lint checks
Follow existing code patterns
Add tests for new functionality
Update documentation

This project builds upon the excellent work of several open source communities:

WunderIO/csi-rclone - The original rclone CSI driver implementation that inspired this project
Kubernetes CSI NFS Driver - Reference implementation and architectural patterns
Rclone - The powerful cloud storage sync tool that makes this driver possible
Kubernetes CSI Community - For the Container Storage Interface specification and ecosystem

Special thanks to the maintainers and contributors of these projects for their dedication to open source software.

Read Entire Article