Sync Secrets from K8s to Vault

A Kubernetes operator that automatically syncs secrets from Kubernetes to HashiCorp Vault using annotations on Deployments.

The Vault Sync Operator watches for Kubernetes Deployments with specific annotations and automatically pushes the referenced secrets to HashiCorp Vault. It uses Vault's Kubernetes authentication backend for secure authentication.

• Automatic Secret Synchronization: Sync Kubernetes secrets to Vault based on deployment annotations
• Kubernetes Authentication: Uses Vault's Kubernetes auth backend for secure authentication
• Selective Secret Keys: Choose specific keys from secrets to sync to Vault
• Key Prefixing: Add prefixes to secret keys when storing in Vault
• Cleanup on Deletion: Automatically removes secrets from Vault when deployments are deleted
• RBAC Support: Proper Kubernetes RBAC permissions for secure operation

• Kubernetes cluster (v1.20+)
• HashiCorp Vault server with Kubernetes auth backend configured
• kubectl configured to access your cluster

The Vault Sync Operator supports multiple deployment methods:

1. Helm Chart (Recommended):

2. Kustomize:

3. Manual kubectl:

4. Make (Development):

For detailed deployment instructions, see DEPLOYMENT.md.

1. Configure Vault Kubernetes Auth Backend:

For detailed Vault setup instructions, see VAULT-SETUP-GUIDE.md.

Quick setup:

The Vault Sync Operator provides comprehensive monitoring and error handling capabilities to ensure reliable operation.

The operator exposes standard Kubernetes health and readiness endpoints:

• Health Check (/healthz): Validates connectivity to Vault server
• Readiness Check (/readyz): Ensures Vault authentication is working correctly

These endpoints are automatically configured and can be used by Kubernetes for container health monitoring.

The operator exposes Prometheus metrics on port :8080 by default. Available metrics include:

• vault_sync_operator_sync_attempts_total: Total number of secret sync attempts (labeled by namespace, deployment, result)
• vault_sync_operator_sync_duration_seconds: Duration of secret sync operations in seconds
• vault_sync_operator_secrets_discovered: Number of secrets auto-discovered in deployments

• vault_sync_operator_secret_not_found_errors_total: Kubernetes secrets that couldn't be found
• vault_sync_operator_secret_key_missing_errors_total: Missing keys within secrets
• vault_sync_operator_config_parse_errors_total: Configuration parsing errors
• vault_sync_operator_vault_write_errors_total: Vault write errors (categorized by error type)

• vault_sync_operator_auth_attempts_total: Vault authentication attempts and results

The operator provides detailed error reporting for common failure scenarios:

• Secret Not Found: When a referenced Kubernetes secret doesn't exist
• Key Not Found: When a specified key doesn't exist within a secret
• Available Keys Reporting: Error logs include available keys to aid troubleshooting

• Authentication Failures: When Vault authentication fails or tokens expire
• Permission Denied: When the operator lacks write permissions to the specified Vault path
• Invalid Path: When the Vault path doesn't exist or is malformed
• Connection Issues: Network connectivity problems with Vault

• JSON Parse Errors: When the vault-sync.io/secrets annotation contains invalid JSON
• Invalid Annotation Format: When required annotations are malformed

All errors are logged with structured logging including relevant context (namespace, deployment, secret names, etc.) and are tracked via Prometheus metrics for alerting and monitoring.

1. Create a Secret:

2. Create a Deployment with Vault Sync Annotations:

This will sync the username and password keys from the my-app-secrets secret to Vault at the path secret/data/my-app with the keys prefixed as app_username and app_password.

The operator uses the following annotations to control secret synchronization:

The operator supports two synchronization modes:

When no vault-sync.io/secrets annotation is provided, the operator automatically discovers all secrets referenced in the deployment pod template and writes each secret to its own sub-path.

Result: Each secret gets its own sub-path:

• secret/data/my-app/my-app-secrets → { "username": "...", "password": "..." }
• secret/data/my-app/db-secrets → { "host": "...", "port": "..." }

When vault-sync.io/secrets annotation is provided, all specified keys are written directly to the main vault path with optional prefixes.

Result: All keys written to the main path:

• secret/data/my-app → { "db_username": "...", "db_password": "...", "token": "..." }

Enable periodic reconciliation to automatically restore secrets that are accidentally deleted from Vault:

Benefits:

• Automatically restores deleted secrets
• Provides resilience against manual vault operations
• Configurable interval (minimum 30 seconds)
• Disabled by default for optimal performance

Configuration Examples:

• "5m" - Check every 5 minutes
• "1h" - Check every hour
• "30s" - Check every 30 seconds (minimum)
• "off" - Disabled (default)

The operator follows the standard Kubernetes pattern of per-cluster deployment. Each cluster runs its own operator instance, which is the recommended approach for:

• Security isolation between clusters
• Simple architecture without cross-cluster networking
• Independent cluster operations
• Following Kubernetes operator best practices

Deploy the operator in each cluster with cluster-specific configuration:

Vault paths will be automatically organized by cluster:

• Cluster A: clusters/production-us-east/secret/data/my-app
• Cluster B: clusters/production-eu-west/secret/data/my-app

See Multi-Cluster Deployment Guide for complete setup instructions.

The operator works with secret generators (Kustomize, Helm, etc.) as it:

1. Reads secrets at runtime: Doesn't watch secret creation, only reads when syncing
2. Auto-discovers references: Finds secrets referenced in deployment pod templates
3. Provides helpful error messages: Suggests checking if secret generators have run

Recommendation: Ensure secret generators run before the operator reconciles deployments.

• Go 1.22+
• Docker
• kubectl
• A Kubernetes cluster for testing

1. Run the operator locally:

The operator supports the following command-line flags:

1. RBAC: The operator only requires read access to Deployments and Secrets in the namespaces it operates in.

2. Vault Authentication: Uses Kubernetes service account tokens for authentication with Vault.

3. Finalizers: Uses finalizers to ensure cleanup of Vault secrets when deployments are deleted.

4. Least Privilege: Configure Vault policies to give the operator only the minimum required permissions.

The operator provides detailed error messages and metrics to help diagnose issues. Here are common problems and their solutions:

Error: failed to get secret mysecret: secrets "mysecret" not found

Cause: The Kubernetes secret referenced in the deployment doesn't exist.

Solution:

• Check if the secret exists: kubectl get secret mysecret -n <namespace>
• Create the missing secret or fix the reference in your deployment annotations

Metrics: Tracked in vault_sync_operator_secret_not_found_errors_total

Error: key mykey not found in secret mysecret

Cause: The specified key doesn't exist in the secret.

Solution:

• Check available keys: kubectl get secret mysecret -o yaml
• The error log will include available keys to help identify the correct key name
• Update the annotation to use the correct key name

Metrics: Tracked in vault_sync_operator_secret_key_missing_errors_total

Error: failed to authenticate with vault: permission denied

Cause: The operator can't authenticate with Vault.

Solution:

• Verify Vault's Kubernetes auth backend is configured correctly
• Check that the service account has the correct role assigned
• Ensure the JWT token is valid and accessible

Metrics: Tracked in vault_sync_operator_auth_attempts_total{result="failed"}

Error: failed to write secret to vault: permission denied

Cause: The authenticated role lacks write permissions to the specified path.

Solution:

• Update the Vault policy to allow write access to the path
• Verify the role-policy binding in Vault

Metrics: Tracked in vault_sync_operator_vault_write_errors_total{error_type="permission_denied"}

Error: failed to parse secrets annotation: invalid character

Cause: The JSON in the vault-sync.io/secrets annotation is malformed.

Solution:

• Validate the JSON syntax in your annotation
• Use tools like jq to validate: echo '<annotation-value>' | jq .

Metrics: Tracked in vault_sync_operator_config_parse_errors_total{error_type="json_parse_error"}

1. Check operator logs:

2. Monitor metrics (if Prometheus is available):

3. Verify deployment annotations:

4. Test with troubleshooting example:

You can manually check the operator's health:

The operator is optimized for Kubernetes container environments with automatic Go runtime configuration:

• GOMAXPROCS: Automatically configured based on container CPU limits using go.uber.org/automaxprocs
• GOMEMLIMIT: Set from container memory limits to optimize garbage collection
• Container-aware GC: Prevents memory usage beyond container limits

The default deployment includes:

The operator logs its runtime configuration at startup:

For production workloads, adjust resources based on:

• CPU: Increase for high-volume secret synchronization
• Memory: Increase for deployments with many large secrets
• Replicas: Enable leader election for high availability

Example production configuration: