Backup & Restore

Protect your Laminar data with comprehensive backup and restore procedures.

Overview

Critical data to backup:

RocksDB data - Pipeline state and metadata
Checkpoints - Pipeline processing state (in object storage)
Artifacts - Pipeline outputs (in object storage)
Configuration - Helm values, secrets

RocksDB Backup

Manual Backup

Use Kubernetes volume snapshots for consistent backups:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: laminar-rocksdb-backup
  namespace: laminar
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: laminar-controller-data

# Create snapshot
kubectl apply -f volume-snapshot.yaml
 
# List snapshots
kubectl get volumesnapshot -n laminar

Automated Backup with CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: rocksdb-backup
  namespace: laminar
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: alpine
              command:
                - /bin/sh
                - -c
                - |
                  # Create snapshot and upload to S3
                  tar czf /tmp/rocksdb-$(date +%Y%m%d-%H%M%S).tar.gz /data
                  # Upload to S3 (requires aws cli)
              volumeMounts:
                - name: data
                  mountPath: /data
                  readOnly: true
          volumes:
            - name: data
              persistentVolumeClaim:
                claimName: laminar-controller-data
          restartPolicy: OnFailure

Cloud Provider Snapshots

# Get volume ID
VOLUME_ID=$(kubectl get pv -o jsonpath='{.items[?(@.spec.claimRef.name=="laminar-controller-data")].spec.awsElasticBlockStore.volumeID}')
 
# Create snapshot
aws ec2 create-snapshot \
  --volume-id $VOLUME_ID \
  --description "Laminar RocksDB backup $(date +%Y%m%d)"

Checkpoint & Artifact Backup

Checkpoints and artifacts are stored in object storage (S3, GCS, Azure Blob).

S3 Versioning & Replication

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket laminar-data \
  --versioning-configuration Status=Enabled
 
# Create replication rule for DR
aws s3api put-bucket-replication \
  --bucket laminar-data \
  --replication-configuration file://replication.json

replication.json:

{
  "Role": "arn:aws:iam::123456789:role/replication-role",
  "Rules": [{
    "Status": "Enabled",
    "Priority": 1,
    "Filter": {},
    "Destination": {
      "Bucket": "arn:aws:s3:::laminar-data-backup",
      "StorageClass": "STANDARD_IA"
    }
  }]
}

GCS Backup

# Transfer to backup bucket
gsutil -m rsync -r gs://laminar-data gs://laminar-data-backup

Configuration Backup

Helm Values

# Export current values
helm get values laminar -n laminar > laminar-values-backup.yaml
 
# Store in version control
git add laminar-values-backup.yaml
git commit -m "Backup Laminar values $(date +%Y%m%d)"

Secrets Backup

# Export secrets (encrypted)
kubectl get secrets -n laminar -o yaml | \
  kubeseal --format yaml > sealed-secrets-backup.yaml
 
# Or backup to secrets manager
kubectl get secret laminar-credentials -n laminar -o jsonpath='{.data}' | \
  aws secretsmanager put-secret-value \
    --secret-id laminar/backup \
    --secret-string "$(cat -)"

Restore Procedures

RocksDB Restore

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: laminar-controller-data-restored
  namespace: laminar
spec:
  dataSource:
    name: laminar-rocksdb-backup
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

# Scale down
kubectl scale deployment -n laminar laminar-controller --replicas=0
 
# Apply restored PVC
kubectl apply -f restored-pvc.yaml
 
# Update deployment to use restored PVC
kubectl patch deployment laminar-controller -n laminar \
  --patch '{"spec":{"template":{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"laminar-controller-data-restored"}}]}}}}'
 
# Scale back up
kubectl scale deployment -n laminar laminar-controller --replicas=2

Full Disaster Recovery

Provision new infrastructure

# Create new cluster
eksctl create cluster --name laminar-dr --region us-west-2

Restore RocksDB volume

# Restore from cross-region snapshot
aws ec2 copy-snapshot \
  --source-region us-east-1 \
  --source-snapshot-id snap-xxx \
  --destination-region us-west-2

Install Laminar

helm install laminar laminar/laminar \
  -f laminar-values-backup.yaml \
  --set controller.persistence.existingClaim=restored-pvc

Verify data

kubectl exec -n laminar deploy/laminar-api -- laminar pipelines list

Backup Retention

Data Type	Retention	Storage Class
RocksDB (hourly snapshots)	24 hours	Standard
RocksDB (daily snapshots)	30 days	Standard
RocksDB (weekly snapshots)	90 days	Infrequent Access
Checkpoints	7 days	Standard
Artifacts	Per policy	Varies

Lifecycle Policy (S3)

{
  "Rules": [{
    "ID": "BackupRetention",
    "Status": "Enabled",
    "Filter": {"Prefix": "backups/"},
    "Transitions": [{
      "Days": 30,
      "StorageClass": "STANDARD_IA"
    }, {
      "Days": 90,
      "StorageClass": "GLACIER"
    }],
    "Expiration": {"Days": 365}
  }]
}

Backup Verification

Test Restore

Schedule monthly restore tests:

# Create test environment
helm install laminar-test laminar/laminar \
  --namespace laminar-test \
  --create-namespace \
  --set controller.persistence.existingClaim=test-restored-pvc
 
# Verify data
kubectl exec -n laminar-test deploy/laminar-api -- \
  laminar pipelines list
 
# Clean up
helm uninstall laminar-test -n laminar-test

Backup Monitoring

# Alert if backup is too old
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: backup-alerts
spec:
  groups:
    - name: backup
      rules:
        - alert: BackupTooOld
          expr: time() - backup_last_success_timestamp > 86400
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: RocksDB backup is more than 24 hours old

Backup Checklist

RocksDB volume snapshots configured (daily minimum)
Backup retention policy defined
Cross-region/cross-account backup enabled
Checkpoint storage with versioning
Helm values in version control
Secrets backed up securely
Restore procedure documented
Monthly restore tests scheduled
Backup monitoring and alerting

Next Steps

High Availability - HA configuration
Troubleshooting - Common issues