Recovery
This guide explains how to restore your system to a previous state using backup data. The label used throughout the KIOPS codebase is Recovery History (restoration history).
When do you need recovery?
- System failure: Data loss due to server failure, disk corruption, etc.
- Accidental deletion: Important resources or data were mistakenly deleted.
- Update rollback: Rolling back a problematic update to a previous state.
- Environment replication: Using backups to set up an identical environment elsewhere.
Recovery Types
KIOPS supports the following recovery flows.
- Velero recovery: Restores workloads and PV data from a Velero backup at namespace/resource granularity.
- Control plane recovery: Restores the cluster from a control plane backup that includes an etcd snapshot and PKI certificates.
- Docker/Podman recovery: Restores volumes, configuration, and containers.
- Restoration may overwrite current data. Create a backup of the current state if needed.
- If possible, perform recovery during low-traffic hours.
- For production environments, validate recovery in a test environment first.
Velero Recovery
Restore workloads and PV data from a Velero backup. Recovery runs through the Restore Wizard: connection method selection → SSH credentials → Restore Settings.
Step 1: Select a Backup and Start the Restore
- In [Backup Management] > the Backup List tab, keep the K8s backup segment on Velero and find the backup row you want to restore.
- Click the Restore icon in that row.
Step 2: Restore Settings
In the final Restore Settings step, configure the following options.
| Option | Description |
|---|---|
| Select namespaces to restore | For a multi-namespace backup, you can choose which of the backed-up namespaces to restore (at least one) and restore only some of them. For a single-namespace backup, you can map the original namespace to a target namespace. |
| Resource restore scope | Choose what to restore (4 options below). |
| Restore PVC data | Shown only when the backup includes PVC volume data. Clear it to restore only K8s resources (Deployment, Service, etc.). |
Resource restore scope (4 options)
- All resources: Restores every resource in the backup.
- Exclude PV/PVC: Does not restore the PV/PVC objects themselves, preserving existing storage.
- App resources only: Restores only Deployment, Service, ConfigMap, Secret, etc.
- Custom selection: Choose exactly which resource types to restore.
Resource filtering is chosen at restore time — The backup captures the entire selected namespace; you narrow down what to roll back here, in Resource restore scope. "Exclude PV/PVC" is useful when you want to roll back only workloads while preserving existing in-use volumes.
Step 3: Run and Verify
- Review the options, then click Start Restore.
- Track restore progress in real time in the Recovery History tab (Velero segment).
- After completion, confirm workloads came up with
kubectl get pods -n <namespace>.
Control Plane Recovery
Restore your cluster from a control plane backup that includes an etcd snapshot and PKI certificates.
Control plane recovery completely replaces the cluster to the selected backup point. All changes after the backup (resource creation/modification/deletion) will be lost. Make sure to back up the current state first.
Control plane recovery runs through the restore wizard: connection method selection → SSH credentials → confirmation.
Step 1: Select a Backup and Click the Restore Icon
- Click [Backup Management] in the left menu.
- In the Backup List tab, switch the K8s backup Segmented control to Control Plane, then find the backup row you want to restore.
- Click the Restore icon in that row.
Step 2: Select Connection Method and Enter SSH Credentials
The restore is performed by connecting directly to the master node via SSH. First choose the connection method (direct SSH / via VPN), then enter the SSH connection details. Multi-hop SSH through a gateway is supported.
- SSH connection details: host/port/username/password for the master node (and any gateway hop)
- Sudo password: Enter the sudo password required to stop/restore etcd and apply the PKI.
Step 3: Enter the Confirmation Text and Start the Restore
The restore button becomes enabled only after you type the backup name exactly for confirmation. Enter the backup name as-is, then click Start Restore. (This operation completely replaces the cluster with the selected backup point in time.)
Step 4: Monitor Progress and Confirm Completion
- The backend downloads the backup bundle from object storage.
- etcd restore (
etcdctl snapshot restore) and PKI certificate restoration run in sequence. - After completion, check the result status in the Recovery History tab.
Step 5: Verify Cluster Status
Once restoration completes, verify that the cluster is operating normally.
kubectl get nodes
kubectl get pods --all-namespaces
kubectl get pods -n kube-system
Post-restore checklist
- Are all nodes in
Readystate? - Are all kube-system Pods
Running? - Are application Pods running normally?
- Can you reach services normally?
Docker/Podman Recovery
Restore volumes, configuration, and containers from a Docker/Podman backup. There are four restore options.
Restore Options (UI)
| UI Field Label | Option Key | Description |
|---|---|---|
| Restore Volume Data | restoreVolumes | Restores the volume data mounted to containers. |
| Restore Network Settings | restoreNetworks | Restores container network settings. |
| Overwrite Existing Containers | overwriteExisting | Deletes and recreates existing containers with the same name. |
| Containers to Restore | selectedContainers | Restores only the selected containers (all if none selected). |
Three behaviors that the screen does not show are handled with fixed values: configuration is always restored, and neither Compose redeployment nor stopping existing containers is performed.
Note on location options — The "original location / new location" options from earlier versions of the guide do not exist in the product. There is also no standalone "image restore (docker load)" flow.
Restore Procedure
- Open [Backup Management] > the Backup List tab and find the target Docker/Podman backup row. The Restore History tab is for viewing results only and has no "start restore" action — restores must always be started from a backup row in the Backup List tab.
- Click the Restore icon on the backup row to open the restore modal.
- Configure the options above (Restore Volume Data / Restore Network Settings / Overwrite Existing Containers).
- Enter SSH credentials in the Remote Connection dialog when prompted.
- Click Start Restore.
Container restore detail
Once a restore starts, the restore detail dialog shows:
- Restore start/finish times
- Summary of selected options
- Per-step progress logs
- Final status (
pending,in_progress,completed,failed)
Recovery History
The [Backup Management] > Recovery History tab tracks every restore operation.
- Status values:
pending,in_progress,completed,failed - Detail view: Clicking a row opens either the container restore detail or the control plane restore detail.
- Compose Project Filter: Filter the Docker/Podman recovery history by Docker Compose project name. Useful on Docker hosts that run multiple projects when you want to view only one project's restore history.
Recovery Verification
After recovery, always verify that the system is operating correctly.
Verification Checklist
- Service status: Check Pod/container status. Expected state: Running.
- Data integrity: Verify application data. Expected state: Data exists from backup point.
- Network: Test service access. Expected state: Normal response.
- Logs: Check application logs. Expected state: No errors.
Automate recovery verification — For critical systems, prepare scripts that automatically perform health checks after recovery to reduce verification time.
Troubleshooting
Restore Failure: "snapshot file corrupted"
restore failed: snapshot file corrupted
Why does this happen? The backup file is corrupted and cannot be used for restoration.
Resolution
- Use a different backup file: Pick a valid backup from a different date.
- Verify backup file integrity: Check the checksum on the storage side.
- Check backup copies: If backups were replicated elsewhere, use that file.
SSH Authentication Failure
The master node SSH credentials are incorrect or the user lacks permissions. Re-check that the SSH username, password, and sudo password you entered are correct.
Data Mismatch
This occurs when restored data differs from expectations or applications produce errors.
How to check and resolve
- Verify backup point: Confirm the restored data matches the backup point state.
- Check schema compatibility: Verify the application version is compatible with the data schema.
- Sync external systems: Check if databases or other external systems also need restoration.
Caution with partial restoration — Restoring only part of a system can cause data inconsistencies. When possible, restore all related components together.