# Cloud and Kubernetes recovery exercise

Move from Pod Ready to demonstrated business recovery. Plan recovery of cluster records, persistent data, identities, keys and dependencies together.

Original AI-assisted SecNet educational content. Fictional local examples; not a standards provider publication, endorsement or compliance evidence. Page exercises do not create account progress.

Sources checked: 2026-10-21

## What the snapshot does not cover

An etcd snapshot can cover cluster records, not automatically application volume data, external databases or identity providers. Available managed-control-plane recovery options vary by provider. For each asset, record backup method, key, owner, frequency and restoration dependency. Replace a single “backup exists” row with the component chain that makes the service work and expose missing links before the exercise.

## Clean and isolated restoration

Restoring the old environment unchanged after an attack may restore attacker access too. Start isolated with reviewed images, refreshed identities, verified data and controlled connectivity. Kubernetes Ready does not establish correct business transactions. For Harbor, test permitted ticket reads/writes, tenant isolation and external dependencies. An authorized business owner should decide traffic reopening based on evidence.

## Measure RTO and RPO

This fictional example uses a 60-minute RTO and 15-minute RPO, not universal targets. Fifty minutes restoring plus twenty validating exceeds the RTO at seventy. Twenty-five minutes of data loss exceeds RPO even if service returns in fifty-five minutes. Define clock boundaries and measurement method beforehand. Waiting for identity or DNS does not pause the clock; record when all dependencies become functionally ready.

## Worked example: green cluster, failed business

Every Pod in the new cluster appears Ready, but historical tickets cannot be read because the old key is unavailable. Do not close this as success. The inventory model separates keys and persistent data, the clean model requires business validation, and the objective model checks elapsed time and data loss. Record missing-dependency ownership, remediation date and retest in the exercise report. No real snapshot or restore command runs here; downloaded workpapers support planning an authorized exercise.

## Working sequence

1. Map the service and dependency restoration chain.
2. Approve RTO/RPO and clock boundaries.
3. Plan isolated controlled restoration.
4. Test business transactions and authority boundaries.
5. Plan improvements from actual time and data loss.

## 1. Only an etcd snapshot

An etcd snapshot does not automatically restore application data or external identity. Persistent data, identity configuration and recoverable keys need separate evidence.

This code is a local policy model; it does not replace target/cluster testing.

### Weak implementation
```js
s=>s.etcdSnapshot
```

### Corrected implementation
```js
s=>s.etcdSnapshot&&s.persistentData&&s.identityConfig&&s.keysRecoverable
```

### Synthetic input and actual model output
```json
{
  "before": [
    {
      "name": "dependencies-covered",
      "input": {
        "etcdSnapshot": true,
        "persistentData": true,
        "identityConfig": true,
        "keysRecoverable": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    },
    {
      "name": "snapshot-only",
      "input": {
        "etcdSnapshot": true,
        "persistentData": false,
        "identityConfig": false,
        "keysRecoverable": false
      },
      "expected": false,
      "observed": true,
      "matched": false
    },
    {
      "name": "missing-volume-data",
      "input": {
        "etcdSnapshot": true,
        "persistentData": false,
        "identityConfig": true,
        "keysRecoverable": true
      },
      "expected": false,
      "observed": true,
      "matched": false
    }
  ],
  "after": [
    {
      "name": "dependencies-covered",
      "input": {
        "etcdSnapshot": true,
        "persistentData": true,
        "identityConfig": true,
        "keysRecoverable": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    },
    {
      "name": "snapshot-only",
      "input": {
        "etcdSnapshot": true,
        "persistentData": false,
        "identityConfig": false,
        "keysRecoverable": false
      },
      "expected": false,
      "observed": false,
      "matched": true
    },
    {
      "name": "missing-volume-data",
      "input": {
        "etcdSnapshot": true,
        "persistentData": false,
        "identityConfig": true,
        "keysRecoverable": true
      },
      "expected": false,
      "observed": false,
      "matched": true
    }
  ]
}
```

## 2. Ready but unverified

Pod Ready is not business validation. Isolation, rotated identities, integrity and business tests are jointly required by this fictional recovery profile.

This code is a local policy model; it does not replace target/cluster testing.

### Weak implementation
```js
s=>s.podsReady
```

### Corrected implementation
```js
s=>s.podsReady&&s.isolated&&s.credentialsRotated&&s.integrityVerified&&s.businessTested
```

### Synthetic input and actual model output
```json
{
  "before": [
    {
      "name": "clean-recovery",
      "input": {
        "podsReady": true,
        "isolated": true,
        "credentialsRotated": true,
        "integrityVerified": true,
        "businessTested": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    },
    {
      "name": "ready-only",
      "input": {
        "podsReady": true,
        "isolated": false,
        "credentialsRotated": false,
        "integrityVerified": false,
        "businessTested": false
      },
      "expected": false,
      "observed": true,
      "matched": false
    },
    {
      "name": "no-business-test",
      "input": {
        "podsReady": true,
        "isolated": true,
        "credentialsRotated": true,
        "integrityVerified": true,
        "businessTested": false
      },
      "expected": false,
      "observed": true,
      "matched": false
    }
  ],
  "after": [
    {
      "name": "clean-recovery",
      "input": {
        "podsReady": true,
        "isolated": true,
        "credentialsRotated": true,
        "integrityVerified": true,
        "businessTested": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    },
    {
      "name": "ready-only",
      "input": {
        "podsReady": true,
        "isolated": false,
        "credentialsRotated": false,
        "integrityVerified": false,
        "businessTested": false
      },
      "expected": false,
      "observed": false,
      "matched": true
    },
    {
      "name": "no-business-test",
      "input": {
        "podsReady": true,
        "isolated": true,
        "credentialsRotated": true,
        "integrityVerified": true,
        "businessTested": false
      },
      "expected": false,
      "observed": false,
      "matched": true
    }
  ]
}
```

## 3. Validation time counts too

50+20 minutes exceeds the 60-minute RTO; 25 minutes of data loss exceeds the 15-minute RPO. Starting a restore does not establish objective achievement.

This code is a local policy model; it does not replace target/cluster testing.

### Weak implementation
```js
s=>s.restoreStarted
```

### Corrected implementation
```js
s=>s.restoreStarted&&s.elapsedMinutes+s.validationMinutes<=s.rtoMinutes&&s.dataLossMinutes<=s.rpoMinutes&&s.dependenciesReady
```

### Synthetic input and actual model output
```json
{
  "before": [
    {
      "name": "within-objectives",
      "input": {
        "restoreStarted": true,
        "elapsedMinutes": 40,
        "validationMinutes": 15,
        "rtoMinutes": 60,
        "dataLossMinutes": 8,
        "rpoMinutes": 15,
        "dependenciesReady": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    },
    {
      "name": "validation-overrun",
      "input": {
        "restoreStarted": true,
        "elapsedMinutes": 50,
        "validationMinutes": 20,
        "rtoMinutes": 60,
        "dataLossMinutes": 8,
        "rpoMinutes": 15,
        "dependenciesReady": true
      },
      "expected": false,
      "observed": true,
      "matched": false
    },
    {
      "name": "data-loss-exceeded",
      "input": {
        "restoreStarted": true,
        "elapsedMinutes": 40,
        "validationMinutes": 15,
        "rtoMinutes": 60,
        "dataLossMinutes": 25,
        "rpoMinutes": 15,
        "dependenciesReady": true
      },
      "expected": false,
      "observed": true,
      "matched": false
    }
  ],
  "after": [
    {
      "name": "within-objectives",
      "input": {
        "restoreStarted": true,
        "elapsedMinutes": 40,
        "validationMinutes": 15,
        "rtoMinutes": 60,
        "dataLossMinutes": 8,
        "rpoMinutes": 15,
        "dependenciesReady": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    },
    {
      "name": "validation-overrun",
      "input": {
        "restoreStarted": true,
        "elapsedMinutes": 50,
        "validationMinutes": 20,
        "rtoMinutes": 60,
        "dataLossMinutes": 8,
        "rpoMinutes": 15,
        "dependenciesReady": true
      },
      "expected": false,
      "observed": false,
      "matched": true
    },
    {
      "name": "data-loss-exceeded",
      "input": {
        "restoreStarted": true,
        "elapsedMinutes": 40,
        "validationMinutes": 15,
        "rtoMinutes": 60,
        "dataLossMinutes": 25,
        "rpoMinutes": 15,
        "dependenciesReady": true
      },
      "expected": false,
      "observed": false,
      "matched": true
    }
  ]
}
```

## Exit evidence

- Data/identity/key/dependency restoration matrix
- Measured RTO/RPO and business-validation timeline

## References

- [Kubernetes · Operating etcd](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
- [Kubernetes · At-rest encryption](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/)