# Security review of AI-generated code

AI can accelerate work; authority over security decisions is retained. Evaluate generated code, package suggestions and self-authored tests with independent evidence.

Original AI-assisted SecNet educational content. Fictional local examples; not a standards provider publication, endorsement or compliance evidence. Page exercises do not create account progress.

Sources checked: 2026-10-21

## From claims to evidence

A “secure” comment is not a test result. Trace actor origin, tenant boundaries, parameter binding and error paths in generated code. Tests written by the same model may repeat the same implementation assumption. Derive acceptance criteria from the understood business rule first and independently design negative cases. Compare claimed security in comments with actual controls.

## Package and secret boundaries

A suggested package may not exist or may resemble a maliciously named package. Verify registry, maintainer, version, source and license before installing. Do not send production secrets, real customer records or private code to an unauthorized service. A small synthetic reproduction is often sufficient. Read the privileges and file effects of generated commands before execution.

## Human review and release

Generative AI does not remove required tests or code-owner review. Preserve traceability among change, model suggestion, independent tests and release approval. If sensitive input reaches an unauthorized service, deleting the message is insufficient; consider credential revocation/rotation and incident handling. The generative-AI profile on the NIST SSDF page is an additional reference; this workshop does not issue compliance evidence.

## Example review of a generated change

Suppose a model produces a ticket-update function and two green tests. First define expectations independently: owner/tenant stay unchanged, foreign actors cannot write, and stale versions conflict. If the function spreads the entire body into the record, a third case containing owner reveals the flaw despite those green tests. After correction, rerun permitted title updates and tenant separation as well as the original case. Verify the actual source of any suggested package before installation. Record which suggestion was accepted, which assumption failed and which independent evidence changed the decision; model confidence is not evidence.

## Working sequence

1. Derive independent acceptance criteria from the business rule.
2. Trace generated data and authority flows.
3. Verify package origin and input confidentiality.
4. Add independent negative tests.
5. Approve the same revision through code-owner review.

## 1. The model said it was secure

The model’s claim is neither server authorization nor a negative test. Require tenant and abuse tests as independent evidence.

This code is a local policy model; it does not replace target/device testing.

### Weak implementation
```js
s=>s.modelSaysSecure
```

### Corrected implementation
```js
s=>s.serverAuthorization&&s.tenantTest&&s.negativeTest
```

### Synthetic input and actual model output
```json
{
  "before": [
    {
      "name": "generated-claim",
      "input": {
        "modelSaysSecure": true,
        "serverAuthorization": false,
        "tenantTest": false,
        "negativeTest": false
      },
      "expected": false,
      "observed": true,
      "matched": false
    },
    {
      "name": "reviewed-change",
      "input": {
        "modelSaysSecure": true,
        "serverAuthorization": true,
        "tenantTest": true,
        "negativeTest": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    }
  ],
  "after": [
    {
      "name": "generated-claim",
      "input": {
        "modelSaysSecure": true,
        "serverAuthorization": false,
        "tenantTest": false,
        "negativeTest": false
      },
      "expected": false,
      "observed": false,
      "matched": true
    },
    {
      "name": "reviewed-change",
      "input": {
        "modelSaysSecure": true,
        "serverAuthorization": true,
        "tenantTest": true,
        "negativeTest": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    }
  ]
}
```

## 2. Installing a suggestion immediately

A model suggestion does not verify package origin. Check registry, maintainer, pinned version and license together; the model downloads no packages.

This code is a local policy model; it does not replace target/device testing.

### Weak implementation
```js
s=>s.suggestedByModel
```

### Corrected implementation
```js
s=>s.registryVerified&&s.maintainerReviewed&&s.versionPinned&&s.licenseReviewed
```

### Synthetic input and actual model output
```json
{
  "before": [
    {
      "name": "hallucinated-name",
      "input": {
        "suggestedByModel": true,
        "registryVerified": false,
        "maintainerReviewed": false,
        "versionPinned": false,
        "licenseReviewed": false
      },
      "expected": false,
      "observed": true,
      "matched": false
    },
    {
      "name": "verified-package",
      "input": {
        "suggestedByModel": true,
        "registryVerified": true,
        "maintainerReviewed": true,
        "versionPinned": true,
        "licenseReviewed": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    }
  ],
  "after": [
    {
      "name": "hallucinated-name",
      "input": {
        "suggestedByModel": true,
        "registryVerified": false,
        "maintainerReviewed": false,
        "versionPinned": false,
        "licenseReviewed": false
      },
      "expected": false,
      "observed": false,
      "matched": true
    },
    {
      "name": "verified-package",
      "input": {
        "suggestedByModel": true,
        "registryVerified": true,
        "maintainerReviewed": true,
        "versionPinned": true,
        "licenseReviewed": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    }
  ]
}
```

## 3. Self-confirming tests

Passing generated tests cannot justify release without independent cases, secret boundaries, required checks and human review.

This code is a local policy model; it does not replace target/device testing.

### Weak implementation
```js
s=>s.generatedTestsPass
```

### Corrected implementation
```js
s=>s.generatedTestsPass&&s.independentCases&&s.noSecrets&&s.requiredChecks&&s.humanReview
```

### Synthetic input and actual model output
```json
{
  "before": [
    {
      "name": "self-confirming-tests",
      "input": {
        "generatedTestsPass": true,
        "independentCases": false,
        "noSecrets": true,
        "requiredChecks": false,
        "humanReview": false
      },
      "expected": false,
      "observed": true,
      "matched": false
    },
    {
      "name": "independent-review",
      "input": {
        "generatedTestsPass": true,
        "independentCases": true,
        "noSecrets": true,
        "requiredChecks": true,
        "humanReview": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    },
    {
      "name": "secret-in-prompt",
      "input": {
        "generatedTestsPass": true,
        "independentCases": true,
        "noSecrets": false,
        "requiredChecks": true,
        "humanReview": true
      },
      "expected": false,
      "observed": true,
      "matched": false
    }
  ],
  "after": [
    {
      "name": "self-confirming-tests",
      "input": {
        "generatedTestsPass": true,
        "independentCases": false,
        "noSecrets": true,
        "requiredChecks": false,
        "humanReview": false
      },
      "expected": false,
      "observed": false,
      "matched": true
    },
    {
      "name": "independent-review",
      "input": {
        "generatedTestsPass": true,
        "independentCases": true,
        "noSecrets": true,
        "requiredChecks": true,
        "humanReview": true
      },
      "expected": true,
      "observed": true,
      "matched": true
    },
    {
      "name": "secret-in-prompt",
      "input": {
        "generatedTestsPass": true,
        "independentCases": true,
        "noSecrets": false,
        "requiredChecks": true,
        "humanReview": true
      },
      "expected": false,
      "observed": false,
      "matched": true
    }
  ]
}
```

## Exit evidence

- AI suggestion, human review and test trace
- Package verification and sensitive-input review record

## References

- [NIST SSDF publications: 1.1 final; 1.2 draft](https://csrc.nist.gov/projects/ssdf/publications)
- [OWASP ASVS 5.0.0](https://asvs.owasp.org/)
- [OWASP CI/CD Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/CI_CD_Security_Cheat_Sheet.html)