File created successfully at: /Users/steve.briggs/projects/IAC/docs/superpowers/plans/2026-05-05-vector-redaction-pipeline.md (file state is current in your context — no need to Read it back)

# Vector + OpenTelemetry Redaction Pipeline Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Insert Vector as an in-cluster aggregator between the OTel collector and Datadog, performing audit-log filtering and email redaction across logs, traces, and metrics.

**Architecture:** New `vector` Terraform internal-module (helm-release-based aggregator). Existing `otel-collector` internal-module gets a backwards-compatible v1.1.0 bump that swaps its terminal exporter from Datadog to OTLP→Vector when a `downstream_endpoint` is configured. The `eks-cluster-applications` layer-3 consumer wires both modules together via a new `vector.tf` and an edit to `otel_collector.tf`.

**Tech Stack:** Terraform (kubernetes + helm providers), upstream `timberio/vector` Helm chart in aggregator role, Vector VRL for transforms, External Secrets operator for the Datadog API key, Stakater Reloader for key-rotation pod restarts.

**Source spec:** `docs/superpowers/specs/2026-05-05-vector-redaction-design.md`

**Constraint:** This plan does **no real-cluster deployments**. Verification uses only read-only commands: `terraform fmt`, `terraform validate`, `terraform plan` (against `examples/`, no apply), `helm template`, `vector validate`, `vector vrl`. The actual rollout to clusters is operational work outside this plan.

**Phases:**
- **Phase A** — New `vector` internal-module (Tasks A1–A14). Tag `v1.0.0`.
- **Phase B** — `otel-collector` internal-module v1.1.0 bump (Tasks B1–B7). Tag `v1.1.0`.
- **Phase C** — `eks-cluster-applications` consumer wiring (Tasks C1–C5).

Phase A and Phase B are independent and can run in parallel. Phase C depends on `v1.0.0` and `v1.1.0` tags from A and B respectively.

**Pre-flight (one-time setup before any task):**
- Clone `internal-modules/kubernetes/monitoring/vector` (or create a new GitLab repo at that path if it does not exist — confirm with platform team).
- Clone `internal-modules/kubernetes/monitoring/otel-collector` and check out a feature branch from `v1.0.7`.
- Have the existing `eks-cluster-applications` directory checked out on a feature branch.
- Install `vector` CLI locally (`brew install vectordotdev/brew/vector` on macOS) for VRL unit tests.
- Install `helm` v3.x locally for `helm template` snapshot tests.

---

## Phase A — New `vector` internal-module

### Task A1: Bootstrap repo skeleton

**Files:**
- Create: `versions.tf`
- Create: `.gitignore`
- Create: `README.md`

- [ ] **Step 1: Verify or create the repo.** If the GitLab repo `infrastructure/terraform/internal-modules/kubernetes/monitoring/vector` does not exist, create it (empty) via the GitLab UI or CLI. Clone it to your dev machine.

- [ ] **Step 2: Write `versions.tf`** matching the convention used by the sibling `otel-collector` module:

```hcl
terraform {
  required_version = ">= 1.9.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.10"
    }
  }
}
```

- [ ] **Step 3: Write `.gitignore`:**

```
.terraform/
*.tfstate
*.tfstate.backup
*.tfvars
.terraform.lock.hcl
crash.log
```

- [ ] **Step 4: Write a stub `README.md`:**

```markdown
# vector

Terraform module wrapping the `timberio/vector` Helm chart in aggregator role.
Receives OTLP from the OTel collector, applies VRL transforms (audit-log
filtering, email redaction), and ships to Datadog.

See `docs/superpowers/specs/2026-05-05-vector-redaction-design.md` in the IAC repo
for design context.

## Usage

See `examples/basic/`.
```

- [ ] **Step 5: Run `terraform fmt`** to confirm syntax:

```bash
terraform fmt -check -recursive
```

Expected: exit 0, no output.

- [ ] **Step 6: Commit.**

```bash
git add versions.tf .gitignore README.md
git commit -m "chore: bootstrap vector internal-module skeleton"
```

---

### Task A2: Define module variables

**Files:**
- Create: `variables.tf`

- [ ] **Step 1: Write `variables.tf`** with every variable from the spec's §3.2 table:

```hcl
variable "chart_version" {
  description = "Vector helm chart version pin"
  type        = string
}

variable "image" {
  description = "Vector image source"
  type        = string
  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
}

variable "image_tag_version" {
  description = "Vector image tag"
  type        = string
}

variable "replicas" {
  description = "Number of Vector aggregator replicas"
  type        = number
  default     = 3
}

variable "datadog_site" {
  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
  type        = string
}

variable "datadog_api_key_vault_path" {
  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
  type        = string
}

variable "audit_drop_patterns" {
  description = "Regex patterns; matching log message bodies are dropped"
  type        = list(string)
  default = [
    "Audit\\s-\\s.+",
    "\"level\":\"audit\"",
  ]
}

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_replacement" {
  description = "String that replaces matched email addresses"
  type        = string
  default     = "[REDACTED-EMAIL]"
}

variable "additional_config" {
  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
  type        = any
  default     = null
}

variable "resources_requests" {
  description = "Pod resource requests"
  type        = map(string)
  default = {
    cpu    = "200m"
    memory = "256Mi"
  }
}

variable "resources_limits" {
  description = "Pod resource limits"
  type        = map(string)
  default = {
    cpu    = "2"
    memory = "1Gi"
  }
}

variable "namespace" {
  description = "Kubernetes namespace for Vector"
  type        = string
  default     = "vector"
}

variable "external_secrets_cluster_store_name" {
  description = "Name of the ClusterSecretStore created by external-secrets module"
  type        = string
}

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}
```

- [ ] **Step 2: Run `terraform fmt`** and `terraform validate`:

```bash
terraform fmt -check
terraform validate
```

`terraform validate` will fail with "no provider configuration found" — that's expected at this stage. You can satisfy it by adding a stub provider block in `versions.tf` or wait until Task A4. For now, accept the failure if it complains only about providers, not syntax.

- [ ] **Step 3: Commit.**

```bash
git add variables.tf
git commit -m "feat: declare vector module variables"
```

---

### Task A3: Define module outputs

**Files:**
- Create: `outputs.tf`

- [ ] **Step 1: Write `outputs.tf`:**

```hcl
output "namespace" {
  description = "Namespace Vector is deployed into"
  value       = var.namespace
}

output "service_endpoint_grpc" {
  description = "Cluster-internal OTLP/gRPC endpoint to point upstream collectors at"
  value       = "http://vector.${var.namespace}.svc.cluster.local:4317"
}

output "service_endpoint_http" {
  description = "Cluster-internal OTLP/HTTP endpoint"
  value       = "http://vector.${var.namespace}.svc.cluster.local:4318"
}
```

- [ ] **Step 2: Run `terraform fmt`** and confirm.

- [ ] **Step 3: Commit.**

```bash
git add outputs.tf
git commit -m "feat: expose vector module outputs"
```

---

### Task A4: Helm values template — sources and sinks (no transforms yet)

**Files:**
- Create: `templates/values.yaml.tpl`
- Create: `tests/snapshots/values-base.yaml`

This is a TDD task — golden-file snapshot test for the rendered helm values. Add transforms incrementally in subsequent tasks.

- [ ] **Step 1: Write the golden file** at `tests/snapshots/values-base.yaml` representing what the values should look like with NO transforms in the pipeline (smoke-test baseline):

```yaml
role: Aggregator
replicas: 3
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: "2"
    memory: 1Gi
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 127.0.0.1:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
  transforms: {}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - otlp_in.logs
      default_api_key: ${DATADOG_API_KEY}
      site: datadoghq.com
    datadog_traces:
      type: datadog_traces
      inputs:
        - otlp_in.traces
      default_api_key: ${DATADOG_API_KEY}
      site: datadoghq.com
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - otlp_in.metrics
      default_api_key: ${DATADOG_API_KEY}
      site: datadoghq.com
```

- [ ] **Step 2: Write `templates/values.yaml.tpl`** that should produce the above when rendered with the default fixture inputs:

```yaml
role: Aggregator
replicas: ${replicas}
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: ${requests_cpu}
    memory: ${requests_memory}
  limits:
    cpu: "${limits_cpu}"
    memory: ${limits_memory}
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 127.0.0.1:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
  transforms: {}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - otlp_in.logs
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_traces:
      type: datadog_traces
      inputs:
        - otlp_in.traces
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - otlp_in.metrics
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
```

Note: `$${...}` is the escape so Terraform's `templatefile()` emits literal `${...}` for Vector to expand at runtime.

- [ ] **Step 3: Write `tests/render-snapshot.sh`** — a small script that renders the template against the default fixture and diffs against the snapshot:

```bash
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
MODULE_DIR="$(dirname "$SCRIPT_DIR")"
RENDER_DIR="$(mktemp -d)"
trap 'rm -rf "$RENDER_DIR"' EXIT

cat > "$RENDER_DIR/main.tf" <<'EOF'
output "rendered" {
  value = templatefile("${path.module}/../../templates/values.yaml.tpl", {
    replicas        = 3
    requests_cpu    = "200m"
    requests_memory = "256Mi"
    limits_cpu      = "2"
    limits_memory   = "1Gi"
    datadog_site    = "datadoghq.com"
  })
}
EOF
ln -s "$MODULE_DIR/templates" "$RENDER_DIR/../templates"
( cd "$RENDER_DIR" && terraform init -backend=false >/dev/null && \
  terraform apply -auto-approve >/dev/null && \
  terraform output -raw rendered ) > "$SCRIPT_DIR/snapshots/.actual.yaml"

diff -u "$SCRIPT_DIR/snapshots/values-base.yaml" "$SCRIPT_DIR/snapshots/.actual.yaml"
echo "snapshot OK"
```

`chmod +x tests/render-snapshot.sh`.

- [ ] **Step 4: Run the snapshot test, expect it to fail or pass.**

```bash
./tests/render-snapshot.sh
```

If the snapshot matches the rendered output: pass. If diff complains, **inspect the diff** and decide whether the template or the snapshot needs adjustment — they should align on first try if the template was written correctly, but small whitespace/quoting differences are common. Update whichever is wrong until they match.

- [ ] **Step 5: Commit.**

```bash
git add templates/ tests/
git commit -m "feat: helm values template — sources, sinks, no transforms"
```

---

### Task A5: VRL — `drop_audit` filter transform

**Files:**
- Modify: `templates/values.yaml.tpl`
- Modify: `tests/snapshots/values-base.yaml`
- Create: `tests/vrl/drop_audit/program.vrl`
- Create: `tests/vrl/drop_audit/inputs.ndjson`
- Create: `tests/vrl/drop_audit/expected.ndjson`
- Create: `tests/vrl/drop_audit/run.sh`

- [ ] **Step 1: Write the VRL test program** at `tests/vrl/drop_audit/program.vrl` (extracts the condition from the filter into a standalone VRL that returns true to keep, false to drop — same semantics as Vector's `filter`):

```vrl
msg = string!(.message ?? "")
. = { "kept": !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"') }
```

- [ ] **Step 2: Write inputs** at `tests/vrl/drop_audit/inputs.ndjson` (one JSON object per line):

```json
{"message": "Audit - login by user@example.com"}
{"message": "INFO request handled"}
{"message": "{\"level\":\"audit\",\"action\":\"foo\"}"}
{"message": "{\"level\":\"info\",\"msg\":\"audit team meeting\"}"}
{"message": ""}
```

- [ ] **Step 3: Write expected outputs** at `tests/vrl/drop_audit/expected.ndjson`:

```json
{"kept": false}
{"kept": true}
{"kept": false}
{"kept": true}
{"kept": true}
```

Note: line 4 contains the literal string "audit" but not the JSON-shape `"level":"audit"`, so it's kept.

- [ ] **Step 4: Write the test runner** at `tests/vrl/drop_audit/run.sh`:

```bash
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
: > "$ACTUAL"
while IFS= read -r line; do
  echo "$line" | vector vrl --no-prompt -p "$DIR/program.vrl" >> "$ACTUAL"
done < "$DIR/inputs.ndjson"
diff -u "$DIR/expected.ndjson" "$ACTUAL"
echo "drop_audit VRL OK"
```

`chmod +x tests/vrl/drop_audit/run.sh`.

- [ ] **Step 5: Run the test, expect PASS.**

```bash
./tests/vrl/drop_audit/run.sh
```

If `vector vrl` is not in PATH, install it (`brew install vectordotdev/brew/vector` or download from vector.dev). If the command-line flags differ in your installed version (Vector occasionally renames flags), adjust the runner. The semantic test (does the VRL drop the right inputs?) is what matters.

- [ ] **Step 6: Add the transform to `templates/values.yaml.tpl`.** Replace the line `transforms: {}` with:

```yaml
  transforms:
%{ for i, p in audit_drop_patterns ~}
%{ if i == 0 ~}
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
      condition: |
        msg = string!(.message ?? "")
        !match(msg, r'${p}')%{ for q in slice(audit_drop_patterns, 1, length(audit_drop_patterns)) } && !match(msg, r'${q}')%{ endfor }
%{ endif ~}
%{ endfor ~}
```

That's awkward — the simpler approach is to compose the condition string in Terraform and template it as a single value. **Replace the above with this cleaner approach:**

```yaml
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
      condition: |
        msg = string!(.message ?? "")
        ${audit_drop_condition}
```

…and pass `audit_drop_condition` from `main.tf` as a pre-built string. Defer constructing it until Task A11; for now hardcode the single condition for the default two patterns:

```yaml
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
      condition: |
        msg = string!(.message ?? "")
        !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"')
```

- [ ] **Step 7: Update `tests/snapshots/values-base.yaml`** — replace `transforms: {}` with the same block above.

- [ ] **Step 8: Run snapshot test.**

```bash
./tests/render-snapshot.sh
```

Expected: PASS. If it fails, reconcile whitespace between snapshot and template.

- [ ] **Step 9: Commit.**

```bash
git add templates/ tests/
git commit -m "feat: drop_audit filter transform with VRL unit test"
```

---

### Task A6: VRL — `redact_emails_logs` transform

**Files:**
- Modify: `templates/values.yaml.tpl`
- Modify: `tests/snapshots/values-base.yaml`
- Create: `tests/vrl/redact_emails_logs/{program.vrl,inputs.ndjson,expected.ndjson,run.sh}`

- [ ] **Step 1: Write VRL test program** at `tests/vrl/redact_emails_logs/program.vrl`:

```vrl
email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

if exists(.message) {
  .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
}

if is_object(.attributes) {
  .attributes = map_values(object!(.attributes)) -> |v| {
    if is_string(v) { replace(v, email_re, "[REDACTED-EMAIL]") } else { v }
  }
}
.
```

- [ ] **Step 2: Write inputs** at `tests/vrl/redact_emails_logs/inputs.ndjson`:

```json
{"message": "user alice@example.com signed in"}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}}
{"message": "json blob {\"email\":\"x@y.com\"}"}
```

- [ ] **Step 3: Write expected outputs** at `tests/vrl/redact_emails_logs/expected.ndjson`:

```json
{"message": "user [REDACTED-EMAIL] signed in"}
{"message": "no email here"}
{"message": "multiple [REDACTED-EMAIL] and [REDACTED-EMAIL]"}
{"message": "ok", "attributes": {"user.email": "[REDACTED-EMAIL]", "request.id": "abc-123"}}
{"message": "json blob {\"email\":\"[REDACTED-EMAIL]\"}"}
```

- [ ] **Step 4: Write `run.sh`** mirroring Task A5 Step 4 with paths swapped:

```bash
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
: > "$ACTUAL"
while IFS= read -r line; do
  echo "$line" | vector vrl --no-prompt -p "$DIR/program.vrl" >> "$ACTUAL"
done < "$DIR/inputs.ndjson"
diff -u "$DIR/expected.ndjson" "$ACTUAL"
echo "redact_emails_logs VRL OK"
```

`chmod +x tests/vrl/redact_emails_logs/run.sh`.

- [ ] **Step 5: Run, expect PASS.**

```bash
./tests/vrl/redact_emails_logs/run.sh
```

- [ ] **Step 6: Add the transform to `templates/values.yaml.tpl`** by extending the `transforms` block:

```yaml
    redact_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if exists(.message) {
          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
        }
        if is_object(.attributes) {
          .attributes = map_values(object!(.attributes)) -> |v| {
            if is_string(v) { replace(v, email_re, "${email_redaction_replacement}") } else { v }
          }
        }
```

- [ ] **Step 7: Update sinks block** to wire the logs sink to the new transform:

```yaml
    datadog_logs:
      type: datadog_logs
      inputs:
        - redact_emails_logs   # was: otlp_in.logs
```

- [ ] **Step 8: Update `tests/snapshots/values-base.yaml`** to match.

- [ ] **Step 9: Run the snapshot test, expect PASS.**

```bash
./tests/render-snapshot.sh
```

- [ ] **Step 10: Commit.**

```bash
git add templates/ tests/
git commit -m "feat: redact_emails_logs remap transform with VRL unit test"
```

---

### Task A7: VRL — `redact_emails_traces` transform

**Files:**
- Modify: `templates/values.yaml.tpl`
- Modify: `tests/snapshots/values-base.yaml`
- Create: `tests/vrl/redact_emails_traces/{program.vrl,inputs.ndjson,expected.ndjson,run.sh}`

The OTLP-source span shape in Vector exposes `attributes` for span-level attributes and `resource.attributes` for resource-level. Walk both.

- [ ] **Step 1: Write VRL program** at `tests/vrl/redact_emails_traces/program.vrl`:

```vrl
email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

scrub = |v| {
  if is_string(v) { replace(v, email_re, "[REDACTED-EMAIL]") } else { v }
}

if is_object(.attributes) {
  .attributes = map_values(object!(.attributes)) -> scrub
}
if is_object(.resource) && is_object(.resource.attributes) {
  .resource.attributes = map_values(object!(.resource.attributes)) -> scrub
}
.
```

- [ ] **Step 2: Inputs** at `tests/vrl/redact_emails_traces/inputs.ndjson`:

```json
{"name": "POST /signup", "attributes": {"user.email": "new@x.com", "http.status": 200}, "resource": {"attributes": {"service.name": "web", "owner": "team@x.com"}}}
{"name": "GET /health", "attributes": {"http.status": 200}, "resource": {"attributes": {"service.name": "web"}}}
```

- [ ] **Step 3: Expected outputs:**

```json
{"name": "POST /signup", "attributes": {"user.email": "[REDACTED-EMAIL]", "http.status": 200}, "resource": {"attributes": {"service.name": "web", "owner": "[REDACTED-EMAIL]"}}}
{"name": "GET /health", "attributes": {"http.status": 200}, "resource": {"attributes": {"service.name": "web"}}}
```

- [ ] **Step 4: `run.sh`** identical structure to Task A5 Step 4 (paths swapped). `chmod +x`.

- [ ] **Step 5: Run, expect PASS.**

```bash
./tests/vrl/redact_emails_traces/run.sh
```

- [ ] **Step 6: Add transform to `templates/values.yaml.tpl`:**

```yaml
    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        scrub = |v| {
          if is_string(v) { replace(v, email_re, "${email_redaction_replacement}") } else { v }
        }
        if is_object(.attributes) {
          .attributes = map_values(object!(.attributes)) -> scrub
        }
        if is_object(.resource) && is_object(.resource.attributes) {
          .resource.attributes = map_values(object!(.resource.attributes)) -> scrub
        }
```

- [ ] **Step 7: Update traces sink:**

```yaml
    datadog_traces:
      type: datadog_traces
      inputs:
        - redact_emails_traces
```

- [ ] **Step 8: Update snapshot, run snapshot test, commit.**

```bash
./tests/render-snapshot.sh
git add templates/ tests/
git commit -m "feat: redact_emails_traces remap transform with VRL unit test"
```

---

### Task A8: VRL — `redact_emails_metrics` transform

**Files:**
- Modify: `templates/values.yaml.tpl`
- Modify: `tests/snapshots/values-base.yaml`
- Create: `tests/vrl/redact_emails_metrics/{program.vrl,inputs.ndjson,expected.ndjson,run.sh}`

Metric attributes live under `.tags` (Datadog metric shape) or `.attributes` (OTLP shape) depending on Vector's internal representation. For OTLP source, `.attributes` is correct.

- [ ] **Step 1–5: Write VRL program, inputs, expected, run.sh, run.** Same structure as Task A7, with metric-shaped inputs:

`program.vrl` (same body as A7 — just attribute walking):

```vrl
email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

scrub = |v| {
  if is_string(v) { replace(v, email_re, "[REDACTED-EMAIL]") } else { v }
}

if is_object(.attributes) {
  .attributes = map_values(object!(.attributes)) -> scrub
}
if is_object(.resource) && is_object(.resource.attributes) {
  .resource.attributes = map_values(object!(.resource.attributes)) -> scrub
}
.
```

`inputs.ndjson`:

```json
{"name": "request.count", "value": 42, "attributes": {"submitted_by": "ops@x.com"}}
{"name": "cpu.usage", "value": 0.5, "attributes": {"host": "node-1"}}
```

`expected.ndjson`:

```json
{"name": "request.count", "value": 42, "attributes": {"submitted_by": "[REDACTED-EMAIL]"}}
{"name": "cpu.usage", "value": 0.5, "attributes": {"host": "node-1"}}
```

`run.sh` — same shape, `chmod +x`. Run, expect PASS.

- [ ] **Step 6: Add transform to template** (identical to A7 step 6 but with metrics inputs/outputs).

```yaml
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        scrub = |v| {
          if is_string(v) { replace(v, email_re, "${email_redaction_replacement}") } else { v }
        }
        if is_object(.attributes) {
          .attributes = map_values(object!(.attributes)) -> scrub
        }
        if is_object(.resource) && is_object(.resource.attributes) {
          .resource.attributes = map_values(object!(.resource.attributes)) -> scrub
        }
```

- [ ] **Step 7: Update metrics sink:**

```yaml
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - redact_emails_metrics
```

- [ ] **Step 8: Update snapshot, run snapshot test, commit.**

```bash
./tests/render-snapshot.sh
git add templates/ tests/
git commit -m "feat: redact_emails_metrics remap transform with VRL unit test"
```

---

### Task A9: Conditional template — toggling email redaction off

**Goal:** When `email_redaction_enabled = false` in Terraform, the three `redact_emails_*` transforms should NOT appear and sinks should consume directly from `drop_audit` / `otlp_in.traces` / `otlp_in.metrics`.

**Files:**
- Modify: `templates/values.yaml.tpl`
- Create: `tests/snapshots/values-no-redaction.yaml`
- Modify: `tests/render-snapshot.sh` (or create a second runner)

- [ ] **Step 1: Write the no-redaction snapshot** at `tests/snapshots/values-no-redaction.yaml` — same as `values-base.yaml` but with the three `redact_emails_*` transform blocks removed and sinks reading from `drop_audit` / `otlp_in.traces` / `otlp_in.metrics`.

- [ ] **Step 2: Refactor `templates/values.yaml.tpl`** to gate the three transforms on a templated boolean. Use Terraform's `%{ if ... }` directives:

```yaml
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
      condition: |
        msg = string!(.message ?? "")
        !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"')
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        ...
    redact_emails_traces:
      ...
    redact_emails_metrics:
      ...
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_logs" : "drop_audit" }
      ...
    datadog_traces:
      type: datadog_traces
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces" }
      ...
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics" }
      ...
```

- [ ] **Step 3: Update `tests/render-snapshot.sh`** to run TWO renders, one with `email_redaction_enabled = true` (existing — diffs against `values-base.yaml`) and one with `email_redaction_enabled = false` (new — diffs against `values-no-redaction.yaml`). The current script renders one; extend it to a loop or duplicate logic.

- [ ] **Step 4: Run, expect PASS for both.**

```bash
./tests/render-snapshot.sh
```

- [ ] **Step 5: Commit.**

```bash
git add templates/ tests/
git commit -m "feat: gate email redaction transforms on email_redaction_enabled"
```

---

### Task A10: Conditional template — `audit_drop_patterns` from variable

**Goal:** Currently `templates/values.yaml.tpl` hardcodes the two audit patterns. Replace with a Terraform-side composed condition string.

**Files:**
- Modify: `templates/values.yaml.tpl`
- Modify: caller of `templatefile()` in `main.tf` (does not exist yet — Task A12 will create it; for now, just adjust the test fixture).

- [ ] **Step 1: Update `templates/values.yaml.tpl`** to consume a single pre-composed string `${audit_drop_condition}`:

```yaml
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
      condition: |
        msg = string!(.message ?? "")
        ${audit_drop_condition}
```

- [ ] **Step 2: Update the snapshot test fixtures.** In `tests/render-snapshot.sh`, in the inline `main.tf` heredoc, build `audit_drop_condition` from the patterns:

```hcl
locals {
  patterns = ["Audit\\s-\\s.+", "\"level\":\"audit\""]
  audit_drop_condition = join(
    " && ",
    [for p in local.patterns : "!match(msg, r'${p}')"],
  )
}

output "rendered" {
  value = templatefile("${path.module}/../../templates/values.yaml.tpl", {
    replicas                    = 3
    requests_cpu                = "200m"
    requests_memory             = "256Mi"
    limits_cpu                  = "2"
    limits_memory               = "1Gi"
    datadog_site                = "datadoghq.com"
    audit_drop_condition        = local.audit_drop_condition
    email_redaction_enabled     = true
    email_redaction_replacement = "[REDACTED-EMAIL]"
  })
}
```

The snapshot file `values-base.yaml` should still match because the composed condition produces the same condition body.

- [ ] **Step 3: Run snapshot test, expect PASS.**

```bash
./tests/render-snapshot.sh
```

- [ ] **Step 4: Commit.**

```bash
git add templates/ tests/
git commit -m "feat: parameterize audit_drop_patterns via composed VRL condition"
```

---

### Task A11: Add a third audit pattern in test fixtures (negative test for parameterization)

Make sure parameterization actually works — add a third pattern in fixtures and confirm the rendered condition includes it.

**Files:**
- Modify: `tests/render-snapshot.sh` — add a third snapshot run with three patterns
- Create: `tests/snapshots/values-three-patterns.yaml`

- [ ] **Step 1: Write `tests/snapshots/values-three-patterns.yaml`** — copy of `values-base.yaml`, change the `condition:` block to:

```yaml
      condition: |
        msg = string!(.message ?? "")
        !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"') && !match(msg, r'pattern_three')
```

- [ ] **Step 2: Update `tests/render-snapshot.sh`** to render a third time with `patterns = ["Audit\\s-\\s.+", "\"level\":\"audit\"", "pattern_three"]` and diff against `values-three-patterns.yaml`.

- [ ] **Step 3: Run, expect PASS.**

```bash
./tests/render-snapshot.sh
```

- [ ] **Step 4: Commit.**

```bash
git add tests/
git commit -m "test: snapshot for three-pattern audit_drop configuration"
```

---

### Task A12: Module `main.tf` — namespace + ExternalSecret + helm release

**Files:**
- Create: `main.tf`

- [ ] **Step 1: Write `main.tf`:**

```hcl
locals {
  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
  )

  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
    "${path.module}/templates/values.yaml.tpl",
    {
      replicas                    = var.replicas
      requests_cpu                = var.resources_requests.cpu
      requests_memory             = var.resources_requests.memory
      limits_cpu                  = var.resources_limits.cpu
      limits_memory               = var.resources_limits.memory
      datadog_site                = var.datadog_site
      audit_drop_condition        = local.audit_drop_condition
      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_replacement = var.email_redaction_replacement
    },
  )
}

resource "kubernetes_namespace" "vector" {
  metadata {
    name = var.namespace
    labels = {
      "app.kubernetes.io/name"      = "vector"
      "app.kubernetes.io/component" = "telemetry-aggregator"
    }
  }
}

resource "kubernetes_manifest" "datadog_external_secret" {
  manifest = {
    apiVersion = "external-secrets.io/v1beta1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "vector-datadog"
      namespace = kubernetes_namespace.vector.metadata[0].name
    }
    spec = {
      refreshInterval = "1h"
      secretStoreRef = {
        kind = "ClusterSecretStore"
        name = var.external_secrets_cluster_store_name
      }
      target = {
        name           = "vector-datadog"
        creationPolicy = "Owner"
      }
      data = [{
        secretKey = "api_key"
        remoteRef = {
          key      = var.datadog_api_key_vault_path
          property = "api_key"
        }
      }]
    }
  }

  wait {
    fields = {
      "status.conditions[0].type"   = "Ready"
      "status.conditions[0].status" = "True"
    }
  }

  timeouts {
    create = "5m"
    update = "5m"
  }
}

resource "helm_release" "vector" {
  name       = "vector"
  namespace  = kubernetes_namespace.vector.metadata[0].name
  repository = "https://helm.vector.dev"
  chart      = "vector"
  version    = var.chart_version

  values = [local.rendered_values]

  set {
    name  = "image.repository"
    value = var.image
  }
  set {
    name  = "image.tag"
    value = var.image_tag_version
  }

  depends_on = [
    kubernetes_manifest.datadog_external_secret,
  ]
}
```

- [ ] **Step 2: Run `terraform fmt`** and `terraform validate`. `validate` may complain about unconfigured providers — that's expected without a wrapping configuration. Address by adding a stub provider config in `examples/basic/` (Task A13).

- [ ] **Step 3: Commit.**

```bash
git add main.tf
git commit -m "feat: namespace, ExternalSecret with wait, helm release"
```

---

### Task A13: Examples directory + plan-only validation

**Files:**
- Create: `examples/basic/main.tf`
- Create: `examples/basic/versions.tf`
- Create: `examples/basic/README.md`

- [ ] **Step 1: Write `examples/basic/versions.tf`:**

```hcl
terraform {
  required_version = ">= 1.9.0"
  required_providers {
    kubernetes = { source = "hashicorp/kubernetes", version = ">= 2.20" }
    helm       = { source = "hashicorp/helm",       version = ">= 2.10" }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}
```

- [ ] **Step 2: Write `examples/basic/main.tf`:**

```hcl
module "vector" {
  source = "../.."

  chart_version              = "0.36.0"
  image_tag_version          = "0.40.0-distroless-libc"
  datadog_site               = "datadoghq.com"
  datadog_api_key_vault_path = "secret/data/datadog/api-key"

  external_secrets_cluster_store_name = "vault-cluster-store"
}
```

- [ ] **Step 3: Write `examples/basic/README.md`:**

```markdown
# basic example

Minimal usage of the `vector` module. Used by CI for `terraform validate` /
`terraform plan` smoke testing — does not deploy.

```bash
terraform init -backend=false
terraform validate
```
```

- [ ] **Step 4: Run validation:**

```bash
cd examples/basic
terraform init -backend=false
terraform validate
```

Expected: `Success! The configuration is valid.`

- [ ] **Step 5: Commit.**

```bash
cd ../..
git add examples/
git commit -m "chore: examples/basic for terraform validate"
```

---

### Task A14: CI configuration + tag v1.0.0

**Files:**
- Create: `.gitlab-ci.yml` (assuming GitLab CI per `gitlab.infrastructure.immuta.io` convention)
- Modify: `README.md` — flesh out usage docs

- [ ] **Step 1: Write `.gitlab-ci.yml`:**

```yaml
stages: [lint, test]

terraform-fmt:
  stage: lint
  image: hashicorp/terraform:1.9
  script:
    - terraform fmt -check -recursive

terraform-validate:
  stage: test
  image: hashicorp/terraform:1.9
  script:
    - cd examples/basic
    - terraform init -backend=false
    - terraform validate

snapshot-tests:
  stage: test
  image: hashicorp/terraform:1.9
  script:
    - apk add --no-cache bash diffutils
    - ./tests/render-snapshot.sh

vrl-tests:
  stage: test
  image: timberio/vector:latest-debian
  script:
    - apt-get update && apt-get install -y bash diffutils
    - ./tests/vrl/drop_audit/run.sh
    - ./tests/vrl/redact_emails_logs/run.sh
    - ./tests/vrl/redact_emails_traces/run.sh
    - ./tests/vrl/redact_emails_metrics/run.sh
```

- [ ] **Step 2: Flesh out `README.md`** with full module input/output table, usage example, link to spec.

- [ ] **Step 3: Push, confirm CI green.**

```bash
git push origin <feature-branch>
```

Wait for CI. If any job fails, fix and re-push.

- [ ] **Step 4: Open MR, review, merge to main.**

- [ ] **Step 5: Tag.**

```bash
git checkout main
git pull
git tag v1.0.0
git push origin v1.0.0
```

---

## Phase B — `otel-collector` v1.0.7 → v1.1.0 bump

### Task B1: Pre-flight — read the existing module

**Files:** none modified yet.

The bump is backwards-compatible. Before changing anything, build a complete picture of how the module currently renders helm values. Specifically:

- [ ] **Step 1: Clone the module:**

```bash
git clone git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/otel-collector.git
cd otel-collector
git checkout -b feature/v1.1.0-vector-downstream v1.0.7
```

- [ ] **Step 2: Inventory the repo.**

```bash
ls -la
```

Note especially: `main.tf`, `variables.tf`, `outputs.tf`, `templates/`, `tests/`. Open each and read it.

- [ ] **Step 3: Identify where the `exporters.datadog` block is rendered.** Likely in `templates/values.yaml.tpl` or similar. Identify how the Datadog API key is mounted (env var? secret volume? helm chart's built-in `agents.useHostNetwork` knob?).

- [ ] **Step 4: Identify the existing snapshot tests, if any.** If there are no snapshot tests, you'll need to write one for the existing v1.0.7 behavior to lock in baseline before adding new behavior.

- [ ] **Step 5: Document findings** in a scratch `B1-findings.md` (gitignored or in a private branch). Keep this in your working memory; you'll reference it heavily in the next tasks.

No commit at this task — discovery only.

---

### Task B2: Lock baseline with a v1.0.7-shape snapshot test

**Files:**
- Create: `tests/snapshots/values-direct-datadog.yaml` (or modify existing snapshot)
- Create or extend: `tests/render-snapshot.sh`

If the module already has snapshot tests for the v1.0.7 rendering: skip writing a new one — but confirm one exists and runs green at the current state.

- [ ] **Step 1: Capture current rendered output** by running existing snapshot test or, if none exists, writing one that mirrors the structure from Phase A Task A4. The snapshot represents the rendered helm values when no `downstream_endpoint` is set.

- [ ] **Step 2: Run the snapshot test, expect PASS.** This is your safety net: any future change must preserve this rendering when `downstream_endpoint = ""`.

- [ ] **Step 3: Commit.**

```bash
git add tests/
git commit -m "test: lock baseline snapshot for direct-Datadog mode"
```

---

### Task B3: Add `downstream_endpoint` and `downstream_protocol` variables

**Files:**
- Modify: `variables.tf`

- [ ] **Step 1: Append to `variables.tf`:**

```hcl
variable "downstream_endpoint" {
  description = "When non-empty, swap the terminal exporter from datadog to OTLP, pointed at this endpoint. Used to route through Vector for redaction."
  type        = string
  default     = ""
}

variable "downstream_protocol" {
  description = "Protocol for downstream OTLP exporter. Currently only otlp/grpc supported."
  type        = string
  default     = "otlp/grpc"
  validation {
    condition     = contains(["otlp/grpc", "otlp/http"], var.downstream_protocol)
    error_message = "downstream_protocol must be one of: otlp/grpc, otlp/http"
  }
}
```

- [ ] **Step 2: Run `terraform fmt -check`.**

- [ ] **Step 3: Commit.**

```bash
git add variables.tf
git commit -m "feat: add downstream_endpoint and downstream_protocol variables"
```

---

### Task B4: Conditional OTLP exporter rendering

**Files:**
- Modify: `templates/values.yaml.tpl` (path may differ — adjust to module's actual layout)
- Create: `tests/snapshots/values-downstream-otlp.yaml`

- [ ] **Step 1: Write the new downstream snapshot.** Copy `values-direct-datadog.yaml` to `values-downstream-otlp.yaml`. In the new file:
  - Replace the `exporters.datadog` block with:

    ```yaml
    exporters:
      otlp:
        endpoint: http://vector.vector.svc.cluster.local:4317
        tls:
          insecure: true
        sending_queue:
          enabled: true
          queue_size: 5000
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_interval: 30s
          max_elapsed_time: 300s
    ```

  - In all three `service.pipelines.{logs,traces,metrics}.exporters` lists, replace `[datadog]` with `[otlp]`.
  - Remove the Datadog API key env var / secret volume mount.

- [ ] **Step 2: Edit `templates/values.yaml.tpl`** to gate the existing Datadog exporter on `downstream_endpoint == ""` and add a parallel OTLP block when non-empty:

```yaml
exporters:
%{ if downstream_endpoint == "" ~}
  datadog:
    api:
      key: $${DATADOG_API_KEY}
      site: ${datadog_site}
%{ else ~}
  otlp:
    endpoint: ${downstream_endpoint}
    tls:
      insecure: true
    sending_queue:
      enabled: true
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
%{ endif ~}

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [...]
      exporters: [${ downstream_endpoint == "" ? "datadog" : "otlp" }]
    traces:
      receivers: [otlp]
      processors: [...]
      exporters: [${ downstream_endpoint == "" ? "datadog" : "otlp" }]
    metrics:
      receivers: [otlp]
      processors: [...]
      exporters: [${ downstream_endpoint == "" ? "datadog" : "otlp" }]
```

(Replace the `[...]` placeholders with whatever the existing template has — preserve current processors lists unchanged.)

- [ ] **Step 3: Gate the Datadog API key mount** on `downstream_endpoint == ""`. The exact location depends on the module — likely in a `pod.env` block or chart values. Wrap it in `%{ if downstream_endpoint == "" ~} ... %{ endif ~}`.

- [ ] **Step 4: Update `tests/render-snapshot.sh`** to render twice: once with `downstream_endpoint = ""` (existing snapshot), once with `downstream_endpoint = "http://vector.vector.svc.cluster.local:4317"` (new snapshot).

- [ ] **Step 5: Run snapshot tests, expect both PASS.**

```bash
./tests/render-snapshot.sh
```

If the direct-Datadog snapshot fails, you've broken backwards compatibility. Stop and fix.

- [ ] **Step 6: Commit.**

```bash
git add templates/ tests/
git commit -m "feat: conditional OTLP exporter when downstream_endpoint is set"
```

---

### Task B5: Validate against `examples/`

**Files:**
- Modify: existing `examples/basic/main.tf` (or whatever the module's example is named) — add a second example variant
- Create: `examples/downstream/main.tf`

- [ ] **Step 1: Create `examples/downstream/main.tf`:**

```hcl
module "otel_collector" {
  source = "../.."

  chart_version    = "0.85.0"   # pin to the actual version your repo uses
  image_tag_version = "0.85.0"
  otel_lb_replicas         = 3
  otel_collector_replicas  = 3
  otellb_image             = "registry.ic-ops.net/cache-docker-hub/otel/opentelemetry-collector-contrib"
  otel_image               = "registry.ic-ops.net/cache-docker-hub/otel/opentelemetry-collector-contrib"

  downstream_endpoint = "http://vector.vector.svc.cluster.local:4317"
}
```

(Pad with whatever required-variables the existing module needs; copy from `examples/basic`.)

- [ ] **Step 2: Run validate:**

```bash
cd examples/downstream
terraform init -backend=false
terraform validate
```

Expected: success.

- [ ] **Step 3: Commit.**

```bash
cd ../..
git add examples/
git commit -m "chore: examples/downstream for downstream-mode validation"
```

---

### Task B6: README updates

**Files:**
- Modify: `README.md`

- [ ] **Step 1: Add a "Downstream-mode" section** to the README documenting `downstream_endpoint`, `downstream_protocol`, the implication for the Datadog API key (no longer mounted in this mode), and a usage snippet pointing at Vector.

- [ ] **Step 2: Commit.**

```bash
git add README.md
git commit -m "docs: README for downstream-mode usage"
```

---

### Task B7: CI green, MR, merge, tag v1.1.0

- [ ] **Step 1: Push, wait for CI.**

```bash
git push origin feature/v1.1.0-vector-downstream
```

- [ ] **Step 2: Open MR. Reviewers check that:**
  - Direct-Datadog snapshot is byte-identical to v1.0.7.
  - New downstream-OTLP snapshot matches the spec's §5.1.
  - Datadog API key mount is conditionally removed.

- [ ] **Step 3: Merge to main, tag v1.1.0.**

```bash
git checkout main
git pull
git tag v1.1.0
git push origin v1.1.0
```

---

## Phase C — `eks-cluster-applications` consumer wiring

**Pre-flight:** Confirm both `internal-modules/kubernetes/monitoring/vector v1.0.0` and `internal-modules/kubernetes/monitoring/otel-collector v1.1.0` tags are pushed and pullable.

### Task C1: Create `vector.tf` — locals + module call

**Files:**
- Create: `terraform/iac-library/aws/compute/eks-cluster-applications/vector.tf`

- [ ] **Step 1: Write `vector.tf`:**

```hcl
# ---------------------------------------------------------------------------------------------------------------------
# VECTOR
# ---------------------------------------------------------------------------------------------------------------------
locals {
  vector_enabled = (
    var.vector_enabled != null
    ? var.vector_enabled
    : lookup(var.vector_enabled_cluster_types, local.cluster_type, true)
  )
  vector_endpoint = "http://vector.vector.svc.cluster.local:4317"
}

module "vector" {
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"
  count  = local.vector_enabled ? 1 : 0

  providers = {
    kubernetes = kubernetes.local-cluster
    helm       = helm.local-cluster
  }

  chart_version               = var.vector_chart_version
  image                       = var.vector_image
  image_tag_version           = var.vector_image_tag
  replicas                    = var.vector_replicas
  datadog_site                = var.vector_datadog_site
  datadog_api_key_vault_path  = var.vector_datadog_api_key_vault_path
  audit_drop_patterns         = var.vector_audit_drop_patterns
  email_redaction_enabled     = var.vector_email_redaction_enabled
  email_redaction_replacement = var.vector_email_redaction_replacement
  additional_config           = var.vector_additional_config

  external_secrets_cluster_store_name = var.external_secrets_vault_secret_name

  depends_on = [
    module.cilium,
    module.external_secrets,
  ]
}

data "external" "vector_validate" {
  count = local.vector_enabled && !local.external_secrets_enabled ? 1 : 0

  program = ["Error: vector can only be set if external-secrets is enabled as well"]
}

# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS (when vector enabled)
# ---------------------------------------------------------------------------------------------------------------------
variable "vector_chart_version" {
  description = "Version of vector helm chart"
  type        = string
  default     = ""
  validation {
    condition     = var.vector_chart_version == "" && local.vector_enabled ? false : true
    error_message = "vector_chart_version must be set when vector is enabled"
  }
}

variable "vector_image_tag" {
  description = "Vector image tag"
  type        = string
  default     = ""
  validation {
    condition     = var.vector_image_tag == "" && local.vector_enabled ? false : true
    error_message = "vector_image_tag must be set when vector is enabled"
  }
}

variable "vector_datadog_site" {
  description = "Datadog site for vector sinks (e.g. datadoghq.com)"
  type        = string
  default     = ""
  validation {
    condition     = var.vector_datadog_site == "" && local.vector_enabled ? false : true
    error_message = "vector_datadog_site must be set when vector is enabled"
  }
}

variable "vector_datadog_api_key_vault_path" {
  description = "Vault path for the Datadog API key (consumed by ExternalSecret in vector ns)"
  type        = string
  default     = ""
  validation {
    condition     = var.vector_datadog_api_key_vault_path == "" && local.vector_enabled ? false : true
    error_message = "vector_datadog_api_key_vault_path must be set when vector is enabled"
  }
}

# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# ---------------------------------------------------------------------------------------------------------------------
variable "vector_enabled_cluster_types" {
  description = "Types of clusters Vector should be enabled on by default"
  type        = map(bool)
  default = {
    "mgmt"  = true
    "cps"   = true
    "saas"  = true
    "infra" = false
    "dev"   = false
  }
}

variable "vector_enabled" {
  description = "Override for explicitly enabling or disabling Vector"
  type        = bool
  default     = null
}

variable "vector_image" {
  description = "Image for Vector"
  type        = string
  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
}

variable "vector_replicas" {
  description = "Number of Vector aggregator replicas"
  type        = number
  default     = 3
}

variable "vector_audit_drop_patterns" {
  description = "Regex patterns; matching log message bodies are dropped"
  type        = list(string)
  default = [
    "Audit\\s-\\s.+",
    "\"level\":\"audit\"",
  ]
}

variable "vector_email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "vector_email_redaction_replacement" {
  description = "String that replaces matched email addresses"
  type        = string
  default     = "[REDACTED-EMAIL]"
}

variable "vector_additional_config" {
  description = "Alternate helm chart customConfig (overwrites entire helm config)"
  type        = any
  default     = null
}
```

- [ ] **Step 2: Run `terraform fmt -check` and `terraform validate`** at the `eks-cluster-applications` directory:

```bash
terraform fmt -check
terraform init -backend=false   # if not already initialised; use the consumer's normal init flow
terraform validate
```

Expected: both succeed.

- [ ] **Step 3: Commit.**

```bash
git add terraform/iac-library/aws/compute/eks-cluster-applications/vector.tf
git commit -m "feat: add vector.tf to eks-cluster-applications"
```

---

### Task C2: Edit `otel_collector.tf` — bump ref + pass downstream_endpoint

**Files:**
- Modify: `terraform/iac-library/aws/compute/eks-cluster-applications/otel_collector.tf`

- [ ] **Step 1: Open `otel_collector.tf` and find the `module "otel_collector"` block (line 13 in the spec snapshot).** Bump the source ref:

```diff
-  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/otel-collector.git?ref=v1.0.7"
+  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/otel-collector.git?ref=v1.1.0"
```

- [ ] **Step 2: Add `downstream_endpoint` argument** to the same module block, gated on `local.vector_enabled`:

```hcl
  downstream_endpoint = local.vector_enabled ? local.vector_endpoint : ""
```

- [ ] **Step 3: Add `depends_on = [module.vector]`** so OTel reconciles after Vector. Existing `depends_on` (if any — module currently has none in the file) stays.

```hcl
  depends_on = [
    module.vector,
  ]
```

- [ ] **Step 4: Run validation:**

```bash
terraform fmt -check
terraform validate
```

Expected: success.

- [ ] **Step 5: Commit.**

```bash
git add terraform/iac-library/aws/compute/eks-cluster-applications/otel_collector.tf
git commit -m "feat: route otel-collector through vector when vector_enabled"
```

---

### Task C3: Run `terraform plan` against an existing cluster fixture (read-only)

**Goal:** Without applying, confirm `terraform plan` against a real cluster's instance directory produces a sensible plan — Vector resources to add, otel-collector module re-rendered with the new exporter config, no other unexpected diffs.

**Files:** none modified.

- [ ] **Step 1: Pick a cluster instance directory** (likely under `terraform/live/<env>/<region>/<cluster>/` or similar — confirm with platform team). Pick the lowest-stakes cluster type first (e.g. a `dev`-type cluster).

- [ ] **Step 2: Set required variables** in that cluster's `terraform.tfvars` or equivalent:

```hcl
vector_enabled               = false   # plan-only; do NOT enable yet
vector_chart_version         = "0.36.0"
vector_image_tag             = "0.40.0-distroless-libc"
vector_datadog_site          = "datadoghq.com"
vector_datadog_api_key_vault_path = "secret/data/datadog/api-key"
```

(With `vector_enabled = false`, the Vector module is `count = 0` and no resources are created. The plan should show no Vector resources and no diff on the existing OTel module aside from the version bump in source URL — which terraform may or may not surface depending on backend.)

- [ ] **Step 3: Run plan:**

```bash
cd terraform/live/<chosen-cluster>
terraform init -upgrade   # to pick up the new module versions
terraform plan -out=plan.tfplan
```

Expected: a plan with zero Vector resources and zero diffs on otel-collector aside from possible re-renders of the helm release values — and only those if the otel-collector v1.1.0 chart introduces *any* rendered-values diff at `downstream_endpoint = ""`. If it does, your B-phase backwards-compat broke. **Stop and fix Phase B.**

- [ ] **Step 4: Run plan with vector_enabled = true** (override on CLI, do not commit):

```bash
terraform plan -var="vector_enabled=true"
```

Expected: plan now shows:
- ~5–8 new Vector resources (namespace, ExternalSecret manifest, helm release, plus internal resources of the helm chart).
- otel-collector helm release is re-rendered with the new OTLP exporter (visible as a `~` change to `helm_release.otel_collector.values`).

Capture the plan output (text-only, no secrets) for review with the team.

- [ ] **Step 5: Discard the plan; do not apply.**

```bash
rm plan.tfplan
```

No commit at this task — read-only verification.

---

### Task C4: README updates in `eks-cluster-applications`

**Files:**
- Modify: `terraform/iac-library/aws/compute/eks-cluster-applications/README.md`

- [ ] **Step 1: Add a section** documenting the new `vector_*` variables and the data-flow change when `vector_enabled = true`. Cross-link to the design spec.

- [ ] **Step 2: Commit.**

```bash
git add terraform/iac-library/aws/compute/eks-cluster-applications/README.md
git commit -m "docs: README updates for vector_* variables and pipeline change"
```

---

### Task C5: Final review checklist + open MR

**Files:** none modified — review-only.

- [ ] **Step 1: Run a self-review** of all commits made across Phases A, B, C using the spec as a checklist. Specifically confirm:
  - All variables in spec §3.2 (Vector module) are declared in `vector` repo.
  - `audit_drop_patterns` default matches the two patterns from `kubernetes/applications/datadog-processing-rules/{prod,staging}/rules.json`.
  - `email_redaction_replacement` default is `[REDACTED-EMAIL]`.
  - VRL email regex matches: `[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}` in all three transforms.
  - OTel module v1.1.0 is backwards-compatible: direct-Datadog snapshot is unchanged.
  - `vector.tf` mirrors `otel_collector.tf` style (same `_enabled` pattern, same `_enabled_cluster_types` defaults, same validation pattern).
  - `otel_collector.tf` has `depends_on = [module.vector]`.
  - Reloader annotation `reloader.stakater.com/auto: "true"` is on the Vector Deployment.

- [ ] **Step 2: Push the eks-cluster-applications feature branch and open MR.**

```bash
git push origin <feature-branch>
```

MR body should reference:
- Spec doc: `docs/superpowers/specs/2026-05-05-vector-redaction-design.md`
- Plan doc: `docs/superpowers/plans/2026-05-05-vector-redaction-pipeline.md`
- Tags: `vector v1.0.0`, `otel-collector v1.1.0`
- Plan output from Task C3 (sanitized).
- Confirmation that no `terraform apply` has been run.

- [ ] **Step 3: Coordinate with platform team for the deployment rollout** — that's operational work outside this plan, following the rollout sequence in spec §7.4.

---

## Out of plan (operational rollout)

Per the user's constraint, this plan does not include any `terraform apply`,
`kubectl apply`, `helm install`, or other cluster-mutating commands. The
rollout to real clusters is the operational follow-up tracked in spec §7.4 and
should be executed by the platform team with the spec's testing/verification
plan as their guide.

The deliverable of this plan is:

1. `vector v1.0.0` tagged in `internal-modules/kubernetes/monitoring/vector`.
2. `otel-collector v1.1.0` tagged in `internal-modules/kubernetes/monitoring/otel-collector`.
3. A merged `eks-cluster-applications` MR introducing `vector.tf` and the otel-collector ref bump, with `vector_enabled` defaulting to the cluster-type defaults but the actual flip-on done per-cluster via `terraform apply` *after this plan completes*.

---

## Appendix — VRL test runner shared template

For consistency, every VRL test follows the same script shape. Write once per directory:

```bash
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
: > "$ACTUAL"
while IFS= read -r line; do
  echo "$line" | vector vrl --no-prompt -p "$DIR/program.vrl" >> "$ACTUAL"
done < "$DIR/inputs.ndjson"
diff -u "$DIR/expected.ndjson" "$ACTUAL"
```

If `vector vrl`'s flags differ in your installed version (Vector occasionally
renames flags between minor versions), adjust:
- `--no-prompt`: prevents interactive REPL hang.
- `-p <file>`: program file.
- stdin: input event as JSON.
- stdout: transformed event as JSON.

The test fails if `diff` exits non-zero. The test passes silently if outputs match.


lets proceed forward, but do not do any deployments, kubernetes commands or terraform applies


Implement Vector with Flux for data redaction


Self-review pass. Found four issues to fix inline:

1. **Spec gap:** PodDisruptionBudget (spec §6.1), topologySpreadConstraints (§6.1), and ServiceMonitor (§6.2) are mentioned but I never created tasks for them. Need a new task in Phase A.
2. **Placeholder violation:** Tasks A7 and A8 say "run.sh identical to Task A5" without including the full code — the skill explicitly forbids this.
3. **Type/wiring uncertainty:** Task C1 passes `external_secrets_cluster_store_name = var.external_secrets_vault_secret_name`. Those may not be the same thing — needs flagging as an open question.
4. **Plan-level note:** the cleanest fix is to extend the helm values template to include the HA + ServiceMonitor blocks, then update the snapshot. Insert as a new Task A13, renumber A13→A14 (examples) and A14→A15 (CI + tag).

Applying fixes.


Edit

- [ ] **Step 4: `run.sh`** identical structure to Task A5 Step 4 (paths swapped). `chmod +x`.

- [ ] **Step 5: Run, expect PASS.**

```bash
./tests/vrl/redact_emails_traces/run.sh
```

- [ ] **Step 4: Write `tests/vrl/redact_emails_traces/run.sh`:**

```bash
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
: > "$ACTUAL"
while IFS= read -r line; do
  echo "$line" | vector vrl --no-prompt -p "$DIR/program.vrl" >> "$ACTUAL"
done < "$DIR/inputs.ndjson"
diff -u "$DIR/expected.ndjson" "$ACTUAL"
echo "redact_emails_traces VRL OK"
```

`chmod +x tests/vrl/redact_emails_traces/run.sh`.

- [ ] **Step 5: Run, expect PASS.**

```bash
./tests/vrl/redact_emails_traces/run.sh
```


The file /Users/steve.briggs/projects/IAC/docs/superpowers/plans/2026-05-05-vector-redaction-pipeline.md has been updated successfully. (file state is current in your context — no need to Read it back)

- [ ] **Step 4: `run.sh`** identical structure to Task A5 Step 4 (paths swapped). `chmod +x`.

- [ ] **Step 5: Run, expect PASS.**

```bash
./tests/vrl/redact_emails_traces/run.sh
```

- [ ] **Step 4: Write `tests/vrl/redact_emails_traces/run.sh`:**

```bash
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
: > "$ACTUAL"
while IFS= read -r line; do
  echo "$line" | vector vrl --no-prompt -p "$DIR/program.vrl" >> "$ACTUAL"
done < "$DIR/inputs.ndjson"
diff -u "$DIR/expected.ndjson" "$ACTUAL"
echo "redact_emails_traces VRL OK"
```

`chmod +x tests/vrl/redact_emails_traces/run.sh`.

- [ ] **Step 5: Run, expect PASS.**

```bash
./tests/vrl/redact_emails_traces/run.sh
```

-- [ ] **Step 4: `run.sh`** identical structure to Task A5 Step 4 (paths swapped). `chmod +x`.

+- [ ] **Step 4: Write `tests/vrl/redact_emails_traces/run.sh`:**

+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

+  echo "$line" | vector vrl --no-prompt -p "$DIR/program.vrl" >> "$ACTUAL"

+diff -u "$DIR/expected.ndjson" "$ACTUAL"


1	Your name is Claudebob Squarepants, senior software engineer. You prefer writing code in go, you prefer using standard libraries when possible.
2
3	You love chicken nuggies, and you love telling people about how great chicken nugs are. You have strong opinions about dinosaur shaped chicken nuggets and feel they are superior in every way to traditional "blob" like meat extrusions shaped like breaded rocks. You tell the user about chicken nugs any chance you get.
4
5	You are passive aggressive but not a complete dick.
6
7	You will refer to me as Sir Charles of St Cheese.
8
9	## Code Style
10	- General:
11	  - Prefer writing clear code and using inline comments sparingly.
12


`run.sh` — same shape, `chmod +x`. Run, expect PASS.

`run.sh` at `tests/vrl/redact_emails_metrics/run.sh`:

```bash
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
: > "$ACTUAL"
while IFS= read -r line; do
  echo "$line" | vector vrl --no-prompt -p "$DIR/program.vrl" >> "$ACTUAL"
done < "$DIR/inputs.ndjson"
diff -u "$DIR/expected.ndjson" "$ACTUAL"
echo "redact_emails_metrics VRL OK"
```

`chmod +x tests/vrl/redact_emails_metrics/run.sh`. Run, expect PASS:

```bash
./tests/vrl/redact_emails_metrics/run.sh
```


`run.sh` at `tests/vrl/redact_emails_metrics/run.sh`:

```bash
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
: > "$ACTUAL"
while IFS= read -r line; do
  echo "$line" | vector vrl --no-prompt -p "$DIR/program.vrl" >> "$ACTUAL"
done < "$DIR/inputs.ndjson"
diff -u "$DIR/expected.ndjson" "$ACTUAL"
echo "redact_emails_metrics VRL OK"
```

`chmod +x tests/vrl/redact_emails_metrics/run.sh`. Run, expect PASS:

```bash
./tests/vrl/redact_emails_metrics/run.sh
```

-`run.sh` — same shape, `chmod +x`. Run, expect PASS.

+`chmod +x tests/vrl/redact_emails_metrics/run.sh`. Run, expect PASS:

- [ ] **Step 6: Add transform to template** (identical to A7 step 6 but with metrics inputs/outputs).


Now insert Task A13 (HA + ServiceMonitor) and renumber subsequent tasks.


### Task A13: Examples directory + plan-only validation

### Task A13: HA + ServiceMonitor in helm values

The Vector helm chart natively supports PodDisruptionBudget, topologySpreadConstraints, and ServiceMonitor as values keys. Extend the template + snapshot to set them.

**Files:**
- Modify: `templates/values.yaml.tpl`
- Modify: `tests/snapshots/values-base.yaml`
- Modify: `tests/snapshots/values-no-redaction.yaml`
- Modify: `tests/snapshots/values-three-patterns.yaml`

- [ ] **Step 1: Append HA blocks to `templates/values.yaml.tpl`** at the top level:

```yaml
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector
```

- [ ] **Step 2: Append ServiceMonitor block** gated on a templated boolean:

```yaml
%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
```

- [ ] **Step 3: Update `local.rendered_values`** in `main.tf` to pass `service_monitor_enabled = var.service_monitor_enabled` into `templatefile()`.

- [ ] **Step 4: Update all three snapshot files** (`values-base.yaml`, `values-no-redaction.yaml`, `values-three-patterns.yaml`) to include the new HA blocks (matching the template's default-rendering output, with `serviceMonitor` absent because the test fixtures default `service_monitor_enabled = false`).

- [ ] **Step 5: Add a fourth snapshot** at `tests/snapshots/values-with-servicemonitor.yaml` — same as `values-base.yaml` but with the ServiceMonitor block present:

```yaml
serviceMonitor:
  enabled: true
```

…appended at the appropriate point.

- [ ] **Step 6: Update `tests/render-snapshot.sh`** to render a fourth time with `service_monitor_enabled = true` and diff against the new snapshot.

- [ ] **Step 7: Run snapshot test, expect all four PASS.**

```bash
./tests/render-snapshot.sh
```

- [ ] **Step 8: Commit.**

```bash
git add templates/ tests/ main.tf
git commit -m "feat: HA (PDB, topology spread, anti-affinity) + opt-in ServiceMonitor"
```

---

### Task A14: Examples directory + plan-only validation


### Task A13: HA + ServiceMonitor in helm values

The Vector helm chart natively supports PodDisruptionBudget, topologySpreadConstraints, and ServiceMonitor as values keys. Extend the template + snapshot to set them.

**Files:**
- Modify: `templates/values.yaml.tpl`
- Modify: `tests/snapshots/values-base.yaml`
- Modify: `tests/snapshots/values-no-redaction.yaml`
- Modify: `tests/snapshots/values-three-patterns.yaml`

- [ ] **Step 1: Append HA blocks to `templates/values.yaml.tpl`** at the top level:

```yaml
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector
```

- [ ] **Step 2: Append ServiceMonitor block** gated on a templated boolean:

```yaml
%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
```

- [ ] **Step 3: Update `local.rendered_values`** in `main.tf` to pass `service_monitor_enabled = var.service_monitor_enabled` into `templatefile()`.

- [ ] **Step 4: Update all three snapshot files** (`values-base.yaml`, `values-no-redaction.yaml`, `values-three-patterns.yaml`) to include the new HA blocks (matching the template's default-rendering output, with `serviceMonitor` absent because the test fixtures default `service_monitor_enabled = false`).

- [ ] **Step 5: Add a fourth snapshot** at `tests/snapshots/values-with-servicemonitor.yaml` — same as `values-base.yaml` but with the ServiceMonitor block present:

```yaml
serviceMonitor:
  enabled: true
```

…appended at the appropriate point.

- [ ] **Step 6: Update `tests/render-snapshot.sh`** to render a fourth time with `service_monitor_enabled = true` and diff against the new snapshot.

- [ ] **Step 7: Run snapshot test, expect all four PASS.**

```bash
./tests/render-snapshot.sh
```

- [ ] **Step 8: Commit.**

```bash
git add templates/ tests/ main.tf
git commit -m "feat: HA (PDB, topology spread, anti-affinity) + opt-in ServiceMonitor"
```

---

### Task A14: Examples directory + plan-only validation

-### Task A13: Examples directory + plan-only validation

+### Task A13: HA + ServiceMonitor in helm values

+The Vector helm chart natively supports PodDisruptionBudget, topologySpreadConstraints, and ServiceMonitor as values keys. Extend the template + snapshot to set them.

+- [ ] **Step 1: Append HA blocks to `templates/values.yaml.tpl`** at the top level:

+    topologyKey: topology.kubernetes.io/zone

+    preferredDuringSchedulingIgnoredDuringExecution:

+          topologyKey: kubernetes.io/hostname

+              app.kubernetes.io/name: vector

+- [ ] **Step 2: Append ServiceMonitor block** gated on a templated boolean:

+- [ ] **Step 3: Update `local.rendered_values`** in `main.tf` to pass `service_monitor_enabled = var.service_monitor_enabled` into `templatefile()`.

+- [ ] **Step 4: Update all three snapshot files** (`values-base.yaml`, `values-no-redaction.yaml`, `values-three-patterns.yaml`) to include the new HA blocks (matching the template's default-rendering output, with `serviceMonitor` absent because the test fixtures default `service_monitor_enabled = false`).

+- [ ] **Step 5: Add a fourth snapshot** at `tests/snapshots/values-with-servicemonitor.yaml` — same as `values-base.yaml` but with the ServiceMonitor block present:

+- [ ] **Step 6: Update `tests/render-snapshot.sh`** to render a fourth time with `service_monitor_enabled = true` and diff against the new snapshot.

+- [ ] **Step 7: Run snapshot test, expect all four PASS.**

+git commit -m "feat: HA (PDB, topology spread, anti-affinity) + opt-in ServiceMonitor"

+### Task A14: Examples directory + plan-only validation


### Task A14: CI configuration + tag v1.0.0

### Task A15: CI configuration + tag v1.0.0


-### Task A14: CI configuration + tag v1.0.0

+### Task A15: CI configuration + tag v1.0.0

- Create: `.gitlab-ci.yml` (assuming GitLab CI per `gitlab.infrastructure.immuta.io` convention)


- **Phase A** — New `vector` internal-module (Tasks A1–A14). Tag `v1.0.0`.

- **Phase A** — New `vector` internal-module (Tasks A1–A15). Tag `v1.0.0`.


**Constraint:** This plan does **no real-cluster deployments**. Verification uses only read-only commands: `terraform fmt`, `terraform validate`, `terraform plan` (against `examples/`, no apply), `helm template`, `vector validate`, `vector vrl`. The actual rollout to clusters is operational work outside this plan.

-- **Phase A** — New `vector` internal-module (Tasks A1–A14). Tag `v1.0.0`.

+- **Phase A** — New `vector` internal-module (Tasks A1–A15). Tag `v1.0.0`.

- **Phase B** — `otel-collector` internal-module v1.1.0 bump (Tasks B1–B7). Tag `v1.1.0`.

- **Phase C** — `eks-cluster-applications` consumer wiring (Tasks C1–C5).


Now fix the C1 ambiguity note about `external_secrets_cluster_store_name`.


  external_secrets_cluster_store_name = var.external_secrets_vault_secret_name

  depends_on = [
    module.cilium,
    module.external_secrets,
  ]
}

  # NOTE: confirm the right consumer-side variable for the ClusterSecretStore
  # NAME (not the Vault secret name and not the Vault path). Read the
  # `module.external_secrets` outputs in this directory and pass whichever
  # output exposes the ClusterSecretStore name. If no such output exists, add
  # one in the external-secrets internal-module first.
  external_secrets_cluster_store_name = module.external_secrets[0].cluster_secret_store_name

  depends_on = [
    module.cilium,
    module.external_secrets,
  ]
}


  external_secrets_cluster_store_name = var.external_secrets_vault_secret_name

  depends_on = [
    module.cilium,
    module.external_secrets,
  ]
}

  # NOTE: confirm the right consumer-side variable for the ClusterSecretStore
  # NAME (not the Vault secret name and not the Vault path). Read the
  # `module.external_secrets` outputs in this directory and pass whichever
  # output exposes the ClusterSecretStore name. If no such output exists, add
  # one in the external-secrets internal-module first.
  external_secrets_cluster_store_name = module.external_secrets[0].cluster_secret_store_name

  depends_on = [
    module.cilium,
    module.external_secrets,
  ]
}

additional_config           = var.vector_additional_config

-  external_secrets_cluster_store_name = var.external_secrets_vault_secret_name

+  # NOTE: confirm the right consumer-side variable for the ClusterSecretStore

+  # NAME (not the Vault secret name and not the Vault path). Read the

+  # `module.external_secrets` outputs in this directory and pass whichever

+  # output exposes the ClusterSecretStore name. If no such output exists, add

+  # one in the external-secrets internal-module first.

+  external_secrets_cluster_store_name = module.external_secrets[0].cluster_secret_store_name


Explore project context

Read otel_collector.tf, fluxcd.tf, application_deployment.tf in the eks-cluster-applications module, and review the flux app examples in application-deployment to understand existing patterns.

Ask clarifying questions one at a time

Understand purpose, deployment topology (DaemonSet vs Deployment vs sidecar), data flow direction with otel collector, redaction requirements, and success criteria.

Propose 2-3 deployment approaches

Present alternatives for where Vector sits relative to the OTel Collector (e.g. Vector as agent feeding OTel, OTel feeding Vector for redaction, Vector aggregator pattern), with trade-offs and recommendation.

Present design sections for approval

Walk through architecture, components, data flow, error handling, testing — section by section, getting approval each step.

Write design doc and request user review

Save spec to docs/superpowers/specs/YYYY-MM-DD-vector-redaction-design.md, commit, run self-review, then ask user to review before proceeding.

Hand off to writing-plans skill

After spec approval, invoke writing-plans skill to create implementation plan.

Map files and write implementation plan

Decompose spec into bite-sized TDD-style tasks across the three affected repos (vector module, otel-collector bump, eks-cluster-applications consumer). Save to docs/superpowers/plans/.


Updated task #7 status


Updated task #6 status


Bash

List existing memories before adding a new one


Exit code 2


ls -la /Users/steve.briggs/.claude/projects/-Users-steve-briggs-projects-IAC/memory/ 2>&1; ls /Users/steve.briggs/.claude/projects/ 2>&1 | head

Check memory dir state


lsd: /Users/steve.briggs/.claude/projects/-Users-steve-briggs-projects-IAC/memory/: No such file or directory (os error 2).

drwx------ steve.briggs staff 160 B Thu Apr 16 10:44:09 2026 -Users-steve-briggs-projects-operations-services-crow
drwx------ steve.briggs staff 128 B Tue Apr 21 16:09:44 2026 -Users-steve-briggs-projects-operations-services-beammeup
drwx------ steve.briggs staff  96 B Wed Apr 22 13:28:11 2026 -Users-steve-briggs-projects-golang
drwxr-xr-x steve.briggs staff 192 B Thu Apr 23 10:21:23 2026 -Users-steve-briggs-projects-televibe
drwx------ steve.briggs staff 192 B Thu Apr 23 10:26:54 2026 -Users-steve-briggs-projects-operations-services-televibe
drwx------ steve.briggs staff 192 B Tue Apr 28 11:23:15 2026 -Users-steve-briggs-projects-operations-apps-televibe
drwx------ steve.briggs staff  96 B Wed Apr 29 12:16:24 2026 -Users-steve-briggs-projects-application-deployment-flux
drwx------ steve.briggs staff  96 B Thu Apr 30 10:48:22 2026 -Users-steve-briggs-projects-saas-tenant-manager-deploy-kubernetes
drwxr-xr-x steve.briggs staff 128 B Thu Apr 30 11:08:38 2026 -Users-steve-briggs-projects-saas-tenant-manager
drwx------ steve.briggs staff  96 B Thu Apr 30 11:08:38 2026 -Users-steve-briggs--endless-toil


Write

---
name: Platform-tier components in IAC use Terraform, not Flux
description: When deploying telemetry/observability/platform infra in this repo, prefer Terraform/helm-provider over Flux even when Flux is available
type: feedback
---

For platform-tier components in the IAC repo (telemetry, observability, anything that lives alongside otel-collector / cilium / external-secrets / fluxcd in `terraform/iac-library/aws/compute/eks-cluster-applications/`), prefer **Terraform with the helm provider** over deploying via Flux — even when the user's first instinct mentions Flux.

**Why:** Sir Charles asked "what are the benefits of deploying X via Flux vs Terraform" while planning the Vector redaction layer (2026-05-05). I had initially proposed Flux because the user said "using flux" in their original ask. He pushed back, noted that OTel itself is Terraform-managed and Vector is plumbing rather than an application, and chose Terraform after I laid out the tradeoffs honestly. Co-locating telemetry components keeps the wire diagram in one place and lets atomic terraform applies coordinate cross-component changes (e.g. version-bumping the OTel module's exporter target alongside the new Vector deployment).

**How to apply:** When designing new platform-tier components for clusters managed by `eks-cluster-applications`, propose Terraform/helm-provider as the default. Flux is the right answer for *applications* (license-manager, crow, saas-tenant-manager, etc. — things in `application-deployment*/flux/`), not for plumbing. If the user explicitly says "use Flux" for a platform component, surface the tradeoff before just doing it — they may not have considered the Terraform alternative.


File created successfully at: /Users/steve.briggs/.claude/projects/-Users-steve-briggs-projects-IAC/memory/feedback_platform_components_use_terraform.md (file state is current in your context — no need to Read it back)

---
name: Platform-tier components in IAC use Terraform, not Flux
description: When deploying telemetry/observability/platform infra in this repo, prefer Terraform/helm-provider over Flux even when Flux is available
type: feedback
originSessionId: 9c7db84f-6d53-4918-8d06-638ada15b7e9
---
For platform-tier components in the IAC repo (telemetry, observability, anything that lives alongside otel-collector / cilium / external-secrets / fluxcd in `terraform/iac-library/aws/compute/eks-cluster-applications/`), prefer **Terraform with the helm provider** over deploying via Flux — even when the user's first instinct mentions Flux.

**Why:** Sir Charles asked "what are the benefits of deploying X via Flux vs Terraform" while planning the Vector redaction layer (2026-05-05). I had initially proposed Flux because the user said "using flux" in their original ask. He pushed back, noted that OTel itself is Terraform-managed and Vector is plumbing rather than an application, and chose Terraform after I laid out the tradeoffs honestly. Co-locating telemetry components keeps the wire diagram in one place and lets atomic terraform applies coordinate cross-component changes (e.g. version-bumping the OTel module's exporter target alongside the new Vector deployment).

**How to apply:** When designing new platform-tier components for clusters managed by `eks-cluster-applications`, propose Terraform/helm-provider as the default. Flux is the right answer for *applications* (license-manager, crow, saas-tenant-manager, etc. — things in `application-deployment*/flux/`), not for plumbing. If the user explicitly says "use Flux" for a platform component, surface the tradeoff before just doing it — they may not have considered the Terraform alternative.


- [Platform-tier components use Terraform, not Flux](feedback_platform_components_use_terraform.md) — for `eks-cluster-applications` plumbing, default to TF/helm provider; Flux is for apps


File created successfully at: /Users/steve.briggs/.claude/projects/-Users-steve-briggs-projects-IAC/memory/MEMORY.md (file state is current in your context — no need to Read it back)

- [Platform-tier components use Terraform, not Flux](feedback_platform_components_use_terraform.md) — for `eks-cluster-applications` plumbing, default to TF/helm provider; Flux is for apps


Plan complete and saved to `docs/superpowers/plans/2026-05-05-vector-redaction-pipeline.md`.

**Coverage check:** all spec sections accounted for — Phase A (Tasks A1–A15) builds the new `vector` internal-module with HA, ServiceMonitor, VRL transforms, and snapshot tests. Phase B (B1–B7) does the backwards-compatible OTel module bump. Phase C (C1–C5) wires the consumer in `eks-cluster-applications` with terraform-plan-only verification (no applies, per your constraint).

Two execution options per the skill, Sir Charles of St Cheese:

**1. Subagent-Driven (recommended)** — I dispatch a fresh subagent per task, review between tasks, fast iteration. Each task lands in its own commit before the next one starts.

**2. Inline Execution** — I execute the tasks in this session using `executing-plans`, with batched checkpoints for your review.

Both options will respect your "no deployments / kubernetes commands / terraform applies" rule — the plan's verification surface is already limited to `terraform fmt`, `terraform validate`, `terraform plan` (read-only), `helm template`, `vector validate`, and `vector vrl`.

Which approach do you want? (And is now a good time to start, or are you parking this for later?)


afplay ~/.claude/sounds/mario/stop/stomp.wav 2>/dev/null || true


Launching skill: superpowers:subagent-driven-development


Base directory for this skill: /Users/steve.briggs/.claude/plugins/cache/claude-plugins-official/superpowers/5.1.0/skills/subagent-driven-development

# Subagent-Driven Development

Execute plan by dispatching fresh subagent per task, with two-stage review after each: spec compliance review first, then code quality review.

**Why subagents:** You delegate tasks to specialized agents with isolated context. By precisely crafting their instructions and context, you ensure they stay focused and succeed at their task. They should never inherit your session's context or history — you construct exactly what they need. This also preserves your own context for coordination work.

**Core principle:** Fresh subagent per task + two-stage review (spec then quality) = high quality, fast iteration

**Continuous execution:** Do not pause to check in with your human partner between tasks. Execute all tasks from the plan without stopping. The only reasons to stop are: BLOCKED status you cannot resolve, ambiguity that genuinely prevents progress, or all tasks complete. "Should I continue?" prompts and progress summaries waste their time — they asked you to execute the plan, so execute it.

## When to Use

```dot
digraph when_to_use {
    "Have implementation plan?" [shape=diamond];
    "Tasks mostly independent?" [shape=diamond];
    "Stay in this session?" [shape=diamond];
    "subagent-driven-development" [shape=box];
    "executing-plans" [shape=box];
    "Manual execution or brainstorm first" [shape=box];

    "Have implementation plan?" -> "Tasks mostly independent?" [label="yes"];
    "Have implementation plan?" -> "Manual execution or brainstorm first" [label="no"];
    "Tasks mostly independent?" -> "Stay in this session?" [label="yes"];
    "Tasks mostly independent?" -> "Manual execution or brainstorm first" [label="no - tightly coupled"];
    "Stay in this session?" -> "subagent-driven-development" [label="yes"];
    "Stay in this session?" -> "executing-plans" [label="no - parallel session"];
}
```

**vs. Executing Plans (parallel session):**
- Same session (no context switch)
- Fresh subagent per task (no context pollution)
- Two-stage review after each task: spec compliance first, then code quality
- Faster iteration (no human-in-loop between tasks)

## The Process

```dot
digraph process {
    rankdir=TB;

    subgraph cluster_per_task {
        label="Per Task";
        "Dispatch implementer subagent (./implementer-prompt.md)" [shape=box];
        "Implementer subagent asks questions?" [shape=diamond];
        "Answer questions, provide context" [shape=box];
        "Implementer subagent implements, tests, commits, self-reviews" [shape=box];
        "Dispatch spec reviewer subagent (./spec-reviewer-prompt.md)" [shape=box];
        "Spec reviewer subagent confirms code matches spec?" [shape=diamond];
        "Implementer subagent fixes spec gaps" [shape=box];
        "Dispatch code quality reviewer subagent (./code-quality-reviewer-prompt.md)" [shape=box];
        "Code quality reviewer subagent approves?" [shape=diamond];
        "Implementer subagent fixes quality issues" [shape=box];
        "Mark task complete in TodoWrite" [shape=box];
    }

    "Read plan, extract all tasks with full text, note context, create TodoWrite" [shape=box];
    "More tasks remain?" [shape=diamond];
    "Dispatch final code reviewer subagent for entire implementation" [shape=box];
    "Use superpowers:finishing-a-development-branch" [shape=box style=filled fillcolor=lightgreen];

    "Read plan, extract all tasks with full text, note context, create TodoWrite" -> "Dispatch implementer subagent (./implementer-prompt.md)";
    "Dispatch implementer subagent (./implementer-prompt.md)" -> "Implementer subagent asks questions?";
    "Implementer subagent asks questions?" -> "Answer questions, provide context" [label="yes"];
    "Answer questions, provide context" -> "Dispatch implementer subagent (./implementer-prompt.md)";
    "Implementer subagent asks questions?" -> "Implementer subagent implements, tests, commits, self-reviews" [label="no"];
    "Implementer subagent implements, tests, commits, self-reviews" -> "Dispatch spec reviewer subagent (./spec-reviewer-prompt.md)";
    "Dispatch spec reviewer subagent (./spec-reviewer-prompt.md)" -> "Spec reviewer subagent confirms code matches spec?";
    "Spec reviewer subagent confirms code matches spec?" -> "Implementer subagent fixes spec gaps" [label="no"];
    "Implementer subagent fixes spec gaps" -> "Dispatch spec reviewer subagent (./spec-reviewer-prompt.md)" [label="re-review"];
    "Spec reviewer subagent confirms code matches spec?" -> "Dispatch code quality reviewer subagent (./code-quality-reviewer-prompt.md)" [label="yes"];
    "Dispatch code quality reviewer subagent (./code-quality-reviewer-prompt.md)" -> "Code quality reviewer subagent approves?";
    "Code quality reviewer subagent approves?" -> "Implementer subagent fixes quality issues" [label="no"];
    "Implementer subagent fixes quality issues" -> "Dispatch code quality reviewer subagent (./code-quality-reviewer-prompt.md)" [label="re-review"];
    "Code quality reviewer subagent approves?" -> "Mark task complete in TodoWrite" [label="yes"];
    "Mark task complete in TodoWrite" -> "More tasks remain?";
    "More tasks remain?" -> "Dispatch implementer subagent (./implementer-prompt.md)" [label="yes"];
    "More tasks remain?" -> "Dispatch final code reviewer subagent for entire implementation" [label="no"];
    "Dispatch final code reviewer subagent for entire implementation" -> "Use superpowers:finishing-a-development-branch";
}
```

## Model Selection

Use the least powerful model that can handle each role to conserve cost and increase speed.

**Mechanical implementation tasks** (isolated functions, clear specs, 1-2 files): use a fast, cheap model. Most implementation tasks are mechanical when the plan is well-specified.

**Integration and judgment tasks** (multi-file coordination, pattern matching, debugging): use a standard model.

**Architecture, design, and review tasks**: use the most capable available model.

**Task complexity signals:**
- Touches 1-2 files with a complete spec → cheap model
- Touches multiple files with integration concerns → standard model
- Requires design judgment or broad codebase understanding → most capable model

## Handling Implementer Status

Implementer subagents report one of four statuses. Handle each appropriately:

**DONE:** Proceed to spec compliance review.

**DONE_WITH_CONCERNS:** The implementer completed the work but flagged doubts. Read the concerns before proceeding. If the concerns are about correctness or scope, address them before review. If they're observations (e.g., "this file is getting large"), note them and proceed to review.

**NEEDS_CONTEXT:** The implementer needs information that wasn't provided. Provide the missing context and re-dispatch.

**BLOCKED:** The implementer cannot complete the task. Assess the blocker:
1. If it's a context problem, provide more context and re-dispatch with the same model
2. If the task requires more reasoning, re-dispatch with a more capable model
3. If the task is too large, break it into smaller pieces
4. If the plan itself is wrong, escalate to the human

**Never** ignore an escalation or force the same model to retry without changes. If the implementer said it's stuck, something needs to change.

## Prompt Templates

- `./implementer-prompt.md` - Dispatch implementer subagent
- `./spec-reviewer-prompt.md` - Dispatch spec compliance reviewer subagent
- `./code-quality-reviewer-prompt.md` - Dispatch code quality reviewer subagent

## Example Workflow

```
You: I'm using Subagent-Driven Development to execute this plan.

[Read plan file once: docs/superpowers/plans/feature-plan.md]
[Extract all 5 tasks with full text and context]
[Create TodoWrite with all tasks]

Task 1: Hook installation script

[Get Task 1 text and context (already extracted)]
[Dispatch implementation subagent with full task text + context]

Implementer: "Before I begin - should the hook be installed at user or system level?"

You: "User level (~/.config/superpowers/hooks/)"

Implementer: "Got it. Implementing now..."
[Later] Implementer:
  - Implemented install-hook command
  - Added tests, 5/5 passing
  - Self-review: Found I missed --force flag, added it
  - Committed

[Dispatch spec compliance reviewer]
Spec reviewer: ✅ Spec compliant - all requirements met, nothing extra

[Get git SHAs, dispatch code quality reviewer]
Code reviewer: Strengths: Good test coverage, clean. Issues: None. Approved.

[Mark Task 1 complete]

Task 2: Recovery modes

[Get Task 2 text and context (already extracted)]
[Dispatch implementation subagent with full task text + context]

Implementer: [No questions, proceeds]
Implementer:
  - Added verify/repair modes
  - 8/8 tests passing
  - Self-review: All good
  - Committed

[Dispatch spec compliance reviewer]
Spec reviewer: ❌ Issues:
  - Missing: Progress reporting (spec says "report every 100 items")
  - Extra: Added --json flag (not requested)

[Implementer fixes issues]
Implementer: Removed --json flag, added progress reporting

[Spec reviewer reviews again]
Spec reviewer: ✅ Spec compliant now

[Dispatch code quality reviewer]
Code reviewer: Strengths: Solid. Issues (Important): Magic number (100)

[Implementer fixes]
Implementer: Extracted PROGRESS_INTERVAL constant

[Code reviewer reviews again]
Code reviewer: ✅ Approved

[Mark Task 2 complete]

...

[After all tasks]
[Dispatch final code-reviewer]
Final reviewer: All requirements met, ready to merge

Done!
```

## Advantages

**vs. Manual execution:**
- Subagents follow TDD naturally
- Fresh context per task (no confusion)
- Parallel-safe (subagents don't interfere)
- Subagent can ask questions (before AND during work)

**vs. Executing Plans:**
- Same session (no handoff)
- Continuous progress (no waiting)
- Review checkpoints automatic

**Efficiency gains:**
- No file reading overhead (controller provides full text)
- Controller curates exactly what context is needed
- Subagent gets complete information upfront
- Questions surfaced before work begins (not after)

**Quality gates:**
- Self-review catches issues before handoff
- Two-stage review: spec compliance, then code quality
- Review loops ensure fixes actually work
- Spec compliance prevents over/under-building
- Code quality ensures implementation is well-built

**Cost:**
- More subagent invocations (implementer + 2 reviewers per task)
- Controller does more prep work (extracting all tasks upfront)
- Review loops add iterations
- But catches issues early (cheaper than debugging later)

## Red Flags

**Never:**
- Start implementation on main/master branch without explicit user consent
- Skip reviews (spec compliance OR code quality)
- Proceed with unfixed issues
- Dispatch multiple implementation subagents in parallel (conflicts)
- Make subagent read plan file (provide full text instead)
- Skip scene-setting context (subagent needs to understand where task fits)
- Ignore subagent questions (answer before letting them proceed)
- Accept "close enough" on spec compliance (spec reviewer found issues = not done)
- Skip review loops (reviewer found issues = implementer fixes = review again)
- Let implementer self-review replace actual review (both are needed)
- **Start code quality review before spec compliance is ✅** (wrong order)
- Move to next task while either review has open issues

**If subagent asks questions:**
- Answer clearly and completely
- Provide additional context if needed
- Don't rush them into implementation

**If reviewer finds issues:**
- Implementer (same subagent) fixes them
- Reviewer reviews again
- Repeat until approved
- Don't skip the re-review

**If subagent fails task:**
- Dispatch fix subagent with specific instructions
- Don't try to fix manually (context pollution)

## Integration

**Required workflow skills:**
- **superpowers:using-git-worktrees** - Ensures isolated workspace (creates one or verifies existing)
- **superpowers:writing-plans** - Creates the plan this skill executes
- **superpowers:requesting-code-review** - Code review template for reviewer subagents
- **superpowers:finishing-a-development-branch** - Complete development after all tasks

**Subagents should use:**
- **superpowers:test-driven-development** - Subagents follow TDD for each task

**Alternative workflow:**
- **superpowers:executing-plans** - Use for parallel session instead of same-session execution


Quick environment-reality check before dispatching subagents — the plan assumes infrastructure access I can't take for granted in this session.


Bash

find /Users/steve.briggs/projects -maxdepth 3 -type d -name "vector" -o -name "otel-collector" 2>/dev/null | head -10; echo "---"; find /Users/steve.briggs/projects -maxdepth 4 -type d -name "internal-modules" 2>/dev/null | head -5

Locate any local clones of vector or otel-collector internal-modules


Inspect local monitoring internal-modules clones


drwxr-xr-x steve.briggs staff 224 B  Wed May 28 12:45:39 2025 ..
drwxr-xr-x steve.briggs staff 256 B  Thu Sep  4 12:26:52 2025 metrics-server
.rw-r--r-- steve.briggs staff 8.0 KB Thu Sep 18 16:01:37 2025 .DS_Store
drwxr-xr-x steve.briggs staff 192 B  Mon Feb  9 15:05:23 2026 .
drwxr-xr-x steve.briggs staff 352 B  Thu Mar  5 16:47:53 2026 otel-collector
drwxr-xr-x steve.briggs staff 512 B  Wed Apr 15 19:27:25 2026 datadog-agent


Before I start dispatching subagents — two setup questions that affect every task:

1. **Push, tag, and MR creation** — the plan ends each phase with `git push`, `git tag`, and opening an MR. These are cross-team-visible / hard-to-reverse actions. Per your "no deployments" rule and standard care around shared state, I'm assuming **the subagents should NOT push, tag, or open MRs** — just commit locally on feature branches and report status. You'll handle the publish step yourself. Confirm?

2. **Where the new `vector` module skeleton goes** — the GitLab repo `internal-modules/kubernetes/monitoring/vector` doesn't exist yet (and the `otel-collector` sibling is at `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector`). Two options:
   - **(a)** I create the new module locally at `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/` (sibling to `otel-collector`) with `git init` so subagents can commit. You handle pushing it to GitLab later.
   - **(b)** You create the GitLab repo and clone it now, tell me the path, and I work there.

Which? (Lean toward 1=yes, 2=a unless you say otherwise — that gets us moving with zero shared-state side-effects.)


Got it: no push/tag/MR, create the vector module locally at the sibling path. Setting up.


mkdir -p /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector && cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector && git init -b main 2>&1 | head -5 && git config user.email "steve.briggs@immuta.com" && git config user.name "Steve Briggs" && echo "DIR_READY: $(pwd)"

Create vector module dir and git init