Fair question. I quietly assumed Flux because you said "using flux" in the original ask, but you're right to interrogate it — the real answer is "it depends, and honestly the case for Terraform here is stronger than I gave it credit for." Let me lay both sides out properly.

### Terraform (mirror `otel_collector.tf`)

**Pros:**
- **Atomicity with OTel.** The OTel module bump and the Vector deploy land in the same `terraform apply`. No window where OTel is exporting to a Service that hasn't been reconciled yet. Same blast radius, same rollback surface.
- **Lifecycle match.** Vector is plumbing, not an app. Its chart version, image, replicas, and resource limits are platform concerns — exactly what `eks-cluster-applications` already manages for OTel, Cilium, ExternalSecrets, etc.
- **Less ceremony.** No new `internal-tooling/vector` OCI source repo. No per-cluster `flux/<cluster>/apps/vector/` directories to keep in sync across N clusters. One module, one variable surface.
- **Consistency.** Telemetry pipeline lives in one place. Today: split brain between `otel_collector.tf` (terraform) and `vector/` (flux). Cognitive cost.
- **Day-1 rules in TF aren't unusual here.** `otel_collector_additional_config` already exists as a TF-passed-through-to-helm config block. VRL rules can ride the same paved road.

### Flux (the path I sketched)

**Pros:**
- **Faster iteration on rules.** Update a regex, bump the OCI tag, Flux reconciles in minutes. No TF plan/apply gate. This matters if redaction rules are expected to change a lot.
- **Per-cluster overlay flexibility.** If `cps-prod` needs different rules than `saas-prod`, Flux/Kustomize handles that natively. Today's `datadog-processing-rules` already has prod/staging splits, so there's precedent for cluster-specific rules.
- **Matches the home of the rules today.** `kubernetes/applications/datadog-processing-rules` is already kustomize-managed. Continuing that lineage in Flux is the smaller conceptual leap.
- **Decouples operator changes from infra changes.** A platform engineer bumping CoreDNS doesn't accidentally touch redaction rules.

### My honest read

Given your stack — and especially given OTel itself is in TF — **pure Terraform is the cleaner answer for the deployment of Vector**. The argument for Flux is mostly about iteration speed on the VRL rules, and the shape of the day-1 rule set ("drop these two patterns + redact email") doesn't suggest weekly churn.

If rule iteration speed becomes a real pain point later, the **hybrid** is also reasonable: TF deploys the Vector HelmRelease (platform), Flux manages a separate `vector-rules` ConfigMap that Vector reloads on change. But that's premature absent the pain.

**Recommendation: switch to Terraform.** Want me to redo Section 2 with that layout?


afplay ~/.claude/sounds/mario/stop/stomp.wav 2>/dev/null || true


yes redo section 2 with terraform


## Section 2 / 7 (revised) — Terraform file layout

**Three changes across two repos:**

### 1. `internal-modules/kubernetes/monitoring/otel-collector` v1.0.7 → v1.1.0 (version bump)

Backwards-compatible. Two new variables:
- `downstream_endpoint` (string, default `""`)
- `downstream_protocol` (string, default `"otlp/grpc"`)

When `downstream_endpoint` is non-empty, the rendered helm values gain an `exporters.otlp` block and the `service.pipelines.{logs,traces,metrics}.exporters` lists swap from `[datadog]` to `[otlp]`. Empty preserves today's behavior.

### 2. `internal-modules/kubernetes/monitoring/vector` v1.0.0 (new module)

Mirrors the shape of the `otel-collector` module — same provider wiring (`kubernetes`, `helm`), same variable patterns, same approach to image/chart/replica overrides. Wraps the upstream Vector helm chart (`timberio/vector`).

What it creates:
- `kubernetes_namespace.vector` — the `vector` namespace.
- `helm_release.vector` — the chart, with values templated from variables.
- `kubernetes_manifest` for an `ExternalSecret` pulling the Datadog API key from Vault, using the same external-secrets cluster store the rest of the platform uses.

Module variables (initial set):
| Variable | Type | Default | Purpose |
|---|---|---|---|
| `chart_version` | string | required | Vector helm chart version pin |
| `image` | string | `registry.ic-ops.net/cache-docker-hub/timberio/vector` | Image source |
| `image_tag_version` | string | required | Image tag |
| `replicas` | number | `3` | HA matches OTel collector tier |
| `datadog_site` | string | required | e.g. `datadoghq.com` / `us3.datadoghq.com` |
| `datadog_api_key_vault_path` | string | required | Vault path for ExternalSecret |
| `audit_drop_patterns` | list(string) | `["Audit\\s-\\s.+", "\"level\":\"audit\""]` | Day-1 exclude rules |
| `email_redaction_enabled` | bool | `true` | Toggle the email-redact transform |
| `email_redaction_replacement` | string | `"[REDACTED-EMAIL]"` | The replacement marker |
| `additional_config` | any | `null` | Escape-hatch matching `otel_collector_additional_config` semantics |

### 3. `eks-cluster-applications` (this repo)

**3a. New file `vector.tf`** — pure mirror of `otel_collector.tf`'s shape (locals + `module "vector"` call + variable declarations):

```hcl
locals {
  vector_enabled = (
    var.vector_enabled != null
    ? var.vector_enabled
    : lookup(var.vector_enabled_cluster_types, local.cluster_type, true)
  )
  vector_endpoint = "http://vector.vector.svc.cluster.local:4317"
}

module "vector" {
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"
  count  = local.vector_enabled ? 1 : 0

  providers = {
    kubernetes = kubernetes.local-cluster
    helm       = helm.local-cluster
  }

  chart_version              = var.vector_chart_version
  image                      = var.vector_image
  image_tag_version          = var.vector_image_tag
  replicas                   = var.vector_replicas
  datadog_site               = var.vector_datadog_site
  datadog_api_key_vault_path = var.vector_datadog_api_key_vault_path
  audit_drop_patterns        = var.vector_audit_drop_patterns
  email_redaction_enabled    = var.vector_email_redaction_enabled
  email_redaction_replacement = var.vector_email_redaction_replacement
  additional_config          = var.vector_additional_config

  depends_on = [
    module.cilium,
    module.external_secrets,
  ]
}
```

Plus variable declarations matching the otel module's style. `vector_enabled_cluster_types` defaults match OTel: `mgmt/cps/saas = true`, `infra/dev = false`. A `data "external"` validation block enforces `vector_enabled ⇒ external_secrets_enabled` (Vault-backed key requires it).

**3b. Edit `otel_collector.tf`:**
- Bump `?ref=v1.0.7` → `?ref=v1.1.0`.
- Pass `downstream_endpoint = local.vector_enabled ? local.vector_endpoint : ""`.
- Add `depends_on = [module.vector]` so Vector's Service exists before OTel reconciles. (Belt-and-suspenders — OTLP exporter retries handle the runtime race anyway.)

### Out of scope (to call out in spec)
- **Retiring `kubernetes/applications/datadog-processing-rules`** — that's a separate decommissioning task once Vector is proven; the existing Datadog Agent rules ConfigMap stays put for now.
- **Per-cluster rule overrides** — the day-1 rules are identical across `prod` and `staging` in the existing repo, so we don't introduce overlay machinery yet. Operators override per-cluster via the `vector_audit_drop_patterns` Terraform variable if they need to.
- **Dynamic / runtime-loaded rules** — out of scope.

---

**Sanity check before Section 3:** does this layout look right? The one place I'd particularly welcome pushback is the `module "vector"` variable surface — I tried to mirror the otel-collector module's style verbatim, but if there's a different paved-road convention you prefer for new internal-modules, tell me before I commit it to the spec.


Implement Vector with Flux for data redaction


this looks good, continue please good sir


## Section 3 / 7 — Vector configuration (helm values + VRL)

Rendered by the new internal-module; this is the day-1 shape. Vector runs in **aggregator role** (it receives OTLP from another in-cluster service rather than tailing files).

### Pipeline topology

```
                ┌─ logs    ──▶ drop_audit ──▶ redact_emails_logs    ──▶ datadog_logs
otlp_in (4317   │
gRPC + 4318    ─┼─ traces  ──────────────────▶ redact_emails_traces  ──▶ datadog_traces
HTTP)          │
                └─ metrics ──────────────────▶ redact_emails_metrics ──▶ datadog_metrics
```

Three pipelines, one per signal. The `drop_audit` filter runs on logs only (the day-1 rules are message-text patterns, no equivalent semantic on traces/metrics). Email redaction runs on all three.

### VRL — the audit-drop filter (logs only)

Vector's `filter` transform passes events for which the condition returns `true`, so the condition is the *inverse* of what gets dropped:

```toml
[transforms.drop_audit]
type = "filter"
inputs = ["otlp_in.logs"]
condition = '''
  msg = string!(.message ?? "")
  !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"')
'''
```

`audit_drop_patterns` is templated into the helm values from the Terraform variable, so the regex set is operator-controllable. Default ships the two patterns from `datadog-processing-rules/{prod,staging}/rules.json`.

### VRL — the email-redact transform (all three signals)

Per-signal `remap` transforms wrap a shared VRL helper. For the log shape (the simple case):

```toml
[transforms.redact_emails_logs]
type = "remap"
inputs = ["drop_audit"]
source = '''
  email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

  if exists(.message) {
    .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
  }

  # walk top-level attributes for string values containing emails
  if is_object(.attributes) {
    .attributes = map_values(object!(.attributes)) -> |v| {
      if is_string(v) { replace(v, email_re, "[REDACTED-EMAIL]") } else { v }
    }
  }
'''
```

Trace and metric variants apply the same `map_values` walk over `attributes` and `resource.attributes`, plus span-event attributes for traces. The email regex (`[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}`) and the replacement string are both templated from Terraform variables (`vector_email_redaction_replacement` defaults to `[REDACTED-EMAIL]`).

**Limitation called out in spec:** if an email is buried inside a stringified JSON attribute value (e.g. an `http.request.body` attribute holding raw JSON text), the regex still catches it because it operates on the string content. If it's deeply nested in a structured map, only the values reachable by `map_values` are scanned. We don't recurse into nested object trees in v1 — kept simple to avoid a runaway transform on payload-heavy spans. Recursion can be added when there's a concrete need.

### Helm values surface

The internal-module renders these to the upstream `timberio/vector` chart:

```yaml
role: Aggregator
replicas: {{ .Values.replicas }}            # 3 by default

# OTLP source — receives from OTel Collector
customConfig:
  data_dir: /vector-data-dir
  api: { enabled: true, address: 127.0.0.1:8686 }
  sources:
    otlp_in:
      type: opentelemetry
      grpc: { address: 0.0.0.0:4317 }
      http: { address: 0.0.0.0:4318 }
  transforms:
    drop_audit:        { ... }              # see above
    redact_emails_logs: { ... }
    redact_emails_traces: { ... }
    redact_emails_metrics: { ... }
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs: [redact_emails_logs]
      default_api_key: "${DATADOG_API_KEY}"
      site: {{ .Values.datadogSite }}
    datadog_traces:    { type: datadog_traces,  inputs: [redact_emails_traces],  default_api_key: "${DATADOG_API_KEY}", site: ... }
    datadog_metrics:   { type: datadog_metrics, inputs: [redact_emails_metrics], default_api_key: "${DATADOG_API_KEY}", site: ... }

service:
  enabled: true
  type: ClusterIP
  ports:
    - { name: otlp-grpc, port: 4317, protocol: TCP }
    - { name: otlp-http, port: 4318, protocol: TCP }

env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog                # populated by the ExternalSecret
        key: api_key

resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits:   { cpu: 2,    memory: 1Gi }       # tunable via TF vars
```

`additional_config` (escape-hatch variable, mirroring `otel_collector_additional_config`) when set overwrites `customConfig` wholesale — same semantics as OTel.

---

**Move on to Section 4 (the OTel exporter swap detail) when ready, or push back here.** One thing I want to flag: the `[REDACTED-EMAIL]` replacement is a *destructive* transform — once Vector strips the email, it is gone from Datadog forever. If you ever need a "preview mode" (log what *would* have been redacted but pass it through), say so now and I'll add a `vector_email_redaction_dry_run` variable to v1 instead of bolting it on later.


this looks good to me, continue


## Section 4 / 7 — OTel Collector exporter swap

### What changes in the otel-collector helm values

**Before (`v1.0.7`, today):**
```yaml
exporters:
  datadog:
    api:
      key:  ${DATADOG_API_KEY}
      site: datadoghq.com
service:
  pipelines:
    logs:    { receivers: [otlp], processors: [...], exporters: [datadog] }
    traces:  { receivers: [otlp], processors: [...], exporters: [datadog] }
    metrics: { receivers: [otlp], processors: [...], exporters: [datadog] }
```

**After (`v1.1.0` when `downstream_endpoint != ""`):**
```yaml
exporters:
  otlp:
    endpoint: http://vector.vector.svc.cluster.local:4317
    tls:
      insecure: true                  # in-cluster only; Cilium handles transit encryption
    sending_queue:
      enabled: true
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
service:
  pipelines:
    logs:    { receivers: [otlp], processors: [...], exporters: [otlp] }
    traces:  { receivers: [otlp], processors: [...], exporters: [otlp] }
    metrics: { receivers: [otlp], processors: [...], exporters: [otlp] }
```

The whole `exporters.datadog` block disappears when the swap is active. Pipeline `processors` lists are untouched — anything OTel does today (batching, `k8sattributes`, tail-sampling, etc.) stays put.

### Datadog API key ownership

Today the Datadog API key is mounted into the OTel collector pods (presumably via the existing external-secrets stack). After the swap:
- **Vector module owns the secret.** Its ExternalSecret creates `vector-datadog` in the `vector` namespace; Vector pods consume it via `env.DATADOG_API_KEY`.
- **OTel module stops mounting the Datadog secret** when `downstream_endpoint` is set. The `v1.1.0` helm template gates the secret mount on the same conditional that gates the `exporters.datadog` block — both fall away together.
- No flag-day overlap: the Terraform `depends_on = [module.vector]` on the OTel module ensures the Vector Service (and its secret pipeline) is reconciled before OTel reconciles with the new exporter config.

### Failure-mode behavior

What happens if Vector is down or unhealthy:
- OTel's `sending_queue` buffers up to `queue_size: 5000` batches in-memory per pipeline.
- `retry_on_failure` retries with exponential backoff up to 5 minutes total.
- Past that, OTel drops batches and increments `otelcol_exporter_send_failed_*` counters — visible in OTel's own self-telemetry.
- **No persistent queue in v1.** Adding the `file_storage` extension (durable queue across restarts) is a future improvement; in v1, a Vector outage longer than ~5 minutes loses the tail of the buffer. Documented limitation, mitigatable by Vector replica HA (3 replicas, `PodDisruptionBudget`).

### TLS posture

OTLP between OTel and Vector runs as plain gRPC, both endpoints in-cluster. Cilium provides the transport encryption layer (the cluster already runs Cilium). No mTLS between the two components in v1 — that's a future hardening item if compliance later requires defense-in-depth at the application layer.

### Rollout shape

Single `terraform apply` per cluster:

1. Apply provisions Vector (new module) — namespace, helm release, ExternalSecret, all reconcile to ready.
2. Same apply re-renders OTel collector helm release with the new `v1.1.0` chart and `downstream_endpoint` set — OTel pods rolling-restart with the new exporter config.
3. Vector now receives all traffic; Datadog continues to receive (via Vector's sinks).

Rollback: set `vector_enabled = false`. The `local.vector_endpoint` ternary in `otel_collector.tf` falls back to `""`, OTel re-renders with the original `datadog` exporter, OTel pods rolling-restart back to direct-to-Datadog. Vector module destroys cleanly. **The Datadog API key needs to be available to OTel again** during rollback — this means we cannot delete the OTel-side secret reference in the same release that introduces Vector. Captured as a sequencing note in the spec.

---

**Move to Section 5 (operational concerns — HA, observability, alerting on Vector itself) when ready, or push back here.**


this looks good to me, continue to the next section but do not run any terraform commands


## Section 5 / 7 — Operational concerns

### HA topology

- **Replicas:** 3 by default (`vector_replicas`), matching the OTel collector tier.
- **PodDisruptionBudget:** `minAvailable: 2` — voluntary disruption can never take more than one Vector pod at a time.
- **Topology spread:** `topologySpreadConstraints` on `topology.kubernetes.io/zone` with `maxSkew: 1` — pods land in different AZs by default. Module gives operators a passthrough variable to override on small clusters where AZ spread is impossible.
- **Anti-affinity:** preferred (not required) on `kubernetes.io/hostname` — soft preference for separate nodes, but doesn't block scheduling on tight clusters.

### Vector's own telemetry

Vector emits Prometheus-format internal metrics on `:8686/metrics`. Module ships a `ServiceMonitor` (assuming the platform's existing Prometheus operator picks it up — same pattern Datadog/Sysdig agents would use today). Key metrics worth dashboards for:

| Metric | What it tells you |
|---|---|
| `vector_component_received_events_total` | Inbound rate per source/transform/sink |
| `vector_component_sent_events_total` | Outbound rate at each Datadog sink |
| `vector_component_errors_total` | Failed sends (rate of errors per sink) |
| `vector_component_discarded_events_total` | Events the `drop_audit` filter dropped (sanity-check the rule isn't over-matching) |
| `vector_buffer_events` | In-memory queue depth toward each Datadog sink |
| `vector_utilization` | Per-component utilization, 0–1 |

Vector's own logs go to stdout — captured by whatever log path the cluster uses today (the existing OTel pipeline... which means a Vector outage means we can't see Vector's logs through Vector. Worth noting). For first-line debugging during a Vector outage, `kubectl logs -n vector` is the path of last resort.

### Alerting (sketch — actual alert wiring lives in your Datadog monitors, not in the module)

| Condition | Severity | Why |
|---|---|---|
| `vector_component_errors_total{sink=~"datadog_.*"}` rate > 0 for 5m | warn | Datadog sink failing — telemetry is queueing or dropping |
| `vector_buffer_events / vector_buffer_max_events > 0.8` for 5m | warn | Outbound queue filling; backpressure imminent |
| `vector_buffer_events / vector_buffer_max_events > 0.95` for 2m | page | About to drop |
| Vector pod ready replicas < 2 for 5m | page | Below PDB floor; one more failure and we're down |
| `otelcol_exporter_send_failed_*{exporter="otlp"}` rate > 0 for 5m on otel-collector | warn | OTel→Vector path broken (Vector unreachable or rejecting) |

These live as Datadog monitors, not Terraform — out of scope for the module itself, but the spec calls them out as *required follow-up* before declaring v1 production-ready.

### Sizing & resource defaults

Day-1 defaults in the module:
```yaml
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits:   { cpu: 2,    memory: 1Gi }
```

These are conservative starting points. Vector's CPU usage scales with VRL transform complexity (regex on every event isn't free — the email regex runs on every log message and every string-valued attribute in every span). Real sizing comes from the metrics above after a week or two in `mgmt`/`cps` and tuning before rolling to `saas`. The spec recommends staging to a low-traffic cluster type first.

### Vault / ExternalSecret dependency chain

```
Vault ──▶ external-secrets operator ──▶ ExternalSecret (vector ns)
                                              │
                                              ▼
                                         Secret: vector-datadog
                                              │
                                              ▼
                                  Vector pods (env DATADOG_API_KEY)
```

The Flux-app `healthChecks` pattern from `license-manager`'s bootstrap doesn't apply here because we're in Terraform — instead, `module "vector"` uses `kubernetes_manifest.wait` to block until the resulting Secret object exists. Without that wait, Vector's helm release can race ahead of secret hydration and the pods CrashLoop on startup. Captured in the module's reconcile order:

1. Namespace
2. ExternalSecret (and wait until the underlying Secret resource is observed)
3. Helm release for Vector

### Bootstrap-order edge case (first-time cluster provisioning)

A net-new cluster runs `terraform apply` once for layer-3 (`eks-cluster-applications`). Within that apply:
- `external_secrets` reconciles first (existing dependency).
- `vector` reconciles next (new — `depends_on = [module.cilium, module.external_secrets]`).
- `otel_collector` reconciles last because of the new `depends_on = [module.vector]`.

This ordering means there's never an OTel collector running with `downstream_endpoint` set before Vector is ready to receive. Terraform handles it. The day-1 ordering is also the "Vector outage" recovery path — toggle `vector_enabled=false`, OTel reverts to direct-to-Datadog, fix Vector, re-enable.

### Data-loss budget for v1

| Scenario | Loss |
|---|---|
| Vector single-pod restart | None (other replicas serve via Service load-balancing) |
| Vector full-outage < 5 min | None (OTel `sending_queue` + `retry_on_failure` covers) |
| Vector full-outage > 5 min | Tail of OTel queue dropped; Datadog gap proportional to outage minus 5 min |
| OTel collector full-outage | Same shape as today — apps' own OTLP exporters retry/buffer per their config |
| Cluster Datadog API key rotation | None if rotated in Vault first; ExternalSecret refreshes the K8s secret; Vector picks up via [secret-reload mechanism — Vector watches env-from-secret, but doesn't auto-reload by default; needs sidecar reloader OR pod restart on secret change]. **Action item: confirm reload mechanism in module impl, document.** |

The last line is a real open question I want to flag now rather than sweep under: Vector doesn't natively re-read env-mounted secrets. We either (a) use the existing `reloader` operator (I see `reloader.tf` in the module — it's already there!) by annotating the HelmRelease, or (b) accept manual rolling-restart on key rotation. Option (a) wins for free. Module sets the `reloader.stakater.com/auto: "true"` annotation on the Vector Deployment.

---

**Section 6 next (testing & verification plan), or push back here.**


yes use the stakater reloader annotation


## Section 6 / 7 — Testing & verification plan

### Pre-deploy (local / CI)

1. **Terraform plan** in a sandbox state to confirm the module shape is sane and `count = local.vector_enabled ? 1 : 0` toggles cleanly.
2. **Helm template render** of both modules (`vector` and the bumped `otel-collector`) — diff against current rendered manifests in CI to catch unintended changes.
3. **`vector validate`** on the rendered config — Vector has a config-validation subcommand that catches VRL syntax errors and source/transform/sink misconfiguration before the chart is applied. CI step in the new `internal-tooling/vector` (terraform module) repo runs this against the templated values.
4. **VRL unit tests** via `vector vrl --input ... --program ...`:
   - Audit-drop: feed `{"message":"Audit - login by user"}` → expect filtered. Feed `{"message":"INFO request handled"}` → expect passed.
   - Email redact: feed `{"message":"contact alice@example.com for info"}` → expect `.message == "contact [REDACTED-EMAIL] for info"`. Feed event with no email → expect unchanged.
   - Edge: dotted addresses (`a.b.c@d.e.co.uk`), plus-tag (`a+tag@example.com`), email inside a JSON-stringified attribute. Test corpus committed alongside the module.

### Integration (in a dev/sandbox EKS cluster)

Goal: prove the wire end-to-end before any prod cluster sees it. Use one of the `dev`-type clusters (where `vector_enabled` defaults to `false`) and explicitly flip `vector_enabled = true`.

1. **Smoke test:** apply the module, confirm Vector pods Ready, OTel pods rolled with new `otlp` exporter, no `otelcol_exporter_send_failed` errors.
2. **Audit-drop verification:** emit a log event from a test pod with body `Audit - test-event-<uuid>`. Wait 60s. Search Datadog for the UUID. **Pass = no result.** Repeat for `"level":"audit"` payload.
3. **Email-redact verification:** emit a log event with body `Reach me at e2e-<uuid>@example.com`. Search Datadog for the UUID. **Pass = log appears, body contains `[REDACTED-EMAIL]`, does not contain the original local-part.**
4. **Span-attribute redact:** emit a trace with span attribute `user.email = e2e-<uuid>@example.com`. Inspect span in Datadog APM. **Pass = attribute value is `[REDACTED-EMAIL]`.**
5. **Negative test (no over-matching):** emit logs containing email-shaped strings inside backticks/quoted code blocks (`` `regex: [a-z]+@[a-z]+` ``). Confirm the regex matches and redacts (we expect this — the regex isn't context-aware, and a code-snippet redaction is acceptable v1 behavior). Documented as known limitation, not a failure.

### Failure-mode tests (sandbox cluster)

1. **Vector pod restart:** `kubectl delete pod -n vector -l app=vector`. Watch `vector_component_sent_events_total` — expect a brief dip during rolling restart, no Datadog gap visible after 1m.
2. **Vector full-stop:** `kubectl scale deploy/vector -n vector --replicas=0`. Generate ~60s of telemetry. Scale back to 3. **Pass = OTel collector queue depth grows then drains; no permanent drop visible in Datadog.**
3. **Vector full-stop > 5 min:** same scale-to-zero, hold for 7 minutes. **Expect** Datadog gap of ~2 min (the overage past the retry budget). Confirms the documented data-loss budget.
4. **Rollback drill:** set `vector_enabled = false`, `terraform apply`. Confirm OTel re-renders with the `datadog` exporter, telemetry resumes direct-to-Datadog, Vector resources destroyed cleanly.
5. **Datadog API key rotation:** rotate the secret in Vault. Confirm reloader operator restarts Vector pods (we annotated the Deployment with `reloader.stakater.com/auto: "true"`). Verify telemetry resumes.

### Production rollout sequence

The `vector_enabled_cluster_types` defaults give us natural staging:

1. **Sandbox / `dev`-type cluster** — explicit opt-in, full integration suite above.
2. **One `mgmt` cluster** (lowest customer-traffic of the enabled types) — soak 3-7 days. Watch `vector_component_errors_total`, queue depths, and Datadog ingestion volumes. Tune resource limits if needed.
3. **Remaining `mgmt` clusters + all `cps` clusters** — incremental.
4. **`saas` clusters last**, region-by-region. Customer-facing telemetry path; lowest-risk regions first.

Each promotion gate: green dashboards (no error-rate spikes, queue depths healthy) for 48h on the previous tier.

### Acceptance criteria for v1

- All `*_enabled` clusters running Vector with no `otelcol_exporter_send_failed` errors over a 7-day window.
- Datadog ingestion volume for logs/traces/metrics post-rollout matches pre-rollout within ±5% (modulo the audit-drop and redact-driven reduction, which is expected and quantifiable).
- A representative sample of logs/spans pulled from Datadog shows zero unredacted email addresses (sample query: `@message:*@*.* AND -@message:"\[REDACTED-EMAIL\]"`).
- The two day-1 audit patterns produce zero hits in a Datadog log search post-rollout.
- Datadog monitors from Section 5 deployed and silent for the first 7 days outside of the planned rollout-induced restarts.
- Runbook entry added (TBD location — note this in spec) covering: Vector pod debug, queue-full triage, key rotation, rollback to direct-Datadog.

---

**Section 7 (out-of-scope / phased work) is the last one. Continue, or stop here?**


continue to section 7, do not run any kubernetes or terraform commands


## Section 7 / 7 — Out of scope, phased work, open questions

### Explicitly out of scope for v1 (called out in earlier sections, restated here for the doc)

- **Decommissioning `kubernetes/applications/datadog-processing-rules`.** That directory's prod/staging ConfigMaps remain in place. They become dead weight once Vector is doing the equivalent work, but removing them is a separate change with its own coordination (confirming nothing else consumes them, deleting the kustomize app from the relevant Datadog Agent deployments). Tracked separately.
- **Per-cluster rule overlay machinery** (Kustomize-style per-environment rule sets). Defaults today are identical between prod and staging, so we ship a single Terraform variable surface. Overlays are added the first time a cluster genuinely needs a different rule set.
- **Dynamic / runtime-loaded rules** (rules pulled from a config service, customer/tenant-driven scrubbing). VRL config is static-at-deploy-time in v1.
- **Persistent (`file_storage`) queue on the OTel collector.** In-memory `sending_queue` only. Durable cross-restart queueing is a hardening item if the v1 data-loss budget proves insufficient.
- **mTLS between OTel collector and Vector.** Plain gRPC in-cluster, Cilium handles transit. App-layer mTLS is added when/if compliance asks.
- **Recursive descent in the email-redact VRL transform.** v1 walks one level of attributes via `map_values`. If a nested object tree turns out to commonly carry emails (e.g. structured request bodies as nested objects), recursion can be added — but it's a measurable CPU cost and isn't justified absent a specific case.
- **Vector Agent-mode (DaemonSet) for node-level collection.** Aggregator-only. We collect via OTel; Vector is the redaction stage, not a log-tailer.
- **Datadog monitor wiring.** Called out in Section 5 as required follow-up. Lives in whatever Datadog-as-Code surface this org uses, not in the Terraform module.
- **Operations runbook.** Required follow-up. Captures: Vector pod debug, queue-full triage, key rotation, rollback to direct-Datadog. Lives wherever your existing runbooks live (TBD — placeholder in spec for you to fill).

### Phased work — what we'd do *after* v1 ships, in roughly priority order

1. **Expand redaction patterns beyond email.** Once the email pattern is proven in prod, add SSN / credit-card / API-token / JWT / phone-number patterns. Each ships as an opt-in `vector_<category>_redaction_enabled` flag in the module. Same VRL shape — just more `replace` calls, ordered carefully (most-specific first).
2. **Field/attribute name allow-deny lists.** Some attributes — `db.statement`, `http.request.body`, `aws.s3.key` — are higher-risk and worth wholesale stripping rather than pattern-matching. A `vector_drop_attributes` list variable hashes/removes those by attribute name.
3. **Service-level allow/deny.** Drop telemetry from specific services entirely (e.g., a service that handles raw customer payloads where redaction can't be exhaustive). Implemented as a `route` transform on the `service.name` resource attribute.
4. **Decommission `datadog-processing-rules`.** Once Vector has been clean in prod for a known soak period (suggest: one quarter), remove the legacy Datadog Agent ConfigMap path.
5. **Persistent queue on OTel.** If the data-loss budget proves insufficient (Vector outages > 5min observed > some threshold), add `file_storage` extension on the OTel side.
6. **Runtime-driven rule sources.** If the redaction surface keeps growing such that Terraform applies become a friction point, move rules behind a `kubernetes` or `http` source that Vector polls — no chart restart on rule change.

### Open questions to resolve in implementation (flagged but not blocking the spec)

1. **Datadog site value.** I assumed it's templated through, but the actual value (`datadoghq.com`, `us3.datadoghq.com`, etc.) needs to come from somewhere — likely an existing variable on the OTel module that gets re-used as `vector_datadog_site`. To confirm during implementation by reading the OTel internal-module repo (which I haven't read in this session — only the wrapper in `eks-cluster-applications`).
2. **Vault path for the Datadog API key.** Same as above — find where OTel reads it today, point Vector at the same path. Likely the existing `external_secrets_*` variable cluster.
3. **Whether the OTel internal-module already mounts the Datadog secret as env or via a separate secret-volume.** Affects how cleanly the v1.1.0 bump removes that mount when `downstream_endpoint` is set.
4. **`datadog_agent.tf` overlap.** I noted `datadog_agent.tf` and `sysdig_agent.tf` exist alongside the OTel module. Question 2 confirmed the data flow is `apps → OTel → Datadog`, but the Datadog Agent module itself may still be deployed for non-OTel reasons (host metrics, kubelet stats, processes). Vector + OTel doesn't replace it; the Datadog Agent side of the world is untouched.

### Transition to writing the design doc

That's the seven-section walkthrough.

If the design as a whole holds together for you, my next move is:

1. Write the spec to `docs/superpowers/specs/2026-05-05-vector-redaction-design.md` (per the brainstorming skill convention — date-stamped under your IAC repo's `docs/` tree, or wherever you'd like specs to live; tell me if a different home is preferred).
2. Commit it.
3. Run an inline self-review for placeholders, contradictions, ambiguity, scope.
4. Hand it back to you to read in full before we move to writing the implementation plan.

**Ready for me to write the spec, or are there sections you want revisited first?**


write the spec


Updated task #4 status


Updated task #5 status


Bash

ls /Users/steve.briggs/projects/IAC/docs 2>/dev/null && echo "DOCS_EXISTS" || echo "DOCS_MISSING"; ls /Users/steve.briggs/projects/IAC/ | head -20

Check if docs directory exists in IAC repo


DOCS_MISSING
drwxr-xr-x steve.briggs staff 128 B  Tue Jul 16 11:06:00 2024 applications
drwxr-xr-x steve.briggs staff 160 B  Tue Jul 16 11:06:10 2024 vault
drwxr-xr-x steve.briggs staff 160 B  Tue Jul 16 11:06:38 2024 testing
drwxr-xr-x steve.briggs staff 128 B  Tue Jul 16 11:08:38 2024 retool-protected
drwxr-xr-x steve.briggs staff 128 B  Tue Jul 16 11:09:00 2024 lambda
drwxr-xr-x steve.briggs staff 384 B  Tue Jul 16 11:09:32 2024 devdb-bootstrap
drwxr-xr-x steve.briggs staff 128 B  Tue Jul 16 11:09:35 2024 ci-cd
drwxr-xr-x steve.briggs staff 448 B  Tue Jul 16 11:09:38 2024 retool
drwxr-xr-x steve.briggs staff 416 B  Thu Oct 24 10:04:14 2024 immanager
drwxr-xr-x steve.briggs staff 192 B  Mon Oct 28 14:38:05 2024 deployment-actions
drwxr-xr-x steve.briggs staff 256 B  Wed Jun 18 13:19:52 2025 containers
drwxr-xr-x steve.briggs staff 320 B  Fri Feb 20 13:17:28 2026 github-immuta
drwxr-xr-x steve.briggs staff 320 B  Tue Mar  3 16:40:40 2026 kubernetes
drwxr-xr-x steve.briggs staff 224 B  Thu Mar 26 12:04:23 2026 terraform
.rw-r--r-- steve.briggs staff 6.0 KB Thu Mar 26 12:45:39 2026 .DS_Store
drwxr-xr-x steve.briggs staff 224 B  Tue Apr 28 11:42:17 2026 application-deployment
drwxr-xr-x steve.briggs staff 1.5 KB Thu Apr 30 14:28:26 2026 ..
drwxr-xr-x steve.briggs staff 608 B  Tue May  5 11:34:59 2026 .
drwxr-xr-x steve.briggs staff 128 B  Tue May  5 12:02:00 2026 .claude


Write

# Vector + OpenTelemetry Redaction Pipeline — Design Spec

**Date:** 2026-05-05
**Author:** steve.briggs@immuta.com (with Claudebob)
**Status:** Draft, pending review

---

## 1. Overview

### Problem

Today the telemetry path on EKS clusters in this org is:

```
apps → otel-lb → otel-collector → Datadog (logs, APM, metrics)
```

Sensitive-data exclusion lives in **Datadog Agent processing rules** under
`kubernetes/applications/datadog-processing-rules`. Both `prod` and `staging` ship
the same `rules.json` containing two `exclude_at_match` entries that drop Immuta
audit log lines before they reach Datadog. `apm.json` is empty.

These rules are tied to the Datadog Agent's processing pipeline and only operate
on logs. They cannot redact span attributes or metric labels, and there is no
straightforward way to add modify-style transforms (e.g., replace email
addresses with a placeholder) within that surface.

### Goal

Insert **Vector** as a new in-cluster aggregator between the OTel collector and
Datadog. Vector replaces today's Datadog-side exclusion rules with VRL-based
filter and remap transforms, applied uniformly across logs, traces, and metrics.

The day-1 rule set replicates today's two audit exclusions and adds email
redaction across all three signals. The architecture leaves room to grow
additional patterns (SSN, credit cards, tokens, attribute drops, service-level
allow/deny) without further structural change.

### Non-goals (v1)

- Replacing the OTel collector.
- Removing the Datadog Agent (it remains for host metrics / kubelet stats).
- Decommissioning `kubernetes/applications/datadog-processing-rules` (separate effort).
- Per-cluster rule overlays beyond the Terraform variable surface.
- Dynamic / runtime-loaded rules.
- mTLS between OTel and Vector.
- Vector Agent-mode / DaemonSet collection.

---

## 2. Architecture

```
                                     [ in-cluster ]
  ┌──────────┐  OTLP  ┌─────────┐  OTLP  ┌────────────────┐  OTLP  ┌──────────────────┐ Datadog
  │   apps   │ ─────▶ │ otel-lb │ ─────▶ │ otel-collector │ ─────▶ │ vector aggregator│ ──────▶  Datadog SaaS
  └──────────┘        └─────────┘        └────────────────┘        │ - OTLP source    │  (logs, APM,
                                                                   │ - VRL transforms │   metrics)
                                                                   │ - Datadog sinks  │
                                                                   └──────────────────┘
```

**Components:**

- **OTel collector** keeps its current shape (`otel-lb` front tier + `otel-collector` back tier). Its terminal exporter switches from `datadog` to `otlp` (gRPC), pointed at the Vector Service.
- **Vector** runs as a Deployment with N replicas (default 3, matching OTel's tier), fronted by a ClusterIP Service. Receives OTLP, applies VRL transforms, ships to Datadog via Vector's native `datadog_logs` / `datadog_traces` / `datadog_metrics` sinks.
- **Datadog API key** moves from the OTel exporter config to Vector. Sourced from Vault via the existing `external-secrets` operator.

**Cluster scope.** Vector is enabled on the same cluster types as the OTel
collector by default: `mgmt`, `cps`, `saas` true; `infra`, `dev` false. A single
`vector_enabled` bool variable overrides the per-cluster-type default, matching
the established pattern in this module.

---

## 3. Terraform layout

Three changes across two repos.

### 3.1 `internal-modules/kubernetes/monitoring/otel-collector` v1.0.7 → v1.1.0

Backwards-compatible bump. New variables:

| Variable | Type | Default | Purpose |
|---|---|---|---|
| `downstream_endpoint` | string | `""` | When non-empty, swaps terminal exporter to OTLP |
| `downstream_protocol` | string | `"otlp/grpc"` | Wire protocol |

When `downstream_endpoint` is non-empty, the rendered helm values gain an
`exporters.otlp` block and `service.pipelines.{logs,traces,metrics}.exporters`
swap from `[datadog]` to `[otlp]`. The Datadog exporter block and its API-key
mount fall away under the same conditional. Empty preserves today's
direct-to-Datadog behavior — clusters that don't enable Vector see no behavioral
change.

### 3.2 `internal-modules/kubernetes/monitoring/vector` v1.0.0 (new module)

Mirrors the shape of the `otel-collector` internal-module: same provider wiring
(`kubernetes`, `helm`), same variable patterns, same approach to image / chart /
replica overrides.

What it creates:

- `kubernetes_namespace.vector` — the `vector` namespace.
- `kubernetes_manifest` for an `ExternalSecret` pulling the Datadog API key from Vault, using the same external-secrets cluster store the rest of the platform uses. Terraform waits for the underlying Secret object before proceeding.
- `helm_release.vector` — wraps the upstream `timberio/vector` chart with values templated from the variables below. Deployment annotated with `reloader.stakater.com/auto: "true"` so the existing reloader operator restarts pods on Datadog-key rotation.

Module variables (initial set):

| Variable | Type | Default | Purpose |
|---|---|---|---|
| `chart_version` | string | required | Vector helm chart version pin |
| `image` | string | `registry.ic-ops.net/cache-docker-hub/timberio/vector` | Image source |
| `image_tag_version` | string | required | Image tag |
| `replicas` | number | `3` | HA matches OTel collector tier |
| `datadog_site` | string | required | e.g. `datadoghq.com` / `us3.datadoghq.com` |
| `datadog_api_key_vault_path` | string | required | Vault path for ExternalSecret |
| `audit_drop_patterns` | list(string) | `["Audit\\s-\\s.+", "\"level\":\"audit\""]` | Day-1 exclude rules |
| `email_redaction_enabled` | bool | `true` | Toggle email redaction |
| `email_redaction_replacement` | string | `"[REDACTED-EMAIL]"` | Replacement marker |
| `additional_config` | any | `null` | Escape-hatch — overwrites `customConfig` wholesale, mirroring `otel_collector_additional_config` semantics |
| `resources_requests` / `resources_limits` | map | see Section 6 | Resource tuning |

### 3.3 `eks-cluster-applications` (this repo)

**New file `vector.tf`** — pure mirror of `otel_collector.tf`'s shape:

```hcl
locals {
  vector_enabled = (
    var.vector_enabled != null
    ? var.vector_enabled
    : lookup(var.vector_enabled_cluster_types, local.cluster_type, true)
  )
  vector_endpoint = "http://vector.vector.svc.cluster.local:4317"
}

module "vector" {
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"
  count  = local.vector_enabled ? 1 : 0

  providers = {
    kubernetes = kubernetes.local-cluster
    helm       = helm.local-cluster
  }

  chart_version               = var.vector_chart_version
  image                       = var.vector_image
  image_tag_version           = var.vector_image_tag
  replicas                    = var.vector_replicas
  datadog_site                = var.vector_datadog_site
  datadog_api_key_vault_path  = var.vector_datadog_api_key_vault_path
  audit_drop_patterns         = var.vector_audit_drop_patterns
  email_redaction_enabled     = var.vector_email_redaction_enabled
  email_redaction_replacement = var.vector_email_redaction_replacement
  additional_config           = var.vector_additional_config

  depends_on = [
    module.cilium,
    module.external_secrets,
  ]
}
```

Plus variable declarations matching the otel-collector module's style.
`vector_enabled_cluster_types` defaults match OTel: `mgmt/cps/saas = true`,
`infra/dev = false`. A `data "external"` validation block enforces
`vector_enabled ⇒ external_secrets_enabled` (the Vault-backed key requires it).

**Edits to `otel_collector.tf`:**

- Bump `?ref=v1.0.7` → `?ref=v1.1.0`.
- Pass `downstream_endpoint = local.vector_enabled ? local.vector_endpoint : ""`.
- Add `depends_on = [module.vector]` so Vector's Service exists before OTel reconciles. (Belt-and-suspenders — OTLP exporter retries handle the runtime race.)

---

## 4. Vector configuration

Vector runs in **aggregator role** (it receives OTLP from another in-cluster
service rather than tailing files).

### 4.1 Pipeline topology

```
                ┌─ logs    ──▶ drop_audit ──▶ redact_emails_logs    ──▶ datadog_logs
otlp_in (4317   │
gRPC + 4318    ─┼─ traces  ──────────────────▶ redact_emails_traces  ──▶ datadog_traces
HTTP)          │
                └─ metrics ──────────────────▶ redact_emails_metrics ──▶ datadog_metrics
```

Three pipelines, one per signal. The `drop_audit` filter runs on logs only (the
day-1 rules are message-text patterns). Email redaction runs on all three.

### 4.2 VRL — audit-drop filter (logs only)

Vector's `filter` transform passes events for which the condition returns
`true`, so the condition is the inverse of what gets dropped:

```toml
[transforms.drop_audit]
type = "filter"
inputs = ["otlp_in.logs"]
condition = '''
  msg = string!(.message ?? "")
  !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"')
'''
```

`audit_drop_patterns` is templated into the helm values from the Terraform
variable. The default ships the two patterns from
`datadog-processing-rules/{prod,staging}/rules.json`.

### 4.3 VRL — email-redact transform (all three signals)

Per-signal `remap` transforms apply the same shared pattern. Log shape (the
simple case):

```toml
[transforms.redact_emails_logs]
type = "remap"
inputs = ["drop_audit"]
source = '''
  email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

  if exists(.message) {
    .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
  }

  # walk top-level attributes for string values containing emails
  if is_object(.attributes) {
    .attributes = map_values(object!(.attributes)) -> |v| {
      if is_string(v) { replace(v, email_re, "[REDACTED-EMAIL]") } else { v }
    }
  }
'''
```

Trace and metric variants apply the same `map_values` walk over `attributes`
and `resource.attributes`, plus span-event attributes for traces. The email
regex and the replacement string are both templated from Terraform variables.

**Limitation called out:** if an email is buried inside a stringified JSON
attribute value (e.g. `http.request.body` holding raw JSON text), the regex
catches it because it operates on string content. If it's deeply nested in a
structured object map, only values reachable by `map_values` are scanned. v1
does **not** recurse into nested object trees — kept simple to avoid runaway
transform CPU on payload-heavy spans. Recursion can be added when justified.

### 4.4 Helm values surface

The internal-module renders to the upstream `timberio/vector` chart:

```yaml
role: Aggregator
replicas: {{ .Values.replicas }}            # 3 by default

customConfig:
  data_dir: /vector-data-dir
  api: { enabled: true, address: 127.0.0.1:8686 }
  sources:
    otlp_in:
      type: opentelemetry
      grpc: { address: 0.0.0.0:4317 }
      http: { address: 0.0.0.0:4318 }
  transforms:
    drop_audit:           { ... }            # see 4.2
    redact_emails_logs:    { ... }
    redact_emails_traces:  { ... }
    redact_emails_metrics: { ... }
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs: [redact_emails_logs]
      default_api_key: "${DATADOG_API_KEY}"
      site: {{ .Values.datadogSite }}
    datadog_traces:    { type: datadog_traces,  inputs: [redact_emails_traces],  default_api_key: "${DATADOG_API_KEY}", site: ... }
    datadog_metrics:   { type: datadog_metrics, inputs: [redact_emails_metrics], default_api_key: "${DATADOG_API_KEY}", site: ... }

service:
  enabled: true
  type: ClusterIP
  ports:
    - { name: otlp-grpc, port: 4317, protocol: TCP }
    - { name: otlp-http, port: 4318, protocol: TCP }

env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key

podAnnotations:
  reloader.stakater.com/auto: "true"

resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits:   { cpu: 2,    memory: 1Gi }
```

`additional_config` (escape-hatch variable) when set overwrites `customConfig`
wholesale, matching `otel_collector_additional_config` semantics.

---

## 5. OTel collector exporter swap

### 5.1 Helm values diff

**Before (`v1.0.7`, today):**

```yaml
exporters:
  datadog:
    api:
      key:  ${DATADOG_API_KEY}
      site: datadoghq.com
service:
  pipelines:
    logs:    { receivers: [otlp], processors: [...], exporters: [datadog] }
    traces:  { receivers: [otlp], processors: [...], exporters: [datadog] }
    metrics: { receivers: [otlp], processors: [...], exporters: [datadog] }
```

**After (`v1.1.0` when `downstream_endpoint != ""`):**

```yaml
exporters:
  otlp:
    endpoint: http://vector.vector.svc.cluster.local:4317
    tls:
      insecure: true                  # in-cluster only; Cilium handles transit
    sending_queue:
      enabled: true
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
service:
  pipelines:
    logs:    { receivers: [otlp], processors: [...], exporters: [otlp] }
    traces:  { receivers: [otlp], processors: [...], exporters: [otlp] }
    metrics: { receivers: [otlp], processors: [...], exporters: [otlp] }
```

The `exporters.datadog` block disappears. Pipeline `processors` lists are
untouched — anything OTel does today (batching, `k8sattributes`, tail-sampling)
stays put.

### 5.2 Datadog API key ownership

- **Before:** OTel collector pods consume the Datadog API key (mount mechanism per the OTel internal-module's existing pattern).
- **After:** Vector module owns the secret. Its ExternalSecret creates `vector-datadog` in the `vector` namespace; Vector pods consume it via `env.DATADOG_API_KEY`. The OTel module's secret mount falls away under the same `downstream_endpoint != ""` conditional that gates the exporter swap.
- **Sequencing:** `module.otel_collector` has `depends_on = [module.vector]`. Single `terraform apply` brings up Vector first, then re-renders OTel.

### 5.3 Failure-mode behavior

If Vector is down or unhealthy:

- OTel's `sending_queue` buffers up to `queue_size: 5000` batches in-memory per pipeline.
- `retry_on_failure` retries with exponential backoff up to 5 minutes total.
- Past that, OTel drops batches and increments `otelcol_exporter_send_failed_*`.
- **No persistent queue in v1.** A Vector outage longer than ~5 minutes loses the buffer tail. Mitigation: Vector replica HA (3 replicas, PDB minAvailable 2). `file_storage` extension is a future hardening item.

### 5.4 TLS posture

OTLP between OTel and Vector runs as plain gRPC, both endpoints in-cluster.
Cilium provides transport encryption. App-layer mTLS is a future hardening
item.

### 5.5 Rollout shape (single cluster)

Single `terraform apply`:

1. `module.vector` provisions — namespace, ExternalSecret, helm release reconcile to ready.
2. Same apply re-renders OTel collector helm release with `v1.1.0` and `downstream_endpoint` set — OTel pods rolling-restart with new exporter config.
3. Vector now receives all traffic; Datadog continues to receive (via Vector's sinks).

**Rollback:** set `vector_enabled = false`. The `local.vector_endpoint` ternary
in `otel_collector.tf` falls back to `""`; OTel re-renders with the original
`datadog` exporter, OTel pods rolling-restart back to direct-to-Datadog. Vector
module destroys cleanly. The Datadog API key path on the OTel side must remain
available across the rollback — this is a pre-condition; the OTel-side secret
reference is not deleted in the same release that introduces Vector.

---

## 6. Operational concerns

### 6.1 HA topology

- **Replicas:** 3 default (`vector_replicas`).
- **PodDisruptionBudget:** `minAvailable: 2`.
- **Topology spread:** `topologySpreadConstraints` on `topology.kubernetes.io/zone`, `maxSkew: 1`. Module gives operators a passthrough variable to override on small clusters where AZ spread is impossible.
- **Anti-affinity:** preferred (not required) on `kubernetes.io/hostname`.

### 6.2 Vector's own telemetry

Vector emits Prometheus-format internal metrics on `:8686/metrics`. Module
ships a `ServiceMonitor`. Key metrics:

| Metric | What it tells you |
|---|---|
| `vector_component_received_events_total` | Inbound rate per source/transform/sink |
| `vector_component_sent_events_total` | Outbound rate at each Datadog sink |
| `vector_component_errors_total` | Failed sends per sink |
| `vector_component_discarded_events_total` | Events dropped by `drop_audit` |
| `vector_buffer_events` | In-memory queue depth toward each Datadog sink |
| `vector_utilization` | Per-component utilization, 0–1 |

Vector's own logs go to stdout and are captured by the existing OTel pipeline.
A Vector outage means we cannot see Vector's logs through Vector. First-line
debug: `kubectl logs -n vector`.

### 6.3 Alerting (sketch — actual monitor wiring out of scope)

| Condition | Severity | Why |
|---|---|---|
| `vector_component_errors_total{sink=~"datadog_.*"}` rate > 0 for 5m | warn | Datadog sink failing |
| `vector_buffer_events / vector_buffer_max_events > 0.8` for 5m | warn | Outbound queue filling |
| `vector_buffer_events / vector_buffer_max_events > 0.95` for 2m | page | About to drop |
| Vector ready replicas < 2 for 5m | page | Below PDB floor |
| `otelcol_exporter_send_failed_*{exporter="otlp"}` rate > 0 for 5m | warn | OTel→Vector path broken |

Datadog monitor wiring lives in this org's Datadog-as-Code surface, not the
Terraform module. **Required follow-up before declaring v1 production-ready.**

### 6.4 Sizing & resource defaults

```yaml
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits:   { cpu: 2,    memory: 1Gi }
```

Conservative starting points. Vector CPU usage scales with VRL transform
complexity (regex on every event isn't free). Real sizing comes from
observed metrics after a soak in `mgmt`/`cps` before rolling to `saas`.

### 6.5 Vault / ExternalSecret dependency chain

```
Vault ──▶ external-secrets operator ──▶ ExternalSecret (vector ns)
                                              │
                                              ▼
                                         Secret: vector-datadog
                                              │
                                              ▼
                                  Vector pods (env DATADOG_API_KEY)
```

The Vector module uses `kubernetes_manifest.wait` on the ExternalSecret's
materialized Secret to block the helm release until secret hydration completes.
Without that wait, the helm release races ahead and Vector pods CrashLoop on
startup. Reconcile order within the module:

1. Namespace
2. ExternalSecret (and wait until the underlying Secret resource is observed)
3. Helm release for Vector

### 6.6 Datadog API key rotation

Rotate the secret in Vault. ExternalSecret refreshes the in-cluster Secret. The
existing **`reloader` operator** picks up the change and restarts Vector pods —
the Vector Deployment is annotated with `reloader.stakater.com/auto: "true"`.
No manual intervention required.

### 6.7 Bootstrap-order edge case (first-time cluster provisioning)

A net-new cluster runs `terraform apply` once for layer-3
(`eks-cluster-applications`). Within that apply:

1. `external_secrets` reconciles first (existing dependency).
2. `vector` reconciles next (new — `depends_on = [module.cilium, module.external_secrets]`).
3. `otel_collector` reconciles last (`depends_on = [module.vector]`).

There is never an OTel collector running with `downstream_endpoint` set before
Vector is ready to receive.

### 6.8 Data-loss budget for v1

| Scenario | Loss |
|---|---|
| Vector single-pod restart | None — other replicas serve via Service load-balancing |
| Vector full-outage < 5 min | None — OTel `sending_queue` + `retry_on_failure` covers |
| Vector full-outage > 5 min | Tail of OTel queue dropped; Datadog gap proportional to outage minus 5 min |
| OTel collector full-outage | Same shape as today — apps' OTLP exporters retry/buffer per their config |
| Datadog API key rotation | None — reloader-driven pod restart |

---

## 7. Testing & verification plan

### 7.1 Pre-deploy (local / CI)

1. **Terraform plan** in a sandbox state to confirm module shape, count toggles cleanly.
2. **Helm template render** of both modules, diff against current rendered manifests in CI.
3. **`vector validate`** on the rendered config — catches VRL syntax errors. CI step in the new `internal-modules/kubernetes/monitoring/vector` repo.
4. **VRL unit tests** via `vector vrl --input ... --program ...`:
   - Audit-drop: `{"message":"Audit - login by user"}` → filtered. `{"message":"INFO request handled"}` → passed.
   - Email redact: `{"message":"contact alice@example.com"}` → `.message == "contact [REDACTED-EMAIL]"`.
   - Edge: dotted addresses, plus-tag, email inside JSON-stringified attribute. Test corpus committed alongside the module.

### 7.2 Integration (sandbox EKS cluster)

Use a `dev`-type cluster with `vector_enabled = true` explicit override.

1. **Smoke:** apply, confirm Vector pods Ready, OTel rolled with new `otlp` exporter, no `otelcol_exporter_send_failed`.
2. **Audit-drop:** emit log with body `Audit - test-event-<uuid>`. Search Datadog for the UUID. **Pass = no result.** Repeat for `"level":"audit"`.
3. **Email-redact (logs):** emit log `Reach me at e2e-<uuid>@example.com`. **Pass = log appears, body contains `[REDACTED-EMAIL]`, original local-part not present.**
4. **Email-redact (traces):** emit trace with span attribute `user.email = e2e-<uuid>@example.com`. **Pass = attribute is `[REDACTED-EMAIL]` in Datadog APM.**
5. **Negative test:** emit logs containing email-shaped strings inside code blocks. Confirm regex matches and redacts. Documented as known limitation, not a failure.

### 7.3 Failure-mode tests (sandbox)

1. **Vector pod restart:** `kubectl delete pod -n vector -l app=vector`. Brief dip, no Datadog gap > 1m.
2. **Vector full-stop ≤ 1m:** scale to zero, wait, scale back. Queue grows then drains; no permanent drop.
3. **Vector full-stop > 5m:** expect Datadog gap of (outage − 5m). Confirms documented data-loss budget.
4. **Rollback drill:** `vector_enabled = false`, `terraform apply`. OTel reverts to `datadog` exporter, telemetry resumes.
5. **Key rotation:** rotate in Vault. Confirm reloader-driven pod restart, telemetry resumes.

### 7.4 Production rollout sequence

1. **Sandbox / `dev`-type cluster** — explicit opt-in, full integration suite.
2. **One `mgmt` cluster** — soak 3-7 days. Tune resource limits if needed.
3. **Remaining `mgmt` + all `cps` clusters** — incremental.
4. **`saas` clusters last**, region-by-region, lowest-risk regions first.

Each promotion gate: green dashboards (no error-rate spikes, healthy queue
depths) for 48h on the previous tier.

### 7.5 Acceptance criteria for v1

- All `*_enabled` clusters running Vector with no `otelcol_exporter_send_failed` over a 7-day window.
- Datadog ingestion volume post-rollout matches pre-rollout within ±5% (modulo the audit-drop and redaction reductions, expected and quantifiable).
- Sample query in Datadog (`@message:*@*.* AND -@message:"\[REDACTED-EMAIL\]"`) shows zero unredacted email addresses.
- Day-1 audit patterns produce zero hits in Datadog log search post-rollout.
- Datadog monitors from §6.3 deployed and silent for the first 7 days outside of planned rollout-induced restarts.
- Runbook entry added covering: Vector pod debug, queue-full triage, key rotation, rollback to direct-Datadog. (Runbook home TBD — placeholder.)

---

## 8. Out of scope, phased work, open questions

### 8.1 Out of scope for v1 (restated for cross-reference)

- Decommissioning `kubernetes/applications/datadog-processing-rules` — separate effort.
- Per-cluster rule overlay machinery beyond the Terraform variable surface.
- Dynamic / runtime-loaded rules.
- Persistent (`file_storage`) queue on the OTel collector.
- mTLS between OTel and Vector.
- Recursive descent in the email-redact VRL transform.
- Vector Agent-mode (DaemonSet) for node-level collection.
- Datadog monitor wiring (required follow-up, lives in Datadog-as-Code surface).
- Operations runbook (required follow-up, home TBD).

### 8.2 Phased work — post-v1, in rough priority order

1. **Expand redaction patterns:** SSN, credit cards, tokens, JWT, phone numbers. Each ships as opt-in `vector_<category>_redaction_enabled` flag in the module.
2. **Field/attribute name allow-deny lists.** `db.statement`, `http.request.body`, `aws.s3.key` — wholesale strip by attribute name via `vector_drop_attributes`.
3. **Service-level allow/deny.** Drop telemetry from services where redaction can't be exhaustive. `route` transform on `service.name`.
4. **Decommission `datadog-processing-rules`.** After Vector has been clean in prod for one quarter.
5. **Persistent OTel queue.** If the data-loss budget proves insufficient.
6. **Runtime-driven rule sources.** If redaction surface keeps growing such that Terraform applies become friction, move rules behind a `kubernetes` or `http` source.

### 8.3 Open questions to resolve in implementation

1. **Datadog site value origin.** Likely re-use whatever variable feeds the OTel module today; confirm during implementation by reading the OTel internal-module repo.
2. **Vault path for Datadog API key.** Same — find where OTel reads it today, point Vector at the same path. Likely the existing `external_secrets_*` variable cluster.
3. **OTel internal-module's existing secret-mount mechanism.** Affects how cleanly `v1.1.0` removes the mount when `downstream_endpoint` is set.
4. **`datadog_agent.tf` overlap.** Datadog Agent module remains for non-OTel reasons (host metrics, kubelet stats). Vector + OTel does not replace it; the Agent side is untouched.

---

## Appendix A — File touch list

| Repo | File | Action |
|---|---|---|
| `internal-modules/kubernetes/monitoring/otel-collector` | `main.tf`, `variables.tf`, helm values template | Add `downstream_endpoint`, `downstream_protocol` vars; conditionally render `exporters.otlp` block and gate Datadog secret mount. Tag `v1.1.0`. |
| `internal-modules/kubernetes/monitoring/vector` | (new repo) | New module v1.0.0. Namespace, ExternalSecret, helm release, ServiceMonitor, PDB. |
| `terraform/iac-library/aws/compute/eks-cluster-applications` | `vector.tf` | New file — locals, vars, `module "vector"` call, validation. |
| `terraform/iac-library/aws/compute/eks-cluster-applications` | `otel_collector.tf` | Bump module ref to v1.1.0; pass `downstream_endpoint`; add `depends_on = [module.vector]`. |

## Appendix B — Day-1 rule equivalence

Source rules in `kubernetes/applications/datadog-processing-rules/{prod,staging}/rules.json`:

```json
[
  { "type": "exclude_at_match", "name": "exclude_immuta_audit",   "pattern": "Audit\\s-\\s.+" },
  { "type": "exclude_at_match", "name": "exclude_immuta_audit_2", "pattern": "\"level":\"audit\"" }
]
```

Becomes (in Vector VRL):

```vrl
condition = !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"')
```

Plus new email redaction across all signals (no equivalent in source rules):

```vrl
.message = replace(string!(.message), r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}', "[REDACTED-EMAIL]")
```


File created successfully at: /Users/steve.briggs/projects/IAC/docs/superpowers/specs/2026-05-05-vector-redaction-design.md (file state is current in your context — no need to Read it back)

# Vector + OpenTelemetry Redaction Pipeline — Design Spec

**Date:** 2026-05-05
**Author:** steve.briggs@immuta.com (with Claudebob)
**Status:** Draft, pending review

---

## 1. Overview

### Problem

Today the telemetry path on EKS clusters in this org is:

```
apps → otel-lb → otel-collector → Datadog (logs, APM, metrics)
```

Sensitive-data exclusion lives in **Datadog Agent processing rules** under
`kubernetes/applications/datadog-processing-rules`. Both `prod` and `staging` ship
the same `rules.json` containing two `exclude_at_match` entries that drop Immuta
audit log lines before they reach Datadog. `apm.json` is empty.

These rules are tied to the Datadog Agent's processing pipeline and only operate
on logs. They cannot redact span attributes or metric labels, and there is no
straightforward way to add modify-style transforms (e.g., replace email
addresses with a placeholder) within that surface.

### Goal

Insert **Vector** as a new in-cluster aggregator between the OTel collector and
Datadog. Vector replaces today's Datadog-side exclusion rules with VRL-based
filter and remap transforms, applied uniformly across logs, traces, and metrics.

The day-1 rule set replicates today's two audit exclusions and adds email
redaction across all three signals. The architecture leaves room to grow
additional patterns (SSN, credit cards, tokens, attribute drops, service-level
allow/deny) without further structural change.

### Non-goals (v1)

- Replacing the OTel collector.
- Removing the Datadog Agent (it remains for host metrics / kubelet stats).
- Decommissioning `kubernetes/applications/datadog-processing-rules` (separate effort).
- Per-cluster rule overlays beyond the Terraform variable surface.
- Dynamic / runtime-loaded rules.
- mTLS between OTel and Vector.
- Vector Agent-mode / DaemonSet collection.

---

## 2. Architecture

```
                                     [ in-cluster ]
  ┌──────────┐  OTLP  ┌─────────┐  OTLP  ┌────────────────┐  OTLP  ┌──────────────────┐ Datadog
  │   apps   │ ─────▶ │ otel-lb │ ─────▶ │ otel-collector │ ─────▶ │ vector aggregator│ ──────▶  Datadog SaaS
  └──────────┘        └─────────┘        └────────────────┘        │ - OTLP source    │  (logs, APM,
                                                                   │ - VRL transforms │   metrics)
                                                                   │ - Datadog sinks  │
                                                                   └──────────────────┘
```

**Components:**

- **OTel collector** keeps its current shape (`otel-lb` front tier + `otel-collector` back tier). Its terminal exporter switches from `datadog` to `otlp` (gRPC), pointed at the Vector Service.
- **Vector** runs as a Deployment with N replicas (default 3, matching OTel's tier), fronted by a ClusterIP Service. Receives OTLP, applies VRL transforms, ships to Datadog via Vector's native `datadog_logs` / `datadog_traces` / `datadog_metrics` sinks.
- **Datadog API key** moves from the OTel exporter config to Vector. Sourced from Vault via the existing `external-secrets` operator.

**Cluster scope.** Vector is enabled on the same cluster types as the OTel
collector by default: `mgmt`, `cps`, `saas` true; `infra`, `dev` false. A single
`vector_enabled` bool variable overrides the per-cluster-type default, matching
the established pattern in this module.

---

## 3. Terraform layout

Three changes across two repos.

### 3.1 `internal-modules/kubernetes/monitoring/otel-collector` v1.0.7 → v1.1.0

Backwards-compatible bump. New variables:

| Variable | Type | Default | Purpose |
|---|---|---|---|
| `downstream_endpoint` | string | `""` | When non-empty, swaps terminal exporter to OTLP |
| `downstream_protocol` | string | `"otlp/grpc"` | Wire protocol |

When `downstream_endpoint` is non-empty, the rendered helm values gain an
`exporters.otlp` block and `service.pipelines.{logs,traces,metrics}.exporters`
swap from `[datadog]` to `[otlp]`. The Datadog exporter block and its API-key
mount fall away under the same conditional. Empty preserves today's
direct-to-Datadog behavior — clusters that don't enable Vector see no behavioral
change.

### 3.2 `internal-modules/kubernetes/monitoring/vector` v1.0.0 (new module)

Mirrors the shape of the `otel-collector` internal-module: same provider wiring
(`kubernetes`, `helm`), same variable patterns, same approach to image / chart /
replica overrides.

What it creates:

- `kubernetes_namespace.vector` — the `vector` namespace.
- `kubernetes_manifest` for an `ExternalSecret` pulling the Datadog API key from Vault, using the same external-secrets cluster store the rest of the platform uses. Terraform waits for the underlying Secret object before proceeding.
- `helm_release.vector` — wraps the upstream `timberio/vector` chart with values templated from the variables below. Deployment annotated with `reloader.stakater.com/auto: "true"` so the existing reloader operator restarts pods on Datadog-key rotation.

Module variables (initial set):

| Variable | Type | Default | Purpose |
|---|---|---|---|
| `chart_version` | string | required | Vector helm chart version pin |
| `image` | string | `registry.ic-ops.net/cache-docker-hub/timberio/vector` | Image source |
| `image_tag_version` | string | required | Image tag |
| `replicas` | number | `3` | HA matches OTel collector tier |
| `datadog_site` | string | required | e.g. `datadoghq.com` / `us3.datadoghq.com` |
| `datadog_api_key_vault_path` | string | required | Vault path for ExternalSecret |
| `audit_drop_patterns` | list(string) | `["Audit\\s-\\s.+", "\"level\":\"audit\""]` | Day-1 exclude rules |
| `email_redaction_enabled` | bool | `true` | Toggle email redaction |
| `email_redaction_replacement` | string | `"[REDACTED-EMAIL]"` | Replacement marker |
| `additional_config` | any | `null` | Escape-hatch — overwrites `customConfig` wholesale, mirroring `otel_collector_additional_config` semantics |
| `resources_requests` / `resources_limits` | map | see Section 6 | Resource tuning |

### 3.3 `eks-cluster-applications` (this repo)

**New file `vector.tf`** — pure mirror of `otel_collector.tf`'s shape:

```hcl
locals {
  vector_enabled = (
    var.vector_enabled != null
    ? var.vector_enabled
    : lookup(var.vector_enabled_cluster_types, local.cluster_type, true)
  )
  vector_endpoint = "http://vector.vector.svc.cluster.local:4317"
}

module "vector" {
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"
  count  = local.vector_enabled ? 1 : 0

  providers = {
    kubernetes = kubernetes.local-cluster
    helm       = helm.local-cluster
  }

  chart_version               = var.vector_chart_version
  image                       = var.vector_image
  image_tag_version           = var.vector_image_tag
  replicas                    = var.vector_replicas
  datadog_site                = var.vector_datadog_site
  datadog_api_key_vault_path  = var.vector_datadog_api_key_vault_path
  audit_drop_patterns         = var.vector_audit_drop_patterns
  email_redaction_enabled     = var.vector_email_redaction_enabled
  email_redaction_replacement = var.vector_email_redaction_replacement
  additional_config           = var.vector_additional_config

  depends_on = [
    module.cilium,
    module.external_secrets,
  ]
}
```

Plus variable declarations matching the otel-collector module's style.
`vector_enabled_cluster_types` defaults match OTel: `mgmt/cps/saas = true`,
`infra/dev = false`. A `data "external"` validation block enforces
`vector_enabled ⇒ external_secrets_enabled` (the Vault-backed key requires it).

**Edits to `otel_collector.tf`:**

- Bump `?ref=v1.0.7` → `?ref=v1.1.0`.
- Pass `downstream_endpoint = local.vector_enabled ? local.vector_endpoint : ""`.
- Add `depends_on = [module.vector]` so Vector's Service exists before OTel reconciles. (Belt-and-suspenders — OTLP exporter retries handle the runtime race.)

---

## 4. Vector configuration

Vector runs in **aggregator role** (it receives OTLP from another in-cluster
service rather than tailing files).

### 4.1 Pipeline topology

```
                ┌─ logs    ──▶ drop_audit ──▶ redact_emails_logs    ──▶ datadog_logs
otlp_in (4317   │
gRPC + 4318    ─┼─ traces  ──────────────────▶ redact_emails_traces  ──▶ datadog_traces
HTTP)          │
                └─ metrics ──────────────────▶ redact_emails_metrics ──▶ datadog_metrics
```

Three pipelines, one per signal. The `drop_audit` filter runs on logs only (the
day-1 rules are message-text patterns). Email redaction runs on all three.

### 4.2 VRL — audit-drop filter (logs only)

Vector's `filter` transform passes events for which the condition returns
`true`, so the condition is the inverse of what gets dropped:

```toml
[transforms.drop_audit]
type = "filter"
inputs = ["otlp_in.logs"]
condition = '''
  msg = string!(.message ?? "")
  !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"')
'''
```

`audit_drop_patterns` is templated into the helm values from the Terraform
variable. The default ships the two patterns from
`datadog-processing-rules/{prod,staging}/rules.json`.

### 4.3 VRL — email-redact transform (all three signals)

Per-signal `remap` transforms apply the same shared pattern. Log shape (the
simple case):

```toml
[transforms.redact_emails_logs]
type = "remap"
inputs = ["drop_audit"]
source = '''
  email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

  if exists(.message) {
    .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
  }

  # walk top-level attributes for string values containing emails
  if is_object(.attributes) {
    .attributes = map_values(object!(.attributes)) -> |v| {
      if is_string(v) { replace(v, email_re, "[REDACTED-EMAIL]") } else { v }
    }
  }
'''
```

Trace and metric variants apply the same `map_values` walk over `attributes`
and `resource.attributes`, plus span-event attributes for traces. The email
regex and the replacement string are both templated from Terraform variables.

**Limitation called out:** if an email is buried inside a stringified JSON
attribute value (e.g. `http.request.body` holding raw JSON text), the regex
catches it because it operates on string content. If it's deeply nested in a
structured object map, only values reachable by `map_values` are scanned. v1
does **not** recurse into nested object trees — kept simple to avoid runaway
transform CPU on payload-heavy spans. Recursion can be added when justified.

### 4.4 Helm values surface

The internal-module renders to the upstream `timberio/vector` chart:

```yaml
role: Aggregator
replicas: {{ .Values.replicas }}            # 3 by default

customConfig:
  data_dir: /vector-data-dir
  api: { enabled: true, address: 127.0.0.1:8686 }
  sources:
    otlp_in:
      type: opentelemetry
      grpc: { address: 0.0.0.0:4317 }
      http: { address: 0.0.0.0:4318 }
  transforms:
    drop_audit:           { ... }            # see 4.2
    redact_emails_logs:    { ... }
    redact_emails_traces:  { ... }
    redact_emails_metrics: { ... }
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs: [redact_emails_logs]
      default_api_key: "${DATADOG_API_KEY}"
      site: {{ .Values.datadogSite }}
    datadog_traces:    { type: datadog_traces,  inputs: [redact_emails_traces],  default_api_key: "${DATADOG_API_KEY}", site: ... }
    datadog_metrics:   { type: datadog_metrics, inputs: [redact_emails_metrics], default_api_key: "${DATADOG_API_KEY}", site: ... }

service:
  enabled: true
  type: ClusterIP
  ports:
    - { name: otlp-grpc, port: 4317, protocol: TCP }
    - { name: otlp-http, port: 4318, protocol: TCP }

env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key

podAnnotations:
  reloader.stakater.com/auto: "true"

resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits:   { cpu: 2,    memory: 1Gi }
```

`additional_config` (escape-hatch variable) when set overwrites `customConfig`
wholesale, matching `otel_collector_additional_config` semantics.

---

## 5. OTel collector exporter swap

### 5.1 Helm values diff

**Before (`v1.0.7`, today):**

```yaml
exporters:
  datadog:
    api:
      key:  ${DATADOG_API_KEY}
      site: datadoghq.com
service:
  pipelines:
    logs:    { receivers: [otlp], processors: [...], exporters: [datadog] }
    traces:  { receivers: [otlp], processors: [...], exporters: [datadog] }
    metrics: { receivers: [otlp], processors: [...], exporters: [datadog] }
```

**After (`v1.1.0` when `downstream_endpoint != ""`):**

```yaml
exporters:
  otlp:
    endpoint: http://vector.vector.svc.cluster.local:4317
    tls:
      insecure: true                  # in-cluster only; Cilium handles transit
    sending_queue:
      enabled: true
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
service:
  pipelines:
    logs:    { receivers: [otlp], processors: [...], exporters: [otlp] }
    traces:  { receivers: [otlp], processors: [...], exporters: [otlp] }
    metrics: { receivers: [otlp], processors: [...], exporters: [otlp] }
```

The `exporters.datadog` block disappears. Pipeline `processors` lists are
untouched — anything OTel does today (batching, `k8sattributes`, tail-sampling)
stays put.

### 5.2 Datadog API key ownership

- **Before:** OTel collector pods consume the Datadog API key (mount mechanism per the OTel internal-module's existing pattern).
- **After:** Vector module owns the secret. Its ExternalSecret creates `vector-datadog` in the `vector` namespace; Vector pods consume it via `env.DATADOG_API_KEY`. The OTel module's secret mount falls away under the same `downstream_endpoint != ""` conditional that gates the exporter swap.
- **Sequencing:** `module.otel_collector` has `depends_on = [module.vector]`. Single `terraform apply` brings up Vector first, then re-renders OTel.

### 5.3 Failure-mode behavior

If Vector is down or unhealthy:

- OTel's `sending_queue` buffers up to `queue_size: 5000` batches in-memory per pipeline.
- `retry_on_failure` retries with exponential backoff up to 5 minutes total.
- Past that, OTel drops batches and increments `otelcol_exporter_send_failed_*`.
- **No persistent queue in v1.** A Vector outage longer than ~5 minutes loses the buffer tail. Mitigation: Vector replica HA (3 replicas, PDB minAvailable 2). `file_storage` extension is a future hardening item.

### 5.4 TLS posture

OTLP between OTel and Vector runs as plain gRPC, both endpoints in-cluster.
Cilium provides transport encryption. App-layer mTLS is a future hardening
item.

### 5.5 Rollout shape (single cluster)

Single `terraform apply`:

1. `module.vector` provisions — namespace, ExternalSecret, helm release reconcile to ready.
2. Same apply re-renders OTel collector helm release with `v1.1.0` and `downstream_endpoint` set — OTel pods rolling-restart with new exporter config.
3. Vector now receives all traffic; Datadog continues to receive (via Vector's sinks).

**Rollback:** set `vector_enabled = false`. The `local.vector_endpoint` ternary
in `otel_collector.tf` falls back to `""`; OTel re-renders with the original
`datadog` exporter, OTel pods rolling-restart back to direct-to-Datadog. Vector
module destroys cleanly. The Datadog API key path on the OTel side must remain
available across the rollback — this is a pre-condition; the OTel-side secret
reference is not deleted in the same release that introduces Vector.

---

## 6. Operational concerns

### 6.1 HA topology

- **Replicas:** 3 default (`vector_replicas`).
- **PodDisruptionBudget:** `minAvailable: 2`.
- **Topology spread:** `topologySpreadConstraints` on `topology.kubernetes.io/zone`, `maxSkew: 1`. Module gives operators a passthrough variable to override on small clusters where AZ spread is impossible.
- **Anti-affinity:** preferred (not required) on `kubernetes.io/hostname`.

### 6.2 Vector's own telemetry

Vector emits Prometheus-format internal metrics on `:8686/metrics`. Module
ships a `ServiceMonitor`. Key metrics:

| Metric | What it tells you |
|---|---|
| `vector_component_received_events_total` | Inbound rate per source/transform/sink |
| `vector_component_sent_events_total` | Outbound rate at each Datadog sink |
| `vector_component_errors_total` | Failed sends per sink |
| `vector_component_discarded_events_total` | Events dropped by `drop_audit` |
| `vector_buffer_events` | In-memory queue depth toward each Datadog sink |
| `vector_utilization` | Per-component utilization, 0–1 |

Vector's own logs go to stdout and are captured by the existing OTel pipeline.
A Vector outage means we cannot see Vector's logs through Vector. First-line
debug: `kubectl logs -n vector`.

### 6.3 Alerting (sketch — actual monitor wiring out of scope)

| Condition | Severity | Why |
|---|---|---|
| `vector_component_errors_total{sink=~"datadog_.*"}` rate > 0 for 5m | warn | Datadog sink failing |
| `vector_buffer_events / vector_buffer_max_events > 0.8` for 5m | warn | Outbound queue filling |
| `vector_buffer_events / vector_buffer_max_events > 0.95` for 2m | page | About to drop |
| Vector ready replicas < 2 for 5m | page | Below PDB floor |
| `otelcol_exporter_send_failed_*{exporter="otlp"}` rate > 0 for 5m | warn | OTel→Vector path broken |

Datadog monitor wiring lives in this org's Datadog-as-Code surface, not the
Terraform module. **Required follow-up before declaring v1 production-ready.**

### 6.4 Sizing & resource defaults

```yaml
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits:   { cpu: 2,    memory: 1Gi }
```

Conservative starting points. Vector CPU usage scales with VRL transform
complexity (regex on every event isn't free). Real sizing comes from
observed metrics after a soak in `mgmt`/`cps` before rolling to `saas`.

### 6.5 Vault / ExternalSecret dependency chain

```
Vault ──▶ external-secrets operator ──▶ ExternalSecret (vector ns)
                                              │
                                              ▼
                                         Secret: vector-datadog
                                              │
                                              ▼
                                  Vector pods (env DATADOG_API_KEY)
```

The Vector module uses `kubernetes_manifest.wait` on the ExternalSecret's
materialized Secret to block the helm release until secret hydration completes.
Without that wait, the helm release races ahead and Vector pods CrashLoop on
startup. Reconcile order within the module:

1. Namespace
2. ExternalSecret (and wait until the underlying Secret resource is observed)
3. Helm release for Vector

### 6.6 Datadog API key rotation

Rotate the secret in Vault. ExternalSecret refreshes the in-cluster Secret. The
existing **`reloader` operator** picks up the change and restarts Vector pods —
the Vector Deployment is annotated with `reloader.stakater.com/auto: "true"`.
No manual intervention required.

### 6.7 Bootstrap-order edge case (first-time cluster provisioning)

A net-new cluster runs `terraform apply` once for layer-3
(`eks-cluster-applications`). Within that apply:

1. `external_secrets` reconciles first (existing dependency).
2. `vector` reconciles next (new — `depends_on = [module.cilium, module.external_secrets]`).
3. `otel_collector` reconciles last (`depends_on = [module.vector]`).

There is never an OTel collector running with `downstream_endpoint` set before
Vector is ready to receive.

### 6.8 Data-loss budget for v1

| Scenario | Loss |
|---|---|
| Vector single-pod restart | None — other replicas serve via Service load-balancing |
| Vector full-outage < 5 min | None — OTel `sending_queue` + `retry_on_failure` covers |
| Vector full-outage > 5 min | Tail of OTel queue dropped; Datadog gap proportional to outage minus 5 min |
| OTel collector full-outage | Same shape as today — apps' OTLP exporters retry/buffer per their config |
| Datadog API key rotation | None — reloader-driven pod restart |

---

## 7. Testing & verification plan

### 7.1 Pre-deploy (local / CI)

1. **Terraform plan** in a sandbox state to confirm module shape, count toggles cleanly.
2. **Helm template render** of both modules, diff against current rendered manifests in CI.
3. **`vector validate`** on the rendered config — catches VRL syntax errors. CI step in the new `internal-modules/kubernetes/monitoring/vector` repo.
4. **VRL unit tests** via `vector vrl --input ... --program ...`:
   - Audit-drop: `{"message":"Audit - login by user"}` → filtered. `{"message":"INFO request handled"}` → passed.
   - Email redact: `{"message":"contact alice@example.com"}` → `.message == "contact [REDACTED-EMAIL]"`.
   - Edge: dotted addresses, plus-tag, email inside JSON-stringified attribute. Test corpus committed alongside the module.

### 7.2 Integration (sandbox EKS cluster)

Use a `dev`-type cluster with `vector_enabled = true` explicit override.

1. **Smoke:** apply, confirm Vector pods Ready, OTel rolled with new `otlp` exporter, no `otelcol_exporter_send_failed`.
2. **Audit-drop:** emit log with body `Audit - test-event-<uuid>`. Search Datadog for the UUID. **Pass = no result.** Repeat for `"level":"audit"`.
3. **Email-redact (logs):** emit log `Reach me at e2e-<uuid>@example.com`. **Pass = log appears, body contains `[REDACTED-EMAIL]`, original local-part not present.**
4. **Email-redact (traces):** emit trace with span attribute `user.email = e2e-<uuid>@example.com`. **Pass = attribute is `[REDACTED-EMAIL]` in Datadog APM.**
5. **Negative test:** emit logs containing email-shaped strings inside code blocks. Confirm regex matches and redacts. Documented as known limitation, not a failure.

### 7.3 Failure-mode tests (sandbox)

1. **Vector pod restart:** `kubectl delete pod -n vector -l app=vector`. Brief dip, no Datadog gap > 1m.
2. **Vector full-stop ≤ 1m:** scale to zero, wait, scale back. Queue grows then drains; no permanent drop.
3. **Vector full-stop > 5m:** expect Datadog gap of (outage − 5m). Confirms documented data-loss budget.
4. **Rollback drill:** `vector_enabled = false`, `terraform apply`. OTel reverts to `datadog` exporter, telemetry resumes.
5. **Key rotation:** rotate in Vault. Confirm reloader-driven pod restart, telemetry resumes.

### 7.4 Production rollout sequence

1. **Sandbox / `dev`-type cluster** — explicit opt-in, full integration suite.
2. **One `mgmt` cluster** — soak 3-7 days. Tune resource limits if needed.
3. **Remaining `mgmt` + all `cps` clusters** — incremental.
4. **`saas` clusters last**, region-by-region, lowest-risk regions first.

Each promotion gate: green dashboards (no error-rate spikes, healthy queue
depths) for 48h on the previous tier.

### 7.5 Acceptance criteria for v1

- All `*_enabled` clusters running Vector with no `otelcol_exporter_send_failed` over a 7-day window.
- Datadog ingestion volume post-rollout matches pre-rollout within ±5% (modulo the audit-drop and redaction reductions, expected and quantifiable).
- Sample query in Datadog (`@message:*@*.* AND -@message:"\[REDACTED-EMAIL\]"`) shows zero unredacted email addresses.
- Day-1 audit patterns produce zero hits in Datadog log search post-rollout.
- Datadog monitors from §6.3 deployed and silent for the first 7 days outside of planned rollout-induced restarts.
- Runbook entry added covering: Vector pod debug, queue-full triage, key rotation, rollback to direct-Datadog. (Runbook home TBD — placeholder.)

---

## 8. Out of scope, phased work, open questions

### 8.1 Out of scope for v1 (restated for cross-reference)

- Decommissioning `kubernetes/applications/datadog-processing-rules` — separate effort.
- Per-cluster rule overlay machinery beyond the Terraform variable surface.
- Dynamic / runtime-loaded rules.
- Persistent (`file_storage`) queue on the OTel collector.
- mTLS between OTel and Vector.
- Recursive descent in the email-redact VRL transform.
- Vector Agent-mode (DaemonSet) for node-level collection.
- Datadog monitor wiring (required follow-up, lives in Datadog-as-Code surface).
- Operations runbook (required follow-up, home TBD).

### 8.2 Phased work — post-v1, in rough priority order

1. **Expand redaction patterns:** SSN, credit cards, tokens, JWT, phone numbers. Each ships as opt-in `vector_<category>_redaction_enabled` flag in the module.
2. **Field/attribute name allow-deny lists.** `db.statement`, `http.request.body`, `aws.s3.key` — wholesale strip by attribute name via `vector_drop_attributes`.
3. **Service-level allow/deny.** Drop telemetry from services where redaction can't be exhaustive. `route` transform on `service.name`.
4. **Decommission `datadog-processing-rules`.** After Vector has been clean in prod for one quarter.
5. **Persistent OTel queue.** If the data-loss budget proves insufficient.
6. **Runtime-driven rule sources.** If redaction surface keeps growing such that Terraform applies become friction, move rules behind a `kubernetes` or `http` source.

### 8.3 Open questions to resolve in implementation

1. **Datadog site value origin.** Likely re-use whatever variable feeds the OTel module today; confirm during implementation by reading the OTel internal-module repo.
2. **Vault path for Datadog API key.** Same — find where OTel reads it today, point Vector at the same path. Likely the existing `external_secrets_*` variable cluster.
3. **OTel internal-module's existing secret-mount mechanism.** Affects how cleanly `v1.1.0` removes the mount when `downstream_endpoint` is set.
4. **`datadog_agent.tf` overlap.** Datadog Agent module remains for non-OTel reasons (host metrics, kubelet stats). Vector + OTel does not replace it; the Agent side is untouched.

---

## Appendix A — File touch list

| Repo | File | Action |
|---|---|---|
| `internal-modules/kubernetes/monitoring/otel-collector` | `main.tf`, `variables.tf`, helm values template | Add `downstream_endpoint`, `downstream_protocol` vars; conditionally render `exporters.otlp` block and gate Datadog secret mount. Tag `v1.1.0`. |
| `internal-modules/kubernetes/monitoring/vector` | (new repo) | New module v1.0.0. Namespace, ExternalSecret, helm release, ServiceMonitor, PDB. |
| `terraform/iac-library/aws/compute/eks-cluster-applications` | `vector.tf` | New file — locals, vars, `module "vector"` call, validation. |
| `terraform/iac-library/aws/compute/eks-cluster-applications` | `otel_collector.tf` | Bump module ref to v1.1.0; pass `downstream_endpoint`; add `depends_on = [module.vector]`. |

## Appendix B — Day-1 rule equivalence

Source rules in `kubernetes/applications/datadog-processing-rules/{prod,staging}/rules.json`:

```json
[
  { "type": "exclude_at_match", "name": "exclude_immuta_audit",   "pattern": "Audit\\s-\\s.+" },
  { "type": "exclude_at_match", "name": "exclude_immuta_audit_2", "pattern": "\"level":\"audit\"" }
]
```

Becomes (in Vector VRL):

```vrl
condition = !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"')
```

Plus new email redaction across all signals (no equivalent in source rules):

```vrl
.message = replace(string!(.message), r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}', "[REDACTED-EMAIL]")
```


Spec written. Now running the inline self-review pass per the brainstorming skill.


425	| `vector_buffer_events / vector_buffer_max_events > 0.8` for 5m | warn | Outbound queue filling |
426	| `vector_buffer_events / vector_buffer_max_events > 0.95` for 2m | page | About to drop |
427	| Vector ready replicas < 2 for 5m | page | Below PDB floor |
428	| `otelcol_exporter_send_failed_*{exporter="otlp"}` rate > 0 for 5m | warn | OTel→Vector path broken |
429
430	Datadog monitor wiring lives in this org's Datadog-as-Code surface, not the
431	Terraform module. **Required follow-up before declaring v1 production-ready.**
432
433	### 6.4 Sizing & resource defaults
434
435	```yaml
436	resources:
437	  requests: { cpu: 200m, memory: 256Mi }
438	  limits:   { cpu: 2,    memory: 1Gi }
439	```
440
441	Conservative starting points. Vector CPU usage scales with VRL transform
442	complexity (regex on every event isn't free). Real sizing comes from
443	observed metrics after a soak in `mgmt`/`cps` before rolling to `saas`.
444
445	### 6.5 Vault / ExternalSecret dependency chain
446
447	```
448	Vault ──▶ external-secrets operator ──▶ ExternalSecret (vector ns)
449	                                              │
450	                                              ▼
451	                                         Secret: vector-datadog
452	                                              │
453	                                              ▼
454	                                  Vector pods (env DATADOG_API_KEY)
455	```
456
457	The Vector module uses `kubernetes_manifest.wait` on the ExternalSecret's
458	materialized Secret to block the helm release until secret hydration completes.
459	Without that wait, the helm release races ahead and Vector pods CrashLoop on
460	startup. Reconcile order within the module:
461
462	1. Namespace
463	2. ExternalSecret (and wait until the underlying Secret resource is observed)
464	3. Helm release for Vector

| `vector_buffer_events / vector_buffer_max_events > 0.8` for 5m | warn | Outbound queue filling |
| `vector_buffer_events / vector_buffer_max_events > 0.95` for 2m | page | About to drop |
| Vector ready replicas < 2 for 5m | page | Below PDB floor |
| `otelcol_exporter_send_failed_*{exporter="otlp"}` rate > 0 for 5m | warn | OTel→Vector path broken |

Datadog monitor wiring lives in this org's Datadog-as-Code surface, not the
Terraform module. **Required follow-up before declaring v1 production-ready.**

### 6.4 Sizing & resource defaults

```yaml
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits:   { cpu: 2,    memory: 1Gi }
```

Conservative starting points. Vector CPU usage scales with VRL transform
complexity (regex on every event isn't free). Real sizing comes from
observed metrics after a soak in `mgmt`/`cps` before rolling to `saas`.

### 6.5 Vault / ExternalSecret dependency chain

```
Vault ──▶ external-secrets operator ──▶ ExternalSecret (vector ns)
                                              │
                                              ▼
                                         Secret: vector-datadog
                                              │
                                              ▼
                                  Vector pods (env DATADOG_API_KEY)
```

The Vector module uses `kubernetes_manifest.wait` on the ExternalSecret's
materialized Secret to block the helm release until secret hydration completes.
Without that wait, the helm release races ahead and Vector pods CrashLoop on
startup. Reconcile order within the module:

1. Namespace
2. ExternalSecret (and wait until the underlying Secret resource is observed)
3. Helm release for Vector


Found two issues to fix inline:

1. **Typo in Appendix B** — escaped JSON in the `rules.json` quote is wrong (missing a backslash).
2. **ServiceMonitor assumption** — I assumed a Prometheus operator is present without verifying. Should soften and add to open questions.


Edit

{ "type": "exclude_at_match", "name": "exclude_immuta_audit_2", "pattern": "\"level":\"audit\"" }

{ "type": "exclude_at_match", "name": "exclude_immuta_audit_2", "pattern": "\"level\":\"audit\"" }


The file /Users/steve.briggs/projects/IAC/docs/superpowers/specs/2026-05-05-vector-redaction-design.md has been updated successfully. (file state is current in your context — no need to Read it back)

{ "type": "exclude_at_match", "name": "exclude_immuta_audit",   "pattern": "Audit\\s-\\s.+" },

-  { "type": "exclude_at_match", "name": "exclude_immuta_audit_2", "pattern": "\"level":\"audit\"" }

+  { "type": "exclude_at_match", "name": "exclude_immuta_audit_2", "pattern": "\"level\":\"audit\"" }


Vector emits Prometheus-format internal metrics on `:8686/metrics`. Module
ships a `ServiceMonitor`. Key metrics:

Vector emits Prometheus-format internal metrics on `:8686/metrics`. If a
Prometheus operator (kube-prometheus-stack CRDs) is present in the cluster, the
module ships a `ServiceMonitor` for scraping. Otherwise, the metrics are
consumable via the Datadog Agent's OpenMetrics check or the OTel collector
itself (annotation-based scraping). The shipping mechanism is a v1
implementation question — see §8.3. Key metrics:


Vector emits Prometheus-format internal metrics on `:8686/metrics`. Module
ships a `ServiceMonitor`. Key metrics:

Vector emits Prometheus-format internal metrics on `:8686/metrics`. If a
Prometheus operator (kube-prometheus-stack CRDs) is present in the cluster, the
module ships a `ServiceMonitor` for scraping. Otherwise, the metrics are
consumable via the Datadog Agent's OpenMetrics check or the OTel collector
itself (annotation-based scraping). The shipping mechanism is a v1
implementation question — see §8.3. Key metrics:

-Vector emits Prometheus-format internal metrics on `:8686/metrics`. Module

+Vector emits Prometheus-format internal metrics on `:8686/metrics`. If a

+Prometheus operator (kube-prometheus-stack CRDs) is present in the cluster, the

+module ships a `ServiceMonitor` for scraping. Otherwise, the metrics are

+consumable via the Datadog Agent's OpenMetrics check or the OTel collector

+itself (annotation-based scraping). The shipping mechanism is a v1

+implementation question — see §8.3. Key metrics:


4. **`datadog_agent.tf` overlap.** Datadog Agent module remains for non-OTel reasons (host metrics, kubelet stats). Vector + OTel does not replace it; the Agent side is untouched.

4. **`datadog_agent.tf` overlap.** Datadog Agent module remains for non-OTel reasons (host metrics, kubelet stats). Vector + OTel does not replace it; the Agent side is untouched.
5. **Vector self-metrics scrape mechanism.** Confirm whether a Prometheus operator with kube-prometheus-stack CRDs is present in the target cluster types. If yes, ship a `ServiceMonitor`. If no, fall back to Datadog Agent's OpenMetrics check (annotation-driven) or OTel-collector-side scraping. Decide before tagging v1.0.0 of the Vector module.


4. **`datadog_agent.tf` overlap.** Datadog Agent module remains for non-OTel reasons (host metrics, kubelet stats). Vector + OTel does not replace it; the Agent side is untouched.
5. **Vector self-metrics scrape mechanism.** Confirm whether a Prometheus operator with kube-prometheus-stack CRDs is present in the target cluster types. If yes, ship a `ServiceMonitor`. If no, fall back to Datadog Agent's OpenMetrics check (annotation-driven) or OTel-collector-side scraping. Decide before tagging v1.0.0 of the Vector module.

2. **Vault path for Datadog API key.** Same — find where OTel reads it today, point Vector at the same path. Likely the existing `external_secrets_*` variable cluster.

3. **OTel internal-module's existing secret-mount mechanism.** Affects how cleanly `v1.1.0` removes the mount when `downstream_endpoint` is set.

+5. **Vector self-metrics scrape mechanism.** Confirm whether a Prometheus operator with kube-prometheus-stack CRDs is present in the target cluster types. If yes, ship a `ServiceMonitor`. If no, fall back to Datadog Agent's OpenMetrics check (annotation-driven) or OTel-collector-side scraping. Decide before tagging v1.0.0 of the Vector module.


Updated task #5


Bash

git rev-parse --is-inside-work-tree 2>&1 || echo "NOT_A_GIT_REPO"

Check whether IAC umbrella is git-tracked


fatal: not a git repository (or any of the parent directories): .git
NOT_A_GIT_REPO