Write

# Datadog Agent Logs via Vector (OPW Intake) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Route DD-Agent-collected container logs through Vector (using Datadog's OPW intake protocol) so the existing email-redaction VRL transform applies to them before they reach Datadog SaaS.

**Architecture:** Three additive module changes. Vector module adds a `datadog_agent` source on port 8282 and exposes a new output. DD Agent module adds a `logs_intake_url` variable that, when non-null, renders two OPW env vars (`DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED`, `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL`) into the agent's helm values. Consumer wires them via a new `vector_dd_agent_logs_enabled` flag that defaults to follow `local.vector_enabled`.

**Tech Stack:** Terraform (>= 1.3.0, < 2.0), Helm provider (~> 2.0), Kubernetes provider (~> 2.0), Vector (timberio/vector chart), Datadog Agent helm chart, VRL (Vector Remap Language), Cilium NetworkPolicy.

**Spec:** `docs/superpowers/specs/2026-05-06-datadog-agent-logs-via-vector-design.md`

**Sir Charles handles all git operations** — branches, tags (`vector v1.1.0`, `datadog-agent v1.2.0`), and MRs. Each "Commit" step is something Sir Charles will run himself or skip; the plan still calls them out so the work is broken cleanly.

---

## File Structure

### Vector module (`/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector`)

| Path | Action | Responsibility |
|---|---|---|
| `templates/values.yaml.tpl` | Modify | Add 8282 port to service.ports; add `dd_agent_in` source block; expand `drop_audit.inputs` to include the new source |
| `netpol.tf` | Modify | Add port 8282 to existing `fromEntities = ["cluster"]` ingress rule |
| `outputs.tf` | Modify | Add `service_endpoint_dd_agent` output |
| `tests/vrl/redact_emails_dd_agent_logs/inputs.ndjson` | Create | DD-Agent-shape event fixtures (top-level message + ddtags/service/hostname metadata) |
| `tests/vrl/redact_emails_dd_agent_logs/expected.ndjson` | Create | Expected-output fixtures with emails redacted |
| `tests/vrl/redact_emails_dd_agent_logs/program.vrl` | Create | VRL transform (identical logic to `redact_emails_logs/program.vrl`, source-agnostic) |
| `tests/vrl/redact_emails_dd_agent_logs/run.sh` | Create | Shell runner mirroring sibling tests |
| `.gitlab-ci.yml` | Modify | Add the new VRL test to the `vrl-tests` job script list |
| `README.md` | Modify | Add architecture line about DD Agent intake; new output row; new port row |

### DD Agent module (`/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent`)

| Path | Action | Responsibility |
|---|---|---|
| `variables.tf` | Modify | Add `logs_intake_url` variable + URL-prefix validation |
| `values.yaml.tftpl` | Modify | Append two OPW env vars to `datadog.env`, gated by `%{ if logs_intake_url != null ~}` |
| `main.tf` | Modify | Pass `logs_intake_url = var.logs_intake_url` into the templatefile call |
| `tests/render/expected-direct.yaml` | Create | Expected render output for `logs_intake_url = null` (the "no churn" baseline) |
| `tests/render/expected-via-vector.yaml` | Create | Expected render output for `logs_intake_url = "http://vector.vector.svc.cluster.local:8282"` |
| `tests/render/run.sh` | Create | Renders the template under both inputs and diffs against expected outputs |
| `README.md` | Modify | Add `logs_intake_url` to the inputs table; one paragraph describing the OPW intake path |

### Consumer (`/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications`)

| Path | Action | Responsibility |
|---|---|---|
| `datadog_agent.tf` | Modify | Add `vector_dd_agent_logs_enabled` variable + local + plan-time guard; pass `logs_intake_url` into `module.datadog_agent`; bump module ref from `v1.1.2` → `v1.2.0` |
| `vector.tf` | Modify | Bump `module "vector"` source ref from `v1.0.0` → `v1.1.0` (replace the commented-out git ref + remove local-path source) |
| `README.md` | Modify | Add `vector_dd_agent_logs_enabled` row to inputs table; note dual-layer log filtering |

---

## Phase A: Vector module v1.1.0

### Task A1: Add VRL test fixture for DD-Agent-shape logs (failing)

**Files:**
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/inputs.ndjson`
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/expected.ndjson`
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/program.vrl`
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/run.sh`

- [ ] **Step 1: Write the input fixture**

Create `tests/vrl/redact_emails_dd_agent_logs/inputs.ndjson` with five DD-Agent-shape events. The `datadog_agent` source emits events with a top-level `.message` string plus DD-Agent-attached metadata (`ddtags`, `service`, `hostname`, `status`). The VRL transform should redact emails in `.message` and string-typed `.attributes` values, but leave metadata fields untouched.

```json
{"message": "user alice@example.com signed in", "ddtags": "env:prod,service:api", "service": "api", "hostname": "ip-10-0-1-23"}
{"message": "no email here", "ddtags": "env:prod", "service": "worker", "hostname": "ip-10-0-1-24"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "service": "api", "hostname": "ip-10-0-1-25"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "ddtags": "env:staging", "service": "auth", "hostname": "ip-10-0-2-99"}
{"message": "json blob {\"email\":\"x@y.com\"}", "service": "api", "hostname": "ip-10-0-1-50"}
```

- [ ] **Step 2: Write the expected-output fixture**

Create `tests/vrl/redact_emails_dd_agent_logs/expected.ndjson`. Emails in `.message` and `.attributes.user.email` should become `[REDACTED-EMAIL]`. Metadata fields (`ddtags`, `service`, `hostname`) should pass through unchanged. Note: `jq` canonicalizes object key order — write attributes alphabetically.

```json
{"ddtags": "env:prod,service:api", "hostname": "ip-10-0-1-23", "message": "user [REDACTED-EMAIL] signed in", "service": "api"}
{"ddtags": "env:prod", "hostname": "ip-10-0-1-24", "message": "no email here", "service": "worker"}
{"hostname": "ip-10-0-1-25", "message": "multiple [REDACTED-EMAIL] and [REDACTED-EMAIL]", "service": "api"}
{"attributes": {"request.id": "abc-123", "user.email": "[REDACTED-EMAIL]"}, "ddtags": "env:staging", "hostname": "ip-10-0-2-99", "message": "ok", "service": "auth"}
{"hostname": "ip-10-0-1-50", "message": "json blob {\"email\":\"[REDACTED-EMAIL]\"}", "service": "api"}
```

- [ ] **Step 3: Write the VRL program**

Create `tests/vrl/redact_emails_dd_agent_logs/program.vrl`. Identical logic to the existing `redact_emails_logs/program.vrl` — the transform is source-agnostic, but having a dedicated test file makes per-source assertions explicit.

```vrl
email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

if is_string(.message) {
  .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
}

if is_object(.attributes) {
  attrs, _err = object(.attributes)
  if is_object(attrs) {
    .attributes = map_values(attrs) -> |v| {
      if is_string(v) { replace(string!(v), email_re, "[REDACTED-EMAIL]") } else { v }
    }
  }
}
.
```

- [ ] **Step 4: Write the runner script**

Create `tests/vrl/redact_emails_dd_agent_logs/run.sh` (mirrors siblings), then make it executable:

```bash
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
echo "redact_emails_dd_agent_logs VRL OK"
```

Then:
```bash
chmod +x /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/run.sh
```

- [ ] **Step 5: Run the test (expect: passes immediately, since the transform is identical to the existing one)**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

Expected: prints `redact_emails_dd_agent_logs VRL OK`.

If diff fails, check object-key alphabetization in `expected.ndjson` (jq canonicalizes — keys must be sorted).

- [ ] **Step 6: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add tests/vrl/redact_emails_dd_agent_logs/
git commit -m "test: add VRL test for DD-Agent-shape log redaction"
```

---

### Task A2: Wire the new VRL test into CI

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/.gitlab-ci.yml`

- [ ] **Step 1: Add the new test invocation to the `vrl-tests` job**

Open `.gitlab-ci.yml`, locate the `vrl-tests:` job's `script:` list, and append a fifth line:

```yaml
vrl-tests:
  stage: test
  image: registry.ic-ops.net/cache-docker-hub/timberio/vector:latest-debian
  before_script:
    - apt-get update -qq && apt-get install -y -qq bash diffutils jq
  script:
    - ./tests/vrl/drop_audit/run.sh
    - ./tests/vrl/redact_emails_logs/run.sh
    - ./tests/vrl/redact_emails_traces/run.sh
    - ./tests/vrl/redact_emails_metrics/run.sh
    - ./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

- [ ] **Step 2: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add .gitlab-ci.yml
git commit -m "ci: run new DD-Agent-shape VRL test"
```

---

### Task A3: Add the `dd_agent_in` source to the helm values template

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/templates/values.yaml.tpl`

- [ ] **Step 1: Add port 8282 to the service**

Locate the `service.ports:` block (around lines 5-14 of `templates/values.yaml.tpl`). Append a third port entry:

```yaml
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
```

- [ ] **Step 2: Add the `dd_agent_in` source under `customConfig.sources`**

Locate the `customConfig.sources:` block (around lines 59-65). Add a sibling source after `otlp_in`:

```yaml
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
```

`multiple_outputs: true` is required so downstream transforms can reference `dd_agent_in.logs` (the named-output suffix).

- [ ] **Step 3: Update `drop_audit.inputs` to be multi-input**

Locate the `drop_audit:` transform (around lines 67-73). Change its `inputs` list:

```yaml
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
```

- [ ] **Step 4: Validate the example renders**

The template change is gated only by template variables already present in the consumer; no new template inputs added. Run validation:

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/examples/basic
terraform init -backend=false
terraform validate
```

Expected: `Success! The configuration is valid.`

If validate fails, check YAML indentation in the new source block (it must match the 4-space indent of the surrounding map).

- [ ] **Step 5: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add templates/values.yaml.tpl
git commit -m "feat: add datadog_agent OPW source on port 8282"
```

---

### Task A4: Open netpol ingress for port 8282

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/netpol.tf`

- [ ] **Step 1: Add port 8282 to the existing cluster-from ingress rule**

Open `netpol.tf`. Locate the first ingress rule (the one with `fromEntities = ["cluster"]` and the OTLP ports list, around lines 12-21). Add a third port entry:

```hcl
      {
        # OTLP from any cluster-internal sender (primarily otel-collector).
        # Port 8282 is the OPW-compatible Datadog Agent intake (datadog_agent source).
        fromEntities = ["cluster"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
          ]
        }]
      },
```

Do not add 8282 to the `host` (kubelet) ingress rule — kubelet doesn't probe this port.

- [ ] **Step 2: Validate**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
terraform fmt -check -recursive
cd examples/basic
terraform init -backend=false
terraform validate
```

Expected: both commands pass cleanly.

- [ ] **Step 3: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add netpol.tf
git commit -m "feat: open netpol ingress for OPW intake on port 8282"
```

---

### Task A5: Expose the new endpoint as a module output

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/outputs.tf`

- [ ] **Step 1: Add `service_endpoint_dd_agent` output**

Open `outputs.tf` and append:

```hcl
output "service_endpoint_dd_agent" {
  description = "OPW-compatible Datadog Agent intake endpoint (http, cluster-internal). Pass to datadog-agent module's logs_intake_url to redirect agent logs through Vector."
  value       = "http://vector.${var.namespace}.svc.cluster.local:8282"
}
```

- [ ] **Step 2: Validate**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
terraform fmt -check -recursive
cd examples/basic
terraform init -backend=false
terraform validate
```

Expected: pass.

- [ ] **Step 3: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add outputs.tf
git commit -m "feat: expose service_endpoint_dd_agent output"
```

---

### Task A6: Update Vector module README

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/README.md`

- [ ] **Step 1: Update the architecture description**

In `README.md`, replace the existing architecture paragraph (lines 14-32) with:

```markdown
Vector sits between the OTel collector tier and Datadog, and additionally
serves as an OPW-compatible logs intake for the Datadog Agent:

```
apps → otel-lb → otel-collector ──OTLP/gRPC──▶ vector ──▶ Datadog SaaS
                                                  ▲
                                          DD Agent (logs)
```

Vector runs as a Deployment (default 3 replicas) fronted by a ClusterIP
Service. It exposes:

- OTLP/gRPC on port 4317 (from OTel collector)
- OTLP/HTTP on port 4318 (from OTel collector)
- Datadog Agent intake on port 8282 (from DD Agent, OPW protocol)

Three signal pipelines run in parallel:

- **logs** — `drop_audit` filter (multi-input: OTLP logs + DD Agent logs;
  drops lines matching audit patterns) → `redact_emails_logs` remap →
  `datadog_logs` sink.
- **traces** — `redact_emails_traces` remap → `datadog_traces` sink.
- **metrics** — `redact_emails_metrics` remap → `datadog_metrics` sink.

The Datadog API key is sourced from Vault via the existing `external-secrets`
operator. The Deployment is annotated with `reloader.stakater.com/auto: "true"`
so key rotation triggers a rolling restart automatically.
```

- [ ] **Step 2: Add the new output to the Outputs table**

Locate the Outputs table (around lines 59-63). Append:

```markdown
| `service_endpoint_dd_agent` | OPW-compatible Datadog Agent intake endpoint (`http://vector.<namespace>.svc.cluster.local:8282`) |
```

- [ ] **Step 3: Update the example version pin**

Locate the Usage example (around line 73). Change the source ref from `?ref=v1.0.0` to `?ref=v1.1.0`.

- [ ] **Step 4: Add a usage hint for the DD Agent wire-up**

After the existing OTel-collector usage hint (around line 89-93), add:

```markdown
Point the Datadog Agent at Vector's OPW intake by passing the new endpoint to
the `datadog-agent` module (v1.2.0+):

```hcl
logs_intake_url = module.vector.service_endpoint_dd_agent
```
```

- [ ] **Step 5: Update the Testing section**

In the Testing section, add a fifth runner to the VRL test list (around line 109):

```sh
./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

- [ ] **Step 6: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add README.md
git commit -m "docs: document DD Agent OPW intake on port 8282"
```

---

### Task A7: Run all module tests + final validation

- [ ] **Step 1: Run all VRL tests**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
./tests/vrl/drop_audit/run.sh
./tests/vrl/redact_emails_logs/run.sh
./tests/vrl/redact_emails_traces/run.sh
./tests/vrl/redact_emails_metrics/run.sh
./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

Expected: each prints `<test_name> VRL OK` and exits 0.

- [ ] **Step 2: Terraform fmt + validate**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
terraform fmt -check -recursive
cd examples/basic
rm -rf .terraform .terraform.lock.hcl
terraform init -backend=false
terraform validate
```

Expected: fmt prints nothing, validate prints `Success!`.

- [ ] **Step 3: Sir Charles handles tagging**

Sir Charles tags `vector v1.1.0` and pushes (separate from this plan; not Claude's responsibility).

---

## Phase B: DD Agent module v1.2.0

### Task B1: Capture the current ("direct shipping") render as a baseline

**Files:**
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/tests/render/expected-direct.yaml`
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/tests/render/run.sh`

The point of this task is to lock in the current rendered output BEFORE making any changes. After Phase B's template change, rendering with `logs_intake_url = null` must produce byte-identical output.

- [ ] **Step 1: Create the test directory**

```bash
mkdir -p /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/tests/render
```

- [ ] **Step 2: Write a runner script that uses Terraform's templatefile() to render the template under fixed inputs**

Create `tests/render/run.sh`:

```bash
#!/usr/bin/env bash
# Renders values.yaml.tftpl under two input scenarios and diffs against the
# locked expected outputs. Catches accidental drift in the helm values output.
set -euo pipefail

DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
MODULE_DIR="$(cd "$DIR/../.." && pwd)"
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT

render() {
  local intake_url="$1"
  local out="$2"

  # logs_intake_url is rendered as a literal "null" sentinel via jsonencode
  # when null; the template uses `!= null` to gate, so we pass the raw value
  # via terraform's tftpl_render harness.

  cat > "$TMP/main.tf" <<EOF
output "rendered" {
  value = templatefile("\${path.module}/values.yaml.tftpl", {
    cluster_name                    = "test-cluster"
    env                             = "nonprod"
    global_segment                  = "test"
    image_tag                       = "7.55.0"
    watch_namespace                 = "datadog"
    tags                            = ["env:test"]
    agents_priority_class_name      = "cloud-ops-system"
    apm_enabled                     = true
    container_exclude               = ""
    container_exclude_logs          = ""
    enable_mrds_monitoring          = false
    enable_nowrds_monitoring        = false
    enable_now_obfusicator_options  = false
    now_obfusicator_options         = { replace_digits = "true" }
    datadog_api_key_secret_name     = "datadog-api-key"
    cluster_agent_token_secret_name = "datadog-clusteragent-token"
    log_level                       = "ERROR"
    logs_intake_url                 = ${intake_url}
  })
}
EOF
  ln -sf "$MODULE_DIR/values.yaml.tftpl" "$TMP/values.yaml.tftpl"
  (cd "$TMP" && terraform init -backend=false -input=false >/dev/null && terraform apply -auto-approve -input=false >/dev/null)
  (cd "$TMP" && terraform output -raw rendered) > "$out"
  rm -rf "$TMP"/.terraform "$TMP"/terraform.tfstate*
}

render "null"                                                "$DIR/.actual-direct.yaml"
render '"http://vector.vector.svc.cluster.local:8282"'       "$DIR/.actual-via-vector.yaml"

diff -u "$DIR/expected-direct.yaml"     "$DIR/.actual-direct.yaml"
diff -u "$DIR/expected-via-vector.yaml" "$DIR/.actual-via-vector.yaml"

echo "datadog-agent values render OK"
```

Then make it executable:
```bash
chmod +x /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/tests/render/run.sh
```

- [ ] **Step 3: Generate the baseline `expected-direct.yaml` from the CURRENT (pre-change) template**

Before the template change lands, capture today's output as the locked baseline:

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
TMP="$(mktemp -d)"
ln -sf "$(pwd)/values.yaml.tftpl" "$TMP/values.yaml.tftpl"
cat > "$TMP/main.tf" <<'EOF'
output "rendered" {
  value = templatefile("${path.module}/values.yaml.tftpl", {
    cluster_name                    = "test-cluster"
    env                             = "nonprod"
    global_segment                  = "test"
    image_tag                       = "7.55.0"
    watch_namespace                 = "datadog"
    tags                            = ["env:test"]
    agents_priority_class_name      = "cloud-ops-system"
    apm_enabled                     = true
    container_exclude               = ""
    container_exclude_logs          = ""
    enable_mrds_monitoring          = false
    enable_nowrds_monitoring        = false
    enable_now_obfusicator_options  = false
    now_obfusicator_options         = { replace_digits = "true" }
    datadog_api_key_secret_name     = "datadog-api-key"
    cluster_agent_token_secret_name = "datadog-clusteragent-token"
    log_level                       = "ERROR"
  })
}
EOF
(cd "$TMP" && terraform init -backend=false -input=false >/dev/null && terraform apply -auto-approve -input=false >/dev/null)
(cd "$TMP" && terraform output -raw rendered) > tests/render/expected-direct.yaml
rm -rf "$TMP"
```

This produces `tests/render/expected-direct.yaml` containing the current rendered helm values (without the new template variable).

- [ ] **Step 4: Verify the baseline file is non-empty and looks right**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
wc -l tests/render/expected-direct.yaml
head -5 tests/render/expected-direct.yaml
```

Expected: ~230 lines (the existing template is 234 lines), starts with `targetSystem: "linux"`.

- [ ] **Step 5: Commit the baseline + runner BEFORE the template change**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
git add tests/render/run.sh tests/render/expected-direct.yaml
git commit -m "test: lock current values render as baseline before OPW change"
```

This commit is critical — it establishes the "before" state so the next task can verify the change is byte-identical when the new variable is null.

---

### Task B2: Add the `logs_intake_url` variable

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/variables.tf`

- [ ] **Step 1: Append the new variable to `variables.tf`**

Open `variables.tf` and append:

```hcl
variable "logs_intake_url" {
  description = "OPW-compatible logs intake URL. When set, the agent ships logs to this endpoint via DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL. For cluster-internal Vector use module.vector.service_endpoint_dd_agent. When null (default), logs ship direct to Datadog SaaS."
  type        = string
  default     = null

  validation {
    condition     = var.logs_intake_url == null || can(regex("^https?://", var.logs_intake_url))
    error_message = "logs_intake_url must be a URL starting with http:// or https://, or null."
  }
}
```

- [ ] **Step 2: Run terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
terraform fmt variables.tf
```

Expected: no output (or it auto-formats and exits 0).

- [ ] **Step 3: Commit**

```bash
git add variables.tf
git commit -m "feat: add logs_intake_url variable for OPW redirection"
```

---

### Task B3: Wire the new variable into the templatefile call

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/main.tf`

- [ ] **Step 1: Add `logs_intake_url` to the templatefile arguments**

Open `main.tf`. Locate the `templatefile(...)` call inside `helm_release.this.values` (around lines 242-265). Add `logs_intake_url = var.logs_intake_url` to the arguments map. After the change, the templatefile block should look like:

```hcl
  values = [
    templatefile("${path.module}/values.yaml.tftpl", {
      cluster_name   = var.cluster_name
      env            = var.env == "staging" ? "nonprod" : var.env
      global_segment = var.global_segment
      image_tag      = var.image_tag
      watch_namespace          = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
      tags                     = concat(["global_segment:${var.global_segment}"], var.datadog_tags)
      agents_priority_class_name = var.agents_priority_class_name

      apm_enabled = var.apm_enabled

      container_exclude = var.container_exclude
      container_exclude_logs = var.container_exclude_logs

      enable_mrds_monitoring   = var.enable_mrds_monitoring
      enable_nowrds_monitoring = var.enable_nowrds_monitoring
      enable_now_obfusicator_options = var.enable_now_obfusicator_options
      now_obfusicator_options = var.now_obfusicator_options

      datadog_api_key_secret_name = var.datadog_api_key_secret_name
      cluster_agent_token_secret_name = var.cluster_agent_token_secret_name
      log_level = var.log_level

      logs_intake_url = var.logs_intake_url
    })
  ]
```

- [ ] **Step 2: Validate (note: this will fail at apply time without a real provider, but `validate` should pass)**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
terraform fmt -check
terraform init -backend=false -input=false 2>&1 | head -20 || true
terraform validate
```

Expected: `terraform validate` exits 0 with `Success!`. The `init` may complain about missing remote module sources — that's OK as long as `validate` passes.

- [ ] **Step 3: Commit**

```bash
git add main.tf
git commit -m "feat: pass logs_intake_url into helm values render"
```

---

### Task B4: Add the OPW env-var block to the values template

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/values.yaml.tftpl`

- [ ] **Step 1: Locate the `datadog.env` list**

In `values.yaml.tftpl`, find the `datadog.env:` block (lines 57-81). It currently ends with the `DD_APM_PROBABILISTIC_SAMPLER_SAMPLING_PERCENTAGE` entry on line 81.

- [ ] **Step 2: Append the gated OPW env vars**

After line 81 (the `value: 15` line for `DD_APM_PROBABILISTIC_SAMPLER_SAMPLING_PERCENTAGE`) and before `clusterAgent:` on line 82, insert:

```yaml
%{ if logs_intake_url != null ~}
    - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED
      value: "true"
    - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL
      value: "${logs_intake_url}"
%{ endif ~}
```

The 4-space indentation matches the surrounding `datadog.env` list items.

- [ ] **Step 3: Generate the "via-vector" expected render**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
TMP="$(mktemp -d)"
ln -sf "$(pwd)/values.yaml.tftpl" "$TMP/values.yaml.tftpl"
cat > "$TMP/main.tf" <<'EOF'
output "rendered" {
  value = templatefile("${path.module}/values.yaml.tftpl", {
    cluster_name                    = "test-cluster"
    env                             = "nonprod"
    global_segment                  = "test"
    image_tag                       = "7.55.0"
    watch_namespace                 = "datadog"
    tags                            = ["env:test"]
    agents_priority_class_name      = "cloud-ops-system"
    apm_enabled                     = true
    container_exclude               = ""
    container_exclude_logs          = ""
    enable_mrds_monitoring          = false
    enable_nowrds_monitoring        = false
    enable_now_obfusicator_options  = false
    now_obfusicator_options         = { replace_digits = "true" }
    datadog_api_key_secret_name     = "datadog-api-key"
    cluster_agent_token_secret_name = "datadog-clusteragent-token"
    log_level                       = "ERROR"
    logs_intake_url                 = "http://vector.vector.svc.cluster.local:8282"
  })
}
EOF
(cd "$TMP" && terraform init -backend=false -input=false >/dev/null && terraform apply -auto-approve -input=false >/dev/null)
(cd "$TMP" && terraform output -raw rendered) > tests/render/expected-via-vector.yaml
rm -rf "$TMP"
```

- [ ] **Step 4: Verify the via-vector render contains the OPW env vars**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
grep -A1 "DD_OBSERVABILITY_PIPELINES_WORKER" tests/render/expected-via-vector.yaml
```

Expected output:
```
    - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED
      value: "true"
    - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL
      value: "http://vector.vector.svc.cluster.local:8282"
```

- [ ] **Step 5: Run the render harness — both scenarios must pass**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
./tests/render/run.sh
```

Expected: prints `datadog-agent values render OK`.

The "direct" diff must be **empty** (proving byte-identical when null). The "via-vector" diff must be empty (proving expected env vars present).

If the "direct" diff shows ANY change, your `%{ if ~}` directive isn't strip-newlining correctly — the template should produce zero output when `logs_intake_url == null`. Common cause: missing `~` on either delimiter.

- [ ] **Step 6: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
git add values.yaml.tftpl tests/render/expected-via-vector.yaml
git commit -m "feat: render OPW env vars when logs_intake_url is set"
```

---

### Task B5: Update DD Agent README

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/README.md`

- [ ] **Step 1: Add the `logs_intake_url` row to the Inputs table**

In `README.md`, locate the Inputs table (around lines 33-58). Find the row for `log_level` (alphabetical position) and insert immediately after it:

```markdown
| <a name="input_logs_intake_url"></a> [logs\_intake\_url](#input\_logs\_intake\_url) | OPW-compatible logs intake URL. When set, the agent ships logs to this endpoint via DD\_OBSERVABILITY\_PIPELINES\_WORKER\_LOGS\_URL. For cluster-internal Vector use module.vector.service\_endpoint\_dd\_agent. When null (default), logs ship direct to Datadog SaaS. | `string` | `null` | no |
```

- [ ] **Step 2: Add an "OPW Logs Intake" section after the Outputs table**

Append at the end of `README.md`:

```markdown

## OPW Logs Intake (optional)

By default, the agent ships logs directly to Datadog SaaS over HTTPS. To
redirect logs through a Datadog Observability Pipelines Worker (OPW), or
through a Vector deployment functioning as one, set `logs_intake_url`:

```hcl
module "datadog_agent" {
  # ...
  logs_intake_url = module.vector.service_endpoint_dd_agent
  # e.g. "http://vector.vector.svc.cluster.local:8282"
}
```

This sets two env vars on the agent containers:

- `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED=true`
- `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL=<logs_intake_url>`

Agent-side processing rules (`DD_LOGS_CONFIG_PROCESSING_RULES`) continue to
execute before transmission, regardless of destination. Metrics and APM
(traces) are unaffected and continue shipping direct to Datadog.

When `logs_intake_url` is `null`, the rendered helm values are byte-identical
to the v1.1.x line — there is no behavior change for clusters that do not opt in.

Reference: <https://docs.datadoghq.com/observability_pipelines/sources/datadog_agent/>
```

- [ ] **Step 3: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
git add README.md
git commit -m "docs: document logs_intake_url for OPW redirection"
```

---

### Task B6: Final module validation

- [ ] **Step 1: Run the full render harness**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
./tests/render/run.sh
```

Expected: `datadog-agent values render OK`.

- [ ] **Step 2: terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
terraform fmt -check -recursive
```

Expected: no output, exit 0.

- [ ] **Step 3: terraform validate**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
terraform init -backend=false -input=false 2>&1 | tail -5 || true
terraform validate
```

Expected: `Success!`

- [ ] **Step 4: Sir Charles handles tagging**

Sir Charles tags `datadog-agent v1.2.0` and pushes (separate from this plan).

---

## Phase C: Consumer (eks-cluster-applications) wire-up

### Task C1: Add the `vector_dd_agent_logs_enabled` flag

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/datadog_agent.tf`

- [ ] **Step 1: Add the variable, local, and plan-time guard**

Open `datadog_agent.tf`. At the bottom of the file (after the existing `variable "datadog_agent_enabled"` block, around line 117), append:

```hcl
# ---------------------------------------------------------------------------------------------------------------------
# DD AGENT → VECTOR LOGS ROUTING (OPW)
# ---------------------------------------------------------------------------------------------------------------------
locals {
  vector_dd_agent_logs_enabled = (
    var.vector_dd_agent_logs_enabled != null
    ? var.vector_dd_agent_logs_enabled
    : local.vector_enabled
  )
}

variable "vector_dd_agent_logs_enabled" {
  description = "Override for routing DD Agent logs through Vector. When null (default), inherits from local.vector_enabled (i.e., on wherever Vector is on). Set to false to keep DD Agent shipping logs directly to DD SaaS even when Vector is deployed."
  type        = bool
  default     = null
}

# Plan-time guard: vector_dd_agent_logs_enabled cannot be true if vector itself is off.
# Mirrors the existing data "external" "vector_validate" pattern in vector.tf.
data "external" "vector_dd_agent_logs_validate" {
  count = local.vector_dd_agent_logs_enabled && !local.vector_enabled ? 1 : 0

  program = ["Error: vector_dd_agent_logs_enabled requires vector_enabled"]
}
```

- [ ] **Step 2: Run terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform fmt datadog_agent.tf
```

- [ ] **Step 3: Commit**

```bash
git add datadog_agent.tf
git commit -m "feat: add vector_dd_agent_logs_enabled flag with vector_enabled inheritance"
```

---

### Task C2: Bump the `module.datadog_agent` ref and pass `logs_intake_url`

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/datadog_agent.tf`

- [ ] **Step 1: Bump the module source ref**

In `datadog_agent.tf`, line 14, change:

```hcl
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/datadog-agent?ref=v1.1.2"
```

to:

```hcl
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/datadog-agent?ref=v1.2.0"
```

- [ ] **Step 2: Add the `logs_intake_url` argument to the module call**

In the same `module "datadog_agent"` block (around lines 13-34), add `logs_intake_url` as the last argument before the closing `}`:

```hcl
module "datadog_agent" {
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/datadog-agent?ref=v1.2.0"
  count  = local.datadog_agent_enabled ? 1 : 0

  providers = {
    kubernetes = kubernetes.local-cluster
    helm       = helm.local-cluster
  }

  chart_version                  = var.datadog_agent_version
  image_tag                      = var.datadog_agent_image_tag
  create_namespace               = var.datadog_create_namespace
  cluster_name                   = var.cluster_name
  global_segment                 = var.global_segment
  datadog_tags                   = var.datadog_tags
  enable_nowrds_monitoring       = var.datadog_enable_nowrds_monitoring
  enable_now_obfusicator_options = var.datadog_enable_now_obfusicator_options
  enable_mrds_monitoring         = var.datadog_enable_mrds_monitoring
  now_obfusicator_options        = var.datadog_now_obfusicator_options
  tags                           = var.tags
  env                            = var.env

  logs_intake_url = local.vector_dd_agent_logs_enabled ? module.vector[0].service_endpoint_dd_agent : null
}
```

The ternary handles the `count = 0` case correctly: when `local.vector_dd_agent_logs_enabled = false`, the right-hand side evaluates to `null` without dereferencing `module.vector[0]`.

- [ ] **Step 3: Run terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform fmt datadog_agent.tf
```

- [ ] **Step 4: Commit**

```bash
git add datadog_agent.tf
git commit -m "feat: route DD Agent logs through Vector when enabled"
```

---

### Task C3: Bump `module.vector` ref to v1.1.0

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/vector.tf`

- [ ] **Step 1: Replace the local-path source with the tagged git ref**

In `vector.tf`, lines 14-15 currently read:

```hcl
  #source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"
  source = "/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector"
```

Change to:

```hcl
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.1.0"
```

(Removes the local-path source and the commented-out v1.0.0 line; replaces with the v1.1.0 tagged git ref. The local-path source was only present during development, per the prior session note.)

- [ ] **Step 2: Run terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform fmt vector.tf
```

- [ ] **Step 3: Commit**

```bash
git add vector.tf
git commit -m "chore: pin vector module to v1.1.0 (OPW intake)"
```

---

### Task C4: Update consumer README

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/README.md`

- [ ] **Step 1: Read the existing inputs table**

```bash
grep -n "vector_email_redaction_enabled\|vector_enabled\|input_vector" /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/README.md | head -30
```

This locates the Vector-related inputs table rows so the new entry slots in alphabetically.

- [ ] **Step 2: Add the new input row**

In `README.md`, in the inputs table where Vector variables are listed, insert (alphabetically — between `vector_chart_version` and `vector_email_redaction_enabled` or wherever it sorts):

```markdown
| <a name="input_vector_dd_agent_logs_enabled"></a> [vector\_dd\_agent\_logs\_enabled](#input\_vector\_dd\_agent\_logs\_enabled) | Override for routing DD Agent logs through Vector. When null (default), inherits from local.vector\_enabled (i.e., on wherever Vector is on). Set to false to keep DD Agent shipping logs directly to DD SaaS even when Vector is deployed. | `bool` | `null` | no |
```

- [ ] **Step 3: Add a paragraph about dual-layer log filtering**

Find the existing Vector-related section in the README (added in the prior session per the spec) and append a new paragraph:

```markdown

### DD Agent → Vector log routing (OPW)

When `vector_dd_agent_logs_enabled` is true (defaults to `local.vector_enabled`),
the Datadog Agent ships container logs to Vector via the OPW HTTP intake on
`vector.vector.svc.cluster.local:8282`. Logs traverse two filtering layers
before reaching Datadog SaaS:

1. **Agent-side** — `DD_LOGS_CONFIG_PROCESSING_RULES` (see
   `datadog-agent/processing-rules/{env}/rules.json`) drops audit lines before
   transmission. Authoritative for audit drops.
2. **Vector** — `drop_audit` regex filter (defense-in-depth, same patterns) +
   `redact_emails_logs` substitutes email addresses with `[REDACTED-EMAIL]`.

Metrics and APM (traces) are unaffected and continue their existing OTel-based
path through Vector. To roll back logs to direct DD shipping for a single
cluster while keeping Vector traces, set `vector_dd_agent_logs_enabled = false`
on that cluster — Vector continues serving OTel traffic, agent re-rolls back to
direct logs.
```

- [ ] **Step 4: Commit**

```bash
git add README.md
git commit -m "docs: document vector_dd_agent_logs_enabled and dual-layer log filtering"
```

---

### Task C5: Final consumer validation

- [ ] **Step 1: terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform fmt -check -recursive
```

Expected: no output, exit 0.

- [ ] **Step 2: terraform validate (best-effort, may need cluster auth)**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform init -backend=false -input=false 2>&1 | tail -10 || true
terraform validate
```

Expected: `Success!`. If `init` cannot fetch the new module versions (because Sir Charles hasn't pushed the tags yet), `validate` will fail with module-not-found; that's expected pre-tag and Sir Charles will re-run after tagging.

- [ ] **Step 3: Note for Sir Charles**

Once `vector v1.1.0` and `datadog-agent v1.2.0` are tagged + pushed, re-run:
```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
rm -rf .terraform .terraform.lock.hcl
terraform init -backend=false
terraform validate
```

To confirm clean resolution against the real tags.

---

## Phase D: Deploy verification (post-rollout, ad-hoc)

These steps run against a real cluster after the three repos are tagged, MRs merged, and `terraform apply` has rolled the change out. They are **not** committable steps — they are runbook entries.

### Task D1: Verify DD Agent ingests via OPW path (smoke)

- [ ] **Step 1: Confirm the new env vars are live on agent pods**

```bash
kubectl exec -n datadog ds/immuta-dd -c agent -- env | grep DD_OBSERVABILITY_PIPELINES_WORKER
```

Expected:
```
DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED=true
DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL=http://vector.vector.svc.cluster.local:8282
```

If empty: the helm release didn't roll, or `vector_dd_agent_logs_enabled` is false. Check `terraform plan` output and `kubectl describe ds/immuta-dd`.

- [ ] **Step 2: Confirm Vector is receiving DD Agent traffic**

```bash
kubectl exec -n vector deploy/vector -- curl -s http://localhost:8686/metrics \
  | grep 'vector_component_received_events_total{component_id="dd_agent_in"'
```

Expected: a non-zero counter that increases over time.

If zero: check Vector pod logs for source-startup errors (`kubectl logs -n vector deploy/vector | grep dd_agent_in`).

### Task D2: End-to-end redaction verification

- [ ] **Step 1: Emit an email-tagged log line from a test pod**

```bash
UUID="dd-redact-$(uuidgen)"
kubectl run "${UUID}" --image=busybox --restart=Never --rm -it \
  -- sh -c "echo '${UUID}: contact alice@example.com'; sleep 5"
```

- [ ] **Step 2: Search Datadog Logs UI**

Query: `${UUID}` (the test marker). Within ~30s the matching log should appear.

Expected: log message reads `${UUID}: contact [REDACTED-EMAIL]` (not the original `alice@example.com`).

If unredacted: check `vector_component_errors_total{component_id="redact_emails_logs"}` and the email regex against the actual log shape.

- [ ] **Step 3: Verify metadata preservation**

In the same DD Logs UI result, verify these facets are populated:
- `service` (should match the test pod's service tag, often the pod name)
- `host` (the EC2 instance / node)
- `kubernetes.namespace_name` (= `default`)
- `kubernetes.pod_name` (= `${UUID}`)

If missing: the `datadog_agent` source's `multiple_outputs` may be dropping fields. Check Vector pod logs.

### Task D3: Audit-drop verification

- [ ] **Step 1: Emit an audit-shape log line**

```bash
UUID2="dd-audit-$(uuidgen)"
kubectl run "${UUID2}" --image=busybox --restart=Never --rm -it \
  -- sh -c "echo 'Audit - test ${UUID2}'; sleep 5"
```

- [ ] **Step 2: Confirm zero results in Datadog**

Query: `${UUID2}`. Expected: zero results (dropped at the agent before transmission to Vector).

- [ ] **Step 3: Confirm Vector did NOT receive it**

```bash
kubectl exec -n vector deploy/vector -- curl -s http://localhost:8686/metrics \
  | grep 'vector_component_received_events_total{component_id="dd_agent_in"'
```

Note the value. Wait 30s, re-run. The counter should NOT have incremented for this audit line (since the agent dropped it before transmission).

If the counter ticked up: agent-side processing rules aren't matching the new audit pattern. Check `kubectl get cm -n datadog rules-config -o yaml`.

### Task D4: Rollback drill

- [ ] **Step 1: Flip the flag false on a non-prod cluster**

In the cluster's instance directory, set `vector_dd_agent_logs_enabled = false`. Run `terraform apply`.

- [ ] **Step 2: Confirm env vars removed from agent**

```bash
kubectl exec -n datadog ds/immuta-dd -c agent -- env | grep DD_OBSERVABILITY_PIPELINES_WORKER || echo "no OPW env vars"
```

Expected: prints `no OPW env vars`.

- [ ] **Step 3: Re-run the redaction smoke test from Task D2**

Expected: log appears in DD UI with the email address NOT redacted (proving the agent reverted to direct shipping).

- [ ] **Step 4: Flip the flag back on**

`vector_dd_agent_logs_enabled = null` (or `true`), `terraform apply`. Re-run Task D2 step 2 — redaction should be active again.

---

## Self-Review

**Spec coverage check** (each numbered subsection of the spec → mapped task):

| Spec section | Implementing task(s) |
|---|---|
| §3.1 Pipeline before/after | Phase A + B (mechanics), Phase C (gating) |
| §3.2 Module split | Phase A (Vector), B (DD Agent), C (consumer) |
| §4.1 Vector module deltas — `service.ports` | A3 step 1 |
| §4.1 Vector module deltas — `dd_agent_in` source | A3 step 2 |
| §4.1 Vector module deltas — `drop_audit.inputs` | A3 step 3 |
| §4.1 netpol port 8282 | A4 |
| §4.1 outputs | A5 |
| §4.1 README | A6 |
| §4.2 DD Agent module — `logs_intake_url` variable | B2 |
| §4.2 DD Agent module — values template | B4 |
| §4.2 DD Agent module — main.tf wiring | B3 |
| §4.2 DD Agent module — README | B5 |
| §4.3 Consumer — `vector_dd_agent_logs_enabled` flag | C1 |
| §4.3 Consumer — module ref bumps | C2, C3 |
| §4.3 Consumer — wire `logs_intake_url` | C2 |
| §4.3 Consumer — README | C4 |
| §6.1 Module input validation — `logs_intake_url` regex | B2 |
| §6.1 Module input validation — consumer guard | C1 |
| §7.1 Unit tests — VRL | A1 |
| §7.1 Unit tests — DD Agent template render | B1, B4 |
| §7.1 Unit tests — terraform fmt/validate | A7, B6, C5 |
| §7.2 Integration verification | D1, D2, D3 |
| §7.3 Cross-cutting — metadata preservation | D2 step 3 |
| §7.3 Cross-cutting — JSON-shape redaction | (covered by A1 fixture line 5) |
| §7.4 Soak | (Sir Charles's operational responsibility, not implementation) |

No gaps.

**Type/name consistency check:**
- Variable name `logs_intake_url` consistent across DD Agent module + consumer wire-up.
- Variable name `vector_dd_agent_logs_enabled` consistent across consumer var + local + guard.
- Output name `service_endpoint_dd_agent` consistent across Vector module + consumer wire-up.
- Port `8282` consistent across Vector service.ports + dd_agent_in.address + netpol + output URL + DD Agent docs/README.
- Module versions: `vector v1.1.0` and `datadog-agent v1.2.0` consistent across spec + plan + consumer ref bumps.


File created successfully at: /Users/steve.briggs/projects/IAC/docs/superpowers/plans/2026-05-06-datadog-agent-logs-via-vector.md (file state is current in your context — no need to Read it back)

# Datadog Agent Logs via Vector (OPW Intake) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Route DD-Agent-collected container logs through Vector (using Datadog's OPW intake protocol) so the existing email-redaction VRL transform applies to them before they reach Datadog SaaS.

**Architecture:** Three additive module changes. Vector module adds a `datadog_agent` source on port 8282 and exposes a new output. DD Agent module adds a `logs_intake_url` variable that, when non-null, renders two OPW env vars (`DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED`, `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL`) into the agent's helm values. Consumer wires them via a new `vector_dd_agent_logs_enabled` flag that defaults to follow `local.vector_enabled`.

**Tech Stack:** Terraform (>= 1.3.0, < 2.0), Helm provider (~> 2.0), Kubernetes provider (~> 2.0), Vector (timberio/vector chart), Datadog Agent helm chart, VRL (Vector Remap Language), Cilium NetworkPolicy.

**Spec:** `docs/superpowers/specs/2026-05-06-datadog-agent-logs-via-vector-design.md`

**Sir Charles handles all git operations** — branches, tags (`vector v1.1.0`, `datadog-agent v1.2.0`), and MRs. Each "Commit" step is something Sir Charles will run himself or skip; the plan still calls them out so the work is broken cleanly.

---

## File Structure

### Vector module (`/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector`)

| Path | Action | Responsibility |
|---|---|---|
| `templates/values.yaml.tpl` | Modify | Add 8282 port to service.ports; add `dd_agent_in` source block; expand `drop_audit.inputs` to include the new source |
| `netpol.tf` | Modify | Add port 8282 to existing `fromEntities = ["cluster"]` ingress rule |
| `outputs.tf` | Modify | Add `service_endpoint_dd_agent` output |
| `tests/vrl/redact_emails_dd_agent_logs/inputs.ndjson` | Create | DD-Agent-shape event fixtures (top-level message + ddtags/service/hostname metadata) |
| `tests/vrl/redact_emails_dd_agent_logs/expected.ndjson` | Create | Expected-output fixtures with emails redacted |
| `tests/vrl/redact_emails_dd_agent_logs/program.vrl` | Create | VRL transform (identical logic to `redact_emails_logs/program.vrl`, source-agnostic) |
| `tests/vrl/redact_emails_dd_agent_logs/run.sh` | Create | Shell runner mirroring sibling tests |
| `.gitlab-ci.yml` | Modify | Add the new VRL test to the `vrl-tests` job script list |
| `README.md` | Modify | Add architecture line about DD Agent intake; new output row; new port row |

### DD Agent module (`/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent`)

| Path | Action | Responsibility |
|---|---|---|
| `variables.tf` | Modify | Add `logs_intake_url` variable + URL-prefix validation |
| `values.yaml.tftpl` | Modify | Append two OPW env vars to `datadog.env`, gated by `%{ if logs_intake_url != null ~}` |
| `main.tf` | Modify | Pass `logs_intake_url = var.logs_intake_url` into the templatefile call |
| `tests/render/expected-direct.yaml` | Create | Expected render output for `logs_intake_url = null` (the "no churn" baseline) |
| `tests/render/expected-via-vector.yaml` | Create | Expected render output for `logs_intake_url = "http://vector.vector.svc.cluster.local:8282"` |
| `tests/render/run.sh` | Create | Renders the template under both inputs and diffs against expected outputs |
| `README.md` | Modify | Add `logs_intake_url` to the inputs table; one paragraph describing the OPW intake path |

### Consumer (`/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications`)

| Path | Action | Responsibility |
|---|---|---|
| `datadog_agent.tf` | Modify | Add `vector_dd_agent_logs_enabled` variable + local + plan-time guard; pass `logs_intake_url` into `module.datadog_agent`; bump module ref from `v1.1.2` → `v1.2.0` |
| `vector.tf` | Modify | Bump `module "vector"` source ref from `v1.0.0` → `v1.1.0` (replace the commented-out git ref + remove local-path source) |
| `README.md` | Modify | Add `vector_dd_agent_logs_enabled` row to inputs table; note dual-layer log filtering |

---

## Phase A: Vector module v1.1.0

### Task A1: Add VRL test fixture for DD-Agent-shape logs (failing)

**Files:**
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/inputs.ndjson`
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/expected.ndjson`
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/program.vrl`
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/run.sh`

- [ ] **Step 1: Write the input fixture**

Create `tests/vrl/redact_emails_dd_agent_logs/inputs.ndjson` with five DD-Agent-shape events. The `datadog_agent` source emits events with a top-level `.message` string plus DD-Agent-attached metadata (`ddtags`, `service`, `hostname`, `status`). The VRL transform should redact emails in `.message` and string-typed `.attributes` values, but leave metadata fields untouched.

```json
{"message": "user alice@example.com signed in", "ddtags": "env:prod,service:api", "service": "api", "hostname": "ip-10-0-1-23"}
{"message": "no email here", "ddtags": "env:prod", "service": "worker", "hostname": "ip-10-0-1-24"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "service": "api", "hostname": "ip-10-0-1-25"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "ddtags": "env:staging", "service": "auth", "hostname": "ip-10-0-2-99"}
{"message": "json blob {\"email\":\"x@y.com\"}", "service": "api", "hostname": "ip-10-0-1-50"}
```

- [ ] **Step 2: Write the expected-output fixture**

Create `tests/vrl/redact_emails_dd_agent_logs/expected.ndjson`. Emails in `.message` and `.attributes.user.email` should become `[REDACTED-EMAIL]`. Metadata fields (`ddtags`, `service`, `hostname`) should pass through unchanged. Note: `jq` canonicalizes object key order — write attributes alphabetically.

```json
{"ddtags": "env:prod,service:api", "hostname": "ip-10-0-1-23", "message": "user [REDACTED-EMAIL] signed in", "service": "api"}
{"ddtags": "env:prod", "hostname": "ip-10-0-1-24", "message": "no email here", "service": "worker"}
{"hostname": "ip-10-0-1-25", "message": "multiple [REDACTED-EMAIL] and [REDACTED-EMAIL]", "service": "api"}
{"attributes": {"request.id": "abc-123", "user.email": "[REDACTED-EMAIL]"}, "ddtags": "env:staging", "hostname": "ip-10-0-2-99", "message": "ok", "service": "auth"}
{"hostname": "ip-10-0-1-50", "message": "json blob {\"email\":\"[REDACTED-EMAIL]\"}", "service": "api"}
```

- [ ] **Step 3: Write the VRL program**

Create `tests/vrl/redact_emails_dd_agent_logs/program.vrl`. Identical logic to the existing `redact_emails_logs/program.vrl` — the transform is source-agnostic, but having a dedicated test file makes per-source assertions explicit.

```vrl
email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

if is_string(.message) {
  .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
}

if is_object(.attributes) {
  attrs, _err = object(.attributes)
  if is_object(attrs) {
    .attributes = map_values(attrs) -> |v| {
      if is_string(v) { replace(string!(v), email_re, "[REDACTED-EMAIL]") } else { v }
    }
  }
}
.
```

- [ ] **Step 4: Write the runner script**

Create `tests/vrl/redact_emails_dd_agent_logs/run.sh` (mirrors siblings), then make it executable:

```bash
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
echo "redact_emails_dd_agent_logs VRL OK"
```

Then:
```bash
chmod +x /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_emails_dd_agent_logs/run.sh
```

- [ ] **Step 5: Run the test (expect: passes immediately, since the transform is identical to the existing one)**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

Expected: prints `redact_emails_dd_agent_logs VRL OK`.

If diff fails, check object-key alphabetization in `expected.ndjson` (jq canonicalizes — keys must be sorted).

- [ ] **Step 6: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add tests/vrl/redact_emails_dd_agent_logs/
git commit -m "test: add VRL test for DD-Agent-shape log redaction"
```

---

### Task A2: Wire the new VRL test into CI

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/.gitlab-ci.yml`

- [ ] **Step 1: Add the new test invocation to the `vrl-tests` job**

Open `.gitlab-ci.yml`, locate the `vrl-tests:` job's `script:` list, and append a fifth line:

```yaml
vrl-tests:
  stage: test
  image: registry.ic-ops.net/cache-docker-hub/timberio/vector:latest-debian
  before_script:
    - apt-get update -qq && apt-get install -y -qq bash diffutils jq
  script:
    - ./tests/vrl/drop_audit/run.sh
    - ./tests/vrl/redact_emails_logs/run.sh
    - ./tests/vrl/redact_emails_traces/run.sh
    - ./tests/vrl/redact_emails_metrics/run.sh
    - ./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

- [ ] **Step 2: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add .gitlab-ci.yml
git commit -m "ci: run new DD-Agent-shape VRL test"
```

---

### Task A3: Add the `dd_agent_in` source to the helm values template

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/templates/values.yaml.tpl`

- [ ] **Step 1: Add port 8282 to the service**

Locate the `service.ports:` block (around lines 5-14 of `templates/values.yaml.tpl`). Append a third port entry:

```yaml
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
```

- [ ] **Step 2: Add the `dd_agent_in` source under `customConfig.sources`**

Locate the `customConfig.sources:` block (around lines 59-65). Add a sibling source after `otlp_in`:

```yaml
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
```

`multiple_outputs: true` is required so downstream transforms can reference `dd_agent_in.logs` (the named-output suffix).

- [ ] **Step 3: Update `drop_audit.inputs` to be multi-input**

Locate the `drop_audit:` transform (around lines 67-73). Change its `inputs` list:

```yaml
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
```

- [ ] **Step 4: Validate the example renders**

The template change is gated only by template variables already present in the consumer; no new template inputs added. Run validation:

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/examples/basic
terraform init -backend=false
terraform validate
```

Expected: `Success! The configuration is valid.`

If validate fails, check YAML indentation in the new source block (it must match the 4-space indent of the surrounding map).

- [ ] **Step 5: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add templates/values.yaml.tpl
git commit -m "feat: add datadog_agent OPW source on port 8282"
```

---

### Task A4: Open netpol ingress for port 8282

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/netpol.tf`

- [ ] **Step 1: Add port 8282 to the existing cluster-from ingress rule**

Open `netpol.tf`. Locate the first ingress rule (the one with `fromEntities = ["cluster"]` and the OTLP ports list, around lines 12-21). Add a third port entry:

```hcl
      {
        # OTLP from any cluster-internal sender (primarily otel-collector).
        # Port 8282 is the OPW-compatible Datadog Agent intake (datadog_agent source).
        fromEntities = ["cluster"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
          ]
        }]
      },
```

Do not add 8282 to the `host` (kubelet) ingress rule — kubelet doesn't probe this port.

- [ ] **Step 2: Validate**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
terraform fmt -check -recursive
cd examples/basic
terraform init -backend=false
terraform validate
```

Expected: both commands pass cleanly.

- [ ] **Step 3: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add netpol.tf
git commit -m "feat: open netpol ingress for OPW intake on port 8282"
```

---

### Task A5: Expose the new endpoint as a module output

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/outputs.tf`

- [ ] **Step 1: Add `service_endpoint_dd_agent` output**

Open `outputs.tf` and append:

```hcl
output "service_endpoint_dd_agent" {
  description = "OPW-compatible Datadog Agent intake endpoint (http, cluster-internal). Pass to datadog-agent module's logs_intake_url to redirect agent logs through Vector."
  value       = "http://vector.${var.namespace}.svc.cluster.local:8282"
}
```

- [ ] **Step 2: Validate**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
terraform fmt -check -recursive
cd examples/basic
terraform init -backend=false
terraform validate
```

Expected: pass.

- [ ] **Step 3: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add outputs.tf
git commit -m "feat: expose service_endpoint_dd_agent output"
```

---

### Task A6: Update Vector module README

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/README.md`

- [ ] **Step 1: Update the architecture description**

In `README.md`, replace the existing architecture paragraph (lines 14-32) with:

```markdown
Vector sits between the OTel collector tier and Datadog, and additionally
serves as an OPW-compatible logs intake for the Datadog Agent:

```
apps → otel-lb → otel-collector ──OTLP/gRPC──▶ vector ──▶ Datadog SaaS
                                                  ▲
                                          DD Agent (logs)
```

Vector runs as a Deployment (default 3 replicas) fronted by a ClusterIP
Service. It exposes:

- OTLP/gRPC on port 4317 (from OTel collector)
- OTLP/HTTP on port 4318 (from OTel collector)
- Datadog Agent intake on port 8282 (from DD Agent, OPW protocol)

Three signal pipelines run in parallel:

- **logs** — `drop_audit` filter (multi-input: OTLP logs + DD Agent logs;
  drops lines matching audit patterns) → `redact_emails_logs` remap →
  `datadog_logs` sink.
- **traces** — `redact_emails_traces` remap → `datadog_traces` sink.
- **metrics** — `redact_emails_metrics` remap → `datadog_metrics` sink.

The Datadog API key is sourced from Vault via the existing `external-secrets`
operator. The Deployment is annotated with `reloader.stakater.com/auto: "true"`
so key rotation triggers a rolling restart automatically.
```

- [ ] **Step 2: Add the new output to the Outputs table**

Locate the Outputs table (around lines 59-63). Append:

```markdown
| `service_endpoint_dd_agent` | OPW-compatible Datadog Agent intake endpoint (`http://vector.<namespace>.svc.cluster.local:8282`) |
```

- [ ] **Step 3: Update the example version pin**

Locate the Usage example (around line 73). Change the source ref from `?ref=v1.0.0` to `?ref=v1.1.0`.

- [ ] **Step 4: Add a usage hint for the DD Agent wire-up**

After the existing OTel-collector usage hint (around line 89-93), add:

```markdown
Point the Datadog Agent at Vector's OPW intake by passing the new endpoint to
the `datadog-agent` module (v1.2.0+):

```hcl
logs_intake_url = module.vector.service_endpoint_dd_agent
```
```

- [ ] **Step 5: Update the Testing section**

In the Testing section, add a fifth runner to the VRL test list (around line 109):

```sh
./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

- [ ] **Step 6: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
git add README.md
git commit -m "docs: document DD Agent OPW intake on port 8282"
```

---

### Task A7: Run all module tests + final validation

- [ ] **Step 1: Run all VRL tests**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
./tests/vrl/drop_audit/run.sh
./tests/vrl/redact_emails_logs/run.sh
./tests/vrl/redact_emails_traces/run.sh
./tests/vrl/redact_emails_metrics/run.sh
./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

Expected: each prints `<test_name> VRL OK` and exits 0.

- [ ] **Step 2: Terraform fmt + validate**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector
terraform fmt -check -recursive
cd examples/basic
rm -rf .terraform .terraform.lock.hcl
terraform init -backend=false
terraform validate
```

Expected: fmt prints nothing, validate prints `Success!`.

- [ ] **Step 3: Sir Charles handles tagging**

Sir Charles tags `vector v1.1.0` and pushes (separate from this plan; not Claude's responsibility).

---

## Phase B: DD Agent module v1.2.0

### Task B1: Capture the current ("direct shipping") render as a baseline

**Files:**
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/tests/render/expected-direct.yaml`
- Create: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/tests/render/run.sh`

The point of this task is to lock in the current rendered output BEFORE making any changes. After Phase B's template change, rendering with `logs_intake_url = null` must produce byte-identical output.

- [ ] **Step 1: Create the test directory**

```bash
mkdir -p /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/tests/render
```

- [ ] **Step 2: Write a runner script that uses Terraform's templatefile() to render the template under fixed inputs**

Create `tests/render/run.sh`:

```bash
#!/usr/bin/env bash
# Renders values.yaml.tftpl under two input scenarios and diffs against the
# locked expected outputs. Catches accidental drift in the helm values output.
set -euo pipefail

DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
MODULE_DIR="$(cd "$DIR/../.." && pwd)"
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT

render() {
  local intake_url="$1"
  local out="$2"

  # logs_intake_url is rendered as a literal "null" sentinel via jsonencode
  # when null; the template uses `!= null` to gate, so we pass the raw value
  # via terraform's tftpl_render harness.

  cat > "$TMP/main.tf" <<EOF
output "rendered" {
  value = templatefile("\${path.module}/values.yaml.tftpl", {
    cluster_name                    = "test-cluster"
    env                             = "nonprod"
    global_segment                  = "test"
    image_tag                       = "7.55.0"
    watch_namespace                 = "datadog"
    tags                            = ["env:test"]
    agents_priority_class_name      = "cloud-ops-system"
    apm_enabled                     = true
    container_exclude               = ""
    container_exclude_logs          = ""
    enable_mrds_monitoring          = false
    enable_nowrds_monitoring        = false
    enable_now_obfusicator_options  = false
    now_obfusicator_options         = { replace_digits = "true" }
    datadog_api_key_secret_name     = "datadog-api-key"
    cluster_agent_token_secret_name = "datadog-clusteragent-token"
    log_level                       = "ERROR"
    logs_intake_url                 = ${intake_url}
  })
}
EOF
  ln -sf "$MODULE_DIR/values.yaml.tftpl" "$TMP/values.yaml.tftpl"
  (cd "$TMP" && terraform init -backend=false -input=false >/dev/null && terraform apply -auto-approve -input=false >/dev/null)
  (cd "$TMP" && terraform output -raw rendered) > "$out"
  rm -rf "$TMP"/.terraform "$TMP"/terraform.tfstate*
}

render "null"                                                "$DIR/.actual-direct.yaml"
render '"http://vector.vector.svc.cluster.local:8282"'       "$DIR/.actual-via-vector.yaml"

diff -u "$DIR/expected-direct.yaml"     "$DIR/.actual-direct.yaml"
diff -u "$DIR/expected-via-vector.yaml" "$DIR/.actual-via-vector.yaml"

echo "datadog-agent values render OK"
```

Then make it executable:
```bash
chmod +x /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/tests/render/run.sh
```

- [ ] **Step 3: Generate the baseline `expected-direct.yaml` from the CURRENT (pre-change) template**

Before the template change lands, capture today's output as the locked baseline:

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
TMP="$(mktemp -d)"
ln -sf "$(pwd)/values.yaml.tftpl" "$TMP/values.yaml.tftpl"
cat > "$TMP/main.tf" <<'EOF'
output "rendered" {
  value = templatefile("${path.module}/values.yaml.tftpl", {
    cluster_name                    = "test-cluster"
    env                             = "nonprod"
    global_segment                  = "test"
    image_tag                       = "7.55.0"
    watch_namespace                 = "datadog"
    tags                            = ["env:test"]
    agents_priority_class_name      = "cloud-ops-system"
    apm_enabled                     = true
    container_exclude               = ""
    container_exclude_logs          = ""
    enable_mrds_monitoring          = false
    enable_nowrds_monitoring        = false
    enable_now_obfusicator_options  = false
    now_obfusicator_options         = { replace_digits = "true" }
    datadog_api_key_secret_name     = "datadog-api-key"
    cluster_agent_token_secret_name = "datadog-clusteragent-token"
    log_level                       = "ERROR"
  })
}
EOF
(cd "$TMP" && terraform init -backend=false -input=false >/dev/null && terraform apply -auto-approve -input=false >/dev/null)
(cd "$TMP" && terraform output -raw rendered) > tests/render/expected-direct.yaml
rm -rf "$TMP"
```

This produces `tests/render/expected-direct.yaml` containing the current rendered helm values (without the new template variable).

- [ ] **Step 4: Verify the baseline file is non-empty and looks right**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
wc -l tests/render/expected-direct.yaml
head -5 tests/render/expected-direct.yaml
```

Expected: ~230 lines (the existing template is 234 lines), starts with `targetSystem: "linux"`.

- [ ] **Step 5: Commit the baseline + runner BEFORE the template change**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
git add tests/render/run.sh tests/render/expected-direct.yaml
git commit -m "test: lock current values render as baseline before OPW change"
```

This commit is critical — it establishes the "before" state so the next task can verify the change is byte-identical when the new variable is null.

---

### Task B2: Add the `logs_intake_url` variable

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/variables.tf`

- [ ] **Step 1: Append the new variable to `variables.tf`**

Open `variables.tf` and append:

```hcl
variable "logs_intake_url" {
  description = "OPW-compatible logs intake URL. When set, the agent ships logs to this endpoint via DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL. For cluster-internal Vector use module.vector.service_endpoint_dd_agent. When null (default), logs ship direct to Datadog SaaS."
  type        = string
  default     = null

  validation {
    condition     = var.logs_intake_url == null || can(regex("^https?://", var.logs_intake_url))
    error_message = "logs_intake_url must be a URL starting with http:// or https://, or null."
  }
}
```

- [ ] **Step 2: Run terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
terraform fmt variables.tf
```

Expected: no output (or it auto-formats and exits 0).

- [ ] **Step 3: Commit**

```bash
git add variables.tf
git commit -m "feat: add logs_intake_url variable for OPW redirection"
```

---

### Task B3: Wire the new variable into the templatefile call

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/main.tf`

- [ ] **Step 1: Add `logs_intake_url` to the templatefile arguments**

Open `main.tf`. Locate the `templatefile(...)` call inside `helm_release.this.values` (around lines 242-265). Add `logs_intake_url = var.logs_intake_url` to the arguments map. After the change, the templatefile block should look like:

```hcl
  values = [
    templatefile("${path.module}/values.yaml.tftpl", {
      cluster_name   = var.cluster_name
      env            = var.env == "staging" ? "nonprod" : var.env
      global_segment = var.global_segment
      image_tag      = var.image_tag
      watch_namespace          = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
      tags                     = concat(["global_segment:${var.global_segment}"], var.datadog_tags)
      agents_priority_class_name = var.agents_priority_class_name

      apm_enabled = var.apm_enabled

      container_exclude = var.container_exclude
      container_exclude_logs = var.container_exclude_logs

      enable_mrds_monitoring   = var.enable_mrds_monitoring
      enable_nowrds_monitoring = var.enable_nowrds_monitoring
      enable_now_obfusicator_options = var.enable_now_obfusicator_options
      now_obfusicator_options = var.now_obfusicator_options

      datadog_api_key_secret_name = var.datadog_api_key_secret_name
      cluster_agent_token_secret_name = var.cluster_agent_token_secret_name
      log_level = var.log_level

      logs_intake_url = var.logs_intake_url
    })
  ]
```

- [ ] **Step 2: Validate (note: this will fail at apply time without a real provider, but `validate` should pass)**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
terraform fmt -check
terraform init -backend=false -input=false 2>&1 | head -20 || true
terraform validate
```

Expected: `terraform validate` exits 0 with `Success!`. The `init` may complain about missing remote module sources — that's OK as long as `validate` passes.

- [ ] **Step 3: Commit**

```bash
git add main.tf
git commit -m "feat: pass logs_intake_url into helm values render"
```

---

### Task B4: Add the OPW env-var block to the values template

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/values.yaml.tftpl`

- [ ] **Step 1: Locate the `datadog.env` list**

In `values.yaml.tftpl`, find the `datadog.env:` block (lines 57-81). It currently ends with the `DD_APM_PROBABILISTIC_SAMPLER_SAMPLING_PERCENTAGE` entry on line 81.

- [ ] **Step 2: Append the gated OPW env vars**

After line 81 (the `value: 15` line for `DD_APM_PROBABILISTIC_SAMPLER_SAMPLING_PERCENTAGE`) and before `clusterAgent:` on line 82, insert:

```yaml
%{ if logs_intake_url != null ~}
    - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED
      value: "true"
    - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL
      value: "${logs_intake_url}"
%{ endif ~}
```

The 4-space indentation matches the surrounding `datadog.env` list items.

- [ ] **Step 3: Generate the "via-vector" expected render**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
TMP="$(mktemp -d)"
ln -sf "$(pwd)/values.yaml.tftpl" "$TMP/values.yaml.tftpl"
cat > "$TMP/main.tf" <<'EOF'
output "rendered" {
  value = templatefile("${path.module}/values.yaml.tftpl", {
    cluster_name                    = "test-cluster"
    env                             = "nonprod"
    global_segment                  = "test"
    image_tag                       = "7.55.0"
    watch_namespace                 = "datadog"
    tags                            = ["env:test"]
    agents_priority_class_name      = "cloud-ops-system"
    apm_enabled                     = true
    container_exclude               = ""
    container_exclude_logs          = ""
    enable_mrds_monitoring          = false
    enable_nowrds_monitoring        = false
    enable_now_obfusicator_options  = false
    now_obfusicator_options         = { replace_digits = "true" }
    datadog_api_key_secret_name     = "datadog-api-key"
    cluster_agent_token_secret_name = "datadog-clusteragent-token"
    log_level                       = "ERROR"
    logs_intake_url                 = "http://vector.vector.svc.cluster.local:8282"
  })
}
EOF
(cd "$TMP" && terraform init -backend=false -input=false >/dev/null && terraform apply -auto-approve -input=false >/dev/null)
(cd "$TMP" && terraform output -raw rendered) > tests/render/expected-via-vector.yaml
rm -rf "$TMP"
```

- [ ] **Step 4: Verify the via-vector render contains the OPW env vars**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
grep -A1 "DD_OBSERVABILITY_PIPELINES_WORKER" tests/render/expected-via-vector.yaml
```

Expected output:
```
    - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED
      value: "true"
    - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL
      value: "http://vector.vector.svc.cluster.local:8282"
```

- [ ] **Step 5: Run the render harness — both scenarios must pass**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
./tests/render/run.sh
```

Expected: prints `datadog-agent values render OK`.

The "direct" diff must be **empty** (proving byte-identical when null). The "via-vector" diff must be empty (proving expected env vars present).

If the "direct" diff shows ANY change, your `%{ if ~}` directive isn't strip-newlining correctly — the template should produce zero output when `logs_intake_url == null`. Common cause: missing `~` on either delimiter.

- [ ] **Step 6: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
git add values.yaml.tftpl tests/render/expected-via-vector.yaml
git commit -m "feat: render OPW env vars when logs_intake_url is set"
```

---

### Task B5: Update DD Agent README

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent/README.md`

- [ ] **Step 1: Add the `logs_intake_url` row to the Inputs table**

In `README.md`, locate the Inputs table (around lines 33-58). Find the row for `log_level` (alphabetical position) and insert immediately after it:

```markdown
| <a name="input_logs_intake_url"></a> [logs\_intake\_url](#input\_logs\_intake\_url) | OPW-compatible logs intake URL. When set, the agent ships logs to this endpoint via DD\_OBSERVABILITY\_PIPELINES\_WORKER\_LOGS\_URL. For cluster-internal Vector use module.vector.service\_endpoint\_dd\_agent. When null (default), logs ship direct to Datadog SaaS. | `string` | `null` | no |
```

- [ ] **Step 2: Add an "OPW Logs Intake" section after the Outputs table**

Append at the end of `README.md`:

```markdown

## OPW Logs Intake (optional)

By default, the agent ships logs directly to Datadog SaaS over HTTPS. To
redirect logs through a Datadog Observability Pipelines Worker (OPW), or
through a Vector deployment functioning as one, set `logs_intake_url`:

```hcl
module "datadog_agent" {
  # ...
  logs_intake_url = module.vector.service_endpoint_dd_agent
  # e.g. "http://vector.vector.svc.cluster.local:8282"
}
```

This sets two env vars on the agent containers:

- `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED=true`
- `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL=<logs_intake_url>`

Agent-side processing rules (`DD_LOGS_CONFIG_PROCESSING_RULES`) continue to
execute before transmission, regardless of destination. Metrics and APM
(traces) are unaffected and continue shipping direct to Datadog.

When `logs_intake_url` is `null`, the rendered helm values are byte-identical
to the v1.1.x line — there is no behavior change for clusters that do not opt in.

Reference: <https://docs.datadoghq.com/observability_pipelines/sources/datadog_agent/>
```

- [ ] **Step 3: Commit**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
git add README.md
git commit -m "docs: document logs_intake_url for OPW redirection"
```

---

### Task B6: Final module validation

- [ ] **Step 1: Run the full render harness**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
./tests/render/run.sh
```

Expected: `datadog-agent values render OK`.

- [ ] **Step 2: terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
terraform fmt -check -recursive
```

Expected: no output, exit 0.

- [ ] **Step 3: terraform validate**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/datadog-agent
terraform init -backend=false -input=false 2>&1 | tail -5 || true
terraform validate
```

Expected: `Success!`

- [ ] **Step 4: Sir Charles handles tagging**

Sir Charles tags `datadog-agent v1.2.0` and pushes (separate from this plan).

---

## Phase C: Consumer (eks-cluster-applications) wire-up

### Task C1: Add the `vector_dd_agent_logs_enabled` flag

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/datadog_agent.tf`

- [ ] **Step 1: Add the variable, local, and plan-time guard**

Open `datadog_agent.tf`. At the bottom of the file (after the existing `variable "datadog_agent_enabled"` block, around line 117), append:

```hcl
# ---------------------------------------------------------------------------------------------------------------------
# DD AGENT → VECTOR LOGS ROUTING (OPW)
# ---------------------------------------------------------------------------------------------------------------------
locals {
  vector_dd_agent_logs_enabled = (
    var.vector_dd_agent_logs_enabled != null
    ? var.vector_dd_agent_logs_enabled
    : local.vector_enabled
  )
}

variable "vector_dd_agent_logs_enabled" {
  description = "Override for routing DD Agent logs through Vector. When null (default), inherits from local.vector_enabled (i.e., on wherever Vector is on). Set to false to keep DD Agent shipping logs directly to DD SaaS even when Vector is deployed."
  type        = bool
  default     = null
}

# Plan-time guard: vector_dd_agent_logs_enabled cannot be true if vector itself is off.
# Mirrors the existing data "external" "vector_validate" pattern in vector.tf.
data "external" "vector_dd_agent_logs_validate" {
  count = local.vector_dd_agent_logs_enabled && !local.vector_enabled ? 1 : 0

  program = ["Error: vector_dd_agent_logs_enabled requires vector_enabled"]
}
```

- [ ] **Step 2: Run terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform fmt datadog_agent.tf
```

- [ ] **Step 3: Commit**

```bash
git add datadog_agent.tf
git commit -m "feat: add vector_dd_agent_logs_enabled flag with vector_enabled inheritance"
```

---

### Task C2: Bump the `module.datadog_agent` ref and pass `logs_intake_url`

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/datadog_agent.tf`

- [ ] **Step 1: Bump the module source ref**

In `datadog_agent.tf`, line 14, change:

```hcl
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/datadog-agent?ref=v1.1.2"
```

to:

```hcl
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/datadog-agent?ref=v1.2.0"
```

- [ ] **Step 2: Add the `logs_intake_url` argument to the module call**

In the same `module "datadog_agent"` block (around lines 13-34), add `logs_intake_url` as the last argument before the closing `}`:

```hcl
module "datadog_agent" {
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/datadog-agent?ref=v1.2.0"
  count  = local.datadog_agent_enabled ? 1 : 0

  providers = {
    kubernetes = kubernetes.local-cluster
    helm       = helm.local-cluster
  }

  chart_version                  = var.datadog_agent_version
  image_tag                      = var.datadog_agent_image_tag
  create_namespace               = var.datadog_create_namespace
  cluster_name                   = var.cluster_name
  global_segment                 = var.global_segment
  datadog_tags                   = var.datadog_tags
  enable_nowrds_monitoring       = var.datadog_enable_nowrds_monitoring
  enable_now_obfusicator_options = var.datadog_enable_now_obfusicator_options
  enable_mrds_monitoring         = var.datadog_enable_mrds_monitoring
  now_obfusicator_options        = var.datadog_now_obfusicator_options
  tags                           = var.tags
  env                            = var.env

  logs_intake_url = local.vector_dd_agent_logs_enabled ? module.vector[0].service_endpoint_dd_agent : null
}
```

The ternary handles the `count = 0` case correctly: when `local.vector_dd_agent_logs_enabled = false`, the right-hand side evaluates to `null` without dereferencing `module.vector[0]`.

- [ ] **Step 3: Run terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform fmt datadog_agent.tf
```

- [ ] **Step 4: Commit**

```bash
git add datadog_agent.tf
git commit -m "feat: route DD Agent logs through Vector when enabled"
```

---

### Task C3: Bump `module.vector` ref to v1.1.0

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/vector.tf`

- [ ] **Step 1: Replace the local-path source with the tagged git ref**

In `vector.tf`, lines 14-15 currently read:

```hcl
  #source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"
  source = "/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector"
```

Change to:

```hcl
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.1.0"
```

(Removes the local-path source and the commented-out v1.0.0 line; replaces with the v1.1.0 tagged git ref. The local-path source was only present during development, per the prior session note.)

- [ ] **Step 2: Run terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform fmt vector.tf
```

- [ ] **Step 3: Commit**

```bash
git add vector.tf
git commit -m "chore: pin vector module to v1.1.0 (OPW intake)"
```

---

### Task C4: Update consumer README

**Files:**
- Modify: `/Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/README.md`

- [ ] **Step 1: Read the existing inputs table**

```bash
grep -n "vector_email_redaction_enabled\|vector_enabled\|input_vector" /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications/README.md | head -30
```

This locates the Vector-related inputs table rows so the new entry slots in alphabetically.

- [ ] **Step 2: Add the new input row**

In `README.md`, in the inputs table where Vector variables are listed, insert (alphabetically — between `vector_chart_version` and `vector_email_redaction_enabled` or wherever it sorts):

```markdown
| <a name="input_vector_dd_agent_logs_enabled"></a> [vector\_dd\_agent\_logs\_enabled](#input\_vector\_dd\_agent\_logs\_enabled) | Override for routing DD Agent logs through Vector. When null (default), inherits from local.vector\_enabled (i.e., on wherever Vector is on). Set to false to keep DD Agent shipping logs directly to DD SaaS even when Vector is deployed. | `bool` | `null` | no |
```

- [ ] **Step 3: Add a paragraph about dual-layer log filtering**

Find the existing Vector-related section in the README (added in the prior session per the spec) and append a new paragraph:

```markdown

### DD Agent → Vector log routing (OPW)

When `vector_dd_agent_logs_enabled` is true (defaults to `local.vector_enabled`),
the Datadog Agent ships container logs to Vector via the OPW HTTP intake on
`vector.vector.svc.cluster.local:8282`. Logs traverse two filtering layers
before reaching Datadog SaaS:

1. **Agent-side** — `DD_LOGS_CONFIG_PROCESSING_RULES` (see
   `datadog-agent/processing-rules/{env}/rules.json`) drops audit lines before
   transmission. Authoritative for audit drops.
2. **Vector** — `drop_audit` regex filter (defense-in-depth, same patterns) +
   `redact_emails_logs` substitutes email addresses with `[REDACTED-EMAIL]`.

Metrics and APM (traces) are unaffected and continue their existing OTel-based
path through Vector. To roll back logs to direct DD shipping for a single
cluster while keeping Vector traces, set `vector_dd_agent_logs_enabled = false`
on that cluster — Vector continues serving OTel traffic, agent re-rolls back to
direct logs.
```

- [ ] **Step 4: Commit**

```bash
git add README.md
git commit -m "docs: document vector_dd_agent_logs_enabled and dual-layer log filtering"
```

---

### Task C5: Final consumer validation

- [ ] **Step 1: terraform fmt**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform fmt -check -recursive
```

Expected: no output, exit 0.

- [ ] **Step 2: terraform validate (best-effort, may need cluster auth)**

```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
terraform init -backend=false -input=false 2>&1 | tail -10 || true
terraform validate
```

Expected: `Success!`. If `init` cannot fetch the new module versions (because Sir Charles hasn't pushed the tags yet), `validate` will fail with module-not-found; that's expected pre-tag and Sir Charles will re-run after tagging.

- [ ] **Step 3: Note for Sir Charles**

Once `vector v1.1.0` and `datadog-agent v1.2.0` are tagged + pushed, re-run:
```bash
cd /Users/steve.briggs/projects/IAC/terraform/iac-library/aws/compute/eks-cluster-applications
rm -rf .terraform .terraform.lock.hcl
terraform init -backend=false
terraform validate
```

To confirm clean resolution against the real tags.

---

## Phase D: Deploy verification (post-rollout, ad-hoc)

These steps run against a real cluster after the three repos are tagged, MRs merged, and `terraform apply` has rolled the change out. They are **not** committable steps — they are runbook entries.

### Task D1: Verify DD Agent ingests via OPW path (smoke)

- [ ] **Step 1: Confirm the new env vars are live on agent pods**

```bash
kubectl exec -n datadog ds/immuta-dd -c agent -- env | grep DD_OBSERVABILITY_PIPELINES_WORKER
```

Expected:
```
DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED=true
DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL=http://vector.vector.svc.cluster.local:8282
```

If empty: the helm release didn't roll, or `vector_dd_agent_logs_enabled` is false. Check `terraform plan` output and `kubectl describe ds/immuta-dd`.

- [ ] **Step 2: Confirm Vector is receiving DD Agent traffic**

```bash
kubectl exec -n vector deploy/vector -- curl -s http://localhost:8686/metrics \
  | grep 'vector_component_received_events_total{component_id="dd_agent_in"'
```

Expected: a non-zero counter that increases over time.

If zero: check Vector pod logs for source-startup errors (`kubectl logs -n vector deploy/vector | grep dd_agent_in`).

### Task D2: End-to-end redaction verification

- [ ] **Step 1: Emit an email-tagged log line from a test pod**

```bash
UUID="dd-redact-$(uuidgen)"
kubectl run "${UUID}" --image=busybox --restart=Never --rm -it \
  -- sh -c "echo '${UUID}: contact alice@example.com'; sleep 5"
```

- [ ] **Step 2: Search Datadog Logs UI**

Query: `${UUID}` (the test marker). Within ~30s the matching log should appear.

Expected: log message reads `${UUID}: contact [REDACTED-EMAIL]` (not the original `alice@example.com`).

If unredacted: check `vector_component_errors_total{component_id="redact_emails_logs"}` and the email regex against the actual log shape.

- [ ] **Step 3: Verify metadata preservation**

In the same DD Logs UI result, verify these facets are populated:
- `service` (should match the test pod's service tag, often the pod name)
- `host` (the EC2 instance / node)
- `kubernetes.namespace_name` (= `default`)
- `kubernetes.pod_name` (= `${UUID}`)

If missing: the `datadog_agent` source's `multiple_outputs` may be dropping fields. Check Vector pod logs.

### Task D3: Audit-drop verification

- [ ] **Step 1: Emit an audit-shape log line**

```bash
UUID2="dd-audit-$(uuidgen)"
kubectl run "${UUID2}" --image=busybox --restart=Never --rm -it \
  -- sh -c "echo 'Audit - test ${UUID2}'; sleep 5"
```

- [ ] **Step 2: Confirm zero results in Datadog**

Query: `${UUID2}`. Expected: zero results (dropped at the agent before transmission to Vector).

- [ ] **Step 3: Confirm Vector did NOT receive it**

```bash
kubectl exec -n vector deploy/vector -- curl -s http://localhost:8686/metrics \
  | grep 'vector_component_received_events_total{component_id="dd_agent_in"'
```

Note the value. Wait 30s, re-run. The counter should NOT have incremented for this audit line (since the agent dropped it before transmission).

If the counter ticked up: agent-side processing rules aren't matching the new audit pattern. Check `kubectl get cm -n datadog rules-config -o yaml`.

### Task D4: Rollback drill

- [ ] **Step 1: Flip the flag false on a non-prod cluster**

In the cluster's instance directory, set `vector_dd_agent_logs_enabled = false`. Run `terraform apply`.

- [ ] **Step 2: Confirm env vars removed from agent**

```bash
kubectl exec -n datadog ds/immuta-dd -c agent -- env | grep DD_OBSERVABILITY_PIPELINES_WORKER || echo "no OPW env vars"
```

Expected: prints `no OPW env vars`.

- [ ] **Step 3: Re-run the redaction smoke test from Task D2**

Expected: log appears in DD UI with the email address NOT redacted (proving the agent reverted to direct shipping).

- [ ] **Step 4: Flip the flag back on**

`vector_dd_agent_logs_enabled = null` (or `true`), `terraform apply`. Re-run Task D2 step 2 — redaction should be active again.

---

## Self-Review

**Spec coverage check** (each numbered subsection of the spec → mapped task):

| Spec section | Implementing task(s) |
|---|---|
| §3.1 Pipeline before/after | Phase A + B (mechanics), Phase C (gating) |
| §3.2 Module split | Phase A (Vector), B (DD Agent), C (consumer) |
| §4.1 Vector module deltas — `service.ports` | A3 step 1 |
| §4.1 Vector module deltas — `dd_agent_in` source | A3 step 2 |
| §4.1 Vector module deltas — `drop_audit.inputs` | A3 step 3 |
| §4.1 netpol port 8282 | A4 |
| §4.1 outputs | A5 |
| §4.1 README | A6 |
| §4.2 DD Agent module — `logs_intake_url` variable | B2 |
| §4.2 DD Agent module — values template | B4 |
| §4.2 DD Agent module — main.tf wiring | B3 |
| §4.2 DD Agent module — README | B5 |
| §4.3 Consumer — `vector_dd_agent_logs_enabled` flag | C1 |
| §4.3 Consumer — module ref bumps | C2, C3 |
| §4.3 Consumer — wire `logs_intake_url` | C2 |
| §4.3 Consumer — README | C4 |
| §6.1 Module input validation — `logs_intake_url` regex | B2 |
| §6.1 Module input validation — consumer guard | C1 |
| §7.1 Unit tests — VRL | A1 |
| §7.1 Unit tests — DD Agent template render | B1, B4 |
| §7.1 Unit tests — terraform fmt/validate | A7, B6, C5 |
| §7.2 Integration verification | D1, D2, D3 |
| §7.3 Cross-cutting — metadata preservation | D2 step 3 |
| §7.3 Cross-cutting — JSON-shape redaction | (covered by A1 fixture line 5) |
| §7.4 Soak | (Sir Charles's operational responsibility, not implementation) |

No gaps.

**Type/name consistency check:**
- Variable name `logs_intake_url` consistent across DD Agent module + consumer wire-up.
- Variable name `vector_dd_agent_logs_enabled` consistent across consumer var + local + guard.
- Output name `service_endpoint_dd_agent` consistent across Vector module + consumer wire-up.
- Port `8282` consistent across Vector service.ports + dd_agent_in.address + netpol + output URL + DD Agent docs/README.
- Module versions: `vector v1.1.0` and `datadog-agent v1.2.0` consistent across spec + plan + consumer ref bumps.