---
date: 2026-05-06
project: IAC
tags:
  - claude-context
  - "#iac"
  - "#vector"
  - "#datadog"
  - "#opw"
  - "#observability"
---

# Datadog Agent Logs via Vector (OPW Intake)

## Summary

Designed and implemented routing of Datadog-Agent-collected container logs through Vector, so the existing email-redaction VRL transform applies to log bodies the agent collects (today they bypass Vector and ship direct to DD SaaS). Used the Datadog **Observability Pipelines Worker** (OPW) intake protocol — Vector's native `datadog_agent` source is the same wire protocol Datadog's OPW product uses (Datadog acquired Vector). Three additive module changes spanning vector module v1.1.0 → v1.1.0, datadog-agent module v1.1.2 → v1.2.0, and the eks-cluster-applications consumer.

Companion notes:
- `2026-05-05-vector-redaction-pipeline.md` — original Vector module design + Phase A/B/C of the OTel-based redaction pipeline.
- `2026-05-05-vector-deploy-fixes.md` — helm v2/v3 syntax fixes, VRL E620 cleanups, netpol additions discovered during Vector deploy.

## Context

**Where logs flowed before this work:**
```
DD Agent (DaemonSet) → [agent-side processing rules drop audits] → datadoghq.com:443
```
DD Agent's `DD_LOGS_CONFIG_PROCESSING_RULES` (sourced from `processing-rules/{env}/rules.json` config map) dropped audit lines BEFORE transmission. No redaction applied to DD-Agent-collected logs.

**Where they flow after:**
```
DD Agent → [agent-side audit drops] → vector.vector.svc.cluster.local:8282 (OPW HTTP)
       → Vector dd_agent_in source
       → drop_audit (multi-input: otlp_in.logs + dd_agent_in.logs, defense-in-depth)
       → redact_emails_logs
       → datadog_logs sink → datadoghq.com:443
```

**Pivot during design (Q4):** Initially proposed setting `logs_config.use_http: true`, `logs_config.logs_no_ssl: true`, `logs_config.logs_dd_url: <url>` directly in the agent's helm values. Sir Charles redirected to the **OPW env-var path** (`DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED=true` + `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL=<url>`), which is the documented Datadog approach and cleaner. URL scheme `http://` implies plaintext — no separate `logs_no_ssl` flag needed.

**Port 8282** is the Datadog-documented OPW default (verified via WebFetch of the Datadog OPW docs). Initial design recommended 8080; revised to 8282 for OPW consistency.

**Reference:** <https://docs.datadoghq.com/observability_pipelines/sources/datadog_agent/?tab=secretsmanagement>

## Decisions

**1. OPW env-var path over helm `logs_config` block.** Cleaner gating in the values template (just two env vars appended to `datadog.env`), no `use_http`/`logs_no_ssl` knobs to expose, future-proof if Immuta later runs a real OPW alongside Vector.

**2. Single bool variable `vector_dd_agent_logs_enabled`, defaults to inheriting from `local.vector_enabled`.** Considered the three-part construct (per-cluster-type map + override variable + computed local) that mirrors the existing `vector_enabled` pattern, but this flag has no meaning where Vector isn't deployed — so the cluster-type granularity is already provided by the inherited Vector enablement. One variable, fewer knobs, same outcome. The override (set explicitly to `true`/`false`) is preserved as a safety hatch.

**3. `drop_audit` becomes multi-input (defense-in-depth) — Q3 option (b).** The transform's `inputs` list goes from `["otlp_in.logs"]` to `["otlp_in.logs", "dd_agent_in.logs"]`. The agent rules already drop audits before transmission so Vector's drop is technically redundant, but cheap belt-and-suspenders catches anything the agent rules miss/misconfigure. Sir Charles explicitly chose this over option (a) "skip drop_audit on the new path" or option (c) "two separate filters per source".

**4. Goal scoped to redaction only — Q1 option (a).** Did NOT migrate the agent-side audit-drop rules (`processing-rules/{env}/rules.json`) into Vector. Those rules stay where they are; Vector adds redaction only. Long-term consolidation is flagged as an open question but explicitly out of scope.

**5. Vector module owns the `datadog_agent` source unconditionally.** No new module variable for "is the source on?" — the receiver is harmless when no agent is pointed at it (it just listens), and adding a flag would be churn for no benefit. The OPW intake URL output (`service_endpoint_dd_agent`) is the only new public interface.

**6. DD Agent module's new variable `logs_intake_url` is a single string (default null) — Q4 option (a).** When non-null the values template renders the two OPW env vars; when null the rendered output is byte-identical to today (proven by the render harness diff in `tests/render/`). Not a structured object — `use_http`/`logs_no_ssl` would be forced for cluster-internal Vector anyway and exposing them invites misconfiguration (TLS off when pointing at real DD SaaS = broken).

**7. Module version bumps:** `vector` v1.0.0 → **v1.1.0**, `datadog-agent` v1.1.2 → **v1.2.0**. Consumer ref bumps both. Note: vector v1.0.0 tag itself is still owed from the prior session (Sir Charles handles git).

## Task State

**Spec + plan written and committed to disk:**
- `/Users/steve.briggs/projects/IAC/docs/superpowers/specs/2026-05-06-datadog-agent-logs-via-vector-design.md`
- `/Users/steve.briggs/projects/IAC/docs/superpowers/plans/2026-05-06-datadog-agent-logs-via-vector.md`

**Implementation executed via subagent-driven-development skill** — 18 tasks (A1–A7, B1–B6, C1–C5), each through implementer + spec/quality review cycles, then a final cross-repo Opus review. Final review: ✅ Ready to tag and ship, no Critical/Important issues.

**Vector module changes (ready for `v1.1.0` tag):**
- `templates/values.yaml.tpl` — added `dd-agent` port 8282 to `service.ports`; added `dd_agent_in` source under `customConfig.sources` with `multiple_outputs: true`, `disable_metrics/disable_traces: true`, `store_api_key: false`, `acknowledgements.enabled: true`; expanded `drop_audit.inputs` to multi-input.
- `netpol.tf` — added port 8282 to the existing `fromEntities = ["cluster"]` ingress rule. Did NOT touch the kubelet `host` rule (kubelet doesn't probe 8282).
- `outputs.tf` — added `service_endpoint_dd_agent` output (`http://vector.<namespace>.svc.cluster.local:8282`).
- `tests/vrl/redact_emails_dd_agent_logs/` — new VRL test directory (inputs, expected, program.vrl, run.sh) covering DD-Agent-shape events with `ddtags`/`service`/`hostname` metadata.
- `.gitlab-ci.yml` — appended new test runner to `vrl-tests` job.
- `README.md` — architecture diagram updated, outputs table row added, usage hint for DD Agent wire-up added, testing list updated, version pin in example bumped to v1.1.0.

**DD Agent module changes (ready for `v1.2.0` tag):**
- `variables.tf` — added `logs_intake_url` (string, default null, URL-prefix validation `^https?://`).
- `main.tf` — passes `logs_intake_url = var.logs_intake_url` into the `templatefile()` call. (Note: `terraform fmt` re-aligned equals signs in the existing args map as a side effect — pre-existing fmt drift now incorporated.)
- `values.yaml.tftpl` — appended gated block (`%{ if logs_intake_url != null ~}...%{ endif ~}`) for two OPW env vars in `datadog.env`. Both `~}` strip-newline directives present (verified empirically).
- `tests/render/` — new render harness (`run.sh`, `expected-direct.yaml`, `expected-via-vector.yaml`). The 130-line `expected-direct.yaml` was locked from the unmodified template BEFORE the template change so the byte-identical assertion is meaningful.
- `README.md` — added `logs_intake_url` row to inputs table, new "OPW Logs Intake (optional)" section.

**Consumer (`eks-cluster-applications`) changes (working tree, uncommitted):**
- `datadog_agent.tf` — module ref bumped `v1.1.2 → v1.2.0`; `logs_intake_url = local.vector_dd_agent_logs_enabled ? module.vector[0].service_endpoint_dd_agent : null` ternary on the module call (short-circuits safely when Vector count=0); new `vector_dd_agent_logs_enabled` variable + local + plan-time guard via `data "external" "vector_dd_agent_logs_validate"` (mirrors the existing `vector_validate` pattern).
- `vector.tf` — currently using the local-path source (`/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector`) with the v1.1.0 git ref commented out. The Plan task C3 had bumped to the git ref; this is the development-iteration shape until the v1.1.0 tag is actually published.
- `README.md` — added `vector_dd_agent_logs_enabled` to the friendly Vector variables table; added the new "DD Agent → Vector log routing (OPW)" section with the two-layer filtering description; added a row to the auto-generated inputs table (alphabetically between `vector_chart_version` and `vector_datadog_api_key_vault_path`).

**Verifications run:**
- All 5 VRL tests pass (`drop_audit`, `redact_emails_logs`, `_traces`, `_metrics`, `_dd_agent_logs`).
- DD Agent render harness passes both diff scenarios: `logs_intake_url = null` → byte-identical to baseline; URL set → OPW env vars correctly rendered.
- `terraform fmt -check` clean on all 4 .tf files we touched. Pre-existing fmt drift exists in untouched files (`otel_collector.tf`, `sysdig_agent.tf` in consumer; `netpols.tf` in DD Agent module) — not part of this MR's diff.
- `terraform validate` deferred for the consumer until Sir Charles publishes the v1.1.0 / v1.2.0 tags; init can't fetch the new module versions before then.

## Open Questions

1. **Vector module v1.0.0 tag still owed from prior session.** v1.1.0 changes sit on top of an unpublished v1.0.0 codebase. Sir Charles needs to decide: bundle v1.0.0 + v1.1.0 changes into a single v1.1.0 tag (since v1.0.0 was never public), or tag v1.0.0 first then a separate v1.1.0 immediately after.

2. **Auto-generated `terraform-docs` section in the consumer README is stale.** The Modules table still shows `module_datadog_agent | v1.1.0` (should be v1.2.0) and lacks a `module_vector` row entirely. Run `terraform-docs` regeneration as part of the consumer MR. The friendly tables we added by hand are correct.

3. **Agent buffer behavior under sustained Vector outage.** DD Agent's on-disk log buffer is bounded; if Vector is down for hours the oldest buffered logs may drop. Existing alerting on `datadog.agent.logs.bytes_buffered` should catch this case but worth confirming during soak.

4. **Audit-drop rule consolidation (long-term).** The agent-side `processing-rules/{env}/rules.json` and Vector's `drop_audit` regex now both exist. Drift between them is silent. Long-term, may want to make one authoritative — out of scope here, flagged for later cleanup.

5. **DD Logs UI metadata preservation** needs an in-cluster verification once deployed. The `datadog_agent` source preserves `ddtags`/`service`/`hostname`/`kubernetes.*` and the redaction transforms only touch `.message` and string `.attributes`, but ad-hoc verification through the DD UI is the proof.

## Next Steps

**Sir Charles's git/tag side (carries over from prior session, expanded):**

1. Decide branch strategy for `vector v1.0.0`: bundle into v1.1.0 or tag separately first.
2. Tag `vector v1.1.0` and push.
3. Tag `datadog-agent v1.2.0` and push.
4. Re-point `eks-cluster-applications/vector.tf` source from local-path to `git@.../vector.git?ref=v1.1.0` once the tag is published.
5. Open MRs for all three repos (vector, datadog-agent bump, eks-cluster-applications consumer wire-up).
6. Regenerate `terraform-docs` in the consumer to refresh the Modules table.

**Once tags are remote:**

7. `cd eks-cluster-applications && rm -rf .terraform .terraform.lock.hcl && terraform init -backend=false && terraform validate` — confirm clean module resolution.

**Deploy verification (Plan §D — runbook entries, not committable):**

8. Confirm OPW env vars are live on agent pods: `kubectl exec -n datadog ds/immuta-dd -c agent -- env | grep DD_OBSERVABILITY_PIPELINES_WORKER`.
9. Confirm Vector receiving DD Agent traffic: `kubectl exec -n vector deploy/vector -- curl -s http://localhost:8686/metrics | grep 'vector_component_received_events_total{component_id="dd_agent_in"'`.
10. UUID-tagged log line e2e test: emit `dd-redact-<UUID>: contact alice@example.com` from a busybox pod, search DD Logs UI for the UUID, confirm `[REDACTED-EMAIL]` substituted.
11. Audit-drop verification: emit `Audit - test-<UUID>`, expect zero DD UI results AND no Vector `dd_agent_in` ingest counter increment (proving the agent dropped it before transmission).
12. Rollback drill: flip `vector_dd_agent_logs_enabled = false` on a non-prod cluster, terraform apply, confirm env vars removed and emails appear unredacted in DD UI; flip back.

**Operational rollout (per spec §7.4):**

13. Sandbox/`dev` cluster — 1 day full smoke suite + Vector self-metrics watch.
14. One `mgmt` cluster — 3-7 day soak; watch `vector_component_errors_total{component_id="dd_agent_in"}`, `datadog.agent.logs.bytes_buffered`, DD Logs ingest delta vs. baseline.
15. Remaining `mgmt` + all `cps` clusters.
16. `saas` clusters region-by-region.

**Future work (post-v1):**

17. Audit-drop consolidation — move the `exclude_at_match` patterns either fully into Vector (and decommission the agent-side config map) or keep agent-side authoritative and document Vector's drop_audit as redundant defense-in-depth only. Pick one.
18. After Vector clean in prod for one quarter, decommission `kubernetes/applications/datadog-processing-rules/` (carry-over from prior session).


Configure Datadog to ship logs to Vector


File created successfully

File created successfully


Saved to `Claude/IAC/2026-05-06-dd-agent-logs-via-vector-opw.md`, Sir Charles of St Cheese.

The note captures the OPW pivot decision, the single-bool flag with `vector_enabled` inheritance choice, the multi-input `drop_audit` defense-in-depth call, the byte-identical baseline render proof, all 18 task outcomes across the three repos, and the carry-over git work (vector v1.0.0 tag still owed, vector v1.1.0 + datadog-agent v1.2.0 to tag, MRs to open, terraform-docs regen for the consumer README, plus the §D smoke-test runbook entries for post-deploy).

Future me (or future you) can `/obsidian-load` to pick up exactly where we left off — context preserved, decisions reasoned-out with their why, open questions surfaced.

Now if you'll excuse me, three repos worth of OPW wire-up clearly warrants a celebratory dino-nug session. Stegosaurus first — best sauce-adhesion-to-bite ratio in the species.


afplay ~/.claude/sounds/mario/stop/stomp.wav 2>/dev/null || true


We routed Datadog Agent logs through Vector via OPW intake (port 8282) so Vector's email redaction applies; all 18 implementation tasks across three repos are done and verified. Next action: tag vector v1.1.0 and datadog-agent v1.2.0, then open the three MRs. (disable recaps in /config)


<local-command-caveat>Caveat: The messages below were generated by the user while running local commands. DO NOT respond to these messages or otherwise consider them in your response unless the user explicitly asks you to.</local-command-caveat>


<command-name>/model</command-name>
            <command-message>model</command-message>
            <command-args></command-args>


<local-command-stdout>Set model to Sonnet 4.6 (default)</local-command-stdout>


<command-name>/exit</command-name>
            <command-message>exit</command-message>
            <command-args></command-args>


<local-command-stdout>Goodbye!</local-command-stdout>