<EXTREMELY_IMPORTANT>
You have superpowers.

**Below is the full content of your 'superpowers:using-superpowers' skill - your introduction to using skills. For all other skills, use the 'Skill' tool:**

---
name: using-superpowers
description: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
---

<SUBAGENT-STOP>
If you were dispatched as a subagent to execute a specific task, skip this skill.
</SUBAGENT-STOP>

<EXTREMELY-IMPORTANT>
If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.

IF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.

This is not negotiable. This is not optional. You cannot rationalize your way out of this.
</EXTREMELY-IMPORTANT>

## Instruction Priority

Superpowers skills override default system prompt behavior, but **user instructions always take precedence**:

1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority
2. **Superpowers skills** — override default system behavior where they conflict
3. **Default system prompt** — lowest priority

If CLAUDE.md, GEMINI.md, or AGENTS.md says "don't use TDD" and a skill says "always use TDD," follow the user's instructions. The user is in control.

## How to Access Skills

**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.

**In Copilot CLI:** Use the `skill` tool. Skills are auto-discovered from installed plugins. The `skill` tool works the same as Claude Code's `Skill` tool.

**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.

**In other environments:** Check your platform's documentation for how skills are loaded.

## Platform Adaptation

Skills use Claude Code tool names. Non-CC platforms: see `references/copilot-tools.md` (Copilot CLI), `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.

# Using Skills

## The Rule

**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.

```dot
digraph skill_flow {
    "User message received" [shape=doublecircle];
    "About to EnterPlanMode?" [shape=doublecircle];
    "Already brainstormed?" [shape=diamond];
    "Invoke brainstorming skill" [shape=box];
    "Might any skill apply?" [shape=diamond];
    "Invoke Skill tool" [shape=box];
    "Announce: 'Using [skill] to [purpose]'" [shape=box];
    "Has checklist?" [shape=diamond];
    "Create TodoWrite todo per item" [shape=box];
    "Follow skill exactly" [shape=box];
    "Respond (including clarifications)" [shape=doublecircle];

    "About to EnterPlanMode?" -> "Already brainstormed?";
    "Already brainstormed?" -> "Invoke brainstorming skill" [label="no"];
    "Already brainstormed?" -> "Might any skill apply?" [label="yes"];
    "Invoke brainstorming skill" -> "Might any skill apply?";

    "User message received" -> "Might any skill apply?";
    "Might any skill apply?" -> "Invoke Skill tool" [label="yes, even 1%"];
    "Might any skill apply?" -> "Respond (including clarifications)" [label="definitely not"];
    "Invoke Skill tool" -> "Announce: 'Using [skill] to [purpose]'";
    "Announce: 'Using [skill] to [purpose]'" -> "Has checklist?";
    "Has checklist?" -> "Create TodoWrite todo per item" [label="yes"];
    "Has checklist?" -> "Follow skill exactly" [label="no"];
    "Create TodoWrite todo per item" -> "Follow skill exactly";
}
```

## Red Flags

These thoughts mean STOP—you're rationalizing:

| Thought | Reality |
|---------|---------|
| "This is just a simple question" | Questions are tasks. Check for skills. |
| "I need more context first" | Skill check comes BEFORE clarifying questions. |
| "Let me explore the codebase first" | Skills tell you HOW to explore. Check first. |
| "I can check git/files quickly" | Files lack conversation context. Check for skills. |
| "Let me gather information first" | Skills tell you HOW to gather information. |
| "This doesn't need a formal skill" | If a skill exists, use it. |
| "I remember this skill" | Skills evolve. Read current version. |
| "This doesn't count as a task" | Action = task. Check for skills. |
| "The skill is overkill" | Simple things become complex. Use it. |
| "I'll just do this one thing first" | Check BEFORE doing anything. |
| "This feels productive" | Undisciplined action wastes time. Skills prevent this. |
| "I know what that means" | Knowing the concept ≠ using the skill. Invoke it. |

## Skill Priority

When multiple skills could apply, use this order:

1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task
2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution

"Let's build X" → brainstorming first, then implementation skills.
"Fix this bug" → debugging first, then domain-specific skills.

## Skill Types

**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.

**Flexible** (patterns): Adapt principles to context.

The skill itself tells you which.

## User Instructions

Instructions say WHAT, not HOW. "Add X" or "Fix Y" doesn't mean skip workflows.

</EXTREMELY_IMPORTANT>


<memvid-mind-context>
# 🧠 Claude Mind Ready

📁 Project: **vector**
💾 Memory will be created at: `.claude/mind.mv2`

_Your observations will be automatically captured._
</memvid-mind-context>


<EXTREMELY_IMPORTANT>
You have superpowers.

**Below is the full content of your 'superpowers:using-superpowers' skill - your introduction to using skills. For all other skills, use the 'Skill' tool:**

---
name: using-superpowers
description: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
---

<SUBAGENT-STOP>
If you were dispatched as a subagent to execute a specific task, skip this skill.
</SUBAGENT-STOP>

<EXTREMELY-IMPORTANT>
If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.

IF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.

This is not negotiable. This is not optional. You cannot rationalize your way out of this.
</EXTREMELY-IMPORTANT>

## Instruction Priority

Superpowers skills override default system prompt behavior, but **user instructions always take precedence**:

1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority
2. **Superpowers skills** — override default system behavior where they conflict
3. **Default system prompt** — lowest priority

If CLAUDE.md, GEMINI.md, or AGENTS.md says "don't use TDD" and a skill says "always use TDD," follow the user's instructions. The user is in control.

## How to Access Skills

**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.

**In Copilot CLI:** Use the `skill` tool. Skills are auto-discovered from installed plugins. The `skill` tool works the same as Claude Code's `Skill` tool.

**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.

**In other environments:** Check your platform's documentation for how skills are loaded.

## Platform Adaptation

Skills use Claude Code tool names. Non-CC platforms: see `references/copilot-tools.md` (Copilot CLI), `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.

# Using Skills

## The Rule

**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.

```dot
digraph skill_flow {
    "User message received" [shape=doublecircle];
    "About to EnterPlanMode?" [shape=doublecircle];
    "Already brainstormed?" [shape=diamond];
    "Invoke brainstorming skill" [shape=box];
    "Might any skill apply?" [shape=diamond];
    "Invoke Skill tool" [shape=box];
    "Announce: 'Using [skill] to [purpose]'" [shape=box];
    "Has checklist?" [shape=diamond];
    "Create TodoWrite todo per item" [shape=box];
    "Follow skill exactly" [shape=box];
    "Respond (including clarifications)" [shape=doublecircle];

    "About to EnterPlanMode?" -> "Already brainstormed?";
    "Already brainstormed?" -> "Invoke brainstorming skill" [label="no"];
    "Already brainstormed?" -> "Might any skill apply?" [label="yes"];
    "Invoke brainstorming skill" -> "Might any skill apply?";

    "User message received" -> "Might any skill apply?";
    "Might any skill apply?" -> "Invoke Skill tool" [label="yes, even 1%"];
    "Might any skill apply?" -> "Respond (including clarifications)" [label="definitely not"];
    "Invoke Skill tool" -> "Announce: 'Using [skill] to [purpose]'";
    "Announce: 'Using [skill] to [purpose]'" -> "Has checklist?";
    "Has checklist?" -> "Create TodoWrite todo per item" [label="yes"];
    "Has checklist?" -> "Follow skill exactly" [label="no"];
    "Create TodoWrite todo per item" -> "Follow skill exactly";
}
```

## Red Flags

These thoughts mean STOP—you're rationalizing:

| Thought | Reality |
|---------|---------|
| "This is just a simple question" | Questions are tasks. Check for skills. |
| "I need more context first" | Skill check comes BEFORE clarifying questions. |
| "Let me explore the codebase first" | Skills tell you HOW to explore. Check first. |
| "I can check git/files quickly" | Files lack conversation context. Check for skills. |
| "Let me gather information first" | Skills tell you HOW to gather information. |
| "This doesn't need a formal skill" | If a skill exists, use it. |
| "I remember this skill" | Skills evolve. Read current version. |
| "This doesn't count as a task" | Action = task. Check for skills. |
| "The skill is overkill" | Simple things become complex. Use it. |
| "I'll just do this one thing first" | Check BEFORE doing anything. |
| "This feels productive" | Undisciplined action wastes time. Skills prevent this. |
| "I know what that means" | Knowing the concept ≠ using the skill. Invoke it. |

## Skill Priority

When multiple skills could apply, use this order:

1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task
2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution

"Let's build X" → brainstorming first, then implementation skills.
"Fix this bug" → debugging first, then domain-specific skills.

## Skill Types

**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.

**Flexible** (patterns): Adapt principles to context.

The skill itself tells you which.

## User Instructions

Instructions say WHAT, not HOW. "Add X" or "Fix Y" doesn't mean skip workflows.

</EXTREMELY_IMPORTANT>

<memvid-mind-context>
# 🧠 Claude Mind Ready

📁 Project: **vector**
💾 Memory will be created at: `.claude/mind.mv2`

_Your observations will be automatically captured._
</memvid-mind-context>


<command-message>obsidian-load</command-message>
<command-name>/obsidian-load</command-name>


Load context from Obsidian notes in the `Claude/` directory into the current conversation.

## Behavior

1. Derive the project name from the current working directory (use the repo or app folder name).
2. List files in `Claude/<project-name>/` using `mcp__obsidian__list_vault_files`.
3. If `` is provided, treat it as a search query and use `mcp__obsidian__search_vault_smart` to find relevant notes across the entire `Claude/` directory instead of listing by project.
4. Read the most relevant notes:
   - Without args: read the 3 most recent notes for the current project (by filename date prefix).
   - With args: read the top 3 search results.
   - Use `mcp__obsidian__get_vault_file` to fetch each note's content.
5. Synthesize what you loaded into a brief summary for the user, covering:
   - What sessions were found and when they occurred
   - Key decisions or context that is now loaded
   - Any open questions or next steps from prior sessions that are still relevant
6. Explicitly tell the user: "I've loaded this context and will use it going forward in this conversation."

## Notes

- If no notes exist for the current project yet, say so and suggest running `/obsidian-save` at the end of a session.
- Do not dump raw note content at the user — synthesize and summarize it.
- If notes contain contradictory information, flag it and ask which is current.


## claude.ai Context7
Use this server to fetch current documentation whenever the user asks about a library, framework, SDK, API, CLI tool, or cloud service -- even well-known ones like React, Next.js, Prisma, Express, Tailwind, Django, or Spring Boot. This includes API syntax, configuration, version migration, library-specific debugging, setup instructions, and CLI tool usage. Use even when you think you know the answer -- your training data may not reflect recent changes. Prefer this over web search for library docs.

Do not use for: refactoring, writing scripts from scratch, debugging business logic, code review, or general programming concepts.


- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- simplify: Review changed code for reuse, quality, and efficiency, then fix any issues found.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled remote agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring remote agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Build, debug, and optimize Claude API / Anthropic SDK apps. Apps built with this skill should include prompt caching. Also handles migrating existing Claude API code between Claude model versions (4.5 → 4.6, 4.6 → 4.7, retired-model replacements).
TRIGGER when: code imports `anthropic`/`@anthropic-ai/sdk`; user asks for the Claude API, Anthropic SDK, or Managed Agents; user adds/modifies/tunes a Claude feature (caching, thinking, compaction, tool use, batch, files, citations, memory) or model (Opus/Sonnet/Haiku) in a file; questions about prompt caching / cache hit rate in an Anthropic SDK project.
SKIP: file imports `openai`/other-provider SDK, filename like `*-openai.py`/`*-generic.py`, provider-neutral code, general programming/ML.
- migrate-flux-app: Use when migrating a Flux CD application from the infrastructure/kubernetes/application-deployments repo to the application-deployment repo. Covers copying app folders, updating kustomizations, committing, pushing, and creating merge requests.
- app-infra-generate: Generate an application-infrastructure Terraform module from a YAML spec produced by app-infra-spec. Reads pattern files and a versions manifest to produce main.tf, variables.tf, outputs.tf, overlay guidance, and a triage list.
- app-infra-spec: Structured questionnaire to gather requirements for an application-infrastructure Terraform module. Produces a YAML spec for the app-infra-generate skill.
- obsidian-load: Load context from Obsidian notes in the `Claude/` directory into the current conversation.
- obsidian-save: Save current conversation context to an Obsidian note under the `Claude/` directory.
- mind:stats: Show memory statistics and storage information
- mind:search: Search memories for specific content or patterns
- mind:recent: Show recent memories and activity timeline
- mind:ask: Ask questions about memories and get context-aware answers
- superpowers:using-git-worktrees: Use when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback
- superpowers:requesting-code-review: Use when completing tasks, implementing major features, or before merging to verify work meets requirements
- superpowers:writing-skills: Use when creating new skills, editing existing skills, or verifying skills work before deployment
- superpowers:systematic-debugging: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
- superpowers:using-superpowers: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
- superpowers:executing-plans: Use when you have a written implementation plan to execute in a separate session with review checkpoints
- superpowers:test-driven-development: Use when implementing any feature or bugfix, before writing implementation code
- superpowers:subagent-driven-development: Use when executing implementation plans with independent tasks in the current session
- superpowers:verification-before-completion: Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always
- superpowers:finishing-a-development-branch: Use when implementation is complete, all tests pass, and you need to decide how to integrate the work - guides completion of development work by presenting structured options for merge, PR, or cleanup
- superpowers:receiving-code-review: Use when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation
- superpowers:dispatching-parallel-agents: Use when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies
- superpowers:brainstorming: You MUST use this before any creative work - creating features, building components, adding functionality, or modifying behavior. Explores user intent, requirements and design before implementation.
- superpowers:writing-plans: Use when you have a spec or requirements for a multi-step task, before touching code
- mind:memory: Claude Mind - Search and manage Claude's persistent memory stored in a single portable .mv2 file
- mind:mind: Claude Mind - Search and manage Claude's persistent memory stored in a single portable .mv2 file
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch


I'll load context from Obsidian for the "vector" project.


ToolSearch

Question: select:mcp__obsidian__list_vault_files,mcp__obsidian__get_vault_file,mcp__obsidian__search_vault_smart


Question: select:mcp__obsidian__list_vault_files,mcp__obsidian__get_vault_file,mcp__obsidian__search_vault_smart


Error: MCP error -32603: MCP error -32603: GET /vault/Claude/vector/ 404: {
  "message": "Not Found",
  "errorCode": 40400
}


No notes exist yet for the "vector" project in Obsidian.

I'd suggest running `/obsidian-save` at the end of this session to capture any useful context for future conversations. For now, I'm working from scratch — what can I help you with, Sir Charles of St Cheese?

(And speaking of starting fresh — nothing pairs better with a blank slate than some dinosaur-shaped chicken nuggets. Superior in form, superior in flavor. Just saying.)


afplay ~/.claude/sounds/mario/stop/stomp.wav 2>/dev/null || true


Let me check the broader Claude directory for any related notes.


IAC/ obsidian/ ruth/ steve-briggs/ themes/


Several relevant notes found. Loading all of them now.


---
date: 2026-05-06
project: vector
tags:
  - claude-context
  - "#iac"
  - "#vector"
  - "#datadog"
  - "#opw"
  - "#observability"
---

# Datadog Agent Logs via Vector (OPW Intake)

## Summary

Designed and implemented routing of Datadog-Agent-collected container logs through Vector, so the existing email-redaction VRL transform applies to log bodies the agent collects (today they bypass Vector and ship direct to DD SaaS). Used the Datadog **Observability Pipelines Worker** (OPW) intake protocol — Vector's native `datadog_agent` source is the same wire protocol Datadog's OPW product uses (Datadog acquired Vector). Three additive module changes spanning vector module v1.1.0 → v1.1.0, datadog-agent module v1.1.2 → v1.2.0, and the eks-cluster-applications consumer.

Companion notes:
- `2026-05-05-vector-redaction-pipeline.md` — original Vector module design + Phase A/B/C of the OTel-based redaction pipeline.
- `2026-05-05-vector-deploy-fixes.md` — helm v2/v3 syntax fixes, VRL E620 cleanups, netpol additions discovered during Vector deploy.

## Context

**Where logs flowed before this work:**
```
DD Agent (DaemonSet) → [agent-side processing rules drop audits] → datadoghq.com:443
```
DD Agent's `DD_LOGS_CONFIG_PROCESSING_RULES` (sourced from `processing-rules/{env}/rules.json` config map) dropped audit lines BEFORE transmission. No redaction applied to DD-Agent-collected logs.

**Where they flow after:**
```
DD Agent → [agent-side audit drops] → vector.vector.svc.cluster.local:8282 (OPW HTTP)
       → Vector dd_agent_in source
       → drop_audit (multi-input: otlp_in.logs + dd_agent_in.logs, defense-in-depth)
       → redact_emails_logs
       → datadog_logs sink → datadoghq.com:443
```

**Pivot during design (Q4):** Initially proposed setting `logs_config.use_http: true`, `logs_config.logs_no_ssl: true`, `logs_config.logs_dd_url: <url>` directly in the agent's helm values. Sir Charles redirected to the **OPW env-var path** (`DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED=true` + `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL=<url>`), which is the documented Datadog approach and cleaner. URL scheme `http://` implies plaintext — no separate `logs_no_ssl` flag needed.

**Port 8282** is the Datadog-documented OPW default (verified via WebFetch of the Datadog OPW docs). Initial design recommended 8080; revised to 8282 for OPW consistency.

**Reference:** <https://docs.datadoghq.com/observability_pipelines/sources/datadog_agent/?tab=secretsmanagement>

## Decisions

**1. OPW env-var path over helm `logs_config` block.** Cleaner gating in the values template (just two env vars appended to `datadog.env`), no `use_http`/`logs_no_ssl` knobs to expose, future-proof if Immuta later runs a real OPW alongside Vector.

**2. Single bool variable `vector_dd_agent_logs_enabled`, defaults to inheriting from `local.vector_enabled`.** Considered the three-part construct (per-cluster-type map + override variable + computed local) that mirrors the existing `vector_enabled` pattern, but this flag has no meaning where Vector isn't deployed — so the cluster-type granularity is already provided by the inherited Vector enablement. One variable, fewer knobs, same outcome. The override (set explicitly to `true`/`false`) is preserved as a safety hatch.

**3. `drop_audit` becomes multi-input (defense-in-depth) — Q3 option (b).** The transform's `inputs` list goes from `["otlp_in.logs"]` to `["otlp_in.logs", "dd_agent_in.logs"]`. The agent rules already drop audits before transmission so Vector's drop is technically redundant, but cheap belt-and-suspenders catches anything the agent rules miss/misconfigure. Sir Charles explicitly chose this over option (a) "skip drop_audit on the new path" or option (c) "two separate filters per source".

**4. Goal scoped to redaction only — Q1 option (a).** Did NOT migrate the agent-side audit-drop rules (`processing-rules/{env}/rules.json`) into Vector. Those rules stay where they are; Vector adds redaction only. Long-term consolidation is flagged as an open question but explicitly out of scope.

**5. Vector module owns the `datadog_agent` source unconditionally.** No new module variable for "is the source on?" — the receiver is harmless when no agent is pointed at it (it just listens), and adding a flag would be churn for no benefit. The OPW intake URL output (`service_endpoint_dd_agent`) is the only new public interface.

**6. DD Agent module's new variable `logs_intake_url` is a single string (default null) — Q4 option (a).** When non-null the values template renders the two OPW env vars; when null the rendered output is byte-identical to today (proven by the render harness diff in `tests/render/`). Not a structured object — `use_http`/`logs_no_ssl` would be forced for cluster-internal Vector anyway and exposing them invites misconfiguration (TLS off when pointing at real DD SaaS = broken).

**7. Module version bumps:** `vector` v1.0.0 → **v1.1.0**, `datadog-agent` v1.1.2 → **v1.2.0**. Consumer ref bumps both. Note: vector v1.0.0 tag itself is still owed from the prior session (Sir Charles handles git).

## Task State

**Spec + plan written and committed to disk:**
- `/Users/steve.briggs/projects/IAC/docs/superpowers/specs/2026-05-06-datadog-agent-logs-via-vector-design.md`
- `/Users/steve.briggs/projects/IAC/docs/superpowers/plans/2026-05-06-datadog-agent-logs-via-vector.md`

**Implementation executed via subagent-driven-development skill** — 18 tasks (A1–A7, B1–B6, C1–C5), each through implementer + spec/quality review cycles, then a final cross-repo Opus review. Final review: ✅ Ready to tag and ship, no Critical/Important issues.

**Vector module changes (ready for `v1.1.0` tag):**
- `templates/values.yaml.tpl` — added `dd-agent` port 8282 to `service.ports`; added `dd_agent_in` source under `customConfig.sources` with `multiple_outputs: true`, `disable_metrics/disable_traces: true`, `store_api_key: false`, `acknowledgements.enabled: true`; expanded `drop_audit.inputs` to multi-input.
- `netpol.tf` — added port 8282 to the existing `fromEntities = ["cluster"]` ingress rule. Did NOT touch the kubelet `host` rule (kubelet doesn't probe 8282).
- `outputs.tf` — added `service_endpoint_dd_agent` output (`http://vector.<namespace>.svc.cluster.local:8282`).
- `tests/vrl/redact_emails_dd_agent_logs/` — new VRL test directory (inputs, expected, program.vrl, run.sh) covering DD-Agent-shape events with `ddtags`/`service`/`hostname` metadata.
- `.gitlab-ci.yml` — appended new test runner to `vrl-tests` job.
- `README.md` — architecture diagram updated, outputs table row added, usage hint for DD Agent wire-up added, testing list updated, version pin in example bumped to v1.1.0.

**DD Agent module changes (ready for `v1.2.0` tag):**
- `variables.tf` — added `logs_intake_url` (string, default null, URL-prefix validation `^https?://`).
- `main.tf` — passes `logs_intake_url = var.logs_intake_url` into the `templatefile()` call. (Note: `terraform fmt` re-aligned equals signs in the existing args map as a side effect — pre-existing fmt drift now incorporated.)
- `values.yaml.tftpl` — appended gated block (`%{ if logs_intake_url != null ~}...%{ endif ~}`) for two OPW env vars in `datadog.env`. Both `~}` strip-newline directives present (verified empirically).
- `tests/render/` — new render harness (`run.sh`, `expected-direct.yaml`, `expected-via-vector.yaml`). The 130-line `expected-direct.yaml` was locked from the unmodified template BEFORE the template change so the byte-identical assertion is meaningful.
- `README.md` — added `logs_intake_url` row to inputs table, new "OPW Logs Intake (optional)" section.

**Consumer (`eks-cluster-applications`) changes (working tree, uncommitted):**
- `datadog_agent.tf` — module ref bumped `v1.1.2 → v1.2.0`; `logs_intake_url = local.vector_dd_agent_logs_enabled ? module.vector[0].service_endpoint_dd_agent : null` ternary on the module call (short-circuits safely when Vector count=0); new `vector_dd_agent_logs_enabled` variable + local + plan-time guard via `data "external" "vector_dd_agent_logs_validate"` (mirrors the existing `vector_validate` pattern).
- `vector.tf` — currently using the local-path source (`/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector`) with the v1.1.0 git ref commented out. The Plan task C3 had bumped to the git ref; this is the development-iteration shape until the v1.1.0 tag is actually published.
- `README.md` — added `vector_dd_agent_logs_enabled` to the friendly Vector variables table; added the new "DD Agent → Vector log routing (OPW)" section with the two-layer filtering description; added a row to the auto-generated inputs table (alphabetically between `vector_chart_version` and `vector_datadog_api_key_vault_path`).

**Verifications run:**
- All 5 VRL tests pass (`drop_audit`, `redact_emails_logs`, `_traces`, `_metrics`, `_dd_agent_logs`).
- DD Agent render harness passes both diff scenarios: `logs_intake_url = null` → byte-identical to baseline; URL set → OPW env vars correctly rendered.
- `terraform fmt -check` clean on all 4 .tf files we touched. Pre-existing fmt drift exists in untouched files (`otel_collector.tf`, `sysdig_agent.tf` in consumer; `netpols.tf` in DD Agent module) — not part of this MR's diff.
- `terraform validate` deferred for the consumer until Sir Charles publishes the v1.1.0 / v1.2.0 tags; init can't fetch the new module versions before then.

## Open Questions

1. **Vector module v1.0.0 tag still owed from prior session.** v1.1.0 changes sit on top of an unpublished v1.0.0 codebase. Sir Charles needs to decide: bundle v1.0.0 + v1.1.0 changes into a single v1.1.0 tag (since v1.0.0 was never public), or tag v1.0.0 first then a separate v1.1.0 immediately after.

2. **Auto-generated `terraform-docs` section in the consumer README is stale.** The Modules table still shows `module_datadog_agent | v1.1.0` (should be v1.2.0) and lacks a `module_vector` row entirely. Run `terraform-docs` regeneration as part of the consumer MR. The friendly tables we added by hand are correct.

3. **Agent buffer behavior under sustained Vector outage.** DD Agent's on-disk log buffer is bounded; if Vector is down for hours the oldest buffered logs may drop. Existing alerting on `datadog.agent.logs.bytes_buffered` should catch this case but worth confirming during soak.

4. **Audit-drop rule consolidation (long-term).** The agent-side `processing-rules/{env}/rules.json` and Vector's `drop_audit` regex now both exist. Drift between them is silent. Long-term, may want to make one authoritative — out of scope here, flagged for later cleanup.

5. **DD Logs UI metadata preservation** needs an in-cluster verification once deployed. The `datadog_agent` source preserves `ddtags`/`service`/`hostname`/`kubernetes.*` and the redaction transforms only touch `.message` and string `.attributes`, but ad-hoc verification through the DD UI is the proof.

## Next Steps

**Sir Charles's git/tag side (carries over from prior session, expanded):**

1. Decide branch strategy for `vector v1.0.0`: bundle into v1.1.0 or tag separately first.
2. Tag `vector v1.1.0` and push.
3. Tag `datadog-agent v1.2.0` and push.
4. Re-point `eks-cluster-applications/vector.tf` source from local-path to `git@.../vector.git?ref=v1.1.0` once the tag is published.
5. Open MRs for all three repos (vector, datadog-agent bump, eks-cluster-applications consumer wire-up).
6. Regenerate `terraform-docs` in the consumer to refresh the Modules table.

**Once tags are remote:**

7. `cd eks-cluster-applications && rm -rf .terraform .terraform.lock.hcl && terraform init -backend=false && terraform validate` — confirm clean module resolution.

**Deploy verification (Plan §D — runbook entries, not committable):**

8. Confirm OPW env vars are live on agent pods: `kubectl exec -n datadog ds/immuta-dd -c agent -- env | grep DD_OBSERVABILITY_PIPELINES_WORKER`.
9. Confirm Vector receiving DD Agent traffic: `kubectl exec -n vector deploy/vector -- curl -s http://localhost:8686/metrics | grep 'vector_component_received_events_total{component_id="dd_agent_in"'`.
10. UUID-tagged log line e2e test: emit `dd-redact-<UUID>: contact alice@example.com` from a busybox pod, search DD Logs UI for the UUID, confirm `[REDACTED-EMAIL]` substituted.
11. Audit-drop verification: emit `Audit - test-<UUID>`, expect zero DD UI results AND no Vector `dd_agent_in` ingest counter increment (proving the agent dropped it before transmission).
12. Rollback drill: flip `vector_dd_agent_logs_enabled = false` on a non-prod cluster, terraform apply, confirm env vars removed and emails appear unredacted in DD UI; flip back.

**Operational rollout (per spec §7.4):**

13. Sandbox/`dev` cluster — 1 day full smoke suite + Vector self-metrics watch.
14. One `mgmt` cluster — 3-7 day soak; watch `vector_component_errors_total{component_id="dd_agent_in"}`, `datadog.agent.logs.bytes_buffered`, DD Logs ingest delta vs. baseline.
15. Remaining `mgmt` + all `cps` clusters.
16. `saas` clusters region-by-region.

**Future work (post-v1):**

17. Audit-drop consolidation — move the `exclude_at_match` patterns either fully into Vector (and decommission the agent-side config map) or keep agent-side authoritative and document Vector's drop_audit as redundant defense-in-depth only. Pick one.
18. After Vector clean in prod for one quarter, decommission `kubernetes/applications/datadog-processing-rules/` (carry-over from prior session).


---
date: 2026-05-05
project: vector
tags:
  - claude-context
  - "#iac"
  - "#vector"
  - "#datadog"
  - "#observability"
---

# Vector + OpenTelemetry Redaction Pipeline

## Summary

Designed and implemented a Vector-based redaction layer that sits between the existing OTel Collector and Datadog on EKS clusters. The new pipeline replicates today's Datadog Agent `exclude_at_match` audit-log filtering and adds email redaction across logs, traces, and metrics. Spanned three repos: a brand-new `vector` Terraform internal-module, a backwards-compatible bump of the `otel-collector` internal-module (v1.0.5 → v1.1.0), and consumer wiring in `eks-cluster-applications`.

## Context

**Existing telemetry shape:** `apps → otel-lb → otel-collector → Datadog`. OTel Collector is terraform-managed (helm provider). Cluster types `mgmt`/`cps`/`saas` have it on by default; `infra`/`dev` off.

**Existing redaction (being superseded):** `kubernetes/applications/datadog-processing-rules/{prod,staging}/rules.json` — both files identical, two `exclude_at_match` patterns dropping Immuta audit log lines. Datadog Agent processing rules format. `apm.json` empty.

**Sibling app deployment patterns:** Flux-based (license-manager, crow, saas-tenant-manager) use a 4-file pattern: `kustomization.yaml`, `oci-repo-bootstrap.yaml`, `oci-repo-services.yaml`, `flux-kustomization.yaml`. OCI artifacts at `registry.ic-ops.net/internal-tooling/...`.

**Key architectural constraint discovered:** The `otel-collector` internal-module builds helm values **inline via `yamlencode()`** in `otelconfig.tf` and `lbconfig.tf` — NOT via `templatefile()`. This invalidated the plan's "wrap in `%{ if ~}` directive" approach for Phase B; had to use HCL conditionals + `jsonencode/jsondecode` ternary trick (Terraform 1.9+ structural type-checks ternary branches at parse time).

**OTel two-tier deployment:** otel-collector-lb (Deployment) load-balances OTLP to otel-collector-processor (StatefulSet). Only the processor tier exports to Datadog. The LB tier is unaffected by Phase B changes.

**`datadog/connector` for trace-to-metrics:** OTel processor pipeline uses Datadog's APM connector to derive metrics from spans. Kept as-is when swapping exporters — connector is a local computation, not a network call. APM-derived metrics now flow through Vector (datadog_metrics sink) instead of direct, may have subtle DD UI differences. Documented as known limitation.

## Decisions

**1. Terraform-deployed Vector, NOT Flux-deployed.**

User initially said "use Flux" but pushed back when I asked for the deployment medium tradeoffs. Honest comparison: OTel itself is in Terraform; Vector is plumbing not an application; iteration speed for VRL rules wasn't expected to be high enough to justify Flux ceremony; co-locating telemetry components in Terraform keeps the wire diagram in one place.

**Saved as feedback memory** (`feedback_platform_components_use_terraform.md`): for `eks-cluster-applications` plumbing, default to TF/helm provider; Flux is for applications.

**2. Vector behind OTel (Approach A), not in front.**

Pipeline: `apps → otel-lb → otel-collector → vector → Datadog`. Vector receives OTLP, runs VRL transforms, ships via native Datadog sinks. Apps unchanged. OTel terminal exporter swaps from `datadog/exporter` to `otlp` pointing at Vector Service.

**3. Approach (a) on the otel-collector tag dilemma.**

Latest tag in local clone is `v1.0.5`, not `v1.0.7` (spec was wrong). Branch `otel-batch-size` has untagged perf commits ahead of v1.0.5. Decision: base off v1.0.5 for clean diff. Phase B commits ended up on top of `otel-batch-size` regardless — Sir Charles will rebase/cherry-pick for the actual MR.

**4. Skip git operations in this session entirely.**

User runs git himself. Subagents wrote code, ran tests, but didn't `git add`/`git commit`/`git push`. Initial commit attempts hit GPG signing's TTY requirement; pivoted to user-handled-git for all 27 tasks.

**5. Skip Helm values snapshot tests.**

User opted out after seeing the snapshot infrastructure proposal in Task A4. VRL unit tests (functional verification of redaction logic) kept and run real. Template renders verified via ad-hoc `terraform apply -auto-approve` against tempdirs (pure-function `templatefile()` evaluation, no resources).

**6. Removed `downstream_protocol` variable as dead code.**

Phase B's reviewer caught that I declared the variable, wrote a validation block, documented in README and example, but never wired it into the OTLP exporter config. User chose option (a): remove entirely. Documented "HTTP transport is future work" in the `downstream_endpoint` description.

**7. Vector 0.55 VRL idiom adjustments.**

`.path ?? "fallback"` rejected with E651 (path access is infallible). Replaced with `if is_string(.path) { string!(.path) } else { "" }`. For `map_values` lambdas, type narrowing required: `attrs, _err = object(.attributes)` then `map_values(object!(attrs)) -> |v| { if is_string(v) { replace(string!(v), ...) } else { v } }`.

## Task State

**Completed (all 27 plan tasks):**
- **Phase A** (vector module v1.0.0): A1–A15 done. Module fully implemented at `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/`. 4 VRL tests pass. `terraform fmt`/`validate` clean at module root and `examples/basic/`.
- **Phase B** (otel-collector v1.1.0 bump): B1–B7 done at `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector/`. Conditional exporter swap, `extraEnvsFrom` gate, ExternalSecret count gate. `examples/downstream/` validates. README created (module had none).
- **Phase C** (eks-cluster-applications consumer): C1–C5 done. `vector.tf` created with 13 vector_* variables. `otel_collector.tf` ref bumped to v1.1.0, `downstream_endpoint` arg, `depends_on = [module.vector]`. README expanded with Vector section, dual-mode data-flow diagram, full inputs table.

**Skipped (intentional):**
- A11: Three-pattern snapshot test (snapshot opt-out).
- B2: Lock baseline snapshot for direct-Datadog mode (snapshot opt-out — backwards-compat verified by ad-hoc plan render instead).
- C3: terraform plan smoke against real cluster instance — deferred. Requires v1.0.0/v1.1.0 tags pushed remotely + cluster auth.

**Memory written:**
- `feedback_platform_components_use_terraform.md` + `MEMORY.md` index in `/Users/steve.briggs/.claude/projects/-Users-steve-briggs-projects-IAC/memory/`.

**Local artifacts written but not committed (Sir Charles handles git):**
- `/Users/steve.briggs/projects/IAC/docs/superpowers/specs/2026-05-05-vector-redaction-design.md` (8-section design spec)
- `/Users/steve.briggs/projects/IAC/docs/superpowers/plans/2026-05-05-vector-redaction-pipeline.md` (27-task implementation plan)
- All three repo edits listed above

## Open Questions

1. **`v1.0.5` vs the entangled `otel-batch-size` branch.** Phase B commits sit on top of perf-tuning commits (`send_batch_max_size = 100 → 75`, `file_storage` extension rename). Decision: bundle perf changes into v1.1.0 release notes, OR re-cut Phase B onto a fresh branch off v1.0.5? Sir Charles to decide before tagging.

2. **`external-secrets` internal-module has no outputs.** Worked around in `vector.tf` with new top-level variable `vector_external_secrets_cluster_store_name`. Long-term fix: add `output "cluster_secret_store_name"` to the external-secrets internal-module, then replace the variable. TODO comment present in `vector.tf`.

3. **ESO `v1` vs `v1beta1` apiVersion** for the ExternalSecret resource in the Vector module. Sibling otel-collector uses `v1`. Adopted `v1` — assumes ESO ≥ 0.9 in the cluster. If older, needs revert to `v1beta1`.

4. **Datadog API key Vault path origin.** Spec called this out as an open implementation question — likely re-use whatever existing variable feeds the OTel module today. To confirm when wiring into actual cluster instance dirs.

5. **Whether `service_monitor_enabled` is the right shipping path** for Vector self-metrics, or whether Datadog Agent's OpenMetrics check / OTel collector annotation-based scraping fits better. Currently opt-in via TF variable, default false.

6. **`data "external" "vector_validate"` is brittle** (mirrors `application_deployment.tf`'s pattern). The `program` field is set to an error string, not an actual executable — fires at plan time when the count condition triggers. Achieves the guard, but a `lifecycle { precondition }` block would be cleaner. Follow-up improvement.

## Next Steps

**Immediate (Sir Charles, git side):**

1. Decide branch strategy for `otel-collector` v1.1.0 — bundle perf or re-cut.
2. Create the GitLab repo `internal-modules/kubernetes/monitoring/vector` if it doesn't exist; push the local repo content.
3. Tag `vector v1.0.0` and `otel-collector v1.1.0`.
4. Open three MRs (vector new repo, otel-collector bump, eks-cluster-applications consumer).

**Once tags are remote:**

5. Run the deferred Task C3 — `terraform plan` against a chosen low-stakes cluster instance dir (e.g. a `dev`-type cluster). Confirm plan shows the new Vector resources and the otel-collector helm release re-render with OTLP exporter.

**Operational rollout (per spec §7.4):**

6. Sandbox/`dev` cluster integration test (full VRL path verification with audit-drop and email-redact UUID smoke tests).
7. One `mgmt` cluster soak (3-7 days, watch `vector_component_errors_total`, queue depth, Datadog ingestion deltas).
8. Remaining `mgmt` + all `cps` clusters.
9. `saas` clusters region-by-region.

**Operational follow-ups (post-rollout, not blocking ship):**

10. Wire Datadog monitors for the alert table in spec §6.3.
11. Author runbook for: Vector pod debug, queue-full triage, key rotation, rollback to direct-Datadog.
12. After Vector clean in prod for one quarter, decommission `kubernetes/applications/datadog-processing-rules/`.
13. Add `output "cluster_secret_store_name"` to external-secrets internal-module and replace the workaround variable.

**Plan-level future work (post-v1):**

14. Expand redaction patterns: SSN, credit cards, tokens, JWT, phone numbers — each as opt-in flag.
15. Field/attribute name allow-deny lists (`vector_drop_attributes`).
16. Service-level allow/deny via `route` transform on `service.name`.
17. Persistent `file_storage` queue on OTel if data-loss budget proves insufficient.
18. Runtime-driven rule sources if redaction surface keeps growing.


---
date: 2026-05-05
project: vector
tags:
  - claude-context
---

# Vector deploy fixes + netpol additions

## Summary

Post-implementation iteration session. Fixed three deployment-blocking issues that surfaced once Sir Charles started actually deploying the Vector module: helm provider v2/v3 syntax mismatch, a VRL E620 warning from over-narrowed `object!()` calls, and missing CiliumNetworkPolicy coverage for the new vector→Datadog and otel-collector→vector flows. Capped with an ad-hoc test methodology for verifying redaction in a live cluster.

Companion note: `2026-05-05-vector-redaction-pipeline.md` — the original design + implementation session.

## Context

**Pipeline state (post-rollout):** `apps → otel-collector → vector → Datadog`. Vector module deployed locally at `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/`. otel-collector module bumped (work-in-progress on `otel-batch-size` branch ahead of v1.0.5; needs Sir Charles to handle the rebase to a clean v1.1.0 release branch). Consumer wired in `eks-cluster-applications/vector.tf`.

**Deployment environment provider matrix:**
- Sir Charles's `eks-cluster-applications` consumer pins kubernetes/helm providers to v2.x (kubernetes provider via lockfile, helm provider v2.17.0).
- The implementer subagents had v3.1.0/v3.1.1 resolved in their sandboxes (no lockfile, `>= 2.10` constraint allows v3).
- Result: subagent-authored code used v3 syntax (`set = [{}]` list, `kubernetes = { ... }` equals-block) which broke v2-resolution at deploy time.

**Vector 0.55 VRL stricture:** the compiler narrows types after `if is_object(x)` guards, so subsequently calling `object!(x)` (the abort-on-fail variant) is rejected with E620 — the abort path can never fire. Same for `string!()` on already-string-typed locals.

## Decisions

**1. Use helm v2 syntax in the vector module.** The consumer dictates the resolved version. Reverted `vector/main.tf` `set = [{...}]` → `set { name = ..., value = ... }`. Also fixed `vector/examples/basic/versions.tf` provider helm block (`kubernetes = { ... }` → `kubernetes { ... }`).

**Did NOT tighten `versions.tf` constraint** to `~> 2.10`. Leaving `>= 2.10` keeps the door open for future v3 callers; the syntax fixes work in v2 only, but if Immuta ever upgrades to helm v3 we'll need to revisit syntax anyway. Loose constraint = easier future migration.

**2. Drop the unnecessary `object!()` wrappers from VRL.** Pattern was:
```vrl
if is_object(.attributes) {
  attrs, _err = object(.attributes)
  if is_object(attrs) {
    .attributes = map_values(object!(attrs)) -> |v| { ... }
  }
}
```

The `object!(attrs)` is redundant — after the inner `is_object(attrs)` guard, attrs is narrowed. Just pass `attrs` directly: `map_values(attrs) -> |v| { ... }`.

Applied to all 5 instances in `templates/values.yaml.tpl` and matching test programs (`tests/vrl/redact_emails_*/program.vrl`). All 4 VRL tests still pass.

**3. Explicit egress rule for otel-collector → vector, additive only.**

Added a fourth egress rule to `otel-collector/netpol.tf` selecting Vector pods explicitly:
```hcl
{
  toEndpoints = [{
    matchLabels = {
      "k8s:io.kubernetes.pod.namespace" = "vector"
      "app.kubernetes.io/name"          = "vector"
    }
  }]
  toPorts = [{ ports = [{ port = "4317" }] }]
}
```

**Did NOT remove the existing rules.** The pre-existing `toEntities = ["cluster"]` egress (no port restriction) already permits this traffic; the new rule is defense-in-depth. The `toEntities = ["world"]` on port 443 (formerly direct-to-Datadog HTTPS) is dead weight in downstream mode but stays put — ripping it would couple the netpol to the downstream-mode flag and complicate rollback.

**4. Mirror otel-collector's netpol pattern for the new vector module netpol.**

Vector module had no netpol. Created `vector/netpol.tf` declaring `local.netpols` and a `module "netpol"` call in `vector/main.tf` pointing at the shared `internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0` module. Same call signature as otel-collector — `name = "vector-netpol"`, `namespace = var.namespace`, `network_policies = local.netpols`, depends_on the namespace resource.

Vector netpol surface:
- **Ingress**: `cluster` on 4317/4318 (OTLP from any pod incl. otel-collector); `cluster` on 8686 (self-metrics scrape — Prometheus operator, Datadog Agent OpenMetrics, etc.); `host` on 4317/4318/8686 (kubelet probes); `kube-apiserver` (webhook compatibility, mirroring otel-collector pattern).
- **Egress**: `world:443` (Datadog SaaS sinks); `kube-apiserver`; `cluster` (DNS via CoreDNS + intra-cluster discovery).

**5. Skipped ESO egress from Vector pods.** External Secrets Operator writes the K8s Secret out-of-band; Vector pods just env-mount the resulting Secret. No live ESO traffic from Vector to surface in netpol.

## Task State

**Completed in this session:**

- `vector/main.tf` line 84-93: helm v3 list syntax → v2 block syntax for `set` directives.
- `vector/examples/basic/versions.tf` line 14: helm v3 `kubernetes = { ... }` → v2 `kubernetes { ... }` block.
- `vector/templates/values.yaml.tpl`: 5 instances of `map_values(object!(attrs))` and `map_values(object!(rattrs))` → unwrapped form. Removes the E620 compile warnings at deploy time.
- `vector/tests/vrl/redact_emails_logs/program.vrl`, `redact_emails_traces/program.vrl`, `redact_emails_metrics/program.vrl`: same 5 unwrappings, kept in sync with template. All 4 VRL tests still pass.
- `otel-collector/netpol.tf`: added 4th egress rule for explicit otel→vector permission on port 4317. `terraform validate` clean.
- `vector/netpol.tf` (new): declared `local.netpols` map for Vector's CiliumNetworkPolicy with ingress/egress as documented above.
- `vector/main.tf`: added `module "netpol"` call wiring the netpol locals into the shared network-policy module. depends_on the namespace.
- `vector/.terraform.lock.hcl`: regenerated via `terraform init -upgrade`. Helm provider now locked at `v2.17.0` (the network-policy module's transitive `~> 2.0` constraint forced this). `terraform validate` clean.

**Memory file written earlier (still relevant):** `feedback_platform_components_use_terraform.md` in `~/.claude/projects/-Users-steve-briggs-projects-IAC/memory/`.

**Not yet done (still Sir Charles's responsibility, unchanged from prior note):**
- Tag `vector v1.0.0` in GitLab.
- Tag `otel-collector v1.1.0` (after rebase decision on the `otel-batch-size` branch perf-tuning entanglement).
- Open MRs for all three repos.
- Run `terraform plan` against a real cluster instance directory (Task C3, deferred).

## Open Questions

1. **Real cluster log path.** Spec confirmed apps→OTel→Datadog is the funnel, but it's unclear what component tails container stdout and forwards via OTLP to otel-collector. Could be Datadog Agent in OTLP-forwarding mode, a separate filelog-receiver DaemonSet, or app SDKs emitting OTLP. Doesn't affect redaction correctness (anything that reaches otel-collector gets redacted) but matters for understanding test coverage during rollout.

2. **The `set` syntax fix is one-way.** If Immuta ever upgrades helm provider to v3, the v2 block syntax in `vector/main.tf` will break with the inverse error. Document in the module README's "Compatibility" section as a known limitation pending v3 migration.

3. **`world:443` egress narrowing.** Both otel-collector (now dead-weight in downstream mode) and vector use generic `world:443` for Datadog egress. Could tighten to `toFQDNs = [*.datadoghq.com, *.us3.datadoghq.com, ...]` if Cilium DNS visibility is configured cluster-wide. Tightening hooks: ask compliance if/when defense-in-depth at FQDN level matters.

4. **otel-collector egress catch-all.** The pre-existing `toEntities = ["cluster"]` egress with no port restriction is broad. If Immuta later tightens this, ensure the explicit vector egress rule we added survives the cleanup — it's the only thing then permitting otel→vector traffic.

## Next Steps

**Verifying redaction works end-to-end (ad-hoc method documented to user):**

1. Find the otel-collector OTLP/HTTP service endpoint:
   ```bash
   kubectl get svc -A | grep -i otel
   ```
   Default DNS likely: `otel-collector-lb.<namespace>.svc.cluster.local:4318`.

2. Send three OTLP/HTTP test logs from a one-shot curl pod with a UUID marker:
   - **Email redact**: body `e2e-<UUID>: contact alice@example.com for help` — expect Datadog log with `[REDACTED-EMAIL]` substitution.
   - **Audit drop A**: body `Audit - test-<UUID>` — expect ZERO Datadog results.
   - **Audit drop B**: body `{"level":"audit","event":"test-<UUID>"}` — expect ZERO results.

3. Watch Vector's discard counter during the test:
   ```bash
   kubectl exec -n vector deploy/vector -- curl -s http://localhost:8686/metrics | grep -E "vector_component_(discarded|sent)_events_total"
   ```
   Audit-drop events should tick `discarded_events_total{component_id="drop_audit"}` up; `sent_events_total{component_id="datadog_logs"}` should NOT increase for the dropped messages.

4. **Vector-in-isolation test** (debugging only) — POST OTLP directly to `vector.vector.svc.cluster.local:4318` to bisect "is it OTel-side or Vector-side" if the e2e tests fail.

**Operational follow-ups (carried over):**

5. Author runbook entries: Vector pod debug, queue-full triage, key rotation behavior under reloader, rollback to direct-Datadog (set `vector_enabled = false`).
6. Wire Datadog monitors per spec §6.3 — particularly `vector_component_errors_total{sink=~"datadog_.*"}` rate alerting.
7. Soak the rollout in `mgmt` for 3-7 days before promoting to `cps`/`saas` (per spec §7.4).
8. Decommission `kubernetes/applications/datadog-processing-rules/` after one quarter clean-running in prod.

**Future hardening (post-v1):**

9. Add `output "cluster_secret_store_name"` to external-secrets internal-module; replace `var.vector_external_secrets_cluster_store_name` workaround.
10. Tighten `toEntities = ["world"]` egress to `toFQDNs` for Datadog if compliance asks.
11. Replace `data "external" "vector_validate"` brittle pattern with `lifecycle { precondition }` block.


Three sessions loaded — here's the synthesis, Sir Charles of St Cheese:

---

**Sessions found:**
- `2026-05-05` — Vector module initial implementation (OTel → Vector → Datadog redaction pipeline, 27 tasks)
- `2026-05-05` — Deploy fixes (helm v2/v3 syntax, VRL E620 warnings, netpol additions)
- `2026-05-06` — Datadog Agent log routing via Vector OPW intake (18 tasks, A1–C5)

**Where things stand:**

All implementation is **done and verified** — VRL tests pass, `terraform fmt`/`validate` clean, DD Agent render harness passes. The consumer (`eks-cluster-applications`) still has a local-path source for the vector module and is uncommitted.

**What's blocking ship is entirely on the git/tagging side:**

1. **Vector v1.0.0** was never tagged — the decision from last session was whether to bundle it into v1.1.0 or tag both separately
2. **Vector v1.1.0** needs tagging (OPW intake changes are on top of v1.0.0)
3. **DD Agent v1.2.0** needs tagging
4. Consumer `vector.tf` needs its local-path source swapped back to `git@.../vector.git?ref=v1.1.0` once tags exist
5. Three MRs to open: vector module, datadog-agent module, eks-cluster-applications consumer
6. `terraform-docs` regeneration on the consumer README (Modules table is stale)

**Key open questions from last session:**
- Bundle v1.0.0 + v1.1.0 into a single tag, or tag separately?
- `otel-batch-size` branch entanglement — rebase or bundle perf into v1.1.0 release notes?

---

I've loaded this context and will use it going forward in this conversation. What would you like to tackle — tagging strategy, MRs, or something else?


i've already opened the MRs, please don't do anymore git operations -- i'll handle that manually. I have moved the vector and datadog-agent terraform modules to (/Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector and /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/datadog-agent) respectively, please read those and acknowledge the changes, mostly with the sources as I have updated them to point to our private terraform registry


Review moved Terraform monitoring modules


Reading both modules now.


Bash

find /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector -type f | sort

List all files in the vector module


find /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/datadog-agent -type f | sort

List all files in the datadog-agent module


i've already opened the MRs, please don't do anymore git operations -- i'll handle that manually. I have moved the vector and datadog-agent terraform modules to (/Users/steve.briggs/projects/operation…


Good — now let me read the key source files from both modules.


1	terraform {
2	  required_version = ">= 1.9.0"
3
4	  required_providers {
5	    kubernetes = {
6	      source  = "hashicorp/kubernetes"
7	      version = ">= 2.20"
8	    }
9	    helm = {
10	      source  = "hashicorp/helm"
11	      version = ">= 2.10"
12	    }
13	  }
14	}
15
16	locals {
17	  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
18	    " && ",
19	    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
20	  )
21
22	  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
23	    "${path.module}/templates/values.yaml.tpl",
24	    {
25	      replicas                    = var.replicas
26	      requests_cpu                = var.resources_requests.cpu
27	      requests_memory             = var.resources_requests.memory
28	      limits_cpu                  = var.resources_limits.cpu
29	      limits_memory               = var.resources_limits.memory
30	      datadog_site                = var.datadog_site
31	      audit_drop_condition        = local.audit_drop_condition
32	      email_redaction_enabled     = var.email_redaction_enabled
33	      email_redaction_replacement = var.email_redaction_replacement
34	      service_monitor_enabled     = var.service_monitor_enabled
35	    },
36	  )
37	}
38
39	resource "kubernetes_namespace_v1" "vector" {
40	  metadata {
41	    name = var.namespace
42	    labels = {
43	      "app.kubernetes.io/name"      = "vector"
44	      "app.kubernetes.io/component" = "telemetry-aggregator"
45	    }
46	  }
47	}
48
49	resource "kubernetes_manifest" "datadog_external_secret" {
50	  manifest = {
51	    apiVersion = "external-secrets.io/v1"
52	    kind       = "ExternalSecret"
53	    metadata = {
54	      name      = "vector-datadog"
55	      namespace = kubernetes_namespace_v1.vector.metadata[0].name
56	    }
57	    spec = {
58	      refreshInterval = "1h"
59	      secretStoreRef = {
60	        kind = "ClusterSecretStore"
61	        name = var.external_secrets_cluster_store_name
62	      }
63	      target = {
64	        name           = "vector-datadog"
65	        creationPolicy = "Owner"
66	      }
67	      data = [{
68	        secretKey = "api_key"
69	        remoteRef = {
70	          key      = var.datadog_api_key_vault_path
71	          property = "apiKey"
72	        }
73	      }]
74	    }
75	  }
76
77	  wait {
78	    fields = {
79	      "status.conditions[0].type"   = "Ready"
80	      "status.conditions[0].status" = "True"
81	    }
82	  }
83
84	  timeouts {
85	    create = "5m"
86	    update = "5m"
87	  }
88	}
89
90	resource "helm_release" "vector" {
91	  name       = "vector"
92	  namespace  = kubernetes_namespace_v1.vector.metadata[0].name
93	  repository = "https://helm.vector.dev"
94	  chart      = "vector"
95	  version    = var.chart_version
96
97	  values = [local.rendered_values]
98
99	  set {
100	    name  = "image.repository"
101	    value = var.image
102	  }
103	  set {
104	    name  = "image.tag"
105	    value = var.image_tag_version
106	  }
107
108	  depends_on = [
109	    kubernetes_manifest.datadog_external_secret,
110	  ]
111	}
112
113	module "netpol" {
114	  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0"
115
116	  name             = "vector-netpol"
117	  namespace        = var.namespace
118	  network_policies = local.netpols
119
120	  depends_on = [
121	    kubernetes_namespace_v1.vector,
122	  ]
123	}
124

terraform {
  required_version = ">= 1.9.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.10"
    }
  }
}

locals {
  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
  )

  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
    "${path.module}/templates/values.yaml.tpl",
    {
      replicas                    = var.replicas
      requests_cpu                = var.resources_requests.cpu
      requests_memory             = var.resources_requests.memory
      limits_cpu                  = var.resources_limits.cpu
      limits_memory               = var.resources_limits.memory
      datadog_site                = var.datadog_site
      audit_drop_condition        = local.audit_drop_condition
      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_replacement = var.email_redaction_replacement
      service_monitor_enabled     = var.service_monitor_enabled
    },
  )
}

resource "kubernetes_namespace_v1" "vector" {
  metadata {
    name = var.namespace
    labels = {
      "app.kubernetes.io/name"      = "vector"
      "app.kubernetes.io/component" = "telemetry-aggregator"
    }
  }
}

resource "kubernetes_manifest" "datadog_external_secret" {
  manifest = {
    apiVersion = "external-secrets.io/v1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "vector-datadog"
      namespace = kubernetes_namespace_v1.vector.metadata[0].name
    }
    spec = {
      refreshInterval = "1h"
      secretStoreRef = {
        kind = "ClusterSecretStore"
        name = var.external_secrets_cluster_store_name
      }
      target = {
        name           = "vector-datadog"
        creationPolicy = "Owner"
      }
      data = [{
        secretKey = "api_key"
        remoteRef = {
          key      = var.datadog_api_key_vault_path
          property = "apiKey"
        }
      }]
    }
  }

  wait {
    fields = {
      "status.conditions[0].type"   = "Ready"
      "status.conditions[0].status" = "True"
    }
  }

  timeouts {
    create = "5m"
    update = "5m"
  }
}

resource "helm_release" "vector" {
  name       = "vector"
  namespace  = kubernetes_namespace_v1.vector.metadata[0].name
  repository = "https://helm.vector.dev"
  chart      = "vector"
  version    = var.chart_version

  values = [local.rendered_values]

  set {
    name  = "image.repository"
    value = var.image
  }
  set {
    name  = "image.tag"
    value = var.image_tag_version
  }

  depends_on = [
    kubernetes_manifest.datadog_external_secret,
  ]
}

module "netpol" {
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0"

  name             = "vector-netpol"
  namespace        = var.namespace
  network_policies = local.netpols

  depends_on = [
    kubernetes_namespace_v1.vector,
  ]
}


1	# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2	# Deploys the Datadog Agent to a Kubernetes cluster using the official Helm chart.
3	# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4
5	# ---------------------------------------------------------------------------------------------------------------------
6	# SET TERRAFORM REQUIREMENTS FOR RUNNING THIS MODULE
7	# ---------------------------------------------------------------------------------------------------------------------
8
9	terraform {
10	  required_version = ">= 1.3.0, < 2.0"
11
12	  required_providers {
13	    helm = {
14	      source  = "hashicorp/helm"
15	      version = "~> 2.0"
16	    }
17	    kubernetes = {
18	      source  = "hashicorp/kubernetes"
19	      version = "~> 2.0"
20	    }
21	  }
22	}
23
24	locals {
25	  apm_rules = file("${path.module}/processing-rules/${var.env}/apm.json")
26	  log_rules = file("${path.module}/processing-rules/${var.env}/rules.json")
27	  nowrds_instance = var.enable_nowrds_monitoring ? (<<-EON
28	    - dbm: true
29	      ssl: allow
30	      host: "%%env_NOWRDS_HOST%%"
31	      port: "%%env_NOWRDS_PORT%%"
32	      username: "%%env_NOWRDS_USERNAME%%"
33	      password: "%%env_NOWRDS_PASSWORD%%"
34	      collect_schemas:
35	        enabled: true
36	      database_autodiscovery:
37	        enabled: true
38	      tags:
39	        - "dbinstanceidentifier:%%env_NOWRDS_INSTANCE_NAME%%"
40	        - "env:%%env_DD_ENV%%"
41	  EON
42	  ) : null
43	}
44	# ---------------------------------------------------------------------------------------------------------------------
45	# CREATE A NAMESPACE
46	# ---------------------------------------------------------------------------------------------------------------------
47	resource "kubernetes_namespace" "this" {
48	  count = var.create_namespace ? 1 : 0
49	  metadata {
50	    name = var.namespace
51	  }
52	}
53
54	resource "kubernetes_config_map" "datadog_processing_rules" {
55	  metadata {
56	    name      = "rules-config"
57	    namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
58	  }
59	  data = {
60	    //"apm"  = local.apm_rules
61	    "apm" = file("${path.module}/processing-rules/${var.env}/apm.json")
62	    //"logs" = local.log_rules
63	    "logs" = file("${path.module}/processing-rules/${var.env}/rules.json")
64	  }
65	  depends_on = [kubernetes_namespace.this]
66	}
67
68	resource "kubernetes_manifest" "external-secret-dd-api-key" {
69	  manifest = {
70	    apiVersion = "external-secrets.io/v1"
71	    kind       = "ExternalSecret"
72	    metadata = {
73	      name      = "datadog-api-key"
74	      namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
75	    }
76	    spec = {
77	      refreshInterval = "1h0m0s"
78	      secretStoreRef = {
79	        name = "vault"
80	        kind = "ClusterSecretStore"
81	      }
82	      target = {
83	        name           = "datadog-api-key"
84	        creationPolicy = "Owner"
85	      }
86	      data = [{
87	        secretKey = "api-key"
88	        remoteRef = {
89	          key      = "saas/infrastructure/datadog-api-key"
90	          property = "apiKey"
91	        }
92	      }]
93	    }
94	  }
95	}
96
97	resource "kubernetes_manifest" "external-secret-dd-api-token" {
98	  manifest = {
99	    apiVersion = "external-secrets.io/v1"
100	    kind       = "ExternalSecret"
101	    metadata = {
102	      name      = "datadog-clusteragent-token"
103	      namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
104	    }
105	    spec = {
106	      refreshInterval = "1h0m0s"
107	      secretStoreRef = {
108	        name = "vault"
109	        kind = "ClusterSecretStore"
110	      }
111	      target = {
112	        name           = "datadog-clusteragent-token"
113	        creationPolicy = "Owner"
114	      }
115	      data = [{
116	        secretKey = "token"
117	        remoteRef = {
118	          key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
119	          property = "token"
120	        }
121	      }]
122	    }
123	  }
124	}
125
126	resource "kubernetes_manifest" "external-secret" {
127	  count = (var.enable_mrds_monitoring || var.enable_nowrds_monitoring) ? 1 : 0
128
129	  manifest = {
130	    apiVersion = "external-secrets.io/v1"
131	    kind       = "ExternalSecret"
132	    metadata = {
133	      name      = "datadog-env"
134	      namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
135	    }
136	    spec = {
137	      refreshInterval = "1h0m0s"
138	      secretStoreRef = {
139	        name = "vault"
140	        kind = "ClusterSecretStore"
141	      }
142	      target = {
143	        name           = "datadog-env"
144	        creationPolicy = "Owner"
145	      }
146	      data = concat(var.enable_mrds_monitoring ? [
147	        {
148	          secretKey = "MRDS_HOST"
149	          remoteRef = {
150	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
151	            property = "rds-host"
152	          }
153	        },
154	        {
155	          secretKey = "MRDS_PORT"
156	          remoteRef = {
157	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
158	            property = "rds-port"
159	          }
160	        },
161	        {
162	          secretKey = "MRDS_USERNAME"
163	          remoteRef = {
164	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
165	            property = "rds-username"
166	          }
167	        },
168	        {
169	          secretKey = "MRDS_PASSWORD"
170	          remoteRef = {
171	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
172	            property = "rds-password"
173	          }
174	        },
175	        {
176	          secretKey = "MRDS_REGION"
177	          remoteRef = {
178	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
179	            property = "rds-region"
180	          }
181	        },
182	        {
183	          secretKey = "MRDS_INSTANCE_NAME"
184	          remoteRef = {
185	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
186	            property = "rds-instance"
187	          }
188	        },
189	        ] : [], var.enable_nowrds_monitoring ? [
190	        {
191	          secretKey = "NOWRDS_HOST"
192	          remoteRef = {
193	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
194	            property = "NOWRDS_HOST"
195	          }
196	        },
197	        {
198	          secretKey = "NOWRDS_PORT"
199	          remoteRef = {
200	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
201	            property = "NOWRDS_PORT"
202	          }
203	        },
204	        {
205	          secretKey = "NOWRDS_USERNAME"
206	          remoteRef = {
207	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
208	            property = "NOWRDS_USERNAME"
209	          }
210	        },
211	        {
212	          secretKey = "NOWRDS_PASSWORD"
213	          remoteRef = {
214	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
215	            property = "NOWRDS_PASSWORD"
216	          }
217	        },
218	        {
219	          secretKey = "NOWRDS_INSTANCE_NAME"
220	          remoteRef = {
221	            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
222	            property = "NOWRDS_INSTANCE_NAME"
223	          }
224	        }
225	      ] : [])
226	    }
227	  }
228	  depends_on = [kubernetes_namespace.this]
229	}
230	# ---------------------------------------------------------------------------------------------------------------------
231	# DEPLOY A HELM CHART
232	# ---------------------------------------------------------------------------------------------------------------------
233	resource "helm_release" "this" {
234	  name = var.name
235	  #namespace = kubernetes_namespace.this.metadata[0].name
236	  namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
237	  wait      = false # Set to true if you want Terraform to wait for resources to be ready
238
239	  repository = "https://helm.datadoghq.com"
240	  chart      = "datadog"
241	  version    = var.chart_version
242	  values = [
243	    templatefile("${path.module}/values.yaml.tftpl", {
244	      cluster_name               = var.cluster_name
245	      env                        = var.env == "staging" ? "nonprod" : var.env
246	      global_segment             = var.global_segment
247	      image_tag                  = var.image_tag
248	      watch_namespace            = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
249	      tags                       = concat(["global_segment:${var.global_segment}"], var.datadog_tags)
250	      agents_priority_class_name = var.agents_priority_class_name
251
252	      apm_enabled = var.apm_enabled
253
254	      container_exclude      = var.container_exclude
255	      container_exclude_logs = var.container_exclude_logs
256
257	      enable_mrds_monitoring         = var.enable_mrds_monitoring
258	      enable_nowrds_monitoring       = var.enable_nowrds_monitoring
259	      enable_now_obfusicator_options = var.enable_now_obfusicator_options
260	      now_obfusicator_options        = var.now_obfusicator_options
261
262	      datadog_api_key_secret_name     = var.datadog_api_key_secret_name
263	      cluster_agent_token_secret_name = var.cluster_agent_token_secret_name
264	      log_level                       = var.log_level
265
266	      logs_intake_url = var.logs_intake_url
267	    })
268	  ]
269
270	  depends_on = [
271	    kubernetes_namespace.this,
272	    kubernetes_manifest.external-secret,
273	    kubernetes_manifest.external-secret-dd-api-key,
274	    kubernetes_manifest.external-secret-dd-api-token
275	  ]
276	}
277
278	module "netpol" {
279	  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0"
280
281	  name             = join("-", [var.name, "netpol"])
282	  namespace        = var.namespace
283	  network_policies = local.netpols
284
285	  tags = merge(var.tags, { "sourceModule" = "/internal-modules/kubernetes/monitoring/datadog" })
286
287	  depends_on = [
288	    kubernetes_namespace.this
289	  ]
290	}
291
292

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Deploys the Datadog Agent to a Kubernetes cluster using the official Helm chart.
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# ---------------------------------------------------------------------------------------------------------------------
# SET TERRAFORM REQUIREMENTS FOR RUNNING THIS MODULE
# ---------------------------------------------------------------------------------------------------------------------

terraform {
  required_version = ">= 1.3.0, < 2.0"

  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
  }
}

locals {
  apm_rules = file("${path.module}/processing-rules/${var.env}/apm.json")
  log_rules = file("${path.module}/processing-rules/${var.env}/rules.json")
  nowrds_instance = var.enable_nowrds_monitoring ? (<<-EON
    - dbm: true
      ssl: allow
      host: "%%env_NOWRDS_HOST%%"
      port: "%%env_NOWRDS_PORT%%"
      username: "%%env_NOWRDS_USERNAME%%"
      password: "%%env_NOWRDS_PASSWORD%%"
      collect_schemas:
        enabled: true
      database_autodiscovery:
        enabled: true
      tags:
        - "dbinstanceidentifier:%%env_NOWRDS_INSTANCE_NAME%%"
        - "env:%%env_DD_ENV%%"
  EON
  ) : null
}
# ---------------------------------------------------------------------------------------------------------------------
# CREATE A NAMESPACE
# ---------------------------------------------------------------------------------------------------------------------
resource "kubernetes_namespace" "this" {
  count = var.create_namespace ? 1 : 0
  metadata {
    name = var.namespace
  }
}

resource "kubernetes_config_map" "datadog_processing_rules" {
  metadata {
    name      = "rules-config"
    namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
  }
  data = {
    //"apm"  = local.apm_rules
    "apm" = file("${path.module}/processing-rules/${var.env}/apm.json")
    //"logs" = local.log_rules
    "logs" = file("${path.module}/processing-rules/${var.env}/rules.json")
  }
  depends_on = [kubernetes_namespace.this]
}

resource "kubernetes_manifest" "external-secret-dd-api-key" {
  manifest = {
    apiVersion = "external-secrets.io/v1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "datadog-api-key"
      namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
    }
    spec = {
      refreshInterval = "1h0m0s"
      secretStoreRef = {
        name = "vault"
        kind = "ClusterSecretStore"
      }
      target = {
        name           = "datadog-api-key"
        creationPolicy = "Owner"
      }
      data = [{
        secretKey = "api-key"
        remoteRef = {
          key      = "saas/infrastructure/datadog-api-key"
          property = "apiKey"
        }
      }]
    }
  }
}

resource "kubernetes_manifest" "external-secret-dd-api-token" {
  manifest = {
    apiVersion = "external-secrets.io/v1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "datadog-clusteragent-token"
      namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
    }
    spec = {
      refreshInterval = "1h0m0s"
      secretStoreRef = {
        name = "vault"
        kind = "ClusterSecretStore"
      }
      target = {
        name           = "datadog-clusteragent-token"
        creationPolicy = "Owner"
      }
      data = [{
        secretKey = "token"
        remoteRef = {
          key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
          property = "token"
        }
      }]
    }
  }
}

resource "kubernetes_manifest" "external-secret" {
  count = (var.enable_mrds_monitoring || var.enable_nowrds_monitoring) ? 1 : 0

  manifest = {
    apiVersion = "external-secrets.io/v1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "datadog-env"
      namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
    }
    spec = {
      refreshInterval = "1h0m0s"
      secretStoreRef = {
        name = "vault"
        kind = "ClusterSecretStore"
      }
      target = {
        name           = "datadog-env"
        creationPolicy = "Owner"
      }
      data = concat(var.enable_mrds_monitoring ? [
        {
          secretKey = "MRDS_HOST"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "rds-host"
          }
        },
        {
          secretKey = "MRDS_PORT"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "rds-port"
          }
        },
        {
          secretKey = "MRDS_USERNAME"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "rds-username"
          }
        },
        {
          secretKey = "MRDS_PASSWORD"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "rds-password"
          }
        },
        {
          secretKey = "MRDS_REGION"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "rds-region"
          }
        },
        {
          secretKey = "MRDS_INSTANCE_NAME"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "rds-instance"
          }
        },
        ] : [], var.enable_nowrds_monitoring ? [
        {
          secretKey = "NOWRDS_HOST"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "NOWRDS_HOST"
          }
        },
        {
          secretKey = "NOWRDS_PORT"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "NOWRDS_PORT"
          }
        },
        {
          secretKey = "NOWRDS_USERNAME"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "NOWRDS_USERNAME"
          }
        },
        {
          secretKey = "NOWRDS_PASSWORD"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "NOWRDS_PASSWORD"
          }
        },
        {
          secretKey = "NOWRDS_INSTANCE_NAME"
          remoteRef = {
            key      = "saas/infrastructure/datadog-helper/${var.env}/${var.global_segment}"
            property = "NOWRDS_INSTANCE_NAME"
          }
        }
      ] : [])
    }
  }
  depends_on = [kubernetes_namespace.this]
}
# ---------------------------------------------------------------------------------------------------------------------
# DEPLOY A HELM CHART
# ---------------------------------------------------------------------------------------------------------------------
resource "helm_release" "this" {
  name = var.name
  #namespace = kubernetes_namespace.this.metadata[0].name
  namespace = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
  wait      = false # Set to true if you want Terraform to wait for resources to be ready

  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  version    = var.chart_version
  values = [
    templatefile("${path.module}/values.yaml.tftpl", {
      cluster_name               = var.cluster_name
      env                        = var.env == "staging" ? "nonprod" : var.env
      global_segment             = var.global_segment
      image_tag                  = var.image_tag
      watch_namespace            = var.create_namespace ? kubernetes_namespace.this[0].metadata[0].name : var.namespace
      tags                       = concat(["global_segment:${var.global_segment}"], var.datadog_tags)
      agents_priority_class_name = var.agents_priority_class_name

      apm_enabled = var.apm_enabled

      container_exclude      = var.container_exclude
      container_exclude_logs = var.container_exclude_logs

      enable_mrds_monitoring         = var.enable_mrds_monitoring
      enable_nowrds_monitoring       = var.enable_nowrds_monitoring
      enable_now_obfusicator_options = var.enable_now_obfusicator_options
      now_obfusicator_options        = var.now_obfusicator_options

      datadog_api_key_secret_name     = var.datadog_api_key_secret_name
      cluster_agent_token_secret_name = var.cluster_agent_token_secret_name
      log_level                       = var.log_level

      logs_intake_url = var.logs_intake_url
    })
  ]

  depends_on = [
    kubernetes_namespace.this,
    kubernetes_manifest.external-secret,
    kubernetes_manifest.external-secret-dd-api-key,
    kubernetes_manifest.external-secret-dd-api-token
  ]
}

module "netpol" {
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0"

  name             = join("-", [var.name, "netpol"])
  namespace        = var.namespace
  network_policies = local.netpols

  tags = merge(var.tags, { "sourceModule" = "/internal-modules/kubernetes/monitoring/datadog" })

  depends_on = [
    kubernetes_namespace.this
  ]
}


Let me also check the netpol.tf, variables.tf, outputs.tf, and the `.gitlab-ci.yml` for vector since those are the files most likely to have changed.


1	locals {
2	  netpols = {
3	    "vector" = local.vector_netpol
4	  }
5	  vector_netpol = {
6	    endpointSelector = {
7	      matchLabels = {
8	        "app.kubernetes.io/name" = "vector"
9	      }
10	    }
11	    ingress = [
12	      {
13	        # OTLP from any cluster-internal sender (primarily otel-collector).
14	        # Port 8282 is the OPW-compatible Datadog Agent intake (datadog_agent source).
15	        fromEntities = ["cluster"]
16	        toPorts = [{
17	          ports = [
18	            { port = "4317" },
19	            { port = "4318" },
20	            { port = "8282" },
21	          ]
22	        }]
23	      },
24	      {
25	        # Vector self-metrics scrape (Prometheus operator, Datadog Agent, etc.).
26	        fromEntities = ["cluster"]
27	        toPorts = [{
28	          ports = [{ port = "8686" }]
29	        }]
30	      },
31	      {
32	        # kubelet health probes from the node.
33	        fromEntities = ["host"]
34	        toPorts = [{
35	          ports = [
36	            { port = "4317" },
37	            { port = "4318" },
38	            { port = "8686" },
39	          ]
40	        }]
41	      },
42	      {
43	        fromEntities = ["kube-apiserver"]
44	      }
45	    ]
46	    egress = [
47	      {
48	        # Datadog SaaS HTTPS for logs/traces/metrics sinks.
49	        toEntities = ["world"]
50	        toPorts = [{
51	          ports = [{ port = "443" }]
52	        }]
53	      },
54	      {
55	        toEntities = ["kube-apiserver"]
56	      },
57	      {
58	        # DNS resolution + service discovery (CoreDNS lives here).
59	        toEntities = ["cluster"]
60	      }
61	    ]
62	  }
63	}
64

locals {
  netpols = {
    "vector" = local.vector_netpol
  }
  vector_netpol = {
    endpointSelector = {
      matchLabels = {
        "app.kubernetes.io/name" = "vector"
      }
    }
    ingress = [
      {
        # OTLP from any cluster-internal sender (primarily otel-collector).
        # Port 8282 is the OPW-compatible Datadog Agent intake (datadog_agent source).
        fromEntities = ["cluster"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
          ]
        }]
      },
      {
        # Vector self-metrics scrape (Prometheus operator, Datadog Agent, etc.).
        fromEntities = ["cluster"]
        toPorts = [{
          ports = [{ port = "8686" }]
        }]
      },
      {
        # kubelet health probes from the node.
        fromEntities = ["host"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8686" },
          ]
        }]
      },
      {
        fromEntities = ["kube-apiserver"]
      }
    ]
    egress = [
      {
        # Datadog SaaS HTTPS for logs/traces/metrics sinks.
        toEntities = ["world"]
        toPorts = [{
          ports = [{ port = "443" }]
        }]
      },
      {
        toEntities = ["kube-apiserver"]
      },
      {
        # DNS resolution + service discovery (CoreDNS lives here).
        toEntities = ["cluster"]
      }
    ]
  }
}


1	1.1.0
2


1	1.1.3
2


1	stages:
2	  - lint
3	  - test
4
5	terraform-fmt:
6	  stage: lint
7	  image: registry.ic-ops.net/cache-docker-hub/hashicorp/terraform:1.9
8	  script:
9	    - terraform fmt -check -recursive
10
11	terraform-validate:
12	  stage: test
13	  image: registry.ic-ops.net/cache-docker-hub/hashicorp/terraform:1.9
14	  script:
15	    - cd examples/basic
16	    - terraform init -backend=false
17	    - terraform validate
18
19	vrl-tests:
20	  stage: test
21	  image: registry.ic-ops.net/cache-docker-hub/timberio/vector:latest-debian
22	  before_script:
23	    - apt-get update -qq && apt-get install -y -qq bash diffutils jq
24	  script:
25	    - ./tests/vrl/drop_audit/run.sh
26	    - ./tests/vrl/redact_emails_logs/run.sh
27	    - ./tests/vrl/redact_emails_traces/run.sh
28	    - ./tests/vrl/redact_emails_metrics/run.sh
29

stages:
  - lint
  - test

terraform-fmt:
  stage: lint
  image: registry.ic-ops.net/cache-docker-hub/hashicorp/terraform:1.9
  script:
    - terraform fmt -check -recursive

terraform-validate:
  stage: test
  image: registry.ic-ops.net/cache-docker-hub/hashicorp/terraform:1.9
  script:
    - cd examples/basic
    - terraform init -backend=false
    - terraform validate

vrl-tests:
  stage: test
  image: registry.ic-ops.net/cache-docker-hub/timberio/vector:latest-debian
  before_script:
    - apt-get update -qq && apt-get install -y -qq bash diffutils jq
  script:
    - ./tests/vrl/drop_audit/run.sh
    - ./tests/vrl/redact_emails_logs/run.sh
    - ./tests/vrl/redact_emails_traces/run.sh
    - ./tests/vrl/redact_emails_metrics/run.sh


Let me check the examples and look more carefully at the source references.


1	module "vector" {
2	  source = "../.."
3
4	  chart_version              = "0.36.0"
5	  image_tag_version          = "0.40.0-distroless-libc"
6	  datadog_site               = "datadoghq.com"
7	  datadog_api_key_vault_path = "secret/data/datadog/api-key"
8
9	  external_secrets_cluster_store_name = "vault-cluster-store"
10	}
11

module "vector" {
  source = "../.."

  chart_version              = "0.36.0"
  image_tag_version          = "0.40.0-distroless-libc"
  datadog_site               = "datadoghq.com"
  datadog_api_key_vault_path = "secret/data/datadog/api-key"

  external_secrets_cluster_store_name = "vault-cluster-store"
}


1	terraform {
2	  required_version = ">= 1.9.0"
3	  required_providers {
4	    kubernetes = { source = "hashicorp/kubernetes", version = ">= 2.20" }
5	    helm       = { source = "hashicorp/helm", version = ">= 2.10" }
6	  }
7	}
8
9	provider "kubernetes" {
10	  config_path = "~/.kube/config"
11	}
12
13	provider "helm" {
14	  kubernetes {
15	    config_path = "~/.kube/config"
16	  }
17	}
18

terraform {
  required_version = ">= 1.9.0"
  required_providers {
    kubernetes = { source = "hashicorp/kubernetes", version = ">= 2.20" }
    helm       = { source = "hashicorp/helm", version = ">= 2.10" }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}


1	# This file is maintained automatically by "terraform init".
2	# Manual edits may be lost in future updates.
3
4	provider "registry.terraform.io/hashicorp/helm" {
5	  version     = "2.17.0"
6	  constraints = "~> 2.0, >= 2.10.0"
7	  hashes = [
8	    "h1:kQMkcPVvHOguOqnxoEU2sm1ND9vCHiT8TvZ2x6v/Rsw=",
9	    "zh:06fb4e9932f0afc1904d2279e6e99353c2ddac0d765305ce90519af410706bd4",
10	    "zh:104eccfc781fc868da3c7fec4385ad14ed183eb985c96331a1a937ac79c2d1a7",
11	    "zh:129345c82359837bb3f0070ce4891ec232697052f7d5ccf61d43d818912cf5f3",
12	    "zh:3956187ec239f4045975b35e8c30741f701aa494c386aaa04ebabffe7749f81c",
13	    "zh:66a9686d92a6b3ec43de3ca3fde60ef3d89fb76259ed3313ca4eb9bb8c13b7dd",
14	    "zh:88644260090aa621e7e8083585c468c8dd5e09a3c01a432fb05da5c4623af940",
15	    "zh:a248f650d174a883b32c5b94f9e725f4057e623b00f171936dcdcc840fad0b3e",
16	    "zh:aa498c1f1ab93be5c8fbf6d48af51dc6ef0f10b2ea88d67bcb9f02d1d80d3930",
17	    "zh:bf01e0f2ec2468c53596e027d376532a2d30feb72b0b5b810334d043109ae32f",
18	    "zh:c46fa84cc8388e5ca87eb575a534ebcf68819c5a5724142998b487cb11246654",
19	    "zh:d0c0f15ffc115c0965cbfe5c81f18c2e114113e7a1e6829f6bfd879ce5744fbb",
20	    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
21	  ]
22	}
23
24	provider "registry.terraform.io/hashicorp/kubernetes" {
25	  version     = "2.38.0"
26	  constraints = "~> 2.0, >= 2.20.0"
27	  hashes = [
28	    "h1:soK8Lt0SZ6dB+HsypFRDzuX/npqlMU6M0fvyaR1yW0k=",
29	    "zh:0af928d776eb269b192dc0ea0f8a3f0f5ec117224cd644bdacdc682300f84ba0",
30	    "zh:1be998e67206f7cfc4ffe77c01a09ac91ce725de0abaec9030b22c0a832af44f",
31	    "zh:326803fe5946023687d603f6f1bab24de7af3d426b01d20e51d4e6fbe4e7ec1b",
32	    "zh:4a99ec8d91193af961de1abb1f824be73df07489301d62e6141a656b3ebfff12",
33	    "zh:5136e51765d6a0b9e4dbcc3b38821e9736bd2136cf15e9aac11668f22db117d2",
34	    "zh:63fab47349852d7802fb032e4f2b6a101ee1ce34b62557a9ad0f0f0f5b6ecfdc",
35	    "zh:924fb0257e2d03e03e2bfe9c7b99aa73c195b1f19412ca09960001bee3c50d15",
36	    "zh:b63a0be5e233f8f6727c56bed3b61eb9456ca7a8bb29539fba0837f1badf1396",
37	    "zh:d39861aa21077f1bc899bc53e7233262e530ba8a3a2d737449b100daeb303e4d",
38	    "zh:de0805e10ebe4c83ce3b728a67f6b0f9d18be32b25146aa89116634df5145ad4",
39	    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
40	    "zh:faf23e45f0090eef8ba28a8aac7ec5d4fdf11a36c40a8d286304567d71c1e7db",
41	  ]
42	}
43

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/helm" {
  version     = "2.17.0"
  constraints = "~> 2.0, >= 2.10.0"
  hashes = [
    "h1:kQMkcPVvHOguOqnxoEU2sm1ND9vCHiT8TvZ2x6v/Rsw=",
    "zh:06fb4e9932f0afc1904d2279e6e99353c2ddac0d765305ce90519af410706bd4",
    "zh:104eccfc781fc868da3c7fec4385ad14ed183eb985c96331a1a937ac79c2d1a7",
    "zh:129345c82359837bb3f0070ce4891ec232697052f7d5ccf61d43d818912cf5f3",
    "zh:3956187ec239f4045975b35e8c30741f701aa494c386aaa04ebabffe7749f81c",
    "zh:66a9686d92a6b3ec43de3ca3fde60ef3d89fb76259ed3313ca4eb9bb8c13b7dd",
    "zh:88644260090aa621e7e8083585c468c8dd5e09a3c01a432fb05da5c4623af940",
    "zh:a248f650d174a883b32c5b94f9e725f4057e623b00f171936dcdcc840fad0b3e",
    "zh:aa498c1f1ab93be5c8fbf6d48af51dc6ef0f10b2ea88d67bcb9f02d1d80d3930",
    "zh:bf01e0f2ec2468c53596e027d376532a2d30feb72b0b5b810334d043109ae32f",
    "zh:c46fa84cc8388e5ca87eb575a534ebcf68819c5a5724142998b487cb11246654",
    "zh:d0c0f15ffc115c0965cbfe5c81f18c2e114113e7a1e6829f6bfd879ce5744fbb",
    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
  ]
}

provider "registry.terraform.io/hashicorp/kubernetes" {
  version     = "2.38.0"
  constraints = "~> 2.0, >= 2.20.0"
  hashes = [
    "h1:soK8Lt0SZ6dB+HsypFRDzuX/npqlMU6M0fvyaR1yW0k=",
    "zh:0af928d776eb269b192dc0ea0f8a3f0f5ec117224cd644bdacdc682300f84ba0",
    "zh:1be998e67206f7cfc4ffe77c01a09ac91ce725de0abaec9030b22c0a832af44f",
    "zh:326803fe5946023687d603f6f1bab24de7af3d426b01d20e51d4e6fbe4e7ec1b",
    "zh:4a99ec8d91193af961de1abb1f824be73df07489301d62e6141a656b3ebfff12",
    "zh:5136e51765d6a0b9e4dbcc3b38821e9736bd2136cf15e9aac11668f22db117d2",
    "zh:63fab47349852d7802fb032e4f2b6a101ee1ce34b62557a9ad0f0f0f5b6ecfdc",
    "zh:924fb0257e2d03e03e2bfe9c7b99aa73c195b1f19412ca09960001bee3c50d15",
    "zh:b63a0be5e233f8f6727c56bed3b61eb9456ca7a8bb29539fba0837f1badf1396",
    "zh:d39861aa21077f1bc899bc53e7233262e530ba8a3a2d737449b100daeb303e4d",
    "zh:de0805e10ebe4c83ce3b728a67f6b0f9d18be32b25146aa89116634df5145ad4",
    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
    "zh:faf23e45f0090eef8ba28a8aac7ec5d4fdf11a36c40a8d286304567d71c1e7db",
  ]
}


Let me also check the examples terraform modules.json and look if there's a snapshot test file that hints at what changed.


1	role: Aggregator
2	replicas: 3
3	podAnnotations:
4	  reloader.stakater.com/auto: "true"
5	service:
6	  enabled: true
7	  type: ClusterIP
8	  ports:
9	    - name: otlp-grpc
10	      port: 4317
11	      protocol: TCP
12	    - name: otlp-http
13	      port: 4318
14	      protocol: TCP
15	env:
16	  - name: DATADOG_API_KEY
17	    valueFrom:
18	      secretKeyRef:
19	        name: vector-datadog
20	        key: api_key
21	resources:
22	  requests:
23	    cpu: 200m
24	    memory: 256Mi
25	  limits:
26	    cpu: "2"
27	    memory: 1Gi
28	customConfig:
29	  data_dir: /vector-data-dir
30	  api:
31	    enabled: true
32	    address: 127.0.0.1:8686
33	  sources:
34	    otlp_in:
35	      type: opentelemetry
36	      grpc:
37	        address: 0.0.0.0:4317
38	      http:
39	        address: 0.0.0.0:4318
40	  transforms: {}
41	  sinks:
42	    datadog_logs:
43	      type: datadog_logs
44	      inputs:
45	        - otlp_in.logs
46	      default_api_key: ${DATADOG_API_KEY}
47	      site: datadoghq.com
48	    datadog_traces:
49	      type: datadog_traces
50	      inputs:
51	        - otlp_in.traces
52	      default_api_key: ${DATADOG_API_KEY}
53	      site: datadoghq.com
54	    datadog_metrics:
55	      type: datadog_metrics
56	      inputs:
57	        - otlp_in.metrics
58	      default_api_key: ${DATADOG_API_KEY}
59	      site: datadoghq.com
60

role: Aggregator
replicas: 3
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: "2"
    memory: 1Gi
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 127.0.0.1:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
  transforms: {}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - otlp_in.logs
      default_api_key: ${DATADOG_API_KEY}
      site: datadoghq.com
    datadog_traces:
      type: datadog_traces
      inputs:
        - otlp_in.traces
      default_api_key: ${DATADOG_API_KEY}
      site: datadoghq.com
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - otlp_in.metrics
      default_api_key: ${DATADOG_API_KEY}
      site: datadoghq.com