<memvid-mind-context>
# 🧠 Claude Mind Active

📁 Project: **vector**
💾 Memory: `.claude/mind.mv2` (5492 KB)

**Commands:**
- `/mind:search <query>` - Search memories
- `/mind:ask <question>` - Ask your memory
- `/mind:recent` - View timeline
- `/mind:stats` - View statistics

_Memories are captured automatically from your tool use._
</memvid-mind-context>


<EXTREMELY_IMPORTANT>
You have superpowers.

**Below is the full content of your 'superpowers:using-superpowers' skill - your introduction to using skills. For all other skills, use the 'Skill' tool:**

---
name: using-superpowers
description: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
---

<SUBAGENT-STOP>
If you were dispatched as a subagent to execute a specific task, skip this skill.
</SUBAGENT-STOP>

<EXTREMELY-IMPORTANT>
If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.

IF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.

This is not negotiable. This is not optional. You cannot rationalize your way out of this.
</EXTREMELY-IMPORTANT>

## Instruction Priority

Superpowers skills override default system prompt behavior, but **user instructions always take precedence**:

1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority
2. **Superpowers skills** — override default system behavior where they conflict
3. **Default system prompt** — lowest priority

If CLAUDE.md, GEMINI.md, or AGENTS.md says "don't use TDD" and a skill says "always use TDD," follow the user's instructions. The user is in control.

## How to Access Skills

**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.

**In Copilot CLI:** Use the `skill` tool. Skills are auto-discovered from installed plugins. The `skill` tool works the same as Claude Code's `Skill` tool.

**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.

**In other environments:** Check your platform's documentation for how skills are loaded.

## Platform Adaptation

Skills use Claude Code tool names. Non-CC platforms: see `references/copilot-tools.md` (Copilot CLI), `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.

# Using Skills

## The Rule

**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.

```dot
digraph skill_flow {
    "User message received" [shape=doublecircle];
    "About to EnterPlanMode?" [shape=doublecircle];
    "Already brainstormed?" [shape=diamond];
    "Invoke brainstorming skill" [shape=box];
    "Might any skill apply?" [shape=diamond];
    "Invoke Skill tool" [shape=box];
    "Announce: 'Using [skill] to [purpose]'" [shape=box];
    "Has checklist?" [shape=diamond];
    "Create TodoWrite todo per item" [shape=box];
    "Follow skill exactly" [shape=box];
    "Respond (including clarifications)" [shape=doublecircle];

    "About to EnterPlanMode?" -> "Already brainstormed?";
    "Already brainstormed?" -> "Invoke brainstorming skill" [label="no"];
    "Already brainstormed?" -> "Might any skill apply?" [label="yes"];
    "Invoke brainstorming skill" -> "Might any skill apply?";

    "User message received" -> "Might any skill apply?";
    "Might any skill apply?" -> "Invoke Skill tool" [label="yes, even 1%"];
    "Might any skill apply?" -> "Respond (including clarifications)" [label="definitely not"];
    "Invoke Skill tool" -> "Announce: 'Using [skill] to [purpose]'";
    "Announce: 'Using [skill] to [purpose]'" -> "Has checklist?";
    "Has checklist?" -> "Create TodoWrite todo per item" [label="yes"];
    "Has checklist?" -> "Follow skill exactly" [label="no"];
    "Create TodoWrite todo per item" -> "Follow skill exactly";
}
```

## Red Flags

These thoughts mean STOP—you're rationalizing:

| Thought | Reality |
|---------|---------|
| "This is just a simple question" | Questions are tasks. Check for skills. |
| "I need more context first" | Skill check comes BEFORE clarifying questions. |
| "Let me explore the codebase first" | Skills tell you HOW to explore. Check first. |
| "I can check git/files quickly" | Files lack conversation context. Check for skills. |
| "Let me gather information first" | Skills tell you HOW to gather information. |
| "This doesn't need a formal skill" | If a skill exists, use it. |
| "I remember this skill" | Skills evolve. Read current version. |
| "This doesn't count as a task" | Action = task. Check for skills. |
| "The skill is overkill" | Simple things become complex. Use it. |
| "I'll just do this one thing first" | Check BEFORE doing anything. |
| "This feels productive" | Undisciplined action wastes time. Skills prevent this. |
| "I know what that means" | Knowing the concept ≠ using the skill. Invoke it. |

## Skill Priority

When multiple skills could apply, use this order:

1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task
2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution

"Let's build X" → brainstorming first, then implementation skills.
"Fix this bug" → debugging first, then domain-specific skills.

## Skill Types

**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.

**Flexible** (patterns): Adapt principles to context.

The skill itself tells you which.

## User Instructions

Instructions say WHAT, not HOW. "Add X" or "Fix Y" doesn't mean skip workflows.

</EXTREMELY_IMPORTANT>


<memvid-mind-context>
# 🧠 Claude Mind Active

📁 Project: **vector**
💾 Memory: `.claude/mind.mv2` (5492 KB)

**Commands:**
- `/mind:search <query>` - Search memories
- `/mind:ask <question>` - Ask your memory
- `/mind:recent` - View timeline
- `/mind:stats` - View statistics

_Memories are captured automatically from your tool use._
</memvid-mind-context>

<EXTREMELY_IMPORTANT>
You have superpowers.

**Below is the full content of your 'superpowers:using-superpowers' skill - your introduction to using skills. For all other skills, use the 'Skill' tool:**

---
name: using-superpowers
description: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
---

<SUBAGENT-STOP>
If you were dispatched as a subagent to execute a specific task, skip this skill.
</SUBAGENT-STOP>

<EXTREMELY-IMPORTANT>
If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.

IF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.

This is not negotiable. This is not optional. You cannot rationalize your way out of this.
</EXTREMELY-IMPORTANT>

## Instruction Priority

Superpowers skills override default system prompt behavior, but **user instructions always take precedence**:

1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority
2. **Superpowers skills** — override default system behavior where they conflict
3. **Default system prompt** — lowest priority

If CLAUDE.md, GEMINI.md, or AGENTS.md says "don't use TDD" and a skill says "always use TDD," follow the user's instructions. The user is in control.

## How to Access Skills

**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.

**In Copilot CLI:** Use the `skill` tool. Skills are auto-discovered from installed plugins. The `skill` tool works the same as Claude Code's `Skill` tool.

**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.

**In other environments:** Check your platform's documentation for how skills are loaded.

## Platform Adaptation

Skills use Claude Code tool names. Non-CC platforms: see `references/copilot-tools.md` (Copilot CLI), `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.

# Using Skills

## The Rule

**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.

```dot
digraph skill_flow {
    "User message received" [shape=doublecircle];
    "About to EnterPlanMode?" [shape=doublecircle];
    "Already brainstormed?" [shape=diamond];
    "Invoke brainstorming skill" [shape=box];
    "Might any skill apply?" [shape=diamond];
    "Invoke Skill tool" [shape=box];
    "Announce: 'Using [skill] to [purpose]'" [shape=box];
    "Has checklist?" [shape=diamond];
    "Create TodoWrite todo per item" [shape=box];
    "Follow skill exactly" [shape=box];
    "Respond (including clarifications)" [shape=doublecircle];

    "About to EnterPlanMode?" -> "Already brainstormed?";
    "Already brainstormed?" -> "Invoke brainstorming skill" [label="no"];
    "Already brainstormed?" -> "Might any skill apply?" [label="yes"];
    "Invoke brainstorming skill" -> "Might any skill apply?";

    "User message received" -> "Might any skill apply?";
    "Might any skill apply?" -> "Invoke Skill tool" [label="yes, even 1%"];
    "Might any skill apply?" -> "Respond (including clarifications)" [label="definitely not"];
    "Invoke Skill tool" -> "Announce: 'Using [skill] to [purpose]'";
    "Announce: 'Using [skill] to [purpose]'" -> "Has checklist?";
    "Has checklist?" -> "Create TodoWrite todo per item" [label="yes"];
    "Has checklist?" -> "Follow skill exactly" [label="no"];
    "Create TodoWrite todo per item" -> "Follow skill exactly";
}
```

## Red Flags

These thoughts mean STOP—you're rationalizing:

| Thought | Reality |
|---------|---------|
| "This is just a simple question" | Questions are tasks. Check for skills. |
| "I need more context first" | Skill check comes BEFORE clarifying questions. |
| "Let me explore the codebase first" | Skills tell you HOW to explore. Check first. |
| "I can check git/files quickly" | Files lack conversation context. Check for skills. |
| "Let me gather information first" | Skills tell you HOW to gather information. |
| "This doesn't need a formal skill" | If a skill exists, use it. |
| "I remember this skill" | Skills evolve. Read current version. |
| "This doesn't count as a task" | Action = task. Check for skills. |
| "The skill is overkill" | Simple things become complex. Use it. |
| "I'll just do this one thing first" | Check BEFORE doing anything. |
| "This feels productive" | Undisciplined action wastes time. Skills prevent this. |
| "I know what that means" | Knowing the concept ≠ using the skill. Invoke it. |

## Skill Priority

When multiple skills could apply, use this order:

1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task
2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution

"Let's build X" → brainstorming first, then implementation skills.
"Fix this bug" → debugging first, then domain-specific skills.

## Skill Types

**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.

**Flexible** (patterns): Adapt principles to context.

The skill itself tells you which.

## User Instructions

Instructions say WHAT, not HOW. "Add X" or "Fix Y" doesn't mean skip workflows.

</EXTREMELY_IMPORTANT>


For this vector deployment I need to add another redaction filter for Bearer Token Scanner, in Datadog we have the Sensitive Data Scanner query for this setup as the following; env:prod AND (service:service OR service:secure-metadata-database) AND NOT "internal-webhook" AND NOT "starting crawl" AND NOT "Databricks automatic subscription" AND NOT "audit/enrich" AND NOT "Background Job" AND NOT community-organization-id:immuta AND NOT resource_name:"pg - pool.connect"


## claude.ai Context7
Use this server to fetch current documentation whenever the user asks about a library, framework, SDK, API, CLI tool, or cloud service -- even well-known ones like React, Next.js, Prisma, Express, Tailwind, Django, or Spring Boot. This includes API syntax, configuration, version migration, library-specific debugging, setup instructions, and CLI tool usage. Use even when you think you know the answer -- your training data may not reflect recent changes. Prefer this over web search for library docs.

Do not use for: refactoring, writing scripts from scratch, debugging business logic, code review, or general programming concepts.


- migrate-flux-app: Use when migrating a Flux CD application from the infrastructure/kubernetes/application-deployments repo to the application-deployment repo. Covers copying app folders, updating kustomizations, committing, pushing, and creating merge requests.
- app-infra-generate: Generate an application-infrastructure Terraform module from a YAML spec produced by app-infra-spec. Reads pattern files and a versions manifest to produce main.tf, variables.tf, outputs.tf, overlay guidance, and a triage list.
- app-infra-spec: Structured questionnaire to gather requirements for an application-infrastructure Terraform module. Produces a YAML spec for the app-infra-generate skill.
- obsidian-load: Load context from Obsidian notes in the `Claude/` directory into the current conversation.
- obsidian-save: Save current conversation context to an Obsidian note under the `Claude/` directory.
- mind:search: Search memories for specific content or patterns
- mind:stats: Show memory statistics and storage information
- mind:recent: Show recent memories and activity timeline
- mind:ask: Ask questions about memories and get context-aware answers
- superpowers:using-git-worktrees: Use when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback
- superpowers:test-driven-development: Use when implementing any feature or bugfix, before writing implementation code
- superpowers:executing-plans: Use when you have a written implementation plan to execute in a separate session with review checkpoints
- superpowers:systematic-debugging: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
- superpowers:dispatching-parallel-agents: Use when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies
- superpowers:using-superpowers: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
- superpowers:finishing-a-development-branch: Use when implementation is complete, all tests pass, and you need to decide how to integrate the work - guides completion of development work by presenting structured options for merge, PR, or cleanup
- superpowers:writing-skills: Use when creating new skills, editing existing skills, or verifying skills work before deployment
- superpowers:receiving-code-review: Use when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation
- superpowers:writing-plans: Use when you have a spec or requirements for a multi-step task, before touching code
- superpowers:verification-before-completion: Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always
- superpowers:requesting-code-review: Use when completing tasks, implementing major features, or before merging to verify work meets requirements
- superpowers:brainstorming: You MUST use this before any creative work - creating features, building components, adding functionality, or modifying behavior. Explores user intent, requirements and design before implementation.
- superpowers:subagent-driven-development: Use when executing implementation plans with independent tasks in the current session
- mind:memory: Claude Mind - Search and manage Claude's persistent memory stored in a single portable .mv2 file
- mind:mind: Claude Mind - Search and manage Claude's persistent memory stored in a single portable .mv2 file
- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- simplify: Review changed code for reuse, quality, and efficiency, then fix any issues found.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled remote agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring remote agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Build, debug, and optimize Claude API / Anthropic SDK apps. Apps built with this skill should include prompt caching. Also handles migrating existing Claude API code between Claude model versions (4.5 → 4.6, 4.6 → 4.7, retired-model replacements).
TRIGGER when: code imports `anthropic`/`@anthropic-ai/sdk`; user asks for the Claude API, Anthropic SDK, or Managed Agents; user adds/modifies/tunes a Claude feature (caching, thinking, compaction, tool use, batch, files, citations, memory) or model (Opus/Sonnet/Haiku) in a file; questions about prompt caching / cache hit rate in an Anthropic SDK project.
SKIP: file imports `openai`/other-provider SDK, filename like `*-openai.py`/`*-generic.py`, provider-neutral code, general programming/ML.
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch


Add Bearer Token Scanner redaction filter


Let me check the existing vector configuration to understand the current setup.


Bash

find /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector -type f | sort

List all files in vector module


For this vector deployment I need to add another redaction filter for Bearer Token Scanner, in Datadog we have the Sensitive Data Scanner query for this setup as the following; env:prod AND (service:s…


1	role: Aggregator
2	replicas: ${replicas}
3	podAnnotations:
4	  reloader.stakater.com/auto: "true"
5	service:
6	  enabled: true
7	  type: ClusterIP
8	  ports:
9	    - name: otlp-grpc
10	      port: 4317
11	      protocol: TCP
12	    - name: otlp-http
13	      port: 4318
14	      protocol: TCP
15	    - name: dd-agent
16	      port: 8282
17	      protocol: TCP
18	env:
19	  - name: DATADOG_API_KEY
20	    valueFrom:
21	      secretKeyRef:
22	        name: vector-datadog
23	        key: api_key
24	resources:
25	  requests:
26	    cpu: ${requests_cpu}
27	    memory: ${requests_memory}
28	  limits:
29	    cpu: "${limits_cpu}"
30	    memory: ${limits_memory}
31	podDisruptionBudget:
32	  enabled: true
33	  minAvailable: 2
34
35	topologySpreadConstraints:
36	  - maxSkew: 1
37	    topologyKey: topology.kubernetes.io/zone
38	    whenUnsatisfiable: ScheduleAnyway
39	    labelSelector:
40	      matchLabels:
41	        app.kubernetes.io/name: vector
42
43	affinity:
44	  podAntiAffinity:
45	    preferredDuringSchedulingIgnoredDuringExecution:
46	      - weight: 100
47	        podAffinityTerm:
48	          topologyKey: kubernetes.io/hostname
49	          labelSelector:
50	            matchLabels:
51	              app.kubernetes.io/name: vector
52
53	%{ if service_monitor_enabled ~}
54	serviceMonitor:
55	  enabled: true
56	%{ endif ~}
57	customConfig:
58	  data_dir: /vector-data-dir
59	  api:
60	    enabled: true
61	    address: 0.0.0.0:8686
62	  sources:
63	    otlp_in:
64	      type: opentelemetry
65	      grpc:
66	        address: 0.0.0.0:4317
67	      http:
68	        address: 0.0.0.0:4318
69	    dd_agent_in:
70	      type: datadog_agent
71	      address: 0.0.0.0:8282
72	      multiple_outputs: true
73	      disable_metrics: true
74	      disable_traces: true
75	      store_api_key: false
76	      acknowledgements:
77	        enabled: true
78	  transforms:
79	    drop_audit:
80	      type: filter
81	      inputs:
82	        - otlp_in.logs
83	        - dd_agent_in.logs
84	      condition: |
85	        msg = if is_string(.message) { string!(.message) } else { "" }
86	        ${audit_drop_condition}
87	    normalize_logs:
88	      type: remap
89	      inputs:
90	        - drop_audit
91	      source: |
92	        if exists(.severity) {
93	          .status = downcase(to_string!(.severity))
94	          del(.severity)
95	        }
96	%{ if email_redaction_enabled ~}
97	    redact_emails_logs:
98	      type: remap
99	      inputs:
100	        - normalize_logs
101	      source: |
102	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
103	        if is_string(.message) {
104	          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
105	        }
106	        if is_object(.attributes) {
107	          attrs, _err = object(.attributes)
108	          if is_object(attrs) {
109	            .attributes = map_values(attrs) -> |v| {
110	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
111	            }
112	          }
113	        }
114	    redact_emails_traces:
115	      type: remap
116	      inputs:
117	        - otlp_in.traces
118	      source: |
119	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
120	        if is_object(.attributes) {
121	          attrs, _err = object(.attributes)
122	          if is_object(attrs) {
123	            .attributes = map_values(attrs) -> |v| {
124	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
125	            }
126	          }
127	        }
128	        if is_object(.resource) {
129	          res, _err = object(.resource)
130	          if is_object(res) && is_object(res.attributes) {
131	            rattrs, _err2 = object(res.attributes)
132	            if is_object(rattrs) {
133	              .resource.attributes = map_values(rattrs) -> |v| {
134	                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
135	              }
136	            }
137	          }
138	        }
139	    redact_emails_metrics:
140	      type: remap
141	      inputs:
142	        - otlp_in.metrics
143	      source: |
144	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
145	        if is_object(.attributes) {
146	          attrs, _err = object(.attributes)
147	          if is_object(attrs) {
148	            .attributes = map_values(attrs) -> |v| {
149	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
150	            }
151	          }
152	        }
153	        if is_object(.resource) {
154	          res, _err = object(.resource)
155	          if is_object(res) && is_object(res.attributes) {
156	            rattrs, _err2 = object(res.attributes)
157	            if is_object(rattrs) {
158	              .resource.attributes = map_values(rattrs) -> |v| {
159	                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
160	              }
161	            }
162	          }
163	        }
164	%{ endif ~}
165	%{ if !email_redaction_enabled && email_redaction_dry_run ~}
166	    tag_emails_logs:
167	      type: remap
168	      inputs:
169	        - normalize_logs
170	      source: |
171	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
172	        detected = false
173	        if is_string(.message) {
174	          detected = match(string!(.message), email_re)
175	        }
176	        if !detected && is_object(.attributes) {
177	          detected = match(encode_json(.attributes), email_re)
178	        }
179	        if detected {
180	          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
181	        }
182	%{ endif ~}
183	  sinks:
184	    datadog_logs:
185	      type: datadog_logs
186	      inputs:
187	        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "normalize_logs" }
188	      default_api_key: $${DATADOG_API_KEY}
189	      site: ${datadog_site}
190	    datadog_traces:
191	      type: datadog_traces
192	      inputs:
193	        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
194	      default_api_key: $${DATADOG_API_KEY}
195	      site: ${datadog_site}
196	    datadog_metrics:
197	      type: datadog_metrics
198	      inputs:
199	        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
200	      default_api_key: $${DATADOG_API_KEY}
201	      site: ${datadog_site}
202

role: Aggregator
replicas: ${replicas}
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: ${requests_cpu}
    memory: ${requests_memory}
  limits:
    cpu: "${limits_cpu}"
    memory: ${limits_memory}
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector

%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 0.0.0.0:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
    normalize_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        if exists(.severity) {
          .status = downcase(to_string!(.severity))
          del(.severity)
        }
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - normalize_logs
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_string(.message) {
          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
        }
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - normalize_logs
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "normalize_logs" }
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_traces:
      type: datadog_traces
      inputs:
        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}


1	# ---------------------------------------------------------------------------------------------------------------------
2	# REQUIRED PARAMETERS
3	# These variables must be set.
4	# ---------------------------------------------------------------------------------------------------------------------
5
6	variable "chart_version" {
7	  description = "Vector helm chart version pin"
8	  type        = string
9	}
10
11	variable "image_tag_version" {
12	  description = "Vector image tag"
13	  type        = string
14	}
15
16	# ---------------------------------------------------------------------------------------------------------------------
17	# OPTIONAL PARAMETERS
18	# These variables are configurable, but have reasonable defaults set.
19	# ---------------------------------------------------------------------------------------------------------------------
20	variable "datadog_site" {
21	  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
22	  type        = string
23	  default     = "datadoghq.com"
24	}
25
26	variable "image" {
27	  description = "Vector image source"
28	  type        = string
29	  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
30	}
31
32	variable "replicas" {
33	  description = "Number of Vector aggregator replicas"
34	  type        = number
35	  default     = 3
36	}
37
38	variable "audit_drop_patterns" {
39	  description = "Regex patterns; matching log message bodies are dropped"
40	  type        = list(string)
41	  default = [
42	    "Audit\\s-\\s.+",
43	    "\"level\":\"audit\"",
44	  ]
45	}
46
47	variable "datadog_api_key_vault_path" {
48	  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
49	  type        = string
50	  default     = "secret/data/saas/infrastructure/datadog-api-key"
51	}
52
53	variable "external_secrets_cluster_store_name" {
54	  description = "Name of the ClusterSecretStore created by external-secrets module"
55	  type        = string
56	  default     = "vault"
57	}
58
59	variable "email_redaction_enabled" {
60	  description = "Toggle the email-redact VRL transform across all signals"
61	  type        = bool
62	  default     = true
63	}
64
65	variable "email_redaction_dry_run" {
66	  description = "When true (and email_redaction_enabled is false), tag logs that would have been redacted with vector_email_detected=true instead of replacing. Useful for non-prod visibility without masking."
67	  type        = bool
68	  default     = false
69	}
70
71	variable "email_redaction_replacement" {
72	  description = "String that replaces matched email addresses"
73	  type        = string
74	  default     = "[REDACTED-EMAIL]"
75	}
76
77	variable "additional_config" {
78	  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
79	  type        = any
80	  default     = null
81	}
82
83	variable "resources_requests" {
84	  description = "Pod resource requests"
85	  type        = map(string)
86	  default = {
87	    cpu    = "200m"
88	    memory = "256Mi"
89	  }
90	}
91
92	variable "resources_limits" {
93	  description = "Pod resource limits"
94	  type        = map(string)
95	  default = {
96	    cpu    = "2"
97	    memory = "1Gi"
98	  }
99	}
100
101	variable "namespace" {
102	  description = "Kubernetes namespace for Vector"
103	  type        = string
104	  default     = "vector"
105	}
106
107	variable "service_monitor_enabled" {
108	  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
109	  type        = bool
110	  default     = false
111	}
112

# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# These variables must be set.
# ---------------------------------------------------------------------------------------------------------------------

variable "chart_version" {
  description = "Vector helm chart version pin"
  type        = string
}

variable "image_tag_version" {
  description = "Vector image tag"
  type        = string
}

# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# These variables are configurable, but have reasonable defaults set.
# ---------------------------------------------------------------------------------------------------------------------
variable "datadog_site" {
  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "image" {
  description = "Vector image source"
  type        = string
  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
}

variable "replicas" {
  description = "Number of Vector aggregator replicas"
  type        = number
  default     = 3
}

variable "audit_drop_patterns" {
  description = "Regex patterns; matching log message bodies are dropped"
  type        = list(string)
  default = [
    "Audit\\s-\\s.+",
    "\"level\":\"audit\"",
  ]
}

variable "datadog_api_key_vault_path" {
  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
  type        = string
  default     = "secret/data/saas/infrastructure/datadog-api-key"
}

variable "external_secrets_cluster_store_name" {
  description = "Name of the ClusterSecretStore created by external-secrets module"
  type        = string
  default     = "vault"
}

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_dry_run" {
  description = "When true (and email_redaction_enabled is false), tag logs that would have been redacted with vector_email_detected=true instead of replacing. Useful for non-prod visibility without masking."
  type        = bool
  default     = false
}

variable "email_redaction_replacement" {
  description = "String that replaces matched email addresses"
  type        = string
  default     = "[REDACTED-EMAIL]"
}

variable "additional_config" {
  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
  type        = any
  default     = null
}

variable "resources_requests" {
  description = "Pod resource requests"
  type        = map(string)
  default = {
    cpu    = "200m"
    memory = "256Mi"
  }
}

variable "resources_limits" {
  description = "Pod resource limits"
  type        = map(string)
  default = {
    cpu    = "2"
    memory = "1Gi"
  }
}

variable "namespace" {
  description = "Kubernetes namespace for Vector"
  type        = string
  default     = "vector"
}

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}


1	terraform {
2	  required_version = ">= 1.9.0"
3
4	  required_providers {
5	    kubernetes = {
6	      source  = "hashicorp/kubernetes"
7	      version = ">= 2.20"
8	    }
9	    helm = {
10	      source  = "hashicorp/helm"
11	      version = ">= 2.10"
12	    }
13	  }
14	}
15
16	locals {
17	  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
18	    " && ",
19	    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
20	  )
21
22	  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
23	    "${path.module}/templates/values.yaml.tpl",
24	    {
25	      replicas                    = var.replicas
26	      requests_cpu                = var.resources_requests.cpu
27	      requests_memory             = var.resources_requests.memory
28	      limits_cpu                  = var.resources_limits.cpu
29	      limits_memory               = var.resources_limits.memory
30	      datadog_site                = var.datadog_site
31	      audit_drop_condition        = local.audit_drop_condition
32	      email_redaction_enabled     = var.email_redaction_enabled
33	      email_redaction_dry_run     = var.email_redaction_dry_run
34	      email_redaction_replacement = var.email_redaction_replacement
35	      service_monitor_enabled     = var.service_monitor_enabled
36	    },
37	  )
38	}
39
40	resource "kubernetes_namespace_v1" "vector" {
41	  metadata {
42	    name = var.namespace
43	    labels = {
44	      "app.kubernetes.io/name"      = "vector"
45	      "app.kubernetes.io/component" = "telemetry-aggregator"
46	    }
47	  }
48	}
49
50	resource "kubernetes_manifest" "datadog_external_secret" {
51	  manifest = {
52	    apiVersion = "external-secrets.io/v1"
53	    kind       = "ExternalSecret"
54	    metadata = {
55	      name      = "vector-datadog"
56	      namespace = kubernetes_namespace_v1.vector.metadata[0].name
57	    }
58	    spec = {
59	      refreshInterval = "1h"
60	      secretStoreRef = {
61	        kind = "ClusterSecretStore"
62	        name = var.external_secrets_cluster_store_name
63	      }
64	      target = {
65	        name           = "vector-datadog"
66	        creationPolicy = "Owner"
67	      }
68	      data = [{
69	        secretKey = "api_key"
70	        remoteRef = {
71	          key      = var.datadog_api_key_vault_path
72	          property = "apiKey"
73	        }
74	      }]
75	    }
76	  }
77
78	  wait {
79	    fields = {
80	      "status.conditions[0].type"   = "Ready"
81	      "status.conditions[0].status" = "True"
82	    }
83	  }
84
85	  timeouts {
86	    create = "5m"
87	    update = "5m"
88	  }
89	}
90
91	resource "helm_release" "vector" {
92	  name       = "vector"
93	  namespace  = kubernetes_namespace_v1.vector.metadata[0].name
94	  repository = "https://helm.vector.dev"
95	  chart      = "vector"
96	  version    = var.chart_version
97
98	  values = [local.rendered_values]
99
100	  set {
101	    name  = "image.repository"
102	    value = var.image
103	  }
104	  set {
105	    name  = "image.tag"
106	    value = var.image_tag_version
107	  }
108
109	  depends_on = [
110	    kubernetes_manifest.datadog_external_secret,
111	  ]
112	}
113
114	module "netpol" {
115	  source = "crow.infrastructure.immuta.io/internal/security-network-policy/kubernetes"
116	  version = "1.1.0"
117
118	  name             = "vector-netpol"
119	  namespace        = var.namespace
120	  network_policies = local.netpols
121
122	  depends_on = [
123	    kubernetes_namespace_v1.vector,
124	  ]
125	}
126

terraform {
  required_version = ">= 1.9.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.10"
    }
  }
}

locals {
  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
  )

  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
    "${path.module}/templates/values.yaml.tpl",
    {
      replicas                    = var.replicas
      requests_cpu                = var.resources_requests.cpu
      requests_memory             = var.resources_requests.memory
      limits_cpu                  = var.resources_limits.cpu
      limits_memory               = var.resources_limits.memory
      datadog_site                = var.datadog_site
      audit_drop_condition        = local.audit_drop_condition
      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_dry_run     = var.email_redaction_dry_run
      email_redaction_replacement = var.email_redaction_replacement
      service_monitor_enabled     = var.service_monitor_enabled
    },
  )
}

resource "kubernetes_namespace_v1" "vector" {
  metadata {
    name = var.namespace
    labels = {
      "app.kubernetes.io/name"      = "vector"
      "app.kubernetes.io/component" = "telemetry-aggregator"
    }
  }
}

resource "kubernetes_manifest" "datadog_external_secret" {
  manifest = {
    apiVersion = "external-secrets.io/v1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "vector-datadog"
      namespace = kubernetes_namespace_v1.vector.metadata[0].name
    }
    spec = {
      refreshInterval = "1h"
      secretStoreRef = {
        kind = "ClusterSecretStore"
        name = var.external_secrets_cluster_store_name
      }
      target = {
        name           = "vector-datadog"
        creationPolicy = "Owner"
      }
      data = [{
        secretKey = "api_key"
        remoteRef = {
          key      = var.datadog_api_key_vault_path
          property = "apiKey"
        }
      }]
    }
  }

  wait {
    fields = {
      "status.conditions[0].type"   = "Ready"
      "status.conditions[0].status" = "True"
    }
  }

  timeouts {
    create = "5m"
    update = "5m"
  }
}

resource "helm_release" "vector" {
  name       = "vector"
  namespace  = kubernetes_namespace_v1.vector.metadata[0].name
  repository = "https://helm.vector.dev"
  chart      = "vector"
  version    = var.chart_version

  values = [local.rendered_values]

  set {
    name  = "image.repository"
    value = var.image
  }
  set {
    name  = "image.tag"
    value = var.image_tag_version
  }

  depends_on = [
    kubernetes_manifest.datadog_external_secret,
  ]
}

module "netpol" {
  source = "crow.infrastructure.immuta.io/internal/security-network-policy/kubernetes"
  version = "1.1.0"

  name             = "vector-netpol"
  namespace        = var.namespace
  network_policies = local.netpols

  depends_on = [
    kubernetes_namespace_v1.vector,
  ]
}


Let me also look at the test inputs to understand the log data structure.


1	{"message": "user alice@example.com signed in"}
2	{"message": "no email here"}
3	{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk"}
4	{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}}
5	{"message": "json blob {\"email\":\"x@y.com\"}"}
6

{"message": "user alice@example.com signed in"}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}}
{"message": "json blob {\"email\":\"x@y.com\"}"}


1	email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
2
3	if is_string(.message) {
4	  .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
5	}
6
7	if is_object(.attributes) {
8	  attrs, _err = object(.attributes)
9	  if is_object(attrs) {
10	    .attributes = map_values(attrs) -> |v| {
11	      if is_string(v) { replace(string!(v), email_re, "[REDACTED-EMAIL]") } else { v }
12	    }
13	  }
14	}
15	.
16

email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

if is_string(.message) {
  .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
}

if is_object(.attributes) {
  attrs, _err = object(.attributes)
  if is_object(attrs) {
    .attributes = map_values(attrs) -> |v| {
      if is_string(v) { replace(string!(v), email_re, "[REDACTED-EMAIL]") } else { v }
    }
  }
}
.


1	#!/usr/bin/env bash
2	set -euo pipefail
3	DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
4	ACTUAL="$DIR/.actual.ndjson"
5	vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
6	diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
7	echo "redact_emails_logs VRL OK"
8

#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
echo "redact_emails_logs VRL OK"


1	{"message": "user [REDACTED-EMAIL] signed in"}
2	{"message": "no email here"}
3	{"message": "multiple [REDACTED-EMAIL] and [REDACTED-EMAIL]"}
4	{"attributes": {"request.id": "abc-123", "user.email": "[REDACTED-EMAIL]"}, "message": "ok"}
5	{"message": "json blob {\"email\":\"[REDACTED-EMAIL]\"}"}
6

{"message": "user [REDACTED-EMAIL] signed in"}
{"message": "no email here"}
{"message": "multiple [REDACTED-EMAIL] and [REDACTED-EMAIL]"}
{"attributes": {"request.id": "abc-123", "user.email": "[REDACTED-EMAIL]"}, "message": "ok"}
{"message": "json blob {\"email\":\"[REDACTED-EMAIL]\"}"}


Let me check the snapshot and the examples to understand the full picture.


1	role: Aggregator
2	replicas: 3
3	podAnnotations:
4	  reloader.stakater.com/auto: "true"
5	service:
6	  enabled: true
7	  type: ClusterIP
8	  ports:
9	    - name: otlp-grpc
10	      port: 4317
11	      protocol: TCP
12	    - name: otlp-http
13	      port: 4318
14	      protocol: TCP
15	env:
16	  - name: DATADOG_API_KEY
17	    valueFrom:
18	      secretKeyRef:
19	        name: vector-datadog
20	        key: api_key
21	resources:
22	  requests:
23	    cpu: 200m
24	    memory: 256Mi
25	  limits:
26	    cpu: "2"
27	    memory: 1Gi
28	customConfig:
29	  data_dir: /vector-data-dir
30	  api:
31	    enabled: true
32	    address: 127.0.0.1:8686
33	  sources:
34	    otlp_in:
35	      type: opentelemetry
36	      grpc:
37	        address: 0.0.0.0:4317
38	      http:
39	        address: 0.0.0.0:4318
40	  transforms: {}
41	  sinks:
42	    datadog_logs:
43	      type: datadog_logs
44	      inputs:
45	        - otlp_in.logs
46	      default_api_key: ${DATADOG_API_KEY}
47	      site: datadoghq.com
48	    datadog_traces:
49	      type: datadog_traces
50	      inputs:
51	        - otlp_in.traces
52	      default_api_key: ${DATADOG_API_KEY}
53	      site: datadoghq.com
54	    datadog_metrics:
55	      type: datadog_metrics
56	      inputs:
57	        - otlp_in.metrics
58	      default_api_key: ${DATADOG_API_KEY}
59	      site: datadoghq.com
60

role: Aggregator
replicas: 3
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: "2"
    memory: 1Gi
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 127.0.0.1:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
  transforms: {}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - otlp_in.logs
      default_api_key: ${DATADOG_API_KEY}
      site: datadoghq.com
    datadog_traces:
      type: datadog_traces
      inputs:
        - otlp_in.traces
      default_api_key: ${DATADOG_API_KEY}
      site: datadoghq.com
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - otlp_in.metrics
      default_api_key: ${DATADOG_API_KEY}
      site: datadoghq.com


1	module "vector" {
2	  source = "../.."
3
4	  chart_version              = "0.36.0"
5	  image_tag_version          = "0.40.0-distroless-libc"
6	  datadog_site               = "datadoghq.com"
7	  datadog_api_key_vault_path = "secret/data/datadog/api-key"
8
9	  external_secrets_cluster_store_name = "vault-cluster-store"
10	}
11

module "vector" {
  source = "../.."

  chart_version              = "0.36.0"
  image_tag_version          = "0.40.0-distroless-libc"
  datadog_site               = "datadoghq.com"
  datadog_api_key_vault_path = "secret/data/datadog/api-key"

  external_secrets_cluster_store_name = "vault-cluster-store"
}


Now I have a clear picture of the setup. Let me implement the bearer token redaction filter following the same pattern as email redaction.


1	{"message": "user alice@example.com signed in", "ddtags": "env:prod,service:api", "service": "api", "hostname": "ip-10-0-1-23"}
2	{"message": "no email here", "ddtags": "env:prod", "service": "worker", "hostname": "ip-10-0-1-24"}
3	{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "service": "api", "hostname": "ip-10-0-1-25"}
4	{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "ddtags": "env:staging", "service": "auth", "hostname": "ip-10-0-2-99"}
5	{"message": "json blob {\"email\":\"x@y.com\"}", "service": "api", "hostname": "ip-10-0-1-50"}
6

{"message": "user alice@example.com signed in", "ddtags": "env:prod,service:api", "service": "api", "hostname": "ip-10-0-1-23"}
{"message": "no email here", "ddtags": "env:prod", "service": "worker", "hostname": "ip-10-0-1-24"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "service": "api", "hostname": "ip-10-0-1-25"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "ddtags": "env:staging", "service": "auth", "hostname": "ip-10-0-2-99"}
{"message": "json blob {\"email\":\"x@y.com\"}", "service": "api", "hostname": "ip-10-0-1-50"}


Good, I have all the context I need. Here's the plan: add a `redact_bearer_tokens_logs` VRL transform that filters by the same conditions as the SDS query (env:prod, specific services, excluding specific message patterns), and wire it into the transform chain. I'll follow the same pattern as email redaction.


Edit

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}

variable "bearer_token_redaction_enabled" {
  description = "Toggle bearer token redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "bearer_token_redaction_replacement" {
  description = "String that replaces matched bearer tokens (replaces the full 'Bearer <token>' match)"
  type        = string
  default     = "Bearer [REDACTED-BEARER-TOKEN]"
}

variable "bearer_token_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply bearer token redaction to"
  type        = list(string)
  default     = ["service", "secure-metadata-database"]
}

variable "bearer_token_redaction_exclusion_patterns" {
  description = "Skip bearer token redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default = [
    "internal-webhook",
    "starting crawl",
    "Databricks automatic subscription",
    "audit/enrich",
    "Background Job",
    "community-organization-id:immuta",
    "pg - pool.connect",
  ]
}


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/variables.tf has been updated successfully. (file state is current in your context — no need to Read it back)

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}

variable "bearer_token_redaction_enabled" {
  description = "Toggle bearer token redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "bearer_token_redaction_replacement" {
  description = "String that replaces matched bearer tokens (replaces the full 'Bearer <token>' match)"
  type        = string
  default     = "Bearer [REDACTED-BEARER-TOKEN]"
}

variable "bearer_token_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply bearer token redaction to"
  type        = list(string)
  default     = ["service", "secure-metadata-database"]
}

variable "bearer_token_redaction_exclusion_patterns" {
  description = "Skip bearer token redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default = [
    "internal-webhook",
    "starting crawl",
    "Databricks automatic subscription",
    "audit/enrich",
    "Background Job",
    "community-organization-id:immuta",
    "pg - pool.connect",
  ]
}

# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# These variables must be set.
# ---------------------------------------------------------------------------------------------------------------------

variable "chart_version" {
  description = "Vector helm chart version pin"
  type        = string
}

variable "image_tag_version" {
  description = "Vector image tag"
  type        = string
}

# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# These variables are configurable, but have reasonable defaults set.
# ---------------------------------------------------------------------------------------------------------------------
variable "datadog_site" {
  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "image" {
  description = "Vector image source"
  type        = string
  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
}

variable "replicas" {
  description = "Number of Vector aggregator replicas"
  type        = number
  default     = 3
}

variable "audit_drop_patterns" {
  description = "Regex patterns; matching log message bodies are dropped"
  type        = list(string)
  default = [
    "Audit\\s-\\s.+",
    "\"level\":\"audit\"",
  ]
}

variable "datadog_api_key_vault_path" {
  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
  type        = string
  default     = "secret/data/saas/infrastructure/datadog-api-key"
}

variable "external_secrets_cluster_store_name" {
  description = "Name of the ClusterSecretStore created by external-secrets module"
  type        = string
  default     = "vault"
}

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_dry_run" {
  description = "When true (and email_redaction_enabled is false), tag logs that would have been redacted with vector_email_detected=true instead of replacing. Useful for non-prod visibility without masking."
  type        = bool
  default     = false
}

variable "email_redaction_replacement" {
  description = "String that replaces matched email addresses"
  type        = string
  default     = "[REDACTED-EMAIL]"
}

variable "additional_config" {
  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
  type        = any
  default     = null
}

variable "resources_requests" {
  description = "Pod resource requests"
  type        = map(string)
  default = {
    cpu    = "200m"
    memory = "256Mi"
  }
}

variable "resources_limits" {
  description = "Pod resource limits"
  type        = map(string)
  default = {
    cpu    = "2"
    memory = "1Gi"
  }
}

variable "namespace" {
  description = "Kubernetes namespace for Vector"
  type        = string
  default     = "vector"
}

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}

+  description = "Toggle bearer token redact VRL transform on logs"

+  description = "String that replaces matched bearer tokens (replaces the full 'Bearer <token>' match)"

+  default     = "Bearer [REDACTED-BEARER-TOKEN]"

+  description = "Service names (.service field on DD agent logs) to apply bearer token redaction to"

+  default     = ["service", "secure-metadata-database"]

+  description = "Skip bearer token redaction for any log whose message or ddtags contains one of these strings"

+    "Databricks automatic subscription",

+    "community-organization-id:immuta",


Now update `main.tf` to add the new locals and template variables:


locals {
  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
  )

  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
    "${path.module}/templates/values.yaml.tpl",
    {
      replicas                    = var.replicas
      requests_cpu                = var.resources_requests.cpu
      requests_memory             = var.resources_requests.memory
      limits_cpu                  = var.resources_limits.cpu
      limits_memory               = var.resources_limits.memory
      datadog_site                = var.datadog_site
      audit_drop_condition        = local.audit_drop_condition
      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_dry_run     = var.email_redaction_dry_run
      email_redaction_replacement = var.email_redaction_replacement
      service_monitor_enabled     = var.service_monitor_enabled
    },
  )
}

locals {
  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
  )

  bearer_token_exclusion_condition = length(var.bearer_token_redaction_exclusion_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.bearer_token_redaction_exclusion_patterns : "!contains(combined, \"${p}\")"],
  )

  bearer_token_services_arr = "[\"${join("\", \"", var.bearer_token_redaction_services)}\"]"

  # The input to the bearer token transform is whatever the last logs transform in the chain is
  bearer_token_logs_input = (
    var.email_redaction_enabled ? "redact_emails_logs" :
    var.email_redaction_dry_run ? "tag_emails_logs" :
    "normalize_logs"
  )

  # The datadog_logs sink takes from the last transform in the full chain
  logs_sink_input = (
    var.bearer_token_redaction_enabled ? "redact_bearer_tokens_logs" :
    var.email_redaction_enabled ? "redact_emails_logs" :
    var.email_redaction_dry_run ? "tag_emails_logs" :
    "normalize_logs"
  )

  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
    "${path.module}/templates/values.yaml.tpl",
    {
      replicas                             = var.replicas
      requests_cpu                         = var.resources_requests.cpu
      requests_memory                      = var.resources_requests.memory
      limits_cpu                           = var.resources_limits.cpu
      limits_memory                        = var.resources_limits.memory
      datadog_site                         = var.datadog_site
      audit_drop_condition                 = local.audit_drop_condition
      email_redaction_enabled              = var.email_redaction_enabled
      email_redaction_dry_run              = var.email_redaction_dry_run
      email_redaction_replacement          = var.email_redaction_replacement
      service_monitor_enabled              = var.service_monitor_enabled
      bearer_token_redaction_enabled       = var.bearer_token_redaction_enabled
      bearer_token_redaction_replacement   = var.bearer_token_redaction_replacement
      bearer_token_services_arr            = local.bearer_token_services_arr
      bearer_token_exclusion_condition     = local.bearer_token_exclusion_condition
      bearer_token_logs_input              = local.bearer_token_logs_input
      logs_sink_input                      = local.logs_sink_input
    },
  )
}


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/main.tf has been updated successfully. (file state is current in your context — no need to Read it back)

locals {
  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
  )

  bearer_token_exclusion_condition = length(var.bearer_token_redaction_exclusion_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.bearer_token_redaction_exclusion_patterns : "!contains(combined, \"${p}\")"],
  )

  bearer_token_services_arr = "[\"${join("\", \"", var.bearer_token_redaction_services)}\"]"

  # The input to the bearer token transform is whatever the last logs transform in the chain is
  bearer_token_logs_input = (
    var.email_redaction_enabled ? "redact_emails_logs" :
    var.email_redaction_dry_run ? "tag_emails_logs" :
    "normalize_logs"
  )

  # The datadog_logs sink takes from the last transform in the full chain
  logs_sink_input = (
    var.bearer_token_redaction_enabled ? "redact_bearer_tokens_logs" :
    var.email_redaction_enabled ? "redact_emails_logs" :
    var.email_redaction_dry_run ? "tag_emails_logs" :
    "normalize_logs"
  )

  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
    "${path.module}/templates/values.yaml.tpl",
    {
      replicas                             = var.replicas
      requests_cpu                         = var.resources_requests.cpu
      requests_memory                      = var.resources_requests.memory
      limits_cpu                           = var.resources_limits.cpu
      limits_memory                        = var.resources_limits.memory
      datadog_site                         = var.datadog_site
      audit_drop_condition                 = local.audit_drop_condition
      email_redaction_enabled              = var.email_redaction_enabled
      email_redaction_dry_run              = var.email_redaction_dry_run
      email_redaction_replacement          = var.email_redaction_replacement
      service_monitor_enabled              = var.service_monitor_enabled
      bearer_token_redaction_enabled       = var.bearer_token_redaction_enabled
      bearer_token_redaction_replacement   = var.bearer_token_redaction_replacement
      bearer_token_services_arr            = local.bearer_token_services_arr
      bearer_token_exclusion_condition     = local.bearer_token_exclusion_condition
      bearer_token_logs_input              = local.bearer_token_logs_input
      logs_sink_input                      = local.logs_sink_input
    },
  )
}

terraform {
  required_version = ">= 1.9.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.10"
    }
  }
}

locals {
  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
  )

  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
    "${path.module}/templates/values.yaml.tpl",
    {
      replicas                    = var.replicas
      requests_cpu                = var.resources_requests.cpu
      requests_memory             = var.resources_requests.memory
      limits_cpu                  = var.resources_limits.cpu
      limits_memory               = var.resources_limits.memory
      datadog_site                = var.datadog_site
      audit_drop_condition        = local.audit_drop_condition
      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_dry_run     = var.email_redaction_dry_run
      email_redaction_replacement = var.email_redaction_replacement
      service_monitor_enabled     = var.service_monitor_enabled
    },
  )
}

resource "kubernetes_namespace_v1" "vector" {
  metadata {
    name = var.namespace
    labels = {
      "app.kubernetes.io/name"      = "vector"
      "app.kubernetes.io/component" = "telemetry-aggregator"
    }
  }
}

resource "kubernetes_manifest" "datadog_external_secret" {
  manifest = {
    apiVersion = "external-secrets.io/v1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "vector-datadog"
      namespace = kubernetes_namespace_v1.vector.metadata[0].name
    }
    spec = {
      refreshInterval = "1h"
      secretStoreRef = {
        kind = "ClusterSecretStore"
        name = var.external_secrets_cluster_store_name
      }
      target = {
        name           = "vector-datadog"
        creationPolicy = "Owner"
      }
      data = [{
        secretKey = "api_key"
        remoteRef = {
          key      = var.datadog_api_key_vault_path
          property = "apiKey"
        }
      }]
    }
  }

  wait {
    fields = {
      "status.conditions[0].type"   = "Ready"
      "status.conditions[0].status" = "True"
    }
  }

  timeouts {
    create = "5m"
    update = "5m"
  }
}

resource "helm_release" "vector" {
  name       = "vector"
  namespace  = kubernetes_namespace_v1.vector.metadata[0].name
  repository = "https://helm.vector.dev"
  chart      = "vector"
  version    = var.chart_version

  values = [local.rendered_values]

  set {
    name  = "image.repository"
    value = var.image
  }
  set {
    name  = "image.tag"
    value = var.image_tag_version
  }

  depends_on = [
    kubernetes_manifest.datadog_external_secret,
  ]
}

module "netpol" {
  source = "crow.infrastructure.immuta.io/internal/security-network-policy/kubernetes"
  version = "1.1.0"

  name             = "vector-netpol"
  namespace        = var.namespace
  network_policies = local.netpols

  depends_on = [
    kubernetes_namespace_v1.vector,
  ]
}

+  bearer_token_exclusion_condition = length(var.bearer_token_redaction_exclusion_patterns) == 0 ? "true" : join(

+    [for p in var.bearer_token_redaction_exclusion_patterns : "!contains(combined, \"${p}\")"],

+  bearer_token_services_arr = "[\"${join("\", \"", var.bearer_token_redaction_services)}\"]"

+  # The input to the bearer token transform is whatever the last logs transform in the chain is

+    var.email_redaction_enabled ? "redact_emails_logs" :

+    var.email_redaction_dry_run ? "tag_emails_logs" :

+  # The datadog_logs sink takes from the last transform in the full chain

+    var.bearer_token_redaction_enabled ? "redact_bearer_tokens_logs" :

-      replicas                    = var.replicas

-      requests_cpu                = var.resources_requests.cpu

-      requests_memory             = var.resources_requests.memory

-      limits_cpu                  = var.resources_limits.cpu

-      limits_memory               = var.resources_limits.memory

-      datadog_site                = var.datadog_site

-      audit_drop_condition        = local.audit_drop_condition

-      email_redaction_enabled     = var.email_redaction_enabled

-      email_redaction_dry_run     = var.email_redaction_dry_run

-      email_redaction_replacement = var.email_redaction_replacement

-      service_monitor_enabled     = var.service_monitor_enabled

+      replicas                             = var.replicas

+      requests_cpu                         = var.resources_requests.cpu

+      requests_memory                      = var.resources_requests.memory

+      limits_cpu                           = var.resources_limits.cpu

+      limits_memory                        = var.resources_limits.memory

+      datadog_site                         = var.datadog_site

+      audit_drop_condition                 = local.audit_drop_condition

+      email_redaction_enabled              = var.email_redaction_enabled

+      email_redaction_dry_run              = var.email_redaction_dry_run

+      email_redaction_replacement          = var.email_redaction_replacement

+      service_monitor_enabled              = var.service_monitor_enabled

+      bearer_token_redaction_enabled       = var.bearer_token_redaction_enabled

+      bearer_token_redaction_replacement   = var.bearer_token_redaction_replacement

+      bearer_token_services_arr            = local.bearer_token_services_arr

+      bearer_token_exclusion_condition     = local.bearer_token_exclusion_condition

+      bearer_token_logs_input              = local.bearer_token_logs_input

+      logs_sink_input                      = local.logs_sink_input


Now update the template — add the bearer token transform block and fix the sink input:


%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - normalize_logs
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "normalize_logs" }
      default_api_key: $${DATADOG_API_KEY}

%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - normalize_logs
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
%{ if bearer_token_redaction_enabled ~}
    redact_bearer_tokens_logs:
      type: remap
      inputs:
        - ${bearer_token_logs_input}
      source: |
        bearer_re = r'(?i)Bearer\s+[A-Za-z0-9+/._~-]+=*'
        svc = to_string(.service) ?? ""
        ddtags = to_string(.ddtags) ?? ""
        msg = if is_string(.message) { string!(.message) } else { "" }
        combined = msg + " " + ddtags
        target_services = ${bearer_token_services_arr}
        should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${bearer_token_exclusion_condition}
        if should_redact && is_string(.message) {
          .message = replace(string!(.message), bearer_re, "${bearer_token_redaction_replacement}")
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${logs_sink_input}
      default_api_key: $${DATADOG_API_KEY}


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/templates/values.yaml.tpl has been updated successfully. (file state is current in your context — no need to Read it back)

%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - normalize_logs
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
%{ if bearer_token_redaction_enabled ~}
    redact_bearer_tokens_logs:
      type: remap
      inputs:
        - ${bearer_token_logs_input}
      source: |
        bearer_re = r'(?i)Bearer\s+[A-Za-z0-9+/._~-]+=*'
        svc = to_string(.service) ?? ""
        ddtags = to_string(.ddtags) ?? ""
        msg = if is_string(.message) { string!(.message) } else { "" }
        combined = msg + " " + ddtags
        target_services = ${bearer_token_services_arr}
        should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${bearer_token_exclusion_condition}
        if should_redact && is_string(.message) {
          .message = replace(string!(.message), bearer_re, "${bearer_token_redaction_replacement}")
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${logs_sink_input}
      default_api_key: $${DATADOG_API_KEY}

role: Aggregator
replicas: ${replicas}
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: ${requests_cpu}
    memory: ${requests_memory}
  limits:
    cpu: "${limits_cpu}"
    memory: ${limits_memory}
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector

%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 0.0.0.0:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
    normalize_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        if exists(.severity) {
          .status = downcase(to_string!(.severity))
          del(.severity)
        }
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - normalize_logs
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_string(.message) {
          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
        }
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - normalize_logs
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "normalize_logs" }
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_traces:
      type: datadog_traces
      inputs:
        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}

+%{ if bearer_token_redaction_enabled ~}

+        svc = to_string(.service) ?? ""

+        ddtags = to_string(.ddtags) ?? ""

+        msg = if is_string(.message) { string!(.message) } else { "" }

+        target_services = ${bearer_token_services_arr}

+        should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${bearer_token_exclusion_condition}

+        if should_redact && is_string(.message) {

+          .message = replace(string!(.message), bearer_re, "${bearer_token_redaction_replacement}")

-        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "normalize_logs" }


Now create the test suite for the bearer token redaction:


Write

bearer_re = r'(?i)Bearer\s+[A-Za-z0-9+/._~-]+=*'
svc = to_string(.service) ?? ""
ddtags = to_string(.ddtags) ?? ""
msg = if is_string(.message) { string!(.message) } else { "" }
combined = msg + " " + ddtags
target_services = ["service", "secure-metadata-database"]
should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && !contains(combined, "internal-webhook") && !contains(combined, "starting crawl") && !contains(combined, "Databricks automatic subscription") && !contains(combined, "audit/enrich") && !contains(combined, "Background Job") && !contains(combined, "community-organization-id:immuta") && !contains(combined, "pg - pool.connect")
if should_redact && is_string(.message) {
  .message = replace(string!(.message), bearer_re, "Bearer [REDACTED-BEARER-TOKEN]")
}
.


File created successfully at: /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_bearer_tokens_logs/program.vrl (file state is current in your context — no need to Read it back)

bearer_re = r'(?i)Bearer\s+[A-Za-z0-9+/._~-]+=*'
svc = to_string(.service) ?? ""
ddtags = to_string(.ddtags) ?? ""
msg = if is_string(.message) { string!(.message) } else { "" }
combined = msg + " " + ddtags
target_services = ["service", "secure-metadata-database"]
should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && !contains(combined, "internal-webhook") && !contains(combined, "starting crawl") && !contains(combined, "Databricks automatic subscription") && !contains(combined, "audit/enrich") && !contains(combined, "Background Job") && !contains(combined, "community-organization-id:immuta") && !contains(combined, "pg - pool.connect")
if should_redact && is_string(.message) {
  .message = replace(string!(.message), bearer_re, "Bearer [REDACTED-BEARER-TOKEN]")
}
.


{"message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "secure-metadata-database", "ddtags": "env:prod,service:secure-metadata-database"}
{"message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "other-service", "ddtags": "env:prod,service:other-service"}
{"message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:staging,service:service"}
{"message": "internal-webhook Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "starting crawl Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "Databricks automatic subscription Bearer token", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "audit/enrich Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "Background Job Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "community-organization-id:immuta Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "pg - pool.connect Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "No token here", "service": "service", "ddtags": "env:prod,service:service"}


File created successfully at: /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_bearer_tokens_logs/inputs.ndjson (file state is current in your context — no need to Read it back)

{"message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "secure-metadata-database", "ddtags": "env:prod,service:secure-metadata-database"}
{"message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "other-service", "ddtags": "env:prod,service:other-service"}
{"message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:staging,service:service"}
{"message": "internal-webhook Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "starting crawl Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "Databricks automatic subscription Bearer token", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "audit/enrich Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "Background Job Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "community-organization-id:immuta Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "pg - pool.connect Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service", "ddtags": "env:prod,service:service"}
{"message": "No token here", "service": "service", "ddtags": "env:prod,service:service"}


Now let me run the VRL program against the inputs to generate the correct expected output:


Bash

cd /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_bearer_tokens_logs && vector vrl -p program.vrl -i inputs.ndjson -q -o 2>&1

Run VRL program against inputs to generate expected output


2026-05-07T21:50:42.368919Z  INFO vector::app: Log level is enabled. level="info"
{ "ddtags": "env:prod,service:service", "message": "Authorization: Bearer [REDACTED-BEARER-TOKEN]", "service": "service" }
{ "ddtags": "env:prod,service:secure-metadata-database", "message": "Authorization: Bearer [REDACTED-BEARER-TOKEN]", "service": "secure-metadata-database" }
{ "ddtags": "env:prod,service:other-service", "message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "other-service" }
{ "ddtags": "env:staging,service:service", "message": "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service" }
{ "ddtags": "env:prod,service:service", "message": "internal-webhook Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service" }
{ "ddtags": "env:prod,service:service", "message": "starting crawl Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service" }
{ "ddtags": "env:prod,service:service", "message": "Databricks automatic subscription Bearer token", "service": "service" }
{ "ddtags": "env:prod,service:service", "message": "audit/enrich Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service" }
{ "ddtags": "env:prod,service:service", "message": "Background Job Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service" }
{ "ddtags": "env:prod,service:service", "message": "community-organization-id:immuta Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service" }
{ "ddtags": "env:prod,service:service", "message": "pg - pool.connect Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "service": "service" }
{ "ddtags": "env:prod,service:service", "message": "No token here", "service": "service" }


The logic is correct — tokens are redacted for matching service+env, untouched for wrong service, wrong env, or exclusion pattern matches. Now capture the expected output:


vector vrl -p program.vrl -i inputs.ndjson -q -o | jq -c . > expected.ndjson && cat expected.ndjson

Generate expected.ndjson with compact JSON


2026-05-07T21:50:54.195253Z  INFO vector::app: Log level is enabled. level="info"
{"ddtags":"env:prod,service:service","message":"Authorization: Bearer [REDACTED-BEARER-TOKEN]","service":"service"}
{"ddtags":"env:prod,service:secure-metadata-database","message":"Authorization: Bearer [REDACTED-BEARER-TOKEN]","service":"secure-metadata-database"}
{"ddtags":"env:prod,service:other-service","message":"Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig","service":"other-service"}
{"ddtags":"env:staging,service:service","message":"Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig","service":"service"}
{"ddtags":"env:prod,service:service","message":"internal-webhook Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig","service":"service"}
{"ddtags":"env:prod,service:service","message":"starting crawl Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig","service":"service"}
{"ddtags":"env:prod,service:service","message":"Databricks automatic subscription Bearer token","service":"service"}
{"ddtags":"env:prod,service:service","message":"audit/enrich Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig","service":"service"}
{"ddtags":"env:prod,service:service","message":"Background Job Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig","service":"service"}
{"ddtags":"env:prod,service:service","message":"community-organization-id:immuta Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig","service":"service"}
{"ddtags":"env:prod,service:service","message":"pg - pool.connect Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig","service":"service"}
{"ddtags":"env:prod,service:service","message":"No token here","service":"service"}


Write

#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
echo "redact_bearer_tokens_logs VRL OK"


File created successfully at: /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/redact_bearer_tokens_logs/run.sh (file state is current in your context — no need to Read it back)

#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
echo "redact_bearer_tokens_logs VRL OK"


Bash

Run the bearer token VRL test


2026-05-07T21:51:35.649175Z  INFO vector::app: Log level is enabled. level="info"
redact_bearer_tokens_logs VRL OK


Test passes. Let me also run all existing VRL tests to make sure nothing is broken:


for f in /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/*/run.sh; do bash "$f"; done

Run all VRL tests


Error: Exit code 1
2026-05-07T21:51:58.879411Z  INFO vector::app: Log level is enabled. level="info"
drop_audit VRL OK
2026-05-07T21:51:59.083785Z  INFO vector::app: Log level is enabled. level="info"
redact_bearer_tokens_logs VRL OK
2026-05-07T21:51:59.287488Z  INFO vector::app: Log level is enabled. level="info"
redact_emails_dd_agent_logs VRL OK
2026-05-07T21:51:59.507565Z  INFO vector::app: Log level is enabled. level="info"
redact_emails_logs VRL OK
2026-05-07T21:51:59.703364Z  INFO vector::app: Log level is enabled. level="info"
redact_emails_metrics VRL OK
2026-05-07T21:51:59.902190Z  INFO vector::app: Log level is enabled. level="info"
redact_emails_traces VRL OK
2026-05-07T21:52:00.223814Z  INFO vector::app: Log level is enabled. level="info"
--- /dev/fd/63	2026-05-07 17:52:00
+++ /dev/fd/62	2026-05-07 17:52:00
@@ -1,6 +1,6 @@
-{"message":"user alice@example.com signed in","ddtags":"vector_redaction:true"}
+{"ddtags":"vector_redaction:true","message":"user alice@example.com signed in"}
 {"message":"no email here"}
-{"message":"multiple a@b.com and c.d+tag@sub.example.co.uk","ddtags":"vector_redaction:true"}
-{"message":"ok","attributes":{"user.email":"bob@x.io","request.id":"abc-123"},"ddtags":"vector_redaction:true"}
-{"message":"clean message","attributes":{"request.id":"abc-123"}}
-{"message":"user alice@example.com signed in","ddtags":"env:nonprod,service:myapp,vector_redaction:true"}
+{"ddtags":"vector_redaction:true","message":"multiple a@b.com and c.d+tag@sub.example.co.uk"}
+{"attributes":{"request.id":"abc-123","user.email":"bob@x.io"},"ddtags":"vector_redaction:true","message":"ok"}
+{"attributes":{"request.id":"abc-123"},"message":"clean message"}
+{"ddtags":"env:nonprod,service:myapp,vector_redaction:true","message":"user alice@example.com signed in"}


That `tag_emails_logs` failure is pre-existing (JSON key ordering - `jq -c .` sorts keys but the expected file has them in a different order). Let me verify it was already broken before my changes:


cd /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/tag_emails_logs && git diff expected.ndjson 2>/dev/null || echo "no git diff (file not tracked or no changes)"

Check if tag_emails_logs expected.ndjson was modified by me


(Bash completed with no output)


git -C /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector status -- tests/vrl/tag_emails_logs/

Check git status of tag_emails_logs test


On branch SRE-2806-more-sdd
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	tests/vrl/tag_emails_logs/.actual.ndjson
	tests/vrl/tag_emails_logs/.claude/

nothing added to commit but untracked files present (use "git add" to track)


Good — `tag_emails_logs/expected.ndjson` is unchanged by me (that test was already failing before). All my changes are clean.

Let me do a final review of the modified template to make sure the YAML is correct: