Got a full picture now. Here's what I'm seeing, Sir Charles of St Cheese:

**Vector module (v1.1.0)**

All the prior session work is present: OTel OTLP sources, DD Agent OPW intake on port 8282, email redaction transforms, drop_audit multi-input, the VRL test suite, and the netpol with 8282 added. The `helm_release` pulls from `https://helm.vector.dev`.

The child module source for netpol is:
```
git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0
```

The `.gitlab-ci.yml` uses private Docker image cache (`registry.ic-ops.net/cache-docker-hub/`) for both the Terraform and Vector images — that's consistent with a private registry.

**DD Agent module (v1.1.3)**

`logs_intake_url` is wired through to the `templatefile()` call correctly. Same netpol source pattern (git SSH to the network-policy module).

**One gap I noticed:** The `redact_emails_dd_agent_logs/run.sh` VRL test exists on disk but is **not listed in the `.gitlab-ci.yml` vrl-tests job** — the CI script stops at `redact_emails_metrics`. That test won't run in CI as-is.

---

One question: when you say you updated sources to point to the private terraform registry — I'm currently seeing git SSH URLs for the `module "netpol"` source in both `main.tf` files. Is that the change I should be looking at (i.e., is the private TF registry format different from git SSH and I'm reading a stale state), or are you referring to the Docker image registry in the CI?


afplay ~/.claude/sounds/mario/stop/stomp.wav 2>/dev/null || true


Working on routing Datadog Agent logs through Vector for email redaction via OPW intake. The gitlab-ci.yml is missing the redact_emails_dd_agent_logs VRL test — add it to unblock CI. (disable recaps in /config)


Network policy blocks host-network agent traffic to port 8282

High Severity

Port 8282 was added to the fromEntities: ["cluster"] ingress rule but not to the fromEntities: ["host"] rule. The Datadog agent runs with useHostNetwork: true, so its traffic carries the host (local node) or remote-node identity in Cilium — not cluster. The host rule only permits ports 4317, 4318, and 8686, so DD agent log traffic to Vector on port 8282 would be denied by the network policy, breaking the entire log-forwarding-through-Vector feature.


Clear bug, straightforward fix. Adding 8282 to the `host` rule now.


Edit

      {
        # kubelet health probes from the node.
        fromEntities = ["host"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8686" },
          ]
        }]
      },

      {
        # kubelet health probes and DD Agent (useHostNetwork: true) from the node.
        fromEntities = ["host"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
            { port = "8686" },
          ]
        }]
      },


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/netpol.tf has been updated successfully. (file state is current in your context — no need to Read it back)

      {
        # kubelet health probes and DD Agent (useHostNetwork: true) from the node.
        fromEntities = ["host"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
            { port = "8686" },
          ]
        }]
      },

locals {
  netpols = {
    "vector" = local.vector_netpol
  }
  vector_netpol = {
    endpointSelector = {
      matchLabels = {
        "app.kubernetes.io/name" = "vector"
      }
    }
    ingress = [
      {
        # OTLP from any cluster-internal sender (primarily otel-collector).
        # Port 8282 is the OPW-compatible Datadog Agent intake (datadog_agent source).
        fromEntities = ["cluster"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
          ]
        }]
      },
      {
        # Vector self-metrics scrape (Prometheus operator, Datadog Agent, etc.).
        fromEntities = ["cluster"]
        toPorts = [{
          ports = [{ port = "8686" }]
        }]
      },
      {
        # kubelet health probes from the node.
        fromEntities = ["host"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8686" },
          ]
        }]
      },
      {
        fromEntities = ["kube-apiserver"]
      }
    ]
    egress = [
      {
        # Datadog SaaS HTTPS for logs/traces/metrics sinks.
        toEntities = ["world"]
        toPorts = [{
          ports = [{ port = "443" }]
        }]
      },
      {
        toEntities = ["kube-apiserver"]
      },
      {
        # DNS resolution + service discovery (CoreDNS lives here).
        toEntities = ["cluster"]
      }
    ]
  }
}

-        # kubelet health probes from the node.

+        # kubelet health probes and DD Agent (useHostNetwork: true) from the node.


Done. The `host` entity rule now permits 4317, 4318, 8282, and 8686. Without this, every log the DD Agent tried to ship to Vector over port 8282 would have been silently dropped by Cilium — the entire OPW routing feature would have appeared to work at deploy time but produced nothing in Datadog.


Network policy blocks host-network agent traffic to port 8282  High Severity  Port 8282 was added to the fromEntities: ["cluster"] ingress rule but not to the fromEntities: ["host"] rule. The Datadog…


Review moved Terraform monitoring modules


is port 8686 even being used?


Good question. Let me check whether 8686 is actually exposed.


1	role: Aggregator
2	replicas: ${replicas}
3	podAnnotations:
4	  reloader.stakater.com/auto: "true"
5	service:
6	  enabled: true
7	  type: ClusterIP
8	  ports:
9	    - name: otlp-grpc
10	      port: 4317
11	      protocol: TCP
12	    - name: otlp-http
13	      port: 4318
14	      protocol: TCP
15	    - name: dd-agent
16	      port: 8282
17	      protocol: TCP
18	env:
19	  - name: DATADOG_API_KEY
20	    valueFrom:
21	      secretKeyRef:
22	        name: vector-datadog
23	        key: api_key
24	resources:
25	  requests:
26	    cpu: ${requests_cpu}
27	    memory: ${requests_memory}
28	  limits:
29	    cpu: "${limits_cpu}"
30	    memory: ${limits_memory}
31	podDisruptionBudget:
32	  enabled: true
33	  minAvailable: 2
34
35	topologySpreadConstraints:
36	  - maxSkew: 1
37	    topologyKey: topology.kubernetes.io/zone
38	    whenUnsatisfiable: ScheduleAnyway
39	    labelSelector:
40	      matchLabels:
41	        app.kubernetes.io/name: vector
42
43	affinity:
44	  podAntiAffinity:
45	    preferredDuringSchedulingIgnoredDuringExecution:
46	      - weight: 100
47	        podAffinityTerm:
48	          topologyKey: kubernetes.io/hostname
49	          labelSelector:
50	            matchLabels:
51	              app.kubernetes.io/name: vector
52
53	%{ if service_monitor_enabled ~}
54	serviceMonitor:
55	  enabled: true
56	%{ endif ~}
57	customConfig:
58	  data_dir: /vector-data-dir
59	  api:
60	    enabled: true
61	    address: 127.0.0.1:8686
62	  sources:
63	    otlp_in:
64	      type: opentelemetry
65	      grpc:
66	        address: 0.0.0.0:4317
67	      http:
68	        address: 0.0.0.0:4318
69	    dd_agent_in:
70	      type: datadog_agent
71	      address: 0.0.0.0:8282
72	      multiple_outputs: true
73	      disable_metrics: true
74	      disable_traces: true
75	      store_api_key: false
76	      acknowledgements:
77	        enabled: true
78	  transforms:
79	    drop_audit:
80	      type: filter
81	      inputs:
82	        - otlp_in.logs
83	        - dd_agent_in.logs
84	      condition: |
85	        msg = if is_string(.message) { string!(.message) } else { "" }
86	        ${audit_drop_condition}
87	%{ if email_redaction_enabled ~}
88	    redact_emails_logs:
89	      type: remap
90	      inputs:
91	        - drop_audit
92	      source: |
93	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
94	        if is_string(.message) {
95	          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
96	        }
97	        if is_object(.attributes) {
98	          attrs, _err = object(.attributes)
99	          if is_object(attrs) {
100	            .attributes = map_values(attrs) -> |v| {
101	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
102	            }
103	          }
104	        }
105	    redact_emails_traces:
106	      type: remap
107	      inputs:
108	        - otlp_in.traces
109	      source: |
110	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
111	        if is_object(.attributes) {
112	          attrs, _err = object(.attributes)
113	          if is_object(attrs) {
114	            .attributes = map_values(attrs) -> |v| {
115	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
116	            }
117	          }
118	        }
119	        if is_object(.resource) {
120	          res, _err = object(.resource)
121	          if is_object(res) && is_object(res.attributes) {
122	            rattrs, _err2 = object(res.attributes)
123	            if is_object(rattrs) {
124	              .resource.attributes = map_values(rattrs) -> |v| {
125	                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
126	              }
127	            }
128	          }
129	        }
130	    redact_emails_metrics:
131	      type: remap
132	      inputs:
133	        - otlp_in.metrics
134	      source: |
135	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
136	        if is_object(.attributes) {
137	          attrs, _err = object(.attributes)
138	          if is_object(attrs) {
139	            .attributes = map_values(attrs) -> |v| {
140	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
141	            }
142	          }
143	        }
144	        if is_object(.resource) {
145	          res, _err = object(.resource)
146	          if is_object(res) && is_object(res.attributes) {
147	            rattrs, _err2 = object(res.attributes)
148	            if is_object(rattrs) {
149	              .resource.attributes = map_values(rattrs) -> |v| {
150	                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
151	              }
152	            }
153	          }
154	        }
155	%{ endif ~}
156	  sinks:
157	    datadog_logs:
158	      type: datadog_logs
159	      inputs:
160	        - ${email_redaction_enabled ? "redact_emails_logs" : "drop_audit"}
161	      default_api_key: $${DATADOG_API_KEY}
162	      site: ${datadog_site}
163	    datadog_traces:
164	      type: datadog_traces
165	      inputs:
166	        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
167	      default_api_key: $${DATADOG_API_KEY}
168	      site: ${datadog_site}
169	    datadog_metrics:
170	      type: datadog_metrics
171	      inputs:
172	        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
173	      default_api_key: $${DATADOG_API_KEY}
174	      site: ${datadog_site}
175

role: Aggregator
replicas: ${replicas}
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: ${requests_cpu}
    memory: ${requests_memory}
  limits:
    cpu: "${limits_cpu}"
    memory: ${limits_memory}
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector

%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 127.0.0.1:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_string(.message) {
          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
        }
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${email_redaction_enabled ? "redact_emails_logs" : "drop_audit"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_traces:
      type: datadog_traces
      inputs:
        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}


No — port 8686 is dead in the netpol. The Vector API is bound to `127.0.0.1:8686` (line 61 of the template), and 8686 is not exposed in the Kubernetes Service `ports` list (which only has 4317, 4318, and 8282). Traffic from `cluster` or `host` entities can never reach it because it's localhost-only and has no Service entry to route through.

The `serviceMonitor` conditional would try to scrape it, but even that would fail at the network level — Prometheus would be hitting a Service/pod that doesn't expose the port.

Both the `cluster` and `host` netpol rules for 8686 are dead weight. Want me to remove them from the netpol?


yes remove 8686


      {
        # Vector self-metrics scrape (Prometheus operator, Datadog Agent, etc.).
        fromEntities = ["cluster"]
        toPorts = [{
          ports = [{ port = "8686" }]
        }]
      },
      {
        # kubelet health probes and DD Agent (useHostNetwork: true) from the node.
        fromEntities = ["host"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
            { port = "8686" },
          ]
        }]
      },

      {
        # kubelet health probes and DD Agent (useHostNetwork: true) from the node.
        fromEntities = ["host"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
          ]
        }]
      },


      {
        # kubelet health probes and DD Agent (useHostNetwork: true) from the node.
        fromEntities = ["host"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
          ]
        }]
      },

locals {
  netpols = {
    "vector" = local.vector_netpol
  }
  vector_netpol = {
    endpointSelector = {
      matchLabels = {
        "app.kubernetes.io/name" = "vector"
      }
    }
    ingress = [
      {
        # OTLP from any cluster-internal sender (primarily otel-collector).
        # Port 8282 is the OPW-compatible Datadog Agent intake (datadog_agent source).
        fromEntities = ["cluster"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
          ]
        }]
      },
      {
        # Vector self-metrics scrape (Prometheus operator, Datadog Agent, etc.).
        fromEntities = ["cluster"]
        toPorts = [{
          ports = [{ port = "8686" }]
        }]
      },
      {
        # kubelet health probes and DD Agent (useHostNetwork: true) from the node.
        fromEntities = ["host"]
        toPorts = [{
          ports = [
            { port = "4317" },
            { port = "4318" },
            { port = "8282" },
            { port = "8686" },
          ]
        }]
      },
      {
        fromEntities = ["kube-apiserver"]
      }
    ]
    egress = [
      {
        # Datadog SaaS HTTPS for logs/traces/metrics sinks.
        toEntities = ["world"]
        toPorts = [{
          ports = [{ port = "443" }]
        }]
      },
      {
        toEntities = ["kube-apiserver"]
      },
      {
        # DNS resolution + service discovery (CoreDNS lives here).
        toEntities = ["cluster"]
      }
    ]
  }
}

-        # Vector self-metrics scrape (Prometheus operator, Datadog Agent, etc.).


Done — removed the dedicated `cluster` rule for 8686 entirely and dropped it from the `host` rule. The ingress surface is now 4317/4318/8282 from `cluster`, the same three ports from `host`, and the `kube-apiserver` wildcard.


how to fix? ╰─$ git push origin SRE-2804-vector
To github.com:immuta/operations.git
 ! [rejected]        SRE-2804-vector -> SRE-2804-vector (fetch first)
error: failed to push some refs to 'github.com:immuta/operations.git'
hint: Updates were rejected because the remote contains work that you do not
hint: have locally. This is usually caused by another repository pushing to
hint: the same ref. If you want to integrate the remote changes, use
hint: 'git pull' before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.


Remote has commits you don't have locally. Run:

```
git pull --rebase origin SRE-2804-vector
```

That replays your local commits on top of whatever was pushed remotely. If there are conflicts, resolve them, then `git rebase --continue`. Once clean, `git push origin SRE-2804-vector` will go through.


<command-name>/usage</command-name>
            <command-message>usage</command-message>
            <command-args></command-args>


<local-command-stdout>Status dialog dismissed</local-command-stdout>


can we add a tag to logs that have been redacted by vector? The idea is that we don't want to redact in non-prod environments but still want to see if vector would have taken action


6	    endpointSelector = {
7	      matchLabels = {
8	        "app.kubernetes.io/name" = "vector"
9	      }
10	    }
11	    ingress = [
12	      {
13	        # OTLP from any cluster-internal sender (primarily otel-collector).
14	        fromEntities = ["cluster"]
15	        toPorts = [{
16	          ports = [
17	            { port = "4317" },
18	            { port = "4318" },
19	            { port = "8282" },
20	          ]
21	        }]


Yes, doable. The cleanest approach is a new `email_redaction_dry_run` bool variable (default false). When true, the `redact_emails_logs` transform runs the same regex but instead of replacing, tags the event with something like `vector_email_detected: true` — letting you search in Datadog for events that would have been redacted. You'd set `email_redaction_enabled = false, email_redaction_dry_run = true` in non-prod.

The one tradeoff: the attribute-scanning part is slightly less precise in dry-run — you can't mutate an outer variable from inside a VRL `map_values` closure, so attribute detection would use `encode_json(.attributes)` as a string to match against rather than walking keys individually. For the purpose of "did any attribute contain an email," that's equivalent.

Want me to implement it?


yes


Let me read the variables file before touching it.


1	# ---------------------------------------------------------------------------------------------------------------------
2	# REQUIRED PARAMETERS
3	# These variables must be set.
4	# ---------------------------------------------------------------------------------------------------------------------
5
6	variable "chart_version" {
7	  description = "Vector helm chart version pin"
8	  type        = string
9	}
10
11	variable "image_tag_version" {
12	  description = "Vector image tag"
13	  type        = string
14	}
15
16	# ---------------------------------------------------------------------------------------------------------------------
17	# OPTIONAL PARAMETERS
18	# These variables are configurable, but have reasonable defaults set.
19	# ---------------------------------------------------------------------------------------------------------------------
20	variable "datadog_site" {
21	  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
22	  type        = string
23	  default     = "datadoghq.com"
24	}
25
26	variable "image" {
27	  description = "Vector image source"
28	  type        = string
29	  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
30	}
31
32	variable "replicas" {
33	  description = "Number of Vector aggregator replicas"
34	  type        = number
35	  default     = 3
36	}
37
38	variable "audit_drop_patterns" {
39	  description = "Regex patterns; matching log message bodies are dropped"
40	  type        = list(string)
41	  default = [
42	    "Audit\\s-\\s.+",
43	    "\"level\":\"audit\"",
44	  ]
45	}
46
47	variable "datadog_api_key_vault_path" {
48	  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
49	  type        = string
50	  default     = "secret/data/saas/infrastructure/datadog-api-key"
51	}
52
53	variable "external_secrets_cluster_store_name" {
54	  description = "Name of the ClusterSecretStore created by external-secrets module"
55	  type        = string
56	  default     = "vault"
57	}
58
59	variable "email_redaction_enabled" {
60	  description = "Toggle the email-redact VRL transform across all signals"
61	  type        = bool
62	  default     = true
63	}
64
65	variable "email_redaction_replacement" {
66	  description = "String that replaces matched email addresses"
67	  type        = string
68	  default     = "[REDACTED-EMAIL]"
69	}
70
71	variable "additional_config" {
72	  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
73	  type        = any
74	  default     = null
75	}
76
77	variable "resources_requests" {
78	  description = "Pod resource requests"
79	  type        = map(string)
80	  default = {
81	    cpu    = "200m"
82	    memory = "256Mi"
83	  }
84	}
85
86	variable "resources_limits" {
87	  description = "Pod resource limits"
88	  type        = map(string)
89	  default = {
90	    cpu    = "2"
91	    memory = "1Gi"
92	  }
93	}
94
95	variable "namespace" {
96	  description = "Kubernetes namespace for Vector"
97	  type        = string
98	  default     = "vector"
99	}
100
101	variable "service_monitor_enabled" {
102	  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
103	  type        = bool
104	  default     = false
105	}
106

# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# These variables must be set.
# ---------------------------------------------------------------------------------------------------------------------

variable "chart_version" {
  description = "Vector helm chart version pin"
  type        = string
}

variable "image_tag_version" {
  description = "Vector image tag"
  type        = string
}

# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# These variables are configurable, but have reasonable defaults set.
# ---------------------------------------------------------------------------------------------------------------------
variable "datadog_site" {
  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "image" {
  description = "Vector image source"
  type        = string
  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
}

variable "replicas" {
  description = "Number of Vector aggregator replicas"
  type        = number
  default     = 3
}

variable "audit_drop_patterns" {
  description = "Regex patterns; matching log message bodies are dropped"
  type        = list(string)
  default = [
    "Audit\\s-\\s.+",
    "\"level\":\"audit\"",
  ]
}

variable "datadog_api_key_vault_path" {
  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
  type        = string
  default     = "secret/data/saas/infrastructure/datadog-api-key"
}

variable "external_secrets_cluster_store_name" {
  description = "Name of the ClusterSecretStore created by external-secrets module"
  type        = string
  default     = "vault"
}

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_replacement" {
  description = "String that replaces matched email addresses"
  type        = string
  default     = "[REDACTED-EMAIL]"
}

variable "additional_config" {
  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
  type        = any
  default     = null
}

variable "resources_requests" {
  description = "Pod resource requests"
  type        = map(string)
  default = {
    cpu    = "200m"
    memory = "256Mi"
  }
}

variable "resources_limits" {
  description = "Pod resource limits"
  type        = map(string)
  default = {
    cpu    = "2"
    memory = "1Gi"
  }
}

variable "namespace" {
  description = "Kubernetes namespace for Vector"
  type        = string
  default     = "vector"
}

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}


Let me check an existing VRL test for format reference before writing anything.


1	email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
2
3	if is_string(.message) {
4	  .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
5	}
6
7	if is_object(.attributes) {
8	  attrs, _err = object(.attributes)
9	  if is_object(attrs) {
10	    .attributes = map_values(attrs) -> |v| {
11	      if is_string(v) { replace(string!(v), email_re, "[REDACTED-EMAIL]") } else { v }
12	    }
13	  }
14	}
15	.
16

email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

if is_string(.message) {
  .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
}

if is_object(.attributes) {
  attrs, _err = object(.attributes)
  if is_object(attrs) {
    .attributes = map_values(attrs) -> |v| {
      if is_string(v) { replace(string!(v), email_re, "[REDACTED-EMAIL]") } else { v }
    }
  }
}
.


1	{"message": "user alice@example.com signed in"}
2	{"message": "no email here"}
3	{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk"}
4	{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}}
5	{"message": "json blob {\"email\":\"x@y.com\"}"}
6

{"message": "user alice@example.com signed in"}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}}
{"message": "json blob {\"email\":\"x@y.com\"}"}


1	#!/usr/bin/env bash
2	set -euo pipefail
3	DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
4	ACTUAL="$DIR/.actual.ndjson"
5	vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
6	diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
7	echo "redact_emails_logs VRL OK"
8

#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
echo "redact_emails_logs VRL OK"


Good, I have everything I need. Making all changes now — variables.tf, main.tf, the template, a new VRL test, and CI.


variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_dry_run" {
  description = "When true (and email_redaction_enabled is false), tag logs that would have been redacted with vector_email_detected=true instead of replacing. Useful for non-prod visibility without masking."
  type        = bool
  default     = false
}


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/variables.tf has been updated successfully. (file state is current in your context — no need to Read it back)

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_dry_run" {
  description = "When true (and email_redaction_enabled is false), tag logs that would have been redacted with vector_email_detected=true instead of replacing. Useful for non-prod visibility without masking."
  type        = bool
  default     = false
}

# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# These variables must be set.
# ---------------------------------------------------------------------------------------------------------------------

variable "chart_version" {
  description = "Vector helm chart version pin"
  type        = string
}

variable "image_tag_version" {
  description = "Vector image tag"
  type        = string
}

# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# These variables are configurable, but have reasonable defaults set.
# ---------------------------------------------------------------------------------------------------------------------
variable "datadog_site" {
  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "image" {
  description = "Vector image source"
  type        = string
  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
}

variable "replicas" {
  description = "Number of Vector aggregator replicas"
  type        = number
  default     = 3
}

variable "audit_drop_patterns" {
  description = "Regex patterns; matching log message bodies are dropped"
  type        = list(string)
  default = [
    "Audit\\s-\\s.+",
    "\"level\":\"audit\"",
  ]
}

variable "datadog_api_key_vault_path" {
  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
  type        = string
  default     = "secret/data/saas/infrastructure/datadog-api-key"
}

variable "external_secrets_cluster_store_name" {
  description = "Name of the ClusterSecretStore created by external-secrets module"
  type        = string
  default     = "vault"
}

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_replacement" {
  description = "String that replaces matched email addresses"
  type        = string
  default     = "[REDACTED-EMAIL]"
}

variable "additional_config" {
  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
  type        = any
  default     = null
}

variable "resources_requests" {
  description = "Pod resource requests"
  type        = map(string)
  default = {
    cpu    = "200m"
    memory = "256Mi"
  }
}

variable "resources_limits" {
  description = "Pod resource limits"
  type        = map(string)
  default = {
    cpu    = "2"
    memory = "1Gi"
  }
}

variable "namespace" {
  description = "Kubernetes namespace for Vector"
  type        = string
  default     = "vector"
}

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}

+  description = "When true (and email_redaction_enabled is false), tag logs that would have been redacted with vector_email_detected=true instead of replacing. Useful for non-prod visibility without masking."


      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_replacement = var.email_redaction_replacement
      service_monitor_enabled     = var.service_monitor_enabled

      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_dry_run     = var.email_redaction_dry_run
      email_redaction_replacement = var.email_redaction_replacement
      service_monitor_enabled     = var.service_monitor_enabled


<tool_use_error>File has been modified since read, either by the user or by a linter. Read it again before attempting to write it.</tool_use_error>

Error: File has been modified since read, either by the user or by a linter. Read it again before attempting to write it.


1	terraform {
2	  required_version = ">= 1.9.0"
3
4	  required_providers {
5	    kubernetes = {
6	      source  = "hashicorp/kubernetes"
7	      version = ">= 2.20"
8	    }
9	    helm = {
10	      source  = "hashicorp/helm"
11	      version = ">= 2.10"
12	    }
13	  }
14	}
15
16	locals {
17	  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
18	    " && ",
19	    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
20	  )
21
22	  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
23	    "${path.module}/templates/values.yaml.tpl",
24	    {
25	      replicas                    = var.replicas
26	      requests_cpu                = var.resources_requests.cpu
27	      requests_memory             = var.resources_requests.memory
28	      limits_cpu                  = var.resources_limits.cpu
29	      limits_memory               = var.resources_limits.memory
30	      datadog_site                = var.datadog_site
31	      audit_drop_condition        = local.audit_drop_condition
32	      email_redaction_enabled     = var.email_redaction_enabled
33	      email_redaction_replacement = var.email_redaction_replacement
34	      service_monitor_enabled     = var.service_monitor_enabled
35	    },
36	  )
37	}
38
39	resource "kubernetes_namespace_v1" "vector" {
40	  metadata {
41	    name = var.namespace
42	    labels = {
43	      "app.kubernetes.io/name"      = "vector"
44	      "app.kubernetes.io/component" = "telemetry-aggregator"
45	    }
46	  }
47	}
48
49	resource "kubernetes_manifest" "datadog_external_secret" {
50	  manifest = {
51	    apiVersion = "external-secrets.io/v1"
52	    kind       = "ExternalSecret"
53	    metadata = {
54	      name      = "vector-datadog"
55	      namespace = kubernetes_namespace_v1.vector.metadata[0].name
56	    }
57	    spec = {
58	      refreshInterval = "1h"
59	      secretStoreRef = {
60	        kind = "ClusterSecretStore"
61	        name = var.external_secrets_cluster_store_name
62	      }
63	      target = {
64	        name           = "vector-datadog"
65	        creationPolicy = "Owner"
66	      }
67	      data = [{
68	        secretKey = "api_key"
69	        remoteRef = {
70	          key      = var.datadog_api_key_vault_path
71	          property = "apiKey"
72	        }
73	      }]
74	    }
75	  }
76
77	  wait {
78	    fields = {
79	      "status.conditions[0].type"   = "Ready"
80	      "status.conditions[0].status" = "True"
81	    }
82	  }
83
84	  timeouts {
85	    create = "5m"
86	    update = "5m"
87	  }
88	}
89
90	resource "helm_release" "vector" {
91	  name       = "vector"
92	  namespace  = kubernetes_namespace_v1.vector.metadata[0].name
93	  repository = "https://helm.vector.dev"
94	  chart      = "vector"
95	  version    = var.chart_version
96
97	  values = [local.rendered_values]
98
99	  set {
100	    name  = "image.repository"
101	    value = var.image
102	  }
103	  set {
104	    name  = "image.tag"
105	    value = var.image_tag_version
106	  }
107
108	  depends_on = [
109	    kubernetes_manifest.datadog_external_secret,
110	  ]
111	}
112
113	module "netpol" {
114	  source = "crow.infrastructure.immuta.io/internal/security-network-policy/kubernetes"
115	  version = "1.1.0"
116
117	  name             = "vector-netpol"
118	  namespace        = var.namespace
119	  network_policies = local.netpols
120
121	  depends_on = [
122	    kubernetes_namespace_v1.vector,
123	  ]
124	}
125

terraform {
  required_version = ">= 1.9.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.10"
    }
  }
}

locals {
  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
  )

  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
    "${path.module}/templates/values.yaml.tpl",
    {
      replicas                    = var.replicas
      requests_cpu                = var.resources_requests.cpu
      requests_memory             = var.resources_requests.memory
      limits_cpu                  = var.resources_limits.cpu
      limits_memory               = var.resources_limits.memory
      datadog_site                = var.datadog_site
      audit_drop_condition        = local.audit_drop_condition
      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_replacement = var.email_redaction_replacement
      service_monitor_enabled     = var.service_monitor_enabled
    },
  )
}

resource "kubernetes_namespace_v1" "vector" {
  metadata {
    name = var.namespace
    labels = {
      "app.kubernetes.io/name"      = "vector"
      "app.kubernetes.io/component" = "telemetry-aggregator"
    }
  }
}

resource "kubernetes_manifest" "datadog_external_secret" {
  manifest = {
    apiVersion = "external-secrets.io/v1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "vector-datadog"
      namespace = kubernetes_namespace_v1.vector.metadata[0].name
    }
    spec = {
      refreshInterval = "1h"
      secretStoreRef = {
        kind = "ClusterSecretStore"
        name = var.external_secrets_cluster_store_name
      }
      target = {
        name           = "vector-datadog"
        creationPolicy = "Owner"
      }
      data = [{
        secretKey = "api_key"
        remoteRef = {
          key      = var.datadog_api_key_vault_path
          property = "apiKey"
        }
      }]
    }
  }

  wait {
    fields = {
      "status.conditions[0].type"   = "Ready"
      "status.conditions[0].status" = "True"
    }
  }

  timeouts {
    create = "5m"
    update = "5m"
  }
}

resource "helm_release" "vector" {
  name       = "vector"
  namespace  = kubernetes_namespace_v1.vector.metadata[0].name
  repository = "https://helm.vector.dev"
  chart      = "vector"
  version    = var.chart_version

  values = [local.rendered_values]

  set {
    name  = "image.repository"
    value = var.image
  }
  set {
    name  = "image.tag"
    value = var.image_tag_version
  }

  depends_on = [
    kubernetes_manifest.datadog_external_secret,
  ]
}

module "netpol" {
  source = "crow.infrastructure.immuta.io/internal/security-network-policy/kubernetes"
  version = "1.1.0"

  name             = "vector-netpol"
  namespace        = var.namespace
  network_policies = local.netpols

  depends_on = [
    kubernetes_namespace_v1.vector,
  ]
}


Now I can also see the private registry source — `crow.infrastructure.immuta.io/internal/security-network-policy/kubernetes` with a separate `version` attribute. Acknowledged.

Now making all the remaining changes in parallel.


      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_replacement = var.email_redaction_replacement

      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_dry_run     = var.email_redaction_dry_run
      email_redaction_replacement = var.email_redaction_replacement


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/main.tf has been updated successfully. (file state is current in your context — no need to Read it back)

      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_dry_run     = var.email_redaction_dry_run
      email_redaction_replacement = var.email_redaction_replacement

terraform {
  required_version = ">= 1.9.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.10"
    }
  }
}

locals {
  audit_drop_condition = length(var.audit_drop_patterns) == 0 ? "true" : join(
    " && ",
    [for p in var.audit_drop_patterns : "!match(msg, r'${p}')"],
  )

  rendered_values = var.additional_config != null ? yamlencode({ customConfig = var.additional_config }) : templatefile(
    "${path.module}/templates/values.yaml.tpl",
    {
      replicas                    = var.replicas
      requests_cpu                = var.resources_requests.cpu
      requests_memory             = var.resources_requests.memory
      limits_cpu                  = var.resources_limits.cpu
      limits_memory               = var.resources_limits.memory
      datadog_site                = var.datadog_site
      audit_drop_condition        = local.audit_drop_condition
      email_redaction_enabled     = var.email_redaction_enabled
      email_redaction_replacement = var.email_redaction_replacement
      service_monitor_enabled     = var.service_monitor_enabled
    },
  )
}

resource "kubernetes_namespace_v1" "vector" {
  metadata {
    name = var.namespace
    labels = {
      "app.kubernetes.io/name"      = "vector"
      "app.kubernetes.io/component" = "telemetry-aggregator"
    }
  }
}

resource "kubernetes_manifest" "datadog_external_secret" {
  manifest = {
    apiVersion = "external-secrets.io/v1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "vector-datadog"
      namespace = kubernetes_namespace_v1.vector.metadata[0].name
    }
    spec = {
      refreshInterval = "1h"
      secretStoreRef = {
        kind = "ClusterSecretStore"
        name = var.external_secrets_cluster_store_name
      }
      target = {
        name           = "vector-datadog"
        creationPolicy = "Owner"
      }
      data = [{
        secretKey = "api_key"
        remoteRef = {
          key      = var.datadog_api_key_vault_path
          property = "apiKey"
        }
      }]
    }
  }

  wait {
    fields = {
      "status.conditions[0].type"   = "Ready"
      "status.conditions[0].status" = "True"
    }
  }

  timeouts {
    create = "5m"
    update = "5m"
  }
}

resource "helm_release" "vector" {
  name       = "vector"
  namespace  = kubernetes_namespace_v1.vector.metadata[0].name
  repository = "https://helm.vector.dev"
  chart      = "vector"
  version    = var.chart_version

  values = [local.rendered_values]

  set {
    name  = "image.repository"
    value = var.image
  }
  set {
    name  = "image.tag"
    value = var.image_tag_version
  }

  depends_on = [
    kubernetes_manifest.datadog_external_secret,
  ]
}

module "netpol" {
  source = "crow.infrastructure.immuta.io/internal/security-network-policy/kubernetes"
  version = "1.1.0"

  name             = "vector-netpol"
  namespace        = var.namespace
  network_policies = local.netpols

  depends_on = [
    kubernetes_namespace_v1.vector,
  ]
}

datadog_site                = var.datadog_site

+      email_redaction_dry_run     = var.email_redaction_dry_run


%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${email_redaction_enabled ? "redact_emails_logs" : "drop_audit"}

%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .vector_email_detected = true
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "drop_audit" }


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/templates/values.yaml.tpl has been updated successfully. (file state is current in your context — no need to Read it back)

%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .vector_email_detected = true
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "drop_audit" }

role: Aggregator
replicas: ${replicas}
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: ${requests_cpu}
    memory: ${requests_memory}
  limits:
    cpu: "${limits_cpu}"
    memory: ${limits_memory}
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector

%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 127.0.0.1:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_string(.message) {
          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
        }
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${email_redaction_enabled ? "redact_emails_logs" : "drop_audit"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_traces:
      type: datadog_traces
      inputs:
        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}

+%{ if !email_redaction_enabled && email_redaction_dry_run ~}

+          detected = match(string!(.message), email_re)

+        if !detected && is_object(.attributes) {

+          detected = match(encode_json(.attributes), email_re)

+          .vector_email_detected = true

-        - ${email_redaction_enabled ? "redact_emails_logs" : "drop_audit"}

+        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "drop_audit" }


Now write the VRL test files and update CI.


Write

email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

detected = false

if is_string(.message) {
  detected = match(string!(.message), email_re)
}

if !detected && is_object(.attributes) {
  detected = match(encode_json(.attributes), email_re)
}

if detected {
  .vector_email_detected = true
}

.


{"message": "user alice@example.com signed in"}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}}
{"message": "clean message", "attributes": {"request.id": "abc-123"}}


{"message": "user alice@example.com signed in", "vector_email_detected": true}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "vector_email_detected": true}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "vector_email_detected": true}
{"message": "clean message", "attributes": {"request.id": "abc-123"}}


File created successfully at: /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/tag_emails_logs/program.vrl (file state is current in your context — no need to Read it back)

email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

detected = false

if is_string(.message) {
  detected = match(string!(.message), email_re)
}

if !detected && is_object(.attributes) {
  detected = match(encode_json(.attributes), email_re)
}

if detected {
  .vector_email_detected = true
}

.


#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
echo "tag_emails_logs VRL OK"


File created successfully at: /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/tag_emails_logs/inputs.ndjson (file state is current in your context — no need to Read it back)

{"message": "user alice@example.com signed in"}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}}
{"message": "clean message", "attributes": {"request.id": "abc-123"}}


File created successfully at: /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/tag_emails_logs/expected.ndjson (file state is current in your context — no need to Read it back)

{"message": "user alice@example.com signed in", "vector_email_detected": true}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "vector_email_detected": true}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "vector_email_detected": true}
{"message": "clean message", "attributes": {"request.id": "abc-123"}}


File created successfully at: /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/tag_emails_logs/run.sh (file state is current in your context — no need to Read it back)

#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ACTUAL="$DIR/.actual.ndjson"
vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
echo "tag_emails_logs VRL OK"


Bash

Make run.sh executable


(Bash completed with no output)


Edit

    - ./tests/vrl/redact_emails_metrics/run.sh
    - ./tests/vrl/tag_emails_logs/run.sh


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/.gitlab-ci.yml has been updated successfully. (file state is current in your context — no need to Read it back)

    - ./tests/vrl/redact_emails_metrics/run.sh
    - ./tests/vrl/tag_emails_logs/run.sh

stages:
  - lint
  - test

terraform-fmt:
  stage: lint
  image: registry.ic-ops.net/cache-docker-hub/hashicorp/terraform:1.9
  script:
    - terraform fmt -check -recursive

terraform-validate:
  stage: test
  image: registry.ic-ops.net/cache-docker-hub/hashicorp/terraform:1.9
  script:
    - cd examples/basic
    - terraform init -backend=false
    - terraform validate

vrl-tests:
  stage: test
  image: registry.ic-ops.net/cache-docker-hub/timberio/vector:latest-debian
  before_script:
    - apt-get update -qq && apt-get install -y -qq bash diffutils jq
  script:
    - ./tests/vrl/drop_audit/run.sh
    - ./tests/vrl/redact_emails_logs/run.sh
    - ./tests/vrl/redact_emails_traces/run.sh
    - ./tests/vrl/redact_emails_metrics/run.sh

+    - ./tests/vrl/tag_emails_logs/run.sh


Done. Summary of changes:

- **`variables.tf`** — new `email_redaction_dry_run` bool, default `false`
- **`main.tf`** — passes `email_redaction_dry_run` into `templatefile()`
- **`templates/values.yaml.tpl`** — adds a `tag_emails_logs` remap transform when `!email_redaction_enabled && email_redaction_dry_run`; updates the `datadog_logs` sink input to route through it in that mode
- **`tests/vrl/tag_emails_logs/`** — new VRL test covering message match, attribute match, and clean events
- **`.gitlab-ci.yml`** — runs the new test

Usage in non-prod:
```hcl
module "vector" {
  email_redaction_enabled = false
  email_redaction_dry_run = true
}
```

Logs that contain emails will get `vector_email_detected: true` in Datadog without any masking. Flip `email_redaction_enabled = true` (and drop `dry_run`) for prod.


Adding email redaction via Vector with a dry-run mode for non-prod visibility. Next: run the new VRL test locally to verify tag_emails_logs behaves correctly before committing. (disable recaps in /config)


how can i simulate a json log in a kubernetes pod to test the email redaction dry run


Run a one-shot pod that prints a JSON line to stdout — the DD Agent picks it up and routes it through Vector:

```bash
UUID=$(uuidgen | tr '[:upper:]' '[:lower:]')
kubectl run email-dry-run-test \
  --restart=Never \
  --rm \
  --image=busybox \
  -- sh -c "echo '{\"level\":\"info\",\"message\":\"dry-run-$UUID contact alice@example.com for help\"}'; sleep 2"
```

The `sleep 2` gives the agent a moment to tail the log before the pod exits and the container log file disappears.

**To verify it worked, check two places:**

1. **Vector metrics** — the `tag_emails_logs` component should tick up:
   ```bash
   kubectl exec -n vector deploy/vector -- \
     curl -s http://localhost:8686/metrics \
     | grep 'component_received_events_total{component_id="tag_emails_logs"'
   ```

2. **Datadog Logs UI** — search for `dry-run-$UUID` and check that `vector_email_detected:true` appears as a field on the event.

If the counter doesn't increment, the log either didn't make it through the DD Agent → Vector OPW path (check `dd_agent_in` counter instead) or the pod exited before the agent tailed it (increase the sleep).


this seems to be showing up as an attribute and not a tag? can we add it as a tag and also can we make it more generic? maybe vector_redaction:true instead of vector_email_detected


Two changes: rename to `vector_redaction:true` and write it into `ddtags` (comma-separated string) instead of as a field attribute. Updating the template, the VRL program, and the test expectations.


        if detected {
          .vector_email_detected = true
        }

        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }


        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }

role: Aggregator
replicas: ${replicas}
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: ${requests_cpu}
    memory: ${requests_memory}
  limits:
    cpu: "${limits_cpu}"
    memory: ${limits_memory}
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector

%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 127.0.0.1:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_string(.message) {
          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
        }
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .vector_email_detected = true
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "drop_audit" }
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_traces:
      type: datadog_traces
      inputs:
        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}

-          .vector_email_detected = true

+          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }


if detected {
  .vector_email_detected = true
}

if detected {
  .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
}


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/tag_emails_logs/program.vrl has been updated successfully. (file state is current in your context — no need to Read it back)

if detected {
  .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
}

email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'

detected = false

if is_string(.message) {
  detected = match(string!(.message), email_re)
}

if !detected && is_object(.attributes) {
  detected = match(encode_json(.attributes), email_re)
}

if detected {
  .vector_email_detected = true
}

.

+  .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }


Now update the test data — adding a case with existing `ddtags` to cover the append path.


Write

{"message": "user alice@example.com signed in"}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}}
{"message": "clean message", "attributes": {"request.id": "abc-123"}}
{"message": "user alice@example.com signed in", "ddtags": "env:nonprod,service:myapp"}


{"message": "user alice@example.com signed in", "ddtags": "vector_redaction:true"}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "ddtags": "vector_redaction:true"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "ddtags": "vector_redaction:true"}
{"message": "clean message", "attributes": {"request.id": "abc-123"}}
{"message": "user alice@example.com signed in", "ddtags": "env:nonprod,service:myapp,vector_redaction:true"}


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/tag_emails_logs/inputs.ndjson has been updated successfully. (file state is current in your context — no need to Read it back)

{"message": "user alice@example.com signed in"}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}}
{"message": "clean message", "attributes": {"request.id": "abc-123"}}
{"message": "user alice@example.com signed in", "ddtags": "env:nonprod,service:myapp"}

ok

clean message

+{"message": "user alice@example.com signed in", "ddtags": "env:nonprod,service:myapp"}


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/tests/vrl/tag_emails_logs/expected.ndjson has been updated successfully. (file state is current in your context — no need to Read it back)

{"message": "user alice@example.com signed in", "ddtags": "vector_redaction:true"}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "ddtags": "vector_redaction:true"}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "ddtags": "vector_redaction:true"}
{"message": "clean message", "attributes": {"request.id": "abc-123"}}
{"message": "user alice@example.com signed in", "ddtags": "env:nonprod,service:myapp,vector_redaction:true"}

-{"message": "user alice@example.com signed in", "vector_email_detected": true}

+{"message": "user alice@example.com signed in", "ddtags": "vector_redaction:true"}

no email here

-{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "vector_email_detected": true}

-{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "vector_email_detected": true}

+{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "ddtags": "vector_redaction:true"}

+{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "ddtags": "vector_redaction:true"}

clean message

+{"message": "user alice@example.com signed in", "ddtags": "env:nonprod,service:myapp,vector_redaction:true"}

{"message": "user alice@example.com signed in", "vector_email_detected": true}
{"message": "no email here"}
{"message": "multiple a@b.com and c.d+tag@sub.example.co.uk", "vector_email_detected": true}
{"message": "ok", "attributes": {"user.email": "bob@x.io", "request.id": "abc-123"}, "vector_email_detected": true}
{"message": "clean message", "attributes": {"request.id": "abc-123"}}


Done. The tag now lands in `ddtags` as `vector_redaction:true`, which Datadog surfaces as a proper tag you can filter/facet on. The append case (`env:nonprod,service:myapp,vector_redaction:true`) is also covered in the test.

In Datadog search you'd use `@ddtags:vector_redaction:true` to find affected logs.


Adding email redaction dry-run mode to the Vector Terraform module so non-prod can tag logs that would have been redacted without masking them. Run the VRL test suite locally to verify the new tag_emails_logs transform before committing. (disable recaps in /config)


why do i keep seeing the following log in vector? 2026-05-06T20:09:58.957542Z  WARN vector::sinks::datadog::logs::sink: Semantic meaning is defined, but the event path already exists. Renaming to not overwrite. meaning="severity" renamed="_RESERVED_severity"


This is a semantic meaning collision in Vector. Here's what's happening:

The `opentelemetry` source annotates the `.severity` field on log events with a semantic meaning called `severity`. The `datadog_logs` sink reads that semantic meaning and tries to write it to the Datadog status field — but since `.severity` already exists as a raw event field (put there by the OTel source), Vector refuses to overwrite it, renames the existing value to `_RESERVED_severity`, and logs the warning.

The fix is a small remap transform that normalizes severity before it hits the sink — map `.severity` → `.status` (Datadog's expected field for log level) and delete the original:

```vrl
if exists(.severity) {
  .status = downcase(to_string!(.severity))
  del(.severity)
}
```

This needs to sit on the path that all logs travel, which is a bit awkward right now because `drop_audit` is a filter (no VRL remap) and the redact/tag transforms are only conditionally present. The cleanest way to handle it is a dedicated `normalize_logs` remap transform right after `drop_audit`, with all downstream transforms/sinks pointing at `normalize_logs` instead of `drop_audit`.

Want me to add it?


yes add it


Let me read the current template state before editing.


1	role: Aggregator
2	replicas: ${replicas}
3	podAnnotations:
4	  reloader.stakater.com/auto: "true"
5	service:
6	  enabled: true
7	  type: ClusterIP
8	  ports:
9	    - name: otlp-grpc
10	      port: 4317
11	      protocol: TCP
12	    - name: otlp-http
13	      port: 4318
14	      protocol: TCP
15	    - name: dd-agent
16	      port: 8282
17	      protocol: TCP
18	env:
19	  - name: DATADOG_API_KEY
20	    valueFrom:
21	      secretKeyRef:
22	        name: vector-datadog
23	        key: api_key
24	resources:
25	  requests:
26	    cpu: ${requests_cpu}
27	    memory: ${requests_memory}
28	  limits:
29	    cpu: "${limits_cpu}"
30	    memory: ${limits_memory}
31	podDisruptionBudget:
32	  enabled: true
33	  minAvailable: 2
34
35	topologySpreadConstraints:
36	  - maxSkew: 1
37	    topologyKey: topology.kubernetes.io/zone
38	    whenUnsatisfiable: ScheduleAnyway
39	    labelSelector:
40	      matchLabels:
41	        app.kubernetes.io/name: vector
42
43	affinity:
44	  podAntiAffinity:
45	    preferredDuringSchedulingIgnoredDuringExecution:
46	      - weight: 100
47	        podAffinityTerm:
48	          topologyKey: kubernetes.io/hostname
49	          labelSelector:
50	            matchLabels:
51	              app.kubernetes.io/name: vector
52
53	%{ if service_monitor_enabled ~}
54	serviceMonitor:
55	  enabled: true
56	%{ endif ~}
57	customConfig:
58	  data_dir: /vector-data-dir
59	  api:
60	    enabled: true
61	    address: 127.0.0.1:8686
62	  sources:
63	    otlp_in:
64	      type: opentelemetry
65	      grpc:
66	        address: 0.0.0.0:4317
67	      http:
68	        address: 0.0.0.0:4318
69	    dd_agent_in:
70	      type: datadog_agent
71	      address: 0.0.0.0:8282
72	      multiple_outputs: true
73	      disable_metrics: true
74	      disable_traces: true
75	      store_api_key: false
76	      acknowledgements:
77	        enabled: true
78	  transforms:
79	    drop_audit:
80	      type: filter
81	      inputs:
82	        - otlp_in.logs
83	        - dd_agent_in.logs
84	      condition: |
85	        msg = if is_string(.message) { string!(.message) } else { "" }
86	        ${audit_drop_condition}
87	%{ if email_redaction_enabled ~}
88	    redact_emails_logs:
89	      type: remap
90	      inputs:
91	        - drop_audit
92	      source: |
93	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
94	        if is_string(.message) {
95	          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
96	        }
97	        if is_object(.attributes) {
98	          attrs, _err = object(.attributes)
99	          if is_object(attrs) {
100	            .attributes = map_values(attrs) -> |v| {
101	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
102	            }
103	          }
104	        }
105	    redact_emails_traces:
106	      type: remap
107	      inputs:
108	        - otlp_in.traces
109	      source: |
110	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
111	        if is_object(.attributes) {
112	          attrs, _err = object(.attributes)
113	          if is_object(attrs) {
114	            .attributes = map_values(attrs) -> |v| {
115	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
116	            }
117	          }
118	        }
119	        if is_object(.resource) {
120	          res, _err = object(.resource)
121	          if is_object(res) && is_object(res.attributes) {
122	            rattrs, _err2 = object(res.attributes)
123	            if is_object(rattrs) {
124	              .resource.attributes = map_values(rattrs) -> |v| {
125	                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
126	              }
127	            }
128	          }
129	        }
130	    redact_emails_metrics:
131	      type: remap
132	      inputs:
133	        - otlp_in.metrics
134	      source: |
135	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
136	        if is_object(.attributes) {
137	          attrs, _err = object(.attributes)
138	          if is_object(attrs) {
139	            .attributes = map_values(attrs) -> |v| {
140	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
141	            }
142	          }
143	        }
144	        if is_object(.resource) {
145	          res, _err = object(.resource)
146	          if is_object(res) && is_object(res.attributes) {
147	            rattrs, _err2 = object(res.attributes)
148	            if is_object(rattrs) {
149	              .resource.attributes = map_values(rattrs) -> |v| {
150	                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
151	              }
152	            }
153	          }
154	        }
155	%{ endif ~}
156	%{ if !email_redaction_enabled && email_redaction_dry_run ~}
157	    tag_emails_logs:
158	      type: remap
159	      inputs:
160	        - drop_audit
161	      source: |
162	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
163	        detected = false
164	        if is_string(.message) {
165	          detected = match(string!(.message), email_re)
166	        }
167	        if !detected && is_object(.attributes) {
168	          detected = match(encode_json(.attributes), email_re)
169	        }
170	        if detected {
171	          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
172	        }
173	%{ endif ~}
174	  sinks:
175	    datadog_logs:
176	      type: datadog_logs
177	      inputs:
178	        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "drop_audit" }
179	      default_api_key: $${DATADOG_API_KEY}
180	      site: ${datadog_site}
181	    datadog_traces:
182	      type: datadog_traces
183	      inputs:
184	        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
185	      default_api_key: $${DATADOG_API_KEY}
186	      site: ${datadog_site}
187	    datadog_metrics:
188	      type: datadog_metrics
189	      inputs:
190	        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
191	      default_api_key: $${DATADOG_API_KEY}
192	      site: ${datadog_site}
193

role: Aggregator
replicas: ${replicas}
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: ${requests_cpu}
    memory: ${requests_memory}
  limits:
    cpu: "${limits_cpu}"
    memory: ${limits_memory}
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector

%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 127.0.0.1:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_string(.message) {
          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
        }
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "drop_audit" }
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_traces:
      type: datadog_traces
      inputs:
        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}


Four things to change: add `normalize_logs` after `drop_audit`, then update the three places that reference `drop_audit` downstream.


Edit

    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - drop_audit

    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
    normalize_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        if exists(.severity) {
          .status = downcase(to_string!(.severity))
          del(.severity)
        }
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - normalize_logs


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/templates/values.yaml.tpl has been updated successfully. (file state is current in your context — no need to Read it back)

    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
    normalize_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        if exists(.severity) {
          .status = downcase(to_string!(.severity))
          del(.severity)
        }
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - normalize_logs

role: Aggregator
replicas: ${replicas}
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: ${requests_cpu}
    memory: ${requests_memory}
  limits:
    cpu: "${limits_cpu}"
    memory: ${limits_memory}
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector

%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 127.0.0.1:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
%{ if email_redaction_enabled ~}
    redact_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_string(.message) {
          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
        }
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
%{ endif ~}
%{ if !email_redaction_enabled && email_redaction_dry_run ~}
    tag_emails_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${ email_redaction_enabled ? "redact_emails_logs" : email_redaction_dry_run ? "tag_emails_logs" : "drop_audit" }
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_traces:
      type: datadog_traces
      inputs:
        - ${email_redaction_enabled ? "redact_emails_traces" : "otlp_in.traces"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${email_redaction_enabled ? "redact_emails_metrics" : "otlp_in.metrics"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}

+          .status = downcase(to_string!(.severity))


    tag_emails_logs:
      type: remap
      inputs:
        - drop_audit

    tag_emails_logs:
      type: remap
      inputs:
        - normalize_logs