    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/templates/values.yaml.tpl has been updated successfully. (file state is current in your context — no need to Read it back)

    redact_emails_traces:
      type: remap
      inputs:
        - otlp_in.traces
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }
    redact_emails_metrics:
      type: remap
      inputs:
        - otlp_in.metrics
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
        if is_object(.resource) {
          res, _err = object(.resource)
          if is_object(res) && is_object(res.attributes) {
            rattrs, _err2 = object(res.attributes)
            if is_object(rattrs) {
              .resource.attributes = map_values(rattrs) -> |v| {
                if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
              }
            }
          }
        }

-          attrs, _err = object(.attributes)

-            .attributes = map_values(attrs) -> |v| {

-              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }

-          res, _err = object(.resource)

-          if is_object(res) && is_object(res.attributes) {

-            rattrs, _err2 = object(res.attributes)

-              .resource.attributes = map_values(rattrs) -> |v| {


Edit

    datadog_traces:
      type: datadog_traces
      inputs:
        - ${email_do_redact ? "redact_emails_traces" : "otlp_in.traces"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${email_do_redact ? "redact_emails_metrics" : "otlp_in.metrics"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}


role: Aggregator
replicas: ${replicas}
podAnnotations:
  reloader.stakater.com/auto: "true"
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP
    - name: dd-agent
      port: 8282
      protocol: TCP
env:
  - name: DATADOG_API_KEY
    valueFrom:
      secretKeyRef:
        name: vector-datadog
        key: api_key
resources:
  requests:
    cpu: ${requests_cpu}
    memory: ${requests_memory}
  limits:
    cpu: "${limits_cpu}"
    memory: ${limits_memory}
podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vector

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: vector

%{ if service_monitor_enabled ~}
serviceMonitor:
  enabled: true
%{ endif ~}
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 0.0.0.0:8686
  sources:
    otlp_in:
      type: opentelemetry
      grpc:
        address: 0.0.0.0:4317
      http:
        address: 0.0.0.0:4318
    dd_agent_in:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      disable_metrics: true
      disable_traces: true
      store_api_key: false
      acknowledgements:
        enabled: true
  transforms:
    drop_audit:
      type: filter
      inputs:
        - otlp_in.logs
        - dd_agent_in.logs
      condition: |
        msg = if is_string(.message) { string!(.message) } else { "" }
        ${audit_drop_condition}
    normalize_logs:
      type: remap
      inputs:
        - drop_audit
      source: |
        if exists(.severity) {
          .status = downcase(to_string!(.severity))
          del(.severity)
        }
%{ if email_do_redact ~}
    redact_emails_logs:
      type: remap
      inputs:
        - normalize_logs
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        if is_string(.message) {
          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
        }
        if is_object(.attributes) {
          attrs, _err = object(.attributes)
          if is_object(attrs) {
            .attributes = map_values(attrs) -> |v| {
              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
            }
          }
        }
%{ endif ~}
%{ if email_do_tag ~}
    tag_emails_logs:
      type: remap
      inputs:
        - normalize_logs
      source: |
        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
        detected = false
        if is_string(.message) {
          detected = match(string!(.message), email_re)
        }
        if !detected && is_object(.attributes) {
          detected = match(encode_json(.attributes), email_re)
        }
        if detected {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
%{ if bearer_do_redact ~}
    redact_bearer_tokens_logs:
      type: remap
      inputs:
        - ${bearer_token_logs_input}
      source: |
        bearer_re = r'(?i)Bearer\s+[A-Za-z0-9+/._~-]+=*'
        svc = to_string(.service) ?? ""
        ddtags = to_string(.ddtags) ?? ""
        msg = if is_string(.message) { string!(.message) } else { "" }
        combined = msg + " " + ddtags
        target_services = ${bearer_token_services_arr}
        should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${bearer_token_exclusion_condition}
        if should_redact && is_string(.message) {
          .message = replace(string!(.message), bearer_re, "${bearer_token_redaction_replacement}")
        }
%{ endif ~}
%{ if bearer_do_tag ~}
    tag_bearer_tokens_logs:
      type: remap
      inputs:
        - ${bearer_token_logs_input}
      source: |
        bearer_re = r'(?i)Bearer\s+[A-Za-z0-9+/._~-]+=*'
        svc = to_string(.service) ?? ""
        ddtags = to_string(.ddtags) ?? ""
        msg = if is_string(.message) { string!(.message) } else { "" }
        combined = msg + " " + ddtags
        target_services = ${bearer_token_services_arr}
        would_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${bearer_token_exclusion_condition}
        if would_redact && is_string(.message) && match(string!(.message), bearer_re) {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
%{ if aws_key_do_redact ~}
    redact_aws_access_keys_logs:
      type: remap
      inputs:
        - ${aws_key_logs_input}
      source: |
        aws_key_re = r'(AKIA|ASIA|AROA|AIPA|ANPA|ANVA|APKA)[A-Z0-9]{16}'
        svc = to_string(.service) ?? ""
        ddtags = to_string(.ddtags) ?? ""
        msg = if is_string(.message) { string!(.message) } else { "" }
        combined = msg + " " + ddtags
        target_services = ${aws_key_services_arr}
        attr_excluded = ${aws_key_attr_exclusion_condition}
        should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${aws_key_string_exclusion_condition} && !attr_excluded
        if should_redact && is_string(.message) {
          .message = replace(string!(.message), aws_key_re, "${aws_access_key_redaction_replacement}")
        }
%{ endif ~}
%{ if aws_key_do_tag ~}
    tag_aws_access_keys_logs:
      type: remap
      inputs:
        - ${aws_key_logs_input}
      source: |
        aws_key_re = r'(AKIA|ASIA|AROA|AIPA|ANPA|ANVA|APKA)[A-Z0-9]{16}'
        svc = to_string(.service) ?? ""
        ddtags = to_string(.ddtags) ?? ""
        msg = if is_string(.message) { string!(.message) } else { "" }
        combined = msg + " " + ddtags
        target_services = ${aws_key_services_arr}
        attr_excluded = ${aws_key_attr_exclusion_condition}
        would_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${aws_key_string_exclusion_condition} && !attr_excluded
        if would_redact && is_string(.message) && match(string!(.message), aws_key_re) {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
%{ if jwt_do_redact ~}
    redact_jwts_logs:
      type: remap
      inputs:
        - ${jwt_logs_input}
      source: |
        jwt_re = r'eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'
        svc = to_string(.service) ?? ""
        ddtags = to_string(.ddtags) ?? ""
        msg = if is_string(.message) { string!(.message) } else { "" }
        combined = msg + " " + ddtags
        target_services = ${jwt_services_arr}
        should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${jwt_exclusion_condition}
        if should_redact && is_string(.message) {
          .message = replace(string!(.message), jwt_re, "${jwt_redaction_replacement}")
        }
%{ endif ~}
%{ if jwt_do_tag ~}
    tag_jwts_logs:
      type: remap
      inputs:
        - ${jwt_logs_input}
      source: |
        jwt_re = r'eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'
        svc = to_string(.service) ?? ""
        ddtags = to_string(.ddtags) ?? ""
        msg = if is_string(.message) { string!(.message) } else { "" }
        combined = msg + " " + ddtags
        target_services = ${jwt_services_arr}
        would_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${jwt_exclusion_condition}
        if would_redact && is_string(.message) && match(string!(.message), jwt_re) {
          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
        }
%{ endif ~}
  sinks:
    datadog_logs:
      type: datadog_logs
      inputs:
        - ${logs_sink_input}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_traces:
      type: datadog_traces
      inputs:
        - ${email_do_redact ? "redact_emails_traces" : "otlp_in.traces"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - ${email_do_redact ? "redact_emails_metrics" : "otlp_in.metrics"}
      default_api_key: $${DATADOG_API_KEY}
      site: ${datadog_site}

-        - ${email_do_redact ? "redact_emails_traces" : "otlp_in.traces"}

-      default_api_key: $${DATADOG_API_KEY}

-        - ${email_do_redact ? "redact_emails_metrics" : "otlp_in.metrics"}


can we remove traces and metrics from these? We want vector to only handle logs, metrics and traces will be handled through our existing opentelemetry collector deployment instead


Now fix the stale variable descriptions:


description = "When true, tag logs that would have been redacted with vector_bearer_token_detected:true instead of replacing. Takes priority over bearer_token_redaction_enabled."

description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over bearer_token_redaction_enabled."


description = "When true, tag logs that would have been redacted with vector_aws_key_detected:true instead of replacing. Takes priority over aws_access_key_redaction_enabled."

description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over aws_access_key_redaction_enabled."


description = "When true, tag logs that would have been redacted with vector_jwt_detected:true instead of replacing. Takes priority over jwt_redaction_enabled."

description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over jwt_redaction_enabled."


The file /Users/steve.briggs/projects/operations/terraform/internal-modules/kubernetes/monitoring/vector/variables.tf has been updated successfully. (file state is current in your context — no need to Read it back)

# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# These variables must be set.
# ---------------------------------------------------------------------------------------------------------------------

variable "chart_version" {
  description = "Vector helm chart version pin"
  type        = string
}

variable "image_tag_version" {
  description = "Vector image tag"
  type        = string
}

# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# These variables are configurable, but have reasonable defaults set.
# ---------------------------------------------------------------------------------------------------------------------
variable "datadog_site" {
  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "image" {
  description = "Vector image source"
  type        = string
  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
}

variable "replicas" {
  description = "Number of Vector aggregator replicas"
  type        = number
  default     = 3
}

variable "audit_drop_patterns" {
  description = "Regex patterns; matching log message bodies are dropped"
  type        = list(string)
  default = [
    "Audit\\s-\\s.+",
    "\"level\":\"audit\"",
  ]
}

variable "datadog_api_key_vault_path" {
  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
  type        = string
  default     = "secret/data/saas/infrastructure/datadog-api-key"
}

variable "external_secrets_cluster_store_name" {
  description = "Name of the ClusterSecretStore created by external-secrets module"
  type        = string
  default     = "vault"
}

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over email_redaction_enabled."
  type        = bool
  default     = false
}

variable "email_redaction_replacement" {
  description = "String that replaces matched email addresses"
  type        = string
  default     = "[REDACTED-EMAIL]"
}

variable "additional_config" {
  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
  type        = any
  default     = null
}

variable "resources_requests" {
  description = "Pod resource requests"
  type        = map(string)
  default = {
    cpu    = "200m"
    memory = "256Mi"
  }
}

variable "resources_limits" {
  description = "Pod resource limits"
  type        = map(string)
  default = {
    cpu    = "2"
    memory = "1Gi"
  }
}

variable "namespace" {
  description = "Kubernetes namespace for Vector"
  type        = string
  default     = "vector"
}

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}

variable "bearer_token_redaction_enabled" {
  description = "Toggle bearer token redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "bearer_token_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_bearer_token_detected:true instead of replacing. Takes priority over bearer_token_redaction_enabled."
  type        = bool
  default     = false
}

variable "aws_access_key_redaction_enabled" {
  description = "Toggle AWS Access Key ID redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "aws_access_key_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_aws_key_detected:true instead of replacing. Takes priority over aws_access_key_redaction_enabled."
  type        = bool
  default     = false
}

variable "aws_access_key_redaction_replacement" {
  description = "String that replaces matched AWS Access Key IDs"
  type        = string
  default     = "[REDACTED-AWS-ACCESS-KEY]"
}

variable "aws_access_key_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply AWS Access Key ID redaction to"
  type        = list(string)
  default     = ["marketplace-api"]
}

variable "aws_access_key_redaction_exclusion_patterns" {
  description = "Skip AWS Access Key ID redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default     = []
}

variable "aws_access_key_redaction_exclusion_attributes" {
  description = "Skip AWS Access Key ID redaction for any log where the named attribute equals the given value (keys are attribute paths, values are string values to match)"
  type        = map(string)
  default = {
    "req.headers.x-synthetic-test" = "true"
  }
}

variable "jwt_redaction_enabled" {
  description = "Toggle JSON Web Token redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "jwt_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_jwt_detected:true instead of replacing. Takes priority over jwt_redaction_enabled."
  type        = bool
  default     = false
}

variable "jwt_redaction_replacement" {
  description = "String that replaces matched JSON Web Tokens"
  type        = string
  default     = "[REDACTED-JWT]"
}

variable "jwt_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply JWT redaction to"
  type        = list(string)
  default     = ["service", "secure-metadata-database"]
}

variable "jwt_redaction_exclusion_patterns" {
  description = "Skip JWT redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default = [
    "internal-webhook",
    "starting crawl",
    "Databricks automatic subscription",
    "audit/enrich",
    "Background Job",
    "community-organization-id:immuta",
    "pg - pool.connect",
  ]
}

variable "bearer_token_redaction_replacement" {
  description = "String that replaces matched bearer tokens (replaces the full 'Bearer <token>' match)"
  type        = string
  default     = "Bearer [REDACTED-BEARER-TOKEN]"
}

variable "bearer_token_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply bearer token redaction to"
  type        = list(string)
  default     = ["service", "secure-metadata-database"]
}

variable "bearer_token_redaction_exclusion_patterns" {
  description = "Skip bearer token redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default = [
    "internal-webhook",
    "starting crawl",
    "Databricks automatic subscription",
    "audit/enrich",
    "Background Job",
    "community-organization-id:immuta",
    "pg - pool.connect",
  ]
}

-  description = "When true, tag logs that would have been redacted with vector_bearer_token_detected:true instead of replacing. Takes priority over bearer_token_redaction_enabled."

+  description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over bearer_token_redaction_enabled."


# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# These variables must be set.
# ---------------------------------------------------------------------------------------------------------------------

variable "chart_version" {
  description = "Vector helm chart version pin"
  type        = string
}

variable "image_tag_version" {
  description = "Vector image tag"
  type        = string
}

# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# These variables are configurable, but have reasonable defaults set.
# ---------------------------------------------------------------------------------------------------------------------
variable "datadog_site" {
  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "image" {
  description = "Vector image source"
  type        = string
  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
}

variable "replicas" {
  description = "Number of Vector aggregator replicas"
  type        = number
  default     = 3
}

variable "audit_drop_patterns" {
  description = "Regex patterns; matching log message bodies are dropped"
  type        = list(string)
  default = [
    "Audit\\s-\\s.+",
    "\"level\":\"audit\"",
  ]
}

variable "datadog_api_key_vault_path" {
  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
  type        = string
  default     = "secret/data/saas/infrastructure/datadog-api-key"
}

variable "external_secrets_cluster_store_name" {
  description = "Name of the ClusterSecretStore created by external-secrets module"
  type        = string
  default     = "vault"
}

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over email_redaction_enabled."
  type        = bool
  default     = false
}

variable "email_redaction_replacement" {
  description = "String that replaces matched email addresses"
  type        = string
  default     = "[REDACTED-EMAIL]"
}

variable "additional_config" {
  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
  type        = any
  default     = null
}

variable "resources_requests" {
  description = "Pod resource requests"
  type        = map(string)
  default = {
    cpu    = "200m"
    memory = "256Mi"
  }
}

variable "resources_limits" {
  description = "Pod resource limits"
  type        = map(string)
  default = {
    cpu    = "2"
    memory = "1Gi"
  }
}

variable "namespace" {
  description = "Kubernetes namespace for Vector"
  type        = string
  default     = "vector"
}

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}

variable "bearer_token_redaction_enabled" {
  description = "Toggle bearer token redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "bearer_token_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over bearer_token_redaction_enabled."
  type        = bool
  default     = false
}

variable "aws_access_key_redaction_enabled" {
  description = "Toggle AWS Access Key ID redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "aws_access_key_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_aws_key_detected:true instead of replacing. Takes priority over aws_access_key_redaction_enabled."
  type        = bool
  default     = false
}

variable "aws_access_key_redaction_replacement" {
  description = "String that replaces matched AWS Access Key IDs"
  type        = string
  default     = "[REDACTED-AWS-ACCESS-KEY]"
}

variable "aws_access_key_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply AWS Access Key ID redaction to"
  type        = list(string)
  default     = ["marketplace-api"]
}

variable "aws_access_key_redaction_exclusion_patterns" {
  description = "Skip AWS Access Key ID redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default     = []
}

variable "aws_access_key_redaction_exclusion_attributes" {
  description = "Skip AWS Access Key ID redaction for any log where the named attribute equals the given value (keys are attribute paths, values are string values to match)"
  type        = map(string)
  default = {
    "req.headers.x-synthetic-test" = "true"
  }
}

variable "jwt_redaction_enabled" {
  description = "Toggle JSON Web Token redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "jwt_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_jwt_detected:true instead of replacing. Takes priority over jwt_redaction_enabled."
  type        = bool
  default     = false
}

variable "jwt_redaction_replacement" {
  description = "String that replaces matched JSON Web Tokens"
  type        = string
  default     = "[REDACTED-JWT]"
}

variable "jwt_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply JWT redaction to"
  type        = list(string)
  default     = ["service", "secure-metadata-database"]
}

variable "jwt_redaction_exclusion_patterns" {
  description = "Skip JWT redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default = [
    "internal-webhook",
    "starting crawl",
    "Databricks automatic subscription",
    "audit/enrich",
    "Background Job",
    "community-organization-id:immuta",
    "pg - pool.connect",
  ]
}

variable "bearer_token_redaction_replacement" {
  description = "String that replaces matched bearer tokens (replaces the full 'Bearer <token>' match)"
  type        = string
  default     = "Bearer [REDACTED-BEARER-TOKEN]"
}

variable "bearer_token_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply bearer token redaction to"
  type        = list(string)
  default     = ["service", "secure-metadata-database"]
}

variable "bearer_token_redaction_exclusion_patterns" {
  description = "Skip bearer token redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default = [
    "internal-webhook",
    "starting crawl",
    "Databricks automatic subscription",
    "audit/enrich",
    "Background Job",
    "community-organization-id:immuta",
    "pg - pool.connect",
  ]
}

-  description = "When true, tag logs that would have been redacted with vector_aws_key_detected:true instead of replacing. Takes priority over aws_access_key_redaction_enabled."

+  description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over aws_access_key_redaction_enabled."


# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# These variables must be set.
# ---------------------------------------------------------------------------------------------------------------------

variable "chart_version" {
  description = "Vector helm chart version pin"
  type        = string
}

variable "image_tag_version" {
  description = "Vector image tag"
  type        = string
}

# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# These variables are configurable, but have reasonable defaults set.
# ---------------------------------------------------------------------------------------------------------------------
variable "datadog_site" {
  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "image" {
  description = "Vector image source"
  type        = string
  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
}

variable "replicas" {
  description = "Number of Vector aggregator replicas"
  type        = number
  default     = 3
}

variable "audit_drop_patterns" {
  description = "Regex patterns; matching log message bodies are dropped"
  type        = list(string)
  default = [
    "Audit\\s-\\s.+",
    "\"level\":\"audit\"",
  ]
}

variable "datadog_api_key_vault_path" {
  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
  type        = string
  default     = "secret/data/saas/infrastructure/datadog-api-key"
}

variable "external_secrets_cluster_store_name" {
  description = "Name of the ClusterSecretStore created by external-secrets module"
  type        = string
  default     = "vault"
}

variable "email_redaction_enabled" {
  description = "Toggle the email-redact VRL transform across all signals"
  type        = bool
  default     = true
}

variable "email_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over email_redaction_enabled."
  type        = bool
  default     = false
}

variable "email_redaction_replacement" {
  description = "String that replaces matched email addresses"
  type        = string
  default     = "[REDACTED-EMAIL]"
}

variable "additional_config" {
  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
  type        = any
  default     = null
}

variable "resources_requests" {
  description = "Pod resource requests"
  type        = map(string)
  default = {
    cpu    = "200m"
    memory = "256Mi"
  }
}

variable "resources_limits" {
  description = "Pod resource limits"
  type        = map(string)
  default = {
    cpu    = "2"
    memory = "1Gi"
  }
}

variable "namespace" {
  description = "Kubernetes namespace for Vector"
  type        = string
  default     = "vector"
}

variable "service_monitor_enabled" {
  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
  type        = bool
  default     = false
}

variable "bearer_token_redaction_enabled" {
  description = "Toggle bearer token redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "bearer_token_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over bearer_token_redaction_enabled."
  type        = bool
  default     = false
}

variable "aws_access_key_redaction_enabled" {
  description = "Toggle AWS Access Key ID redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "aws_access_key_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over aws_access_key_redaction_enabled."
  type        = bool
  default     = false
}

variable "aws_access_key_redaction_replacement" {
  description = "String that replaces matched AWS Access Key IDs"
  type        = string
  default     = "[REDACTED-AWS-ACCESS-KEY]"
}

variable "aws_access_key_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply AWS Access Key ID redaction to"
  type        = list(string)
  default     = ["marketplace-api"]
}

variable "aws_access_key_redaction_exclusion_patterns" {
  description = "Skip AWS Access Key ID redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default     = []
}

variable "aws_access_key_redaction_exclusion_attributes" {
  description = "Skip AWS Access Key ID redaction for any log where the named attribute equals the given value (keys are attribute paths, values are string values to match)"
  type        = map(string)
  default = {
    "req.headers.x-synthetic-test" = "true"
  }
}

variable "jwt_redaction_enabled" {
  description = "Toggle JSON Web Token redact VRL transform on logs"
  type        = bool
  default     = false
}

variable "jwt_redaction_dry_run" {
  description = "When true, tag logs that would have been redacted with vector_jwt_detected:true instead of replacing. Takes priority over jwt_redaction_enabled."
  type        = bool
  default     = false
}

variable "jwt_redaction_replacement" {
  description = "String that replaces matched JSON Web Tokens"
  type        = string
  default     = "[REDACTED-JWT]"
}

variable "jwt_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply JWT redaction to"
  type        = list(string)
  default     = ["service", "secure-metadata-database"]
}

variable "jwt_redaction_exclusion_patterns" {
  description = "Skip JWT redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default = [
    "internal-webhook",
    "starting crawl",
    "Databricks automatic subscription",
    "audit/enrich",
    "Background Job",
    "community-organization-id:immuta",
    "pg - pool.connect",
  ]
}

variable "bearer_token_redaction_replacement" {
  description = "String that replaces matched bearer tokens (replaces the full 'Bearer <token>' match)"
  type        = string
  default     = "Bearer [REDACTED-BEARER-TOKEN]"
}

variable "bearer_token_redaction_services" {
  description = "Service names (.service field on DD agent logs) to apply bearer token redaction to"
  type        = list(string)
  default     = ["service", "secure-metadata-database"]
}

variable "bearer_token_redaction_exclusion_patterns" {
  description = "Skip bearer token redaction for any log whose message or ddtags contains one of these strings"
  type        = list(string)
  default = [
    "internal-webhook",
    "starting crawl",
    "Databricks automatic subscription",
    "audit/enrich",
    "Background Job",
    "community-organization-id:immuta",
    "pg - pool.connect",
  ]
}

-  description = "When true, tag logs that would have been redacted with vector_jwt_detected:true instead of replacing. Takes priority over jwt_redaction_enabled."

+  description = "When true, tag logs that would have been redacted with vector_redaction:true instead of replacing. Takes priority over jwt_redaction_enabled."


Done. Here's what changed, Sir Charles of St Cheese:

**`templates/values.yaml.tpl`**
- Removed `redact_emails_traces` and `redact_emails_metrics` transforms
- Removed `datadog_traces` and `datadog_metrics` sinks

The OTLP source still listens on ports 4317/4318 for log ingestion — vector will silently discard the unused `.traces` and `.metrics` sub-outputs. The `dd_agent_in` source was already configured with `disable_metrics: true` and `disable_traces: true`.

**`variables.tf`** — fixed three stale descriptions that still referenced the old per-signal tag names from the previous change.


afplay ~/.claude/sounds/mario/stop/stomp.wav 2>/dev/null || true


<command-message>obsidian-save</command-message>
<command-name>/obsidian-save</command-name>


Save current conversation context to an Obsidian note under the `Claude/` directory.

## What to save

Capture all of the following that are relevant to the current conversation:

- **Conversation summary**: What the user is trying to accomplish and why
- **Project context**: Relevant architecture, design decisions, or constraints discovered
- **Task state**: What has been completed, what is in progress, what is blocked or pending
- **Key decisions**: Any choices made and the reasoning behind them
- **Open questions**: Unresolved questions or things to revisit
- **Next steps**: Concrete actions that should follow from this session

## File location

Derive the project name from the current working directory (use the repo or app folder name). Save to:

```
Claude/<project-name>/<YYYY-MM-DD>-<short-slug>.md
```

Use today's date and a short 2-4 word slug describing the session topic (e.g. `2026-04-29-ruth-plugin-setup.md`).

If `` is provided, use it as the slug instead of generating one.

## Note format

Use this structure:

```markdown
---
date: <YYYY-MM-DD>
project: <project-name>
tags: [claude-context]
---

# <Session Title>

## Summary
<1-3 sentence overview of what this session was about>

## Context
<Relevant project background, architecture notes, constraints>

## Decisions
<Decisions made and rationale>

## Task State
<What was completed, in progress, or blocked>

## Open Questions
<Anything unresolved>

## Next Steps
<Concrete follow-up actions>
```

## Steps

1. Determine the project name from the working directory.
2. Determine today's date.
3. Draft the note content based on the current conversation.
4. Use `mcp__obsidian__create_vault_file` to write the note to the vault.
5. Confirm to the user where the note was saved.


1	role: Aggregator
2	replicas: ${replicas}
3	podAnnotations:
4	  reloader.stakater.com/auto: "true"
5	service:
6	  enabled: true
7	  type: ClusterIP
8	  ports:
9	    - name: otlp-grpc
10	      port: 4317
11	      protocol: TCP
12	    - name: otlp-http
13	      port: 4318
14	      protocol: TCP
15	    - name: dd-agent
16	      port: 8282
17	      protocol: TCP
18	env:
19	  - name: DATADOG_API_KEY
20	    valueFrom:
21	      secretKeyRef:
22	        name: vector-datadog
23	        key: api_key
24	resources:
25	  requests:
26	    cpu: ${requests_cpu}
27	    memory: ${requests_memory}
28	  limits:
29	    cpu: "${limits_cpu}"
30	    memory: ${limits_memory}
31	podDisruptionBudget:
32	  enabled: true
33	  minAvailable: 2
34
35	topologySpreadConstraints:
36	  - maxSkew: 1
37	    topologyKey: topology.kubernetes.io/zone
38	    whenUnsatisfiable: ScheduleAnyway
39	    labelSelector:
40	      matchLabels:
41	        app.kubernetes.io/name: vector
42
43	affinity:
44	  podAntiAffinity:
45	    preferredDuringSchedulingIgnoredDuringExecution:
46	      - weight: 100
47	        podAffinityTerm:
48	          topologyKey: kubernetes.io/hostname
49	          labelSelector:
50	            matchLabels:
51	              app.kubernetes.io/name: vector
52
53	%{ if service_monitor_enabled ~}
54	serviceMonitor:
55	  enabled: true
56	%{ endif ~}
57	customConfig:
58	  data_dir: /vector-data-dir
59	  api:
60	    enabled: true
61	    address: 0.0.0.0:8686
62	  sources:
63	    otlp_in:
64	      type: opentelemetry
65	      grpc:
66	        address: 0.0.0.0:4317
67	      http:
68	        address: 0.0.0.0:4318
69	    dd_agent_in:
70	      type: datadog_agent
71	      address: 0.0.0.0:8282
72	      multiple_outputs: true
73	      disable_metrics: true
74	      disable_traces: true
75	      store_api_key: false
76	      acknowledgements:
77	        enabled: true
78	  transforms:
79	    drop_audit:
80	      type: filter
81	      inputs:
82	        - otlp_in.logs
83	        - dd_agent_in.logs
84	      condition: |
85	        msg = if is_string(.message) { string!(.message) } else { "" }
86	        ${audit_drop_condition}
87	    normalize_logs:
88	      type: remap
89	      inputs:
90	        - drop_audit
91	      source: |
92	        if exists(.severity) {
93	          .status = downcase(to_string!(.severity))
94	          del(.severity)
95	        }
96	%{ if email_do_redact ~}
97	    redact_emails_logs:
98	      type: remap
99	      inputs:
100	        - normalize_logs
101	      source: |
102	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
103	        if is_string(.message) {
104	          .message = replace(string!(.message), email_re, "${email_redaction_replacement}")
105	        }
106	        if is_object(.attributes) {
107	          attrs, _err = object(.attributes)
108	          if is_object(attrs) {
109	            .attributes = map_values(attrs) -> |v| {
110	              if is_string(v) { replace(string!(v), email_re, "${email_redaction_replacement}") } else { v }
111	            }
112	          }
113	        }
114	%{ endif ~}
115	%{ if email_do_tag ~}
116	    tag_emails_logs:
117	      type: remap
118	      inputs:
119	        - normalize_logs
120	      source: |
121	        email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
122	        detected = false
123	        if is_string(.message) {
124	          detected = match(string!(.message), email_re)
125	        }
126	        if !detected && is_object(.attributes) {
127	          detected = match(encode_json(.attributes), email_re)
128	        }
129	        if detected {
130	          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
131	        }
132	%{ endif ~}
133	%{ if bearer_do_redact ~}
134	    redact_bearer_tokens_logs:
135	      type: remap
136	      inputs:
137	        - ${bearer_token_logs_input}
138	      source: |
139	        bearer_re = r'(?i)Bearer\s+[A-Za-z0-9+/._~-]+=*'
140	        svc = to_string(.service) ?? ""
141	        ddtags = to_string(.ddtags) ?? ""
142	        msg = if is_string(.message) { string!(.message) } else { "" }
143	%{ if bearer_needs_combined ~}
144	        combined = msg + " " + ddtags
145	%{ endif ~}
146	        target_services = ${bearer_token_services_arr}
147	        should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${bearer_token_exclusion_condition}
148	        if should_redact && is_string(.message) {
149	          .message = replace(string!(.message), bearer_re, "${bearer_token_redaction_replacement}")
150	        }
151	%{ endif ~}
152	%{ if bearer_do_tag ~}
153	    tag_bearer_tokens_logs:
154	      type: remap
155	      inputs:
156	        - ${bearer_token_logs_input}
157	      source: |
158	        bearer_re = r'(?i)Bearer\s+[A-Za-z0-9+/._~-]+=*'
159	        svc = to_string(.service) ?? ""
160	        ddtags = to_string(.ddtags) ?? ""
161	        msg = if is_string(.message) { string!(.message) } else { "" }
162	%{ if bearer_needs_combined ~}
163	        combined = msg + " " + ddtags
164	%{ endif ~}
165	        target_services = ${bearer_token_services_arr}
166	        would_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${bearer_token_exclusion_condition}
167	        if would_redact && is_string(.message) && match(string!(.message), bearer_re) {
168	          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
169	        }
170	%{ endif ~}
171	%{ if aws_key_do_redact ~}
172	    redact_aws_access_keys_logs:
173	      type: remap
174	      inputs:
175	        - ${aws_key_logs_input}
176	      source: |
177	        aws_key_re = r'(AKIA|ASIA|AROA|AIPA|ANPA|ANVA|APKA)[A-Z0-9]{16}'
178	        svc = to_string(.service) ?? ""
179	        ddtags = to_string(.ddtags) ?? ""
180	        msg = if is_string(.message) { string!(.message) } else { "" }
181	%{ if aws_key_needs_combined ~}
182	        combined = msg + " " + ddtags
183	%{ endif ~}
184	        target_services = ${aws_key_services_arr}
185	        attr_excluded = ${aws_key_attr_exclusion_condition}
186	        should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${aws_key_string_exclusion_condition} && !attr_excluded
187	        if should_redact && is_string(.message) {
188	          .message = replace(string!(.message), aws_key_re, "${aws_access_key_redaction_replacement}")
189	        }
190	%{ endif ~}
191	%{ if aws_key_do_tag ~}
192	    tag_aws_access_keys_logs:
193	      type: remap
194	      inputs:
195	        - ${aws_key_logs_input}
196	      source: |
197	        aws_key_re = r'(AKIA|ASIA|AROA|AIPA|ANPA|ANVA|APKA)[A-Z0-9]{16}'
198	        svc = to_string(.service) ?? ""
199	        ddtags = to_string(.ddtags) ?? ""
200	        msg = if is_string(.message) { string!(.message) } else { "" }
201	%{ if aws_key_needs_combined ~}
202	        combined = msg + " " + ddtags
203	%{ endif ~}
204	        target_services = ${aws_key_services_arr}
205	        attr_excluded = ${aws_key_attr_exclusion_condition}
206	        would_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${aws_key_string_exclusion_condition} && !attr_excluded
207	        if would_redact && is_string(.message) && match(string!(.message), aws_key_re) {
208	          .ddtags = if is_string(.ddtags) { string!(.ddtags) + ",vector_redaction:true" } else { "vector_redaction:true" }
209	        }
210	%{ endif ~}
211	%{ if jwt_do_redact ~}
212	    redact_jwts_logs:
213	      type: remap
214	      inputs:
215	        - ${jwt_logs_input}
216	      source: |
217	        jwt_re = r'eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'
218	        svc = to_string(.service) ?? ""
219	        ddtags = to_string(.ddtags) ?? ""
220	        msg = if is_string(.message) { string!(.message) } else { "" }
221	%{ if jwt_needs_combined ~}
222	        combined = msg + " " + ddtags
223	%{ endif ~}
224	        target_services = ${jwt_services_arr}
225	        should_redact = contains(ddtags, "env:prod") && includes(target_services, svc) && ${jwt_exclusion_condition}
226	        if should_redact && is_string(.message) {

... [30 lines truncated] ...


Saving the session context to Obsidian now.


---
date: 2026-05-12
project: operations
tags: [claude-context]
---

# Vector: Unified Redaction Tag + Logs-Only

## Summary
Two changes made to the Vector Terraform internal module (`terraform/internal-modules/kubernetes/monitoring/vector`) on branch `SRE-2806-more-sdd`: unified all dry-run tagging to use a single `vector_redaction:true` tag, then stripped traces and metrics handling so Vector is logs-only.

## Context
Vector is deployed as a Kubernetes aggregator that receives logs (and previously traces/metrics) from an OTLP source and a Datadog agent source, applies redaction/tagging transforms, and forwards to Datadog. There are four scanner types: email, bearer token, AWS access key, JWT. Each scanner has an `enabled` and `dry_run` mode — dry run tags instead of redacting.

Traces and metrics will now be handled exclusively by the existing OpenTelemetry Collector deployment. The `dd_agent_in` source was already configured with `disable_metrics: true` and `disable_traces: true`; the OTLP source still listens on 4317/4318 for log ingestion (vector silently drops unused `.traces`/`.metrics` sub-outputs).

## Decisions

### Unified dry-run tag: `vector_redaction:true`
- **Before**: each scanner used its own tag (`vector_bearer_token_detected:true`, `vector_aws_key_detected:true`, `vector_jwt_detected:true`); only email used `vector_redaction:true`
- **After**: all four scanners emit `vector_redaction:true` in dry-run mode
- Files changed: `templates/values.yaml.tpl`, `tests/vrl/tag_bearer_tokens_logs/{program.vrl,expected.ndjson}`, `tests/vrl/tag_aws_access_keys_logs/{program.vrl,expected.ndjson}`, `tests/vrl/tag_jwts_logs/{program.vrl,expected.ndjson}`

### Logs-only Vector
- Removed `redact_emails_traces` and `redact_emails_metrics` transforms from the `%{ if email_do_redact }` block
- Removed `datadog_traces` and `datadog_metrics` sinks
- No changes to `main.tf` — the inline ternary expressions for traces/metrics sink inputs were part of the template itself and went away with the sinks
- Also fixed stale variable descriptions in `variables.tf` that still referenced the old per-scanner tag names

## Task State
- [x] Unified dry-run tag to `vector_redaction:true` across all four scanners
- [x] Removed traces and metrics transforms and sinks
- [x] Fixed stale `variables.tf` descriptions

## Open Questions
- None at this time

## Next Steps
- Run VRL tests (`tests/vrl/*/run.sh`) to confirm expected outputs still match
- Apply Terraform and verify Vector pods restart cleanly
- Confirm the OTel collector deployment is handling traces and metrics before removing OTLP ports (4317/4318) if desired


File created successfully

File created successfully


Saved to `Claude/operations/2026-05-12-vector-logs-only-redaction.md`.


<local-command-caveat>Caveat: The messages below were generated by the user while running local commands. DO NOT respond to these messages or otherwise consider them in your response unless the user explicitly asks you to.</local-command-caveat>


<command-name>/exit</command-name>
            <command-message>exit</command-message>
            <command-args></command-args>


<local-command-stdout>Catch you later!</local-command-stdout>