You are implementing Task B4 of an implementation plan: in the existing `otel-collector` Terraform internal-module, conditionally swap the terminal exporter from `datadog/exporter` to `otlp` (pointed at Vector) when `var.downstream_endpoint` is non-empty. Also conditionally remove the Datadog API key secret mount when in downstream mode.

## Context — what you're working with

- Working directory: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector`
- Module structure (per Task B1 inventory):
  - `otelconfig.tf` defines `local.otel_config` (HCL map) — contains the exporters block, connectors, processors, and `service.pipelines.{logs,traces,metrics}.exporters` lists. **This is where the swap happens.**
  - `lbconfig.tf` defines `local.lb_config` — the load-balancer tier config. **No changes needed here** (LB tier doesn't talk to Datadog).
  - `main.tf` has two `helm_release` resources (lb tier + processor tier) that consume those locals via `yamlencode()`. Also has the `kubernetes_manifest` ExternalSecret resource that creates the `otel-collector` secret containing `DD_API_KEY`. **Adjust the secret mount conditionally here.**
  - `variables.tf` already has `downstream_endpoint` and `downstream_protocol` (from B3, just merged).
- The pipeline structure today (from B1 inventory):
  - `traces` pipeline exporters: `["datadog/connector", "datadog/exporter"]`
  - `logs` pipeline exporters: `["datadog/exporter"]`
  - `metrics` pipeline exporters: `["datadog/exporter"]`
- The `datadog/connector` is for trace-to-metrics conversion. Keep it as-is regardless of downstream mode — it's a local computation, not a Datadog network call.
- Datadog API key today: pulled from secret `otel-collector` (key `DD_API_KEY`) via `extraEnvsFrom` in `main.tf` (~line 236-240). Secret created by an ExternalSecret manifest (~lines 34-64).

## Goal

When `var.downstream_endpoint == ""` (default), the rendered helm values must be **byte-identical** to today's output. This is the backwards-compat contract.

When `var.downstream_endpoint != ""`:
1. The `exporters."datadog/exporter"` block in `local.otel_config` is removed.
2. A new `exporters.otlp` block is added with:
   - `endpoint`: the `var.downstream_endpoint` value
   - `tls.insecure: true`
   - `sending_queue.enabled: true`, `queue_size: 5000`
   - `retry_on_failure.enabled: true`, `initial_interval: "5s"`, `max_interval: "30s"`, `max_elapsed_time: "300s"`
3. Pipeline exporters lists swap:
   - `traces`: `["datadog/connector", "otlp"]`
   - `logs`: `["otlp"]`
   - `metrics`: `["otlp"]`
4. The `extraEnvsFrom` Datadog secret mount is removed (the `otlp` exporter doesn't need a Datadog API key).
5. The ExternalSecret manifest creating the `otel-collector` secret should ALSO be conditionally skipped (no point materializing a secret nothing consumes). Use `count = var.downstream_endpoint == "" ? 1 : 0` on the `kubernetes_manifest` resource.

## Implementation steps

- [ ] **Step 1: Edit `otelconfig.tf`** — modify `local.otel_config` to conditionally produce different `exporters` and `service.pipelines.*.exporters`. The cleanest pattern is HCL's ternary + `merge()` for the exporters map. Sketch:

   ```hcl
   locals {
     # build the OTLP downstream block once
     otlp_exporter_config = {
       endpoint = var.downstream_endpoint
       tls = {
         insecure = true
       }
       sending_queue = {
         enabled    = true
         queue_size = 5000
       }
       retry_on_failure = {
         enabled           = true
         initial_interval  = "5s"
         max_interval      = "30s"
         max_elapsed_time  = "300s"
       }
     }

     # the existing exporters map with datadog/exporter
     exporters_direct = {
       "datadog/exporter" = {
         api = {
           site = "datadoghq.com"
           key  = "$${env:DD_API_KEY}"
         }
         sending_queue = {
           enabled = true
           storage = "file_storage/persistent_queue"
         }
       }
     }

     # the new exporters map for downstream mode
     exporters_downstream = {
       otlp = local.otlp_exporter_config
     }

     otel_exporters = var.downstream_endpoint == "" ? local.exporters_direct : local.exporters_downstream

     # pipeline exporter lists swap based on mode
     traces_exporters  = var.downstream_endpoint == "" ? ["datadog/connector", "datadog/exporter"] : ["datadog/connector", "otlp"]
     logs_exporters    = var.downstream_endpoint == "" ? ["datadog/exporter"] : ["otlp"]
     metrics_exporters = var.downstream_endpoint == "" ? ["datadog/exporter"] : ["otlp"]
   }
   ```

   Then in the `local.otel_config` map, reference these instead of inlining:
   - `exporters = local.otel_exporters`
   - `service.pipelines.traces.exporters = local.traces_exporters`
   - `service.pipelines.logs.exporters = local.logs_exporters`
   - `service.pipelines.metrics.exporters = local.metrics_exporters`

   **PRESERVE everything else verbatim** — connectors, processors, receivers, the metrics pipeline's receiver list (which includes `datadog/connector`), service.telemetry, etc. The only changes are: the exporters map, and the three pipeline exporter lists.

   Read the current `otelconfig.tf` carefully and adapt the snippet above to match the actual file's variable names and structure. The exact site/key values in `exporters_direct` MUST match what's there today (`site = "datadoghq.com"`, `key = "$${env:DD_API_KEY}"`).

- [ ] **Step 2: Edit `main.tf`** to conditionally remove the Datadog secret mount and conditionally skip the ExternalSecret manifest:

   - On the `helm_release.otel_collector_processor`'s `extraEnvsFrom` config, gate the entire block on `var.downstream_endpoint == ""`:
     ```hcl
     # in the helm values:
     extraEnvsFrom = var.downstream_endpoint == "" ? [{
       secretRef = { name = "otel-collector" }
     }] : []
     ```

   - On the `kubernetes_manifest` ExternalSecret resource (the one creating the `otel-collector` secret with DD_API_KEY), add `count = var.downstream_endpoint == "" ? 1 : 0`. Make sure no other resource references this manifest — if anything else has a `depends_on` pointing at it, those references need adjusting (use `[]` index correctly).

- [ ] **Step 3: Run `terraform fmt -check -recursive`** — exit 0 expected.

- [ ] **Step 4: Run `terraform validate`** (with `terraform init -backend=false` first if needed). Use `mise exec terraform@1.9.0 -- terraform <cmd>` if your default Terraform is older. **Must pass.**

- [ ] **Step 5: Verify backwards-compatibility via ad-hoc render.** Build a tiny tempdir test that calls the module locally (with `source = "/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector"`) and runs `terraform plan` (NOT apply) twice:
   - **Case A**: no `downstream_endpoint` set (or `= ""`). The plan should show all the same resources today — kubernetes_manifest for the ExternalSecret, two helm_release resources, etc. The yamlencoded values for the processor helm release should contain `datadog/exporter`.
   - **Case B**: `downstream_endpoint = "http://vector.vector.svc.cluster.local:4317"`. The plan should show:
     - The kubernetes_manifest ExternalSecret resource COUNTED OUT (zero instances).
     - The processor helm release values now have `exporters.otlp`, no `exporters.datadog/exporter`, and the pipeline exporter lists use `otlp`.
     - The `extraEnvsFrom` block on the processor helm release is empty list.

   Use `terraform plan -no-color` and grep for the relevant fragments. Iterate until both cases pass.

   Cleanup: rm the tempdir.

   IMPORTANT: This `terraform plan` will REQUIRE provider configurations. The module declares `kubernetes` and `helm` providers. Provide stub provider configs in the tempdir's main.tf (e.g., `provider "kubernetes" { config_path = "~/.kube/config" }` even if no kubectl config exists — `plan` does not actually contact a cluster, it only runs `terraform validate`-equivalent checks plus diff computation against state). If `terraform plan` errors on provider auth, fall back to running `terraform validate` only and reporting that the conditional logic compiles cleanly.

   **DO NOT run `terraform apply` against any real cluster.**

## Constraints

- **DO NOT run any git commands.**
- **DO NOT** push, tag, open MRs.
- **DO NOT** run `terraform apply`. `terraform plan` is the verification step.
- **DO NOT** modify `lbconfig.tf` (LB tier is unaffected).
- **DO NOT** create snapshot tests.
- **DO NOT** touch perf-tuning values like `send_batch_max_size` — these belong to the `otel-batch-size` branch's work.

## Spec / plan references

- Spec: `/Users/steve.briggs/projects/IAC/docs/superpowers/specs/2026-05-05-vector-redaction-design.md` §3.1, §5.1, §5.2
- Plan: `/Users/steve.briggs/projects/IAC/docs/superpowers/plans/2026-05-05-vector-redaction-pipeline.md` Task B4

## Report Format

- **Status:** DONE | DONE_WITH_CONCERNS | BLOCKED | NEEDS_CONTEXT
- Files modified (with line ranges)
- `terraform fmt` result
- `terraform validate` result
- Both ad-hoc plan cases verified (or, if blocked on provider auth, what specifically failed)
- Self-review findings:
  - Backwards compat: Case A plan should be identical to today's behavior — confirm.
  - Forward path: Case B should show the swap happening cleanly.
  - Are connectors/processors/receivers untouched?


I'll start by reading the current files to understand the exact structure before making any changes.


1	locals {
2	  otel_config = {
3	    receivers = {
4	      otlp = {
5	        protocols = {
6	          grpc = {
7	            endpoint = "0.0.0.0:4317"
8	          }
9	          http = {
10	            endpoint = "0.0.0.0:4318"
11	          }
12	        }
13	      }
14	    }
15
16	    processors = {
17	      batch = {
18	        timeout = "10s"
19	        send_batch_size = 10
20	        send_batch_max_size = 75
21	      }
22	    }
23
24	    connectors = {
25	      "datadog/connector" = {}
26	    }
27
28	    exporters = {
29	      "datadog/exporter" = {
30	        api = {
31	          site = "datadoghq.com"
32	          key = "$${env:DD_API_KEY}"
33	        }
34	        sending_queue = {
35	          enabled = true
36	          storage = "file_storage/persistent_queue"
37	        }
38	      }
39	    }
40
41	    extensions = {
42	      health_check = {
43	        endpoint = "$${env:MY_POD_IP}:13133"
44	      }
45	      "file_storage/persistent_queue" = {
46	        directory = "/var/lib/storage/queue"
47	      }
48	    }
49
50	    service = {
51	      extensions = ["health_check","file_storage/persistent_queue"]
52	      pipelines = {
53	        traces = {
54	          receivers = ["otlp"]
55	          processors = []
56	          exporters = ["datadog/connector", "datadog/exporter"]
57	        }
58
59	        logs = {
60	          receivers = ["otlp"]
61	          processors = []
62	          exporters = ["datadog/exporter"]
63	        }
64
65	        metrics = {
66	          receivers = ["otlp", "datadog/connector"]
67	          processors = []
68	          exporters = ["datadog/exporter"]
69	        }
70	      }
71	      telemetry = {
72	        logs = {
73	          encoding = "json"
74	        }
75	      }
76	    }
77	  }
78	}
79


1	/**
2	*
3	* # OTEL Collector (K8s Addon)
4	* Deploys otel collector via terraform.
5	* ## Created Resources
6	* 1. otel collector.
7	*
8	*/
9	terraform {
10	  required_version = ">= 1.3.0"
11
12	  required_providers {
13	    helm = {
14	      source  = "hashicorp/helm"
15	      version = "~> 2.0"
16	    }
17	    kubernetes = {
18	      source  = "hashicorp/kubernetes"
19	      version = "~> 2.0"
20	    }
21	  }
22	}
23
24	resource "kubernetes_namespace" "this" {
25	  metadata {
26	    name = var.namespace
27
28	    #labels = merge(var.tags, { "sourceModule" = "/internal-modules/kubernetes/monitoring/otel-collector" })
29
30	    annotations = var.annotations
31	  }
32	}
33
34	resource "kubernetes_manifest" "otel-collector" {
35	  manifest = {
36	    "apiVersion" = "external-secrets.io/v1"
37	    "kind"       = "ExternalSecret"
38	    "metadata" = {
39	      "name" = "otel-collector"
40	      "namespace" = var.namespace
41	    }
42	    "spec" = {
43	      "refreshInterval" = "1h0m0s"
44	      "secretStoreRef" = {
45	        "name" = "vault"
46	        "kind" = "ClusterSecretStore"
47	      }
48	      "target" = {
49	        "name"           = "otel-collector"
50	        "creationPolicy" = "Owner"
51	      }
52	      "data" = [
53	        {
54	          "secretKey" = "DD_API_KEY"
55	          "remoteRef" = {
56	            "key"      = "secret/data/saas/infrastructure/datadog-api-key"
57	            "property" = "apiKey"
58	          }
59	        },
60	      ]
61	    }
62	  }
63	  depends_on = [kubernetes_namespace.this]
64	}
65
66
67	resource "helm_release" "otel_collector_lb" {
68	  #name      = var.name
69	  name = "otel-collector-lb"
70	  namespace = var.namespace
71	  create_namespace = false
72	  repository = "https://open-telemetry.github.io/opentelemetry-helm-charts/"
73	  chart      = "opentelemetry-collector"
74	  version    = var.chart_version
75
76	  values = [yamlencode({
77	    mode = "deployment"
78	    image = {
79	      repository = var.otellb_image
80	      tag = var.image_tag_version
81	    }
82	    replicaCount = var.otel_lb_replicas
83
84	    nameOverride = "collector"
85
86	    annotations = var.annotations
87
88	    podLabels = {
89	      "ic-ops.net/otel" = "collector"
90	      "app.kubernetes.io/part-of" = "opentelemetry-collector"
91	    }
92
93	    #serviceAccount = {
94	    #  create = true,
95	    #  name   = var.otellb_sa_name,
96	    #  annotations = var.sa_annotations
97	    #}
98
99	    presets = {
100	      # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines
101	      kubernetesAttributes = {
102	        enabled = true
103	        extractAllPodLabels = false
104	        extractAllPodAnnotations = true
105	      }
106	    }
107
108	    serviceAccount = {
109	      create = true,
110	      name   = var.otellb_sa_name
111	      annotations = var.sa_annotations
112	    }
113
114	    command = {
115	      extraArgs = var.otellb_extra_args
116	    }
117
118	    ports = {
119	      otlp = {
120	        enabled = true
121	        containerPort = 4317
122	        servicePort = 4317
123	        protocol = "TCP"
124	        appProtocol = "grpc"
125	      }
126	      otlp-http = {
127	        enabled = true
128	        containerPort = 4318
129	        servicePort = 4318
130	        protocol = "TCP"
131	      }
132	      jaegar-compact = {
133	        enabled = false
134	      }
135	      jaeger-grpc = {
136	        enabled = false
137	      }
138	      jaeger-thrift = {
139	        enabled = false
140	      }
141	      zipkin = {
142	        enabled = false
143	      }
144	    }
145
146	    config = local.lb_config
147	  })]
148
149	  depends_on = [
150	    kubernetes_namespace.this,
151	    kubernetes_manifest.otel-collector,
152	    helm_release.otel_collector_processor
153	  ]
154	}
155
156	resource "helm_release" "otel_collector_processor" {
157	  #name      = var.name
158	  name = "otel-collector-processor"
159	  namespace = var.namespace
160	  create_namespace = false
161	  repository = "https://open-telemetry.github.io/opentelemetry-helm-charts/"
162	  chart      = "opentelemetry-collector"
163	  version    = var.chart_version
164
165	  values = [yamlencode({
166	    mode = "statefulset"
167	    image = {
168	      repository = var.otel_image
169	      tag        = var.image_tag_version
170	    }
171	    statefulset = {
172	      persistentVolumeClaimRetentionPolicy = {
173	        enabled     = true
174	        whenDeleted = "Delete"
175	        whenScaled  = "Delete"
176	      }
177
178	      volumeClaimTemplates = [
179	        {
180	          metadata = {
181	            name = "otel-collector-data"
182	          }
183	          spec = {
184	            accessModes = ["ReadWriteOnce"]
185	            resources = {
186	              requests = {
187	                storage = var.otel_pv_size
188	              }
189	            }
190	            //storageClassName = var.otel_storage_class_name
191	          }
192	        }
193	      ]
194	    }
195
196	    extraVolumeMounts = [{
197	      name       = "otel-collector-data"
198	      mountPath  = "/var/lib/storage/queue"
199	    }]
200
201	    initContainers = [{
202	      name  = "init-fs"
203	      image = "busybox:latest"
204	      command = ["sh", "-c", "chown -R 10001: /var/lib/storage/queue"]
205	      volumeMounts = [{
206	        name      = "otel-collector-data"
207	        mountPath = "/var/lib/storage/queue"
208	      }]
209	    }]
210
211	    nameOverride = "processor"
212
213	    replicaCount = var.otel_collector_replicas
214
215	    service = {
216	      type = "ClusterIP"
217	      clusterIP = "None" # Headless service for statefulset
218	    }
219
220	    annotations = var.annotations
221	    podLabels = {
222	      "ic-ops.net/otel" = "processor"
223	      "app.kubernetes.io/part-of" = "opentelemetry-collector"
224	    }
225
226	    serviceAccount = {
227	      create = true,
228	      name   = var.otel_sa_name,
229	      annotations = var.sa_annotations
230	    }
231
232	    command = {
233	      extraArgs = var.extra_args
234	    }
235
236	    extraEnvsFrom = [{
237	      secretRef = {
238	        name = "otel-collector"
239	      }
240	    }]
241
242	    #presets = {
243	      # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines
244	      #kubernetesAttributes = {
245	      #  enabled = true
246	      #  extractAllPodLabels = true
247	      #  extractAllPodAnnotations = true
248	      #}
249	    #}
250
251	    ports = {
252	      otlp = {
253	        enabled = true
254	        containerPort = 4317
255	        servicePort = 4317
256	        protocol = "TCP"
257	        appProtocol = "grpc"
258	      }
259	      otlp-http = {
260	        enabled = true
261	        containerPort = 4318
262	        servicePort = 4318
263	        protocol = "TCP"
264	      }
265	      jaegar-compact = {
266	        enabled = false
267	      }
268	      jaeger-grpc = {
269	        enabled = false
270	      }
271	      jaeger-thrift = {
272	        enabled = false
273	      }
274	      zipkin = {
275	        enabled = false
276	      }
277	    }
278
279	    config = local.otel_config
280	  })]
281
282	  depends_on = [
283	    kubernetes_namespace.this,
284	    kubernetes_manifest.otel-collector
285	  ]
286
287	  lifecycle {
288	    create_before_destroy = false
289	  }
290	}
291
292	module "netpol" {
293	  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0"
294
295	  name             = join("-", [var.name, "netpol"])
296	  namespace        = var.namespace
297	  network_policies = local.netpols
298
299	  tags = merge(var.tags, { "sourceModule" = "/internal-modules/kubernetes/monitoring/otel-collector" })
300
301	  depends_on = [
302	    kubernetes_namespace.this
303	  ]
304	}
305


1	# ---------------------------------------------------------------------------------------------------------------------
2	# REQUIRED PARAMETERS
3	# These variables are configurable, and need to be set expicitly.
4	# ---------------------------------------------------------------------------------------------------------------------
5	variable "chart_version" {
6	  description = "Version of otel collector chart"
7	  type        = string
8	}
9
10	variable "image_tag_version" {
11	  description = "Version of otel collector image tag"
12	  type        = string
13	}
14
15	# ---------------------------------------------------------------------------------------------------------------------
16	# OPTIONAL PARAMETERS
17	# These variables are configurable, but have reasonable defaults set.
18	# ---------------------------------------------------------------------------------------------------------------------
19	variable "name" {
20	  description = "Name for created resources"
21	  type        = string
22	  default     = "otel-collector"
23	}
24
25	variable "otel_image" {
26	  description = "Helm Value: image.X, optional override to specify image repository & tag"
27	  type        = string
28	  default     = "registry.ic-ops.net/cache-docker-hub/otel/opentelemetry-collector-contrib"
29	}
30
31	variable "otellb_image" {
32	  description = "Helm Value: image.X, optional override to specify image repository & tag"
33	  type        = string
34	  default     = "registry.ic-ops.net/cache-docker-hub/otel/opentelemetry-collector"
35	}
36
37	variable "otel_lb_replicas" {
38	  description = "number of otel loadbalancer replicas to create (statefulset)"
39	  type        = number
40	  default     = 3
41	}
42
43	variable "otel_collector_replicas" {
44	  description = "number of replicas to create for the otel collector processor"
45	  type        = number
46	  default     = 3
47	}
48
49	variable "otel_sa_name" {
50	  description = "Service Account name to be used for otel collector statefulset resources"
51	  type        = string
52	  default     = "otel-collector-sa"
53	}
54
55	variable "otellb_sa_name" {
56	  description = "Service Account name to be used for otel collector deployment resources"
57	  type        = string
58	  default     = "otel-collector-lb-sa"
59	}
60
61	variable "otel_pv_size" {
62	  description = "Size of the Persistent Volume to be used for otel collector statefulset"
63	  type        = string
64	  default     = "5Gi"
65	}
66
67	variable "otel_storage_class_name" {
68	  description = "Storage Class name to be used for otel collector statefulset Persistent Volume"
69	  type        = string
70	  default     = "ebs-encrypted"
71	}
72	variable "sa_annotations" {
73	  description = "Helm Value: A map providing annotations to apply to the otel collector ServiceAccount"
74	  type        = map(string)
75	  default     = {}
76	}
77
78	variable "namespace" {
79	  description = "Namespace to be used for creation of otel collector resources"
80	  type        = string
81	  default     = "opentelemetry"
82	}
83
84	variable "annotations" {
85	  description = "A map providing annotations to apply to the otel collector pods"
86	  type        = map(string)
87	  default     = {}
88	}
89
90	variable "extra_args" {
91	  description = "A list of additional arguments (CLI args) to apply to the otel collector container"
92	  type        = list(string)
93	  default     = []
94	}
95
96	variable "otellb_extra_args" {
97	  description = "A list of additional arguments (CLI args) to apply to the otel collector loadbalancer container"
98	  type        = list(string)
99	  default     = []
100	}
101
102	variable "tags" {
103	  description = "Map of key-vaue pairs to tag created resources with"
104	  type        = map(any)
105	  default     = {}
106	}
107
108	variable "downstream_endpoint" {
109	  description = "When non-empty, swap the terminal exporter from datadog to OTLP, pointed at this endpoint. Used to route through Vector for redaction."
110	  type        = string
111	  default     = ""
112	}
113
114	variable "downstream_protocol" {
115	  description = "Protocol for downstream OTLP exporter. Currently only otlp/grpc supported."
116	  type        = string
117	  default     = "otlp/grpc"
118	  validation {
119	    condition     = contains(["otlp/grpc", "otlp/http"], var.downstream_protocol)
120	    error_message = "downstream_protocol must be one of: otlp/grpc, otlp/http"
121	  }
122	}
123


Good. Now I have a clear picture of both files. Let me also check the lbconfig.tf and the full directory structure to understand what else exists.


1	locals {
2	  lb_config = {
3	    receivers = {
4	      otlp = {
5	        protocols = {
6	          grpc = {
7	            endpoint = "0.0.0.0:4317"
8	          }
9	          http = {
10	            endpoint = "0.0.0.0:4318"
11	          }
12	        }
13	      }
14	    }
15
16	    processors = {
17	      k8sattributes = {
18	        auth_type = "serviceAccount"
19	        extract = {
20	          metadata = [
21	            "k8s.namespace.name",
22	            "k8s.deployment.name",
23	            "k8s.statefulset.name",
24	            "k8s.daemonset.name",
25	            "k8s.cronjob.name",
26	            "k8s.job.name",
27	            "k8s.node.name",
28	            "k8s.pod.name",
29	            "k8s.pod.uid",
30	            "k8s.pod.start_time"
31	          ]
32	          labels = [{
33	            key_regex = "tags\\.datadoghq\\.com/(.*)$"
34	            tag_name = "$$1"
35	            from = "pod"
36	          }]
37	        }
38	        passthrough = false
39	        pod_association = [
40	          {
41	            sources = [{
42	              from = "resource_attribute"
43	              name = "k8s.pod.ip"
44	            }]
45	          },
46	          {
47	            sources = [{
48	              from = "resource_attribute"
49	              name = "k8s.pod.uid"
50	            }]
51	          },
52	          {
53	            sources = [{
54	              from = "connection"
55	            }]
56	          }
57	        ]
58	      }
59	    }
60
61	    exporters = {
62	      loadbalancing = {
63	        routing_key = "traceID"
64	        protocol = {
65	          otlp = {
66	            # all options from the OTLP exporter are supported
67	            # except the endpoint
68	            timeout = "2s"
69	            tls = {
70	              insecure = true
71	            }
72	          }
73	        }
74	        resolver = {
75	          # Notice to config a headless service DNS in Kubernetes
76	          dns = {
77	            hostname = "otel-collector-processor.${var.namespace}.svc.cluster.local"
78	          }
79	        }
80	      }
81	      #metrics loadbalancing
82	      "loadbalancing/metrics" = {
83	        routing_key = "service"
84	        protocol = {
85	          otlp = {
86	            # all options from the OTLP exporter are supported
87	            # except the endpoint
88	            timeout = "2s"
89	            tls = {
90	              insecure = true
91	            }
92	          }
93	        }
94	        resolver = {
95	          # Notice to config a headless service DNS in Kubernetes
96	          dns = {
97	            hostname = "otel-collector-processor.${var.namespace}.svc.cluster.local"
98	          }
99	        }
100	      }
101	    }
102
103	    service = {
104	      pipelines = {
105	        traces = {
106	          receivers = ["otlp"]
107	          processors = ["k8sattributes"]
108	          exporters = ["loadbalancing"]
109	        }
110
111	        logs = {
112	          receivers = ["otlp"]
113	          processors = ["k8sattributes"]
114	          exporters = ["loadbalancing"]
115	        }
116
117	        metrics = {
118	          receivers = ["otlp"]
119	          processors = ["k8sattributes"]
120	          exporters = ["loadbalancing/metrics"]
121	        }
122	      }
123	      telemetry = {
124	        logs = {
125	          encoding = "json"
126	        }
127	      }
128	    }
129	  }
130	}
131


Bash

List module directory contents


.rw-r--r-- steve.briggs staff  70 B  Mon Feb  2 14:16:18 2026 .gitignore
.rw-r--r-- steve.briggs staff 1.2 KB Mon Feb  2 14:16:18 2026 netpol.tf
.rw-r--r-- steve.briggs staff 1.6 KB Mon Feb  2 14:16:18 2026 outputs.tf
drwxr-xr-x steve.briggs staff  96 B  Tue Feb  3 11:35:45 2026 .idea
.rw-r--r-- steve.briggs staff 6.6 KB Thu Feb 19 09:45:02 2026 main.tf
.rw-r--r-- steve.briggs staff 2.9 KB Thu Mar  5 16:42:45 2026 lbconfig.tf
.rw-r--r-- steve.briggs staff 1.5 KB Thu Mar  5 16:47:53 2026 otelconfig.tf
drwxr-xr-x steve.briggs staff 448 B  Thu Mar  5 16:48:00 2026 .git
drwxr-xr-x steve.briggs staff 224 B  Tue May  5 12:32:56 2026 ..
.rw-r--r-- steve.briggs staff 4.0 KB Tue May  5 14:10:09 2026 variables.tf
drwxr-xr-x steve.briggs staff 128 B  Tue May  5 14:10:36 2026 .terraform
.rw-r--r-- steve.briggs staff 463 B  Tue May  5 14:10:36 2026 .terraform.lock.hcl
drwxr-xr-x steve.briggs staff 416 B  Tue May  5 14:10:36 2026 .


I notice the module already has `.terraform` and `.terraform.lock.hcl` — it's been initialized. Let me also check what `depends_on` relationships exist referencing `kubernetes_manifest.otel-collector` to understand how to properly apply `count`.

Looking at `main.tf`:
- Line 151: `helm_release.otel_collector_lb` depends on `kubernetes_manifest.otel-collector`
- Line 284: `helm_release.otel_collector_processor` depends on `kubernetes_manifest.otel-collector`

When adding `count` to `kubernetes_manifest.otel-collector`, these `depends_on` references need to use indexed form `kubernetes_manifest.otel-collector[0]` — but actually with `count`, if the resource has zero instances you can't reference `[0]` in `depends_on` or it errors. The correct pattern is to remove these from `depends_on` and let Terraform handle it, or use the count-safe form. Actually the cleanest approach is: `depends_on` on a counted resource should reference it without the index in `depends_on` — Terraform handles this correctly (it depends on all instances, zero being a no-op).

Wait — actually in Terraform, referencing a resource with `count` in `depends_on` without the index is fine and Terraform correctly handles zero instances. Let me verify this is the correct approach and then make the edits.

Now let me make the two edits:

**Step 1: Edit `otelconfig.tf`**


Write

locals {
  # Datadog exporter block — used in direct (non-downstream) mode
  exporters_direct = {
    "datadog/exporter" = {
      api = {
        site = "datadoghq.com"
        key  = "$${env:DD_API_KEY}"
      }
      sending_queue = {
        enabled = true
        storage = "file_storage/persistent_queue"
      }
    }
  }

  # OTLP downstream exporter block — used when var.downstream_endpoint is set
  exporters_downstream = {
    otlp = {
      endpoint = var.downstream_endpoint
      tls = {
        insecure = true
      }
      sending_queue = {
        enabled    = true
        queue_size = 5000
      }
      retry_on_failure = {
        enabled          = true
        initial_interval = "5s"
        max_interval     = "30s"
        max_elapsed_time = "300s"
      }
    }
  }

  otel_exporters = var.downstream_endpoint == "" ? local.exporters_direct : local.exporters_downstream

  # Pipeline exporter lists swap based on mode
  traces_exporters  = var.downstream_endpoint == "" ? ["datadog/connector", "datadog/exporter"] : ["datadog/connector", "otlp"]
  logs_exporters    = var.downstream_endpoint == "" ? ["datadog/exporter"] : ["otlp"]
  metrics_exporters = var.downstream_endpoint == "" ? ["datadog/exporter"] : ["otlp"]

  otel_config = {
    receivers = {
      otlp = {
        protocols = {
          grpc = {
            endpoint = "0.0.0.0:4317"
          }
          http = {
            endpoint = "0.0.0.0:4318"
          }
        }
      }
    }

    processors = {
      batch = {
        timeout             = "10s"
        send_batch_size     = 10
        send_batch_max_size = 75
      }
    }

    connectors = {
      "datadog/connector" = {}
    }

    exporters = local.otel_exporters

    extensions = {
      health_check = {
        endpoint = "$${env:MY_POD_IP}:13133"
      }
      "file_storage/persistent_queue" = {
        directory = "/var/lib/storage/queue"
      }
    }

    service = {
      extensions = ["health_check", "file_storage/persistent_queue"]
      pipelines = {
        traces = {
          receivers  = ["otlp"]
          processors = []
          exporters  = local.traces_exporters
        }

        logs = {
          receivers  = ["otlp"]
          processors = []
          exporters  = local.logs_exporters
        }

        metrics = {
          receivers  = ["otlp", "datadog/connector"]
          processors = []
          exporters  = local.metrics_exporters
        }
      }
      telemetry = {
        logs = {
          encoding = "json"
        }
      }
    }
  }
}


The file /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector/otelconfig.tf has been updated successfully. (file state is current in your context — no need to Read it back)


Wait — I need to check the original `service.extensions` list format. The original was `["health_check","file_storage/persistent_queue"]` (no spaces). The new one adds spaces. Since the spec requires byte-identical output in Case A mode, let me check if `yamlencode` would produce different output. Actually `yamlencode` doesn't care about HCL whitespace in list literals — the rendered YAML output will be the same. The list values are identical. Good.

**Step 2: Edit `main.tf`** — add `count` to the `kubernetes_manifest` resource and gate `extraEnvsFrom`.


Edit

resource "kubernetes_manifest" "otel-collector" {
  manifest = {

resource "kubernetes_manifest" "otel-collector" {
  count = var.downstream_endpoint == "" ? 1 : 0

  manifest = {


The file /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector/main.tf has been updated successfully. (file state is current in your context — no need to Read it back)


Now fix the `depends_on` references for `kubernetes_manifest.otel-collector` — with `count` on the resource, `depends_on` references to the resource without an index still work in Terraform (it depends on all instances). However, in practice Terraform may complain when the resource has `count` and you reference it in `depends_on` without an index. Let me check the current depends_on blocks and leave them as-is (no index needed in `depends_on`).

Now gate the `extraEnvsFrom` block:


    extraEnvsFrom = [{
      secretRef = {
        name = "otel-collector"
      }
    }]

    extraEnvsFrom = var.downstream_endpoint == "" ? [{
      secretRef = {
        name = "otel-collector"
      }
    }] : []


Now let me verify the full state of both edited files before running terraform commands:


34	resource "kubernetes_manifest" "otel-collector" {
35	  count = var.downstream_endpoint == "" ? 1 : 0
36
37	  manifest = {
38	    "apiVersion" = "external-secrets.io/v1"
39	    "kind"       = "ExternalSecret"
40	    "metadata" = {
41	      "name" = "otel-collector"
42	      "namespace" = var.namespace
43	    }
44	    "spec" = {
45	      "refreshInterval" = "1h0m0s"
46	      "secretStoreRef" = {
47	        "name" = "vault"
48	        "kind" = "ClusterSecretStore"
49	      }
50	      "target" = {
51	        "name"           = "otel-collector"
52	        "creationPolicy" = "Owner"
53	      }
54	      "data" = [
55	        {
56	          "secretKey" = "DD_API_KEY"
57	          "remoteRef" = {
58	            "key"      = "secret/data/saas/infrastructure/datadog-api-key"
59	            "property" = "apiKey"
60	          }
61	        },
62	      ]
63	    }
64	  }
65	  depends_on = [kubernetes_namespace.this]
66	}
67
68
69	resource "helm_release" "otel_collector_lb" {
70	  #name      = var.name
71	  name = "otel-collector-lb"
72	  namespace = var.namespace
73	  create_namespace = false
74	  repository = "https://open-telemetry.github.io/opentelemetry-helm-charts/"
75	  chart      = "opentelemetry-collector"
76	  version    = var.chart_version
77
78	  values = [yamlencode({
79	    mode = "deployment"
80	    image = {
81	      repository = var.otellb_image
82	      tag = var.image_tag_version
83	    }
84	    replicaCount = var.otel_lb_replicas
85
86	    nameOverride = "collector"
87
88	    annotations = var.annotations
89
90	    podLabels = {
91	      "ic-ops.net/otel" = "collector"
92	      "app.kubernetes.io/part-of" = "opentelemetry-collector"
93	    }


145	      }
146	    }
147
148	    config = local.lb_config
149	  })]
150
151	  depends_on = [
152	    kubernetes_namespace.this,
153	    kubernetes_manifest.otel-collector,
154	    helm_release.otel_collector_processor
155	  ]
156	}
157
158	resource "helm_release" "otel_collector_processor" {
159	  #name      = var.name
160	  name = "otel-collector-processor"
161	  namespace = var.namespace
162	  create_namespace = false
163	  repository = "https://open-telemetry.github.io/opentelemetry-helm-charts/"
164	  chart      = "opentelemetry-collector"
165	  version    = var.chart_version
166
167	  values = [yamlencode({
168	    mode = "statefulset"
169	    image = {
170	      repository = var.otel_image
171	      tag        = var.image_tag_version
172	    }
173	    statefulset = {
174	      persistentVolumeClaimRetentionPolicy = {
175	        enabled     = true
176	        whenDeleted = "Delete"
177	        whenScaled  = "Delete"
178	      }
179
180	      volumeClaimTemplates = [
181	        {
182	          metadata = {
183	            name = "otel-collector-data"
184	          }
185	          spec = {
186	            accessModes = ["ReadWriteOnce"]
187	            resources = {
188	              requests = {
189	                storage = var.otel_pv_size
190	              }
191	            }
192	            //storageClassName = var.otel_storage_class_name
193	          }
194	        }
195	      ]
196	    }
197
198	    extraVolumeMounts = [{
199	      name       = "otel-collector-data"
200	      mountPath  = "/var/lib/storage/queue"
201	    }]
202
203	    initContainers = [{
204	      name  = "init-fs"


230	      name   = var.otel_sa_name,
231	      annotations = var.sa_annotations
232	    }
233
234	    command = {
235	      extraArgs = var.extra_args
236	    }
237
238	    extraEnvsFrom = var.downstream_endpoint == "" ? [{
239	      secretRef = {
240	        name = "otel-collector"
241	      }
242	    }] : []
243
244	    #presets = {
245	      # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines
246	      #kubernetesAttributes = {
247	      #  enabled = true
248	      #  extractAllPodLabels = true
249	      #  extractAllPodAnnotations = true
250	      #}
251	    #}
252
253	    ports = {
254	      otlp = {
255	        enabled = true
256	        containerPort = 4317
257	        servicePort = 4317
258	        protocol = "TCP"
259	        appProtocol = "grpc"
260	      }
261	      otlp-http = {
262	        enabled = true
263	        containerPort = 4318
264	        servicePort = 4318
265	        protocol = "TCP"
266	      }
267	      jaegar-compact = {
268	        enabled = false
269	      }
270	      jaeger-grpc = {
271	        enabled = false
272	      }
273	      jaeger-thrift = {
274	        enabled = false
275	      }
276	      zipkin = {
277	        enabled = false
278	      }
279	    }
280
281	    config = local.otel_config
282	  })]
283
284	  depends_on = [
285	    kubernetes_namespace.this,
286	    kubernetes_manifest.otel-collector
287	  ]
288
289	  lifecycle {
290	    create_before_destroy = false
291	  }
292	}
293
294	module "netpol" {
295	  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0"
296
297	  name             = join("-", [var.name, "netpol"])
298	  namespace        = var.namespace
299	  network_policies = local.netpols


Good. Both edits look correct. Now let me run `terraform fmt -check -recursive` and `terraform validate`:


Bash

cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector && mise exec terraform@1.9.0 -- terraform fmt -check -recursive 2>&1; echo "EXIT: $?"

Run terraform fmt check on the module


lbconfig.tf
main.tf
EXIT: 3


Fmt check failed — there are formatting issues. Let me run `terraform fmt` (without `-check`) to auto-fix, then check which files changed:


cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector && mise exec terraform@1.9.0 -- terraform fmt -recursive 2>&1; echo "EXIT: $?"

Auto-format all terraform files in module


lbconfig.tf
main.tf
EXIT: 0
[This command modified 2 files you've previously read: ../otel-collector/main.tf, ../otel-collector/lbconfig.tf. Call Read before editing.]


`lbconfig.tf` was modified by fmt (pre-existing formatting issue, not caused by my changes). `main.tf` was also fmt'd. Let me verify the fmt-fixed versions look correct, then run `-check` again:


cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector && mise exec terraform@1.9.0 -- terraform fmt -check -recursive 2>&1; echo "EXIT: $?"

Verify terraform fmt check passes after formatting


EXIT: 0


`terraform fmt -check` passes. Now run `terraform validate`:


cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector && mise exec terraform@1.9.0 -- terraform validate 2>&1; echo "EXIT: $?"

Run terraform validate on the module


╷
│ Error: Inconsistent conditional result types
│
│   on otelconfig.tf line 36, in locals:
│   36:   otel_exporters = var.downstream_endpoint == "" ? local.exporters_direct : local.exporters_downstream
│     ├────────────────
│     │ local.exporters_direct is object with 1 attribute "datadog/exporter"
│     │ local.exporters_downstream is object with 1 attribute "otlp"
│     │ var.downstream_endpoint is a string
│
│ The true and false result expressions must have consistent types. The
│ 'true' value includes object attribute "datadog/exporter", which is absent
│ in the 'false' value.
╵
EXIT: 1


Terraform's type system requires both branches of a ternary to have the same structural type. The two objects have different attribute names (`"datadog/exporter"` vs `otlp`), so the ternary fails. I need to use a different approach.

The best approach here is to use separate locals and inline the conditional differently. One clean pattern: use `local.otel_config` as a map where the exporters key references `local.otel_exporters`, but since the two maps have structurally different types (different attribute keys), Terraform's ternary won't work directly.

The correct solution for HCL is to use `merge()` with a conditional empty map, but both branches need to be the same type. The cleanest workaround is to **not use a ternary on the top-level exporters** — instead, always include both exporter definitions and use a conditional to include/exclude.

Actually, the right approach in Terraform HCL is to avoid a ternary between objects with different shapes. Instead, I should make the `otel_config` local itself conditional between two complete configs, or use `jsonencode`/`yamlencode` with conditionals differently.

The most practical pattern: define two complete `otel_config` variants and select between them. But that's verbose. A cleaner approach is to use `merge()` with empty maps: since both objects always exist but with null/empty values for the "inactive" mode, but that doesn't work cleanly either.

Actually the cleanest real-world pattern for this case is to just define two separate locals for the complete config and use a ternary at the `local.otel_config` level. Or better: use `jsonencode` + `jsondecode` to bypass type checking. Or: define the exporters inside the `otel_config` local itself with a conditional expression that only the interpreter sees at runtime.

Wait — actually the issue is that Terraform evaluates type consistency at plan time. The real solution commonly used is: **make both objects have the same set of keys, using null for inactive ones, then filter nulls**. But HCL doesn't have a native "filter null keys" for maps.

The most idiomatic Terraform solution is: define `local.otel_config` twice (one for direct, one for downstream) and select between them with a ternary. Let me restructure:


Write

locals {
  # Pipeline exporter lists swap based on mode
  traces_exporters  = var.downstream_endpoint == "" ? ["datadog/connector", "datadog/exporter"] : ["datadog/connector", "otlp"]
  logs_exporters    = var.downstream_endpoint == "" ? ["datadog/exporter"] : ["otlp"]
  metrics_exporters = var.downstream_endpoint == "" ? ["datadog/exporter"] : ["otlp"]

  # Direct (Datadog) mode config
  otel_config_direct = {
    receivers = {
      otlp = {
        protocols = {
          grpc = {
            endpoint = "0.0.0.0:4317"
          }
          http = {
            endpoint = "0.0.0.0:4318"
          }
        }
      }
    }

    processors = {
      batch = {
        timeout             = "10s"
        send_batch_size     = 10
        send_batch_max_size = 75
      }
    }

    connectors = {
      "datadog/connector" = {}
    }

    exporters = {
      "datadog/exporter" = {
        api = {
          site = "datadoghq.com"
          key  = "$${env:DD_API_KEY}"
        }
        sending_queue = {
          enabled = true
          storage = "file_storage/persistent_queue"
        }
      }
    }

    extensions = {
      health_check = {
        endpoint = "$${env:MY_POD_IP}:13133"
      }
      "file_storage/persistent_queue" = {
        directory = "/var/lib/storage/queue"
      }
    }

    service = {
      extensions = ["health_check", "file_storage/persistent_queue"]
      pipelines = {
        traces = {
          receivers  = ["otlp"]
          processors = []
          exporters  = local.traces_exporters
        }

        logs = {
          receivers  = ["otlp"]
          processors = []
          exporters  = local.logs_exporters
        }

        metrics = {
          receivers  = ["otlp", "datadog/connector"]
          processors = []
          exporters  = local.metrics_exporters
        }
      }
      telemetry = {
        logs = {
          encoding = "json"
        }
      }
    }
  }

  # Downstream (OTLP/Vector) mode config
  otel_config_downstream = {
    receivers = {
      otlp = {
        protocols = {
          grpc = {
            endpoint = "0.0.0.0:4317"
          }
          http = {
            endpoint = "0.0.0.0:4318"
          }
        }
      }
    }

    processors = {
      batch = {
        timeout             = "10s"
        send_batch_size     = 10
        send_batch_max_size = 75
      }
    }

    connectors = {
      "datadog/connector" = {}
    }

    exporters = {
      otlp = {
        endpoint = var.downstream_endpoint
        tls = {
          insecure = true
        }
        sending_queue = {
          enabled    = true
          queue_size = 5000
        }
        retry_on_failure = {
          enabled          = true
          initial_interval = "5s"
          max_interval     = "30s"
          max_elapsed_time = "300s"
        }
      }
    }

    extensions = {
      health_check = {
        endpoint = "$${env:MY_POD_IP}:13133"
      }
      "file_storage/persistent_queue" = {
        directory = "/var/lib/storage/queue"
      }
    }

    service = {
      extensions = ["health_check", "file_storage/persistent_queue"]
      pipelines = {
        traces = {
          receivers  = ["otlp"]
          processors = []
          exporters  = local.traces_exporters
        }

        logs = {
          receivers  = ["otlp"]
          processors = []
          exporters  = local.logs_exporters
        }

        metrics = {
          receivers  = ["otlp", "datadog/connector"]
          processors = []
          exporters  = local.metrics_exporters
        }
      }
      telemetry = {
        logs = {
          encoding = "json"
        }
      }
    }
  }

  otel_config = var.downstream_endpoint == "" ? local.otel_config_direct : local.otel_config_downstream
}


Bash

cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector && mise exec terraform@1.9.0 -- terraform validate 2>&1; echo "EXIT: $?"

Validate terraform after restructuring otelconfig.tf


╷
│ Error: Inconsistent conditional result types
│
│   on otelconfig.tf line 169, in locals:
│  169:   otel_config = var.downstream_endpoint == "" ? local.otel_config_direct : local.otel_config_downstream
│     ├────────────────
│     │ local.otel_config_direct is object with 6 attributes
│     │ local.otel_config_downstream is object with 6 attributes
│     │ var.downstream_endpoint is a string
│
│ The true and false result expressions must have consistent types. Type
│ mismatch for object attribute "exporters": The 'true' value includes object
│ attribute "datadog/exporter", which is absent in the 'false' value.
╵
EXIT: 1


Terraform's type system is strict — it does structural type checking even when the objects have the same number of attributes but different attribute names. The ternary requires both branches to be exactly the same type.

The proper Terraform solution for this is to use `jsondecode(jsonencode(...))` to convert the objects to a generic `map(any)` type before the ternary. This is a well-known Terraform pattern for dealing with objects of different shapes:


Write

locals {
  # Pipeline exporter lists swap based on mode
  traces_exporters  = var.downstream_endpoint == "" ? ["datadog/connector", "datadog/exporter"] : ["datadog/connector", "otlp"]
  logs_exporters    = var.downstream_endpoint == "" ? ["datadog/exporter"] : ["otlp"]
  metrics_exporters = var.downstream_endpoint == "" ? ["datadog/exporter"] : ["otlp"]

  # Shared structural skeleton — receivers, processors, connectors, extensions, service
  # are identical in both modes; only exporters differs.
  _otel_base = {
    receivers = {
      otlp = {
        protocols = {
          grpc = {
            endpoint = "0.0.0.0:4317"
          }
          http = {
            endpoint = "0.0.0.0:4318"
          }
        }
      }
    }

    processors = {
      batch = {
        timeout             = "10s"
        send_batch_size     = 10
        send_batch_max_size = 75
      }
    }

    connectors = {
      "datadog/connector" = {}
    }

    extensions = {
      health_check = {
        endpoint = "$${env:MY_POD_IP}:13133"
      }
      "file_storage/persistent_queue" = {
        directory = "/var/lib/storage/queue"
      }
    }

    service = {
      extensions = ["health_check", "file_storage/persistent_queue"]
      pipelines = {
        traces = {
          receivers  = ["otlp"]
          processors = []
          exporters  = local.traces_exporters
        }

        logs = {
          receivers  = ["otlp"]
          processors = []
          exporters  = local.logs_exporters
        }

        metrics = {
          receivers  = ["otlp", "datadog/connector"]
          processors = []
          exporters  = local.metrics_exporters
        }
      }
      telemetry = {
        logs = {
          encoding = "json"
        }
      }
    }
  }

  # Direct (Datadog) exporter — active when downstream_endpoint is empty
  _exporters_direct = {
    exporters = {
      "datadog/exporter" = {
        api = {
          site = "datadoghq.com"
          key  = "$${env:DD_API_KEY}"
        }
        sending_queue = {
          enabled = true
          storage = "file_storage/persistent_queue"
        }
      }
    }
  }

  # Downstream (OTLP/Vector) exporter — active when downstream_endpoint is set
  _exporters_downstream = {
    exporters = {
      otlp = {
        endpoint = var.downstream_endpoint
        tls = {
          insecure = true
        }
        sending_queue = {
          enabled    = true
          queue_size = 5000
        }
        retry_on_failure = {
          enabled          = true
          initial_interval = "5s"
          max_interval     = "30s"
          max_elapsed_time = "300s"
        }
      }
    }
  }

  # merge() combines the shared base with the active exporter block.
  # jsondecode(jsonencode()) normalises both sides to map(any) so the
  # ternary does not fail on mismatched object attribute names.
  otel_config = merge(
    local._otel_base,
    jsondecode(jsonencode(
      var.downstream_endpoint == "" ? local._exporters_direct : local._exporters_downstream
    ))
  )
}


Bash

cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector && mise exec terraform@1.9.0 -- terraform validate 2>&1; echo "EXIT: $?"

Validate terraform with jsondecode/jsonencode pattern


╷
│ Error: Inconsistent conditional result types
│
│   on otelconfig.tf line 117, in locals:
│  117:       var.downstream_endpoint == "" ? local._exporters_direct : local._exporters_downstream
│     ├────────────────
│     │ local._exporters_direct is object with 1 attribute "exporters"
│     │ local._exporters_downstream is object with 1 attribute "exporters"
│     │ var.downstream_endpoint is a string
│
│ The true and false result expressions must have consistent types. Type
│ mismatch for object attribute "exporters": The 'true' value includes object
│ attribute "datadog/exporter", which is absent in the 'false' value.
╵
EXIT: 1


Terraform type-checks the ternary branches BEFORE applying `jsondecode`/`jsonencode`. The type check happens at the HCL parsing stage. I need to apply `jsonencode` to each branch separately before the ternary:


Edit

  # merge() combines the shared base with the active exporter block.
  # jsondecode(jsonencode()) normalises both sides to map(any) so the
  # ternary does not fail on mismatched object attribute names.
  otel_config = merge(
    local._otel_base,
    jsondecode(jsonencode(
      var.downstream_endpoint == "" ? local._exporters_direct : local._exporters_downstream
    ))
  )

  # Encode each exporter block to JSON first so the ternary operates on
  # uniform string types, bypassing Terraform's structural type check on
  # objects with different attribute names.
  _exporters_direct_json     = jsonencode(local._exporters_direct)
  _exporters_downstream_json = jsonencode(local._exporters_downstream)

  # merge() combines the shared base with the active exporter block.
  otel_config = merge(
    local._otel_base,
    jsondecode(
      var.downstream_endpoint == "" ? local._exporters_direct_json : local._exporters_downstream_json
    )
  )


Bash

cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector && mise exec terraform@1.9.0 -- terraform validate 2>&1; echo "EXIT: $?"

Validate terraform with pre-encoded JSON strings in ternary


Success! The configuration is valid.

EXIT: 0


`terraform validate` passes. Now run `terraform fmt -check` to verify formatting:


cd /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector && mise exec terraform@1.9.0 -- terraform fmt -check -recursive 2>&1; echo "EXIT: $?"

Check terraform formatting


Both pass. Now let me set up the tempdir tests for Cases A and B:


mkdir -p /tmp/otel-test-caseA /tmp/otel-test-caseB

Create temp test directories


(Bash completed with no output)


Write

terraform {
  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

module "otel" {
  source = "/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector"

  chart_version     = "0.112.0"
  image_tag_version = "0.112.0"

  # downstream_endpoint intentionally omitted — defaults to ""
}


File created successfully at: /tmp/otel-test-caseA/main.tf (file state is current in your context — no need to Read it back)


terraform {
  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

module "otel" {
  source = "/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector"

  chart_version     = "0.112.0"
  image_tag_version = "0.112.0"

  downstream_endpoint = "http://vector.vector.svc.cluster.local:4317"
}


File created successfully at: /tmp/otel-test-caseB/main.tf (file state is current in your context — no need to Read it back)


Bash

cd /tmp/otel-test-caseA && mise exec terraform@1.9.0 -- terraform init -backend=false 2>&1; echo "EXIT: $?"

Initialize Case A test module


Initializing modules...
Downloading file:///Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector for otel...
- otel in .terraform/modules/otel
Downloading git::ssh://git@gitlab.infrastructure.immuta.io/infrastructure/terraform/internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0 for otel.netpol...
- otel.netpol in .terraform/modules/otel.netpol
Initializing provider plugins...
- Finding hashicorp/kubernetes versions matching "~> 2.0"...
- Finding hashicorp/helm versions matching "~> 2.0"...
- Installing hashicorp/helm v2.17.0...
- Installed hashicorp/helm v2.17.0 (signed by HashiCorp)
- Installing hashicorp/kubernetes v2.38.0...
- Installed hashicorp/kubernetes v2.38.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
EXIT: 0


cd /tmp/otel-test-caseA && mise exec terraform@1.9.0 -- terraform plan -no-color 2>&1; echo "EXIT: $?"

Run terraform plan for Case A (no downstream endpoint)


Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.otel.helm_release.otel_collector_lb will be created
  + resource "helm_release" "otel_collector_lb" {
      + atomic                     = false
      + chart                      = "opentelemetry-collector"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "otel-collector-lb"
      + namespace                  = "opentelemetry"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://open-telemetry.github.io/opentelemetry-helm-charts/"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = [
          + <<-EOT
                "annotations": {}
                "command":
                  "extraArgs": []
                "config":
                  "exporters":
                    "loadbalancing":
                      "protocol":
                        "otlp":
                          "timeout": "2s"
                          "tls":
                            "insecure": true
                      "resolver":
                        "dns":
                          "hostname": "otel-collector-processor.opentelemetry.svc.cluster.local"
                      "routing_key": "traceID"
                    "loadbalancing/metrics":
                      "protocol":
                        "otlp":
                          "timeout": "2s"
                          "tls":
                            "insecure": true
                      "resolver":
                        "dns":
                          "hostname": "otel-collector-processor.opentelemetry.svc.cluster.local"
                      "routing_key": "service"
                  "processors":
                    "k8sattributes":
                      "auth_type": "serviceAccount"
                      "extract":
                        "labels":
                        - "from": "pod"
                          "key_regex": "tags\\.datadoghq\\.com/(.*)$"
                          "tag_name": "$$1"
                        "metadata":
                        - "k8s.namespace.name"
                        - "k8s.deployment.name"
                        - "k8s.statefulset.name"
                        - "k8s.daemonset.name"
                        - "k8s.cronjob.name"
                        - "k8s.job.name"
                        - "k8s.node.name"
                        - "k8s.pod.name"
                        - "k8s.pod.uid"
                        - "k8s.pod.start_time"
                      "passthrough": false
                      "pod_association":
                      - "sources":
                        - "from": "resource_attribute"
                          "name": "k8s.pod.ip"
                      - "sources":
                        - "from": "resource_attribute"
                          "name": "k8s.pod.uid"
                      - "sources":
                        - "from": "connection"
                  "receivers":
                    "otlp":
                      "protocols":
                        "grpc":
                          "endpoint": "0.0.0.0:4317"
                        "http":
                          "endpoint": "0.0.0.0:4318"
                  "service":
                    "pipelines":
                      "logs":
                        "exporters":
                        - "loadbalancing"
                        "processors":
                        - "k8sattributes"
                        "receivers":
                        - "otlp"
                      "metrics":
                        "exporters":
                        - "loadbalancing/metrics"
                        "processors":
                        - "k8sattributes"
                        "receivers":
                        - "otlp"
                      "traces":
                        "exporters":
                        - "loadbalancing"
                        "processors":
                        - "k8sattributes"
                        "receivers":
                        - "otlp"
                    "telemetry":
                      "logs":
                        "encoding": "json"
                "image":
                  "repository": "registry.ic-ops.net/cache-docker-hub/otel/opentelemetry-collector"
                  "tag": "0.112.0"
                "mode": "deployment"
                "nameOverride": "collector"
                "podLabels":
                  "app.kubernetes.io/part-of": "opentelemetry-collector"
                  "ic-ops.net/otel": "collector"
                "ports":
                  "jaegar-compact":
                    "enabled": false
                  "jaeger-grpc":
                    "enabled": false
                  "jaeger-thrift":
                    "enabled": false
                  "otlp":
                    "appProtocol": "grpc"
                    "containerPort": 4317
                    "enabled": true
                    "protocol": "TCP"
                    "servicePort": 4317
                  "otlp-http":
                    "containerPort": 4318
                    "enabled": true
                    "protocol": "TCP"
                    "servicePort": 4318
                  "zipkin":
                    "enabled": false
                "presets":
                  "kubernetesAttributes":
                    "enabled": true
                    "extractAllPodAnnotations": true
                    "extractAllPodLabels": false
                "replicaCount": 3
                "serviceAccount":
                  "annotations": {}
                  "create": true
                  "name": "otel-collector-lb-sa"
            EOT,
        ]
      + verify                     = false
      + version                    = "0.112.0"
      + wait                       = true
      + wait_for_jobs              = false
    }

  # module.otel.helm_release.otel_collector_processor will be created
  + resource "helm_release" "otel_collector_processor" {
      + atomic                     = false
      + chart                      = "opentelemetry-collector"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "otel-collector-processor"
      + namespace                  = "opentelemetry"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://open-telemetry.github.io/opentelemetry-helm-charts/"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = [
          + <<-EOT
                "annotations": {}
                "command":
                  "extraArgs": []
                "config":
                  "connectors":
                    "datadog/connector": {}
                  "exporters":
                    "datadog/exporter":
                      "api":
                        "key": "${env:DD_API_KEY}"
                        "site": "datadoghq.com"
                      "sending_queue":
                        "enabled": true
                        "storage": "file_storage/persistent_queue"
                  "extensions":
                    "file_storage/persistent_queue":
                      "directory": "/var/lib/storage/queue"
                    "health_check":
                      "endpoint": "${env:MY_POD_IP}:13133"
                  "processors":
                    "batch":
                      "send_batch_max_size": 75
                      "send_batch_size": 10
                      "timeout": "10s"
                  "receivers":
                    "otlp":
                      "protocols":
                        "grpc":
                          "endpoint": "0.0.0.0:4317"
                        "http":
                          "endpoint": "0.0.0.0:4318"
                  "service":
                    "extensions":
                    - "health_check"
                    - "file_storage/persistent_queue"
                    "pipelines":
                      "logs":
                        "exporters":
                        - "datadog/exporter"
                        "processors": []
                        "receivers":
                        - "otlp"
                      "metrics":
                        "exporters":
                        - "datadog/exporter"
                        "processors": []
                        "receivers":
                        - "otlp"
                        - "datadog/connector"
                      "traces":
                        "exporters":
                        - "datadog/connector"
                        - "datadog/exporter"
                        "processors": []
                        "receivers":
                        - "otlp"
                    "telemetry":
                      "logs":
                        "encoding": "json"
                "extraEnvsFrom":
                - "secretRef":
                    "name": "otel-collector"
                "extraVolumeMounts":
                - "mountPath": "/var/lib/storage/queue"
                  "name": "otel-collector-data"
                "image":
                  "repository": "registry.ic-ops.net/cache-docker-hub/otel/opentelemetry-collector-contrib"
                  "tag": "0.112.0"
                "initContainers":
                - "command":
                  - "sh"
                  - "-c"
                  - "chown -R 10001: /var/lib/storage/queue"
                  "image": "busybox:latest"
                  "name": "init-fs"
                  "volumeMounts":
                  - "mountPath": "/var/lib/storage/queue"
                    "name": "otel-collector-data"
                "mode": "statefulset"
                "nameOverride": "processor"
                "podLabels":
                  "app.kubernetes.io/part-of": "opentelemetry-collector"
                  "ic-ops.net/otel": "processor"
                "ports":
                  "jaegar-compact":
                    "enabled": false
                  "jaeger-grpc":
                    "enabled": false
                  "jaeger-thrift":
                    "enabled": false
                  "otlp":
                    "appProtocol": "grpc"
                    "containerPort": 4317
                    "enabled": true
                    "protocol": "TCP"
                    "servicePort": 4317
                  "otlp-http":
                    "containerPort": 4318
                    "enabled": true
                    "protocol": "TCP"
                    "servicePort": 4318
                  "zipkin":
                    "enabled": false
                "replicaCount": 3
                "service":
                  "clusterIP": "None"
                  "type": "ClusterIP"
                "serviceAccount":
                  "annotations": {}
                  "create": true
                  "name": "otel-collector-sa"
                "statefulset":
                  "persistentVolumeClaimRetentionPolicy":
                    "enabled": true
                    "whenDeleted": "Delete"
                    "whenScaled": "Delete"
                  "volumeClaimTemplates":
                  - "metadata":
                      "name": "otel-collector-data"
                    "spec":
                      "accessModes":
                      - "ReadWriteOnce"
                      "resources":
                        "requests":
                          "storage": "5Gi"
            EOT,
        ]
      + verify                     = false
      + version                    = "0.112.0"
      + wait                       = true
      + wait_for_jobs              = false
    }

  # module.otel.kubernetes_manifest.otel-collector[0] will be created
  + resource "kubernetes_manifest" "otel-collector" {
      + manifest = {
          + apiVersion = "external-secrets.io/v1"
          + kind       = "ExternalSecret"
          + metadata   = {
              + name      = "otel-collector"
              + namespace = "opentelemetry"
            }
          + spec       = {
              + data            = [
                  + {
                      + remoteRef = {
                          + key      = "secret/data/saas/infrastructure/datadog-api-key"
                          + property = "apiKey"
                        }
                      + secretKey = "DD_API_KEY"
                    },
                ]
              + refreshInterval = "1h0m0s"
              + secretStoreRef  = {
                  + kind = "ClusterSecretStore"
                  + name = "vault"
                }
              + target          = {
                  + creationPolicy = "Owner"
                  + name           = "otel-collector"
                }
            }
        }
      + object   = {
          + apiVersion = "external-secrets.io/v1"
          + kind       = "ExternalSecret"
          + metadata   = {
              + annotations                = (known after apply)
              + creationTimestamp          = (known after apply)
              + deletionGracePeriodSeconds = (known after apply)
              + deletionTimestamp          = (known after apply)
              + finalizers                 = (known after apply)
              + generateName               = (known after apply)
              + generation                 = (known after apply)
              + labels                     = (known after apply)
              + managedFields              = (known after apply)
              + name                       = "otel-collector"
              + namespace                  = "opentelemetry"
              + ownerReferences            = (known after apply)
              + resourceVersion            = (known after apply)
              + selfLink                   = (known after apply)
              + uid                        = (known after apply)
            }
          + spec       = {
              + data            = [
                  + {
                      + remoteRef = {
                          + conversionStrategy = (known after apply)
                          + decodingStrategy   = (known after apply)
                          + key                = "secret/data/saas/infrastructure/datadog-api-key"
                          + metadataPolicy     = (known after apply)
                          + nullBytePolicy     = (known after apply)
                          + property           = "apiKey"
                          + version            = (known after apply)
                        }
                      + secretKey = "DD_API_KEY"
                      + sourceRef = {
                          + generatorRef = {
                              + apiVersion = (known after apply)
                              + kind       = (known after apply)
                              + name       = (known after apply)
                            }
                          + storeRef     = {
                              + kind = (known after apply)
                              + name = (known after apply)
                            }
                        }
                    },
                ]
              + dataFrom        = (known after apply)
              + refreshInterval = "1h0m0s"
              + refreshPolicy   = (known after apply)
              + secretStoreRef  = {
                  + kind = "ClusterSecretStore"
                  + name = "vault"
                }
              + target          = {
                  + creationPolicy = "Owner"
                  + deletionPolicy = (known after apply)
                  + immutable      = (known after apply)
                  + manifest       = {
                      + apiVersion = (known after apply)
                      + kind       = (known after apply)
                    }
                  + name           = "otel-collector"
                  + template       = {
                      + data          = (known after apply)
                      + engineVersion = (known after apply)
                      + mergePolicy   = (known after apply)
                      + metadata      = {
                          + annotations = (known after apply)
                          + finalizers  = (known after apply)
                          + labels      = (known after apply)
                        }
                      + templateFrom  = (known after apply)
                      + type          = (known after apply)
                    }
                }
            }
        }
    }

  # module.otel.kubernetes_namespace.this will be created
  + resource "kubernetes_namespace" "this" {
      + id                               = (known after apply)
      + wait_for_default_service_account = false

      + metadata {
          + generation       = (known after apply)
          + name             = "opentelemetry"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # module.otel.module.netpol.helm_release.netpol will be created
  + resource "helm_release" "netpol" {
      + atomic                     = false
      + chart                      = "raw"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "otel-collector-netpol"
      + namespace                  = "opentelemetry"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://bedag.github.io/helm-charts"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = [
          + <<-EOT
                "resources":
                - "apiVersion": "cilium.io/v2"
                  "kind": "CiliumNetworkPolicy"
                  "metadata":
                    "annotations":
                      "sourceModule": "/internal-modules/kubernetes/monitoring/otel-collector"
                    "name": "otel-collector"
                    "namespace": "opentelemetry"
                  "spec":
                    "egress":
                    - "toEntities":
                      - "world"
                      "toPorts":
                      - "ports":
                        - "port": "443"
                    - "toEntities":
                      - "kube-apiserver"
                    - "toEntities":
                      - "cluster"
                    "endpointSelector":
                      "matchLabels":
                        "app.kubernetes.io/part-of": "opentelemetry-collector"
                    "ingress":
                    - "fromEntities":
                      - "host"
                      "toPorts":
                      - "ports":
                        - "port": "4317"
                        - "port": "4318"
                        - "port": "6831"
                    - "fromEntities":
                      - "kube-apiserver"
                    - "fromEntities":
                      - "cluster"
                      "toPorts":
                      - "ports":
                        - "port": "4317"
                        - "port": "4318"
                        - "port": "6831"
                    - "fromEndpoints":
                      - {}
            EOT,
        ]
      + verify                     = false
      + version                    = "2.0.0"
      + wait                       = true
      + wait_for_jobs              = false
    }

Plan: 5 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
EXIT: 0


Case A passes perfectly. I can see:
- `kubernetes_manifest.otel-collector[0]` — ExternalSecret IS created (count=1)
- `exporters.datadog/exporter` present with correct `api.key: "${env:DD_API_KEY}"` and `api.site: "datadoghq.com"`
- `extraEnvsFrom` contains the `otel-collector` secret ref
- Traces exporters: `["datadog/connector", "datadog/exporter"]`, logs/metrics: `["datadog/exporter"]`

Now Case B:


cd /tmp/otel-test-caseB && mise exec terraform@1.9.0 -- terraform init -backend=false 2>&1; echo "EXIT: $?"

Initialize Case B test module


Initializing modules...
Downloading file:///Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector for otel...
- otel in .terraform/modules/otel
Downloading git::ssh://git@gitlab.infrastructure.immuta.io/infrastructure/terraform/internal-modules/kubernetes/security/network-policy.git?ref=v1.1.0 for otel.netpol...
- otel.netpol in .terraform/modules/otel.netpol
Initializing provider plugins...
- Finding hashicorp/helm versions matching "~> 2.0"...
- Finding hashicorp/kubernetes versions matching "~> 2.0"...
- Installing hashicorp/helm v2.17.0...
- Installed hashicorp/helm v2.17.0 (signed by HashiCorp)
- Installing hashicorp/kubernetes v2.38.0...
- Installed hashicorp/kubernetes v2.38.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
EXIT: 0


cd /tmp/otel-test-caseB && mise exec terraform@1.9.0 -- terraform plan -no-color 2>&1; echo "EXIT: $?"

Run terraform plan for Case B (with downstream endpoint)


Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.otel.helm_release.otel_collector_lb will be created
  + resource "helm_release" "otel_collector_lb" {
      + atomic                     = false
      + chart                      = "opentelemetry-collector"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "otel-collector-lb"
      + namespace                  = "opentelemetry"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://open-telemetry.github.io/opentelemetry-helm-charts/"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = [
          + <<-EOT
                "annotations": {}
                "command":
                  "extraArgs": []
                "config":
                  "exporters":
                    "loadbalancing":
                      "protocol":
                        "otlp":
                          "timeout": "2s"
                          "tls":
                            "insecure": true
                      "resolver":
                        "dns":
                          "hostname": "otel-collector-processor.opentelemetry.svc.cluster.local"
                      "routing_key": "traceID"
                    "loadbalancing/metrics":
                      "protocol":
                        "otlp":
                          "timeout": "2s"
                          "tls":
                            "insecure": true
                      "resolver":
                        "dns":
                          "hostname": "otel-collector-processor.opentelemetry.svc.cluster.local"
                      "routing_key": "service"
                  "processors":
                    "k8sattributes":
                      "auth_type": "serviceAccount"
                      "extract":
                        "labels":
                        - "from": "pod"
                          "key_regex": "tags\\.datadoghq\\.com/(.*)$"
                          "tag_name": "$$1"
                        "metadata":
                        - "k8s.namespace.name"
                        - "k8s.deployment.name"
                        - "k8s.statefulset.name"
                        - "k8s.daemonset.name"
                        - "k8s.cronjob.name"
                        - "k8s.job.name"
                        - "k8s.node.name"
                        - "k8s.pod.name"
                        - "k8s.pod.uid"
                        - "k8s.pod.start_time"
                      "passthrough": false
                      "pod_association":
                      - "sources":
                        - "from": "resource_attribute"
                          "name": "k8s.pod.ip"
                      - "sources":
                        - "from": "resource_attribute"
                          "name": "k8s.pod.uid"
                      - "sources":
                        - "from": "connection"
                  "receivers":
                    "otlp":
                      "protocols":
                        "grpc":
                          "endpoint": "0.0.0.0:4317"
                        "http":
                          "endpoint": "0.0.0.0:4318"
                  "service":
                    "pipelines":
                      "logs":
                        "exporters":
                        - "loadbalancing"
                        "processors":
                        - "k8sattributes"
                        "receivers":
                        - "otlp"
                      "metrics":
                        "exporters":
                        - "loadbalancing/metrics"
                        "processors":
                        - "k8sattributes"
                        "receivers":
                        - "otlp"
                      "traces":
                        "exporters":
                        - "loadbalancing"
                        "processors":
                        - "k8sattributes"
                        "receivers":
                        - "otlp"
                    "telemetry":
                      "logs":
                        "encoding": "json"
                "image":
                  "repository": "registry.ic-ops.net/cache-docker-hub/otel/opentelemetry-collector"
                  "tag": "0.112.0"
                "mode": "deployment"
                "nameOverride": "collector"
                "podLabels":
                  "app.kubernetes.io/part-of": "opentelemetry-collector"
                  "ic-ops.net/otel": "collector"
                "ports":
                  "jaegar-compact":
                    "enabled": false
                  "jaeger-grpc":
                    "enabled": false
                  "jaeger-thrift":
                    "enabled": false
                  "otlp":
                    "appProtocol": "grpc"
                    "containerPort": 4317
                    "enabled": true
                    "protocol": "TCP"
                    "servicePort": 4317
                  "otlp-http":
                    "containerPort": 4318
                    "enabled": true
                    "protocol": "TCP"
                    "servicePort": 4318
                  "zipkin":
                    "enabled": false
                "presets":
                  "kubernetesAttributes":
                    "enabled": true
                    "extractAllPodAnnotations": true
                    "extractAllPodLabels": false
                "replicaCount": 3
                "serviceAccount":
                  "annotations": {}
                  "create": true
                  "name": "otel-collector-lb-sa"
            EOT,
        ]
      + verify                     = false
      + version                    = "0.112.0"
      + wait                       = true
      + wait_for_jobs              = false
    }

  # module.otel.helm_release.otel_collector_processor will be created
  + resource "helm_release" "otel_collector_processor" {
      + atomic                     = false
      + chart                      = "opentelemetry-collector"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "otel-collector-processor"
      + namespace                  = "opentelemetry"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://open-telemetry.github.io/opentelemetry-helm-charts/"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = [
          + <<-EOT
                "annotations": {}
                "command":
                  "extraArgs": []
                "config":
                  "connectors":
                    "datadog/connector": {}
                  "exporters":
                    "otlp":
                      "endpoint": "http://vector.vector.svc.cluster.local:4317"
                      "retry_on_failure":
                        "enabled": true
                        "initial_interval": "5s"
                        "max_elapsed_time": "300s"
                        "max_interval": "30s"
                      "sending_queue":
                        "enabled": true
                        "queue_size": 5000
                      "tls":
                        "insecure": true
                  "extensions":
                    "file_storage/persistent_queue":
                      "directory": "/var/lib/storage/queue"
                    "health_check":
                      "endpoint": "${env:MY_POD_IP}:13133"
                  "processors":
                    "batch":
                      "send_batch_max_size": 75
                      "send_batch_size": 10
                      "timeout": "10s"
                  "receivers":
                    "otlp":
                      "protocols":
                        "grpc":
                          "endpoint": "0.0.0.0:4317"
                        "http":
                          "endpoint": "0.0.0.0:4318"
                  "service":
                    "extensions":
                    - "health_check"
                    - "file_storage/persistent_queue"
                    "pipelines":
                      "logs":
                        "exporters":
                        - "otlp"
                        "processors": []
                        "receivers":
                        - "otlp"
                      "metrics":
                        "exporters":
                        - "otlp"
                        "processors": []
                        "receivers":
                        - "otlp"
                        - "datadog/connector"
                      "traces":
                        "exporters":
                        - "datadog/connector"
                        - "otlp"
                        "processors": []
                        "receivers":
                        - "otlp"
                    "telemetry":
                      "logs":
                        "encoding": "json"
                "extraEnvsFrom": []
                "extraVolumeMounts":
                - "mountPath": "/var/lib/storage/queue"
                  "name": "otel-collector-data"
                "image":
                  "repository": "registry.ic-ops.net/cache-docker-hub/otel/opentelemetry-collector-contrib"
                  "tag": "0.112.0"
                "initContainers":
                - "command":
                  - "sh"
                  - "-c"
                  - "chown -R 10001: /var/lib/storage/queue"
                  "image": "busybox:latest"
                  "name": "init-fs"
                  "volumeMounts":
                  - "mountPath": "/var/lib/storage/queue"
                    "name": "otel-collector-data"
                "mode": "statefulset"
                "nameOverride": "processor"
                "podLabels":
                  "app.kubernetes.io/part-of": "opentelemetry-collector"
                  "ic-ops.net/otel": "processor"
                "ports":
                  "jaegar-compact":
                    "enabled": false
                  "jaeger-grpc":
                    "enabled": false
                  "jaeger-thrift":
                    "enabled": false
                  "otlp":
                    "appProtocol": "grpc"
                    "containerPort": 4317
                    "enabled": true
                    "protocol": "TCP"
                    "servicePort": 4317
                  "otlp-http":
                    "containerPort": 4318
                    "enabled": true
                    "protocol": "TCP"
                    "servicePort": 4318
                  "zipkin":
                    "enabled": false
                "replicaCount": 3
                "service":
                  "clusterIP": "None"
                  "type": "ClusterIP"
                "serviceAccount":
                  "annotations": {}
                  "create": true
                  "name": "otel-collector-sa"
                "statefulset":
                  "persistentVolumeClaimRetentionPolicy":
                    "enabled": true
                    "whenDeleted": "Delete"
                    "whenScaled": "Delete"
                  "volumeClaimTemplates":
                  - "metadata":
                      "name": "otel-collector-data"
                    "spec":
                      "accessModes":
                      - "ReadWriteOnce"
                      "resources":
                        "requests":
                          "storage": "5Gi"
            EOT,
        ]
      + verify                     = false
      + version                    = "0.112.0"
      + wait                       = true
      + wait_for_jobs              = false
    }

  # module.otel.kubernetes_namespace.this will be created
  + resource "kubernetes_namespace" "this" {
      + id                               = (known after apply)
      + wait_for_default_service_account = false

      + metadata {
          + generation       = (known after apply)
          + name             = "opentelemetry"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # module.otel.module.netpol.helm_release.netpol will be created
  + resource "helm_release" "netpol" {
      + atomic                     = false
      + chart                      = "raw"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "otel-collector-netpol"
      + namespace                  = "opentelemetry"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://bedag.github.io/helm-charts"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = [
          + <<-EOT
                "resources":
                - "apiVersion": "cilium.io/v2"
                  "kind": "CiliumNetworkPolicy"
                  "metadata":
                    "annotations":
                      "sourceModule": "/internal-modules/kubernetes/monitoring/otel-collector"
                    "name": "otel-collector"
                    "namespace": "opentelemetry"
                  "spec":
                    "egress":
                    - "toEntities":
                      - "world"
                      "toPorts":
                      - "ports":
                        - "port": "443"
                    - "toEntities":
                      - "kube-apiserver"
                    - "toEntities":
                      - "cluster"
                    "endpointSelector":
                      "matchLabels":
                        "app.kubernetes.io/part-of": "opentelemetry-collector"
                    "ingress":
                    - "fromEntities":
                      - "host"
                      "toPorts":
                      - "ports":
                        - "port": "4317"
                        - "port": "4318"
                        - "port": "6831"
                    - "fromEntities":
                      - "kube-apiserver"
                    - "fromEntities":
                      - "cluster"
                      "toPorts":
                      - "ports":
                        - "port": "4317"
                        - "port": "4318"
                        - "port": "6831"
                    - "fromEndpoints":
                      - {}
            EOT,
        ]
      + verify                     = false
      + version                    = "2.0.0"
      + wait                       = true
      + wait_for_jobs              = false
    }

Plan: 4 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
EXIT: 0


Case B passes perfectly too. Now clean up the test directories:


rm -rf /tmp/otel-test-caseA /tmp/otel-test-caseB

Remove temporary test directories