You are implementing Task A15: write the GitLab CI config for the new `vector` Terraform internal-module and flesh out the README. Stop short of any push/tag operations.

## What's already in place

- Working directory: `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector`
- All module files in place: `versions.tf`, `variables.tf`, `outputs.tf`, `main.tf`, `templates/values.yaml.tpl`, `examples/basic/`, `tests/vrl/{drop_audit,redact_emails_logs,redact_emails_traces,redact_emails_metrics}/`.
- 4 VRL tests pass. `terraform validate` green at module root and in `examples/basic/`.

## Sibling-module reference

Read `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector/.gitlab-ci.yml` if it exists. If the otel-collector module has CI configured, mirror its style. If not, the suggested CI config in the task spec is the starting point.

If the otel-collector module DOES have CI but uses a different image (e.g., a private internal hashicorp/terraform mirror at `registry.ic-ops.net/...`), use the same image source.

## Task Description

### Task A15: CI configuration + README

**Files:**
- Create: `.gitlab-ci.yml`
- Modify: `README.md` (already exists with stub content)

- [ ] **Step 1: Read the sibling otel-collector's `.gitlab-ci.yml`** (if any) at `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector/.gitlab-ci.yml`. Note its image sources, stages, and any house-style conventions (e.g., does it use `before_script` for shared setup, custom runners via `tags:`, etc.).

- [ ] **Step 2: Write `.gitlab-ci.yml`** for the vector module. Suggested baseline (adapt to sibling style if it differs):

```yaml
stages: [lint, test]

terraform-fmt:
  stage: lint
  image: hashicorp/terraform:1.9
  script:
    - terraform fmt -check -recursive

terraform-validate:
  stage: test
  image: hashicorp/terraform:1.9
  script:
    - cd examples/basic
    - terraform init -backend=false
    - terraform validate

vrl-tests:
  stage: test
  image: timberio/vector:latest-debian
  before_script:
    - apt-get update && apt-get install -y bash diffutils jq
  script:
    - ./tests/vrl/drop_audit/run.sh
    - ./tests/vrl/redact_emails_logs/run.sh
    - ./tests/vrl/redact_emails_traces/run.sh
    - ./tests/vrl/redact_emails_metrics/run.sh
```

If the sibling module uses different image registries (e.g., `registry.ic-ops.net/cache-docker-hub/hashicorp/terraform:...`), substitute. The vector image source for VRL tests can be `registry.ic-ops.net/cache-docker-hub/timberio/vector:latest-debian` if a cached registry is used.

NOTE: The plan's original snapshot-tests CI job is OFF the table — the user opted out of snapshot tests. Do not include a `snapshot-tests` job.

- [ ] **Step 3: Flesh out the existing `README.md`.** Currently it's a 7-line stub. Expand it to:

  - Brief description (already there)
  - Architecture / pipeline shape (one paragraph: receives OTLP from OTel collector, applies drop_audit + email redaction, ships to Datadog).
  - Variable input table (read `variables.tf` and tabulate).
  - Output table (read `outputs.tf`).
  - Usage example (link to `examples/basic/`).
  - Testing notes (mention VRL tests and `terraform validate` / `fmt`).
  - Cross-link to the design spec at `docs/superpowers/specs/2026-05-05-vector-redaction-design.md`.

  Keep it concise and useful — this is platform docs, not a sales pitch. Reference the otel-collector module's `README.md` if it has a structure to mirror.

- [ ] **Step 4: Run `terraform fmt -check -recursive`** — exit 0 expected.

## Constraints

- **DO NOT run any git commands.** No `git add`, `git commit`, `git push`, `git tag`. The user handles git manually.
- **DO NOT** push, tag, open MRs, or call any `gh` / GitLab API command.
- **DO NOT** run `terraform apply`, `kubectl`, `helm install`, anything that mutates infra.
- The CI file is just being authored — it won't actually run unless the repo is pushed to GitLab. That's the user's job, post-handoff.

## Spec / plan references

- Spec: `/Users/steve.briggs/projects/IAC/docs/superpowers/specs/2026-05-05-vector-redaction-design.md`
- Plan: `/Users/steve.briggs/projects/IAC/docs/superpowers/plans/2026-05-05-vector-redaction-pipeline.md` Task A15

## Report Format

- **Status:** DONE | DONE_WITH_CONCERNS | BLOCKED | NEEDS_CONTEXT
- Files created/modified
- Sibling CI conventions: any divergences adopted (image sources, tags, etc.)
- README structure summary
- `terraform fmt` result
- Self-review findings


I'll start by reading the sibling module's CI config, README, and the relevant spec/plan files in parallel.


Bash

ls /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector/ 2>/dev/null && echo "EXISTS" || echo "NOT FOUND"

Check if otel-collector directory exists


Error: EISDIR: illegal operation on a directory, read '/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector'


.rw-r--r-- steve.briggs staff  70 B  Mon Feb  2 14:16:18 2026 .gitignore
.rw-r--r-- steve.briggs staff 1.2 KB Mon Feb  2 14:16:18 2026 netpol.tf
.rw-r--r-- steve.briggs staff 1.6 KB Mon Feb  2 14:16:18 2026 outputs.tf
drwxr-xr-x steve.briggs staff  96 B  Tue Feb  3 11:35:45 2026 .idea
.rw-r--r-- steve.briggs staff 6.6 KB Thu Feb 19 09:45:02 2026 main.tf
.rw-r--r-- steve.briggs staff 3.4 KB Thu Feb 19 09:45:02 2026 variables.tf
.rw-r--r-- steve.briggs staff 2.9 KB Thu Mar  5 16:42:45 2026 lbconfig.tf
.rw-r--r-- steve.briggs staff 1.5 KB Thu Mar  5 16:47:53 2026 otelconfig.tf
drwxr-xr-x steve.briggs staff 352 B  Thu Mar  5 16:47:53 2026 .
drwxr-xr-x steve.briggs staff 448 B  Thu Mar  5 16:48:00 2026 .git
drwxr-xr-x steve.briggs staff 224 B  Tue May  5 12:32:56 2026 ..
EXISTS


Error: File does not exist. Note: your current working directory is /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector.


1	# ---------------------------------------------------------------------------------------------------------------------
2	# REQUIRED PARAMETERS
3	# These variables must be set.
4	# ---------------------------------------------------------------------------------------------------------------------
5
6	variable "chart_version" {
7	  description = "Vector helm chart version pin"
8	  type        = string
9	}
10
11	variable "image_tag_version" {
12	  description = "Vector image tag"
13	  type        = string
14	}
15
16	variable "datadog_site" {
17	  description = "Datadog site (e.g. datadoghq.com, us3.datadoghq.com)"
18	  type        = string
19	}
20
21	variable "datadog_api_key_vault_path" {
22	  description = "Vault path for the Datadog API key, consumed by ExternalSecret"
23	  type        = string
24	}
25
26	variable "external_secrets_cluster_store_name" {
27	  description = "Name of the ClusterSecretStore created by external-secrets module"
28	  type        = string
29	}
30
31	# ---------------------------------------------------------------------------------------------------------------------
32	# OPTIONAL PARAMETERS
33	# These variables are configurable, but have reasonable defaults set.
34	# ---------------------------------------------------------------------------------------------------------------------
35
36	variable "image" {
37	  description = "Vector image source"
38	  type        = string
39	  default     = "registry.ic-ops.net/cache-docker-hub/timberio/vector"
40	}
41
42	variable "replicas" {
43	  description = "Number of Vector aggregator replicas"
44	  type        = number
45	  default     = 3
46	}
47
48	variable "audit_drop_patterns" {
49	  description = "Regex patterns; matching log message bodies are dropped"
50	  type        = list(string)
51	  default = [
52	    "Audit\\s-\\s.+",
53	    "\"level\":\"audit\"",
54	  ]
55	}
56
57	variable "email_redaction_enabled" {
58	  description = "Toggle the email-redact VRL transform across all signals"
59	  type        = bool
60	  default     = true
61	}
62
63	variable "email_redaction_replacement" {
64	  description = "String that replaces matched email addresses"
65	  type        = string
66	  default     = "[REDACTED-EMAIL]"
67	}
68
69	variable "additional_config" {
70	  description = "Escape hatch — overwrites helm customConfig wholesale (mirrors otel_collector_additional_config semantics)"
71	  type        = any
72	  default     = null
73	}
74
75	variable "resources_requests" {
76	  description = "Pod resource requests"
77	  type        = map(string)
78	  default = {
79	    cpu    = "200m"
80	    memory = "256Mi"
81	  }
82	}
83
84	variable "resources_limits" {
85	  description = "Pod resource limits"
86	  type        = map(string)
87	  default = {
88	    cpu    = "2"
89	    memory = "1Gi"
90	  }
91	}
92
93	variable "namespace" {
94	  description = "Kubernetes namespace for Vector"
95	  type        = string
96	  default     = "vector"
97	}
98
99	variable "service_monitor_enabled" {
100	  description = "Create a ServiceMonitor for Vector self-metrics. Requires kube-prometheus-stack CRDs in cluster."
101	  type        = bool
102	  default     = false
103	}
104


1	output "namespace" {
2	  description = "Namespace Vector is deployed into"
3	  value       = var.namespace
4	}
5
6	output "service_endpoint_grpc" {
7	  description = "Cluster-internal OTLP/gRPC endpoint to point upstream collectors at"
8	  value       = "http://vector.${var.namespace}.svc.cluster.local:4317"
9	}
10
11	output "service_endpoint_http" {
12	  description = "Cluster-internal OTLP/HTTP endpoint"
13	  value       = "http://vector.${var.namespace}.svc.cluster.local:4318"
14	}
15


1	# vector
2
3	Terraform module wrapping the `timberio/vector` Helm chart in aggregator role.
4	Receives OTLP from the OTel collector, applies VRL transforms (audit-log
5	filtering, email redaction), and ships to Datadog.
6
7	See `docs/superpowers/specs/2026-05-05-vector-redaction-design.md` in the IAC repo
8	for design context.
9
10	## Usage
11
12	See `examples/basic/`.
13


No sibling CI file or README exists for otel-collector. Now let me read the spec/plan for context, and check the otel-collector directory contents and vector tests structure.


1	# Vector + OpenTelemetry Redaction Pipeline — Design Spec
2
3	**Date:** 2026-05-05
4	**Author:** steve.briggs@immuta.com (with Claudebob)
5	**Status:** Draft, pending review
6
7	---
8
9	## 1. Overview
10
11	### Problem
12
13	Today the telemetry path on EKS clusters in this org is:
14
15	```
16	apps → otel-lb → otel-collector → Datadog (logs, APM, metrics)
17	```
18
19	Sensitive-data exclusion lives in **Datadog Agent processing rules** under
20	`kubernetes/applications/datadog-processing-rules`. Both `prod` and `staging` ship
21	the same `rules.json` containing two `exclude_at_match` entries that drop Immuta
22	audit log lines before they reach Datadog. `apm.json` is empty.
23
24	These rules are tied to the Datadog Agent's processing pipeline and only operate
25	on logs. They cannot redact span attributes or metric labels, and there is no
26	straightforward way to add modify-style transforms (e.g., replace email
27	addresses with a placeholder) within that surface.
28
29	### Goal
30
31	Insert **Vector** as a new in-cluster aggregator between the OTel collector and
32	Datadog. Vector replaces today's Datadog-side exclusion rules with VRL-based
33	filter and remap transforms, applied uniformly across logs, traces, and metrics.
34
35	The day-1 rule set replicates today's two audit exclusions and adds email
36	redaction across all three signals. The architecture leaves room to grow
37	additional patterns (SSN, credit cards, tokens, attribute drops, service-level
38	allow/deny) without further structural change.
39
40	### Non-goals (v1)
41
42	- Replacing the OTel collector.
43	- Removing the Datadog Agent (it remains for host metrics / kubelet stats).
44	- Decommissioning `kubernetes/applications/datadog-processing-rules` (separate effort).
45	- Per-cluster rule overlays beyond the Terraform variable surface.
46	- Dynamic / runtime-loaded rules.
47	- mTLS between OTel and Vector.
48	- Vector Agent-mode / DaemonSet collection.
49
50	---
51
52	## 2. Architecture
53
54	```
55	                                     [ in-cluster ]
56	  ┌──────────┐  OTLP  ┌─────────┐  OTLP  ┌────────────────┐  OTLP  ┌──────────────────┐ Datadog
57	  │   apps   │ ─────▶ │ otel-lb │ ─────▶ │ otel-collector │ ─────▶ │ vector aggregator│ ──────▶  Datadog SaaS
58	  └──────────┘        └─────────┘        └────────────────┘        │ - OTLP source    │  (logs, APM,
59	                                                                   │ - VRL transforms │   metrics)
60	                                                                   │ - Datadog sinks  │
61	                                                                   └──────────────────┘
62	```
63
64	**Components:**
65
66	- **OTel collector** keeps its current shape (`otel-lb` front tier + `otel-collector` back tier). Its terminal exporter switches from `datadog` to `otlp` (gRPC), pointed at the Vector Service.
67	- **Vector** runs as a Deployment with N replicas (default 3, matching OTel's tier), fronted by a ClusterIP Service. Receives OTLP, applies VRL transforms, ships to Datadog via Vector's native `datadog_logs` / `datadog_traces` / `datadog_metrics` sinks.
68	- **Datadog API key** moves from the OTel exporter config to Vector. Sourced from Vault via the existing `external-secrets` operator.
69
70	**Cluster scope.** Vector is enabled on the same cluster types as the OTel
71	collector by default: `mgmt`, `cps`, `saas` true; `infra`, `dev` false. A single
72	`vector_enabled` bool variable overrides the per-cluster-type default, matching
73	the established pattern in this module.
74
75	---
76
77	## 3. Terraform layout
78
79	Three changes across two repos.
80
81	### 3.1 `internal-modules/kubernetes/monitoring/otel-collector` v1.0.7 → v1.1.0
82
83	Backwards-compatible bump. New variables:
84
85	| Variable | Type | Default | Purpose |
86	|---|---|---|---|
87	| `downstream_endpoint` | string | `""` | When non-empty, swaps terminal exporter to OTLP |
88	| `downstream_protocol` | string | `"otlp/grpc"` | Wire protocol |
89
90	When `downstream_endpoint` is non-empty, the rendered helm values gain an
91	`exporters.otlp` block and `service.pipelines.{logs,traces,metrics}.exporters`
92	swap from `[datadog]` to `[otlp]`. The Datadog exporter block and its API-key
93	mount fall away under the same conditional. Empty preserves today's
94	direct-to-Datadog behavior — clusters that don't enable Vector see no behavioral
95	change.
96
97	### 3.2 `internal-modules/kubernetes/monitoring/vector` v1.0.0 (new module)
98
99	Mirrors the shape of the `otel-collector` internal-module: same provider wiring
100	(`kubernetes`, `helm`), same variable patterns, same approach to image / chart /
101	replica overrides.
102
103	What it creates:
104
105	- `kubernetes_namespace.vector` — the `vector` namespace.
106	- `kubernetes_manifest` for an `ExternalSecret` pulling the Datadog API key from Vault, using the same external-secrets cluster store the rest of the platform uses. Terraform waits for the underlying Secret object before proceeding.
107	- `helm_release.vector` — wraps the upstream `timberio/vector` chart with values templated from the variables below. Deployment annotated with `reloader.stakater.com/auto: "true"` so the existing reloader operator restarts pods on Datadog-key rotation.
108
109	Module variables (initial set):
110
111	| Variable | Type | Default | Purpose |
112	|---|---|---|---|
113	| `chart_version` | string | required | Vector helm chart version pin |
114	| `image` | string | `registry.ic-ops.net/cache-docker-hub/timberio/vector` | Image source |
115	| `image_tag_version` | string | required | Image tag |
116	| `replicas` | number | `3` | HA matches OTel collector tier |
117	| `datadog_site` | string | required | e.g. `datadoghq.com` / `us3.datadoghq.com` |
118	| `datadog_api_key_vault_path` | string | required | Vault path for ExternalSecret |
119	| `audit_drop_patterns` | list(string) | `["Audit\\s-\\s.+", "\"level\":\"audit\""]` | Day-1 exclude rules |
120	| `email_redaction_enabled` | bool | `true` | Toggle email redaction |
121	| `email_redaction_replacement` | string | `"[REDACTED-EMAIL]"` | Replacement marker |
122	| `additional_config` | any | `null` | Escape-hatch — overwrites `customConfig` wholesale, mirroring `otel_collector_additional_config` semantics |
123	| `resources_requests` / `resources_limits` | map | see Section 6 | Resource tuning |
124
125	### 3.3 `eks-cluster-applications` (this repo)
126
127	**New file `vector.tf`** — pure mirror of `otel_collector.tf`'s shape:
128
129	```hcl
130	locals {
131	  vector_enabled = (
132	    var.vector_enabled != null
133	    ? var.vector_enabled
134	    : lookup(var.vector_enabled_cluster_types, local.cluster_type, true)
135	  )
136	  vector_endpoint = "http://vector.vector.svc.cluster.local:4317"
137	}
138
139	module "vector" {
140	  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"
141	  count  = local.vector_enabled ? 1 : 0
142
143	  providers = {
144	    kubernetes = kubernetes.local-cluster
145	    helm       = helm.local-cluster
146	  }
147
148	  chart_version               = var.vector_chart_version
149	  image                       = var.vector_image
150	  image_tag_version           = var.vector_image_tag
151	  replicas                    = var.vector_replicas
152	  datadog_site                = var.vector_datadog_site
153	  datadog_api_key_vault_path  = var.vector_datadog_api_key_vault_path
154	  audit_drop_patterns         = var.vector_audit_drop_patterns
155	  email_redaction_enabled     = var.vector_email_redaction_enabled
156	  email_redaction_replacement = var.vector_email_redaction_replacement
157	  additional_config           = var.vector_additional_config
158
159	  depends_on = [
160	    module.cilium,
161	    module.external_secrets,
162	  ]
163	}
164	```
165
166	Plus variable declarations matching the otel-collector module's style.
167	`vector_enabled_cluster_types` defaults match OTel: `mgmt/cps/saas = true`,
168	`infra/dev = false`. A `data "external"` validation block enforces
169	`vector_enabled ⇒ external_secrets_enabled` (the Vault-backed key requires it).
170
171	**Edits to `otel_collector.tf`:**
172
173	- Bump `?ref=v1.0.7` → `?ref=v1.1.0`.
174	- Pass `downstream_endpoint = local.vector_enabled ? local.vector_endpoint : ""`.
175	- Add `depends_on = [module.vector]` so Vector's Service exists before OTel reconciles. (Belt-and-suspenders — OTLP exporter retries handle the runtime race.)
176
177	---
178
179	## 4. Vector configuration
180
181	Vector runs in **aggregator role** (it receives OTLP from another in-cluster
182	service rather than tailing files).
183
184	### 4.1 Pipeline topology
185
186	```
187	                ┌─ logs    ──▶ drop_audit ──▶ redact_emails_logs    ──▶ datadog_logs
188	otlp_in (4317   │
189	gRPC + 4318    ─┼─ traces  ──────────────────▶ redact_emails_traces  ──▶ datadog_traces
190	HTTP)          │
191	                └─ metrics ──────────────────▶ redact_emails_metrics ──▶ datadog_metrics
192	```
193
194	Three pipelines, one per signal. The `drop_audit` filter runs on logs only (the
195	day-1 rules are message-text patterns). Email redaction runs on all three.
196
197	### 4.2 VRL — audit-drop filter (logs only)
198
199	Vector's `filter` transform passes events for which the condition returns
200	`true`, so the condition is the inverse of what gets dropped:
201
202	```toml
203	[transforms.drop_audit]
204	type = "filter"
205	inputs = ["otlp_in.logs"]
206	condition = '''
207	  msg = string!(.message ?? "")
208	  !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"')
209	'''
210	```
211
212	`audit_drop_patterns` is templated into the helm values from the Terraform
213	variable. The default ships the two patterns from
214	`datadog-processing-rules/{prod,staging}/rules.json`.
215
216	### 4.3 VRL — email-redact transform (all three signals)
217
218	Per-signal `remap` transforms apply the same shared pattern. Log shape (the
219	simple case):
220
221	```toml
222	[transforms.redact_emails_logs]
223	type = "remap"
224	inputs = ["drop_audit"]
225	source = '''
226	  email_re = r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}'
227
228	  if exists(.message) {
229	    .message = replace(string!(.message), email_re, "[REDACTED-EMAIL]")
230	  }
231
232	  # walk top-level attributes for string values containing emails
233	  if is_object(.attributes) {
234	    .attributes = map_values(object!(.attributes)) -> |v| {
235	      if is_string(v) { replace(v, email_re, "[REDACTED-EMAIL]") } else { v }
236	    }
237	  }
238	'''
239	```
240
241	Trace and metric variants apply the same `map_values` walk over `attributes`
242	and `resource.attributes`, plus span-event attributes for traces. The email
243	regex and the replacement string are both templated from Terraform variables.
244
245	**Limitation called out:** if an email is buried inside a stringified JSON
246	attribute value (e.g. `http.request.body` holding raw JSON text), the regex
247	catches it because it operates on string content. If it's deeply nested in a
248	structured object map, only values reachable by `map_values` are scanned. v1
249	does **not** recurse into nested object trees — kept simple to avoid runaway
250	transform CPU on payload-heavy spans. Recursion can be added when justified.
251
252	### 4.4 Helm values surface
253
254	The internal-module renders to the upstream `timberio/vector` chart:
255
256	```yaml
257	role: Aggregator
258	replicas: {{ .Values.replicas }}            # 3 by default
259
260	customConfig:
261	  data_dir: /vector-data-dir
262	  api: { enabled: true, address: 127.0.0.1:8686 }
263	  sources:
264	    otlp_in:
265	      type: opentelemetry
266	      grpc: { address: 0.0.0.0:4317 }
267	      http: { address: 0.0.0.0:4318 }
268	  transforms:
269	    drop_audit:           { ... }            # see 4.2
270	    redact_emails_logs:    { ... }
271	    redact_emails_traces:  { ... }
272	    redact_emails_metrics: { ... }
273	  sinks:
274	    datadog_logs:
275	      type: datadog_logs
276	      inputs: [redact_emails_logs]
277	      default_api_key: "${DATADOG_API_KEY}"
278	      site: {{ .Values.datadogSite }}
279	    datadog_traces:    { type: datadog_traces,  inputs: [redact_emails_traces],  default_api_key: "${DATADOG_API_KEY}", site: ... }
280	    datadog_metrics:   { type: datadog_metrics, inputs: [redact_emails_metrics], default_api_key: "${DATADOG_API_KEY}", site: ... }
281
282	service:
283	  enabled: true
284	  type: ClusterIP
285	  ports:
286	    - { name: otlp-grpc, port: 4317, protocol: TCP }
287	    - { name: otlp-http, port: 4318, protocol: TCP }
288
289	env:
290	  - name: DATADOG_API_KEY
291	    valueFrom:
292	      secretKeyRef:
293	        name: vector-datadog
294	        key: api_key
295
296	podAnnotations:
297	  reloader.stakater.com/auto: "true"
298
299	resources:
300	  requests: { cpu: 200m, memory: 256Mi }
301	  limits:   { cpu: 2,    memory: 1Gi }
302	```
303
304	`additional_config` (escape-hatch variable) when set overwrites `customConfig`
305	wholesale, matching `otel_collector_additional_config` semantics.
306
307	---
308
309	## 5. OTel collector exporter swap
310
311	### 5.1 Helm values diff
312
313	**Before (`v1.0.7`, today):**
314
315	```yaml
316	exporters:
317	  datadog:
318	    api:
319	      key:  ${DATADOG_API_KEY}
320	      site: datadoghq.com
321	service:
322	  pipelines:
323	    logs:    { receivers: [otlp], processors: [...], exporters: [datadog] }
324	    traces:  { receivers: [otlp], processors: [...], exporters: [datadog] }
325	    metrics: { receivers: [otlp], processors: [...], exporters: [datadog] }
326	```
327
328	**After (`v1.1.0` when `downstream_endpoint != ""`):**
329
330	```yaml
331	exporters:
332	  otlp:
333	    endpoint: http://vector.vector.svc.cluster.local:4317
334	    tls:
335	      insecure: true                  # in-cluster only; Cilium handles transit
336	    sending_queue:
337	      enabled: true
338	      queue_size: 5000
339	    retry_on_failure:
340	      enabled: true
341	      initial_interval: 5s
342	      max_interval: 30s
343	      max_elapsed_time: 300s
344	service:
345	  pipelines:
346	    logs:    { receivers: [otlp], processors: [...], exporters: [otlp] }
347	    traces:  { receivers: [otlp], processors: [...], exporters: [otlp] }
348	    metrics: { receivers: [otlp], processors: [...], exporters: [otlp] }
349	```
350
351	The `exporters.datadog` block disappears. Pipeline `processors` lists are
352	untouched — anything OTel does today (batching, `k8sattributes`, tail-sampling)
353	stays put.
354
355	### 5.2 Datadog API key ownership
356
357	- **Before:** OTel collector pods consume the Datadog API key (mount mechanism per the OTel internal-module's existing pattern).
358	- **After:** Vector module owns the secret. Its ExternalSecret creates `vector-datadog` in the `vector` namespace; Vector pods consume it via `env.DATADOG_API_KEY`. The OTel module's secret mount falls away under the same `downstream_endpoint != ""` conditional that gates the exporter swap.
359	- **Sequencing:** `module.otel_collector` has `depends_on = [module.vector]`. Single `terraform apply` brings up Vector first, then re-renders OTel.
360
361	### 5.3 Failure-mode behavior
362
363	If Vector is down or unhealthy:
364
365	- OTel's `sending_queue` buffers up to `queue_size: 5000` batches in-memory per pipeline.
366	- `retry_on_failure` retries with exponential backoff up to 5 minutes total.
367	- Past that, OTel drops batches and increments `otelcol_exporter_send_failed_*`.
368	- **No persistent queue in v1.** A Vector outage longer than ~5 minutes loses the buffer tail. Mitigation: Vector replica HA (3 replicas, PDB minAvailable 2). `file_storage` extension is a future hardening item.
369
370	### 5.4 TLS posture
371
372	OTLP between OTel and Vector runs as plain gRPC, both endpoints in-cluster.
373	Cilium provides transport encryption. App-layer mTLS is a future hardening
374	item.
375
376	### 5.5 Rollout shape (single cluster)
377
378	Single `terraform apply`:
379
380	1. `module.vector` provisions — namespace, ExternalSecret, helm release reconcile to ready.
381	2. Same apply re-renders OTel collector helm release with `v1.1.0` and `downstream_endpoint` set — OTel pods rolling-restart with new exporter config.
382	3. Vector now receives all traffic; Datadog continues to receive (via Vector's sinks).
383
384	**Rollback:** set `vector_enabled = false`. The `local.vector_endpoint` ternary
385	in `otel_collector.tf` falls back to `""`; OTel re-renders with the original
386	`datadog` exporter, OTel pods rolling-restart back to direct-to-Datadog. Vector
387	module destroys cleanly. The Datadog API key path on the OTel side must remain
388	available across the rollback — this is a pre-condition; the OTel-side secret
389	reference is not deleted in the same release that introduces Vector.
390
391	---
392
393	## 6. Operational concerns
394
395	### 6.1 HA topology
396
397	- **Replicas:** 3 default (`vector_replicas`).
398	- **PodDisruptionBudget:** `minAvailable: 2`.
399	- **Topology spread:** `topologySpreadConstraints` on `topology.kubernetes.io/zone`, `maxSkew: 1`. Module gives operators a passthrough variable to override on small clusters where AZ spread is impossible.
400	- **Anti-affinity:** preferred (not required) on `kubernetes.io/hostname`.
401
402	### 6.2 Vector's own telemetry
403
404	Vector emits Prometheus-format internal metrics on `:8686/metrics`. If a
405	Prometheus operator (kube-prometheus-stack CRDs) is present in the cluster, the
406	module ships a `ServiceMonitor` for scraping. Otherwise, the metrics are
407	consumable via the Datadog Agent's OpenMetrics check or the OTel collector
408	itself (annotation-based scraping). The shipping mechanism is a v1
409	implementation question — see §8.3. Key metrics:
410
411	| Metric | What it tells you |
412	|---|---|
413	| `vector_component_received_events_total` | Inbound rate per source/transform/sink |
414	| `vector_component_sent_events_total` | Outbound rate at each Datadog sink |
415	| `vector_component_errors_total` | Failed sends per sink |
416	| `vector_component_discarded_events_total` | Events dropped by `drop_audit` |
417	| `vector_buffer_events` | In-memory queue depth toward each Datadog sink |
418	| `vector_utilization` | Per-component utilization, 0–1 |
419
420	Vector's own logs go to stdout and are captured by the existing OTel pipeline.
421	A Vector outage means we cannot see Vector's logs through Vector. First-line
422	debug: `kubectl logs -n vector`.
423
424	### 6.3 Alerting (sketch — actual monitor wiring out of scope)
425
426	| Condition | Severity | Why |
427	|---|---|---|
428	| `vector_component_errors_total{sink=~"datadog_.*"}` rate > 0 for 5m | warn | Datadog sink failing |
429	| `vector_buffer_events / vector_buffer_max_events > 0.8` for 5m | warn | Outbound queue filling |
430	| `vector_buffer_events / vector_buffer_max_events > 0.95` for 2m | page | About to drop |
431	| Vector ready replicas < 2 for 5m | page | Below PDB floor |
432	| `otelcol_exporter_send_failed_*{exporter="otlp"}` rate > 0 for 5m | warn | OTel→Vector path broken |
433
434	Datadog monitor wiring lives in this org's Datadog-as-Code surface, not the
435	Terraform module. **Required follow-up before declaring v1 production-ready.**
436
437	### 6.4 Sizing & resource defaults
438
439	```yaml
440	resources:
441	  requests: { cpu: 200m, memory: 256Mi }
442	  limits:   { cpu: 2,    memory: 1Gi }
443	```
444
445	Conservative starting points. Vector CPU usage scales with VRL transform
446	complexity (regex on every event isn't free). Real sizing comes from
447	observed metrics after a soak in `mgmt`/`cps` before rolling to `saas`.
448
449	### 6.5 Vault / ExternalSecret dependency chain
450
451	```
452	Vault ──▶ external-secrets operator ──▶ ExternalSecret (vector ns)
453	                                              │
454	                                              ▼
455	                                         Secret: vector-datadog
456	                                              │
457	                                              ▼
458	                                  Vector pods (env DATADOG_API_KEY)
459	```
460
461	The Vector module uses `kubernetes_manifest.wait` on the ExternalSecret's
462	materialized Secret to block the helm release until secret hydration completes.
463	Without that wait, the helm release races ahead and Vector pods CrashLoop on
464	startup. Reconcile order within the module:
465
466	1. Namespace
467	2. ExternalSecret (and wait until the underlying Secret resource is observed)
468	3. Helm release for Vector
469
470	### 6.6 Datadog API key rotation
471
472	Rotate the secret in Vault. ExternalSecret refreshes the in-cluster Secret. The
473	existing **`reloader` operator** picks up the change and restarts Vector pods —
474	the Vector Deployment is annotated with `reloader.stakater.com/auto: "true"`.
475	No manual intervention required.
476
477	### 6.7 Bootstrap-order edge case (first-time cluster provisioning)
478
479	A net-new cluster runs `terraform apply` once for layer-3
480	(`eks-cluster-applications`). Within that apply:
481
482	1. `external_secrets` reconciles first (existing dependency).
483	2. `vector` reconciles next (new — `depends_on = [module.cilium, module.external_secrets]`).
484	3. `otel_collector` reconciles last (`depends_on = [module.vector]`).
485
486	There is never an OTel collector running with `downstream_endpoint` set before
487	Vector is ready to receive.
488
489	### 6.8 Data-loss budget for v1
490
491	| Scenario | Loss |
492	|---|---|
493	| Vector single-pod restart | None — other replicas serve via Service load-balancing |
494	| Vector full-outage < 5 min | None — OTel `sending_queue` + `retry_on_failure` covers |
495	| Vector full-outage > 5 min | Tail of OTel queue dropped; Datadog gap proportional to outage minus 5 min |
496	| OTel collector full-outage | Same shape as today — apps' OTLP exporters retry/buffer per their config |
497	| Datadog API key rotation | None — reloader-driven pod restart |
498
499	---
500
501	## 7. Testing & verification plan
502
503	### 7.1 Pre-deploy (local / CI)
504
505	1. **Terraform plan** in a sandbox state to confirm module shape, count toggles cleanly.
506	2. **Helm template render** of both modules, diff against current rendered manifests in CI.
507	3. **`vector validate`** on the rendered config — catches VRL syntax errors. CI step in the new `internal-modules/kubernetes/monitoring/vector` repo.
508	4. **VRL unit tests** via `vector vrl --input ... --program ...`:
509	   - Audit-drop: `{"message":"Audit - login by user"}` → filtered. `{"message":"INFO request handled"}` → passed.
510	   - Email redact: `{"message":"contact alice@example.com"}` → `.message == "contact [REDACTED-EMAIL]"`.
511	   - Edge: dotted addresses, plus-tag, email inside JSON-stringified attribute. Test corpus committed alongside the module.
512
513	### 7.2 Integration (sandbox EKS cluster)
514
515	Use a `dev`-type cluster with `vector_enabled = true` explicit override.
516
517	1. **Smoke:** apply, confirm Vector pods Ready, OTel rolled with new `otlp` exporter, no `otelcol_exporter_send_failed`.
518	2. **Audit-drop:** emit log with body `Audit - test-event-<uuid>`. Search Datadog for the UUID. **Pass = no result.** Repeat for `"level":"audit"`.
519	3. **Email-redact (logs):** emit log `Reach me at e2e-<uuid>@example.com`. **Pass = log appears, body contains `[REDACTED-EMAIL]`, original local-part not present.**
520	4. **Email-redact (traces):** emit trace with span attribute `user.email = e2e-<uuid>@example.com`. **Pass = attribute is `[REDACTED-EMAIL]` in Datadog APM.**
521	5. **Negative test:** emit logs containing email-shaped strings inside code blocks. Confirm regex matches and redacts. Documented as known limitation, not a failure.
522
523	### 7.3 Failure-mode tests (sandbox)
524
525	1. **Vector pod restart:** `kubectl delete pod -n vector -l app=vector`. Brief dip, no Datadog gap > 1m.
526	2. **Vector full-stop ≤ 1m:** scale to zero, wait, scale back. Queue grows then drains; no permanent drop.
527	3. **Vector full-stop > 5m:** expect Datadog gap of (outage − 5m). Confirms documented data-loss budget.
528	4. **Rollback drill:** `vector_enabled = false`, `terraform apply`. OTel reverts to `datadog` exporter, telemetry resumes.
529	5. **Key rotation:** rotate in Vault. Confirm reloader-driven pod restart, telemetry resumes.
530
531	### 7.4 Production rollout sequence
532
533	1. **Sandbox / `dev`-type cluster** — explicit opt-in, full integration suite.
534	2. **One `mgmt` cluster** — soak 3-7 days. Tune resource limits if needed.
535	3. **Remaining `mgmt` + all `cps` clusters** — incremental.
536	4. **`saas` clusters last**, region-by-region, lowest-risk regions first.
537
538	Each promotion gate: green dashboards (no error-rate spikes, healthy queue
539	depths) for 48h on the previous tier.
540
541	### 7.5 Acceptance criteria for v1
542
543	- All `*_enabled` clusters running Vector with no `otelcol_exporter_send_failed` over a 7-day window.
544	- Datadog ingestion volume post-rollout matches pre-rollout within ±5% (modulo the audit-drop and redaction reductions, expected and quantifiable).
545	- Sample query in Datadog (`@message:*@*.* AND -@message:"\[REDACTED-EMAIL\]"`) shows zero unredacted email addresses.
546	- Day-1 audit patterns produce zero hits in Datadog log search post-rollout.
547	- Datadog monitors from §6.3 deployed and silent for the first 7 days outside of planned rollout-induced restarts.
548	- Runbook entry added covering: Vector pod debug, queue-full triage, key rotation, rollback to direct-Datadog. (Runbook home TBD — placeholder.)
549
550	---
551
552	## 8. Out of scope, phased work, open questions
553
554	### 8.1 Out of scope for v1 (restated for cross-reference)
555
556	- Decommissioning `kubernetes/applications/datadog-processing-rules` — separate effort.
557	- Per-cluster rule overlay machinery beyond the Terraform variable surface.
558	- Dynamic / runtime-loaded rules.
559	- Persistent (`file_storage`) queue on the OTel collector.
560	- mTLS between OTel and Vector.
561	- Recursive descent in the email-redact VRL transform.
562	- Vector Agent-mode (DaemonSet) for node-level collection.
563	- Datadog monitor wiring (required follow-up, lives in Datadog-as-Code surface).
564	- Operations runbook (required follow-up, home TBD).
565
566	### 8.2 Phased work — post-v1, in rough priority order
567
568	1. **Expand redaction patterns:** SSN, credit cards, tokens, JWT, phone numbers. Each ships as opt-in `vector_<category>_redaction_enabled` flag in the module.
569	2. **Field/attribute name allow-deny lists.** `db.statement`, `http.request.body`, `aws.s3.key` — wholesale strip by attribute name via `vector_drop_attributes`.
570	3. **Service-level allow/deny.** Drop telemetry from services where redaction can't be exhaustive. `route` transform on `service.name`.
571	4. **Decommission `datadog-processing-rules`.** After Vector has been clean in prod for one quarter.
572	5. **Persistent OTel queue.** If the data-loss budget proves insufficient.
573	6. **Runtime-driven rule sources.** If redaction surface keeps growing such that Terraform applies become friction, move rules behind a `kubernetes` or `http` source.
574
575	### 8.3 Open questions to resolve in implementation
576
577	1. **Datadog site value origin.** Likely re-use whatever variable feeds the OTel module today; confirm during implementation by reading the OTel internal-module repo.
578	2. **Vault path for Datadog API key.** Same — find where OTel reads it today, point Vector at the same path. Likely the existing `external_secrets_*` variable cluster.
579	3. **OTel internal-module's existing secret-mount mechanism.** Affects how cleanly `v1.1.0` removes the mount when `downstream_endpoint` is set.
580	4. **`datadog_agent.tf` overlap.** Datadog Agent module remains for non-OTel reasons (host metrics, kubelet stats). Vector + OTel does not replace it; the Agent side is untouched.
581	5. **Vector self-metrics scrape mechanism.** Confirm whether a Prometheus operator with kube-prometheus-stack CRDs is present in the target cluster types. If yes, ship a `ServiceMonitor`. If no, fall back to Datadog Agent's OpenMetrics check (annotation-driven) or OTel-collector-side scraping. Decide before tagging v1.0.0 of the Vector module.
582
583	---
584
585	## Appendix A — File touch list
586
587	| Repo | File | Action |
588	|---|---|---|
589	| `internal-modules/kubernetes/monitoring/otel-collector` | `main.tf`, `variables.tf`, helm values template | Add `downstream_endpoint`, `downstream_protocol` vars; conditionally render `exporters.otlp` block and gate Datadog secret mount. Tag `v1.1.0`. |
590	| `internal-modules/kubernetes/monitoring/vector` | (new repo) | New module v1.0.0. Namespace, ExternalSecret, helm release, ServiceMonitor, PDB. |
591	| `terraform/iac-library/aws/compute/eks-cluster-applications` | `vector.tf` | New file — locals, vars, `module "vector"` call, validation. |
592	| `terraform/iac-library/aws/compute/eks-cluster-applications` | `otel_collector.tf` | Bump module ref to v1.1.0; pass `downstream_endpoint`; add `depends_on = [module.vector]`. |
593
594	## Appendix B — Day-1 rule equivalence
595
596	Source rules in `kubernetes/applications/datadog-processing-rules/{prod,staging}/rules.json`:
597
598	```json
599	[
600	  { "type": "exclude_at_match", "name": "exclude_immuta_audit",   "pattern": "Audit\\s-\\s.+" },
601	  { "type": "exclude_at_match", "name": "exclude_immuta_audit_2", "pattern": "\"level\":\"audit\"" }
602	]
603	```
604
605	Becomes (in Vector VRL):
606
607	```vrl
608	condition = !match(msg, r'Audit\s-\s.+') && !match(msg, r'"level":"audit"')
609	```
610
611	Plus new email redaction across all signals (no equivalent in source rules):
612
613	```vrl
614	.message = replace(string!(.message), r'[\w._%+\-]+@[\w.-]+\.[A-Za-z]{2,}', "[REDACTED-EMAIL]")
615	```
616


find /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/tests -type f | sort

List all test files in vector module


find /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/otel-collector -maxdepth 2 -type f | sort

List otel-collector module files


The otel-collector module has no `.gitlab-ci.yml` and no `README.md`. However, I can see from `variables.tf` that the image source uses `registry.ic-ops.net/cache-docker-hub/timberio/vector`. Let me also check the run.sh scripts to understand how they invoke vector, and check the examples directory.


1	#!/usr/bin/env bash
2	set -euo pipefail
3	DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
4	ACTUAL="$DIR/.actual.ndjson"
5	vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
6	diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
7	echo "drop_audit VRL OK"
8


List examples/basic directory contents


drwxr-xr-x steve.briggs staff  96 B  Tue May  5 13:46:52 2026 ..
.rw-r--r-- steve.briggs staff 307 B  Tue May  5 13:47:15 2026 main.tf
.rw-r--r-- steve.briggs staff 207 B  Tue May  5 13:47:16 2026 README.md
drwxr-xr-x steve.briggs staff 128 B  Tue May  5 13:47:34 2026 .terraform
.rw-r--r-- steve.briggs staff 2.2 KB Tue May  5 13:47:37 2026 .terraform.lock.hcl
.rw-r--r-- steve.briggs staff 352 B  Tue May  5 13:47:52 2026 versions.tf
drwxr-xr-x steve.briggs staff 224 B  Tue May  5 13:47:52 2026 .


Good. The run scripts call `vector vrl` directly — no special flags needed beyond what's already there. The image source from `variables.tf` is `registry.ic-ops.net/cache-docker-hub/timberio/vector`, confirming the private registry mirror pattern. Now let me check one more run.sh and the versions.tf for terraform version info.


1	#!/usr/bin/env bash
2	set -euo pipefail
3	DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
4	ACTUAL="$DIR/.actual.ndjson"
5	vector vrl -p "$DIR/program.vrl" -i "$DIR/inputs.ndjson" -q -o > "$ACTUAL"
6	diff -u <(jq -c . "$DIR/expected.ndjson") <(jq -c . "$ACTUAL")
7	echo "redact_emails_logs VRL OK"
8


1	terraform {
2	  required_version = ">= 1.9.0"
3	  required_providers {
4	    kubernetes = { source = "hashicorp/kubernetes", version = ">= 2.20" }
5	    helm       = { source = "hashicorp/helm", version = ">= 2.10" }
6	  }
7	}
8
9	provider "kubernetes" {
10	  config_path = "~/.kube/config"
11	}
12
13	provider "helm" {
14	  kubernetes = {
15	    config_path = "~/.kube/config"
16	  }
17	}
18


I have everything I need. The run.sh scripts use `vector vrl` directly (no `bash` wrapper needed from outside), and `diff` + `jq` are needed. The terraform version requirement is `>= 1.9.0`. No sibling CI file exists to mirror, so I'll use the task spec's baseline adapted with the private registry image source. Let me now write both files.


Write

stages:
  - lint
  - test

terraform-fmt:
  stage: lint
  image: registry.ic-ops.net/cache-docker-hub/hashicorp/terraform:1.9
  script:
    - terraform fmt -check -recursive

terraform-validate:
  stage: test
  image: registry.ic-ops.net/cache-docker-hub/hashicorp/terraform:1.9
  script:
    - cd examples/basic
    - terraform init -backend=false
    - terraform validate

vrl-tests:
  stage: test
  image: registry.ic-ops.net/cache-docker-hub/timberio/vector:latest-debian
  before_script:
    - apt-get update -qq && apt-get install -y -qq bash diffutils jq
  script:
    - ./tests/vrl/drop_audit/run.sh
    - ./tests/vrl/redact_emails_logs/run.sh
    - ./tests/vrl/redact_emails_traces/run.sh
    - ./tests/vrl/redact_emails_metrics/run.sh


File created successfully at: /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/.gitlab-ci.yml (file state is current in your context — no need to Read it back)


Now flesh out the README. I already have all the variable and output data I need.


# vector

Terraform module wrapping the `timberio/vector` Helm chart in aggregator role.
Receives OTLP from the OTel collector, applies VRL transforms (audit-log
filtering, email redaction), and ships to Datadog.

See `docs/superpowers/specs/2026-05-05-vector-redaction-design.md` in the IAC repo
for design context.

---

## Architecture

Vector sits between the OTel collector tier and Datadog:

```
apps → otel-lb → otel-collector ──OTLP/gRPC──▶ vector aggregator ──▶ Datadog SaaS
```

Vector runs as a Deployment (default 3 replicas) fronted by a ClusterIP Service.
It exposes OTLP/gRPC on port 4317 and OTLP/HTTP on 4318. Three signal pipelines
run in parallel:

- **logs** — `drop_audit` filter (drops lines matching audit patterns) →
  `redact_emails_logs` remap → `datadog_logs` sink.
- **traces** — `redact_emails_traces` remap → `datadog_traces` sink.
- **metrics** — `redact_emails_metrics` remap → `datadog_metrics` sink.

The Datadog API key is sourced from Vault via the existing `external-secrets`
operator. The Deployment is annotated with `reloader.stakater.com/auto: "true"`
so key rotation triggers a rolling restart automatically.

---

## Inputs

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `chart_version` | `string` | yes | — | Vector Helm chart version pin |
| `image_tag_version` | `string` | yes | — | Vector image tag |
| `datadog_site` | `string` | yes | — | Datadog site (e.g. `datadoghq.com`, `us3.datadoghq.com`) |
| `datadog_api_key_vault_path` | `string` | yes | — | Vault path for the Datadog API key, consumed by ExternalSecret |
| `external_secrets_cluster_store_name` | `string` | yes | — | Name of the ClusterSecretStore created by the external-secrets module |
| `image` | `string` | no | `registry.ic-ops.net/cache-docker-hub/timberio/vector` | Vector image source |
| `replicas` | `number` | no | `3` | Number of Vector aggregator replicas |
| `namespace` | `string` | no | `"vector"` | Kubernetes namespace for Vector |
| `audit_drop_patterns` | `list(string)` | no | `["Audit\\s-\\s.+", "\"level\":\"audit\""]` | Regex patterns; matching log message bodies are dropped |
| `email_redaction_enabled` | `bool` | no | `true` | Toggle the email-redact VRL transform across all signals |
| `email_redaction_replacement` | `string` | no | `"[REDACTED-EMAIL]"` | String that replaces matched email addresses |
| `resources_requests` | `map(string)` | no | `{cpu="200m", memory="256Mi"}` | Pod resource requests |
| `resources_limits` | `map(string)` | no | `{cpu="2", memory="1Gi"}` | Pod resource limits |
| `service_monitor_enabled` | `bool` | no | `false` | Create a ServiceMonitor for Vector self-metrics (requires kube-prometheus-stack CRDs) |
| `additional_config` | `any` | no | `null` | Escape hatch — overwrites `customConfig` wholesale |

---

## Outputs

| Name | Description |
|---|---|
| `namespace` | Namespace Vector is deployed into |
| `service_endpoint_grpc` | Cluster-internal OTLP/gRPC endpoint (`http://vector.<namespace>.svc.cluster.local:4317`) |
| `service_endpoint_http` | Cluster-internal OTLP/HTTP endpoint (`http://vector.<namespace>.svc.cluster.local:4318`) |

---

## Usage

See [`examples/basic/`](examples/basic/) for a minimal working example.

```hcl
module "vector" {
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"

  providers = {
    kubernetes = kubernetes.local-cluster
    helm       = helm.local-cluster
  }

  chart_version                        = "0.36.0"
  image_tag_version                    = "0.41.0-debian"
  datadog_site                         = "datadoghq.com"
  datadog_api_key_vault_path           = "secret/data/platform/datadog/api_key"
  external_secrets_cluster_store_name  = "vault-cluster-store"
}
```

Point the OTel collector at Vector's gRPC endpoint via the `downstream_endpoint`
variable in the `otel-collector` module (v1.1.0+):

```hcl
downstream_endpoint = module.vector.service_endpoint_grpc
```

---

## Testing

### VRL unit tests

Each transform has an input/expected fixture pair and a shell runner under
`tests/vrl/<transform-name>/`. Run all four locally (requires `vector` and `jq`
on PATH):

```sh
./tests/vrl/drop_audit/run.sh
./tests/vrl/redact_emails_logs/run.sh
./tests/vrl/redact_emails_traces/run.sh
./tests/vrl/redact_emails_metrics/run.sh
```

### Terraform

```sh
# format check
terraform fmt -check -recursive

# validate the example
cd examples/basic
terraform init -backend=false
terraform validate
```

CI runs all three checks automatically on every push (see `.gitlab-ci.yml`).


The file /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/README.md has been updated successfully. (file state is current in your context — no need to Read it back)


Now run `terraform fmt -check -recursive` from the module root:


Bash

terraform fmt -check -recursive; echo "exit: $?"

Run terraform fmt check recursively from module root


exit: 0


Clean exit. All done, Sir Charles of St Cheese.

---

**Status:** DONE

**Files created/modified:**

- `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/.gitlab-ci.yml` — created
- `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/README.md` — expanded from 7-line stub

**Sibling CI conventions:**

The otel-collector module has no `.gitlab-ci.yml` and no `README.md` — it was the first of its kind in this location. No style to mirror from the sibling. The one concrete divergence adopted from the existing codebase: both the `hashicorp/terraform` and `timberio/vector` images are pulled from the internal cached registry (`registry.ic-ops.net/cache-docker-hub/...`), matching the default `image` variable already baked into `variables.tf`. The `vrl-tests` job's `before_script` uses `-qq` for quieter apt output but otherwise matches the task spec baseline.

**README structure:**

1. One-line description (retained from stub)
2. Architecture section — prose paragraph + ASCII pipeline diagram, covering the three-signal pipeline topology, Vault/ExternalSecret key sourcing, and reloader annotation
3. Inputs table — all 15 variables from `variables.tf` with types, required/optional, defaults, and descriptions
4. Outputs table — all 3 outputs from `outputs.tf`
5. Usage example — HCL snippet + note on wiring the OTel `downstream_endpoint`
6. Testing section — VRL test invocations and `terraform fmt`/`validate` commands, with a note that CI covers all three
7. Cross-link to the design spec is in the opening paragraph (carried from stub, kept prominent)

**`terraform fmt -check -recursive` result:** exit 0, no files reformatted.

**Self-review findings:**

- The `vrl-tests` job calls the run scripts from the repo root. The scripts use `DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` for self-relative paths, so working directory does not matter — this is fine.
- `terraform init -backend=false` in the CI validate job will attempt to pull providers. If the GitLab runner is air-gapped, a provider mirror or `TF_CLI_CONFIG_FILE` pointing at an internal Terraform registry may be needed. That's a runner-environment concern, not a module concern.
- No `snapshot-tests` job included, per the user's opt-out.