type: user

role: user

content:
You are implementing Task A6 of the "DD Agent Logs via Vector" plan.

## Task Description

Update the Vector module's README to document the new DD Agent OPW intake.

**File to modify:** `/Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/README.md`

Five separate edits — make ALL FIVE in this single task.

### Edit 1 — Replace the Architecture section (lines 14-32)

Find this existing block (it starts after the `## Architecture` heading at line 12 and ends just before `---` at line 33):

```
Vector sits between the OTel collector tier and Datadog:

```
apps → otel-lb → otel-collector ──OTLP/gRPC──▶ vector aggregator ──▶ Datadog SaaS
```

Vector runs as a Deployment (default 3 replicas) fronted by a ClusterIP Service.
It exposes OTLP/gRPC on port 4317 and OTLP/HTTP on 4318. Three signal pipelines
run in parallel:

- **logs** — `drop_audit` filter (drops lines matching audit patterns) →
  `redact_emails_logs` remap → `datadog_logs` sink.
- **traces** — `redact_emails_traces` remap → `datadog_traces` sink.
- **metrics** — `redact_emails_metrics` remap → `datadog_metrics` sink.

The Datadog API key is sourced from Vault via the existing `external-secrets`
operator. The Deployment is annotated with `reloader.stakater.com/auto: "true"`
so key rotation triggers a rolling restart automatically.
```

Replace it with:

```
Vector sits between the OTel collector tier and Datadog, and additionally
serves as an OPW-compatible logs intake for the Datadog Agent:

```
apps → otel-lb → otel-collector ──OTLP/gRPC──▶ vector ──▶ Datadog SaaS
                                                  ▲
                                          DD Agent (logs)
```

Vector runs as a Deployment (default 3 replicas) fronted by a ClusterIP
Service. It exposes:

- OTLP/gRPC on port 4317 (from OTel collector)
- OTLP/HTTP on port 4318 (from OTel collector)
- Datadog Agent intake on port 8282 (from DD Agent, OPW protocol)

Three signal pipelines run in parallel:

- **logs** — `drop_audit` filter (multi-input: OTLP logs + DD Agent logs;
  drops lines matching audit patterns) → `redact_emails_logs` remap →
  `datadog_logs` sink.
- **traces** — `redact_emails_traces` remap → `datadog_traces` sink.
- **metrics** — `redact_emails_metrics` remap → `datadog_metrics` sink.

The Datadog API key is sourced from Vault via the existing `external-secrets`
operator. The Deployment is annotated with `reloader.stakater.com/auto: "true"`
so key rotation triggers a rolling restart automatically.
```

### Edit 2 — Append a row to the Outputs table

Locate the Outputs table (around lines 59-63). Currently has 3 rows: `namespace`, `service_endpoint_grpc`, `service_endpoint_http`. Append a 4th:

```
| `service_endpoint_dd_agent` | OPW-compatible Datadog Agent intake endpoint (`http://vector.<namespace>.svc.cluster.local:8282`) |
```

### Edit 3 — Update the version pin in the Usage example

Locate the Usage example (around line 73). The line currently reads:
```
  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"
```
Change `v1.0.0` to `v1.1.0`.

### Edit 4 — Add a usage hint for the DD Agent wire-up

After the existing OTel-collector usage hint (the block around lines 88-93 that talks about `downstream_endpoint = module.vector.service_endpoint_grpc`), insert a new block:

```
Point the Datadog Agent at Vector's OPW intake by passing the new endpoint to
the `datadog-agent` module (v1.2.0+):

```hcl
logs_intake_url = module.vector.service_endpoint_dd_agent
```
```

(That's a markdown code block with the hcl content.)

### Edit 5 — Update the Testing section

In the Testing section (around line 108-110), the current VRL test list has 4 entries. Add a 5th:

```
./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

So the block becomes:

```
./tests/vrl/drop_audit/run.sh
./tests/vrl/redact_emails_logs/run.sh
./tests/vrl/redact_emails_traces/run.sh
./tests/vrl/redact_emails_metrics/run.sh
./tests/vrl/redact_emails_dd_agent_logs/run.sh
```

## Context

This is documentation only. No tests, no terraform validate. Just five separate text edits in README.md.

## CRITICAL — No git operations

Do NOT run any git commands. Sir Charles handles git himself.

## Your Job

1. Read the current README.md to identify exact text to replace.
2. Make all 5 edits.
3. Self-review by re-reading and confirming all 5 changes are present.
4. Report back.

## Report Format

- **Status:** DONE | DONE_WITH_CONCERNS | BLOCKED | NEEDS_CONTEXT
- For each of the 5 edits, confirm it's present (with the line numbers where it now lives)
- Any concerns

userType: external

entrypoint: cli

gitBranch: master


type: message

role: assistant

type: text

text: I'll start by reading the current README.md to identify the exact text and make all five edits.

service_tier: standard

attributionAgent: general-purpose

attributionPlugin: superpowers

type: assistant


Read

type: direct

speed: standard


content:
1	# vector
2
3	Terraform module wrapping the `timberio/vector` Helm chart in aggregator role.
4	Receives OTLP from the OTel collector, applies VRL transforms (audit-log
5	filtering, email redaction), and ships to Datadog.
6
7	See `docs/superpowers/specs/2026-05-05-vector-redaction-design.md` in the IAC repo
8	for design context.
9
10	---
11
12	## Architecture
13
14	Vector sits between the OTel collector tier and Datadog:
15
16	```
17	apps → otel-lb → otel-collector ──OTLP/gRPC──▶ vector aggregator ──▶ Datadog SaaS
18	```
19
20	Vector runs as a Deployment (default 3 replicas) fronted by a ClusterIP Service.
21	It exposes OTLP/gRPC on port 4317 and OTLP/HTTP on 4318. Three signal pipelines
22	run in parallel:
23
24	- **logs** — `drop_audit` filter (drops lines matching audit patterns) →
25	  `redact_emails_logs` remap → `datadog_logs` sink.
26	- **traces** — `redact_emails_traces` remap → `datadog_traces` sink.
27	- **metrics** — `redact_emails_metrics` remap → `datadog_metrics` sink.
28
29	The Datadog API key is sourced from Vault via the existing `external-secrets`
30	operator. The Deployment is annotated with `reloader.stakater.com/auto: "true"`
31	so key rotation triggers a rolling restart automatically.
32
33	---
34
35	## Inputs
36
37	| Name | Type | Required | Default | Description |
38	|---|---|---|---|---|
39	| `chart_version` | `string` | yes | — | Vector Helm chart version pin |
40	| `image_tag_version` | `string` | yes | — | Vector image tag |
41	| `datadog_site` | `string` | yes | — | Datadog site (e.g. `datadoghq.com`, `us3.datadoghq.com`) |
42	| `datadog_api_key_vault_path` | `string` | yes | — | Vault path for the Datadog API key, consumed by ExternalSecret |
43	| `external_secrets_cluster_store_name` | `string` | yes | — | Name of the ClusterSecretStore created by the external-secrets module |
44	| `image` | `string` | no | `registry.ic-ops.net/cache-docker-hub/timberio/vector` | Vector image source |
45	| `replicas` | `number` | no | `3` | Number of Vector aggregator replicas |
46	| `namespace` | `string` | no | `"vector"` | Kubernetes namespace for Vector |
47	| `audit_drop_patterns` | `list(string)` | no | `["Audit\\s-\\s.+", "\"level\":\"audit\""]` | Regex patterns; matching log message bodies are dropped |
48	| `email_redaction_enabled` | `bool` | no | `true` | Toggle the email-redact VRL transform across all signals |
49	| `email_redaction_replacement` | `string` | no | `"[REDACTED-EMAIL]"` | String that replaces matched email addresses |
50	| `resources_requests` | `map(string)` | no | `{cpu="200m", memory="256Mi"}` | Pod resource requests |
51	| `resources_limits` | `map(string)` | no | `{cpu="2", memory="1Gi"}` | Pod resource limits |
52	| `service_monitor_enabled` | `bool` | no | `false` | Create a ServiceMonitor for Vector self-metrics (requires kube-prometheus-stack CRDs) |
53	| `additional_config` | `any` | no | `null` | Escape hatch — overwrites `customConfig` wholesale |
54
55	---
56
57	## Outputs
58
59	| Name | Description |
60	|---|---|
61	| `namespace` | Namespace Vector is deployed into |
62	| `service_endpoint_grpc` | Cluster-internal OTLP/gRPC endpoint (`http://vector.<namespace>.svc.cluster.local:4317`) |
63	| `service_endpoint_http` | Cluster-internal OTLP/HTTP endpoint (`http://vector.<namespace>.svc.cluster.local:4318`) |
64
65	---
66
67	## Usage
68
69	See [`examples/basic/`](examples/basic/) for a minimal working example.
70
71	```hcl
72	module "vector" {
73	  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.0.0"
74
75	  providers = {
76	    kubernetes = kubernetes.local-cluster
77	    helm       = helm.local-cluster
78	  }
79
80	  chart_version                        = "0.36.0"
81	  image_tag_version                    = "0.41.0-debian"
82	  datadog_site                         = "datadoghq.com"
83	  datadog_api_key_vault_path           = "secret/data/platform/datadog/api_key"
84	  external_secrets_cluster_store_name  = "vault-cluster-store"
85	}
86	```
87
88	Point the OTel collector at Vector's gRPC endpoint via the `downstream_endpoint`
89	variable in the `otel-collector` module (v1.1.0+):
90
91	```hcl
92	downstream_endpoint = module.vector.service_endpoint_grpc
93	```
94
95	---
96
97	## Testing
98
99	### VRL unit tests
100
101	Each transform has an input/expected fixture pair and a shell runner under
102	`tests/vrl/<transform-name>/`. Run all four locally (requires `vector` and `jq`
103	on PATH):
104
105	```sh
106	./tests/vrl/drop_audit/run.sh
107	./tests/vrl/redact_emails_logs/run.sh
108	./tests/vrl/redact_emails_traces/run.sh
109	./tests/vrl/redact_emails_metrics/run.sh
110	```
111
112	### Terraform
113
114	```sh
115	# format check
116	terraform fmt -check -recursive
117
118	# validate the example
119	cd examples/basic
120	terraform init -backend=false
121	terraform validate
122	```
123
124	CI runs all three checks automatically on every push (see `.gitlab-ci.yml`).
125


text: Now I'll make all five edits. Let me start with Edit 1 — replacing the Architecture section.


Edit

old_string:
Vector sits between the OTel collector tier and Datadog:

```
apps → otel-lb → otel-collector ──OTLP/gRPC──▶ vector aggregator ──▶ Datadog SaaS
```

Vector runs as a Deployment (default 3 replicas) fronted by a ClusterIP Service.
It exposes OTLP/gRPC on port 4317 and OTLP/HTTP on 4318. Three signal pipelines
run in parallel:

- **logs** — `drop_audit` filter (drops lines matching audit patterns) →
  `redact_emails_logs` remap → `datadog_logs` sink.
- **traces** — `redact_emails_traces` remap → `datadog_traces` sink.
- **metrics** — `redact_emails_metrics` remap → `datadog_metrics` sink.

The Datadog API key is sourced from Vault via the existing `external-secrets`
operator. The Deployment is annotated with `reloader.stakater.com/auto: "true"`
so key rotation triggers a rolling restart automatically.

new_string:
Vector sits between the OTel collector tier and Datadog, and additionally
serves as an OPW-compatible logs intake for the Datadog Agent:

```
apps → otel-lb → otel-collector ──OTLP/gRPC──▶ vector ──▶ Datadog SaaS
                                                  ▲
                                          DD Agent (logs)
```

Vector runs as a Deployment (default 3 replicas) fronted by a ClusterIP
Service. It exposes:

- OTLP/gRPC on port 4317 (from OTel collector)
- OTLP/HTTP on port 4318 (from OTel collector)
- Datadog Agent intake on port 8282 (from DD Agent, OPW protocol)

Three signal pipelines run in parallel:

- **logs** — `drop_audit` filter (multi-input: OTLP logs + DD Agent logs;
  drops lines matching audit patterns) → `redact_emails_logs` remap →
  `datadog_logs` sink.
- **traces** — `redact_emails_traces` remap → `datadog_traces` sink.
- **metrics** — `redact_emails_metrics` remap → `datadog_metrics` sink.

The Datadog API key is sourced from Vault via the existing `external-secrets`
operator. The Deployment is annotated with `reloader.stakater.com/auto: "true"`
so key rotation triggers a rolling restart automatically.


content: The file /Users/steve.briggs/projects/IAC/terraform/internal-modules/kubernetes/monitoring/vector/README.md has been updated successfully. (file state is current in your context — no need to Read it back)


text: Now Edit 2 — append a row to the Outputs table.


old_string:
| Name | Description |
|---|---|
| `namespace` | Namespace Vector is deployed into |
| `service_endpoint_grpc` | Cluster-internal OTLP/gRPC endpoint (`http://vector.<namespace>.svc.cluster.local:4317`) |
| `service_endpoint_http` | Cluster-internal OTLP/HTTP endpoint (`http://vector.<namespace>.svc.cluster.local:4318`) |

new_string:
| Name | Description |
|---|---|
| `namespace` | Namespace Vector is deployed into |
| `service_endpoint_grpc` | Cluster-internal OTLP/gRPC endpoint (`http://vector.<namespace>.svc.cluster.local:4317`) |
| `service_endpoint_http` | Cluster-internal OTLP/HTTP endpoint (`http://vector.<namespace>.svc.cluster.local:4318`) |
| `service_endpoint_dd_agent` | OPW-compatible Datadog Agent intake endpoint (`http://vector.<namespace>.svc.cluster.local:8282`) |


text: Now Edit 3 — update the version pin in the Usage example.


text: Now Edit 4 — add a usage hint for the DD Agent wire-up after the existing OTel-collector hint.


old_string:
Point the OTel collector at Vector's gRPC endpoint via the `downstream_endpoint`
variable in the `otel-collector` module (v1.1.0+):

```hcl
downstream_endpoint = module.vector.service_endpoint_grpc
```

---

new_string:
Point the OTel collector at Vector's gRPC endpoint via the `downstream_endpoint`
variable in the `otel-collector` module (v1.1.0+):

```hcl
downstream_endpoint = module.vector.service_endpoint_grpc
```

Point the Datadog Agent at Vector's OPW intake by passing the new endpoint to
the `datadog-agent` module (v1.2.0+):

```hcl
logs_intake_url = module.vector.service_endpoint_dd_agent
```

---


text: Now Edit 5 — add a 5th entry to the VRL test list in the Testing section.


old_string:
```sh
./tests/vrl/drop_audit/run.sh
./tests/vrl/redact_emails_logs/run.sh
./tests/vrl/redact_emails_traces/run.sh
./tests/vrl/redact_emails_metrics/run.sh
```

new_string:
```sh
./tests/vrl/drop_audit/run.sh
./tests/vrl/redact_emails_logs/run.sh
./tests/vrl/redact_emails_traces/run.sh
./tests/vrl/redact_emails_metrics/run.sh
./tests/vrl/redact_emails_dd_agent_logs/run.sh
```


text: Now I'll verify all five edits by re-reading the modified file.


content:
1	# vector
2
3	Terraform module wrapping the `timberio/vector` Helm chart in aggregator role.
4	Receives OTLP from the OTel collector, applies VRL transforms (audit-log
5	filtering, email redaction), and ships to Datadog.
6
7	See `docs/superpowers/specs/2026-05-05-vector-redaction-design.md` in the IAC repo
8	for design context.
9
10	---
11
12	## Architecture
13
14	Vector sits between the OTel collector tier and Datadog, and additionally
15	serves as an OPW-compatible logs intake for the Datadog Agent:
16
17	```
18	apps → otel-lb → otel-collector ──OTLP/gRPC──▶ vector ──▶ Datadog SaaS
19	                                                  ▲
20	                                          DD Agent (logs)
21	```
22
23	Vector runs as a Deployment (default 3 replicas) fronted by a ClusterIP
24	Service. It exposes:
25
26	- OTLP/gRPC on port 4317 (from OTel collector)
27	- OTLP/HTTP on port 4318 (from OTel collector)
28	- Datadog Agent intake on port 8282 (from DD Agent, OPW protocol)
29
30	Three signal pipelines run in parallel:
31
32	- **logs** — `drop_audit` filter (multi-input: OTLP logs + DD Agent logs;
33	  drops lines matching audit patterns) → `redact_emails_logs` remap →
34	  `datadog_logs` sink.
35	- **traces** — `redact_emails_traces` remap → `datadog_traces` sink.
36	- **metrics** — `redact_emails_metrics` remap → `datadog_metrics` sink.
37
38	The Datadog API key is sourced from Vault via the existing `external-secrets`
39	operator. The Deployment is annotated with `reloader.stakater.com/auto: "true"`
40	so key rotation triggers a rolling restart automatically.
41
42	---
43
44	## Inputs
45
46	| Name | Type | Required | Default | Description |
47	|---|---|---|---|---|
48	| `chart_version` | `string` | yes | — | Vector Helm chart version pin |
49	| `image_tag_version` | `string` | yes | — | Vector image tag |
50	| `datadog_site` | `string` | yes | — | Datadog site (e.g. `datadoghq.com`, `us3.datadoghq.com`) |
51	| `datadog_api_key_vault_path` | `string` | yes | — | Vault path for the Datadog API key, consumed by ExternalSecret |
52	| `external_secrets_cluster_store_name` | `string` | yes | — | Name of the ClusterSecretStore created by the external-secrets module |
53	| `image` | `string` | no | `registry.ic-ops.net/cache-docker-hub/timberio/vector` | Vector image source |
54	| `replicas` | `number` | no | `3` | Number of Vector aggregator replicas |
55	| `namespace` | `string` | no | `"vector"` | Kubernetes namespace for Vector |
56	| `audit_drop_patterns` | `list(string)` | no | `["Audit\\s-\\s.+", "\"level\":\"audit\""]` | Regex patterns; matching log message bodies are dropped |
57	| `email_redaction_enabled` | `bool` | no | `true` | Toggle the email-redact VRL transform across all signals |
58	| `email_redaction_replacement` | `string` | no | `"[REDACTED-EMAIL]"` | String that replaces matched email addresses |
59	| `resources_requests` | `map(string)` | no | `{cpu="200m", memory="256Mi"}` | Pod resource requests |
60	| `resources_limits` | `map(string)` | no | `{cpu="2", memory="1Gi"}` | Pod resource limits |
61	| `service_monitor_enabled` | `bool` | no | `false` | Create a ServiceMonitor for Vector self-metrics (requires kube-prometheus-stack CRDs) |
62	| `additional_config` | `any` | no | `null` | Escape hatch — overwrites `customConfig` wholesale |
63
64	---
65
66	## Outputs
67
68	| Name | Description |
69	|---|---|
70	| `namespace` | Namespace Vector is deployed into |
71	| `service_endpoint_grpc` | Cluster-internal OTLP/gRPC endpoint (`http://vector.<namespace>.svc.cluster.local:4317`) |
72	| `service_endpoint_http` | Cluster-internal OTLP/HTTP endpoint (`http://vector.<namespace>.svc.cluster.local:4318`) |
73	| `service_endpoint_dd_agent` | OPW-compatible Datadog Agent intake endpoint (`http://vector.<namespace>.svc.cluster.local:8282`) |
74
75	---
76
77	## Usage
78
79	See [`examples/basic/`](examples/basic/) for a minimal working example.
80
81	```hcl
82	module "vector" {
83	  source = "git@gitlab.infrastructure.immuta.io:infrastructure/terraform/internal-modules/kubernetes/monitoring/vector.git?ref=v1.1.0"
84
85	  providers = {
86	    kubernetes = kubernetes.local-cluster
87	    helm       = helm.local-cluster
88	  }
89
90	  chart_version                        = "0.36.0"
91	  image_tag_version                    = "0.41.0-debian"
92	  datadog_site                         = "datadoghq.com"
93	  datadog_api_key_vault_path           = "secret/data/platform/datadog/api_key"
94	  external_secrets_cluster_store_name  = "vault-cluster-store"
95	}
96	```
97
98	Point the OTel collector at Vector's gRPC endpoint via the `downstream_endpoint`
99	variable in the `otel-collector` module (v1.1.0+):
100
101	```hcl
102	downstream_endpoint = module.vector.service_endpoint_grpc
103	```
104
105	Point the Datadog Agent at Vector's OPW intake by passing the new endpoint to
106	the `datadog-agent` module (v1.2.0+):
107
108	```hcl
109	logs_intake_url = module.vector.service_endpoint_dd_agent
110	```
111
112	---
113
114	## Testing
115
116	### VRL unit tests
117
118	Each transform has an input/expected fixture pair and a shell runner under
119	`tests/vrl/<transform-name>/`. Run all four locally (requires `vector` and `jq`
120	on PATH):
121
122	```sh
123	./tests/vrl/drop_audit/run.sh
124	./tests/vrl/redact_emails_logs/run.sh
125	./tests/vrl/redact_emails_traces/run.sh
126	./tests/vrl/redact_emails_metrics/run.sh
127	./tests/vrl/redact_emails_dd_agent_logs/run.sh
128	```
129
130	### Terraform
131
132	```sh
133	# format check
134	terraform fmt -check -recursive
135
136	# validate the example
137	cd examples/basic
138	terraform init -backend=false
139	terraform validate
140	```
141
142	CI runs all three checks automatically on every push (see `.gitlab-ci.yml`).
143


text:
Perfect! All five edits have been successfully applied. Let me confirm each one:

---

## Status: DONE

All five edits successfully completed and verified:

**Edit 1 — Architecture section (lines 14-40):** ✓
- Replaced the opening statement to mention OPW-compatible logs intake for DD Agent
- Updated the diagram to show the DD Agent arrow pointing to Vector
- Expanded the "It exposes:" section to list all three ports (4317, 4318, 8282)
- Updated the logs pipeline description to note "multi-input: OTLP logs + DD Agent logs"

**Edit 2 — Outputs table (line 73):** ✓
- Added new row: `| service_endpoint_dd_agent | OPW-compatible Datadog Agent intake endpoint (http://vector.<namespace>.svc.cluster.local:8282) |`

**Edit 3 — Version pin in Usage example (line 83):** ✓
- Changed `ref=v1.0.0` to `ref=v1.1.0`

**Edit 4 — DD Agent usage hint (lines 105-110):** ✓
- Inserted new block after OTel collector hint documenting how to wire up the DD Agent via `logs_intake_url = module.vector.service_endpoint_dd_agent`

**Edit 5 — Testing section (line 127):** ✓
- Added 5th test: `./tests/vrl/redact_emails_dd_agent_logs/run.sh` to the VRL unit tests list

All edits are syntactically correct and logically consistent with the requested changes. No concerns.