<memvid-mind-context>
# 🧠 Claude Mind Active

📁 Project: **churn-renewal-forecast**
💾 Memory: `.claude/mind.mv2` (51523 KB)

**Commands:**
- `/mind:search <query>` - Search memories
- `/mind:ask <question>` - Ask your memory
- `/mind:recent` - View timeline
- `/mind:stats` - View statistics

_Memories are captured automatically from your tool use._
</memvid-mind-context>


<EXTREMELY_IMPORTANT>
You have superpowers.

**Below is the full content of your 'superpowers:using-superpowers' skill - your introduction to using skills. For all other skills, use the 'Skill' tool:**

---
name: using-superpowers
description: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
---

<SUBAGENT-STOP>
If you were dispatched as a subagent to execute a specific task, skip this skill.
</SUBAGENT-STOP>

<EXTREMELY-IMPORTANT>
If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.

IF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.

This is not negotiable. This is not optional. You cannot rationalize your way out of this.
</EXTREMELY-IMPORTANT>

## Instruction Priority

Superpowers skills override default system prompt behavior, but **user instructions always take precedence**:

1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority
2. **Superpowers skills** — override default system behavior where they conflict
3. **Default system prompt** — lowest priority

If CLAUDE.md, GEMINI.md, or AGENTS.md says "don't use TDD" and a skill says "always use TDD," follow the user's instructions. The user is in control.

## How to Access Skills

**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.

**In Copilot CLI:** Use the `skill` tool. Skills are auto-discovered from installed plugins. The `skill` tool works the same as Claude Code's `Skill` tool.

**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.

**In other environments:** Check your platform's documentation for how skills are loaded.

## Platform Adaptation

Skills use Claude Code tool names. Non-CC platforms: see `references/copilot-tools.md` (Copilot CLI), `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.

# Using Skills

## The Rule

**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.

```dot
digraph skill_flow {
    "User message received" [shape=doublecircle];
    "About to EnterPlanMode?" [shape=doublecircle];
    "Already brainstormed?" [shape=diamond];
    "Invoke brainstorming skill" [shape=box];
    "Might any skill apply?" [shape=diamond];
    "Invoke Skill tool" [shape=box];
    "Announce: 'Using [skill] to [purpose]'" [shape=box];
    "Has checklist?" [shape=diamond];
    "Create TodoWrite todo per item" [shape=box];
    "Follow skill exactly" [shape=box];
    "Respond (including clarifications)" [shape=doublecircle];

    "About to EnterPlanMode?" -> "Already brainstormed?";
    "Already brainstormed?" -> "Invoke brainstorming skill" [label="no"];
    "Already brainstormed?" -> "Might any skill apply?" [label="yes"];
    "Invoke brainstorming skill" -> "Might any skill apply?";

    "User message received" -> "Might any skill apply?";
    "Might any skill apply?" -> "Invoke Skill tool" [label="yes, even 1%"];
    "Might any skill apply?" -> "Respond (including clarifications)" [label="definitely not"];
    "Invoke Skill tool" -> "Announce: 'Using [skill] to [purpose]'";
    "Announce: 'Using [skill] to [purpose]'" -> "Has checklist?";
    "Has checklist?" -> "Create TodoWrite todo per item" [label="yes"];
    "Has checklist?" -> "Follow skill exactly" [label="no"];
    "Create TodoWrite todo per item" -> "Follow skill exactly";
}
```

## Red Flags

These thoughts mean STOP—you're rationalizing:

| Thought | Reality |
|---------|---------|
| "This is just a simple question" | Questions are tasks. Check for skills. |
| "I need more context first" | Skill check comes BEFORE clarifying questions. |
| "Let me explore the codebase first" | Skills tell you HOW to explore. Check first. |
| "I can check git/files quickly" | Files lack conversation context. Check for skills. |
| "Let me gather information first" | Skills tell you HOW to gather information. |
| "This doesn't need a formal skill" | If a skill exists, use it. |
| "I remember this skill" | Skills evolve. Read current version. |
| "This doesn't count as a task" | Action = task. Check for skills. |
| "The skill is overkill" | Simple things become complex. Use it. |
| "I'll just do this one thing first" | Check BEFORE doing anything. |
| "This feels productive" | Undisciplined action wastes time. Skills prevent this. |
| "I know what that means" | Knowing the concept ≠ using the skill. Invoke it. |

## Skill Priority

When multiple skills could apply, use this order:

1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task
2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution

"Let's build X" → brainstorming first, then implementation skills.
"Fix this bug" → debugging first, then domain-specific skills.

## Skill Types

**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.

**Flexible** (patterns): Adapt principles to context.

The skill itself tells you which.

## User Instructions

Instructions say WHAT, not HOW. "Add X" or "Fix Y" doesn't mean skip workflows.

</EXTREMELY_IMPORTANT>


<memvid-mind-context>
# 🧠 Claude Mind Active

📁 Project: **churn-renewal-forecast**
💾 Memory: `.claude/mind.mv2` (51523 KB)

**Commands:**
- `/mind:search <query>` - Search memories
- `/mind:ask <question>` - Ask your memory
- `/mind:recent` - View timeline
- `/mind:stats` - View statistics

_Memories are captured automatically from your tool use._
</memvid-mind-context>

<EXTREMELY_IMPORTANT>
You have superpowers.

**Below is the full content of your 'superpowers:using-superpowers' skill - your introduction to using skills. For all other skills, use the 'Skill' tool:**

---
name: using-superpowers
description: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
---

<SUBAGENT-STOP>
If you were dispatched as a subagent to execute a specific task, skip this skill.
</SUBAGENT-STOP>

<EXTREMELY-IMPORTANT>
If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.

IF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.

This is not negotiable. This is not optional. You cannot rationalize your way out of this.
</EXTREMELY-IMPORTANT>

## Instruction Priority

Superpowers skills override default system prompt behavior, but **user instructions always take precedence**:

1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority
2. **Superpowers skills** — override default system behavior where they conflict
3. **Default system prompt** — lowest priority

If CLAUDE.md, GEMINI.md, or AGENTS.md says "don't use TDD" and a skill says "always use TDD," follow the user's instructions. The user is in control.

## How to Access Skills

**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.

**In Copilot CLI:** Use the `skill` tool. Skills are auto-discovered from installed plugins. The `skill` tool works the same as Claude Code's `Skill` tool.

**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.

**In other environments:** Check your platform's documentation for how skills are loaded.

## Platform Adaptation

Skills use Claude Code tool names. Non-CC platforms: see `references/copilot-tools.md` (Copilot CLI), `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.

# Using Skills

## The Rule

**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.

```dot
digraph skill_flow {
    "User message received" [shape=doublecircle];
    "About to EnterPlanMode?" [shape=doublecircle];
    "Already brainstormed?" [shape=diamond];
    "Invoke brainstorming skill" [shape=box];
    "Might any skill apply?" [shape=diamond];
    "Invoke Skill tool" [shape=box];
    "Announce: 'Using [skill] to [purpose]'" [shape=box];
    "Has checklist?" [shape=diamond];
    "Create TodoWrite todo per item" [shape=box];
    "Follow skill exactly" [shape=box];
    "Respond (including clarifications)" [shape=doublecircle];

    "About to EnterPlanMode?" -> "Already brainstormed?";
    "Already brainstormed?" -> "Invoke brainstorming skill" [label="no"];
    "Already brainstormed?" -> "Might any skill apply?" [label="yes"];
    "Invoke brainstorming skill" -> "Might any skill apply?";

    "User message received" -> "Might any skill apply?";
    "Might any skill apply?" -> "Invoke Skill tool" [label="yes, even 1%"];
    "Might any skill apply?" -> "Respond (including clarifications)" [label="definitely not"];
    "Invoke Skill tool" -> "Announce: 'Using [skill] to [purpose]'";
    "Announce: 'Using [skill] to [purpose]'" -> "Has checklist?";
    "Has checklist?" -> "Create TodoWrite todo per item" [label="yes"];
    "Has checklist?" -> "Follow skill exactly" [label="no"];
    "Create TodoWrite todo per item" -> "Follow skill exactly";
}
```

## Red Flags

These thoughts mean STOP—you're rationalizing:

| Thought | Reality |
|---------|---------|
| "This is just a simple question" | Questions are tasks. Check for skills. |
| "I need more context first" | Skill check comes BEFORE clarifying questions. |
| "Let me explore the codebase first" | Skills tell you HOW to explore. Check first. |
| "I can check git/files quickly" | Files lack conversation context. Check for skills. |
| "Let me gather information first" | Skills tell you HOW to gather information. |
| "This doesn't need a formal skill" | If a skill exists, use it. |
| "I remember this skill" | Skills evolve. Read current version. |
| "This doesn't count as a task" | Action = task. Check for skills. |
| "The skill is overkill" | Simple things become complex. Use it. |
| "I'll just do this one thing first" | Check BEFORE doing anything. |
| "This feels productive" | Undisciplined action wastes time. Skills prevent this. |
| "I know what that means" | Knowing the concept ≠ using the skill. Invoke it. |

## Skill Priority

When multiple skills could apply, use this order:

1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task
2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution

"Let's build X" → brainstorming first, then implementation skills.
"Fix this bug" → debugging first, then domain-specific skills.

## Skill Types

**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.

**Flexible** (patterns): Adapt principles to context.

The skill itself tells you which.

## User Instructions

Instructions say WHAT, not HOW. "Add X" or "Fix Y" doesn't mean skip workflows.

</EXTREMELY_IMPORTANT>


i am having issues with the cd portion (.github/workflows/cd.yml) for some reason it seems that the OCI artifact isn't actually updated so flux does not see any changes to deploy?


claude claude-code-guide Explore general-purpose Plan statusline-setup

- claude: Catch-all for any task that doesn't fit a more specific agent. FleetView's default when no agent name is typed. (Tools: *)

- claude-code-guide: Use this agent when the user asks questions ("Can Claude...", "Does Claude...", "How do I...") about: (1) Claude Code (the CLI tool) - features, hooks, slash commands, MCP servers, settings, IDE integrations, keyboard shortcuts; (2) Claude Agent SDK - building custom agents; (3) Claude API (formerly Anthropic API) - API usage, tool use, Anthropic SDK usage. **IMPORTANT:** Before spawning a new agent, check if there is already a running or recently completed claude-code-guide agent that you can continue via SendMessage. (Tools: Bash, Read, WebFetch, WebSearch)

- Explore: Fast read-only search agent for locating code. Use it to find files by pattern (eg. "src/components/**/*.tsx"), grep for symbols or keywords (eg. "API endpoints"), or answer "where is X defined / which files reference Y." Do NOT use it for code review, design-doc auditing, cross-file consistency checks, or open-ended analysis — it reads excerpts rather than whole files and will miss content past its read window. When calling, specify search breadth: "quick" for a single targeted lookup, "medium" for moderate exploration, or "very thorough" to search across multiple locations and naming conventions. (Tools: All tools except Agent, ExitPlanMode, Edit, Write, NotebookEdit)

- general-purpose: General-purpose agent for researching complex questions, searching for code, and executing multi-step tasks. When you are searching for a keyword or file and are not confident that you will find the right match in the first few tries use this agent to perform the search for you. (Tools: *)

- Plan: Software architect agent for designing implementation plans. Use this when you need to plan the implementation strategy for a task. Returns step-by-step plans, identifies critical files, and considers architectural trade-offs. (Tools: All tools except Agent, ExitPlanMode, Edit, Write, NotebookEdit)

- statusline-setup: Use this agent to configure the user's Claude Code status line setting. (Tools: Read, Edit)


- obsidian-load: Load context from Obsidian notes in the `Claude/` directory into the current conversation.
- obsidian-save: Save current conversation context to an Obsidian note under the `Claude/` directory.
- mind:stats: Show memory statistics and storage information
- mind:search: Search memories for specific content or patterns
- mind:recent: Show recent memories and activity timeline
- mind:ask: Ask questions about memories and get context-aware answers
- mind:mind: Claude Mind - Search and manage Claude's persistent memory stored in a single portable .mv2 file
- mind:memory: Claude Mind - Search and manage Claude's persistent memory stored in a single portable .mv2 file
- superpowers:test-driven-development: Use when implementing any feature or bugfix, before writing implementation code
- superpowers:writing-plans: Use when you have a spec or requirements for a multi-step task, before touching code
- superpowers:finishing-a-development-branch: Use when implementation is complete, all tests pass, and you need to decide how to integrate the work - guides completion of development work by presenting structured options for merge, PR, or cleanup
- superpowers:using-git-worktrees: Use when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback
- superpowers:dispatching-parallel-agents: Use when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies
- superpowers:receiving-code-review: Use when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation
- superpowers:using-superpowers: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
- superpowers:requesting-code-review: Use when completing tasks, implementing major features, or before merging to verify work meets requirements
- superpowers:systematic-debugging: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
- superpowers:executing-plans: Use when you have a written implementation plan to execute in a separate session with review checkpoints
- superpowers:brainstorming: You MUST use this before any creative work - creating features, building components, adding functionality, or modifying behavior. Explores user intent, requirements and design before implementation.
- superpowers:writing-skills: Use when creating new skills, editing existing skills, or verifying skills work before deployment
- superpowers:verification-before-completion: Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always
- superpowers:subagent-driven-development: Use when executing implementation plans with independent tasks in the current session
- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
- code-review: Review changed code for reuse, quality, and efficiency, then fix any issues found.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled remote agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring remote agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Build, debug, and optimize Claude API / Anthropic SDK apps. Apps built with this skill should include prompt caching. Also handles migrating existing Claude API code between Claude model versions (4.5 → 4.6, 4.6 → 4.7, retired-model replacements).
TRIGGER when: code imports `anthropic`/`@anthropic-ai/sdk`; user asks for the Claude API, Anthropic SDK, or Managed Agents; user adds/modifies/tunes a Claude feature (caching, thinking, compaction, tool use, batch, files, citations, memory) or model (Opus/Sonnet/Haiku) in a file; questions about prompt caching / cache hit rate in an Anthropic SDK project.
SKIP: file imports `openai`/other-provider SDK, filename like `*-openai.py`/`*-generic.py`, provider-neutral code, general programming/ML.
- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch


Let me check the CD workflow file to understand what's happening.


1	name: CD
2
3	on:
4	  workflow_dispatch:
5	    inputs:
6	      reason:
7	        description: "Reason for running manually"
8	        required: false
9	        default: "Testing"
10	  push:
11	    branches:
12	      - main
13	    paths-ignore:
14	      - "**/*.md"
15	      - "docs/**"
16	      - "LICENSE"
17	      - ".gitignore"
18	      - ".gitattributes"
19	      - ".github/CODEOWNERS"
20
21	env:
22	  REGISTRY: registry.ic-ops.net
23
24	jobs:
25	  build-and-push:
26	    name: Build and Push Docker Image
27	    runs-on: self-hosted-rocky-x64-16g
28	    outputs:
29	      rev_tag: ${{ steps.tag.outputs.rev_tag }}
30	    steps:
31	      - name: Checkout repo
32	        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
33
34	      - name: Generate image tag
35	        id: tag
36	        run: |
37	          REV_TAG="git-$(git rev-parse --short=8 HEAD)"
38	          echo "rev_tag=$REV_TAG" >> "$GITHUB_OUTPUT"
39
40	      - name: Docker login
41	        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v4.0.0
42	        with:
43	          registry: ${{ env.REGISTRY }}
44	          username: ${{ vars.REGISTRY_USERNAME }}
45	          password: ${{ secrets.REGISTRY_PASSWORD }}
46
47	      - name: Set up BuildKit fleet
48	        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
49	        with:
50	          name: eks-fleet
51	          driver: remote
52	          endpoint: tcp://buildkit-amd64-svc.buildkit-fleet.svc.cluster.local:4400
53	          platforms: linux/amd64
54	          append: |
55	            - endpoint: tcp://buildkit-arm64-svc.buildkit-fleet.svc.cluster.local:4400
56	              platforms: linux/arm64
57
58	      - name: Build and push
59	        uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7
60	        with:
61	          builder: eks-fleet
62	          push: true
63	          platforms: linux/amd64,linux/arm64
64	          tags: |
65	            ${{ env.REGISTRY }}/internal-tooling/churn-renewal-forecast:${{ steps.tag.outputs.rev_tag }}
66	            ${{ env.REGISTRY }}/internal-tooling/churn-renewal-forecast:latest
67	          cache-from: type=registry,ref=${{ env.REGISTRY }}/image-build-cache/internal-tooling/churn-renewal-forecast:cache
68	          cache-to: type=registry,ref=${{ env.REGISTRY }}/image-build-cache/internal-tooling/churn-renewal-forecast:cache,mode=max,image-manifest=true,ignore-error=true
69
70	      - name: Build overlays
71	        env:
72	          DEPLOY_DIR: deploy/kubernetes/deploy
73	          BUILD_VARS: ''
74	        run: |
75	          SET_ARGS=()
76	          if [ -n "${BUILD_VARS}" ]; then
77	            IFS=',' read -ra PAIRS <<< "${BUILD_VARS}"
78	            for pair in "${PAIRS[@]}"; do
79	              SET_ARGS+=( --set "${pair}" )
80	            done
81	          fi
82	          pack build \
83	            --directory "${DEPLOY_DIR}" \
84	            --recursive \
85	            --debug \
86	            "${SET_ARGS[@]}"
87
88	      - name: Push overlays
89	        env:
90	          DEPLOY_DIR: deploy/kubernetes/deploy
91	          TAG: ${{ steps.tag.outputs.rev_tag }}
92	          OCI_USERNAME: ${{ vars.REGISTRY_USERNAME }}
93	          OCI_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
94	        run: |
95	          pack push \
96	            --directory "${DEPLOY_DIR}" \
97	            --tag "${TAG}" \
98	            --recursive \
99	            --debug
100
101
102
103

name: CD

on:
  workflow_dispatch:
    inputs:
      reason:
        description: "Reason for running manually"
        required: false
        default: "Testing"
  push:
    branches:
      - main
    paths-ignore:
      - "**/*.md"
      - "docs/**"
      - "LICENSE"
      - ".gitignore"
      - ".gitattributes"
      - ".github/CODEOWNERS"

env:
  REGISTRY: registry.ic-ops.net

jobs:
  build-and-push:
    name: Build and Push Docker Image
    runs-on: self-hosted-rocky-x64-16g
    outputs:
      rev_tag: ${{ steps.tag.outputs.rev_tag }}
    steps:
      - name: Checkout repo
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

      - name: Generate image tag
        id: tag
        run: |
          REV_TAG="git-$(git rev-parse --short=8 HEAD)"
          echo "rev_tag=$REV_TAG" >> "$GITHUB_OUTPUT"

      - name: Docker login
        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v4.0.0
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ vars.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}

      - name: Set up BuildKit fleet
        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
        with:
          name: eks-fleet
          driver: remote
          endpoint: tcp://buildkit-amd64-svc.buildkit-fleet.svc.cluster.local:4400
          platforms: linux/amd64
          append: |
            - endpoint: tcp://buildkit-arm64-svc.buildkit-fleet.svc.cluster.local:4400
              platforms: linux/arm64

      - name: Build and push
        uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7
        with:
          builder: eks-fleet
          push: true
          platforms: linux/amd64,linux/arm64
          tags: |
            ${{ env.REGISTRY }}/internal-tooling/churn-renewal-forecast:${{ steps.tag.outputs.rev_tag }}
            ${{ env.REGISTRY }}/internal-tooling/churn-renewal-forecast:latest
          cache-from: type=registry,ref=${{ env.REGISTRY }}/image-build-cache/internal-tooling/churn-renewal-forecast:cache
          cache-to: type=registry,ref=${{ env.REGISTRY }}/image-build-cache/internal-tooling/churn-renewal-forecast:cache,mode=max,image-manifest=true,ignore-error=true

      - name: Build overlays
        env:
          DEPLOY_DIR: deploy/kubernetes/deploy
          BUILD_VARS: ''
        run: |
          SET_ARGS=()
          if [ -n "${BUILD_VARS}" ]; then
            IFS=',' read -ra PAIRS <<< "${BUILD_VARS}"
            for pair in "${PAIRS[@]}"; do
              SET_ARGS+=( --set "${pair}" )
            done
          fi
          pack build \
            --directory "${DEPLOY_DIR}" \
            --recursive \
            --debug \
            "${SET_ARGS[@]}"

      - name: Push overlays
        env:
          DEPLOY_DIR: deploy/kubernetes/deploy
          TAG: ${{ steps.tag.outputs.rev_tag }}
          OCI_USERNAME: ${{ vars.REGISTRY_USERNAME }}
          OCI_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
        run: |
          pack push \
            --directory "${DEPLOY_DIR}" \
            --tag "${TAG}" \
            --recursive \
            --debug


Let me check the deploy directory structure and the kustomization files.


Bash

find /Users/steve.briggs/projects/churn-renewal-forecast/deploy -type f | sort

List all deploy directory files


1	# Kustomize Deployment Implementation Plan
2
3	> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5	**Goal:** Create a complete Kustomize deployment under `deploy/kubernetes/` with a base layer and a single cluster overlay for `eks-shared-infra-use1-2`.
6
7	**Architecture:** Base layer is split into four sub-directories (bootstrap, init, netpol, services) each with their own `kustomization.yaml`. The cluster overlay at `deploy/kubernetes/deploy/eks-shared-infra-use1-2/` references the base and applies cluster-specific patches (ingress visibility, service account). All resources share a single container image with command overrides per workload.
8
9	**Tech Stack:** Kustomize v4+, Kubernetes manifests (apps/v1, batch/v1, networking.k8s.io/v1), external-secrets.io v1beta1 (ExternalSecret), cilium.io/v2 (CiliumNetworkPolicy)
10
11	---
12
13	## File Map
14
15	**Create — base bootstrap:**
16	- `deploy/kubernetes/base/bootstrap/namespace.yaml`
17	- `deploy/kubernetes/base/bootstrap/serviceaccount.yaml`
18	- `deploy/kubernetes/base/bootstrap/secret.yaml`
19	- `deploy/kubernetes/base/bootstrap/app-config.env`
20	- `deploy/kubernetes/base/bootstrap/kustomization.yaml`
21
22	**Create — base init:**
23	- `deploy/kubernetes/base/init/job.yaml`
24	- `deploy/kubernetes/base/init/kustomization.yaml`
25
26	**Create — base netpol:**
27	- `deploy/kubernetes/base/netpol/allow-dns.yaml`
28	- `deploy/kubernetes/base/netpol/allow-rds.yaml`
29	- `deploy/kubernetes/base/netpol/allow-external.yaml`
30	- `deploy/kubernetes/base/netpol/kustomization.yaml`
31
32	**Create — base services:**
33	- `deploy/kubernetes/base/services/web-deployment.yaml`
34	- `deploy/kubernetes/base/services/worker-deployment.yaml`
35	- `deploy/kubernetes/base/services/service.yaml`
36	- `deploy/kubernetes/base/services/ingress.yaml`
37	- `deploy/kubernetes/base/services/kustomization.yaml`
38
39	**Create — base root:**
40	- `deploy/kubernetes/base/kustomization.yaml`
41
42	**Create — overlay:**
43	- `deploy/kubernetes/deploy/eks-shared-infra-use1-2/kustomization.yaml`
44	- `deploy/kubernetes/deploy/eks-shared-infra-use1-2/.app.yaml`
45	- `deploy/kubernetes/deploy/eks-shared-infra-use1-2/ingress-patch.yaml`
46	- `deploy/kubernetes/deploy/eks-shared-infra-use1-2/serviceaccount.yaml`
47
48	---
49
50	## Task 1: Bootstrap Resources
51
52	Creates the namespace, service account, ExternalSecret, and ConfigMap source.
53
54	**Files:**
55	- Create: `deploy/kubernetes/base/bootstrap/namespace.yaml`
56	- Create: `deploy/kubernetes/base/bootstrap/serviceaccount.yaml`
57	- Create: `deploy/kubernetes/base/bootstrap/secret.yaml`
58	- Create: `deploy/kubernetes/base/bootstrap/app-config.env`
59	- Create: `deploy/kubernetes/base/bootstrap/kustomization.yaml`
60
61	- [ ] **Step 1: Create directory structure**
62
63	```bash
64	mkdir -p deploy/kubernetes/base/bootstrap
65	```
66
67	- [ ] **Step 2: Create namespace.yaml**
68
69	```yaml
70	# deploy/kubernetes/base/bootstrap/namespace.yaml
71	apiVersion: v1
72	kind: Namespace
73	metadata:
74	  name: churn-renewal-forecast
75	  labels:
76	    app.kubernetes.io/name: churn-renewal-forecast
77	```
78
79	- [ ] **Step 3: Create serviceaccount.yaml**
80
81	```yaml
82	# deploy/kubernetes/base/bootstrap/serviceaccount.yaml
83	apiVersion: v1
84	kind: ServiceAccount
85	metadata:
86	  name: churn-renewal-forecast
87	  namespace: churn-renewal-forecast
88	  labels:
89	    app.kubernetes.io/name: churn-renewal-forecast
90	```
91
92	- [ ] **Step 4: Create secret.yaml (ExternalSecret)**
93
94	```yaml
95	# deploy/kubernetes/base/bootstrap/secret.yaml
96	apiVersion: external-secrets.io/v1beta1
97	kind: ExternalSecret
98	metadata:
99	  name: churn-renewal-forecast
100	  namespace: churn-renewal-forecast
101	  labels:
102	    app.kubernetes.io/name: churn-renewal-forecast
103	spec:
104	  refreshInterval: 1h
105	  secretStoreRef:
106	    name: vault
107	    kind: ClusterSecretStore
108	  target:
109	    name: churn-renewal-forecast
110	    creationPolicy: Owner
111	  data:
112	    - secretKey: DATABASE_URL
113	      remoteRef:
114	        key: /internal/churn-renewal-forecast
115	        property: DATABASE_URL
116	    - secretKey: CLERK_SECRET_KEY
117	      remoteRef:
118	        key: /internal/churn-renewal-forecast
119	        property: CLERK_SECRET_KEY
120	    - secretKey: FRONT_API_TOKEN
121	      remoteRef:
122	        key: /internal/churn-renewal-forecast
123	        property: FRONT_API_TOKEN
124	    - secretKey: FRONT_API_TOKEN_2
125	      remoteRef:
126	        key: /internal/churn-renewal-forecast
127	        property: FRONT_API_TOKEN_2
128	    - secretKey: PRODUCTBOARD_API_KEY
129	      remoteRef:
130	        key: /internal/churn-renewal-forecast
131	        property: PRODUCTBOARD_API_KEY
132	    - secretKey: SLACK_BOT_TOKEN
133	      remoteRef:
134	        key: /internal/churn-renewal-forecast
135	        property: SLACK_BOT_TOKEN
136	```
137
138	- [ ] **Step 5: Create app-config.env**
139
140	```env
141	# deploy/kubernetes/base/bootstrap/app-config.env
142	APP_URL=https://churn-forecast.infrastructure.immuta.io
143	NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=PLACEHOLDER
144	NEXT_PUBLIC_PRODUCTBOARD_WORKSPACE=PLACEHOLDER
145	SLACK_TEST_CHANNEL=
146	PG_POOL_MAX=10
147	PG_POOL_IDLE_TIMEOUT_MS=30000
148	PG_POOL_CONNECTION_TIMEOUT_MS=2000
149	NEXT_PUBLIC_CLERK_SIGN_IN_URL=/sign-in
150	NEXT_PUBLIC_CLERK_SIGN_UP_URL=/sign-up
151	NEXT_PUBLIC_CLERK_AFTER_SIGN_IN_URL=/
152	NEXT_PUBLIC_CLERK_AFTER_SIGN_UP_URL=/
153	```
154
155	- [ ] **Step 6: Create bootstrap/kustomization.yaml**
156
157	`configMapGenerator` produces the ConfigMap from `app-config.env`. `disableNameSuffixHash: true` keeps the name stable so `envFrom` references don't need to change on every deploy.
158
159	```yaml
160	# deploy/kubernetes/base/bootstrap/kustomization.yaml
161	apiVersion: kustomize.config.k8s.io/v1beta1
162	kind: Kustomization
163
164	resources:
165	  - namespace.yaml
166	  - serviceaccount.yaml
167	  - secret.yaml
168
169	configMapGenerator:
170	  - name: churn-renewal-forecast-config
171	    envs:
172	      - app-config.env
173
174	generatorOptions:
175	  disableNameSuffixHash: true
176	```
177
178	- [ ] **Step 7: Validate bootstrap kustomization builds**
179
180	```bash
181	kustomize build deploy/kubernetes/base/bootstrap
182	```
183
184	Expected: YAML output containing Namespace, ServiceAccount, ExternalSecret, and ConfigMap. No errors.
185
186	- [ ] **Step 8: Commit**
187
188	```bash
189	git add deploy/kubernetes/base/bootstrap/
190	git commit -m "feat: add k8s bootstrap resources (namespace, sa, secret, configmap)"
191	```
192
193	---
194
195	## Task 2: Init Job
196
197	Creates the one-shot Prisma migration Job that runs before the app starts.
198
199	**Files:**
200	- Create: `deploy/kubernetes/base/init/job.yaml`
201	- Create: `deploy/kubernetes/base/init/kustomization.yaml`
202
203	- [ ] **Step 1: Create directory**
204
205	```bash
206	mkdir -p deploy/kubernetes/base/init
207	```
208
209	- [ ] **Step 2: Create job.yaml**
210
211	`ttlSecondsAfterFinished: 300` auto-deletes the Job pod 5 minutes after it completes. `backoffLimit: 3` retries up to 3 times on failure. `prisma migrate deploy` is idempotent — safe to re-run.
212
213	```yaml
214	# deploy/kubernetes/base/init/job.yaml
215	apiVersion: batch/v1
216	kind: Job
217	metadata:
218	  name: churn-renewal-forecast-migrate
219	  namespace: churn-renewal-forecast
220	  labels:
221	    app.kubernetes.io/name: churn-renewal-forecast
222	    app.kubernetes.io/component: migrate
223	spec:
224	  ttlSecondsAfterFinished: 300
225	  backoffLimit: 3
226	  template:
227	    metadata:
228	      labels:
229	        app.kubernetes.io/name: churn-renewal-forecast
230	        app.kubernetes.io/component: migrate
231	    spec:
232	      serviceAccountName: churn-renewal-forecast
233	      restartPolicy: OnFailure
234	      containers:
235	        - name: migrate
236	          image: registry.ic-ops.net/churn-renewal-forecast:latest
237	          command: ["bunx", "prisma", "migrate", "deploy"]
238	          envFrom:
239	            - configMapRef:
240	                name: churn-renewal-forecast-config
241	            - secretRef:
242	                name: churn-renewal-forecast
243	```
244
245	- [ ] **Step 3: Create init/kustomization.yaml**
246
247	```yaml
248	# deploy/kubernetes/base/init/kustomization.yaml
249	apiVersion: kustomize.config.k8s.io/v1beta1
250	kind: Kustomization
251
252	resources:
253	  - job.yaml
254	```
255
256	- [ ] **Step 4: Validate**
257
258	```bash
259	kustomize build deploy/kubernetes/base/init
260	```
261
262	Expected: YAML output containing a single Job. No errors.
263
264	- [ ] **Step 5: Commit**
265
266	```bash
267	git add deploy/kubernetes/base/init/
268	git commit -m "feat: add k8s init job for prisma migrations"
269	```
270
271	---
272
273	## Task 3: Network Policies
274
275	Creates three CiliumNetworkPolicy resources covering DNS, RDS, and external HTTPS egress.
276
277	**Files:**
278	- Create: `deploy/kubernetes/base/netpol/allow-dns.yaml`
279	- Create: `deploy/kubernetes/base/netpol/allow-rds.yaml`
280	- Create: `deploy/kubernetes/base/netpol/allow-external.yaml`
281	- Create: `deploy/kubernetes/base/netpol/kustomization.yaml`
282
283	- [ ] **Step 1: Create directory**
284
285	```bash
286	mkdir -p deploy/kubernetes/base/netpol
287	```
288
289	- [ ] **Step 2: Create allow-dns.yaml**
290
291	Allows pods with `app.kubernetes.io/name: churn-renewal-forecast` to reach `kube-dns` on UDP 53. The `rules.dns` section tells Cilium to track DNS responses so FQDN policies in `allow-external.yaml` can resolve correctly.
292
293	```yaml
294	# deploy/kubernetes/base/netpol/allow-dns.yaml
295	apiVersion: cilium.io/v2
296	kind: CiliumNetworkPolicy
297	metadata:
298	  name: churn-renewal-forecast-allow-dns
299	  namespace: churn-renewal-forecast
300	  labels:
301	    app.kubernetes.io/name: churn-renewal-forecast
302	spec:
303	  endpointSelector:
304	    matchLabels:
305	      app.kubernetes.io/name: churn-renewal-forecast
306	  egress:
307	    - toEndpoints:
308	        - matchLabels:
309	            k8s:io.kubernetes.pod.namespace: kube-system
310	            k8s-app: kube-dns
311	      toPorts:
312	        - ports:
313	            - port: "53"
314	              protocol: UDP
315	          rules:
316	            dns:
317	              - matchPattern: "*"
318	```
319
320	- [ ] **Step 3: Create allow-rds.yaml**
321
322	Allows outbound TCP 5432 to any IP. Scoped to CIDR 0.0.0.0/0 because RDS runs outside the cluster — you cannot use an endpoint selector for external IPs.
323
324	```yaml
325	# deploy/kubernetes/base/netpol/allow-rds.yaml
326	apiVersion: cilium.io/v2
327	kind: CiliumNetworkPolicy
328	metadata:
329	  name: churn-renewal-forecast-allow-rds
330	  namespace: churn-renewal-forecast
331	  labels:
332	    app.kubernetes.io/name: churn-renewal-forecast
333	spec:
334	  endpointSelector:
335	    matchLabels:
336	      app.kubernetes.io/name: churn-renewal-forecast
337	  egress:
338	    - toCIDR:
339	        - 0.0.0.0/0
340	      toPorts:
341	        - ports:
342	            - port: "5432"
343	              protocol: TCP
344	```
345
346	- [ ] **Step 4: Create allow-external.yaml**
347
348	`toFQDNs` with `matchName` locks to the exact hostname. `matchPattern: "*.clerk.com"` covers Clerk's subdomains (e.g. `accounts.clerk.com`). `*.slack.com` covers Slack API endpoints. Requires `allow-dns.yaml` to be active so Cilium can resolve these FQDNs.
349
350	```yaml
351	# deploy/kubernetes/base/netpol/allow-external.yaml
352	apiVersion: cilium.io/v2
353	kind: CiliumNetworkPolicy
354	metadata:
355	  name: churn-renewal-forecast-allow-external
356	  namespace: churn-renewal-forecast
357	  labels:
358	    app.kubernetes.io/name: churn-renewal-forecast
359	spec:
360	  endpointSelector:
361	    matchLabels:
362	      app.kubernetes.io/name: churn-renewal-forecast
363	  egress:
364	    - toFQDNs:
365	        - matchName: "clerk.com"
366	        - matchPattern: "*.clerk.com"
367	        - matchName: "api.frontapp.com"
368	        - matchName: "api.productboard.com"
369	        - matchName: "slack.com"
370	        - matchPattern: "*.slack.com"
371	      toPorts:
372	        - ports:
373	            - port: "443"
374	              protocol: TCP
375	```
376
377	- [ ] **Step 5: Create netpol/kustomization.yaml**
378
379	```yaml
380	# deploy/kubernetes/base/netpol/kustomization.yaml
381	apiVersion: kustomize.config.k8s.io/v1beta1
382	kind: Kustomization
383
384	resources:
385	  - allow-dns.yaml
386	  - allow-rds.yaml
387	  - allow-external.yaml
388	```
389
390	- [ ] **Step 6: Validate**
391
392	```bash
393	kustomize build deploy/kubernetes/base/netpol
394	```
395
396	Expected: YAML output with three CiliumNetworkPolicy resources. No errors.
397
398	- [ ] **Step 7: Commit**
399
400	```bash
401	git add deploy/kubernetes/base/netpol/
402	git commit -m "feat: add cilium network policies (dns, rds, external egress)"
403	```
404
405	---
406
407	## Task 4: Services (Deployments, Service, Ingress)
408
409	Creates the web Deployment (Next.js), worker Deployment (pg-boss), ClusterIP Service, and Nginx Ingress.
410
411	**Files:**
412	- Create: `deploy/kubernetes/base/services/web-deployment.yaml`
413	- Create: `deploy/kubernetes/base/services/worker-deployment.yaml`
414	- Create: `deploy/kubernetes/base/services/service.yaml`
415	- Create: `deploy/kubernetes/base/services/ingress.yaml`
416	- Create: `deploy/kubernetes/base/services/kustomization.yaml`
417
418	- [ ] **Step 1: Create directory**
419
420	```bash
421	mkdir -p deploy/kubernetes/base/services
422	```
423
424	- [ ] **Step 2: Create web-deployment.yaml**
425
426	`command: ["bun", "run", "start"]` maps to `next start` from the project's `package.json`. Liveness checks `/api/health/live` to detect a hung process; readiness checks `/api/health/ready` to gate traffic until the app is warm.
427
428	```yaml
429	# deploy/kubernetes/base/services/web-deployment.yaml
430	apiVersion: apps/v1
431	kind: Deployment
432	metadata:
433	  name: churn-renewal-forecast-web
434	  namespace: churn-renewal-forecast
435	  labels:
436	    app.kubernetes.io/name: churn-renewal-forecast
437	    app.kubernetes.io/component: web
438	spec:
439	  replicas: 1
440	  selector:
441	    matchLabels:
442	      app.kubernetes.io/name: churn-renewal-forecast
443	      app.kubernetes.io/component: web
444	  template:
445	    metadata:
446	      labels:
447	        app.kubernetes.io/name: churn-renewal-forecast
448	        app.kubernetes.io/component: web
449	    spec:
450	      serviceAccountName: churn-renewal-forecast
451	      containers:
452	        - name: web
453	          image: registry.ic-ops.net/churn-renewal-forecast:latest
454	          command: ["bun", "run", "start"]
455	          ports:
456	            - containerPort: 3000
457	              name: http
458	          envFrom:
459	            - configMapRef:
460	                name: churn-renewal-forecast-config
461	            - secretRef:
462	                name: churn-renewal-forecast
463	          resources:
464	            requests:
465	              memory: 256Mi
466	              cpu: 100m
467	            limits:
468	              memory: 512Mi
469	              cpu: 500m
470	          livenessProbe:
471	            httpGet:
472	              path: /api/health/live
473	              port: 3000
474	            initialDelaySeconds: 15
475	            periodSeconds: 20
476	          readinessProbe:
477	            httpGet:
478	              path: /api/health/ready
479	              port: 3000
480	            initialDelaySeconds: 10
481	            periodSeconds: 10
482	```
483
484	- [ ] **Step 3: Create worker-deployment.yaml**
485
486	`command: ["bunx", "tsx", "src/worker/index.ts"]` maps to `bun run worker` from `package.json`. No HTTP port or probes — the worker is a queue consumer that runs until killed; Kubernetes monitors the process exit code.
487
488	```yaml
489	# deploy/kubernetes/base/services/worker-deployment.yaml
490	apiVersion: apps/v1
491	kind: Deployment
492	metadata:
493	  name: churn-renewal-forecast-worker
494	  namespace: churn-renewal-forecast
495	  labels:
496	    app.kubernetes.io/name: churn-renewal-forecast
497	    app.kubernetes.io/component: worker
498	spec:
499	  replicas: 1
500	  selector:
501	    matchLabels:
502	      app.kubernetes.io/name: churn-renewal-forecast
503	      app.kubernetes.io/component: worker
504	  template:
505	    metadata:
506	      labels:
507	        app.kubernetes.io/name: churn-renewal-forecast
508	        app.kubernetes.io/component: worker
509	    spec:
510	      serviceAccountName: churn-renewal-forecast
511	      containers:
512	        - name: worker
513	          image: registry.ic-ops.net/churn-renewal-forecast:latest
514	          command: ["bunx", "tsx", "src/worker/index.ts"]
515	          envFrom:
516	            - configMapRef:
517	                name: churn-renewal-forecast-config
518	            - secretRef:
519	                name: churn-renewal-forecast
520	          resources:
521	            requests:
522	              memory: 256Mi
523	              cpu: 100m
524	            limits:
525	              memory: 512Mi
526	              cpu: 500m
527	```
528
529	- [ ] **Step 4: Create service.yaml**
530
531	ClusterIP routes traffic to web pods only (selector scoped to `component: web`). Port 80 externally → 3000 on the container.
532
533	```yaml
534	# deploy/kubernetes/base/services/service.yaml
535	apiVersion: v1
536	kind: Service
537	metadata:
538	  name: churn-renewal-forecast
539	  namespace: churn-renewal-forecast
540	  labels:
541	    app.kubernetes.io/name: churn-renewal-forecast
542	    app.kubernetes.io/component: web
543	spec:
544	  type: ClusterIP
545	  selector:
546	    app.kubernetes.io/name: churn-renewal-forecast
547	    app.kubernetes.io/component: web
548	  ports:
549	    - port: 80
550	      targetPort: 3000
551	      name: http
552	```
553
554	- [ ] **Step 5: Create ingress.yaml**
555
556	TLS terminates at the ingress; the `churn-renewal-forecast-tls` secret is managed externally (cert-manager or equivalent). The `ingressClassName` field replaces the deprecated `kubernetes.io/ingress.class` annotation in Kubernetes 1.18+.
557
558	```yaml
559	# deploy/kubernetes/base/services/ingress.yaml
560	apiVersion: networking.k8s.io/v1
561	kind: Ingress
562	metadata:
563	  name: churn-renewal-forecast
564	  namespace: churn-renewal-forecast
565	  labels:
566	    app.kubernetes.io/name: churn-renewal-forecast
567	spec:
568	  ingressClassName: nginx
569	  tls:
570	    - hosts:
571	        - churn-forecast.infrastructure.immuta.io
572	      secretName: churn-renewal-forecast-tls
573	  rules:
574	    - host: churn-forecast.infrastructure.immuta.io
575	      http:
576	        paths:
577	          - path: /
578	            pathType: Prefix
579	            backend:
580	              service:
581	                name: churn-renewal-forecast
582	                port:
583	                  name: http
584	```
585
586	- [ ] **Step 6: Create services/kustomization.yaml**
587
588	```yaml
589	# deploy/kubernetes/base/services/kustomization.yaml
590	apiVersion: kustomize.config.k8s.io/v1beta1
591	kind: Kustomization
592
593	resources:
594	  - web-deployment.yaml
595	  - worker-deployment.yaml
596	  - service.yaml
597	  - ingress.yaml
598	```
599
600	- [ ] **Step 7: Validate**
601
602	```bash
603	kustomize build deploy/kubernetes/base/services
604	```
605
606	Expected: YAML output with two Deployments, one Service, one Ingress. No errors.
607
608	- [ ] **Step 8: Commit**
609
610	```bash
611	git add deploy/kubernetes/base/services/
612	git commit -m "feat: add k8s service resources (web, worker deployments, svc, ingress)"
613	```
614
615	---
616
617	## Task 5: Base Root Kustomization
618
619	Ties all four base sub-directories together into a single root kustomization.
620
621	**Files:**
622	- Create: `deploy/kubernetes/base/kustomization.yaml`
623
624	- [ ] **Step 1: Create base/kustomization.yaml**
625
626	```yaml
627	# deploy/kubernetes/base/kustomization.yaml
628	apiVersion: kustomize.config.k8s.io/v1beta1
629	kind: Kustomization
630
631	resources:
632	  - bootstrap/
633	  - init/
634	  - netpol/
635	  - services/
636	```
637
638	- [ ] **Step 2: Validate full base build**
639
640	```bash
641	kustomize build deploy/kubernetes/base
642	```
643
644	Expected: Combined YAML output with all resources — Namespace, ServiceAccount, ExternalSecret, ConfigMap, Job, three CiliumNetworkPolicies, two Deployments, Service, Ingress. No errors.
645
646	- [ ] **Step 3: Commit**
647
648	```bash
649	git add deploy/kubernetes/base/kustomization.yaml
650	git commit -m "feat: add k8s base kustomization root"
651	```
652
653	---
654
655	## Task 6: Cluster Overlay (eks-shared-infra-use1-2)
656
657	Creates the overlay that references the base and applies cluster-specific patches: private ingress visibility and a service account patch stub for optional IRSA.
658
659	**Files:**
660	- Create: `deploy/kubernetes/deploy/eks-shared-infra-use1-2/kustomization.yaml`
661	- Create: `deploy/kubernetes/deploy/eks-shared-infra-use1-2/.app.yaml`
662	- Create: `deploy/kubernetes/deploy/eks-shared-infra-use1-2/ingress-patch.yaml`
663	- Create: `deploy/kubernetes/deploy/eks-shared-infra-use1-2/serviceaccount.yaml`
664
665	- [ ] **Step 1: Create directory**
666
667	```bash
668	mkdir -p deploy/kubernetes/deploy/eks-shared-infra-use1-2
669	```
670
671	- [ ] **Step 2: Create .app.yaml**
672
673	Flux metadata file consumed by the cluster's GitOps tooling. Update the `cluster` field if the cluster name differs.
674
675	```yaml
676	# deploy/kubernetes/deploy/eks-shared-infra-use1-2/.app.yaml
677	app: churn-renewal-forecast
678	namespace: churn-renewal-forecast
679	cluster: eks-shared-infra-use1-2
680	```
681
682	- [ ] **Step 3: Create ingress-patch.yaml**
683
684	Strategic merge patch that adds the internal-load-balancer annotation to the base Ingress. This restricts the ALB to private subnets only.
685
686	```yaml
687	# deploy/kubernetes/deploy/eks-shared-infra-use1-2/ingress-patch.yaml
688	apiVersion: networking.k8s.io/v1
689	kind: Ingress
690	metadata:
691	  name: churn-renewal-forecast
692	  namespace: churn-renewal-forecast
693	  annotations:
694	    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
695	    nginx.ingress.kubernetes.io/scheme: internal
696	```
697
698	- [ ] **Step 4: Create serviceaccount.yaml**
699
700	Strategic merge patch stub. The IRSA annotation is commented out because this app does not need AWS API access. Uncomment and supply the role ARN if that changes.
701
702	```yaml
703	# deploy/kubernetes/deploy/eks-shared-infra-use1-2/serviceaccount.yaml
704	apiVersion: v1
705	kind: ServiceAccount
706	metadata:
707	  name: churn-renewal-forecast
708	  namespace: churn-renewal-forecast
709	  # annotations:
710	  #   eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/churn-renewal-forecast
711	```
712
713	- [ ] **Step 5: Create overlay kustomization.yaml**
714
715	```yaml
716	# deploy/kubernetes/deploy/eks-shared-infra-use1-2/kustomization.yaml
717	apiVersion: kustomize.config.k8s.io/v1beta1
718	kind: Kustomization
719
720	resources:
721	  - ../../base
722
723	patches:
724	  - path: ingress-patch.yaml
725	  - path: serviceaccount.yaml
726	```
727
728	- [ ] **Step 6: Validate full overlay build**
729
730	This is the final validation — it produces the exact manifests that would be applied to the cluster.
731
732	```bash
733	kustomize build deploy/kubernetes/deploy/eks-shared-infra-use1-2
734	```
735
736	Expected: All base resources with the Ingress now carrying both `service.beta.kubernetes.io/aws-load-balancer-internal: "true"` and `nginx.ingress.kubernetes.io/scheme: internal` annotations. No errors.
737
738	- [ ] **Step 7: Commit**
739
740	```bash
741	git add deploy/kubernetes/deploy/
742	git commit -m "feat: add eks-shared-infra-use1-2 overlay (private ingress, sa stub)"
743	```
744
745	---
746
747	## Self-Review Checklist
748
749	- [x] Namespace `churn-renewal-forecast` created — Task 1
750	- [x] ServiceAccount created — Task 1
751	- [x] ExternalSecret with all 6 vault keys — Task 1
752	- [x] ConfigMap with all non-sensitive env vars — Task 1
753	- [x] Prisma migration Job — Task 2
754	- [x] CiliumNetworkPolicy: DNS UDP 53 — Task 3
755	- [x] CiliumNetworkPolicy: RDS TCP 5432 — Task 3
756	- [x] CiliumNetworkPolicy: External TCP 443 (clerk, front, productboard, slack) — Task 3
757	- [x] Web Deployment with liveness/readiness probes on `/api/health/live` and `/api/health/ready` — Task 4
758	- [x] Worker Deployment, no probes — Task 4
759	- [x] ClusterIP Service port 80→3000 — Task 4
760	- [x] Nginx Ingress for `churn-forecast.infrastructure.immuta.io` — Task 4
761	- [x] `app.kubernetes.io/name: churn-renewal-forecast` on all resources — Tasks 1–4
762	- [x] `app.kubernetes.io/component` on Deployments and Service — Tasks 1, 4
763	- [x] Both Deployments use `envFrom` for ConfigMap and Secret — Tasks 2, 4
764	- [x] Init Job uses `envFrom` for ConfigMap and Secret — Task 2
765	- [x] Single image `registry.ic-ops.net/churn-renewal-forecast:latest` across all workloads — Tasks 2, 4
766	- [x] Overlay adds private ingress annotations — Task 6
767	- [x] IRSA annotation stub (commented out) — Task 6
768	- [x] Every task has a `kustomize build` validation step — Tasks 1–6
769

# Kustomize Deployment Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Create a complete Kustomize deployment under `deploy/kubernetes/` with a base layer and a single cluster overlay for `eks-shared-infra-use1-2`.

**Architecture:** Base layer is split into four sub-directories (bootstrap, init, netpol, services) each with their own `kustomization.yaml`. The cluster overlay at `deploy/kubernetes/deploy/eks-shared-infra-use1-2/` references the base and applies cluster-specific patches (ingress visibility, service account). All resources share a single container image with command overrides per workload.

**Tech Stack:** Kustomize v4+, Kubernetes manifests (apps/v1, batch/v1, networking.k8s.io/v1), external-secrets.io v1beta1 (ExternalSecret), cilium.io/v2 (CiliumNetworkPolicy)

---

## File Map

**Create — base bootstrap:**
- `deploy/kubernetes/base/bootstrap/namespace.yaml`
- `deploy/kubernetes/base/bootstrap/serviceaccount.yaml`
- `deploy/kubernetes/base/bootstrap/secret.yaml`
- `deploy/kubernetes/base/bootstrap/app-config.env`
- `deploy/kubernetes/base/bootstrap/kustomization.yaml`

**Create — base init:**
- `deploy/kubernetes/base/init/job.yaml`
- `deploy/kubernetes/base/init/kustomization.yaml`

**Create — base netpol:**
- `deploy/kubernetes/base/netpol/allow-dns.yaml`
- `deploy/kubernetes/base/netpol/allow-rds.yaml`
- `deploy/kubernetes/base/netpol/allow-external.yaml`
- `deploy/kubernetes/base/netpol/kustomization.yaml`

**Create — base services:**
- `deploy/kubernetes/base/services/web-deployment.yaml`
- `deploy/kubernetes/base/services/worker-deployment.yaml`
- `deploy/kubernetes/base/services/service.yaml`
- `deploy/kubernetes/base/services/ingress.yaml`
- `deploy/kubernetes/base/services/kustomization.yaml`

**Create — base root:**
- `deploy/kubernetes/base/kustomization.yaml`

**Create — overlay:**
- `deploy/kubernetes/deploy/eks-shared-infra-use1-2/kustomization.yaml`
- `deploy/kubernetes/deploy/eks-shared-infra-use1-2/.app.yaml`
- `deploy/kubernetes/deploy/eks-shared-infra-use1-2/ingress-patch.yaml`
- `deploy/kubernetes/deploy/eks-shared-infra-use1-2/serviceaccount.yaml`

---

## Task 1: Bootstrap Resources

Creates the namespace, service account, ExternalSecret, and ConfigMap source.

**Files:**
- Create: `deploy/kubernetes/base/bootstrap/namespace.yaml`
- Create: `deploy/kubernetes/base/bootstrap/serviceaccount.yaml`
- Create: `deploy/kubernetes/base/bootstrap/secret.yaml`
- Create: `deploy/kubernetes/base/bootstrap/app-config.env`
- Create: `deploy/kubernetes/base/bootstrap/kustomization.yaml`

- [ ] **Step 1: Create directory structure**

```bash
mkdir -p deploy/kubernetes/base/bootstrap
```

- [ ] **Step 2: Create namespace.yaml**

```yaml
# deploy/kubernetes/base/bootstrap/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
```

- [ ] **Step 3: Create serviceaccount.yaml**

```yaml
# deploy/kubernetes/base/bootstrap/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: churn-renewal-forecast
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
```

- [ ] **Step 4: Create secret.yaml (ExternalSecret)**

```yaml
# deploy/kubernetes/base/bootstrap/secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: churn-renewal-forecast
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: churn-renewal-forecast
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: /internal/churn-renewal-forecast
        property: DATABASE_URL
    - secretKey: CLERK_SECRET_KEY
      remoteRef:
        key: /internal/churn-renewal-forecast
        property: CLERK_SECRET_KEY
    - secretKey: FRONT_API_TOKEN
      remoteRef:
        key: /internal/churn-renewal-forecast
        property: FRONT_API_TOKEN
    - secretKey: FRONT_API_TOKEN_2
      remoteRef:
        key: /internal/churn-renewal-forecast
        property: FRONT_API_TOKEN_2
    - secretKey: PRODUCTBOARD_API_KEY
      remoteRef:
        key: /internal/churn-renewal-forecast
        property: PRODUCTBOARD_API_KEY
    - secretKey: SLACK_BOT_TOKEN
      remoteRef:
        key: /internal/churn-renewal-forecast
        property: SLACK_BOT_TOKEN
```

- [ ] **Step 5: Create app-config.env**

```env
# deploy/kubernetes/base/bootstrap/app-config.env
APP_URL=https://churn-forecast.infrastructure.immuta.io
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=PLACEHOLDER
NEXT_PUBLIC_PRODUCTBOARD_WORKSPACE=PLACEHOLDER
SLACK_TEST_CHANNEL=
PG_POOL_MAX=10
PG_POOL_IDLE_TIMEOUT_MS=30000
PG_POOL_CONNECTION_TIMEOUT_MS=2000
NEXT_PUBLIC_CLERK_SIGN_IN_URL=/sign-in
NEXT_PUBLIC_CLERK_SIGN_UP_URL=/sign-up
NEXT_PUBLIC_CLERK_AFTER_SIGN_IN_URL=/
NEXT_PUBLIC_CLERK_AFTER_SIGN_UP_URL=/
```

- [ ] **Step 6: Create bootstrap/kustomization.yaml**

`configMapGenerator` produces the ConfigMap from `app-config.env`. `disableNameSuffixHash: true` keeps the name stable so `envFrom` references don't need to change on every deploy.

```yaml
# deploy/kubernetes/base/bootstrap/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - namespace.yaml
  - serviceaccount.yaml
  - secret.yaml

configMapGenerator:
  - name: churn-renewal-forecast-config
    envs:
      - app-config.env

generatorOptions:
  disableNameSuffixHash: true
```

- [ ] **Step 7: Validate bootstrap kustomization builds**

```bash
kustomize build deploy/kubernetes/base/bootstrap
```

Expected: YAML output containing Namespace, ServiceAccount, ExternalSecret, and ConfigMap. No errors.

- [ ] **Step 8: Commit**

```bash
git add deploy/kubernetes/base/bootstrap/
git commit -m "feat: add k8s bootstrap resources (namespace, sa, secret, configmap)"
```

---

## Task 2: Init Job

Creates the one-shot Prisma migration Job that runs before the app starts.

**Files:**
- Create: `deploy/kubernetes/base/init/job.yaml`
- Create: `deploy/kubernetes/base/init/kustomization.yaml`

- [ ] **Step 1: Create directory**

```bash
mkdir -p deploy/kubernetes/base/init
```

- [ ] **Step 2: Create job.yaml**

`ttlSecondsAfterFinished: 300` auto-deletes the Job pod 5 minutes after it completes. `backoffLimit: 3` retries up to 3 times on failure. `prisma migrate deploy` is idempotent — safe to re-run.

```yaml
# deploy/kubernetes/base/init/job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: churn-renewal-forecast-migrate
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
    app.kubernetes.io/component: migrate
spec:
  ttlSecondsAfterFinished: 300
  backoffLimit: 3
  template:
    metadata:
      labels:
        app.kubernetes.io/name: churn-renewal-forecast
        app.kubernetes.io/component: migrate
    spec:
      serviceAccountName: churn-renewal-forecast
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: registry.ic-ops.net/churn-renewal-forecast:latest
          command: ["bunx", "prisma", "migrate", "deploy"]
          envFrom:
            - configMapRef:
                name: churn-renewal-forecast-config
            - secretRef:
                name: churn-renewal-forecast
```

- [ ] **Step 3: Create init/kustomization.yaml**

```yaml
# deploy/kubernetes/base/init/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - job.yaml
```

- [ ] **Step 4: Validate**

```bash
kustomize build deploy/kubernetes/base/init
```

Expected: YAML output containing a single Job. No errors.

- [ ] **Step 5: Commit**

```bash
git add deploy/kubernetes/base/init/
git commit -m "feat: add k8s init job for prisma migrations"
```

---

## Task 3: Network Policies

Creates three CiliumNetworkPolicy resources covering DNS, RDS, and external HTTPS egress.

**Files:**
- Create: `deploy/kubernetes/base/netpol/allow-dns.yaml`
- Create: `deploy/kubernetes/base/netpol/allow-rds.yaml`
- Create: `deploy/kubernetes/base/netpol/allow-external.yaml`
- Create: `deploy/kubernetes/base/netpol/kustomization.yaml`

- [ ] **Step 1: Create directory**

```bash
mkdir -p deploy/kubernetes/base/netpol
```

- [ ] **Step 2: Create allow-dns.yaml**

Allows pods with `app.kubernetes.io/name: churn-renewal-forecast` to reach `kube-dns` on UDP 53. The `rules.dns` section tells Cilium to track DNS responses so FQDN policies in `allow-external.yaml` can resolve correctly.

```yaml
# deploy/kubernetes/base/netpol/allow-dns.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: churn-renewal-forecast-allow-dns
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: churn-renewal-forecast
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*"
```

- [ ] **Step 3: Create allow-rds.yaml**

Allows outbound TCP 5432 to any IP. Scoped to CIDR 0.0.0.0/0 because RDS runs outside the cluster — you cannot use an endpoint selector for external IPs.

```yaml
# deploy/kubernetes/base/netpol/allow-rds.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: churn-renewal-forecast-allow-rds
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: churn-renewal-forecast
  egress:
    - toCIDR:
        - 0.0.0.0/0
      toPorts:
        - ports:
            - port: "5432"
              protocol: TCP
```

- [ ] **Step 4: Create allow-external.yaml**

`toFQDNs` with `matchName` locks to the exact hostname. `matchPattern: "*.clerk.com"` covers Clerk's subdomains (e.g. `accounts.clerk.com`). `*.slack.com` covers Slack API endpoints. Requires `allow-dns.yaml` to be active so Cilium can resolve these FQDNs.

```yaml
# deploy/kubernetes/base/netpol/allow-external.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: churn-renewal-forecast-allow-external
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: churn-renewal-forecast
  egress:
    - toFQDNs:
        - matchName: "clerk.com"
        - matchPattern: "*.clerk.com"
        - matchName: "api.frontapp.com"
        - matchName: "api.productboard.com"
        - matchName: "slack.com"
        - matchPattern: "*.slack.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
```

- [ ] **Step 5: Create netpol/kustomization.yaml**

```yaml
# deploy/kubernetes/base/netpol/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - allow-dns.yaml
  - allow-rds.yaml
  - allow-external.yaml
```

- [ ] **Step 6: Validate**

```bash
kustomize build deploy/kubernetes/base/netpol
```

Expected: YAML output with three CiliumNetworkPolicy resources. No errors.

- [ ] **Step 7: Commit**

```bash
git add deploy/kubernetes/base/netpol/
git commit -m "feat: add cilium network policies (dns, rds, external egress)"
```

---

## Task 4: Services (Deployments, Service, Ingress)

Creates the web Deployment (Next.js), worker Deployment (pg-boss), ClusterIP Service, and Nginx Ingress.

**Files:**
- Create: `deploy/kubernetes/base/services/web-deployment.yaml`
- Create: `deploy/kubernetes/base/services/worker-deployment.yaml`
- Create: `deploy/kubernetes/base/services/service.yaml`
- Create: `deploy/kubernetes/base/services/ingress.yaml`
- Create: `deploy/kubernetes/base/services/kustomization.yaml`

- [ ] **Step 1: Create directory**

```bash
mkdir -p deploy/kubernetes/base/services
```

- [ ] **Step 2: Create web-deployment.yaml**

`command: ["bun", "run", "start"]` maps to `next start` from the project's `package.json`. Liveness checks `/api/health/live` to detect a hung process; readiness checks `/api/health/ready` to gate traffic until the app is warm.

```yaml
# deploy/kubernetes/base/services/web-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: churn-renewal-forecast-web
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
    app.kubernetes.io/component: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: churn-renewal-forecast
      app.kubernetes.io/component: web
  template:
    metadata:
      labels:
        app.kubernetes.io/name: churn-renewal-forecast
        app.kubernetes.io/component: web
    spec:
      serviceAccountName: churn-renewal-forecast
      containers:
        - name: web
          image: registry.ic-ops.net/churn-renewal-forecast:latest
          command: ["bun", "run", "start"]
          ports:
            - containerPort: 3000
              name: http
          envFrom:
            - configMapRef:
                name: churn-renewal-forecast-config
            - secretRef:
                name: churn-renewal-forecast
          resources:
            requests:
              memory: 256Mi
              cpu: 100m
            limits:
              memory: 512Mi
              cpu: 500m
          livenessProbe:
            httpGet:
              path: /api/health/live
              port: 3000
            initialDelaySeconds: 15
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /api/health/ready
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 10
```

- [ ] **Step 3: Create worker-deployment.yaml**

`command: ["bunx", "tsx", "src/worker/index.ts"]` maps to `bun run worker` from `package.json`. No HTTP port or probes — the worker is a queue consumer that runs until killed; Kubernetes monitors the process exit code.

```yaml
# deploy/kubernetes/base/services/worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: churn-renewal-forecast-worker
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
    app.kubernetes.io/component: worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: churn-renewal-forecast
      app.kubernetes.io/component: worker
  template:
    metadata:
      labels:
        app.kubernetes.io/name: churn-renewal-forecast
        app.kubernetes.io/component: worker
    spec:
      serviceAccountName: churn-renewal-forecast
      containers:
        - name: worker
          image: registry.ic-ops.net/churn-renewal-forecast:latest
          command: ["bunx", "tsx", "src/worker/index.ts"]
          envFrom:
            - configMapRef:
                name: churn-renewal-forecast-config
            - secretRef:
                name: churn-renewal-forecast
          resources:
            requests:
              memory: 256Mi
              cpu: 100m
            limits:
              memory: 512Mi
              cpu: 500m
```

- [ ] **Step 4: Create service.yaml**

ClusterIP routes traffic to web pods only (selector scoped to `component: web`). Port 80 externally → 3000 on the container.

```yaml
# deploy/kubernetes/base/services/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: churn-renewal-forecast
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
    app.kubernetes.io/component: web
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: churn-renewal-forecast
    app.kubernetes.io/component: web
  ports:
    - port: 80
      targetPort: 3000
      name: http
```

- [ ] **Step 5: Create ingress.yaml**

TLS terminates at the ingress; the `churn-renewal-forecast-tls` secret is managed externally (cert-manager or equivalent). The `ingressClassName` field replaces the deprecated `kubernetes.io/ingress.class` annotation in Kubernetes 1.18+.

```yaml
# deploy/kubernetes/base/services/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: churn-renewal-forecast
  namespace: churn-renewal-forecast
  labels:
    app.kubernetes.io/name: churn-renewal-forecast
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - churn-forecast.infrastructure.immuta.io
      secretName: churn-renewal-forecast-tls
  rules:
    - host: churn-forecast.infrastructure.immuta.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: churn-renewal-forecast
                port:
                  name: http
```

- [ ] **Step 6: Create services/kustomization.yaml**

```yaml
# deploy/kubernetes/base/services/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - web-deployment.yaml
  - worker-deployment.yaml
  - service.yaml
  - ingress.yaml
```

- [ ] **Step 7: Validate**

```bash
kustomize build deploy/kubernetes/base/services
```

Expected: YAML output with two Deployments, one Service, one Ingress. No errors.

- [ ] **Step 8: Commit**

```bash
git add deploy/kubernetes/base/services/
git commit -m "feat: add k8s service resources (web, worker deployments, svc, ingress)"
```

---

## Task 5: Base Root Kustomization

Ties all four base sub-directories together into a single root kustomization.

**Files:**
- Create: `deploy/kubernetes/base/kustomization.yaml`

- [ ] **Step 1: Create base/kustomization.yaml**

```yaml
# deploy/kubernetes/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - bootstrap/
  - init/
  - netpol/
  - services/
```

- [ ] **Step 2: Validate full base build**

```bash
kustomize build deploy/kubernetes/base
```

Expected: Combined YAML output with all resources — Namespace, ServiceAccount, ExternalSecret, ConfigMap, Job, three CiliumNetworkPolicies, two Deployments, Service, Ingress. No errors.

- [ ] **Step 3: Commit**

```bash
git add deploy/kubernetes/base/kustomization.yaml
git commit -m "feat: add k8s base kustomization root"
```

---

## Task 6: Cluster Overlay (eks-shared-infra-use1-2)

Creates the overlay that references the base and applies cluster-specific patches: private ingress visibility and a service account patch stub for optional IRSA.

**Files:**
- Create: `deploy/kubernetes/deploy/eks-shared-infra-use1-2/kustomization.yaml`
- Create: `deploy/kubernetes/deploy/eks-shared-infra-use1-2/.app.yaml`
- Create: `deploy/kubernetes/deploy/eks-shared-infra-use1-2/ingress-patch.yaml`
- Create: `deploy/kubernetes/deploy/eks-shared-infra-use1-2/serviceaccount.yaml`

- [ ] **Step 1: Create directory**

```bash
mkdir -p deploy/kubernetes/deploy/eks-shared-infra-use1-2
```

- [ ] **Step 2: Create .app.yaml**

Flux metadata file consumed by the cluster's GitOps tooling. Update the `cluster` field if the cluster name differs.

```yaml
# deploy/kubernetes/deploy/eks-shared-infra-use1-2/.app.yaml
app: churn-renewal-forecast
namespace: churn-renewal-forecast
cluster: eks-shared-infra-use1-2
```

- [ ] **Step 3: Create ingress-patch.yaml**

Strategic merge patch that adds the internal-load-balancer annotation to the base Ingress. This restricts the ALB to private subnets only.

```yaml
# deploy/kubernetes/deploy/eks-shared-infra-use1-2/ingress-patch.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: churn-renewal-forecast
  namespace: churn-renewal-forecast
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    nginx.ingress.kubernetes.io/scheme: internal
```

- [ ] **Step 4: Create serviceaccount.yaml**

Strategic merge patch stub. The IRSA annotation is commented out because this app does not need AWS API access. Uncomment and supply the role ARN if that changes.

```yaml
# deploy/kubernetes/deploy/eks-shared-infra-use1-2/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: churn-renewal-forecast
  namespace: churn-renewal-forecast
  # annotations:
  #   eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/churn-renewal-forecast
```

- [ ] **Step 5: Create overlay kustomization.yaml**

```yaml
# deploy/kubernetes/deploy/eks-shared-infra-use1-2/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

patches:
  - path: ingress-patch.yaml
  - path: serviceaccount.yaml
```

- [ ] **Step 6: Validate full overlay build**

This is the final validation — it produces the exact manifests that would be applied to the cluster.

```bash
kustomize build deploy/kubernetes/deploy/eks-shared-infra-use1-2
```

Expected: All base resources with the Ingress now carrying both `service.beta.kubernetes.io/aws-load-balancer-internal: "true"` and `nginx.ingress.kubernetes.io/scheme: internal` annotations. No errors.

- [ ] **Step 7: Commit**

```bash
git add deploy/kubernetes/deploy/
git commit -m "feat: add eks-shared-infra-use1-2 overlay (private ingress, sa stub)"
```

---

## Self-Review Checklist

- [x] Namespace `churn-renewal-forecast` created — Task 1
- [x] ServiceAccount created — Task 1
- [x] ExternalSecret with all 6 vault keys — Task 1
- [x] ConfigMap with all non-sensitive env vars — Task 1
- [x] Prisma migration Job — Task 2
- [x] CiliumNetworkPolicy: DNS UDP 53 — Task 3
- [x] CiliumNetworkPolicy: RDS TCP 5432 — Task 3
- [x] CiliumNetworkPolicy: External TCP 443 (clerk, front, productboard, slack) — Task 3
- [x] Web Deployment with liveness/readiness probes on `/api/health/live` and `/api/health/ready` — Task 4
- [x] Worker Deployment, no probes — Task 4
- [x] ClusterIP Service port 80→3000 — Task 4
- [x] Nginx Ingress for `churn-forecast.infrastructure.immuta.io` — Task 4
- [x] `app.kubernetes.io/name: churn-renewal-forecast` on all resources — Tasks 1–4
- [x] `app.kubernetes.io/component` on Deployments and Service — Tasks 1, 4
- [x] Both Deployments use `envFrom` for ConfigMap and Secret — Tasks 2, 4
- [x] Init Job uses `envFrom` for ConfigMap and Secret — Task 2
- [x] Single image `registry.ic-ops.net/churn-renewal-forecast:latest` across all workloads — Tasks 2, 4
- [x] Overlay adds private ingress annotations — Task 6
- [x] IRSA annotation stub (commented out) — Task 6
- [x] Every task has a `kustomize build` validation step — Tasks 1–6


Now let me read the overlay kustomization files and understand how `pack` is being used.


1	apiVersion: kustomize.config.k8s.io/v1beta1
2	kind: Kustomization
3
4	resources:
5	  - ../../../base/services
6
7	images:
8	  - name: registry.ic-ops.net/internal-tooling/churn-renewal-forecast
9	    newTag: latest
10
11	patches:
12	  - path: ingress-patch.yaml
13

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../../base/services

images:
  - name: registry.ic-ops.net/internal-tooling/churn-renewal-forecast
    newTag: latest

patches:
  - path: ingress-patch.yaml


1	---
2	apiVersion: ic-ops.com/v1
3	kind: KustomizeApp
4	spec:
5	  app: churn-renewal-forecast
6	  phase: services
7	  ociRepo: internal-tooling
8	target:
9	  env: prod
10	  globalSegment: plat
11	  clusterName: eks-platform-mgmt-use1-1
12

---
apiVersion: ic-ops.com/v1
kind: KustomizeApp
spec:
  app: churn-renewal-forecast
  phase: services
  ociRepo: internal-tooling
target:
  env: prod
  globalSegment: plat
  clusterName: eks-platform-mgmt-use1-1


1	---
2	apiVersion: kustomize.config.k8s.io/v1beta1
3	kind: Kustomization
4	namespace: churn-renewal-forecast
5
6	resources:
7	  - ../../../base/bootstrap
8	  - ../../../base/netpol
9
10	patches:
11	  - path: serviceaccount.yaml
12

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: churn-renewal-forecast

resources:
  - ../../../base/bootstrap
  - ../../../base/netpol

patches:
  - path: serviceaccount.yaml


Let me also check the spec and the init overlay.


1	---
2	apiVersion: kustomize.config.k8s.io/v1beta1
3	kind: Kustomization
4
5	namespace: churn-renewal-forecast
6
7	resources:
8	  - ../../../base/init
9
10	images:
11	  - name: registry.ic-ops.net/internal-tooling/churn-renewal-forecast
12	    newTag: latest
13

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: churn-renewal-forecast

resources:
  - ../../../base/init

images:
  - name: registry.ic-ops.net/internal-tooling/churn-renewal-forecast
    newTag: latest


1	# Kustomize Deployment Design
2
3	**Date:** 2026-05-18
4	**Project:** churn-renewal-forecast
5	**Status:** Approved
6
7	## Overview
8
9	Add a Kustomize-based Kubernetes deployment under `deploy/kubernetes/`, modelled after the `saas-tenant-manager` pattern. The app has two runtime components (Next.js web server and pg-boss background worker) that share a single container image with different command overrides. A one-shot init Job handles Prisma migrations before pods come up.
10
11	## Directory Structure
12
13	```
14	deploy/kubernetes/
15	├── base/
16	│   ├── kustomization.yaml
17	│   ├── bootstrap/
18	│   │   ├── kustomization.yaml
19	│   │   ├── namespace.yaml
20	│   │   ├── serviceaccount.yaml
21	│   │   ├── secret.yaml           # ExternalSecret pulling from Vault
22	│   │   └── app-config.env        # ConfigMap values
23	│   ├── init/
24	│   │   ├── kustomization.yaml
25	│   │   └── job.yaml              # Prisma migrate deploy
26	│   ├── netpol/
27	│   │   ├── kustomization.yaml
28	│   │   ├── allow-dns.yaml
29	│   │   ├── allow-rds.yaml
30	│   │   └── allow-external.yaml
31	│   └── services/
32	│       ├── kustomization.yaml
33	│       ├── web-deployment.yaml
34	│       ├── worker-deployment.yaml
35	│       ├── service.yaml
36	│       └── ingress.yaml
37	└── deploy/
38	    └── eks-shared-infra-use1-2/
39	        ├── kustomization.yaml
40	        ├── .app.yaml
41	        ├── ingress-patch.yaml
42	        └── serviceaccount.yaml
43	```
44
45	## Components
46
47	### bootstrap/
48
49	**namespace.yaml** — Creates the `churn-renewal-forecast` namespace with label `app.kubernetes.io/name: churn-renewal-forecast`.
50
51	**serviceaccount.yaml** — ServiceAccount named `churn-renewal-forecast` in the namespace. No IRSA annotation needed at base level (app does not access AWS APIs directly).
52
53	**secret.yaml** — `ExternalSecret` (external-secrets.io v1beta1) that pulls from `ClusterSecretStore` named `vault` at path `/internal/churn-renewal-forecast`. Maps the following keys to a Secret named `churn-renewal-forecast`:
54
55	| Vault key | Secret env key |
56	|---|---|
57	| `DATABASE_URL` | `DATABASE_URL` |
58	| `CLERK_SECRET_KEY` | `CLERK_SECRET_KEY` |
59	| `FRONT_API_TOKEN` | `FRONT_API_TOKEN` |
60	| `FRONT_API_TOKEN_2` | `FRONT_API_TOKEN_2` |
61	| `PRODUCTBOARD_API_KEY` | `PRODUCTBOARD_API_KEY` |
62	| `SLACK_BOT_TOKEN` | `SLACK_BOT_TOKEN` |
63
64	Refresh interval: `1h`.
65
66	**app-config.env** — ConfigMap (generated by Kustomize `configMapGenerator`) with non-sensitive config:
67
68	| Key | Value |
69	|---|---|
70	| `APP_URL` | `https://churn-forecast.infrastructure.immuta.io` |
71	| `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` | `PLACEHOLDER` |
72	| `NEXT_PUBLIC_PRODUCTBOARD_WORKSPACE` | `PLACEHOLDER` |
73	| `SLACK_TEST_CHANNEL` | `""` |
74	| `PG_POOL_MAX` | `10` |
75	| `PG_POOL_IDLE_TIMEOUT_MS` | `30000` |
76	| `PG_POOL_CONNECTION_TIMEOUT_MS` | `2000` |
77	| `NEXT_PUBLIC_CLERK_SIGN_IN_URL` | `/sign-in` |
78	| `NEXT_PUBLIC_CLERK_SIGN_UP_URL` | `/sign-up` |
79	| `NEXT_PUBLIC_CLERK_AFTER_SIGN_IN_URL` | `/` |
80	| `NEXT_PUBLIC_CLERK_AFTER_SIGN_UP_URL` | `/` |
81
82	### init/
83
84	**job.yaml** — One-shot Kubernetes Job:
85	- Image: `registry.ic-ops.net/churn-renewal-forecast:latest`
86	- Command: `["bunx", "prisma", "migrate", "deploy"]`
87	- `restartPolicy: OnFailure`
88	- `backoffLimit: 3`
89	- `envFrom` the ConfigMap and Secret
90	- Uses the `churn-renewal-forecast` ServiceAccount
91	- `ttlSecondsAfterFinished: 300` to clean up completed jobs
92
93	### netpol/
94
95	Three `CiliumNetworkPolicy` resources, all selecting pods with `app.kubernetes.io/name: churn-renewal-forecast`:
96
97	**allow-dns.yaml** — Egress to kube-dns on UDP port 53.
98
99	**allow-rds.yaml** — Egress on TCP port 5432 (PostgreSQL/RDS).
100
101	**allow-external.yaml** — Egress on TCP port 443 to:
102	- `clerk.com`
103	- `api.frontapp.com`
104	- `api.productboard.com`
105	- `slack.com`
106
107	Using `toFQDNs` matchers.
108
109	### services/
110
111	**web-deployment.yaml** — Deployment:
112	- Name: `churn-renewal-forecast-web`
113	- Replicas: `1`
114	- Image: `registry.ic-ops.net/churn-renewal-forecast:latest`
115	- Command override: `["bun", "run", "start"]` (`next start`)
116	- Port: `3000`
117	- `envFrom` ConfigMap + Secret
118	- Resources: `requests: {memory: 256Mi, cpu: 100m}`, `limits: {memory: 512Mi, cpu: 500m}`
119	- Liveness probe: HTTP GET `/api/health/live` port 3000, `initialDelaySeconds: 15`, `periodSeconds: 20`
120	- Readiness probe: HTTP GET `/api/health/ready` port 3000, `initialDelaySeconds: 10`, `periodSeconds: 10`
121	- ServiceAccount: `churn-renewal-forecast`
122
123	**worker-deployment.yaml** — Deployment:
124	- Name: `churn-renewal-forecast-worker`
125	- Replicas: `1`
126	- Image: `registry.ic-ops.net/churn-renewal-forecast:latest`
127	- Command override: `["bunx", "tsx", "src/worker/index.ts"]`
128	- No HTTP port exposed
129	- `envFrom` ConfigMap + Secret
130	- Resources: same as web
131	- No HTTP probes (queue consumer; process liveness is sufficient)
132	- ServiceAccount: `churn-renewal-forecast`
133
134	**service.yaml** — ClusterIP Service targeting `churn-renewal-forecast-web` pods on port 3000.
135
136	**ingress.yaml** — Nginx Ingress:
137	- Host: `churn-forecast.infrastructure.immuta.io`
138	- Backend: the ClusterIP service on port 3000
139	- TLS: references a secret `churn-renewal-forecast-tls` (managed externally / cert-manager)
140
141	### Overlay: deploy/eks-shared-infra-use1-2/
142
143	**kustomization.yaml** — References `../../base` and applies patches.
144
145	**.app.yaml** — Flux app metadata file with app name, namespace, and target cluster.
146
147	**ingress-patch.yaml** — Strategic merge patch adding the private visibility annotation to the base Ingress (e.g., `nginx.ingress.kubernetes.io/scheme: internal` or equivalent annotation used by the cluster).
148
149	**serviceaccount.yaml** — Patch file. IRSA annotation is commented out since no AWS API access is needed. Can be uncommented if requirements change.
150
151	## Labels
152
153	All resources use `app.kubernetes.io/name: churn-renewal-forecast` consistently. Deployments also add `app.kubernetes.io/component: web` or `app.kubernetes.io/component: worker`.
154
155	## Image Strategy
156
157	Single image `registry.ic-ops.net/churn-renewal-forecast:latest` shared across web Deployment, worker Deployment, and init Job. The image tag should be pinned to a specific digest or semver tag in CI via a Kustomize image transform in the overlay.
158
159	## Deployment Ordering
160
161	Kustomize itself does not enforce that the init Job completes before the web/worker Deployments start. Ordering is handled via Flux `Kustomization` resources with `dependsOn` — the init kustomization must complete (health check: Job succeeded) before the services kustomization is applied.
162
163	The overlay's `.app.yaml` captures the Flux app metadata for this pattern.
164
165	## Open Items
166
167	- `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` and `NEXT_PUBLIC_PRODUCTBOARD_WORKSPACE` values need to be filled in for the target environment.
168	- TLS secret name (`churn-renewal-forecast-tls`) assumes cert-manager or equivalent; adjust if cluster uses a different mechanism.
169	- Confirm whether `FRONT_RATE_LIMIT_RPM` should be in the ConfigMap (currently omitted since the spec didn't list it; add if needed).
170	- If the app ever needs AWS API access (S3, Secrets Manager), uncomment the IRSA annotation in the overlay serviceaccount.yaml and supply the role ARN.
171
172

# Kustomize Deployment Design

**Date:** 2026-05-18
**Project:** churn-renewal-forecast
**Status:** Approved

## Overview

Add a Kustomize-based Kubernetes deployment under `deploy/kubernetes/`, modelled after the `saas-tenant-manager` pattern. The app has two runtime components (Next.js web server and pg-boss background worker) that share a single container image with different command overrides. A one-shot init Job handles Prisma migrations before pods come up.

## Directory Structure

```
deploy/kubernetes/
├── base/
│   ├── kustomization.yaml
│   ├── bootstrap/
│   │   ├── kustomization.yaml
│   │   ├── namespace.yaml
│   │   ├── serviceaccount.yaml
│   │   ├── secret.yaml           # ExternalSecret pulling from Vault
│   │   └── app-config.env        # ConfigMap values
│   ├── init/
│   │   ├── kustomization.yaml
│   │   └── job.yaml              # Prisma migrate deploy
│   ├── netpol/
│   │   ├── kustomization.yaml
│   │   ├── allow-dns.yaml
│   │   ├── allow-rds.yaml
│   │   └── allow-external.yaml
│   └── services/
│       ├── kustomization.yaml
│       ├── web-deployment.yaml
│       ├── worker-deployment.yaml
│       ├── service.yaml
│       └── ingress.yaml
└── deploy/
    └── eks-shared-infra-use1-2/
        ├── kustomization.yaml
        ├── .app.yaml
        ├── ingress-patch.yaml
        └── serviceaccount.yaml
```

## Components

### bootstrap/

**namespace.yaml** — Creates the `churn-renewal-forecast` namespace with label `app.kubernetes.io/name: churn-renewal-forecast`.

**serviceaccount.yaml** — ServiceAccount named `churn-renewal-forecast` in the namespace. No IRSA annotation needed at base level (app does not access AWS APIs directly).

**secret.yaml** — `ExternalSecret` (external-secrets.io v1beta1) that pulls from `ClusterSecretStore` named `vault` at path `/internal/churn-renewal-forecast`. Maps the following keys to a Secret named `churn-renewal-forecast`:

| Vault key | Secret env key |
|---|---|
| `DATABASE_URL` | `DATABASE_URL` |
| `CLERK_SECRET_KEY` | `CLERK_SECRET_KEY` |
| `FRONT_API_TOKEN` | `FRONT_API_TOKEN` |
| `FRONT_API_TOKEN_2` | `FRONT_API_TOKEN_2` |
| `PRODUCTBOARD_API_KEY` | `PRODUCTBOARD_API_KEY` |
| `SLACK_BOT_TOKEN` | `SLACK_BOT_TOKEN` |

Refresh interval: `1h`.

**app-config.env** — ConfigMap (generated by Kustomize `configMapGenerator`) with non-sensitive config:

| Key | Value |
|---|---|
| `APP_URL` | `https://churn-forecast.infrastructure.immuta.io` |
| `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` | `PLACEHOLDER` |
| `NEXT_PUBLIC_PRODUCTBOARD_WORKSPACE` | `PLACEHOLDER` |
| `SLACK_TEST_CHANNEL` | `""` |
| `PG_POOL_MAX` | `10` |
| `PG_POOL_IDLE_TIMEOUT_MS` | `30000` |
| `PG_POOL_CONNECTION_TIMEOUT_MS` | `2000` |
| `NEXT_PUBLIC_CLERK_SIGN_IN_URL` | `/sign-in` |
| `NEXT_PUBLIC_CLERK_SIGN_UP_URL` | `/sign-up` |
| `NEXT_PUBLIC_CLERK_AFTER_SIGN_IN_URL` | `/` |
| `NEXT_PUBLIC_CLERK_AFTER_SIGN_UP_URL` | `/` |

### init/

**job.yaml** — One-shot Kubernetes Job:
- Image: `registry.ic-ops.net/churn-renewal-forecast:latest`
- Command: `["bunx", "prisma", "migrate", "deploy"]`
- `restartPolicy: OnFailure`
- `backoffLimit: 3`
- `envFrom` the ConfigMap and Secret
- Uses the `churn-renewal-forecast` ServiceAccount
- `ttlSecondsAfterFinished: 300` to clean up completed jobs

### netpol/

Three `CiliumNetworkPolicy` resources, all selecting pods with `app.kubernetes.io/name: churn-renewal-forecast`:

**allow-dns.yaml** — Egress to kube-dns on UDP port 53.

**allow-rds.yaml** — Egress on TCP port 5432 (PostgreSQL/RDS).

**allow-external.yaml** — Egress on TCP port 443 to:
- `clerk.com`
- `api.frontapp.com`
- `api.productboard.com`
- `slack.com`

Using `toFQDNs` matchers.

### services/

**web-deployment.yaml** — Deployment:
- Name: `churn-renewal-forecast-web`
- Replicas: `1`
- Image: `registry.ic-ops.net/churn-renewal-forecast:latest`
- Command override: `["bun", "run", "start"]` (`next start`)
- Port: `3000`
- `envFrom` ConfigMap + Secret
- Resources: `requests: {memory: 256Mi, cpu: 100m}`, `limits: {memory: 512Mi, cpu: 500m}`
- Liveness probe: HTTP GET `/api/health/live` port 3000, `initialDelaySeconds: 15`, `periodSeconds: 20`
- Readiness probe: HTTP GET `/api/health/ready` port 3000, `initialDelaySeconds: 10`, `periodSeconds: 10`
- ServiceAccount: `churn-renewal-forecast`

**worker-deployment.yaml** — Deployment:
- Name: `churn-renewal-forecast-worker`
- Replicas: `1`
- Image: `registry.ic-ops.net/churn-renewal-forecast:latest`
- Command override: `["bunx", "tsx", "src/worker/index.ts"]`
- No HTTP port exposed
- `envFrom` ConfigMap + Secret
- Resources: same as web
- No HTTP probes (queue consumer; process liveness is sufficient)
- ServiceAccount: `churn-renewal-forecast`

**service.yaml** — ClusterIP Service targeting `churn-renewal-forecast-web` pods on port 3000.

**ingress.yaml** — Nginx Ingress:
- Host: `churn-forecast.infrastructure.immuta.io`
- Backend: the ClusterIP service on port 3000
- TLS: references a secret `churn-renewal-forecast-tls` (managed externally / cert-manager)

### Overlay: deploy/eks-shared-infra-use1-2/

**kustomization.yaml** — References `../../base` and applies patches.

**.app.yaml** — Flux app metadata file with app name, namespace, and target cluster.

**ingress-patch.yaml** — Strategic merge patch adding the private visibility annotation to the base Ingress (e.g., `nginx.ingress.kubernetes.io/scheme: internal` or equivalent annotation used by the cluster).

**serviceaccount.yaml** — Patch file. IRSA annotation is commented out since no AWS API access is needed. Can be uncommented if requirements change.

## Labels

All resources use `app.kubernetes.io/name: churn-renewal-forecast` consistently. Deployments also add `app.kubernetes.io/component: web` or `app.kubernetes.io/component: worker`.

## Image Strategy

Single image `registry.ic-ops.net/churn-renewal-forecast:latest` shared across web Deployment, worker Deployment, and init Job. The image tag should be pinned to a specific digest or semver tag in CI via a Kustomize image transform in the overlay.

## Deployment Ordering

Kustomize itself does not enforce that the init Job completes before the web/worker Deployments start. Ordering is handled via Flux `Kustomization` resources with `dependsOn` — the init kustomization must complete (health check: Job succeeded) before the services kustomization is applied.

The overlay's `.app.yaml` captures the Flux app metadata for this pattern.

## Open Items

- `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` and `NEXT_PUBLIC_PRODUCTBOARD_WORKSPACE` values need to be filled in for the target environment.
- TLS secret name (`churn-renewal-forecast-tls`) assumes cert-manager or equivalent; adjust if cluster uses a different mechanism.
- Confirm whether `FRONT_RATE_LIMIT_RPM` should be in the ConfigMap (currently omitted since the spec didn't list it; add if needed).
- If the app ever needs AWS API access (S3, Secrets Manager), uncomment the IRSA annotation in the overlay serviceaccount.yaml and supply the role ARN.


Found it, Sir Charles of St Cheese. No cap, this issue is bussin' obvious once you clock it.

**Root cause: the kustomize overlays always hardcode `newTag: latest`**

Both overlay kustomization files have this:
```yaml
# deploy/kubernetes/deploy/eks-platform-mgmt-use1-1/services/kustomization.yaml
images:
  - name: registry.ic-ops.net/internal-tooling/churn-renewal-forecast
    newTag: latest   # ← never changes!
```

Same in `init/kustomization.yaml`. So every `pack build` packages OCI artifacts whose manifests are **bit-for-bit identical** regardless of what image was just pushed. Flux compares content digests — same content = same digest = no reconciliation triggered.

The `BUILD_VARS` env var in the CD workflow is the mechanism to fix this, but it's currently empty:
```yaml
BUILD_VARS: ''   # ← this should be passing the rev_tag
```

The fix is to inject the rev_tag into the overlays **before** `pack build` runs. Since the `pack --set` key format for your internal tooling isn't documented in the repo, the safest approach is to update the kustomization files directly:


70	      - name: Build overlays
71	        env:
72	          DEPLOY_DIR: deploy/kubernetes/deploy
73	          BUILD_VARS: ''
74	        run: |
75	          SET_ARGS=()
76	          if [ -n "${BUILD_VARS}" ]; then
77	            IFS=',' read -ra PAIRS <<< "${BUILD_VARS}"
78	            for pair in "${PAIRS[@]}"; do
79	              SET_ARGS+=( --set "${pair}" )
80	            done
81	          fi
82	          pack build \
83	            --directory "${DEPLOY_DIR}" \
84	            --recursive \
85	            --debug \
86	            "${SET_ARGS[@]}"
87
88	      - name: Push overlays
89	        env:
90	          DEPLOY_DIR: deploy/kubernetes/deploy
91	          TAG: ${{ steps.tag.outputs.rev_tag }}
92	          OCI_USERNAME: ${{ vars.REGISTRY_USERNAME }}
93	          OCI_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
94	        run: |
95	          pack push \
96	            --directory "${DEPLOY_DIR}" \
97	            --tag "${TAG}" \
98	            --recursive \
99	            --debug
100
101

      - name: Build overlays
        env:
          DEPLOY_DIR: deploy/kubernetes/deploy
          BUILD_VARS: ''
        run: |
          SET_ARGS=()
          if [ -n "${BUILD_VARS}" ]; then
            IFS=',' read -ra PAIRS <<< "${BUILD_VARS}"
            for pair in "${PAIRS[@]}"; do
              SET_ARGS+=( --set "${pair}" )
            done
          fi
          pack build \
            --directory "${DEPLOY_DIR}" \
            --recursive \
            --debug \
            "${SET_ARGS[@]}"

      - name: Push overlays
        env:
          DEPLOY_DIR: deploy/kubernetes/deploy
          TAG: ${{ steps.tag.outputs.rev_tag }}
          OCI_USERNAME: ${{ vars.REGISTRY_USERNAME }}
          OCI_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
        run: |
          pack push \
            --directory "${DEPLOY_DIR}" \
            --tag "${TAG}" \
            --recursive \
            --debug


Do you know the exact `--set` key format for your internal `pack` tool? If not, I can add a pre-build step that patches the kustomization files directly with `sed`. Here's what the fix looks like either way:


AskUserQuestion

Question: Do you know the --set key format that your `pack` tool uses for overriding image tags?

Choices:
Yes, it's something like `image.tag=<value>`
You know the exact key; we can use BUILD_VARS
Patch kustomization files directly before pack build runs (safer, no guessing)


Your questions have been answered: "Do you know the --set key format that your `pack` tool uses for overriding image tags?"="i am  not too sure, can you check pack to be sure? @~/projects/operations/apps/pack". You can now continue with these answers in mind.

Question: Do you know the --set key format that your `pack` tool uses for overriding image tags?

Choices:
Yes, it's something like `image.tag=<value>`
You know the exact key; we can use BUILD_VARS
Patch kustomization files directly before pack build runs (safer, no guessing)

i am  not too sure, can you check pack to be sure? @~/projects/operations/apps/pack


Let me check the `pack` tool source to find the right `--set` key format.


Bash

find ~/projects/operations/apps/pack -type f | sort | head -60

List pack tool source files


1	package overlay
2
3	import (
4		"fmt"
5
6		jsonpatch "github.com/evanphx/json-patch/v5"
7		"github.com/immuta/operations/apps/pack/internal/artifact"
8		"k8s.io/apimachinery/pkg/util/json"
9		"sigs.k8s.io/yaml"
10	)
11
12	func applyPatchesToDoc(patches []artifact.RFC6902Patch, docBytes []byte, relPath string) ([]byte, error) {
13		// Convert patch structs to JSON bytes for the patch library
14		patchBytes, err := json.Marshal(patches)
15		if err != nil {
16			return nil, fmt.Errorf("failed to marshal patches for %s: %w", relPath, err)
17		}
18
19		// Create the JSON Patch object
20		patch, err := jsonpatch.DecodePatch(patchBytes)
21		if err != nil {
22			return nil, fmt.Errorf("failed to decode patches for %s: %w", relPath, err)
23		}
24
25		// Convert original YAML to JSON to apply the patch
26		jsonContent, err := yaml.YAMLToJSON(docBytes)
27		if err != nil {
28			return nil, fmt.Errorf("failed to convert %s to JSON: %w", relPath, err)
29		}
30
31		modifiedJSON, err := patch.Apply(jsonContent)
32		if err != nil {
33			return nil, fmt.Errorf("failed to apply patch to %s: %w", relPath, err)
34		}
35
36		// Convert back to YAML
37		modifiedYAML, err := yaml.JSONToYAML(modifiedJSON)
38		if err != nil {
39			return nil, fmt.Errorf("failed to convert patched %s back to YAML: %w", relPath, err)
40		}
41
42		return modifiedYAML, nil
43	}
44

package overlay

import (
	"fmt"

	jsonpatch "github.com/evanphx/json-patch/v5"
	"github.com/immuta/operations/apps/pack/internal/artifact"
	"k8s.io/apimachinery/pkg/util/json"
	"sigs.k8s.io/yaml"
)

func applyPatchesToDoc(patches []artifact.RFC6902Patch, docBytes []byte, relPath string) ([]byte, error) {
	// Convert patch structs to JSON bytes for the patch library
	patchBytes, err := json.Marshal(patches)
	if err != nil {
		return nil, fmt.Errorf("failed to marshal patches for %s: %w", relPath, err)
	}

	// Create the JSON Patch object
	patch, err := jsonpatch.DecodePatch(patchBytes)
	if err != nil {
		return nil, fmt.Errorf("failed to decode patches for %s: %w", relPath, err)
	}

	// Convert original YAML to JSON to apply the patch
	jsonContent, err := yaml.YAMLToJSON(docBytes)
	if err != nil {
		return nil, fmt.Errorf("failed to convert %s to JSON: %w", relPath, err)
	}

	modifiedJSON, err := patch.Apply(jsonContent)
	if err != nil {
		return nil, fmt.Errorf("failed to apply patch to %s: %w", relPath, err)
	}

	// Convert back to YAML
	modifiedYAML, err := yaml.JSONToYAML(modifiedJSON)
	if err != nil {
		return nil, fmt.Errorf("failed to convert patched %s back to YAML: %w", relPath, err)
	}

	return modifiedYAML, nil
}