Analyze the project in this directory, it is called Crow, a golang private terraform repository. I need to update this service to accept terraform modules with any given number of paths instead of limited amount


- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, use Config tool.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- simplify: Review changed code for reuse, quality, and efficiency, then fix any issues found.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled remote agents (triggers) that execute on a cron schedule. - When the user wants to schedule a recurring remote agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/triggers.
- claude-api: Build, debug, and optimize Claude API / Anthropic SDK apps. Apps built with this skill should include prompt caching.
TRIGGER when: code imports `anthropic`/`@anthropic-ai/sdk`; user asks for the Claude API, Anthropic SDK, or Managed Agents; user adds/modifies/tunes a Claude feature (caching, thinking, compaction, tool use, batch, files, citations, memory) or model (Opus/Sonnet/Haiku) in a file; questions about prompt caching / cache hit rate in an Anthropic SDK project.
SKIP: file imports `openai`/other-provider SDK, filename like `*-openai.py`/`*-generic.py`, provider-neutral code, general programming/ML.
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch


Agent

Explore Crow terraform repository structure

Prompt:
Explore the Go project at /Users/steve.briggs/projects/operations/services/crow. This is called "Crow" and is a private Terraform registry/repository service. I need to understand:

1. The overall project structure (directories, key files)
2. How terraform modules are handled - specifically how module paths/versions are defined and stored
3. Any structs or types that define module paths (looking for anything that might have a fixed number of path segments like path1, path2, path3 rather than a dynamic slice)
4. The API endpoints for module operations
5. How modules are identified (namespace, name, provider, version, etc.)
6. Database models or storage layer for modules
7. Any path parsing or routing logic

Be thorough - read key source files to understand the architecture. Report: file tree summary, key types/structs for modules, how paths work currently, and what would need to change to support arbitrary numbers of path segments.


Perfect! Now I have a comprehensive understanding. Let me compile the final report.

## Crow Project Exploration Report

I've thoroughly analyzed the Crow project structure, module handling, and current path architecture. Here's a complete breakdown:

### 1. Project Structure

```
/Users/steve.briggs/projects/operations/services/crow/
├── main.go                          # Entry point, sets up Echo HTTP server
├── README.md                        # Complete API documentation
├── go.mod / go.sum                  # Go dependencies (Echo, AWS SDK v2, testify)
├── Dockerfile                       # Container image for service
├── docker-compose.yaml              # Local dev with RustFS (S3-compatible backend)
├── internal/
│   ├── config/
│   │   ├── config.go                # Environment-based configuration loading
│   │   └── config_test.go
│   ├── registry/
│   │   ├── handler.go               # HTTP handlers & routing (the main logic)
│   │   └── handler_test.go          # Comprehensive tests
│   └── storage/
│       ├── storage.go               # Client interface definition
│       ├── s3.go                    # S3 implementation (key file for path handling)
│       ├── s3_test.go               # Tests for path parsing logic
│       └── mock_storage.go          # Mock for testing
├── e2e/
│   ├── runner/
│   │   └── main.tf                  # Example Terraform config
│   └── test-module/                 # Sample module for testing
└── base/ deploy/                    # K8s deployment manifests
```

### 2. Current Module Path & Identification Structure

**Fixed 4-segment path structure:**

- **Namespace** (e.g., "hashicorp", "foo", "myorg")
- **Name** (e.g., "consul", "test", "mymodule")
- **Provider** (e.g., "aws", "gcp", "azure")
- **Version** (e.g., "1.0.0", "2.3.1")

**API Endpoints (all use exactly 4 path segments):**

```
GET  /.well-known/terraform.json                        # Service discovery
GET  /v1/modules/:namespace/:name/:provider/versions    # List versions
GET  /v1/modules/:namespace/:name/:provider/:version/download   # Get presigned URL
PUT  /v1/modules/:namespace/:name/:provider/:version    # Upload module
GET  /health /livez /readyz                            # Health checks
```

**S3 Storage Key Structure:**

```
{namespace}/{name}/{provider}/{version}.tar.gz
Example: hashicorp/consul/aws/1.0.0.tar.gz
         mymodule/mymodule/aws/1.0.0.tar.gz
```

### 3. Key Type Definitions & Structs

**Handler struct** (`/internal/registry/handler.go`, lines 18-23):
```go
type Handler struct {
    store         storage.Client
    presignExpiry time.Duration
    maxUploadSize int64
    rwPassed      atomic.Bool
}
```
All path parameters extracted from Echo context as individual strings (`namespace`, `name`, `provider`, `version`).

**Storage Client Interface** (`/internal/storage/storage.go`):
```go
type Client interface {
    ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error)
    HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error)
    PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error)
    PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error
    HeadBucket(ctx context.Context) error
    CheckReadWrite(ctx context.Context) error
}
```
**All methods receive path segments as individual string parameters, never as slices.**

**S3Client struct** (`/internal/storage/s3.go`, lines 16-20):
```go
type S3Client struct {
    client     *s3.Client
    presigner  *s3.PresignClient
    bucketName string
}
```
Builds paths using helper functions: `objectKey()` and `prefix()`.

### 4. Path Handling Logic

**Currently, all path operations are hardcoded for 4 segments:**

1. **Routing** (`handler.go`, line 31-33):
   ```go
   e.GET("/v1/modules/:namespace/:name/:provider/versions", h.ListVersions)
   e.GET("/v1/modules/:namespace/:name/:provider/:version/download", h.Download)
   e.PUT("/v1/modules/:namespace/:name/:provider/:version", h.Upload)
   ```
   Echo framework extracts exactly these 4 named parameters.

2. **Handler extracts params directly** (`handler.go`, lines 77-79, 106-109, 139-142):
   ```go
   namespace := c.Param("namespace")
   name := c.Param("name")
   provider := c.Param("provider")
   version := c.Param("version")  // or for versions endpoint, omitted
   ```

3. **S3 path building** (`s3.go`, lines 48-54):
   ```go
   func objectKey(namespace, name, provider, version string) string {
       return fmt.Sprintf("%s/%s/%s/%s.tar.gz", namespace, name, provider, version)
   }

   func prefix(namespace, name, provider string) string {
       return fmt.Sprintf("%s/%s/%s/", namespace, name, provider)
   }
   ```
   **Uses simple string slicing with fixed positions, no dynamic handling.**

4. **Version extraction from S3 keys** (`s3.go`, lines 56-69):
   ```go
   func extractVersion(key, pfx string) (string, bool) {
       if !strings.HasPrefix(key, pfx) {
           return "", false
       }
       rest := strings.TrimPrefix(key, pfx)
       if !strings.HasSuffix(rest, ".tar.gz") {
           return "", false
       }
       version := strings.TrimSuffix(rest, ".tar.gz")
       if version == "" {
           return "", false
       }
       return version, true
   }
   ```
   **This one is already dynamic** - it extracts whatever follows the 3-segment prefix and ends in `.tar.gz`.

### 5. HTTP Handler Implementation

**ListVersions** (`handler.go`, lines 76-103):
- Extracts 3 segments (namespace/name/provider)
- Calls storage.ListVersions() with individual strings
- Uses S3 prefix listing to find all matching versions

**Download** (`handler.go`, lines 105-131):
- Extracts 4 segments
- Checks module exists with HeadModule()
- Returns presigned URL in `X-Terraform-Get` header (204 No Content)

**Upload** (`handler.go`, lines 138-184):
- Extracts 4 segments
- Validates gzip magic header
- Checks for duplicate version
- Stores to S3

### 6. Configuration & Dependencies

**Config** (`/internal/config/config.go`):
- Environment-based: `S3_BUCKET_NAME`, `AWS_REGION`, `PORT`, `PRESIGN_EXPIRY`, `S3_PATH_STYLE`, `MAX_UPLOAD_SIZE`
- No path-related configuration
- Supports split endpoints for internal vs. external S3 URLs

**Dependencies:**
- **Echo v4.15.0** - HTTP framework (uses path parameters)
- **AWS SDK v2** - S3 client
- **testify** - Mocking and assertions

### 7. What Would Need to Change for Arbitrary Path Segments

To support dynamic path segments (e.g., `namespace/org/division/name/provider/version` or `namespace/name/provider/version/variant`):

1. **Echo routing** - Change from `:namespace/:name/:provider/:version` to catch-all syntax or change module structure entirely

2. **Storage Client interface** - Replace individual string parameters with a slice or struct:
   ```go
   // Option A: Pass path segments as slice
   ListVersions(ctx context.Context, pathSegments []string) ([]string, error)

   // Option B: Create a ModulePath type
   type ModulePath struct {
       Segments []string
   }
   ListVersions(ctx context.Context, path ModulePath) ([]string, error)
   ```

3. **S3 key building** - Make `objectKey()` and `prefix()` dynamic:
   ```go
   func objectKey(segments []string) string {
       base := strings.Join(segments[:len(segments)-1], "/")
       version := segments[len(segments)-1]
       return fmt.Sprintf("%s/%s.tar.gz", base, version)
   }
   ```

4. **Handler logic** - Parse path as a slice or custom type instead of individual vars:
   ```go
   // Parse entire path as segments
   path := strings.Split(c.Param("path"), "/")
   namespace := path[0]
   // etc.
   ```

5. **Terraform Module Registry Protocol** - The standard protocol expects exactly 3 path segments before version (namespace/name/provider). Supporting arbitrary segments would require either:
   - Reinterpreting the spec (non-standard)
   - Encoding multi-segment identifiers into single segments (e.g., "namespace-org", "name-division")
   - Operating a different module discovery endpoint

6. **API endpoint definitions** - Currently deeply tied to the Terraform spec. Would need redesign if moving away from standard protocol.

### Key Files Summary

- **Entry point**: `/main.go` - Creates Echo server, sets up storage
- **HTTP handlers & routing**: `/internal/registry/handler.go` - All 4 endpoints, path extraction
- **Storage interface**: `/internal/storage/storage.go` - Defines Client contract
- **S3 implementation**: `/internal/storage/s3.go` - Path building (`objectKey`, `prefix`, `extractVersion`)
- **Tests**: `/internal/registry/handler_test.go` and `/internal/storage/s3_test.go` - Comprehensive path parsing examples
- **Mock**: `/internal/storage/mock_storage.go` - For testing

The architecture is clean and follows good Go patterns (interface-based storage, testable handlers), but the 4-segment path model is baked into the HTTP routing layer and would require significant refactoring to support arbitrary path depth.

Prompt:
Explore the Go project at /Users/steve.briggs/projects/operations/services/crow. This is called "Crow" and is a private Terraform registry/repository service. I need to understand:

1. The overall project structure (directories, key files)
2. How terraform modules are handled - specifically how module paths/versions are defined and stored
3. Any structs or types that define module paths (looking for anything that might have a fixed number of path segments like path1, path2, path3 rather than a dynamic slice)
4. The API endpoints for module operations
5. How modules are identified (namespace, name, provider, version, etc.)
6. Database models or storage layer for modules
7. Any path parsing or routing logic

Be thorough - read key source files to understand the architecture. Report: file tree summary, key types/structs for modules, how paths work currently, and what would need to change to support arbitrary numbers of path segments.


Let me read the key source files before planning the changes.


1	package registry
2
3	import (
4		"bytes"
5		"errors"
6		"io"
7		"net/http"
8		"sync/atomic"
9		"time"
10
11		"github.com/immuta/operations/services/crow/internal/storage"
12		"github.com/labstack/echo/v4"
13	)
14
15	// gzipMagic is the two-byte header for gzip files.
16	var gzipMagic = []byte{0x1f, 0x8b}
17
18	type Handler struct {
19		store         storage.Client
20		presignExpiry time.Duration
21		maxUploadSize int64
22		rwPassed      atomic.Bool
23	}
24
25	func NewHandler(store storage.Client, presignExpiry time.Duration, maxUploadSize int64) *Handler {
26		return &Handler{store: store, presignExpiry: presignExpiry, maxUploadSize: maxUploadSize}
27	}
28
29	func (h *Handler) Register(e *echo.Echo) {
30		e.GET("/.well-known/terraform.json", h.ServiceDiscovery)
31		e.GET("/v1/modules/:namespace/:name/:provider/versions", h.ListVersions)
32		e.GET("/v1/modules/:namespace/:name/:provider/:version/download", h.Download)
33		e.PUT("/v1/modules/:namespace/:name/:provider/:version", h.Upload)
34		e.GET("/health", h.Health)
35		e.GET("/livez", h.Livez)
36		e.GET("/readyz", h.Readyz)
37	}
38
39	func (*Handler) Health(c echo.Context) error {
40		return c.NoContent(http.StatusOK)
41	}
42
43	func (*Handler) Livez(c echo.Context) error {
44		return c.NoContent(http.StatusOK)
45	}
46
47	func (h *Handler) Readyz(c echo.Context) error {
48		ctx := c.Request().Context()
49
50		if err := h.store.HeadBucket(ctx); err != nil {
51			c.Logger().Errorf("readyz: %v", err)
52			return c.JSON(http.StatusServiceUnavailable, map[string]string{"error": "head bucket check failed"})
53		}
54
55		if !h.rwPassed.Load() {
56			if err := h.store.CheckReadWrite(ctx); err != nil {
57				c.Logger().Errorf("readyz: %v", err)
58				return c.JSON(http.StatusServiceUnavailable, map[string]string{"error": "read/write check failed"})
59			}
60			h.rwPassed.Store(true)
61		}
62
63		return c.NoContent(http.StatusOK)
64	}
65
66	func (*Handler) ServiceDiscovery(c echo.Context) error {
67		return c.JSON(http.StatusOK, map[string]string{
68			"modules.v1": "/v1/modules/",
69		})
70	}
71
72	type versionEntry struct {
73		Version string `json:"version"`
74	}
75
76	func (h *Handler) ListVersions(c echo.Context) error {
77		namespace := c.Param("namespace")
78		name := c.Param("name")
79		provider := c.Param("provider")
80
81		versions, err := h.store.ListVersions(c.Request().Context(), namespace, name, provider)
82		if err != nil {
83			c.Logger().Errorf("failed to list versions: %v", err)
84			return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
85		}
86
87		if len(versions) == 0 {
88			return c.JSON(http.StatusNotFound, map[string]string{"error": "module not found"})
89		}
90
91		entries := make([]versionEntry, len(versions))
92		for i, v := range versions {
93			entries[i] = versionEntry{Version: v}
94		}
95
96		resp := map[string]any{
97			"modules": []map[string]any{
98				{"versions": entries},
99			},
100		}
101
102		return c.JSON(http.StatusOK, resp)
103	}
104
105	func (h *Handler) Download(c echo.Context) error {
106		namespace := c.Param("namespace")
107		name := c.Param("name")
108		provider := c.Param("provider")
109		version := c.Param("version")
110
111		ctx := c.Request().Context()
112
113		exists, err := h.store.HeadModule(ctx, namespace, name, provider, version)
114		if err != nil {
115			c.Logger().Errorf("failed to check module existence: %v", err)
116			return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
117		}
118
119		if !exists {
120			return c.JSON(http.StatusNotFound, map[string]string{"error": "module version not found"})
121		}
122
123		url, err := h.store.PresignGetModule(ctx, namespace, name, provider, version, h.presignExpiry)
124		if err != nil {
125			c.Logger().Errorf("failed to generate presigned URL: %v", err)
126			return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
127		}
128
129		c.Response().Header().Set("X-Terraform-Get", url)
130		return c.NoContent(http.StatusNoContent)
131	}
132
133	// Upload stores a module archive in S3. The existence check and write are not
134	// atomic (TOCTOU): a concurrent upload between HeadModule and PutModule could
135	// result in a silent overwrite because S3 PutObject is last-write-wins.
136	// Conditional writes (If-None-Match) are not universally supported across
137	// S3-compatible backends, so we accept this trade-off.
138	func (h *Handler) Upload(c echo.Context) error {
139		namespace := c.Param("namespace")
140		name := c.Param("name")
141		provider := c.Param("provider")
142		version := c.Param("version")
143
144		ctx := c.Request().Context()
145
146		exists, err := h.store.HeadModule(ctx, namespace, name, provider, version)
147		if err != nil {
148			c.Logger().Errorf("failed to check module existence: %v", err)
149			return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
150		}
151		if exists {
152			return c.JSON(http.StatusConflict, map[string]string{"error": "module version already exists"})
153		}
154
155		body := http.MaxBytesReader(c.Response(), c.Request().Body, h.maxUploadSize)
156		defer body.Close()
157
158		header := make([]byte, len(gzipMagic))
159		if _, err := io.ReadFull(body, header); err != nil {
160			return c.JSON(http.StatusBadRequest, map[string]string{"error": "request body is too small or empty"})
161		}
162		if header[0] != gzipMagic[0] || header[1] != gzipMagic[1] {
163			return c.JSON(http.StatusBadRequest, map[string]string{"error": "request body is not a valid gzip archive"})
164		}
165
166		// Buffer the full body so S3 PutObject gets a seekable reader, which the
167		// AWS SDK requires for checksum computation over non-TLS connections.
168		// This is bounded by MaxBytesReader so memory usage is capped at maxUploadSize.
169		rest, err := io.ReadAll(body)
170		if err != nil {
171			var maxBytesErr *http.MaxBytesError
172			if errors.As(err, &maxBytesErr) {
173				return c.JSON(http.StatusBadRequest, map[string]string{"error": "request body exceeds maximum upload size"})
174			}
175			return c.JSON(http.StatusBadRequest, map[string]string{"error": "failed to read request body"})
176		}
177		full := append(header, rest...)
178		if err := h.store.PutModule(ctx, namespace, name, provider, version, bytes.NewReader(full)); err != nil {
179			c.Logger().Errorf("failed to store module: %v", err)
180			return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
181		}
182
183		return c.NoContent(http.StatusCreated)
184	}
185

package registry

import (
	"bytes"
	"errors"
	"io"
	"net/http"
	"sync/atomic"
	"time"

	"github.com/immuta/operations/services/crow/internal/storage"
	"github.com/labstack/echo/v4"
)

// gzipMagic is the two-byte header for gzip files.
var gzipMagic = []byte{0x1f, 0x8b}

type Handler struct {
	store         storage.Client
	presignExpiry time.Duration
	maxUploadSize int64
	rwPassed      atomic.Bool
}

func NewHandler(store storage.Client, presignExpiry time.Duration, maxUploadSize int64) *Handler {
	return &Handler{store: store, presignExpiry: presignExpiry, maxUploadSize: maxUploadSize}
}

func (h *Handler) Register(e *echo.Echo) {
	e.GET("/.well-known/terraform.json", h.ServiceDiscovery)
	e.GET("/v1/modules/:namespace/:name/:provider/versions", h.ListVersions)
	e.GET("/v1/modules/:namespace/:name/:provider/:version/download", h.Download)
	e.PUT("/v1/modules/:namespace/:name/:provider/:version", h.Upload)
	e.GET("/health", h.Health)
	e.GET("/livez", h.Livez)
	e.GET("/readyz", h.Readyz)
}

func (*Handler) Health(c echo.Context) error {
	return c.NoContent(http.StatusOK)
}

func (*Handler) Livez(c echo.Context) error {
	return c.NoContent(http.StatusOK)
}

func (h *Handler) Readyz(c echo.Context) error {
	ctx := c.Request().Context()

	if err := h.store.HeadBucket(ctx); err != nil {
		c.Logger().Errorf("readyz: %v", err)
		return c.JSON(http.StatusServiceUnavailable, map[string]string{"error": "head bucket check failed"})
	}

	if !h.rwPassed.Load() {
		if err := h.store.CheckReadWrite(ctx); err != nil {
			c.Logger().Errorf("readyz: %v", err)
			return c.JSON(http.StatusServiceUnavailable, map[string]string{"error": "read/write check failed"})
		}
		h.rwPassed.Store(true)
	}

	return c.NoContent(http.StatusOK)
}

func (*Handler) ServiceDiscovery(c echo.Context) error {
	return c.JSON(http.StatusOK, map[string]string{
		"modules.v1": "/v1/modules/",
	})
}

type versionEntry struct {
	Version string `json:"version"`
}

func (h *Handler) ListVersions(c echo.Context) error {
	namespace := c.Param("namespace")
	name := c.Param("name")
	provider := c.Param("provider")

	versions, err := h.store.ListVersions(c.Request().Context(), namespace, name, provider)
	if err != nil {
		c.Logger().Errorf("failed to list versions: %v", err)
		return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
	}

	if len(versions) == 0 {
		return c.JSON(http.StatusNotFound, map[string]string{"error": "module not found"})
	}

	entries := make([]versionEntry, len(versions))
	for i, v := range versions {
		entries[i] = versionEntry{Version: v}
	}

	resp := map[string]any{
		"modules": []map[string]any{
			{"versions": entries},
		},
	}

	return c.JSON(http.StatusOK, resp)
}

func (h *Handler) Download(c echo.Context) error {
	namespace := c.Param("namespace")
	name := c.Param("name")
	provider := c.Param("provider")
	version := c.Param("version")

	ctx := c.Request().Context()

	exists, err := h.store.HeadModule(ctx, namespace, name, provider, version)
	if err != nil {
		c.Logger().Errorf("failed to check module existence: %v", err)
		return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
	}

	if !exists {
		return c.JSON(http.StatusNotFound, map[string]string{"error": "module version not found"})
	}

	url, err := h.store.PresignGetModule(ctx, namespace, name, provider, version, h.presignExpiry)
	if err != nil {
		c.Logger().Errorf("failed to generate presigned URL: %v", err)
		return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
	}

	c.Response().Header().Set("X-Terraform-Get", url)
	return c.NoContent(http.StatusNoContent)
}

// Upload stores a module archive in S3. The existence check and write are not
// atomic (TOCTOU): a concurrent upload between HeadModule and PutModule could
// result in a silent overwrite because S3 PutObject is last-write-wins.
// Conditional writes (If-None-Match) are not universally supported across
// S3-compatible backends, so we accept this trade-off.
func (h *Handler) Upload(c echo.Context) error {
	namespace := c.Param("namespace")
	name := c.Param("name")
	provider := c.Param("provider")
	version := c.Param("version")

	ctx := c.Request().Context()

	exists, err := h.store.HeadModule(ctx, namespace, name, provider, version)
	if err != nil {
		c.Logger().Errorf("failed to check module existence: %v", err)
		return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
	}
	if exists {
		return c.JSON(http.StatusConflict, map[string]string{"error": "module version already exists"})
	}

	body := http.MaxBytesReader(c.Response(), c.Request().Body, h.maxUploadSize)
	defer body.Close()

	header := make([]byte, len(gzipMagic))
	if _, err := io.ReadFull(body, header); err != nil {
		return c.JSON(http.StatusBadRequest, map[string]string{"error": "request body is too small or empty"})
	}
	if header[0] != gzipMagic[0] || header[1] != gzipMagic[1] {
		return c.JSON(http.StatusBadRequest, map[string]string{"error": "request body is not a valid gzip archive"})
	}

	// Buffer the full body so S3 PutObject gets a seekable reader, which the
	// AWS SDK requires for checksum computation over non-TLS connections.
	// This is bounded by MaxBytesReader so memory usage is capped at maxUploadSize.
	rest, err := io.ReadAll(body)
	if err != nil {
		var maxBytesErr *http.MaxBytesError
		if errors.As(err, &maxBytesErr) {
			return c.JSON(http.StatusBadRequest, map[string]string{"error": "request body exceeds maximum upload size"})
		}
		return c.JSON(http.StatusBadRequest, map[string]string{"error": "failed to read request body"})
	}
	full := append(header, rest...)
	if err := h.store.PutModule(ctx, namespace, name, provider, version, bytes.NewReader(full)); err != nil {
		c.Logger().Errorf("failed to store module: %v", err)
		return c.JSON(http.StatusInternalServerError, map[string]string{"error": "internal server error"})
	}

	return c.NoContent(http.StatusCreated)
}


1	package storage
2
3	import (
4		"context"
5		"errors"
6		"fmt"
7		"io"
8		"strings"
9		"time"
10
11		"github.com/aws/aws-sdk-go-v2/aws"
12		"github.com/aws/aws-sdk-go-v2/service/s3"
13		"github.com/aws/aws-sdk-go-v2/service/s3/types"
14	)
15
16	type S3Client struct {
17		client     *s3.Client
18		presigner  *s3.PresignClient
19		bucketName string
20	}
21
22	func NewS3Client(awsCfg aws.Config, bucketName string, optFns ...func(*s3.Options)) *S3Client {
23		client := s3.NewFromConfig(awsCfg, optFns...)
24		return &S3Client{
25			client:     client,
26			presigner:  s3.NewPresignClient(client),
27			bucketName: bucketName,
28		}
29	}
30
31	// NewS3ClientWithPresignEndpoint creates an S3Client that uses a separate
32	// endpoint for generating presigned URLs. This is needed when the S3 backend
33	// is reachable internally at one address (e.g. http://rustfs:9000) but
34	// clients need presigned URLs pointing to an external address (e.g.
35	// http://localhost:9000).
36	func NewS3ClientWithPresignEndpoint(awsCfg aws.Config, bucketName, presignEndpoint string, optFns ...func(*s3.Options)) *S3Client {
37		client := s3.NewFromConfig(awsCfg, optFns...)
38		presignClient := s3.NewFromConfig(awsCfg, append(optFns, func(o *s3.Options) {
39			o.BaseEndpoint = &presignEndpoint
40		})...)
41		return &S3Client{
42			client:     client,
43			presigner:  s3.NewPresignClient(presignClient),
44			bucketName: bucketName,
45		}
46	}
47
48	func objectKey(namespace, name, provider, version string) string {
49		return fmt.Sprintf("%s/%s/%s/%s.tar.gz", namespace, name, provider, version)
50	}
51
52	func prefix(namespace, name, provider string) string {
53		return fmt.Sprintf("%s/%s/%s/", namespace, name, provider)
54	}
55
56	func extractVersion(key, pfx string) (string, bool) {
57		if !strings.HasPrefix(key, pfx) {
58			return "", false
59		}
60		rest := strings.TrimPrefix(key, pfx)
61		if !strings.HasSuffix(rest, ".tar.gz") {
62			return "", false
63		}
64		version := strings.TrimSuffix(rest, ".tar.gz")
65		if version == "" {
66			return "", false
67		}
68		return version, true
69	}
70
71	func (s *S3Client) ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error) {
72		pfx := prefix(namespace, name, provider)
73		input := &s3.ListObjectsV2Input{
74			Bucket: &s.bucketName,
75			Prefix: &pfx,
76		}
77
78		var versions []string
79		paginator := s3.NewListObjectsV2Paginator(s.client, input)
80		for paginator.HasMorePages() {
81			output, err := paginator.NextPage(ctx)
82			if err != nil {
83				return nil, fmt.Errorf("listing objects: %w", err)
84			}
85			for _, obj := range output.Contents {
86				if obj.Key == nil {
87					return nil, errors.New("listing objects: object key is nil")
88				}
89				if v, ok := extractVersion(*obj.Key, pfx); ok {
90					versions = append(versions, v)
91				}
92			}
93		}
94
95		return versions, nil
96	}
97
98	func (s *S3Client) HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error) {
99		key := objectKey(namespace, name, provider, version)
100		_, err := s.client.HeadObject(ctx, &s3.HeadObjectInput{
101			Bucket: &s.bucketName,
102			Key:    &key,
103		})
104		if err != nil {
105			var notFound *types.NotFound
106			var noSuchKey *types.NoSuchKey
107			if errors.As(err, &notFound) || errors.As(err, &noSuchKey) {
108				return false, nil
109			}
110			return false, fmt.Errorf("head object: %w", err)
111		}
112		return true, nil
113	}
114
115	func (s *S3Client) PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error {
116		key := objectKey(namespace, name, provider, version)
117		contentType := "application/gzip"
118		_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
119			Bucket:      &s.bucketName,
120			Key:         &key,
121			Body:        body,
122			ContentType: &contentType,
123		})
124		if err != nil {
125			return fmt.Errorf("put object: %w", err)
126		}
127		return nil
128	}
129
130	func (s *S3Client) PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error) {
131		key := objectKey(namespace, name, provider, version)
132		req, err := s.presigner.PresignGetObject(ctx, &s3.GetObjectInput{
133			Bucket: &s.bucketName,
134			Key:    &key,
135		}, s3.WithPresignExpires(expiry))
136		if err != nil {
137			return "", fmt.Errorf("presigning URL: %w", err)
138		}
139		return req.URL, nil
140	}
141
142	const readyzKey = ".readyz"
143
144	func (s *S3Client) HeadBucket(ctx context.Context) error {
145		_, err := s.client.HeadBucket(ctx, &s3.HeadBucketInput{
146			Bucket: &s.bucketName,
147		})
148		if err != nil {
149			return fmt.Errorf("head bucket: %w", err)
150		}
151		return nil
152	}
153
154	func (s *S3Client) CheckReadWrite(ctx context.Context) error {
155		body := strings.NewReader(time.Now().UTC().Format(time.RFC3339))
156		_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
157			Bucket: &s.bucketName,
158			Key:    aws.String(readyzKey),
159			Body:   body,
160		})
161		if err != nil {
162			return fmt.Errorf("readyz put object: %w", err)
163		}
164
165		_, err = s.client.HeadObject(ctx, &s3.HeadObjectInput{
166			Bucket: &s.bucketName,
167			Key:    aws.String(readyzKey),
168		})
169		if err != nil {
170			return fmt.Errorf("readyz head object: %w", err)
171		}
172
173		return nil
174	}
175

package storage

import (
	"context"
	"errors"
	"fmt"
	"io"
	"strings"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/service/s3/types"
)

type S3Client struct {
	client     *s3.Client
	presigner  *s3.PresignClient
	bucketName string
}

func NewS3Client(awsCfg aws.Config, bucketName string, optFns ...func(*s3.Options)) *S3Client {
	client := s3.NewFromConfig(awsCfg, optFns...)
	return &S3Client{
		client:     client,
		presigner:  s3.NewPresignClient(client),
		bucketName: bucketName,
	}
}

// NewS3ClientWithPresignEndpoint creates an S3Client that uses a separate
// endpoint for generating presigned URLs. This is needed when the S3 backend
// is reachable internally at one address (e.g. http://rustfs:9000) but
// clients need presigned URLs pointing to an external address (e.g.
// http://localhost:9000).
func NewS3ClientWithPresignEndpoint(awsCfg aws.Config, bucketName, presignEndpoint string, optFns ...func(*s3.Options)) *S3Client {
	client := s3.NewFromConfig(awsCfg, optFns...)
	presignClient := s3.NewFromConfig(awsCfg, append(optFns, func(o *s3.Options) {
		o.BaseEndpoint = &presignEndpoint
	})...)
	return &S3Client{
		client:     client,
		presigner:  s3.NewPresignClient(presignClient),
		bucketName: bucketName,
	}
}

func objectKey(namespace, name, provider, version string) string {
	return fmt.Sprintf("%s/%s/%s/%s.tar.gz", namespace, name, provider, version)
}

func prefix(namespace, name, provider string) string {
	return fmt.Sprintf("%s/%s/%s/", namespace, name, provider)
}

func extractVersion(key, pfx string) (string, bool) {
	if !strings.HasPrefix(key, pfx) {
		return "", false
	}
	rest := strings.TrimPrefix(key, pfx)
	if !strings.HasSuffix(rest, ".tar.gz") {
		return "", false
	}
	version := strings.TrimSuffix(rest, ".tar.gz")
	if version == "" {
		return "", false
	}
	return version, true
}

func (s *S3Client) ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error) {
	pfx := prefix(namespace, name, provider)
	input := &s3.ListObjectsV2Input{
		Bucket: &s.bucketName,
		Prefix: &pfx,
	}

	var versions []string
	paginator := s3.NewListObjectsV2Paginator(s.client, input)
	for paginator.HasMorePages() {
		output, err := paginator.NextPage(ctx)
		if err != nil {
			return nil, fmt.Errorf("listing objects: %w", err)
		}
		for _, obj := range output.Contents {
			if obj.Key == nil {
				return nil, errors.New("listing objects: object key is nil")
			}
			if v, ok := extractVersion(*obj.Key, pfx); ok {
				versions = append(versions, v)
			}
		}
	}

	return versions, nil
}

func (s *S3Client) HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error) {
	key := objectKey(namespace, name, provider, version)
	_, err := s.client.HeadObject(ctx, &s3.HeadObjectInput{
		Bucket: &s.bucketName,
		Key:    &key,
	})
	if err != nil {
		var notFound *types.NotFound
		var noSuchKey *types.NoSuchKey
		if errors.As(err, &notFound) || errors.As(err, &noSuchKey) {
			return false, nil
		}
		return false, fmt.Errorf("head object: %w", err)
	}
	return true, nil
}

func (s *S3Client) PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error {
	key := objectKey(namespace, name, provider, version)
	contentType := "application/gzip"
	_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
		Bucket:      &s.bucketName,
		Key:         &key,
		Body:        body,
		ContentType: &contentType,
	})
	if err != nil {
		return fmt.Errorf("put object: %w", err)
	}
	return nil
}

func (s *S3Client) PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error) {
	key := objectKey(namespace, name, provider, version)
	req, err := s.presigner.PresignGetObject(ctx, &s3.GetObjectInput{
		Bucket: &s.bucketName,
		Key:    &key,
	}, s3.WithPresignExpires(expiry))
	if err != nil {
		return "", fmt.Errorf("presigning URL: %w", err)
	}
	return req.URL, nil
}

const readyzKey = ".readyz"

func (s *S3Client) HeadBucket(ctx context.Context) error {
	_, err := s.client.HeadBucket(ctx, &s3.HeadBucketInput{
		Bucket: &s.bucketName,
	})
	if err != nil {
		return fmt.Errorf("head bucket: %w", err)
	}
	return nil
}

func (s *S3Client) CheckReadWrite(ctx context.Context) error {
	body := strings.NewReader(time.Now().UTC().Format(time.RFC3339))
	_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
		Bucket: &s.bucketName,
		Key:    aws.String(readyzKey),
		Body:   body,
	})
	if err != nil {
		return fmt.Errorf("readyz put object: %w", err)
	}

	_, err = s.client.HeadObject(ctx, &s3.HeadObjectInput{
		Bucket: &s.bucketName,
		Key:    aws.String(readyzKey),
	})
	if err != nil {
		return fmt.Errorf("readyz head object: %w", err)
	}

	return nil
}


1	package storage
2
3	import (
4		"context"
5		"io"
6		"time"
7	)
8
9	type Client interface {
10		ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error)
11		HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error)
12		PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error)
13		PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error
14		HeadBucket(ctx context.Context) error
15		CheckReadWrite(ctx context.Context) error
16	}
17

package storage

import (
	"context"
	"io"
	"time"
)

type Client interface {
	ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error)
	HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error)
	PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error)
	PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error
	HeadBucket(ctx context.Context) error
	CheckReadWrite(ctx context.Context) error
}


Analyze the project in this directory, it is called Crow, a golang private terraform repository. I need to update this service to accept terraform modules with any given number of paths instead of lim…


Now let me read the test files to understand the testing patterns.


1	package registry
2
3	import (
4		"bytes"
5		"compress/gzip"
6		"net/http"
7		"net/http/httptest"
8		"strings"
9		"testing"
10		"time"
11
12		"github.com/immuta/operations/services/crow/internal/storage"
13		"github.com/labstack/echo/v4"
14		"github.com/stretchr/testify/assert"
15		"github.com/stretchr/testify/mock"
16	)
17
18	const testMaxUploadSize = 10 * 1024 * 1024 // 10MB
19
20	func setupTest(t *testing.T) (*echo.Echo, *storage.MockClient) {
21		t.Helper()
22		e := echo.New()
23		mockStore := new(storage.MockClient)
24		h := &Handler{store: mockStore, presignExpiry: 15 * time.Minute, maxUploadSize: testMaxUploadSize}
25		h.Register(e)
26		return e, mockStore
27	}
28
29	func TestHealth(t *testing.T) {
30		e, _ := setupTest(t)
31
32		req := httptest.NewRequest(http.MethodGet, "/health", nil)
33		rec := httptest.NewRecorder()
34		e.ServeHTTP(rec, req)
35
36		assert.Equal(t, http.StatusOK, rec.Code)
37	}
38
39	func TestServiceDiscovery(t *testing.T) {
40		e, _ := setupTest(t)
41
42		req := httptest.NewRequest(http.MethodGet, "/.well-known/terraform.json", nil)
43		rec := httptest.NewRecorder()
44		e.ServeHTTP(rec, req)
45
46		assert.Equal(t, http.StatusOK, rec.Code)
47		assert.JSONEq(t, `{"modules.v1": "/v1/modules/"}`, rec.Body.String())
48	}
49
50	func TestListVersions(t *testing.T) {
51		tests := []struct {
52			name           string
53			namespace      string
54			modName        string
55			provider       string
56			versions       []string
57			storeErr       error
58			expectedStatus int
59			expectedBody   string
60		}{
61			{
62				name:           "returns versions",
63				namespace:      "hashicorp",
64				modName:        "consul",
65				provider:       "aws",
66				versions:       []string{"1.0.0", "1.0.1"},
67				expectedStatus: http.StatusOK,
68				expectedBody:   `{"modules":[{"versions":[{"version":"1.0.0"},{"version":"1.0.1"}]}]}`,
69			},
70			{
71				name:           "returns 404 when no versions found",
72				namespace:      "hashicorp",
73				modName:        "consul",
74				provider:       "aws",
75				versions:       []string{},
76				expectedStatus: http.StatusNotFound,
77			},
78			{
79				name:           "returns 500 on store error",
80				namespace:      "hashicorp",
81				modName:        "consul",
82				provider:       "aws",
83				storeErr:       assert.AnError,
84				expectedStatus: http.StatusInternalServerError,
85			},
86		}
87
88		for _, tt := range tests {
89			t.Run(tt.name, func(t *testing.T) {
90				e, mockStore := setupTest(t)
91
92				mockStore.On("ListVersions", mock.Anything, tt.namespace, tt.modName, tt.provider).
93					Return(tt.versions, tt.storeErr)
94
95				path := "/v1/modules/" + tt.namespace + "/" + tt.modName + "/" + tt.provider + "/versions"
96				req := httptest.NewRequest(http.MethodGet, path, nil)
97				rec := httptest.NewRecorder()
98				e.ServeHTTP(rec, req)
99
100				assert.Equal(t, tt.expectedStatus, rec.Code)
101				if tt.expectedBody != "" {
102					assert.JSONEq(t, tt.expectedBody, rec.Body.String())
103				}
104			})
105		}
106	}
107
108	func TestDownload(t *testing.T) {
109		tests := []struct {
110			name           string
111			namespace      string
112			modName        string
113			provider       string
114			version        string
115			exists         bool
116			headErr        error
117			presignURL     string
118			presignErr     error
119			expectedStatus int
120			expectedHeader string
121		}{
122			{
123				name:           "returns presigned URL",
124				namespace:      "hashicorp",
125				modName:        "consul",
126				provider:       "aws",
127				version:        "1.0.0",
128				exists:         true,
129				presignURL:     "https://s3.amazonaws.com/bucket/hashicorp/consul/aws/1.0.0.tar.gz?signed",
130				expectedStatus: http.StatusNoContent,
131				expectedHeader: "https://s3.amazonaws.com/bucket/hashicorp/consul/aws/1.0.0.tar.gz?signed",
132			},
133			{
134				name:           "returns 404 when module does not exist",
135				namespace:      "hashicorp",
136				modName:        "consul",
137				provider:       "aws",
138				version:        "9.9.9",
139				exists:         false,
140				expectedStatus: http.StatusNotFound,
141			},
142			{
143				name:           "returns 500 on head error",
144				namespace:      "hashicorp",
145				modName:        "consul",
146				provider:       "aws",
147				version:        "1.0.0",
148				headErr:        assert.AnError,
149				expectedStatus: http.StatusInternalServerError,
150			},
151			{
152				name:           "returns 500 on presign error",
153				namespace:      "hashicorp",
154				modName:        "consul",
155				provider:       "aws",
156				version:        "1.0.0",
157				exists:         true,
158				presignErr:     assert.AnError,
159				expectedStatus: http.StatusInternalServerError,
160			},
161		}
162
163		for _, tt := range tests {
164			t.Run(tt.name, func(t *testing.T) {
165				e, mockStore := setupTest(t)
166
167				mockStore.On("HeadModule", mock.Anything, tt.namespace, tt.modName, tt.provider, tt.version).
168					Return(tt.exists, tt.headErr)
169
170				if tt.exists && tt.headErr == nil {
171					mockStore.On("PresignGetModule", mock.Anything, tt.namespace, tt.modName, tt.provider, tt.version, mock.Anything).
172						Return(tt.presignURL, tt.presignErr)
173				}
174
175				path := "/v1/modules/" + tt.namespace + "/" + tt.modName + "/" + tt.provider + "/" + tt.version + "/download"
176				req := httptest.NewRequest(http.MethodGet, path, nil)
177				rec := httptest.NewRecorder()
178				e.ServeHTTP(rec, req)
179
180				assert.Equal(t, tt.expectedStatus, rec.Code)
181				if tt.expectedHeader != "" {
182					assert.Equal(t, tt.expectedHeader, rec.Header().Get("X-Terraform-Get"))
183				}
184			})
185		}
186	}
187
188	func validGzipBody(t *testing.T) *bytes.Buffer {
189		t.Helper()
190		var buf bytes.Buffer
191		gw := gzip.NewWriter(&buf)
192		_, err := gw.Write([]byte("hello"))
193		assert.NoError(t, err)
194		assert.NoError(t, gw.Close())
195		return &buf
196	}
197
198	func TestUpload(t *testing.T) {
199		tests := []struct {
200			name           string
201			namespace      string
202			modName        string
203			provider       string
204			version        string
205			body           func(t *testing.T) *bytes.Buffer
206			exists         bool
207			headErr        error
208			putErr         error
209			expectedStatus int
210		}{
211			{
212				name:           "upload succeeds",
213				namespace:      "hashicorp",
214				modName:        "consul",
215				provider:       "aws",
216				version:        "1.0.0",
217				body:           validGzipBody,
218				exists:         false,
219				expectedStatus: http.StatusCreated,
220			},
221			{
222				name:           "rejects duplicate version",
223				namespace:      "hashicorp",
224				modName:        "consul",
225				provider:       "aws",
226				version:        "1.0.0",
227				body:           validGzipBody,
228				exists:         true,
229				expectedStatus: http.StatusConflict,
230			},
231			{
232				name:      "rejects invalid gzip",
233				namespace: "hashicorp",
234				modName:   "consul",
235				provider:  "aws",
236				version:   "1.0.0",
237				body: func(t *testing.T) *bytes.Buffer {
238					t.Helper()
239					return bytes.NewBufferString("not a gzip file")
240				},
241				exists:         false,
242				expectedStatus: http.StatusBadRequest,
243			},
244			{
245				name:      "rejects empty body",
246				namespace: "hashicorp",
247				modName:   "consul",
248				provider:  "aws",
249				version:   "1.0.0",
250				body: func(t *testing.T) *bytes.Buffer {
251					t.Helper()
252					return bytes.NewBufferString("")
253				},
254				exists:         false,
255				expectedStatus: http.StatusBadRequest,
256			},
257			{
258				name:           "returns 500 on head error",
259				namespace:      "hashicorp",
260				modName:        "consul",
261				provider:       "aws",
262				version:        "1.0.0",
263				body:           validGzipBody,
264				headErr:        assert.AnError,
265				expectedStatus: http.StatusInternalServerError,
266			},
267			{
268				name:           "returns 500 on put error",
269				namespace:      "hashicorp",
270				modName:        "consul",
271				provider:       "aws",
272				version:        "1.0.0",
273				body:           validGzipBody,
274				exists:         false,
275				putErr:         assert.AnError,
276				expectedStatus: http.StatusInternalServerError,
277			},
278		}
279
280		for _, tt := range tests {
281			t.Run(tt.name, func(t *testing.T) {
282				e, mockStore := setupTest(t)
283
284				mockStore.On("HeadModule", mock.Anything, tt.namespace, tt.modName, tt.provider, tt.version).
285					Return(tt.exists, tt.headErr).Maybe()
286
287				if !tt.exists && tt.headErr == nil {
288					mockStore.On("PutModule", mock.Anything, tt.namespace, tt.modName, tt.provider, tt.version, mock.Anything).
289						Return(tt.putErr).Maybe()
290				}
291
292				body := tt.body(t)
293				path := "/v1/modules/" + tt.namespace + "/" + tt.modName + "/" + tt.provider + "/" + tt.version
294				req := httptest.NewRequest(http.MethodPut, path, body)
295				req.Header.Set("Content-Type", "application/octet-stream")
296				rec := httptest.NewRecorder()
297				e.ServeHTTP(rec, req)
298
299				assert.Equal(t, tt.expectedStatus, rec.Code)
300			})
301		}
302	}
303
304	func TestLivez(t *testing.T) {
305		e, _ := setupTest(t)
306
307		req := httptest.NewRequest(http.MethodGet, "/livez", nil)
308		rec := httptest.NewRecorder()
309		e.ServeHTTP(rec, req)
310
311		assert.Equal(t, http.StatusOK, rec.Code)
312	}
313
314	func TestReadyz(t *testing.T) {
315		tests := []struct {
316			name           string
317			headBucketErr  error
318			checkRWErr     error
319			expectedStatus int
320			expectedBody   string
321		}{
322			{
323				name:           "returns 200 when all checks pass",
324				expectedStatus: http.StatusOK,
325			},
326			{
327				name:           "returns 503 when HeadBucket fails",
328				headBucketErr:  assert.AnError,
329				expectedStatus: http.StatusServiceUnavailable,
330				expectedBody:   `{"error":"head bucket check failed"}`,
331			},
332			{
333				name:           "returns 503 when CheckReadWrite fails",
334				checkRWErr:     assert.AnError,
335				expectedStatus: http.StatusServiceUnavailable,
336				expectedBody:   `{"error":"read/write check failed"}`,
337			},
338		}
339
340		for _, tt := range tests {
341			t.Run(tt.name, func(t *testing.T) {
342				e, mockStore := setupTest(t)
343
344				mockStore.On("HeadBucket", mock.Anything).
345					Return(tt.headBucketErr)
346
347				if tt.headBucketErr == nil {
348					mockStore.On("CheckReadWrite", mock.Anything).
349						Return(tt.checkRWErr)
350				}
351
352				req := httptest.NewRequest(http.MethodGet, "/readyz", nil)
353				rec := httptest.NewRecorder()
354				e.ServeHTTP(rec, req)
355
356				assert.Equal(t, tt.expectedStatus, rec.Code)
357				if tt.expectedBody != "" {
358					assert.JSONEq(t, tt.expectedBody, rec.Body.String())
359				}
360			})
361		}
362	}
363
364	func TestReadyzCachesCheckReadWrite(t *testing.T) {
365		e := echo.New()
366		mockStore := new(storage.MockClient)
367		h := &Handler{store: mockStore, presignExpiry: 15 * time.Minute, maxUploadSize: testMaxUploadSize}
368		h.Register(e)
369
370		mockStore.On("HeadBucket", mock.Anything).Return(nil)
371		mockStore.On("CheckReadWrite", mock.Anything).Return(nil)
372
373		// First request — CheckReadWrite is called
374		req := httptest.NewRequest(http.MethodGet, "/readyz", nil)
375		rec := httptest.NewRecorder()
376		e.ServeHTTP(rec, req)
377		assert.Equal(t, http.StatusOK, rec.Code)
378
379		// Second request — CheckReadWrite should NOT be called again
380		req = httptest.NewRequest(http.MethodGet, "/readyz", nil)
381		rec = httptest.NewRecorder()
382		e.ServeHTTP(rec, req)
383		assert.Equal(t, http.StatusOK, rec.Code)
384
385		mockStore.AssertNumberOfCalls(t, "CheckReadWrite", 1)
386		mockStore.AssertNumberOfCalls(t, "HeadBucket", 2)
387	}
388
389	func TestReadyzRetriesCheckReadWriteOnFailure(t *testing.T) {
390		e := echo.New()
391		mockStore := new(storage.MockClient)
392		h := &Handler{store: mockStore, presignExpiry: 15 * time.Minute, maxUploadSize: testMaxUploadSize}
393		h.Register(e)
394
395		mockStore.On("HeadBucket", mock.Anything).Return(nil)
396		mockStore.On("CheckReadWrite", mock.Anything).Return(assert.AnError).Once()
397		mockStore.On("CheckReadWrite", mock.Anything).Return(nil).Once()
398
399		// First request — CheckReadWrite fails, returns 503
400		req := httptest.NewRequest(http.MethodGet, "/readyz", nil)
401		rec := httptest.NewRecorder()
402		e.ServeHTTP(rec, req)
403		assert.Equal(t, http.StatusServiceUnavailable, rec.Code)
404
405		// Second request — CheckReadWrite retries and succeeds
406		req = httptest.NewRequest(http.MethodGet, "/readyz", nil)
407		rec = httptest.NewRecorder()
408		e.ServeHTTP(rec, req)
409		assert.Equal(t, http.StatusOK, rec.Code)
410
411		// Third request — cached, CheckReadWrite not called again
412		req = httptest.NewRequest(http.MethodGet, "/readyz", nil)
413		rec = httptest.NewRecorder()
414		e.ServeHTTP(rec, req)
415		assert.Equal(t, http.StatusOK, rec.Code)
416
417		mockStore.AssertNumberOfCalls(t, "CheckReadWrite", 2)
418		mockStore.AssertNumberOfCalls(t, "HeadBucket", 3)
419	}
420
421	func TestUploadMaxSize(t *testing.T) {
422		e := echo.New()
423		mockStore := new(storage.MockClient)
424		h := &Handler{store: mockStore, presignExpiry: 15 * time.Minute, maxUploadSize: 10}
425		h.Register(e)
426
427		mockStore.On("HeadModule", mock.Anything, "foo", "bar", "aws", "1.0.0").
428			Return(false, nil)
429
430		// Create a valid gzip that exceeds 10 bytes
431		var buf bytes.Buffer
432		gw := gzip.NewWriter(&buf)
433		_, err := gw.Write([]byte(strings.Repeat("x", 100)))
434		assert.NoError(t, err)
435		assert.NoError(t, gw.Close())
436
437		// Body is buffered before PutModule, so MaxBytesReader triggers during
438		// ReadAll — PutModule should not be called at all.
439		req := httptest.NewRequest(http.MethodPut, "/v1/modules/foo/bar/aws/1.0.0", &buf)
440		req.Header.Set("Content-Type", "application/octet-stream")
441		rec := httptest.NewRecorder()
442		e.ServeHTTP(rec, req)
443
444		assert.Equal(t, http.StatusBadRequest, rec.Code)
445		mockStore.AssertNotCalled(t, "PutModule", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything)
446	}
447

package registry

import (
	"bytes"
	"compress/gzip"
	"net/http"
	"net/http/httptest"
	"strings"
	"testing"
	"time"

	"github.com/immuta/operations/services/crow/internal/storage"
	"github.com/labstack/echo/v4"
	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/mock"
)

const testMaxUploadSize = 10 * 1024 * 1024 // 10MB

func setupTest(t *testing.T) (*echo.Echo, *storage.MockClient) {
	t.Helper()
	e := echo.New()
	mockStore := new(storage.MockClient)
	h := &Handler{store: mockStore, presignExpiry: 15 * time.Minute, maxUploadSize: testMaxUploadSize}
	h.Register(e)
	return e, mockStore
}

func TestHealth(t *testing.T) {
	e, _ := setupTest(t)

	req := httptest.NewRequest(http.MethodGet, "/health", nil)
	rec := httptest.NewRecorder()
	e.ServeHTTP(rec, req)

	assert.Equal(t, http.StatusOK, rec.Code)
}

func TestServiceDiscovery(t *testing.T) {
	e, _ := setupTest(t)

	req := httptest.NewRequest(http.MethodGet, "/.well-known/terraform.json", nil)
	rec := httptest.NewRecorder()
	e.ServeHTTP(rec, req)

	assert.Equal(t, http.StatusOK, rec.Code)
	assert.JSONEq(t, `{"modules.v1": "/v1/modules/"}`, rec.Body.String())
}

func TestListVersions(t *testing.T) {
	tests := []struct {
		name           string
		namespace      string
		modName        string
		provider       string
		versions       []string
		storeErr       error
		expectedStatus int
		expectedBody   string
	}{
		{
			name:           "returns versions",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			versions:       []string{"1.0.0", "1.0.1"},
			expectedStatus: http.StatusOK,
			expectedBody:   `{"modules":[{"versions":[{"version":"1.0.0"},{"version":"1.0.1"}]}]}`,
		},
		{
			name:           "returns 404 when no versions found",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			versions:       []string{},
			expectedStatus: http.StatusNotFound,
		},
		{
			name:           "returns 500 on store error",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			storeErr:       assert.AnError,
			expectedStatus: http.StatusInternalServerError,
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			e, mockStore := setupTest(t)

			mockStore.On("ListVersions", mock.Anything, tt.namespace, tt.modName, tt.provider).
				Return(tt.versions, tt.storeErr)

			path := "/v1/modules/" + tt.namespace + "/" + tt.modName + "/" + tt.provider + "/versions"
			req := httptest.NewRequest(http.MethodGet, path, nil)
			rec := httptest.NewRecorder()
			e.ServeHTTP(rec, req)

			assert.Equal(t, tt.expectedStatus, rec.Code)
			if tt.expectedBody != "" {
				assert.JSONEq(t, tt.expectedBody, rec.Body.String())
			}
		})
	}
}

func TestDownload(t *testing.T) {
	tests := []struct {
		name           string
		namespace      string
		modName        string
		provider       string
		version        string
		exists         bool
		headErr        error
		presignURL     string
		presignErr     error
		expectedStatus int
		expectedHeader string
	}{
		{
			name:           "returns presigned URL",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			version:        "1.0.0",
			exists:         true,
			presignURL:     "https://s3.amazonaws.com/bucket/hashicorp/consul/aws/1.0.0.tar.gz?signed",
			expectedStatus: http.StatusNoContent,
			expectedHeader: "https://s3.amazonaws.com/bucket/hashicorp/consul/aws/1.0.0.tar.gz?signed",
		},
		{
			name:           "returns 404 when module does not exist",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			version:        "9.9.9",
			exists:         false,
			expectedStatus: http.StatusNotFound,
		},
		{
			name:           "returns 500 on head error",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			version:        "1.0.0",
			headErr:        assert.AnError,
			expectedStatus: http.StatusInternalServerError,
		},
		{
			name:           "returns 500 on presign error",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			version:        "1.0.0",
			exists:         true,
			presignErr:     assert.AnError,
			expectedStatus: http.StatusInternalServerError,
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			e, mockStore := setupTest(t)

			mockStore.On("HeadModule", mock.Anything, tt.namespace, tt.modName, tt.provider, tt.version).
				Return(tt.exists, tt.headErr)

			if tt.exists && tt.headErr == nil {
				mockStore.On("PresignGetModule", mock.Anything, tt.namespace, tt.modName, tt.provider, tt.version, mock.Anything).
					Return(tt.presignURL, tt.presignErr)
			}

			path := "/v1/modules/" + tt.namespace + "/" + tt.modName + "/" + tt.provider + "/" + tt.version + "/download"
			req := httptest.NewRequest(http.MethodGet, path, nil)
			rec := httptest.NewRecorder()
			e.ServeHTTP(rec, req)

			assert.Equal(t, tt.expectedStatus, rec.Code)
			if tt.expectedHeader != "" {
				assert.Equal(t, tt.expectedHeader, rec.Header().Get("X-Terraform-Get"))
			}
		})
	}
}

func validGzipBody(t *testing.T) *bytes.Buffer {
	t.Helper()
	var buf bytes.Buffer
	gw := gzip.NewWriter(&buf)
	_, err := gw.Write([]byte("hello"))
	assert.NoError(t, err)
	assert.NoError(t, gw.Close())
	return &buf
}

func TestUpload(t *testing.T) {
	tests := []struct {
		name           string
		namespace      string
		modName        string
		provider       string
		version        string
		body           func(t *testing.T) *bytes.Buffer
		exists         bool
		headErr        error
		putErr         error
		expectedStatus int
	}{
		{
			name:           "upload succeeds",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			version:        "1.0.0",
			body:           validGzipBody,
			exists:         false,
			expectedStatus: http.StatusCreated,
		},
		{
			name:           "rejects duplicate version",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			version:        "1.0.0",
			body:           validGzipBody,
			exists:         true,
			expectedStatus: http.StatusConflict,
		},
		{
			name:      "rejects invalid gzip",
			namespace: "hashicorp",
			modName:   "consul",
			provider:  "aws",
			version:   "1.0.0",
			body: func(t *testing.T) *bytes.Buffer {
				t.Helper()
				return bytes.NewBufferString("not a gzip file")
			},
			exists:         false,
			expectedStatus: http.StatusBadRequest,
		},
		{
			name:      "rejects empty body",
			namespace: "hashicorp",
			modName:   "consul",
			provider:  "aws",
			version:   "1.0.0",
			body: func(t *testing.T) *bytes.Buffer {
				t.Helper()
				return bytes.NewBufferString("")
			},
			exists:         false,
			expectedStatus: http.StatusBadRequest,
		},
		{
			name:           "returns 500 on head error",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			version:        "1.0.0",
			body:           validGzipBody,
			headErr:        assert.AnError,
			expectedStatus: http.StatusInternalServerError,
		},
		{
			name:           "returns 500 on put error",
			namespace:      "hashicorp",
			modName:        "consul",
			provider:       "aws",
			version:        "1.0.0",
			body:           validGzipBody,
			exists:         false,
			putErr:         assert.AnError,
			expectedStatus: http.StatusInternalServerError,
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			e, mockStore := setupTest(t)

			mockStore.On("HeadModule", mock.Anything, tt.namespace, tt.modName, tt.provider, tt.version).
				Return(tt.exists, tt.headErr).Maybe()

			if !tt.exists && tt.headErr == nil {
				mockStore.On("PutModule", mock.Anything, tt.namespace, tt.modName, tt.provider, tt.version, mock.Anything).
					Return(tt.putErr).Maybe()
			}

			body := tt.body(t)
			path := "/v1/modules/" + tt.namespace + "/" + tt.modName + "/" + tt.provider + "/" + tt.version
			req := httptest.NewRequest(http.MethodPut, path, body)
			req.Header.Set("Content-Type", "application/octet-stream")
			rec := httptest.NewRecorder()
			e.ServeHTTP(rec, req)

			assert.Equal(t, tt.expectedStatus, rec.Code)
		})
	}
}

func TestLivez(t *testing.T) {
	e, _ := setupTest(t)

	req := httptest.NewRequest(http.MethodGet, "/livez", nil)
	rec := httptest.NewRecorder()
	e.ServeHTTP(rec, req)

	assert.Equal(t, http.StatusOK, rec.Code)
}

func TestReadyz(t *testing.T) {
	tests := []struct {
		name           string
		headBucketErr  error
		checkRWErr     error
		expectedStatus int
		expectedBody   string
	}{
		{
			name:           "returns 200 when all checks pass",
			expectedStatus: http.StatusOK,
		},
		{
			name:           "returns 503 when HeadBucket fails",
			headBucketErr:  assert.AnError,
			expectedStatus: http.StatusServiceUnavailable,
			expectedBody:   `{"error":"head bucket check failed"}`,
		},
		{
			name:           "returns 503 when CheckReadWrite fails",
			checkRWErr:     assert.AnError,
			expectedStatus: http.StatusServiceUnavailable,
			expectedBody:   `{"error":"read/write check failed"}`,
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			e, mockStore := setupTest(t)

			mockStore.On("HeadBucket", mock.Anything).
				Return(tt.headBucketErr)

			if tt.headBucketErr == nil {
				mockStore.On("CheckReadWrite", mock.Anything).
					Return(tt.checkRWErr)
			}

			req := httptest.NewRequest(http.MethodGet, "/readyz", nil)
			rec := httptest.NewRecorder()
			e.ServeHTTP(rec, req)

			assert.Equal(t, tt.expectedStatus, rec.Code)
			if tt.expectedBody != "" {
				assert.JSONEq(t, tt.expectedBody, rec.Body.String())
			}
		})
	}
}

func TestReadyzCachesCheckReadWrite(t *testing.T) {
	e := echo.New()
	mockStore := new(storage.MockClient)
	h := &Handler{store: mockStore, presignExpiry: 15 * time.Minute, maxUploadSize: testMaxUploadSize}
	h.Register(e)

	mockStore.On("HeadBucket", mock.Anything).Return(nil)
	mockStore.On("CheckReadWrite", mock.Anything).Return(nil)

	// First request — CheckReadWrite is called
	req := httptest.NewRequest(http.MethodGet, "/readyz", nil)
	rec := httptest.NewRecorder()
	e.ServeHTTP(rec, req)
	assert.Equal(t, http.StatusOK, rec.Code)

	// Second request — CheckReadWrite should NOT be called again
	req = httptest.NewRequest(http.MethodGet, "/readyz", nil)
	rec = httptest.NewRecorder()
	e.ServeHTTP(rec, req)
	assert.Equal(t, http.StatusOK, rec.Code)

	mockStore.AssertNumberOfCalls(t, "CheckReadWrite", 1)
	mockStore.AssertNumberOfCalls(t, "HeadBucket", 2)
}

func TestReadyzRetriesCheckReadWriteOnFailure(t *testing.T) {
	e := echo.New()
	mockStore := new(storage.MockClient)
	h := &Handler{store: mockStore, presignExpiry: 15 * time.Minute, maxUploadSize: testMaxUploadSize}
	h.Register(e)

	mockStore.On("HeadBucket", mock.Anything).Return(nil)
	mockStore.On("CheckReadWrite", mock.Anything).Return(assert.AnError).Once()
	mockStore.On("CheckReadWrite", mock.Anything).Return(nil).Once()

	// First request — CheckReadWrite fails, returns 503
	req := httptest.NewRequest(http.MethodGet, "/readyz", nil)
	rec := httptest.NewRecorder()
	e.ServeHTTP(rec, req)
	assert.Equal(t, http.StatusServiceUnavailable, rec.Code)

	// Second request — CheckReadWrite retries and succeeds
	req = httptest.NewRequest(http.MethodGet, "/readyz", nil)
	rec = httptest.NewRecorder()
	e.ServeHTTP(rec, req)
	assert.Equal(t, http.StatusOK, rec.Code)

	// Third request — cached, CheckReadWrite not called again
	req = httptest.NewRequest(http.MethodGet, "/readyz", nil)
	rec = httptest.NewRecorder()
	e.ServeHTTP(rec, req)
	assert.Equal(t, http.StatusOK, rec.Code)

	mockStore.AssertNumberOfCalls(t, "CheckReadWrite", 2)
	mockStore.AssertNumberOfCalls(t, "HeadBucket", 3)
}

func TestUploadMaxSize(t *testing.T) {
	e := echo.New()
	mockStore := new(storage.MockClient)
	h := &Handler{store: mockStore, presignExpiry: 15 * time.Minute, maxUploadSize: 10}
	h.Register(e)

	mockStore.On("HeadModule", mock.Anything, "foo", "bar", "aws", "1.0.0").
		Return(false, nil)

	// Create a valid gzip that exceeds 10 bytes
	var buf bytes.Buffer
	gw := gzip.NewWriter(&buf)
	_, err := gw.Write([]byte(strings.Repeat("x", 100)))
	assert.NoError(t, err)
	assert.NoError(t, gw.Close())

	// Body is buffered before PutModule, so MaxBytesReader triggers during
	// ReadAll — PutModule should not be called at all.
	req := httptest.NewRequest(http.MethodPut, "/v1/modules/foo/bar/aws/1.0.0", &buf)
	req.Header.Set("Content-Type", "application/octet-stream")
	rec := httptest.NewRecorder()
	e.ServeHTTP(rec, req)

	assert.Equal(t, http.StatusBadRequest, rec.Code)
	mockStore.AssertNotCalled(t, "PutModule", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything)
}


1	package storage
2
3	import (
4		"testing"
5
6		"github.com/stretchr/testify/assert"
7	)
8
9	func TestObjectKey(t *testing.T) {
10		key := objectKey("hashicorp", "consul", "aws", "1.0.0")
11		assert.Equal(t, "hashicorp/consul/aws/1.0.0.tar.gz", key)
12	}
13
14	func TestPrefix(t *testing.T) {
15		p := prefix("hashicorp", "consul", "aws")
16		assert.Equal(t, "hashicorp/consul/aws/", p)
17	}
18
19	func TestExtractVersion(t *testing.T) {
20		tests := []struct {
21			key      string
22			prefix   string
23			expected string
24			valid    bool
25		}{
26			{"hashicorp/consul/aws/1.0.0.tar.gz", "hashicorp/consul/aws/", "1.0.0", true},
27			{"hashicorp/consul/aws/2.3.1.tar.gz", "hashicorp/consul/aws/", "2.3.1", true},
28			{"hashicorp/consul/aws/readme.md", "hashicorp/consul/aws/", "", false},
29			{"hashicorp/consul/aws/", "hashicorp/consul/aws/", "", false},
30		}
31
32		for _, tt := range tests {
33			t.Run(tt.key, func(t *testing.T) {
34				v, ok := extractVersion(tt.key, tt.prefix)
35				assert.Equal(t, tt.valid, ok)
36				if ok {
37					assert.Equal(t, tt.expected, v)
38				}
39			})
40		}
41	}
42

package storage

import (
	"testing"

	"github.com/stretchr/testify/assert"
)

func TestObjectKey(t *testing.T) {
	key := objectKey("hashicorp", "consul", "aws", "1.0.0")
	assert.Equal(t, "hashicorp/consul/aws/1.0.0.tar.gz", key)
}

func TestPrefix(t *testing.T) {
	p := prefix("hashicorp", "consul", "aws")
	assert.Equal(t, "hashicorp/consul/aws/", p)
}

func TestExtractVersion(t *testing.T) {
	tests := []struct {
		key      string
		prefix   string
		expected string
		valid    bool
	}{
		{"hashicorp/consul/aws/1.0.0.tar.gz", "hashicorp/consul/aws/", "1.0.0", true},
		{"hashicorp/consul/aws/2.3.1.tar.gz", "hashicorp/consul/aws/", "2.3.1", true},
		{"hashicorp/consul/aws/readme.md", "hashicorp/consul/aws/", "", false},
		{"hashicorp/consul/aws/", "hashicorp/consul/aws/", "", false},
	}

	for _, tt := range tests {
		t.Run(tt.key, func(t *testing.T) {
			v, ok := extractVersion(tt.key, tt.prefix)
			assert.Equal(t, tt.valid, ok)
			if ok {
				assert.Equal(t, tt.expected, v)
			}
		})
	}
}


1	package storage
2
3	import (
4		"context"
5		"io"
6		"time"
7
8		"github.com/stretchr/testify/mock"
9	)
10
11	var _ Client = (*MockClient)(nil)
12
13	type MockClient struct {
14		mock.Mock
15	}
16
17	func (m *MockClient) ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error) {
18		args := m.Called(ctx, namespace, name, provider)
19		if val, ok := args.Get(0).([]string); ok {
20			return val, args.Error(1)
21		}
22		return nil, args.Error(1)
23	}
24
25	func (m *MockClient) PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error {
26		args := m.Called(ctx, namespace, name, provider, version, body)
27		return args.Error(0)
28	}
29
30	func (m *MockClient) HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error) {
31		args := m.Called(ctx, namespace, name, provider, version)
32		return args.Bool(0), args.Error(1)
33	}
34
35	func (m *MockClient) PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error) {
36		args := m.Called(ctx, namespace, name, provider, version, expiry)
37		return args.String(0), args.Error(1)
38	}
39
40	func (m *MockClient) HeadBucket(ctx context.Context) error {
41		args := m.Called(ctx)
42		return args.Error(0)
43	}
44
45	func (m *MockClient) CheckReadWrite(ctx context.Context) error {
46		args := m.Called(ctx)
47		return args.Error(0)
48	}
49

package storage

import (
	"context"
	"io"
	"time"

	"github.com/stretchr/testify/mock"
)

var _ Client = (*MockClient)(nil)

type MockClient struct {
	mock.Mock
}

func (m *MockClient) ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error) {
	args := m.Called(ctx, namespace, name, provider)
	if val, ok := args.Get(0).([]string); ok {
		return val, args.Error(1)
	}
	return nil, args.Error(1)
}

func (m *MockClient) PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error {
	args := m.Called(ctx, namespace, name, provider, version, body)
	return args.Error(0)
}

func (m *MockClient) HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error) {
	args := m.Called(ctx, namespace, name, provider, version)
	return args.Bool(0), args.Error(1)
}

func (m *MockClient) PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error) {
	args := m.Called(ctx, namespace, name, provider, version, expiry)
	return args.String(0), args.Error(1)
}

func (m *MockClient) HeadBucket(ctx context.Context) error {
	args := m.Called(ctx)
	return args.Error(0)
}

func (m *MockClient) CheckReadWrite(ctx context.Context) error {
	args := m.Called(ctx)
	return args.Error(0)
}


Now let me also check the e2e test setup and README.


1	module "pet" {
2	  source = "ic-ops.net/foo/test/pet"
3	  version = "1.0.0"
4	}
5
6	output "pet_name" {
7	  value = module.pet.name
8	}

module "pet" {
  source = "ic-ops.net/foo/test/pet"
  version = "1.0.0"
}

output "pet_name" {
  value = module.pet.name
}


1	module github.com/immuta/operations/services/crow
2
3	go 1.25.5
4
5	require (
6		github.com/aws/aws-sdk-go-v2 v1.41.1
7		github.com/aws/aws-sdk-go-v2/config v1.32.9
8		github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0
9		github.com/labstack/echo/v4 v4.15.0
10		github.com/stretchr/testify v1.11.1
11	)
12
13	require (
14		github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 // indirect
15		github.com/aws/aws-sdk-go-v2/credentials v1.19.9 // indirect
16		github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17 // indirect
17		github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17 // indirect
18		github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17 // indirect
19		github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
20		github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.17 // indirect
21		github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 // indirect
22		github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.8 // indirect
23		github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17 // indirect
24		github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.17 // indirect
25		github.com/aws/aws-sdk-go-v2/service/signin v1.0.5 // indirect
26		github.com/aws/aws-sdk-go-v2/service/sso v1.30.10 // indirect
27		github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.14 // indirect
28		github.com/aws/aws-sdk-go-v2/service/sts v1.41.6 // indirect
29		github.com/aws/smithy-go v1.24.1 // indirect
30		github.com/davecgh/go-spew v1.1.1 // indirect
31		github.com/labstack/gommon v0.4.2 // indirect
32		github.com/mattn/go-colorable v0.1.14 // indirect
33		github.com/mattn/go-isatty v0.0.20 // indirect
34		github.com/pmezard/go-difflib v1.0.0 // indirect
35		github.com/stretchr/objx v0.5.3 // indirect
36		github.com/valyala/bytebufferpool v1.0.0 // indirect
37		github.com/valyala/fasttemplate v1.2.2 // indirect
38		golang.org/x/crypto v0.48.0 // indirect
39		golang.org/x/net v0.50.0 // indirect
40		golang.org/x/sys v0.41.0 // indirect
41		golang.org/x/text v0.34.0 // indirect
42		golang.org/x/time v0.14.0 // indirect
43		gopkg.in/yaml.v3 v3.0.1 // indirect
44	)
45

module github.com/immuta/operations/services/crow

go 1.25.5

require (
	github.com/aws/aws-sdk-go-v2 v1.41.1
	github.com/aws/aws-sdk-go-v2/config v1.32.9
	github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0
	github.com/labstack/echo/v4 v4.15.0
	github.com/stretchr/testify v1.11.1
)

require (
	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 // indirect
	github.com/aws/aws-sdk-go-v2/credentials v1.19.9 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.17 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.8 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.17 // indirect
	github.com/aws/aws-sdk-go-v2/service/signin v1.0.5 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.30.10 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.14 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.41.6 // indirect
	github.com/aws/smithy-go v1.24.1 // indirect
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/labstack/gommon v0.4.2 // indirect
	github.com/mattn/go-colorable v0.1.14 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/pmezard/go-difflib v1.0.0 // indirect
	github.com/stretchr/objx v0.5.3 // indirect
	github.com/valyala/bytebufferpool v1.0.0 // indirect
	github.com/valyala/fasttemplate v1.2.2 // indirect
	golang.org/x/crypto v0.48.0 // indirect
	golang.org/x/net v0.50.0 // indirect
	golang.org/x/sys v0.41.0 // indirect
	golang.org/x/text v0.34.0 // indirect
	golang.org/x/time v0.14.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
)


Now I have a complete picture of the codebase. Here's the plan:

**Current state:** Every storage method has fixed named parameters (`namespace`, `name`, `provider`) and routing uses fixed Echo named params (`:namespace/:name/:provider`).

**Approach:**
1. Replace storage interface signatures with `modulePath []string` (capturing all path segments before version)
2. Update S3 path functions to join the slice dynamically — backward compatible (3-segment paths produce same keys as before)
3. Switch HTTP routing to a single `GET /v1/modules/*` + `PUT /v1/modules/*` catch-all; dispatch based on path suffix (`/versions`, `/download`)
4. Update mock and all tests

Let me implement all the changes now.


Write

package storage

import (
	"context"
	"io"
	"time"
)

type Client interface {
	ListVersions(ctx context.Context, modulePath []string) ([]string, error)
	HeadModule(ctx context.Context, modulePath []string, version string) (bool, error)
	PresignGetModule(ctx context.Context, modulePath []string, version string, expiry time.Duration) (string, error)
	PutModule(ctx context.Context, modulePath []string, version string, body io.Reader) error
	HeadBucket(ctx context.Context) error
	CheckReadWrite(ctx context.Context) error
}


The file /Users/steve.briggs/projects/operations/services/crow/internal/storage/storage.go has been updated successfully.

package storage

import (
	"context"
	"io"
	"time"
)

type Client interface {
	ListVersions(ctx context.Context, modulePath []string) ([]string, error)
	HeadModule(ctx context.Context, modulePath []string, version string) (bool, error)
	PresignGetModule(ctx context.Context, modulePath []string, version string, expiry time.Duration) (string, error)
	PutModule(ctx context.Context, modulePath []string, version string, body io.Reader) error
	HeadBucket(ctx context.Context) error
	CheckReadWrite(ctx context.Context) error
}

-  ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error)

-  HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error)

-  PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error)

-  PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error

+  ListVersions(ctx context.Context, modulePath []string) ([]string, error)

+  HeadModule(ctx context.Context, modulePath []string, version string) (bool, error)

+  PresignGetModule(ctx context.Context, modulePath []string, version string, expiry time.Duration) (string, error)

+  PutModule(ctx context.Context, modulePath []string, version string, body io.Reader) error

package storage

import (
	"context"
	"io"
	"time"
)

type Client interface {
	ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error)
	HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error)
	PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error)
	PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error
	HeadBucket(ctx context.Context) error
	CheckReadWrite(ctx context.Context) error
}


package storage

import (
	"context"
	"errors"
	"fmt"
	"io"
	"strings"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/service/s3/types"
)

type S3Client struct {
	client     *s3.Client
	presigner  *s3.PresignClient
	bucketName string
}

func NewS3Client(awsCfg aws.Config, bucketName string, optFns ...func(*s3.Options)) *S3Client {
	client := s3.NewFromConfig(awsCfg, optFns...)
	return &S3Client{
		client:     client,
		presigner:  s3.NewPresignClient(client),
		bucketName: bucketName,
	}
}

// NewS3ClientWithPresignEndpoint creates an S3Client that uses a separate
// endpoint for generating presigned URLs. This is needed when the S3 backend
// is reachable internally at one address (e.g. http://rustfs:9000) but
// clients need presigned URLs pointing to an external address (e.g.
// http://localhost:9000).
func NewS3ClientWithPresignEndpoint(awsCfg aws.Config, bucketName, presignEndpoint string, optFns ...func(*s3.Options)) *S3Client {
	client := s3.NewFromConfig(awsCfg, optFns...)
	presignClient := s3.NewFromConfig(awsCfg, append(optFns, func(o *s3.Options) {
		o.BaseEndpoint = &presignEndpoint
	})...)
	return &S3Client{
		client:     client,
		presigner:  s3.NewPresignClient(presignClient),
		bucketName: bucketName,
	}
}

func objectKey(modulePath []string, version string) string {
	return strings.Join(modulePath, "/") + "/" + version + ".tar.gz"
}

func prefix(modulePath []string) string {
	return strings.Join(modulePath, "/") + "/"
}

func extractVersion(key, pfx string) (string, bool) {
	if !strings.HasPrefix(key, pfx) {
		return "", false
	}
	rest := strings.TrimPrefix(key, pfx)
	if !strings.HasSuffix(rest, ".tar.gz") {
		return "", false
	}
	version := strings.TrimSuffix(rest, ".tar.gz")
	if version == "" {
		return "", false
	}
	return version, true
}

func (s *S3Client) ListVersions(ctx context.Context, modulePath []string) ([]string, error) {
	pfx := prefix(modulePath)
	input := &s3.ListObjectsV2Input{
		Bucket: &s.bucketName,
		Prefix: &pfx,
	}

	var versions []string
	paginator := s3.NewListObjectsV2Paginator(s.client, input)
	for paginator.HasMorePages() {
		output, err := paginator.NextPage(ctx)
		if err != nil {
			return nil, fmt.Errorf("listing objects: %w", err)
		}
		for _, obj := range output.Contents {
			if obj.Key == nil {
				return nil, errors.New("listing objects: object key is nil")
			}
			if v, ok := extractVersion(*obj.Key, pfx); ok {
				versions = append(versions, v)
			}
		}
	}

	return versions, nil
}

func (s *S3Client) HeadModule(ctx context.Context, modulePath []string, version string) (bool, error) {
	key := objectKey(modulePath, version)
	_, err := s.client.HeadObject(ctx, &s3.HeadObjectInput{
		Bucket: &s.bucketName,
		Key:    &key,
	})
	if err != nil {
		var notFound *types.NotFound
		var noSuchKey *types.NoSuchKey
		if errors.As(err, &notFound) || errors.As(err, &noSuchKey) {
			return false, nil
		}
		return false, fmt.Errorf("head object: %w", err)
	}
	return true, nil
}

func (s *S3Client) PutModule(ctx context.Context, modulePath []string, version string, body io.Reader) error {
	key := objectKey(modulePath, version)
	contentType := "application/gzip"
	_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
		Bucket:      &s.bucketName,
		Key:         &key,
		Body:        body,
		ContentType: &contentType,
	})
	if err != nil {
		return fmt.Errorf("put object: %w", err)
	}
	return nil
}

func (s *S3Client) PresignGetModule(ctx context.Context, modulePath []string, version string, expiry time.Duration) (string, error) {
	key := objectKey(modulePath, version)
	req, err := s.presigner.PresignGetObject(ctx, &s3.GetObjectInput{
		Bucket: &s.bucketName,
		Key:    &key,
	}, s3.WithPresignExpires(expiry))
	if err != nil {
		return "", fmt.Errorf("presigning URL: %w", err)
	}
	return req.URL, nil
}

const readyzKey = ".readyz"

func (s *S3Client) HeadBucket(ctx context.Context) error {
	_, err := s.client.HeadBucket(ctx, &s3.HeadBucketInput{
		Bucket: &s.bucketName,
	})
	if err != nil {
		return fmt.Errorf("head bucket: %w", err)
	}
	return nil
}

func (s *S3Client) CheckReadWrite(ctx context.Context) error {
	body := strings.NewReader(time.Now().UTC().Format(time.RFC3339))
	_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
		Bucket: &s.bucketName,
		Key:    aws.String(readyzKey),
		Body:   body,
	})
	if err != nil {
		return fmt.Errorf("readyz put object: %w", err)
	}

	_, err = s.client.HeadObject(ctx, &s3.HeadObjectInput{
		Bucket: &s.bucketName,
		Key:    aws.String(readyzKey),
	})
	if err != nil {
		return fmt.Errorf("readyz head object: %w", err)
	}

	return nil
}


The file /Users/steve.briggs/projects/operations/services/crow/internal/storage/s3.go has been updated successfully.

package storage

import (
	"context"
	"errors"
	"fmt"
	"io"
	"strings"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/service/s3/types"
)

type S3Client struct {
	client     *s3.Client
	presigner  *s3.PresignClient
	bucketName string
}

func NewS3Client(awsCfg aws.Config, bucketName string, optFns ...func(*s3.Options)) *S3Client {
	client := s3.NewFromConfig(awsCfg, optFns...)
	return &S3Client{
		client:     client,
		presigner:  s3.NewPresignClient(client),
		bucketName: bucketName,
	}
}

// NewS3ClientWithPresignEndpoint creates an S3Client that uses a separate
// endpoint for generating presigned URLs. This is needed when the S3 backend
// is reachable internally at one address (e.g. http://rustfs:9000) but
// clients need presigned URLs pointing to an external address (e.g.
// http://localhost:9000).
func NewS3ClientWithPresignEndpoint(awsCfg aws.Config, bucketName, presignEndpoint string, optFns ...func(*s3.Options)) *S3Client {
	client := s3.NewFromConfig(awsCfg, optFns...)
	presignClient := s3.NewFromConfig(awsCfg, append(optFns, func(o *s3.Options) {
		o.BaseEndpoint = &presignEndpoint
	})...)
	return &S3Client{
		client:     client,
		presigner:  s3.NewPresignClient(presignClient),
		bucketName: bucketName,
	}
}

func objectKey(modulePath []string, version string) string {
	return strings.Join(modulePath, "/") + "/" + version + ".tar.gz"
}

func prefix(modulePath []string) string {
	return strings.Join(modulePath, "/") + "/"
}

func extractVersion(key, pfx string) (string, bool) {
	if !strings.HasPrefix(key, pfx) {
		return "", false
	}
	rest := strings.TrimPrefix(key, pfx)
	if !strings.HasSuffix(rest, ".tar.gz") {
		return "", false
	}
	version := strings.TrimSuffix(rest, ".tar.gz")
	if version == "" {
		return "", false
	}
	return version, true
}

func (s *S3Client) ListVersions(ctx context.Context, modulePath []string) ([]string, error) {
	pfx := prefix(modulePath)
	input := &s3.ListObjectsV2Input{
		Bucket: &s.bucketName,
		Prefix: &pfx,
	}

	var versions []string
	paginator := s3.NewListObjectsV2Paginator(s.client, input)
	for paginator.HasMorePages() {
		output, err := paginator.NextPage(ctx)
		if err != nil {
			return nil, fmt.Errorf("listing objects: %w", err)
		}
		for _, obj := range output.Contents {
			if obj.Key == nil {
				return nil, errors.New("listing objects: object key is nil")
			}
			if v, ok := extractVersion(*obj.Key, pfx); ok {
				versions = append(versions, v)
			}
		}
	}

	return versions, nil
}

func (s *S3Client) HeadModule(ctx context.Context, modulePath []string, version string) (bool, error) {
	key := objectKey(modulePath, version)
	_, err := s.client.HeadObject(ctx, &s3.HeadObjectInput{
		Bucket: &s.bucketName,
		Key:    &key,
	})
	if err != nil {
		var notFound *types.NotFound
		var noSuchKey *types.NoSuchKey
		if errors.As(err, &notFound) || errors.As(err, &noSuchKey) {
			return false, nil
		}
		return false, fmt.Errorf("head object: %w", err)
	}
	return true, nil
}

func (s *S3Client) PutModule(ctx context.Context, modulePath []string, version string, body io.Reader) error {
	key := objectKey(modulePath, version)
	contentType := "application/gzip"
	_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
		Bucket:      &s.bucketName,
		Key:         &key,
		Body:        body,
		ContentType: &contentType,
	})
	if err != nil {
		return fmt.Errorf("put object: %w", err)
	}
	return nil
}

func (s *S3Client) PresignGetModule(ctx context.Context, modulePath []string, version string, expiry time.Duration) (string, error) {
	key := objectKey(modulePath, version)
	req, err := s.presigner.PresignGetObject(ctx, &s3.GetObjectInput{
		Bucket: &s.bucketName,
		Key:    &key,
	}, s3.WithPresignExpires(expiry))
	if err != nil {
		return "", fmt.Errorf("presigning URL: %w", err)
	}
	return req.URL, nil
}

const readyzKey = ".readyz"

func (s *S3Client) HeadBucket(ctx context.Context) error {
	_, err := s.client.HeadBucket(ctx, &s3.HeadBucketInput{
		Bucket: &s.bucketName,
	})
	if err != nil {
		return fmt.Errorf("head bucket: %w", err)
	}
	return nil
}

func (s *S3Client) CheckReadWrite(ctx context.Context) error {
	body := strings.NewReader(time.Now().UTC().Format(time.RFC3339))
	_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
		Bucket: &s.bucketName,
		Key:    aws.String(readyzKey),
		Body:   body,
	})
	if err != nil {
		return fmt.Errorf("readyz put object: %w", err)
	}

	_, err = s.client.HeadObject(ctx, &s3.HeadObjectInput{
		Bucket: &s.bucketName,
		Key:    aws.String(readyzKey),
	})
	if err != nil {
		return fmt.Errorf("readyz head object: %w", err)
	}

	return nil
}

-func objectKey(namespace, name, provider, version string) string {

-  return fmt.Sprintf("%s/%s/%s/%s.tar.gz", namespace, name, provider, version)

+func objectKey(modulePath []string, version string) string {

+  return strings.Join(modulePath, "/") + "/" + version + ".tar.gz"

-func prefix(namespace, name, provider string) string {

-  return fmt.Sprintf("%s/%s/%s/", namespace, name, provider)

+func prefix(modulePath []string) string {

-func (s *S3Client) ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error) {

-  pfx := prefix(namespace, name, provider)

+func (s *S3Client) ListVersions(ctx context.Context, modulePath []string) ([]string, error) {

-func (s *S3Client) HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error) {

-  key := objectKey(namespace, name, provider, version)

+func (s *S3Client) HeadModule(ctx context.Context, modulePath []string, version string) (bool, error) {

+  key := objectKey(modulePath, version)

-func (s *S3Client) PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error {

+func (s *S3Client) PutModule(ctx context.Context, modulePath []string, version string, body io.Reader) error {

-func (s *S3Client) PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error) {

+func (s *S3Client) PresignGetModule(ctx context.Context, modulePath []string, version string, expiry time.Duration) (string, error) {

package storage

import (
	"context"
	"errors"
	"fmt"
	"io"
	"strings"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/service/s3/types"
)

type S3Client struct {
	client     *s3.Client
	presigner  *s3.PresignClient
	bucketName string
}

func NewS3Client(awsCfg aws.Config, bucketName string, optFns ...func(*s3.Options)) *S3Client {
	client := s3.NewFromConfig(awsCfg, optFns...)
	return &S3Client{
		client:     client,
		presigner:  s3.NewPresignClient(client),
		bucketName: bucketName,
	}
}

// NewS3ClientWithPresignEndpoint creates an S3Client that uses a separate
// endpoint for generating presigned URLs. This is needed when the S3 backend
// is reachable internally at one address (e.g. http://rustfs:9000) but
// clients need presigned URLs pointing to an external address (e.g.
// http://localhost:9000).
func NewS3ClientWithPresignEndpoint(awsCfg aws.Config, bucketName, presignEndpoint string, optFns ...func(*s3.Options)) *S3Client {
	client := s3.NewFromConfig(awsCfg, optFns...)
	presignClient := s3.NewFromConfig(awsCfg, append(optFns, func(o *s3.Options) {
		o.BaseEndpoint = &presignEndpoint
	})...)
	return &S3Client{
		client:     client,
		presigner:  s3.NewPresignClient(presignClient),
		bucketName: bucketName,
	}
}

func objectKey(namespace, name, provider, version string) string {
	return fmt.Sprintf("%s/%s/%s/%s.tar.gz", namespace, name, provider, version)
}

func prefix(namespace, name, provider string) string {
	return fmt.Sprintf("%s/%s/%s/", namespace, name, provider)
}

func extractVersion(key, pfx string) (string, bool) {
	if !strings.HasPrefix(key, pfx) {
		return "", false
	}
	rest := strings.TrimPrefix(key, pfx)
	if !strings.HasSuffix(rest, ".tar.gz") {
		return "", false
	}
	version := strings.TrimSuffix(rest, ".tar.gz")
	if version == "" {
		return "", false
	}
	return version, true
}

func (s *S3Client) ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error) {
	pfx := prefix(namespace, name, provider)
	input := &s3.ListObjectsV2Input{
		Bucket: &s.bucketName,
		Prefix: &pfx,
	}

	var versions []string
	paginator := s3.NewListObjectsV2Paginator(s.client, input)
	for paginator.HasMorePages() {
		output, err := paginator.NextPage(ctx)
		if err != nil {
			return nil, fmt.Errorf("listing objects: %w", err)
		}
		for _, obj := range output.Contents {
			if obj.Key == nil {
				return nil, errors.New("listing objects: object key is nil")
			}
			if v, ok := extractVersion(*obj.Key, pfx); ok {
				versions = append(versions, v)
			}
		}
	}

	return versions, nil
}

func (s *S3Client) HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error) {
	key := objectKey(namespace, name, provider, version)
	_, err := s.client.HeadObject(ctx, &s3.HeadObjectInput{
		Bucket: &s.bucketName,
		Key:    &key,
	})
	if err != nil {
		var notFound *types.NotFound
		var noSuchKey *types.NoSuchKey
		if errors.As(err, &notFound) || errors.As(err, &noSuchKey) {
			return false, nil
		}
		return false, fmt.Errorf("head object: %w", err)
	}
	return true, nil
}

func (s *S3Client) PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error {
	key := objectKey(namespace, name, provider, version)
	contentType := "application/gzip"
	_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
		Bucket:      &s.bucketName,
		Key:         &key,
		Body:        body,
		ContentType: &contentType,
	})
	if err != nil {
		return fmt.Errorf("put object: %w", err)
	}
	return nil
}

func (s *S3Client) PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error) {
	key := objectKey(namespace, name, provider, version)
	req, err := s.presigner.PresignGetObject(ctx, &s3.GetObjectInput{
		Bucket: &s.bucketName,
		Key:    &key,
	}, s3.WithPresignExpires(expiry))
	if err != nil {
		return "", fmt.Errorf("presigning URL: %w", err)
	}
	return req.URL, nil
}

const readyzKey = ".readyz"

func (s *S3Client) HeadBucket(ctx context.Context) error {
	_, err := s.client.HeadBucket(ctx, &s3.HeadBucketInput{
		Bucket: &s.bucketName,
	})
	if err != nil {
		return fmt.Errorf("head bucket: %w", err)
	}
	return nil
}

func (s *S3Client) CheckReadWrite(ctx context.Context) error {
	body := strings.NewReader(time.Now().UTC().Format(time.RFC3339))
	_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
		Bucket: &s.bucketName,
		Key:    aws.String(readyzKey),
		Body:   body,
	})
	if err != nil {
		return fmt.Errorf("readyz put object: %w", err)
	}

	_, err = s.client.HeadObject(ctx, &s3.HeadObjectInput{
		Bucket: &s.bucketName,
		Key:    aws.String(readyzKey),
	})
	if err != nil {
		return fmt.Errorf("readyz head object: %w", err)
	}

	return nil
}


package storage

import (
	"context"
	"io"
	"time"

	"github.com/stretchr/testify/mock"
)

var _ Client = (*MockClient)(nil)

type MockClient struct {
	mock.Mock
}

func (m *MockClient) ListVersions(ctx context.Context, modulePath []string) ([]string, error) {
	args := m.Called(ctx, modulePath)
	if val, ok := args.Get(0).([]string); ok {
		return val, args.Error(1)
	}
	return nil, args.Error(1)
}

func (m *MockClient) PutModule(ctx context.Context, modulePath []string, version string, body io.Reader) error {
	args := m.Called(ctx, modulePath, version, body)
	return args.Error(0)
}

func (m *MockClient) HeadModule(ctx context.Context, modulePath []string, version string) (bool, error) {
	args := m.Called(ctx, modulePath, version)
	return args.Bool(0), args.Error(1)
}

func (m *MockClient) PresignGetModule(ctx context.Context, modulePath []string, version string, expiry time.Duration) (string, error) {
	args := m.Called(ctx, modulePath, version, expiry)
	return args.String(0), args.Error(1)
}

func (m *MockClient) HeadBucket(ctx context.Context) error {
	args := m.Called(ctx)
	return args.Error(0)
}

func (m *MockClient) CheckReadWrite(ctx context.Context) error {
	args := m.Called(ctx)
	return args.Error(0)
}


The file /Users/steve.briggs/projects/operations/services/crow/internal/storage/mock_storage.go has been updated successfully.

package storage

import (
	"context"
	"io"
	"time"

	"github.com/stretchr/testify/mock"
)

var _ Client = (*MockClient)(nil)

type MockClient struct {
	mock.Mock
}

func (m *MockClient) ListVersions(ctx context.Context, modulePath []string) ([]string, error) {
	args := m.Called(ctx, modulePath)
	if val, ok := args.Get(0).([]string); ok {
		return val, args.Error(1)
	}
	return nil, args.Error(1)
}

func (m *MockClient) PutModule(ctx context.Context, modulePath []string, version string, body io.Reader) error {
	args := m.Called(ctx, modulePath, version, body)
	return args.Error(0)
}

func (m *MockClient) HeadModule(ctx context.Context, modulePath []string, version string) (bool, error) {
	args := m.Called(ctx, modulePath, version)
	return args.Bool(0), args.Error(1)
}

func (m *MockClient) PresignGetModule(ctx context.Context, modulePath []string, version string, expiry time.Duration) (string, error) {
	args := m.Called(ctx, modulePath, version, expiry)
	return args.String(0), args.Error(1)
}

func (m *MockClient) HeadBucket(ctx context.Context) error {
	args := m.Called(ctx)
	return args.Error(0)
}

func (m *MockClient) CheckReadWrite(ctx context.Context) error {
	args := m.Called(ctx)
	return args.Error(0)
}

-func (m *MockClient) ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error) {

-  args := m.Called(ctx, namespace, name, provider)

+func (m *MockClient) ListVersions(ctx context.Context, modulePath []string) ([]string, error) {

-func (m *MockClient) PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error {

-  args := m.Called(ctx, namespace, name, provider, version, body)

+func (m *MockClient) PutModule(ctx context.Context, modulePath []string, version string, body io.Reader) error {

+  args := m.Called(ctx, modulePath, version, body)

-func (m *MockClient) HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error) {

-  args := m.Called(ctx, namespace, name, provider, version)

+func (m *MockClient) HeadModule(ctx context.Context, modulePath []string, version string) (bool, error) {

+  args := m.Called(ctx, modulePath, version)

-func (m *MockClient) PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error) {

-  args := m.Called(ctx, namespace, name, provider, version, expiry)

+func (m *MockClient) PresignGetModule(ctx context.Context, modulePath []string, version string, expiry time.Duration) (string, error) {

+  args := m.Called(ctx, modulePath, version, expiry)

package storage

import (
	"context"
	"io"
	"time"

	"github.com/stretchr/testify/mock"
)

var _ Client = (*MockClient)(nil)

type MockClient struct {
	mock.Mock
}

func (m *MockClient) ListVersions(ctx context.Context, namespace, name, provider string) ([]string, error) {
	args := m.Called(ctx, namespace, name, provider)
	if val, ok := args.Get(0).([]string); ok {
		return val, args.Error(1)
	}
	return nil, args.Error(1)
}

func (m *MockClient) PutModule(ctx context.Context, namespace, name, provider, version string, body io.Reader) error {
	args := m.Called(ctx, namespace, name, provider, version, body)
	return args.Error(0)
}

func (m *MockClient) HeadModule(ctx context.Context, namespace, name, provider, version string) (bool, error) {
	args := m.Called(ctx, namespace, name, provider, version)
	return args.Bool(0), args.Error(1)
}

func (m *MockClient) PresignGetModule(ctx context.Context, namespace, name, provider, version string, expiry time.Duration) (string, error) {
	args := m.Called(ctx, namespace, name, provider, version, expiry)
	return args.String(0), args.Error(1)
}

func (m *MockClient) HeadBucket(ctx context.Context) error {
	args := m.Called(ctx)
	return args.Error(0)
}

func (m *MockClient) CheckReadWrite(ctx context.Context) error {
	args := m.Called(ctx)
	return args.Error(0)
}


Now update the handler with wildcard routing and dynamic path parsing: