<persisted-output>
Output too large (54.2KB). Full output saved to: /Users/steve.briggs/.claude/projects/-Users-steve-briggs-cpi-new-prod-hosting-ap-northeast-1-eks-saas-prod-apne1-2-applications/0d126816-796a-47b1-8588-f1ba2edc9af4/tool-results/boi54qy3j.txt

Preview (first 2KB):
51:   net.peer.name=sts.us-east-1.amazonaws.com http.request_content_length=43 http.request.header.x_amz_security_token="*****" http.request.header.content_type=application/x-www-form-urlencoded http.request.header.amz_sdk_invocation_id=05a93037-eb27-4730-8e41-5f859d61af09 http.method=POST http.url=https://sts.us-east-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/sts#1.28.4" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****"
70:   http.url=https://sts.us-east-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/sts#1.28.4" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20260513T214607Z http.request.header.amz_sdk_invocation_id=ea8c3947-0b07-4275-9e0f-7b104faf9d88 http.method=POST net.peer.name=sts.us-east-1.amazonaws.com http.request_content_length=43 http.request.header.x_amz_security_token="*****" http.request.header.content_type=application/x-www-form-urlencoded
161:   http.method=POST http.request_content_length=43 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.x_amz_date=20260513T214607Z
...
</persisted-output>

51:   net.peer.name=sts.us-east-1.amazonaws.com http.request_content_length=43 http.request.header.x_amz_security_token="*****" http.request.header.content_type=application/x-www-form-urlencoded http.request.header.amz_sdk_invocation_id=05a93037-eb27-4730-8e41-5f859d61af09 http.method=POST http.url=https://sts.us-east-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/sts#1.28.4" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****"
70:   http.url=https://sts.us-east-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/sts#1.28.4" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20260513T214607Z http.request.header.amz_sdk_invocation_id=ea8c3947-0b07-4275-9e0f-7b104faf9d88 http.method=POST net.peer.name=sts.us-east-1.amazonaws.com http.request_content_length=43 http.request.header.x_amz_security_token="*****" http.request.header.content_type=application/x-www-form-urlencoded
161:   http.method=POST http.request_content_length=43 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.x_amz_date=20260513T214607Z
192:   http.method=POST http.url=https://dynamodb.us-east-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/dynamodb#1.30.4" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.request.header.amz_sdk_invocation_id=aa750bee-7b5f-40ac-b6a6-8f926e011a40 http.request_content_length=256 http.request.header.accept_encoding=identity http.request.header.x_amz_target=DynamoDB_20120810.GetItem
226:   http.method=POST http.url=https://dynamodb.us-east-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/dynamodb#1.30.4" http.request_content_length=261 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.request.header.x_amz_security_token="*****" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.content_type=application/x-amz-json-1.0 http.request.header.x_amz_target=DynamoDB_20120810.GetItem
234:   http.request.header.content_type=application/x-amz-json-1.0 net.peer.name=dynamodb.us-east-1.amazonaws.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/dynamodb#1.30.4" http.request.header.x_amz_date=20260513T214608Z http.request.header.x_amz_security_token="*****" http.request.header.amz_sdk_request="attempt=1; max=5" http.method=POST http.request_content_length=241 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.url=https://dynamodb.us-east-1.amazonaws.com/ http.request.header.accept_encoding=identity
263:   http.request_content_length=43 http.request.header.x_amz_security_token="*****" http.request.header.amz_sdk_request="attempt=1; max=5" http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/sts#1.28.4" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************4NGX/20260513/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.amz_sdk_invocation_id=a413d18e-b5dc-4a7e-bc8d-151ff7e10f72 http.request.header.x_amz_date=20260513T214609Z http.request.header.content_type=application/x-www-form-urlencoded http.method=POST http.url=https://sts.us-east-1.amazonaws.com/ net.peer.name=sts.us-east-1.amazonaws.com
338:   http.request_content_length=239 http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_target=DynamoDB_20120810.GetItem http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/dynamodb#1.30.4" net.peer.name=dynamodb.us-east-1.amazonaws.com http.request.header.x_amz_security_token="*****" http.url=https://dynamodb.us-east-1.amazonaws.com/ http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************4NGX/20260513/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.request.header.x_amz_date=20260513T214610Z http.request.header.amz_sdk_invocation_id=018bf678-d4e4-492b-8d98-e99c8d9b7407 http.method=POST
392:   http.method=POST net.peer.name=dynamodb.us-east-1.amazonaws.com http.request.header.x_amz_target=DynamoDB_20120810.GetItem http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/dynamodb#1.30.4" http.request.header.x_amz_security_token="*****" http.url=https://dynamodb.us-east-1.amazonaws.com/ http.request_content_length=260
405:17:46:11.495 STDERR terraform:    http.method=POST http.url=https://sts.us-east-1.amazonaws.com/ http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****"
443:17:46:12.407 STDERR terraform:    http.method=POST http.url=https://dynamodb.us-east-1.amazonaws.com/ http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20260513T214612Z http.request.header.x_amz_security_token="*****" http.request.header.accept_encoding=identity
450:17:46:13.107 STDERR terraform:    http.url=https://dynamodb.us-east-1.amazonaws.com/ net.peer.name=dynamodb.us-east-1.amazonaws.com http.request_content_length=263 http.request.header.amz_sdk_invocation_id=1f3be8b3-a85d-4072-8304-310a4b513da7 http.request.header.x_amz_target=DynamoDB_20120810.GetItem http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/macos lang/go#1.22.7 md/GOOS#darwin md/GOARCH#arm64 api/dynamodb#1.30.4" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.request.header.x_amz_security_token="*****" http.request.header.accept_encoding=identity http.method=POST http.request.header.content_type=application/x-amz-json-1.0 http.request.header.x_amz_date=20260513T214613Z
527:17:46:21.656 STDERR terraform: GET /v1/auth/token/lookup-self HTTP/1.1
549:17:46:21.843 STDERR terraform: POST /v1/auth/token/create HTTP/1.1
627:17:46:22.557 STDERR terraform:    http.request_content_length=77 http.request.header.amz_sdk_invocation_id=9cde731b-0ec3-494e-a42b-ab5d0488c3f0 http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) terraform-provider-aws/6.34.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.41.2 ua/2.1 os/macos lang/go#1.25.7 md/GOOS#darwin md/GOARCH#arm64 api/iam#1.53.3 m/v,w" rpc.system.name=aws-api http.request.header.amz_sdk_request="attempt=1; max=25" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" net.peer.name=iam.amazonaws.com tf_aws.sdk=aws-sdk-go-v2 tf_provider_addr=registry.terraform.io/hashicorp/aws http.request.header.x_amz_date=20260513T214622Z http.url=https://iam.amazonaws.com/ tf_req_id=ab61bbf0-bcd1-6b5f-9f0a-0fa2960c4a83 @module=aws http.method=POST rpc.method=GetRole tf_aws.signing_region="" tf_rpc=ReadDataSource timestamp=2026-05-13T17:46:22.556-0400
634:17:46:22.557 STDERR terraform:    tf_data_source_type=aws_security_group http.request.header.x_amz_date=20260513T214622Z http.request.header.x_amz_security_token="*****" tf_aws.sdk=aws-sdk-go-v2 http.method=POST http.request.header.amz_sdk_invocation_id=f34d70db-9d02-403d-b516-ed309d08c9ac tf_aws.signing_region="" timestamp=2026-05-13T17:46:22.556-0400
644:17:46:22.558 STDERR terraform:    tf_req_id=87daf925-6510-fccf-ab7e-ee3df912b923 http.request.header.amz_sdk_invocation_id=1c139d58-7e52-4db5-8a98-e3842577e092 http.request.header.amz_sdk_request="attempt=1; max=25" net.peer.name=ec2.ap-northeast-1.amazonaws.com http.request_content_length=85 http.url=https://ec2.ap-northeast-1.amazonaws.com/ rpc.system.name=aws-api tf_mux_provider="*schema.GRPCProviderServer" rpc.method=DescribeAddresses tf_rpc=ReadResource @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.70/logging/tf_logger.go:45 http.request.header.content_type=application/x-www-form-urlencoded http.request.header.x_amz_security_token="*****" timestamp=2026-05-13T17:46:22.557-0400
652:17:46:22.563 STDERR terraform:    tf_provider_addr=registry.terraform.io/hashicorp/aws http.url=https://ec2.ap-northeast-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) terraform-provider-aws/6.34.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.41.2 ua/2.1 os/macos lang/go#1.25.7 md/GOOS#darwin md/GOARCH#arm64 api/ec2#1.291.0 m/C,v,w" tf_aws.sdk=aws-sdk-go-v2 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/ap-northeast-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" aws.region=ap-northeast-1 http.request_content_length=96 tf_mux_provider="*proto5server.Server" @module=aws http.request.header.amz_sdk_invocation_id=4a80cea0-bb4d-4b71-8c25-298b1a0fd8fb @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.70/logging/tf_logger.go:45 net.peer.name=ec2.ap-northeast-1.amazonaws.com tf_aws.signing_region="" tf_resource_type=aws_vpc_security_group_ingress_rule tf_rpc=ReadResource http.method=POST http.request.header.x_amz_security_token="*****" tf_req_id=25bb6f3c-2ba0-85fa-a8f9-f6b12bc395a3 http.request.header.content_type=application/x-www-form-urlencoded http.request.header.x_amz_date=20260513T214622Z timestamp=2026-05-13T17:46:22.562-0400
768:17:46:23.047 STDERR terraform:    http.request_content_length=43 tf_aws.signing_region="" aws.region=ap-northeast-1 http.request.header.amz_sdk_invocation_id=bbf706cb-b08e-4492-9a1a-419cbe5818f2 http.request.header.amz_sdk_request="attempt=1; max=25" http.request.header.x_amz_security_token="*****" http.url=https://sts.ap-northeast-1.amazonaws.com/ @module=aws http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/ap-northeast-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.x_amz_date=20260513T214623Z rpc.method=GetCallerIdentity tf_provider_addr=registry.terraform.io/hashicorp/aws timestamp=2026-05-13T17:46:23.046-0400
803:17:46:23.230 STDERR terraform:    http.request.header.x_amz_security_token="*****" tf_rpc=ReadResource http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) terraform-provider-aws/6.34.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.41.2 ua/2.1 os/macos lang/go#1.25.7 md/GOOS#darwin md/GOARCH#arm64 api/iam#1.53.3 m/v,w" rpc.method=GetRole tf_mux_provider="*schema.GRPCProviderServer" @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.70/logging/tf_logger.go:45 http.request.header.content_type=application/x-www-form-urlencoded net.peer.name=iam.amazonaws.com tf_provider_addr=registry.terraform.io/hashicorp/aws aws.region=ap-northeast-1 http.method=POST tf_req_id=bcace08f-1ba6-2b35-bd5f-cdf78ce5eff9 http.request.header.amz_sdk_invocation_id=03b60365-134e-4676-9375-0e0a358ac212 tf_aws.signing_region="" http.request.header.x_amz_date=20260513T214623Z http.request_content_length=77 tf_aws.sdk=aws-sdk-go-v2 tf_resource_type=aws_iam_role http.request.header.amz_sdk_request="attempt=1; max=25" rpc.system.name=aws-api http.url=https://iam.amazonaws.com/ timestamp=2026-05-13T17:46:23.230-0400
820:17:46:23.241 STDERR terraform:    http.request.header.content_type=application/x-www-form-urlencoded http.request.header.x_amz_security_token="*****" timestamp=2026-05-13T17:46:23.241-0400
826:17:46:23.271 STDERR terraform:   |       <AssumeRolePolicyDocument>%7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Effect%22%3A%22Allow%22%2C%22Principal%22%3A%7B%22Federated%22%3A%22arn%3Aaws%3Aiam%3A%3A475806706894%3Aoidc-provider%2Foidc.eks.ap-northeast-1.amazonaws.com%2Fid%2FAB014096DDC9254A905E2AF8A8FA0671%22%7D%2C%22Action%22%3A%22sts%3AAssumeRoleWithWebIdentity%22%2C%22Condition%22%3A%7B%22StringEquals%22%3A%7B%22oidc.eks.ap-northeast-1.amazonaws.com%2Fid%2FAB014096DDC9254A905E2AF8A8FA0671%3Asub%22%3A%22system%3Aserviceaccount%3Akube-system%3Aaws-load-balancer-controller%22%2C%22oidc.eks.ap-northeast-1.amazonaws.com%2Fid%2FAB014096DDC9254A905E2AF8A8FA0671%3Aaud%22%3A%22sts.amazonaws.com%22%7D%7D%7D%5D%7D</AssumeRolePolicyDocument>
859:17:46:23.278 STDERR terraform:    http.url=https://iam.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) terraform-provider-aws/6.34.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.41.2 ua/2.1 os/macos lang/go#1.25.7 md/GOOS#darwin md/GOARCH#arm64 api/iam#1.53.3 m/C,v,w" tf_req_id=d01060e7-4977-85a2-b7cc-7ecabbaf7ebf @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.70/logging/tf_logger.go:45 http.request.header.amz_sdk_request="attempt=1; max=25" http.request.header.content_type=application/x-www-form-urlencoded tf_mux_provider="*schema.GRPCProviderServer" http.method=POST http.request.header.amz_sdk_invocation_id=d679a4ca-263e-4a20-8858-41cfc07c0419 http.request_content_length=102 net.peer.name=iam.amazonaws.com rpc.system.name=aws-api tf_aws.sdk=aws-sdk-go-v2 aws.region=ap-northeast-1 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" tf_rpc=ReadResource http.request.header.x_amz_date=20260513T214623Z http.request.header.x_amz_security_token="*****" tf_resource_type=aws_iam_role timestamp=2026-05-13T17:46:23.271-0400
898:17:46:23.348 STDERR terraform:    tf_mux_provider="*schema.GRPCProviderServer" @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.70/logging/tf_logger.go:45 rpc.method=ListRolePolicies rpc.system.name=aws-api http.request_content_length=86 http.request.header.amz_sdk_invocation_id=e52d1771-434c-463e-a79c-3354469e8f40 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.amz_sdk_request="attempt=1; max=25" http.request.header.x_amz_date=20260513T214623Z tf_resource_type=aws_iam_role aws.region=ap-northeast-1 net.peer.name=iam.amazonaws.com tf_aws.sdk=aws-sdk-go-v2 timestamp=2026-05-13T17:46:23.341-0400
900:17:46:23.354 STDERR terraform:   | {"cluster":{"name":"eks-saas-prod-apne1-2","arn":"arn:aws:eks:ap-northeast-1:475806706894:cluster/eks-saas-prod-apne1-2","createdAt":1.670352973483E9,"version":"1.34","upgradePolicy":{"supportType":"EXTENDED"},"endpoint":"https://AB014096DDC9254A905E2AF8A8FA0671.yl4.ap-northeast-1.eks.amazonaws.com","roleArn":"arn:aws:iam::475806706894:role/eks-role-eks-saas-prod-apne1-2","resourcesVpcConfig":{"subnetIds":["subnet-0a229fd53d7c2f9b9","subnet-03f04c88670656773","subnet-0b5214f5693d9274e"],"securityGroupIds":["sg-098f342154c3aa9c6"],"clusterSecurityGroupId":"sg-01fc8f4b75472a892","vpcId":"vpc-0bb4d5d8ac7aa23c4","endpointPublicAccess":true,"endpointPrivateAccess":true,"publicAccessCidrs":["130.41.129.99/32","130.41.137.252/31","134.238.84.227/32","165.1.234.42/32","208.127.246.98/32","130.41.245.146/32","130.41.233.175/32","208.127.84.6/32","18.214.178.65/32","208.127.181.64/32","208.127.70.146/31","208.127.93.113/32","165.85.67.26/32","134.238.35.199/32","134.238.36.71/32","128.77.115.86/31","134.238.252.46/32","140.209.222.130/31","128.77.31.115/32","130.41.157.27/32","137.83.217.134/32","165.85.41.194/32","208.127.75.177/32","165.1.234.10/32","208.127.243.47/32","165.1.154.138/32","208.127.125.207/32","208.127.125.202/32","130.41.157.152/32","165.85.18.172/31","208.127.181.63/32","134.238.8.60/31","208.127.84.5/32","165.85.41.91/32","162.155.7.34/32","208.127.95.135/32","130.41.129.204/32","130.41.245.145/32","165.85.67.21/32","208.127.128.120/32"]},"kubernetesNetworkConfig":{"serviceIpv4Cidr":"172.20.0.0/16","serviceIpv6Cidr":null,"ipFamily":"ipv4","elasticLoadBalancing":{"enabled":false}},"logging":{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":false}]},"identity":{"oidc":{"issuer":"https://oidc.eks.ap-northeast-1.amazonaws.com/id/AB014096DDC9254A905E2AF8A8FA0671"}},"status":"ACTIVE","certificateAuthority":{"data":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1USXdOakU1TURJd09Wb1hEVE15TVRJd016RTVNREl3T1Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBUEV0Cjd5WDFpaUZhYkQzQ1R2WTI2R01naytUVlhGbmcvckljZHNZYWNnaFI1WDFRYWRjTzNoWWNNMUF1NGJaZUlBaUIKdSsrUkgyQktnTG11MnhkM1hhMFVrbzNqT1FUdGlncVVZaUNuSC9WaHFDdCtSN1pmcU80cHdISzZGaktNczkyUApKWkpMOVpHN0NIZWlCS0U1Mzk1cjE4VEFWQUVqVFVJN3BTNDh0NzdZQjNOTE00aVFKK3MxTDV5L2I0Ymk4cHlBCkY1WnpZd2dqOEE2b1FFT0NpV2FVbm0wL0RETXIzZGRhUzI2RFRTeE1vSU5VaHA3V0t4bDhjVXcva1V6cFV4MjAKb2I1dmxWSHp4WHN1allwMUhyQ2dkQUE2elU0ZjRKMlg3Z2xnU0NMY3lYMUdKZFRQM3pXaDhWRGVaOVZ0MVloQgpCS045ajhxdzFpNTlxb0lhWi9rQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZBSXliVGUvZ3hPTHV5cS9NZ0ZJblF0MzZ5UzFNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBTmdKWEFSWEduRFdndTN3K0lqdgpvcU02b1loNHJ0czV5aTRDc1dwT2NmNGFqZitURlpQbG5XTzdYamlEVzlMMWE1QXRVNzFNcEExM3dXQ1pnTFJWCjBCNTcxc1p1V3ZtQTk5YmhMUDRVWnpwUWYwQnZUSVpadTJPeEtpWlZ2dkJFMGFWLy9lSkNzQjZ5bEdXRFhlZUMKa0tsSTFUVTU0S1hQVDVTeHVIYUo4M01HcmlFeFFzT2hJZjZ0T0RUczlFa3dYT29MTWJxRm9iamxLZHpuYnBBMAphcVA3aFlxRkIvK0h6MDRzMGNpSWQxRXQvU3hPTlRWS1JsR0owcEVpTkJoaDhvSTFvM3c2aFBadkFXUkpOM3NXClRuMFR5T1Z3dzNOckNldUNjQVVWT0tnOEN2QnBNY08rbTY4d2xOWGF3S2U1SC8wNEFxOWhmbWRXY0JzUmdvMDUKbXdNPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="},"clientRequestToken":null,"platformVersion":"eks.20","tags":{"CostCenter":"Platform","tfpath":"/platform-hosting-prod/ap-northeast-1/eks-saas-prod-apne1-2/backplane","ManagedBy":"customer-platform-infrastructure"},"encryptionConfig":[{"resources":["secrets"],"provider":{"keyArn":"arn:aws:kms:ap-northeast-1:475806706894:key/59740dc4-152d-418c-9e69-ea09b994cf63"}}],"connectorConfig":null,"id":null,"health":null,"outpostConfig":null,"accessConfig":{"bootstrapClusterCreatorAdminPermissions":null,"authenticationMode":"CONFIG_MAP"},"remoteNetworkConfig":null,"computeConfig":null,"storageConfig":null,"zonalShiftConfig":null,"deletionProtection":false,"controlPlaneScalingConfig":{"tier":"standard"},"vendedLogs":false}}
916:17:46:23.412 STDERR terraform:   |       <AssumeRolePolicyDocument>%7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Effect%22%3A%22Allow%22%2C%22Principal%22%3A%7B%22Federated%22%3A%22arn%3Aaws%3Aiam%3A%3A475806706894%3Aoidc-provider%2Foidc.eks.ap-northeast-1.amazonaws.com%2Fid%2FAB014096DDC9254A905E2AF8A8FA0671%22%7D%2C%22Action%22%3A%22sts%3AAssumeRoleWithWebIdentity%22%2C%22Condition%22%3A%7B%22StringEquals%22%3A%7B%22oidc.eks.ap-northeast-1.amazonaws.com%2Fid%2FAB014096DDC9254A905E2AF8A8FA0671%3Aaud%22%3A%22sts.amazonaws.com%22%2C%22oidc.eks.ap-northeast-1.amazonaws.com%2Fid%2FAB014096DDC9254A905E2AF8A8FA0671%3Asub%22%3A%22system%3Aserviceaccount%3Akube-system%3Aebs-csi-controller-sa%22%7D%7D%7D%5D%7D</AssumeRolePolicyDocument>
949:17:46:23.414 STDERR terraform:    http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.x_amz_security_token="*****" rpc.system.name=aws-api tf_provider_addr=registry.terraform.io/hashicorp/aws tf_rpc=ReadResource @module=aws tf_req_id=5f78eb8f-8f28-6b56-2c69-c95b0d0fe588 aws.region=ap-northeast-1 tf_resource_type=aws_iam_role http.method=POST http.request.header.content_type=application/x-www-form-urlencoded http.request.header.x_amz_date=20260513T214623Z net.peer.name=iam.amazonaws.com tf_aws.signing_region="" timestamp=2026-05-13T17:46:23.412-0400
977:17:46:23.430 STDERR terraform:    http.request.header.x_amz_date=20260513T214623Z http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) terraform-provider-aws/6.34.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.41.2 ua/2.1 os/macos lang/go#1.25.7 md/GOOS#darwin md/GOARCH#arm64 api/iam#1.53.3 m/C,v,w" tf_aws.sdk=aws-sdk-go-v2 tf_aws.signing_region="" @module=aws tf_mux_provider="*schema.GRPCProviderServer" rpc.method=ListAttachedRolePolicies http.request.header.amz_sdk_request="attempt=1; max=25" @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.70/logging/tf_logger.go:45 http.request.header.x_amz_security_token="*****" tf_rpc=ReadResource http.request.header.content_type=application/x-www-form-urlencoded http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request_content_length=94 http.url=https://iam.amazonaws.com/ net.peer.name=iam.amazonaws.com aws.region=ap-northeast-1 timestamp=2026-05-13T17:46:23.430-0400
1025:17:46:23.470 STDERR terraform:    http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.content_type=application/x-www-form-urlencoded @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.70/logging/tf_logger.go:45 http.request_content_length=92 rpc.method=ListAttachedRolePolicies tf_provider_addr=registry.terraform.io/hashicorp/aws http.request.header.amz_sdk_invocation_id=b9c2cb2e-df56-4ab1-803c-f20dca42145f http.url=https://iam.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) terraform-provider-aws/6.34.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.41.2 ua/2.1 os/macos lang/go#1.25.7 md/GOOS#darwin md/GOARCH#arm64 api/iam#1.53.3 m/C,v,w" tf_aws.signing_region="" tf_aws.sdk=aws-sdk-go-v2 tf_req_id=5f78eb8f-8f28-6b56-2c69-c95b0d0fe588 aws.region=ap-northeast-1 http.method=POST http.request.header.x_amz_security_token="*****" tf_mux_provider="*schema.GRPCProviderServer" http.request.header.x_amz_date=20260513T214623Z http.request.header.amz_sdk_request="attempt=1; max=25" net.peer.name=iam.amazonaws.com rpc.system.name=aws-api tf_rpc=ReadResource timestamp=2026-05-13T17:46:23.469-0400
1029:17:46:23.475 STDERR terraform:    http.request.header.amz_sdk_request="attempt=1; max=25" @module=aws http.method=POST http.request.header.x_amz_security_token="*****" net.peer.name=iam.amazonaws.com aws.region=ap-northeast-1 http.url=https://iam.amazonaws.com/ rpc.method=GetPolicy http.request.header.amz_sdk_invocation_id=1da8ce05-be32-4592-97f5-fb13c5b6a0a7 http.request.header.content_type=application/x-www-form-urlencoded tf_provider_addr=registry.terraform.io/hashicorp/aws timestamp=2026-05-13T17:46:23.474-0400
1051:17:46:23.485 STDERR terraform:   |       <AssumeRolePolicyDocument>%7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Effect%22%3A%22Allow%22%2C%22Principal%22%3A%7B%22Federated%22%3A%22arn%3Aaws%3Aiam%3A%3A475806706894%3Aoidc-provider%2Foidc.eks.ap-northeast-1.amazonaws.com%2Fid%2FAB014096DDC9254A905E2AF8A8FA0671%22%7D%2C%22Action%22%3A%22sts%3AAssumeRoleWithWebIdentity%22%2C%22Condition%22%3A%7B%22StringEquals%22%3A%7B%22oidc.eks.ap-northeast-1.amazonaws.com%2Fid%2FAB014096DDC9254A905E2AF8A8FA0671%3Asub%22%3A%22system%3Aserviceaccount%3Akube-system%3As3-csi-driver%22%2C%22oidc.eks.ap-northeast-1.amazonaws.com%2Fid%2FAB014096DDC9254A905E2AF8A8FA0671%3Aaud%22%3A%22sts.amazonaws.com%22%7D%7D%7D%5D%7D</AssumeRolePolicyDocument>
1081:17:46:23.486 STDERR terraform:    http.request_content_length=87 tf_resource_type=aws_iam_role http.request.header.amz_sdk_request="attempt=1; max=25" rpc.system.name=aws-api tf_aws.signing_region="" tf_mux_provider="*schema.GRPCProviderServer" tf_provider_addr=registry.terraform.io/hashicorp/aws @module=aws http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.content_type=application/x-www-form-urlencoded timestamp=2026-05-13T17:46:23.485-0400
1094:17:46:23.492 STDERR terraform:    tf_mux_provider="*schema.GRPCProviderServer" tf_resource_type=aws_iam_role http.request.header.amz_sdk_request="attempt=1; max=25" http.request.header.x_amz_date=20260513T214623Z http.request.header.x_amz_security_token="*****" http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.8 (+https://www.terraform.io) terraform-provider-aws/6.34.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.41.2 ua/2.1 os/macos lang/go#1.25.7 md/GOOS#darwin md/GOARCH#arm64 api/iam#1.53.3 m/v,w" @module=aws http.request_content_length=83 rpc.method=GetRole tf_aws.signing_region="" http.request.header.amz_sdk_invocation_id=18ad066d-1c42-477a-8289-2750e03ca16c http.url=https://iam.amazonaws.com/ rpc.system.name=aws-api http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" net.peer.name=iam.amazonaws.com timestamp=2026-05-13T17:46:23.491-0400
1125:17:46:23.497 STDERR terraform:    http.request.header.x_amz_security_token="*****" http.request_content_length=133 rpc.system.name=aws-api tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=7f211db8-b920-b858-8c45-eda05e30a9c2 @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.70/logging/tf_logger.go:45 aws.region=ap-northeast-1 http.request.header.amz_sdk_invocation_id=e14df730-da75-4d50-9b35-0a5aaaae9159 http.request.header.amz_sdk_request="attempt=1; max=25" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************J2AR/20260513/us-east-1/iam/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" tf_resource_type=aws_iam_policy http.request.header.content_type=application/x-www-form-urlencoded http.request.header.x_amz_date=20260513T214623Z http.url=https://iam.amazonaws.com/ http.us


Read @output.txt, why is the 'unauthorized' error occuring when running plans on this (and other) kubernetes clusters?


Debug Kubernetes plan unauthorized error


Bash

find /Users/steve.briggs/cpi-new -maxdepth 2 -name "terragrunt.hcl" | grep -v "/.terraform/" | head -5

Find root terragrunt.hcl


1	# ----------------------------------------------------------------------------------------------------------------
2	# This is the root configuration for customer-platform-infrastructure. Its purpose is to:
3	#
4	#   - generate a provider block to configure the Terraform provider for AWS
5	#   - generate a remote state block for storing state in S3
6	#   - define a minimal set of global inputs that may be needed by any file
7	#
8	# Each module within customer-platform-infrastructure includes this file.
9	# ----------------------------------------------------------------------------------------------------------------
10
11	# ----------------------------------------------------------------------------------------------------------------
12	# LOAD COMMON VARIABLES
13	# ----------------------------------------------------------------------------------------------------------------
14	locals {
15	  # Automatically load common variables shared across all accounts
16	  common_vars = read_terragrunt_config(find_in_parent_folders("common.hcl"))
17
18	  # Automatically load account-level variables
19	  account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))
20
21	  # Automatically load region-level variables
22	  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl", find_in_parent_folders("default-region.hcl")))
23	}
24
25	# ----------------------------------------------------------------------------------------------------------------
26	# GENERATED PROVIDER BLOCK
27	# ----------------------------------------------------------------------------------------------------------------
28	generate "provider" {
29	  path      = "provider.tf"
30	  if_exists = "overwrite_terragrunt"
31	  contents  = <<EOF
32	provider aws {
33	  profile = "immuta-${local.account_vars.locals.account_name}"
34	  region  = "${local.region_vars.locals.aws_region}"
35	  ignore_tags {
36	    key_prefixes = [
37	      "kubernetes.io/cluster/"
38	    ]
39	  }
40	}
41
42	provider vault {
43	  address = "${local.account_vars.locals.vault_addr}"
44	}
45
46	provider kubernetes {
47	  alias          = "argocd-cluster"
48	  config_path    = "~/.kube/config"
49	  config_context = "transporter.ic-ops.com-eks-platform-mgmt-use1-1"
50	}
51
52	provider google {
53	  project = "saas-certificates"
54	}
55	EOF
56	}
57	# ----------------------------------------------------------------------------------------------------------------
58	# GENERATED REMOTE STATE BLOCK
59	# ----------------------------------------------------------------------------------------------------------------
60	# Generate the Terraform remote state block for storing state in S3
61	remote_state {
62	  backend = "s3"
63	  config = {
64	    encrypt        = true
65	    bucket         = local.account_vars.locals.state_bucket
66	    key            = "${path_relative_to_include()}/terraform.tfstate"
67	    region         = local.common_vars.locals.default_region
68	    profile        = "immuta-${local.account_vars.locals.account_name}"
69	    dynamodb_table = "${local.account_vars.locals.state_bucket}"
70
71	    disable_bucket_update = true
72	    bucket_sse_algorithm  = "AES256"
73	  }
74	  generate = {
75	    path      = "backend.tf"
76	    if_exists = "overwrite_terragrunt"
77	  }
78	}
79	# ----------------------------------------------------------------------------------------------------------------
80	# DEFAULT INPUTS
81	# ----------------------------------------------------------------------------------------------------------------
82	inputs = {}
83

# ----------------------------------------------------------------------------------------------------------------
# This is the root configuration for customer-platform-infrastructure. Its purpose is to:
#
#   - generate a provider block to configure the Terraform provider for AWS
#   - generate a remote state block for storing state in S3
#   - define a minimal set of global inputs that may be needed by any file
#
# Each module within customer-platform-infrastructure includes this file.
# ----------------------------------------------------------------------------------------------------------------

# ----------------------------------------------------------------------------------------------------------------
# LOAD COMMON VARIABLES
# ----------------------------------------------------------------------------------------------------------------
locals {
  # Automatically load common variables shared across all accounts
  common_vars = read_terragrunt_config(find_in_parent_folders("common.hcl"))

  # Automatically load account-level variables
  account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))

  # Automatically load region-level variables
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl", find_in_parent_folders("default-region.hcl")))
}

# ----------------------------------------------------------------------------------------------------------------
# GENERATED PROVIDER BLOCK
# ----------------------------------------------------------------------------------------------------------------
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider aws {
  profile = "immuta-${local.account_vars.locals.account_name}"
  region  = "${local.region_vars.locals.aws_region}"
  ignore_tags {
    key_prefixes = [
      "kubernetes.io/cluster/"
    ]
  }
}

provider vault {
  address = "${local.account_vars.locals.vault_addr}"
}

provider kubernetes {
  alias          = "argocd-cluster"
  config_path    = "~/.kube/config"
  config_context = "transporter.ic-ops.com-eks-platform-mgmt-use1-1"
}

provider google {
  project = "saas-certificates"
}
EOF
}
# ----------------------------------------------------------------------------------------------------------------
# GENERATED REMOTE STATE BLOCK
# ----------------------------------------------------------------------------------------------------------------
# Generate the Terraform remote state block for storing state in S3
remote_state {
  backend = "s3"
  config = {
    encrypt        = true
    bucket         = local.account_vars.locals.state_bucket
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = local.common_vars.locals.default_region
    profile        = "immuta-${local.account_vars.locals.account_name}"
    dynamodb_table = "${local.account_vars.locals.state_bucket}"

    disable_bucket_update = true
    bucket_sse_algorithm  = "AES256"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}
# ----------------------------------------------------------------------------------------------------------------
# DEFAULT INPUTS
# ----------------------------------------------------------------------------------------------------------------
inputs = {}


Let me check the account.hcl and look at what the module generates for the Kubernetes provider config.


find /Users/steve.briggs/cpi-new/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications -name "*.tf" 2>/dev/null | grep -v ".terraform" | head -20

Find generated terraform files


1	# Set account-wide variables
2	locals {
3	  account_name = "platform-hosting-prod"
4	  domain_name  = "prod-hosting"
5	  env          = "prod"
6	  vault_addr   = "https://vault.ic-ops.net:8200"
7	  state_bucket = "immuta-tf-state-platform-hosting-prod"
8	}
9

# Set account-wide variables
locals {
  account_name = "platform-hosting-prod"
  domain_name  = "prod-hosting"
  env          = "prod"
  vault_addr   = "https://vault.ic-ops.net:8200"
  state_bucket = "immuta-tf-state-platform-hosting-prod"
}


Find tf files with provider/kubernetes auth config


1	# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2	# EKS-CLUSTER APPLICATIONS
3	# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4
5	# ---------------------------------------------------------------------------------------------------------------------
6	# SET TERRAFORM REQUIREMENTS FOR RUNNING THIS MODULE
7	# ---------------------------------------------------------------------------------------------------------------------
8	terraform {
9	  required_version = ">= 1.9.0"
10
11	  required_providers {
12	    aws = {
13	      source                = "hashicorp/aws"
14	      version               = "~> 6.0"
15	      configuration_aliases = [aws.super_certificate_zone]
16	    }
17	    kubernetes = {
18	      source                = "hashicorp/kubernetes"
19	      configuration_aliases = [kubernetes.argocd-cluster]
20	    }
21	    helm = {
22	      source                = "hashicorp/helm"
23	      configuration_aliases = [helm.local-cluster]
24	    }
25	  }
26	}
27
28	# ---------------------------------------------------------------------------------------------------------------------
29	# GET DATA FROM AN EXISTING RESOURCE
30	# ---------------------------------------------------------------------------------------------------------------------
31	data "aws_caller_identity" "current" {}
32
33	data "aws_iam_session_context" "current" {
34	  arn = data.aws_caller_identity.current.arn
35	}
36
37	data "aws_eks_cluster" "this" {
38	  name = var.cluster_id
39	}
40
41	data "aws_eks_cluster_auth" "this" {
42	  name = data.aws_eks_cluster.this.id
43	}
44
45	provider "kubernetes" {
46	  alias = "local-cluster"
47
48	  host                   = var.cluster_endpoint
49	  cluster_ca_certificate = base64decode(var.cluster_ca)
50	  token                  = data.aws_eks_cluster_auth.this.token
51	}
52
53	provider "helm" {
54	  alias = "local-cluster"
55
56	  kubernetes {
57	    host                   = var.cluster_endpoint
58	    cluster_ca_certificate = base64decode(var.cluster_ca)
59	    token                  = data.aws_eks_cluster_auth.this.token
60	  }
61	}
62
63	provider "random" {
64	  alias = "local-cluster"
65	}
66
67	locals {
68	  cluster_type     = var.cluster_type != "" ? var.cluster_type : split("-", var.cluster_name)[1]
69	  account_id       = data.aws_caller_identity.current.account_id
70	  current_role_arn = data.aws_iam_session_context.current.issuer_arn
71	  view_verbs       = ["get", "list", "watch"]
72	  admin_verbs      = ["delete", "deletecollection", "get", "list", "patch", "create", "update", "watch"]
73
74	  app_deploy_repo = {
75	    "mgmt"   = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment.git"
76	    "cps"    = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment.git"
77	    "saas"   = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment.git"
78	    "shared" = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment-infra.git"
79	    "iim"    = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment-infra.git"
80	    "data"   = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment-infra.git"
81	  }
82
83	  app_deploy_repo_github = {
84	    "mgmt"   = "ssh://git@github.com:22/immuta/application-deployment.git"
85	    "cps"    = "ssh://git@github.com:22/immuta/application-deployment.git"
86	    "saas"   = "ssh://git@github.com:22/immuta/application-deployment.git"
87	    "shared" = "ssh://git@github.com:22/immuta/application-deployment-infra.git"
88	    "iim"    = "ssh://git@github.com:22/immuta/application-deployment-infra.git"
89	    "data"   = "ssh://git@github.com:22/immuta/application-deployment-infra.git"
90	  }
91	}
92
93
94	# ---------------------------------------------------------------------------------------------------------------------
95	# MODULE PARAMETERS
96	# These variables are expected to be passed in by the operator when calling this terraform module.
97	# ---------------------------------------------------------------------------------------------------------------------
98
99	# ---------------------------------------------------------------------------------------------------------------------
100	# REQUIRED PARAMETERS
101	# These variables must be set.
102	# ---------------------------------------------------------------------------------------------------------------------
103	variable "cluster_name" {
104	  description = "Cluster name for the EKS cluster"
105	  type        = string
106	}
107
108	variable "cluster_version" {
109	  description = "Cluster version for the EKS cluster"
110	  type        = string
111	}
112
113	variable "env" {
114	  description = "Environment where the cluster is"
115	  type        = string
116	}
117
118	variable "account_name" {
119	  description = "Account name where the cluster is"
120	  type        = string
121	}
122
123	variable "global_segment" {
124	  description = "Global segment for the cluster, i.e. na, eu, ap"
125	  type        = string
126	  default     = ""
127	}
128
129	variable "region" {
130	  description = "Region where the cluster is"
131	  type        = string
132	}
133
134	variable "vpc_id" {
135	  description = "VPC in which to deploy the cluster"
136	  type        = string
137	}
138
139	variable "subnet_ids" {
140	  description = "List of subnets to deploy the cluster into"
141	  type        = list(string)
142	}
143
144	variable "cluster_endpoint" {
145	  description = "Endpoint for the cluster from the EKS CLUSTER module"
146	  type        = string
147	}
148
149	variable "cluster_ca" {
150	  description = "Certificate Authority for the cluster"
151	  type        = string
152	}
153
154	variable "cluster_oidc_provider_issuer" {
155	  description = "ODIC Provider Issuer for the current working cluster"
156	  type        = string
157	}
158
159	variable "cluster_id" {
160	  description = "ID of the working cluster (should be the same as the cluster name)"
161	  type        = string
162	}
163
164	variable "cluster_oidc_provider_arn" {
165	  description = "OIDC Provider ARN for the current working cluster"
166	  type        = string
167	}
168	# ---------------------------------------------------------------------------------------------------------------------
169	# OPTIONAL PARAMETERS
170	# These variables are configurable, but have reasonable defaults set.
171	# ---------------------------------------------------------------------------------------------------------------------
172	variable "cluster_type" {
173	  description = "The type of cluster, i.e. saas, mgmt, cps, will default to extracting the type from the cluster name"
174	  type        = string
175	  default     = ""
176	}
177
178	variable "cluster_clustermesh_id" {
179	  description = "Cluster ID for clustermesh (must be set for ClusterMesh)"
180	  type        = number
181	  default     = 0
182	}
183
184	variable "cluster_worker_sg" {
185	  description = "Security Group for worker nodes"
186	  type        = string
187	  default     = ""
188	  validation {
189	    condition     = var.cluster_worker_sg == "" && local.cilium_enabled && var.cluster_mesh_enabled ? false : true
190	    error_message = "cluster_worker_sg variable must be set when cilium and clustermesh is enabled"
191	  }
192	}
193
194	variable "tags" {
195	  description = "Map of key-value pairs to tag created resources with."
196	  type        = map(any)
197	  default     = {}
198	}
199
200	variable "vault_endpoint" {
201	  description = "Endpoint for vault, used in helm values for vault sidecar injector"
202	  type        = string
203	  default     = "https://vault.infrastructure.immuta.io:8200"
204	}
205

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# EKS-CLUSTER APPLICATIONS
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# ---------------------------------------------------------------------------------------------------------------------
# SET TERRAFORM REQUIREMENTS FOR RUNNING THIS MODULE
# ---------------------------------------------------------------------------------------------------------------------
terraform {
  required_version = ">= 1.9.0"

  required_providers {
    aws = {
      source                = "hashicorp/aws"
      version               = "~> 6.0"
      configuration_aliases = [aws.super_certificate_zone]
    }
    kubernetes = {
      source                = "hashicorp/kubernetes"
      configuration_aliases = [kubernetes.argocd-cluster]
    }
    helm = {
      source                = "hashicorp/helm"
      configuration_aliases = [helm.local-cluster]
    }
  }
}

# ---------------------------------------------------------------------------------------------------------------------
# GET DATA FROM AN EXISTING RESOURCE
# ---------------------------------------------------------------------------------------------------------------------
data "aws_caller_identity" "current" {}

data "aws_iam_session_context" "current" {
  arn = data.aws_caller_identity.current.arn
}

data "aws_eks_cluster" "this" {
  name = var.cluster_id
}

data "aws_eks_cluster_auth" "this" {
  name = data.aws_eks_cluster.this.id
}

provider "kubernetes" {
  alias = "local-cluster"

  host                   = var.cluster_endpoint
  cluster_ca_certificate = base64decode(var.cluster_ca)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  alias = "local-cluster"

  kubernetes {
    host                   = var.cluster_endpoint
    cluster_ca_certificate = base64decode(var.cluster_ca)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "random" {
  alias = "local-cluster"
}

locals {
  cluster_type     = var.cluster_type != "" ? var.cluster_type : split("-", var.cluster_name)[1]
  account_id       = data.aws_caller_identity.current.account_id
  current_role_arn = data.aws_iam_session_context.current.issuer_arn
  view_verbs       = ["get", "list", "watch"]
  admin_verbs      = ["delete", "deletecollection", "get", "list", "patch", "create", "update", "watch"]

  app_deploy_repo = {
    "mgmt"   = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment.git"
    "cps"    = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment.git"
    "saas"   = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment.git"
    "shared" = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment-infra.git"
    "iim"    = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment-infra.git"
    "data"   = "ssh://git@gitlab.infrastructure.immuta.io:22/infrastructure/kubernetes/application-deployment-infra.git"
  }

  app_deploy_repo_github = {
    "mgmt"   = "ssh://git@github.com:22/immuta/application-deployment.git"
    "cps"    = "ssh://git@github.com:22/immuta/application-deployment.git"
    "saas"   = "ssh://git@github.com:22/immuta/application-deployment.git"
    "shared" = "ssh://git@github.com:22/immuta/application-deployment-infra.git"
    "iim"    = "ssh://git@github.com:22/immuta/application-deployment-infra.git"
    "data"   = "ssh://git@github.com:22/immuta/application-deployment-infra.git"
  }
}

# ---------------------------------------------------------------------------------------------------------------------
# MODULE PARAMETERS
# These variables are expected to be passed in by the operator when calling this terraform module.
# ---------------------------------------------------------------------------------------------------------------------

# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# These variables must be set.
# ---------------------------------------------------------------------------------------------------------------------
variable "cluster_name" {
  description = "Cluster name for the EKS cluster"
  type        = string
}

variable "cluster_version" {
  description = "Cluster version for the EKS cluster"
  type        = string
}

variable "env" {
  description = "Environment where the cluster is"
  type        = string
}

variable "account_name" {
  description = "Account name where the cluster is"
  type        = string
}

variable "global_segment" {
  description = "Global segment for the cluster, i.e. na, eu, ap"
  type        = string
  default     = ""
}

variable "region" {
  description = "Region where the cluster is"
  type        = string
}

variable "vpc_id" {
  description = "VPC in which to deploy the cluster"
  type        = string
}

variable "subnet_ids" {
  description = "List of subnets to deploy the cluster into"
  type        = list(string)
}

variable "cluster_endpoint" {
  description = "Endpoint for the cluster from the EKS CLUSTER module"
  type        = string
}

variable "cluster_ca" {
  description = "Certificate Authority for the cluster"
  type        = string
}

variable "cluster_oidc_provider_issuer" {
  description = "ODIC Provider Issuer for the current working cluster"
  type        = string
}

variable "cluster_id" {
  description = "ID of the working cluster (should be the same as the cluster name)"
  type        = string
}

variable "cluster_oidc_provider_arn" {
  description = "OIDC Provider ARN for the current working cluster"
  type        = string
}
# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# These variables are configurable, but have reasonable defaults set.
# ---------------------------------------------------------------------------------------------------------------------
variable "cluster_type" {
  description = "The type of cluster, i.e. saas, mgmt, cps, will default to extracting the type from the cluster name"
  type        = string
  default     = ""
}

variable "cluster_clustermesh_id" {
  description = "Cluster ID for clustermesh (must be set for ClusterMesh)"
  type        = number
  default     = 0
}

variable "cluster_worker_sg" {
  description = "Security Group for worker nodes"
  type        = string
  default     = ""
  validation {
    condition     = var.cluster_worker_sg == "" && local.cilium_enabled && var.cluster_mesh_enabled ? false : true
    error_message = "cluster_worker_sg variable must be set when cilium and clustermesh is enabled"
  }
}

variable "tags" {
  description = "Map of key-value pairs to tag created resources with."
  type        = map(any)
  default     = {}
}

variable "vault_endpoint" {
  description = "Endpoint for vault, used in helm values for vault sidecar injector"
  type        = string
  default     = "https://vault.infrastructure.immuta.io:8200"
}


Found the auth mechanism. Let me check the provider versions and see if there's a known issue.


find /Users/steve.briggs/cpi-new/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications/.terragrunt-cache -name ".terraform.lock.hcl" 2>/dev/null | head -3

Find terraform lock file


grep -A3 "provider\|version" /Users/steve.briggs/cpi-new/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications/.terragrunt-cache/5pBA7tZNQg_9tN_Wbrz7v-Ygpm0/502jehoMJZmlgT3wmortMOb-A9A/.terraform.lock.hcl | grep -E "registry|version" | head -30

Check provider versions in lock file


provider "registry.terraform.io/hashicorp/aws" {
  version     = "6.34.0"
provider "registry.terraform.io/hashicorp/external" {
  version = "2.3.5"
provider "registry.terraform.io/hashicorp/google" {
  version     = "6.50.0"
provider "registry.terraform.io/hashicorp/helm" {
  version     = "2.17.0"
provider "registry.terraform.io/hashicorp/kubernetes" {
  version     = "2.38.0"
provider "registry.terraform.io/hashicorp/random" {
  version = "3.8.1"
provider "registry.terraform.io/hashicorp/vault" {
  version     = "4.8.0"


Let me check what changed in the modified terragrunt.hcl and the blueprint file used.


git -C /Users/steve.briggs/cpi-new diff HEAD -- prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications/terragrunt.hcl

Show diff of modified terragrunt.hcl


find /Users/steve.briggs/cpi-new/prod-hosting -path "*/_blueprints/eks-saas/2026-05-12.hcl" 2>/dev/null | head -3

Find the blueprint file referenced


diff --git a/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications/terragrunt.hcl b/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications/terragrunt.hcl
index fadf31a6..aa569d25 100644
--- a/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications/terragrunt.hcl
+++ b/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications/terragrunt.hcl
@@ -83,7 +83,7 @@ dependency vpc {
 }

 dependency mgmt_vpc_cloudwan {
-  config_path = "../../../../../customer-platform-infrastructure/platform-management/us-east-1/vpc-cloudwan"
+  config_path = "../../../../platform-management/us-east-1/vpc-cloudwan"
 }

 dependency ingress_cert {


(Bash completed with no output)


find /Users/steve.briggs/cpi-new -path "*/_blueprints/eks-saas/2026-05-12.hcl" 2>/dev/null | head -3

Find the blueprint file in the whole repo


Check module version in blueprint


# Check when this blueprint was introduced vs the previous one
ls /Users/steve.briggs/cpi-new/_blueprints/eks-saas/ | sort

List all blueprint versions


  cluster_version = "1.34"
  eks_cluster_applications_module = "7.2.6"


.rw-r--r-- steve.briggs staff 1.8 KB Wed May 13 17:16:25 2026 2026-01-08.hcl
.rw-r--r-- steve.briggs staff 1.8 KB Wed May 13 17:16:25 2026 2026-01-20.hcl
.rw-r--r-- steve.briggs staff 1.8 KB Wed May 13 17:16:25 2026 2026-01-22.hcl
.rw-r--r-- steve.briggs staff 1.8 KB Wed May 13 17:16:25 2026 2026-02-05.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-02-06.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-02-18.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-02-19.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-03-03.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-03-11.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-03-18.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-04-08.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-04-10.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-04-15.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-04-20.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-05-01.hcl
.rw-r--r-- steve.briggs staff 2.0 KB Wed May 13 17:16:25 2026 2026-05-05.hcl
.rw-r--r-- steve.briggs staff 2.1 KB Wed May 13 17:16:25 2026 2026-05-07.hcl
.rw-r--r-- steve.briggs staff 2.1 KB Wed May 13 17:16:25 2026 2026-05-11.hcl
.rw-r--r-- steve.briggs staff 2.1 KB Wed May 13 17:17:11 2026 2026-05-12.hcl
.rw-r--r-- steve.briggs staff 463 B  Wed May 13 17:16:25 2026 .hcl
drwxr-xr-x steve.briggs staff 128 B  Wed May 13 17:16:25 2026 ..
drwxr-xr-x steve.briggs staff 3.8 KB Wed May 13 17:16:25 2026 archive
drwxr-xr-x steve.briggs staff 736 B  Wed May 13 17:17:11 2026 .


Let me compare the blueprint to the previous version and check what IAM role is being used.


Compare yesterday's and today's blueprints


aws sts get-caller-identity --profile immuta-platform-hosting-prod 2>&1

Check which IAM role is being used for this account


eks_cluster_applications_module = "7.2.3"                   |   eks_cluster_applications_module = "7.2.6"


Let me check how the current role ARN is used in the module and look at the backplane access config.


grep -rn "current_role_arn\|access_entry\|aws_auth\|aws-auth" /Users/steve.briggs/cpi-new/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications/.terragrunt-cache/5pBA7tZNQg_9tN_Wbrz7v-Ygpm0/502jehoMJZmlgT3wmortMOb-A9A/ 2>/dev/null | grep -v ".terraform/"

Search for access entry and current_role_arn usage in module


find /Users/steve.briggs/cpi-new/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/backplane -name "*.hcl" 2>/dev/null | head -5

Find backplane hcl config


grep -n "access_entry\|aws_auth\|admin\|sre\|current_role\|iam_role\|caller" /Users/steve.briggs/cpi-new/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/backplane/terragrunt.hcl | head -30

Search backplane for access entry / IAM role config


114:  sso_admin_roles = {


110	    [local.common_vars.locals.office_public_ips.columbus],
111	    [for k, v in dependency.mgmt_vpc_cloudwan.outputs.vpc.nat_gateway_map : format("%s/32", v.public_ip)],
112	  )
113
114	  sso_admin_roles = {
115	    "AdministratorAccess" = ["system:masters"]
116	    "SuperUser"           = ["system:masters"]
117	  }
118
119	  release_version = local.ami_version
120
121	  tags = local.tags
122	}
123

    [local.common_vars.locals.office_public_ips.columbus],
    [for k, v in dependency.mgmt_vpc_cloudwan.outputs.vpc.nat_gateway_map : format("%s/32", v.public_ip)],
  )

  sso_admin_roles = {
    "AdministratorAccess" = ["system:masters"]
    "SuperUser"           = ["system:masters"]
  }

  release_version = local.ami_version

  tags = local.tags
}


grep -n "sso_admin\|access_entry\|aws_auth\|admin_role\|SRE\|sre" /Users/steve.briggs/cpi-new/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/backplane/terragrunt.hcl | head -20

Check how SSO roles are configured in backplane


Check previous blueprint versions


grep -n "sso_admin\|access_entries\|access_entry\|aws_eks_access" /Users/steve.briggs/cpi-new/prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/backplane/terragrunt.hcl | head -20

Find access entry config in backplane


  cluster_version = "1.34"
  eks_cluster_applications_module = "7.2.3"


1	# ---------------------------------------------------------------------------------------------------------------------
2	# TERRAGRUNT CONFIGURATION
3	# ---------------------------------------------------------------------------------------------------------------------
4
5	terraform {
6	  source = "tfr://crow.infrastructure.immuta.io/iac/compute-eks-cluster-backplane/aws?version=${local.eks_cluster_backplane_module}"
7	  # after_hook "kubeconfig" {
8	  #   commands = ["apply", "output"]
9	  #   execute  = ["bash", "-c", "if [[ $(grep -ic transporter ${local.local_kube_cfg_path}) == 0 ]]; then aws eks update-kubeconfig --region ${local.aws_region} --name ${local.cluster_name} --kubeconfig ${local.local_kube_cfg_path}; fi"]
10	  # }
11	}
12
13	# Include all settings from the root terragrunt.hcl file
14	include {
15	  path = find_in_parent_folders()
16	}
17
18	# platform management account provider to access ic-ops.net domain
19	generate "domain_account_provider" {
20	  path      = "domain-account-provider.tf"
21	  if_exists = "overwrite_terragrunt"
22	  contents  = <<EOF
23	provider aws {
24	  alias   = "domain_account"
25	  profile = "${local.accounts["infrastructure"]["profile"]}"
26	  region  = "${local.aws_region}"
27	}
28	EOF
29	}
30
31	# ---------------------------------------------------------------------------------------------------------------------
32	# Locals are named constants that are reusable within the configuration.
33	# ---------------------------------------------------------------------------------------------------------------------
34	locals {
35	  cluster_name = "eks-saas-prod-apne1-2"
36	  cluster_version = local.blueprint_vars.locals.cluster_version
37	  local_kube_cfg_path = format("%s/../%s", get_terragrunt_dir(), "kubeconfig")
38	  # Automatically load common variables shared across all accounts
39	  common_vars = read_terragrunt_config(find_in_parent_folders("common.hcl"))
40
41	  # Automatically load account-level variables
42	  account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))
43
44	  # Automatically load region-level variables
45	  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
46
47	  # Extract the region for easy access
48	  aws_region = local.region_vars.locals.aws_region
49	  env        = local.account_vars.locals.env
50	  cluster_dir = basename(dirname(get_terragrunt_dir()))
51	  #current_dir = run_cmd("--terragrunt-quiet", "basename", "${get_terragrunt_dir()}../")
52	  current_dir = basename(get_terragrunt_dir())
53
54	  # Automatically load blueprint for versioning
55	  blueprint_vars = read_terragrunt_config(find_in_parent_folders("_blueprints/eks-saas/2026-04-20.hcl"))
56
57	  # Extract blueprint
58	  app_blueprint   = local.blueprint_vars.locals.app_versions
59	  ami_version     = local.blueprint_vars.locals.ami_version
60
61	  eks_cluster_backplane_module = local.blueprint_vars.locals.eks_cluster_backplane_module
62	  # A local for more convenient access to the accounts map.
63	  accounts     = local.common_vars.locals.accounts
64	  account_name = local.account_vars.locals.account_name
65	  account_id   = local.accounts[replace(local.account_vars.locals.domain_name, "-", "_")].id
66
67	  spot_types = local.common_vars.locals.saas_spot_instance_types
68
69	  tags = merge(local.common_vars.locals.global_tags, {
70	    tfpath = "/${local.account_vars.locals.account_name}/${local.aws_region}/${local.cluster_dir}/${local.current_dir}"
71	  })
72	}
73
74	# ---------------------------------------------------------------------------------------------------------------------
75	# MODULE DEPENDENCIES
76	# ---------------------------------------------------------------------------------------------------------------------
77
78	dependency vpc {
79	  config_path = "../../vpc-cloudwan"
80	}
81
82	dependency mgmt_vpc_cloudwan {
83	  config_path = "../../../../platform-management/us-east-1/vpc-cloudwan"
84	}
85
86	# ---------------------------------------------------------------------------------------------------------------------
87	# MODULE PARAMETERS
88	# These are the variables we have to pass in to use the module specified in the terragrunt configuration above
89	# ---------------------------------------------------------------------------------------------------------------------
90	inputs = {
91	  cluster_name               = local.cluster_name
92	  cluster_version            = local.cluster_version
93	  env                        = local.env
94	  region                     = local.aws_region
95
96	  public_access       = true
97	  public_access_cidrs = concat(
98	    local.common_vars.locals.globalprotect_public_ips,
99	    [local.common_vars.locals.office_public_ips.columbus],
100	    [for k, v in dependency.mgmt_vpc_cloudwan.outputs.vpc.nat_gateway_map : format("%s/32", v.public_ip)],
101	  )
102
103	  vpc_id = dependency.vpc.outputs.vpc["vpc_id"]
104
105	  private_subnet_ids   = dependency.vpc.outputs.vpc["private_subnet_ids"]
106	  private_access_cidrs = [dependency.mgmt_vpc_cloudwan.outputs.vpc.vpc_cidr_block]
107	  public_access        = true
108	  public_access_cidrs = concat(
109	    local.common_vars.locals.globalprotect_public_ips,
110	    [local.common_vars.locals.office_public_ips.columbus],
111	    [for k, v in dependency.mgmt_vpc_cloudwan.outputs.vpc.nat_gateway_map : format("%s/32", v.public_ip)],
112	  )
113
114	  sso_admin_roles = {
115	    "AdministratorAccess" = ["system:masters"]
116	    "SuperUser"           = ["system:masters"]
117	  }
118
119	  release_version = local.ami_version
120
121	  tags = local.tags
122	}
123

# ---------------------------------------------------------------------------------------------------------------------
# TERRAGRUNT CONFIGURATION
# ---------------------------------------------------------------------------------------------------------------------

terraform {
  source = "tfr://crow.infrastructure.immuta.io/iac/compute-eks-cluster-backplane/aws?version=${local.eks_cluster_backplane_module}"
  # after_hook "kubeconfig" {
  #   commands = ["apply", "output"]
  #   execute  = ["bash", "-c", "if [[ $(grep -ic transporter ${local.local_kube_cfg_path}) == 0 ]]; then aws eks update-kubeconfig --region ${local.aws_region} --name ${local.cluster_name} --kubeconfig ${local.local_kube_cfg_path}; fi"]
  # }
}

# Include all settings from the root terragrunt.hcl file
include {
  path = find_in_parent_folders()
}

# platform management account provider to access ic-ops.net domain
generate "domain_account_provider" {
  path      = "domain-account-provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider aws {
  alias   = "domain_account"
  profile = "${local.accounts["infrastructure"]["profile"]}"
  region  = "${local.aws_region}"
}
EOF
}

# ---------------------------------------------------------------------------------------------------------------------
# Locals are named constants that are reusable within the configuration.
# ---------------------------------------------------------------------------------------------------------------------
locals {
  cluster_name = "eks-saas-prod-apne1-2"
  cluster_version = local.blueprint_vars.locals.cluster_version
  local_kube_cfg_path = format("%s/../%s", get_terragrunt_dir(), "kubeconfig")
  # Automatically load common variables shared across all accounts
  common_vars = read_terragrunt_config(find_in_parent_folders("common.hcl"))

  # Automatically load account-level variables
  account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))

  # Automatically load region-level variables
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))

  # Extract the region for easy access
  aws_region = local.region_vars.locals.aws_region
  env        = local.account_vars.locals.env
  cluster_dir = basename(dirname(get_terragrunt_dir()))
  #current_dir = run_cmd("--terragrunt-quiet", "basename", "${get_terragrunt_dir()}../")
  current_dir = basename(get_terragrunt_dir())

  # Automatically load blueprint for versioning
  blueprint_vars = read_terragrunt_config(find_in_parent_folders("_blueprints/eks-saas/2026-04-20.hcl"))

  # Extract blueprint
  app_blueprint   = local.blueprint_vars.locals.app_versions
  ami_version     = local.blueprint_vars.locals.ami_version

  eks_cluster_backplane_module = local.blueprint_vars.locals.eks_cluster_backplane_module
  # A local for more convenient access to the accounts map.
  accounts     = local.common_vars.locals.accounts
  account_name = local.account_vars.locals.account_name
  account_id   = local.accounts[replace(local.account_vars.locals.domain_name, "-", "_")].id

  spot_types = local.common_vars.locals.saas_spot_instance_types

  tags = merge(local.common_vars.locals.global_tags, {
    tfpath = "/${local.account_vars.locals.account_name}/${local.aws_region}/${local.cluster_dir}/${local.current_dir}"
  })
}

# ---------------------------------------------------------------------------------------------------------------------
# MODULE DEPENDENCIES
# ---------------------------------------------------------------------------------------------------------------------

dependency vpc {
  config_path = "../../vpc-cloudwan"
}

dependency mgmt_vpc_cloudwan {
  config_path = "../../../../platform-management/us-east-1/vpc-cloudwan"
}

# ---------------------------------------------------------------------------------------------------------------------
# MODULE PARAMETERS
# These are the variables we have to pass in to use the module specified in the terragrunt configuration above
# ---------------------------------------------------------------------------------------------------------------------
inputs = {
  cluster_name               = local.cluster_name
  cluster_version            = local.cluster_version
  env                        = local.env
  region                     = local.aws_region

  public_access       = true
  public_access_cidrs = concat(
    local.common_vars.locals.globalprotect_public_ips,
    [local.common_vars.locals.office_public_ips.columbus],
    [for k, v in dependency.mgmt_vpc_cloudwan.outputs.vpc.nat_gateway_map : format("%s/32", v.public_ip)],
  )

  vpc_id = dependency.vpc.outputs.vpc["vpc_id"]

  private_subnet_ids   = dependency.vpc.outputs.vpc["private_subnet_ids"]
  private_access_cidrs = [dependency.mgmt_vpc_cloudwan.outputs.vpc.vpc_cidr_block]
  public_access        = true
  public_access_cidrs = concat(
    local.common_vars.locals.globalprotect_public_ips,
    [local.common_vars.locals.office_public_ips.columbus],
    [for k, v in dependency.mgmt_vpc_cloudwan.outputs.vpc.nat_gateway_map : format("%s/32", v.public_ip)],
  )

  sso_admin_roles = {
    "AdministratorAccess" = ["system:masters"]
    "SuperUser"           = ["system:masters"]
  }

  release_version = local.ami_version

  tags = local.tags
}


Now I have the full picture. Here's the root cause, fr fr no cap Sir Charles of St Cheese:

## Why you're getting 401 Unauthorized

The auth flow is:

1. The `compute-eks-cluster-applications` module (`7.2.6`) uses `data.aws_eks_cluster_auth` (`common.tf:41-43`) to generate a pre-signed STS token
2. That token is minted for your current IAM identity — `AWSReservedSSO_SRE-Admin_dfb454cacb89fc0c`
3. EKS receives the token, verifies it with AWS, identifies the caller... and has no record of it → 401

**The real issue**: The `SRE-Admin` SSO permission set is not in `sso_admin_roles` in the backplane module (`backplane/terragrunt.hcl:114-117`):

```hcl
sso_admin_roles = {
  "AdministratorAccess" = ["system:masters"]
  "SuperUser"           = ["system:masters"]
  # SRE-Admin is missing here
}
```

The backplane module uses `sso_admin_roles` to create EKS access entries (or `aws-auth` mappings) for SSO roles. Since `SRE-Admin` isn't listed, your role has zero EKS RBAC authorization on any of these clusters. The AWS API calls work fine (S3, IAM, STS are all hitting 200s) because those use the same credentials directly — but EKS needs an explicit mapping.

**This hits all clusters** because every backplane config has the same `sso_admin_roles` map that excludes `SRE-Admin`.

## Fix

Add `SRE-Admin` to `sso_admin_roles` in the backplane configs:

```hcl
sso_admin_roles = {
  "AdministratorAccess" = ["system:masters"]
  "SuperUser"           = ["system:masters"]
  "SRE-Admin"           = ["system:masters"]
}
```

Then apply the backplane first, which will create the EKS access entry for your role. After that, the applications plan will be able to authenticate. This needs to happen on every affected cluster's backplane.

Alternatively, if this is a short-term unblock, you can switch to the `AdministratorAccess` or `SuperUser` AWS profile to run the plans — those roles already have access.


afplay ~/.claude/sounds/mario/stop/stomp.wav 2>/dev/null || true


how do you know or how can i tell that my AWS user is using "SRE-Admin" ?


I ran this earlier in the investigation:

```
aws sts get-caller-identity --profile immuta-platform-hosting-prod
```

Which returned:

```json
{
    "UserId": "AROAW5SCKFTHNR6JF6DGD:steve.briggs@immuta.com",
    "Account": "475806706894",
    "Arn": "arn:aws:sts::475806706894:assumed-role/AWSReservedSSO_SRE-Admin_dfb454cacb89fc0c/steve.briggs@immuta.com"
}
```

The `Arn` field tells you everything — `assumed-role/AWSReservedSSO_SRE-Admin_dfb454cacb89fc0c/steve.briggs@immuta.com`. The pattern is `AWSReservedSSO_<PermissionSetName>_<hash>`, so the permission set name is `SRE-Admin`.

That's the command to run any time you want to know what IAM identity your current AWS profile is resolving to.


how can i switch to AdministratorAccess?


grep -A5 "AdministratorAccess\|platform-hosting-prod\|prod-hosting" ~/.aws/config | head -60

Check AWS config for available profiles


sso_role_name = AdministratorAccess
sso_start_url = https://immuta.awsapps.com/start

[profile immuta]
credential_process = aws-vault exec immuta-sso --json
region = us-east-1
--
# sso_role_name = AdministratorAccess
# sso_start_url = https://immuta.awsapps.com/start
#
# [profile immuta-tf]
# credential_process = aws-vault exec immuta-tf-sso --json
# region = us-east-1
--
sso_role_name = AdministratorAccess
sso_start_url = https://immuta.awsapps.com/start

[profile immuta-federal-team-admin]
credential_process = aws-vault exec immuta-federal-team-admin-sso --json
region = us-east-1
--
sso_role_name = AdministratorAccess
sso_start_url = https://immuta.awsapps.com/start

[profile immuta-partner-admin]
credential_process = aws-vault exec immuta-partner-admin-sso --json
region = us-east-1
--
sso_role_name = AdministratorAccess
sso_start_url = https://immuta.awsapps.com/start

[profile immuta-admin]
credential_process = aws-vault exec immuta-admin-sso --json
region = us-east-1
--
sso_role_name = AdministratorAccess
sso_start_url = https://immuta.awsapps.com/start

[profile immuta-infrastructure-admin]
credential_process = aws-vault exec immuta-infrastructure-admin-sso --json
region = us-east-1
--
sso_role_name = AdministratorAccess
sso_start_url = https://immuta.awsapps.com/start

[profile immuta-infrastructure]
credential_process = aws-vault exec immuta-infrastructure-sso --json
region = us-east-1
--
sso_role_name = AdministratorAccess
sso_start_url = https://immuta.awsapps.com/start

[profile immuta-build-automation-admin]
credential_process = aws-vault exec immuta-build-automation-admin-sso --json
region = us-east-1
--
sso_role_name = AdministratorAccess
sso_start_url = https://immuta.awsapps.com/start

[profile immuta-build-automation]


grep -B2 "platform-hosting-prod\|AdministratorAccess" ~/.aws/config | grep -A2 "platform-hosting-prod" | head -30

Find platform-hosting-prod AdministratorAccess profile


[profile immuta-platform-hosting-prod-sso]
--
sso_start_url = https://immuta.awsapps.com/start
--
[profile immuta-platform-hosting-prod]
credential_process = aws-vault exec immuta-platform-hosting-prod-sso --json
--
# sso_account_id = 066661064610


grep -A8 "\[profile immuta-platform-hosting-prod-sso\]" ~/.aws/config

Show full platform-hosting-prod SSO profile config


The user doesn't want to proceed with this tool use. The tool use was rejected (eg. if it was a file edit, the new_string was NOT written to the file). STOP what you are doing and wait for the user to tell you how to proceed.


[Request interrupted by user for tool use]


<command-message>obsidian-save</command-message>
<command-name>/obsidian-save</command-name>


Save current conversation context to an Obsidian note under the `Claude/` directory.

## What to save

Capture all of the following that are relevant to the current conversation:

- **Conversation summary**: What the user is trying to accomplish and why
- **Project context**: Relevant architecture, design decisions, or constraints discovered
- **Task state**: What has been completed, what is in progress, what is blocked or pending
- **Key decisions**: Any choices made and the reasoning behind them
- **Open questions**: Unresolved questions or things to revisit
- **Next steps**: Concrete actions that should follow from this session

## File location

Derive the project name from the current working directory (use the repo or app folder name). Save to:

```
Claude/<project-name>/<YYYY-MM-DD>-<short-slug>.md
```

Use today's date and a short 2-4 word slug describing the session topic (e.g. `2026-04-29-ruth-plugin-setup.md`).

If `` is provided, use it as the slug instead of generating one.

## Note format

Use this structure:

```markdown
---
date: <YYYY-MM-DD>
project: <project-name>
tags: [claude-context]
---

# <Session Title>

## Summary
<1-3 sentence overview of what this session was about>

## Context
<Relevant project background, architecture notes, constraints>

## Decisions
<Decisions made and rationale>

## Task State
<What was completed, in progress, or blocked>

## Open Questions
<Anything unresolved>

## Next Steps
<Concrete follow-up actions>
```

## Steps

1. Determine the project name from the working directory.
2. Determine today's date.
3. Draft the note content based on the current conversation.
4. Use `mcp__obsidian__create_vault_file` to write the note to the vault.
5. Confirm to the user where the note was saved.


---
date: 2026-05-13
project: cpi-new
tags: [claude-context]
---

# EKS Terraform Plan 401 Unauthorized Investigation

## Summary
Investigated why `terragrunt plan` against `eks-saas-prod-apne1-2/applications` (and other EKS clusters) was failing with 401 Unauthorized errors from the Kubernetes and Helm Terraform providers. Root cause identified: the current SSO role (`SRE-Admin`) is not in the EKS cluster access configuration.

## Context
- Repo: `cpi-new` (customer-platform-infrastructure), branch `SRE-2806`
- Working dir: `prod-hosting/ap-northeast-1/eks-saas-prod-apne1-2/applications`
- Blueprint: `_blueprints/eks-saas/2026-05-12.hcl` — module `compute-eks-cluster-applications` version `7.2.6` (bumped from `7.2.3` in `2026-05-11.hcl`)
- AWS account: `475806706894` (platform-hosting-prod), profile: `immuta-platform-hosting-prod`
- Current IAM identity: `arn:aws:sts::475806706894:assumed-role/AWSReservedSSO_SRE-Admin_dfb454cacb89fc0c/steve.briggs@immuta.com`

### Auth mechanism
The `compute-eks-cluster-applications` module (`common.tf`) authenticates to EKS using:
```hcl
data "aws_eks_cluster_auth" "this" {
  name = data.aws_eks_cluster.this.id
}
provider "kubernetes" {
  token = data.aws_eks_cluster_auth.this.token
  ...
}
```
This generates a pre-signed STS token for the current IAM caller identity.

### Access configuration (backplane)
`backplane/terragrunt.hcl` configures EKS access entries via `sso_admin_roles`:
```hcl
sso_admin_roles = {
  "AdministratorAccess" = ["system:masters"]
  "SuperUser"           = ["system:masters"]
  # SRE-Admin is NOT here
}
```
The backplane is pinned to blueprint `2026-04-20.hcl`.

## Decisions
- Root cause confirmed: `SRE-Admin` SSO permission set is absent from `sso_admin_roles` in every cluster's backplane config, so EKS has no access entry for that role → 401 on all Kubernetes API calls.
- AWS API calls (S3, IAM, STS) all succeed — only Kubernetes/Helm provider calls fail, which is consistent with valid AWS creds but missing EKS authorization.

## Task State
- [x] Root cause identified
- [ ] Fix not yet implemented

## Open Questions
- Should `SRE-Admin` be added to `sso_admin_roles` across all cluster backplanes, or should SREs switch to using `AdministratorAccess` for Terraform operations?
- Was `SRE-Admin` ever in `sso_admin_roles` previously, or has the SRE team always needed `AdministratorAccess` for this?
- How to switch AWS profile to `AdministratorAccess` — investigation was interrupted before confirming the exact profile name (likely `immuta-platform-hosting-prod-sso` with `sso_role_name = AdministratorAccess`).

## Next Steps
1. **Short-term unblock**: Switch AWS profile to one that uses `AdministratorAccess` for the `platform-hosting-prod` account (check `~/.aws/config` for the right profile name), then re-run the plan.
2. **Proper fix**: Add `"SRE-Admin" = ["system:masters"]` to `sso_admin_roles` in the backplane `terragrunt.hcl` for all affected clusters, apply backplane, then re-run applications plan.
3. Verify fix with: `aws sts get-caller-identity --profile <profile>` to confirm the correct role, then `terragrunt plan`.


File created successfully

File created successfully


Saved to `Claude/cpi-new/2026-05-13-eks-401-unauthorized.md` in your Obsidian vault.

Now — back to your question about switching to `AdministratorAccess`. Your `~/.aws/config` has a profile `immuta-platform-hosting-prod-sso` that appears to be the SSO profile for that account. To confirm which role it uses and switch, run:

```bash
! grep -A8 "\[profile immuta-platform-hosting-prod-sso\]" ~/.aws/config
```

That'll show if it's already set to `AdministratorAccess`. If so, you'd just need to make sure your aws-vault session is using that profile instead of `immuta-platform-hosting-prod`.


<local-command-caveat>Caveat: The messages below were generated by the user while running local commands. DO NOT respond to these messages or otherwise consider them in your response unless the user explicitly asks you to.</local-command-caveat>


<command-name>/exit</command-name>
            <command-message>exit</command-message>
            <command-args></command-args>


<local-command-stdout>Goodbye!</local-command-stdout>