Terraform AWS EKS ALB Kubernetes Ingress 不会创建侦听器或目标组
Posted
技术标签:
【中文标题】Terraform AWS EKS ALB Kubernetes Ingress 不会创建侦听器或目标组【英文标题】：Terraform AWS EKS ALB Kubernetes Ingress won't create Listeners or Target Groups 【发布时间】：2020-10-18 12:19:53 【问题描述】：我正在尝试使用 Terraform 资源创建一个带有 ALB 入口的 AWS EKS 集群。
This document 表示入口会自动创建一个负载均衡器，关联监听器和目标组。
Kubernetes Ingress 创建 ALB 负载均衡器、安全组和规则，但不创建目标组或侦听器。我尝试过使用网关或应用程序子网，但没有任何区别。我尝试设置安全组但 ALB 设置并使用其自己的自我管理的安全组。
我依赖this guide
卷曲到 ALB 让我感动
连接失败 de59ecbf-default-mainingre-8687-1051686593.ap-southeast-1.elb.amazonaws.com 端口 80：连接被拒绝
我分别创建了 IAM 角色和 ACM 证书，因为 AWS 对它们有配额限制。我对 EKS 集群和节点的角色是标准的，并且节点角色附加了最新的策略。
我使用kubectl 分别应用 kubernetes 入口，但结果相同。它创建 ALB 和一个安全组，其中包含端口规则，但没有目标组或侦听器。
当我将集群端点从 aws eks describe-cluster --name my-tf-eks-cluster --query "cluster.endpoint" 粘贴到浏览器中时，我得到了这个：
“种类”：“状态”，“apiVersion”：“v1”，“元数据”： ，“状态”：“失败”，“消息”：“禁止：用户“系统：匿名”无法获取路径“/”，“原因”：“禁止”， “细节”： , "代码": 403
另外，入口没有IP地址。
kubectl describe ingresses
Name: main-ingress
Namespace: default
Address:
Default backend: go-hello-world:8080 (<none>)
Rules:
Host Path Backends
---- ---- --------
* * go-hello-world:8080 (<none>)
aws eks describe-cluster --name my-tf-eks-cluster --query cluster.endpoint"
"https://88888888B.gr7.ap-southeast-1.eks.amazonaws.com"
curl https://88888888B.gr7.ap-southeast-1.eks.amazonaws.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
edit：IAM 集群策略缺少这些权限。我已经决定使用 ELB 可能会更好，因为他们可以终止 ssl 证书，然后使用 traefik 作为后端代理，所以我现在无法真正测试它。谁能确认 ALB 是否需要这些权限？
"elasticloadbalancing:DescribeListenerCertificates",
"elasticloadbalancing:AddListenerCertificates",
"elasticloadbalancing:RemoveListenerCertificates"
这是我的 EKS 主资源：
data "aws_iam_role" "tf-eks-master"
name = "terraform-eks-cluster"
resource "aws_eks_cluster" "tf_eks"
name = var.cluster_name
role_arn = data.aws_iam_role.tf-eks-master.arn
vpc_config
security_group_ids = [aws_security_group.master.id]
subnet_ids = var.application_subnet_ids
endpoint_private_access = true
endpoint_public_access = true
ALB 入口控制器：
output "vpc_id"
value = data.aws_vpc.selected
data "aws_subnet_ids" "selected"
vpc_id = data.aws_vpc.selected.id
tags = map(
"Name", "application",
)
resource "kubernetes_deployment" "alb-ingress"
metadata
name = "alb-ingress-controller"
labels =
"app.kubernetes.io/name" = "alb-ingress-controller"
namespace = "kube-system"
spec
selector
match_labels =
"app.kubernetes.io/name" = "alb-ingress-controller"
template
metadata
labels =
"app.kubernetes.io/name" = "alb-ingress-controller"
spec
volume
name = kubernetes_service_account.alb-ingress.default_secret_name
secret
secret_name = kubernetes_service_account.alb-ingress.default_secret_name
container
# This is where you change the version when Amazon comes out with a new version of the ingress controller
image = "docker.io/amazon/aws-alb-ingress-controller:v1.1.8"
name = "alb-ingress-controller"
args = [
"--ingress-class=alb",
"--cluster-name=$var.cluster_name",
"--aws-vpc-id=$data.aws_vpc.selected.id",
"--aws-region=$var.aws_region"
]
volume_mount
name = kubernetes_service_account.alb-ingress.default_secret_name
mount_path = "/var/run/secrets/kubernetes.io/serviceaccount"
read_only = true
service_account_name = "alb-ingress-controller"
resource "kubernetes_service_account" "alb-ingress"
metadata
name = "alb-ingress-controller"
namespace = "kube-system"
labels =
"app.kubernetes.io/name" = "alb-ingress-controller"
automount_service_account_token = true
kubernetes_ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: main-ingress
annotations:
kubernetes.io/ingress.class: "alb"
alb.ingress.kubernetes.io/scheme: "internet-facing"
alb.ingress.kubernetes.io/target-type: "ip"
alb.ingress.kubernetes.io/subnets: 'subnet-0ab65d9cec9451287, subnet-034bf8856ab9157b7, subnet-0c16b1d382fadd0b4'
alb.ingress.kubernetes.io/listen-ports: '["HTTP": 80,"HTTPS": 443]'
spec:
backend:
serviceName: go-hello-world
servicePort: 8080
角色
resource "kubernetes_cluster_role" "alb-ingress"
metadata
name = "alb-ingress-controller"
labels =
"app.kubernetes.io/name" = "alb-ingress-controller"
rule
api_groups = ["", "extensions"]
resources = ["configmaps", "endpoints", "events", "ingresses", "ingresses/status", "services"]
verbs = ["create", "get", "list", "update", "watch", "patch"]
rule
api_groups = ["", "extensions"]
resources = ["nodes", "pods", "secrets", "services", "namespaces"]
verbs = ["get", "list", "watch"]
resource "kubernetes_cluster_role_binding" "alb-ingress"
metadata
name = "alb-ingress-controller"
labels =
"app.kubernetes.io/name" = "alb-ingress-controller"
role_ref
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = "alb-ingress-controller"
subject
kind = "ServiceAccount"
name = "alb-ingress-controller"
namespace = "kube-system"
来自 VPC 的一些代码
data "aws_availability_zones" "available"
resource "aws_subnet" "gateway"
count = var.subnet_count
availability_zone = data.aws_availability_zones.available.names[count.index]
cidr_block = "10.0.1$count.index.0/24"
vpc_id = aws_vpc.tf_eks.id
tags = map(
"Name", "gateway",
)
resource "aws_subnet" "application"
count = var.subnet_count
availability_zone = data.aws_availability_zones.available.names[count.index]
cidr_block = "10.0.2$count.index.0/24"
vpc_id = aws_vpc.tf_eks.id
tags = map(
"Name", "application",
"kubernetes.io/cluster/$var.cluster_name", "shared",
"kubernetes.io/role/elb", "1",
)
resource "aws_subnet" "database"
count = var.subnet_count
availability_zone = data.aws_availability_zones.available.names[count.index]
cidr_block = "10.0.3$count.index.0/24"
vpc_id = aws_vpc.tf_eks.id
tags = map(
"Name", "database"
)
resource "aws_route_table" "application"
count = var.subnet_count
vpc_id = aws_vpc.tf_eks.id
route
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.tf_eks.*.id[count.index]
tags =
Name = "application"
resource "aws_route_table" "database"
vpc_id = aws_vpc.tf_eks.id
tags =
Name = "database"
resource "aws_route_table" "gateway"
vpc_id = aws_vpc.tf_eks.id
route
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.tf_eks.id
tags =
Name = "gateway"
resource "aws_route_table_association" "application"
count = var.subnet_count
subnet_id = aws_subnet.application.*.id[count.index]
route_table_id = aws_route_table.application.*.id[count.index]
resource "aws_route_table_association" "database"
count = var.subnet_count
subnet_id = aws_subnet.database.*.id[count.index]
route_table_id = aws_route_table.database.id
resource "aws_route_table_association" "gateway"
count = var.subnet_count
subnet_id = aws_subnet.gateway.*.id[count.index]
route_table_id = aws_route_table.gateway.id
resource "aws_internet_gateway" "tf_eks"
vpc_id = aws_vpc.tf_eks.id
tags =
Name = "internet_gateway"
resource "aws_eip" "nat_gateway"
count = var.subnet_count
vpc = true
resource "aws_nat_gateway" "tf_eks"
count = var.subnet_count
allocation_id = aws_eip.nat_gateway.*.id[count.index]
subnet_id = aws_subnet.gateway.*.id[count.index]
tags =
Name = "nat_gateway"
depends_on = [aws_internet_gateway.tf_eks]
安全组
resource "aws_security_group" "eks"
name = "tf-eks-master"
description = "Cluster communication with worker nodes"
vpc_id = var.vpc_id
egress
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
ingress
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
resource "aws_security_group" "node"
name = "tf-eks-node"
description = "Security group for all nodes in the cluster"
vpc_id = var.vpc_id
egress
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
resource "aws_security_group_rule" "main-node-ingress-self"
type = "ingress"
description = "Allow node to communicate with each other"
from_port = 0
protocol = "-1"
security_group_id = aws_security_group.node.id
to_port = 65535
cidr_blocks = var.subnet_cidrs
resource "aws_security_group_rule" "main-node-ingress-cluster"
type = "ingress"
description = "Allow worker Kubelets and pods to receive communication from the cluster control plane"
from_port = 1025
protocol = "tcp"
security_group_id = aws_security_group.node.id
source_security_group_id = aws_security_group.eks.id
to_port = 65535
kubectl 获取所有 --all-namespaces
kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/go-hello-world-68545f84bc-5st4s 1/1 Running 0 35s
default pod/go-hello-world-68545f84bc-bkwpb 1/1 Running 0 35s
default pod/go-hello-world-68545f84bc-kmfbq 1/1 Running 0 35s
kube-system pod/alb-ingress-controller-5f9cb4b7c4-w858g 1/1 Running 0 2m7s
kube-system pod/aws-node-8jfkf 1/1 Running 0 67m
kube-system pod/aws-node-d7s7w 1/1 Running 0 67m
kube-system pod/aws-node-termination-handler-g5fmj 1/1 Running 0 67m
kube-system pod/aws-node-termination-handler-q5tz5 1/1 Running 0 67m
kube-system pod/aws-node-termination-handler-tmzmr 1/1 Running 0 67m
kube-system pod/aws-node-vswpf 1/1 Running 0 67m
kube-system pod/coredns-5c4dd4cc7-sk474 1/1 Running 0 71m
kube-system pod/coredns-5c4dd4cc7-zplwg 1/1 Running 0 71m
kube-system pod/kube-proxy-5m9dn 1/1 Running 0 67m
kube-system pod/kube-proxy-8tn9l 1/1 Running 0 67m
kube-system pod/kube-proxy-qs652 1/1 Running 0 67m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 71m
kube-system service/kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 71m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/aws-node 3 3 3 3 3 <none> 71m
kube-system daemonset.apps/aws-node-termination-handler 3 3 3 3 3 <none> 68m
kube-system daemonset.apps/kube-proxy 3 3 3 3 3 <none> 71m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default deployment.apps/go-hello-world 3/3 3 3 37s
kube-system deployment.apps/alb-ingress-controller 1/1 1 1 2m9s
kube-system deployment.apps/coredns 2/2 2 2 71m
NAMESPACE NAME DESIRED CURRENT READY AGE
default replicaset.apps/go-hello-world-68545f84bc 3 3 3 37s
kube-system replicaset.apps/alb-ingress-controller-5f9cb4b7c4 1 1 1 2m9s
kube-system replicaset.apps/coredns-5c4dd4cc7 2 2
【问题讨论】：
您是否尝试过：资源：aws_lb_target_group_attachment 提供向应用程序负载均衡器 (ALB) 或网络负载均衡器 (NLB) 目标组注册实例和容器的能力。 我不想走那条路，因为它需要在创建入口后以及每次创建新入口后进行额外的 terraform（或控制台）工作。入口类 alb 应该创建这些资源。 我看到 Ingress 中没有提到rules，这是不正确的。有没有这样的原因或文件不正确。
您的 ec2 节点是否在 alb 目标中？
安全组规则由入口正确创建。
【参考方案1】：
您可以尝试添加这些行并尝试 kubectl 命令
# ALB's Target Group Configurations
alb.ingress.kubernetes.io/backend-protocol: HTTPS
alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
Check this
仍然没有创建目标组，检查控制器的日志
kubectl logs -n kube-system deployment.apps/aws-load-balancer-controller
【讨论】：
您好，我使用 HTTPS:8082 部署了一个应用程序。将入口后端协议更新为 https，但端口仍默认为 8081。如何将后端端口更改为 8082？【参考方案2】：没有对其进行测试，但我将指出我在您的资源中看到的问题。
在 alb 资源参数中，您有以下内容：
"--aws-vpc-id=$data.aws_vpc.selected.id",
但是，您没有任何数据资源来拉取此 VPC。 此外，数据资源在没有任何依赖关系的情况下运行，不会拉取有关尚未创建的 VPC 的任何信息。
如果您的 terraform 已模块化，请从 VPC 模块输出 VPC ID 并改用它。如果所有资源都在同一个文件/文件夹中，只需在此行中直接引用它：aws_vpc.tf_eks.id
【讨论】：
以上是关于Terraform AWS EKS ALB Kubernetes Ingress 不会创建侦听器或目标组的主要内容，如果未能解决你的问题，请参考以下文章
AWS eks绑定alb 使用aws-load-balancer-controller(Ingress Controller)提供服务
AWS eks绑定alb 使用aws-load-balancer-controller(Ingress Controller)提供服务
在 AWS EKS 上扩展 pod 时出现 502 ALB 错误