如何在带有 Terraform 的 AWS VPC 中的两个子网之间进行路由？
Posted
技术标签:
【中文标题】如何在带有 Terraform 的 AWS VPC 中的两个子网之间进行路由？【英文标题】：How to route between two subnets in an AWS VPC w/ Terraform? 【发布时间】：2016-06-19 18:27:21 【问题描述】：更新： 除其他外，一直在努力解决这个问题。似乎无法获得带有两个子网和一个 SSH 堡垒的工作配置。为完整的 .tf 文件配置提供赏金： * 创建两个私有子网 * 创建一个堡垒 * 在通过堡垒配置的每个子网上旋转一个 ec2 实例（通过堡垒运行一些任意 shell 命令） * 已配置互联网网关 * 对私有子网上的主机有一个 nat 网关 * 具有相应配置的路由和安全组
原帖： 我正在尝试学习 Terraform 并构建原型。我有一个通过 Terraform 配置的 AWS VPC。除了 DMZ 子网之外，我还有一个公共子网“web”，它接收来自 Internet 的流量。我有一个无法从 Internet 访问的私有子网“应用程序”。我正在尝试配置堡垒主机，以便 terraform 可以在私有“应用程序”子网上配置实例。我还不能让它工作。
当我 ssh 进入堡垒时，我无法从堡垒主机通过 SSH 连接到私有子网中的任何实例。我怀疑有路由问题。我一直在通过几个可用的示例和文档来构建这个原型。许多示例通过 aws 提供程序使用略有不同的技术和 terraform 路由定义。
谁能提供理想或正确的方法来定义这三个子网（公共“web”、带有堡垒的公共“dmz”和私有“app”），以便“web”子网上的实例可以访问'app' 子网，并且 DMZ 中的堡垒主机可以在私有 'app' 子网中配置实例？
我的配置片段如下：
resource "aws_subnet" "dmz"
vpc_id = "$aws_vpc.vpc-poc.id"
cidr_block = "$var.cidr_block_dmz"
resource "aws_route_table" "dmz"
vpc_id = "$aws_vpc.vpc-poc.id"
route
cidr_block = "0.0.0.0/0"
gateway_id = "$aws_internet_gateway.gateway.id"
resource "aws_route_table_association" "dmz"
subnet_id = "$aws_subnet.dmz.id"
route_table_id = "$aws_route_table.dmz.id"
resource "aws_subnet" "web"
vpc_id = "$aws_vpc.vpc-poc.id"
cidr_block = "10.200.2.0/24"
resource "aws_route_table" "web"
vpc_id = "$aws_vpc.vpc-poc.id"
route
cidr_block = "0.0.0.0/0"
instance_id = "$aws_instance.bastion.id"
resource "aws_route_table_association" "web"
subnet_id = "$aws_subnet.web.id"
route_table_id = "$aws_route_table.web.id"
resource "aws_subnet" "app"
vpc_id = "$aws_vpc.vpc-poc.id"
cidr_block = "10.200.3.0/24"
resource "aws_route_table" "app"
vpc_id = "$aws_vpc.vpc-poc.id"
route
cidr_block = "0.0.0.0/0"
instance_id = "$aws_instance.bastion.id"
resource "aws_route_table_association" "app"
subnet_id = "$aws_subnet.app.id"
route_table_id = "$aws_route_table.app.id"
【问题讨论】：
如果您需要更多帮助，您需要扩展您的 TF 文件以显示任何安全组/NACL 等，因为我不认为这里的路由是一个问题（除了缺少私有子网中的出站 Web 访问，除非您的堡垒机还充当 NAT 网关） 【参考方案1】：这里有一个可以帮助你的 sn-p。这是未经测试的，但是是从我在私有子网中配置 VM 的一个 terraform 文件中提取的。我知道这适用于一个私有子网，我尝试在这里实现两个，就像你原来的问题一样。
我跳过我的 NAT 实例以使用 Terraform 访问和配置私有子网框。如果您的安全组设置正确，它确实可以工作。我做了一些实验。
/* VPC creation */
resource "aws_vpc" "vpc_poc"
cidr_block = "10.200.0.0/16"
/* Internet gateway for the public subnets */
resource "aws_internet_gateway" "gateway"
vpc_id = "$aws_vpc.vpc_poc.id"
/* DMZ subnet - public */
resource "aws_subnet" "dmz"
vpc_id = "$aws_vpc.vpc_poc.id"
cidr_block = "10.200.1.0/24"
/* may help to be explicit here */
map_public_ip_on_launch = true
/* this is recommended in the docs */
depends_on = ["aws_internet_gateway.gateway"]
resource "aws_route_table" "dmz"
vpc_id = "$aws_vpc.vpc_poc.id"
route
cidr_block = "0.0.0.0/0"
gateway_id = "$aws_internet_gateway.gateway.id"
resource "aws_route_table_association" "dmz"
subnet_id = "$aws_subnet.dmz.id"
route_table_id = "$aws_route_table.dmz.id"
/* Web subnet - public */
resource "aws_subnet" "web"
vpc_id = "$aws_vpc.vpc_poc.id"
cidr_block = "10.200.2.0/24"
map_public_ip_on_launch = true
depends_on = ["aws_internet_gateway.gateway"]
resource "aws_route_table" "web"
vpc_id = "$aws_vpc.vpc_poc.id"
route
cidr_block = "0.0.0.0/0"
/* your public web subnet needs access to the gateway */
/* this was set to bastion before so you had a circular arg */
gateway_id = "$aws_internet_gateway.gateway.id"
resource "aws_route_table_association" "web"
subnet_id = "$aws_subnet.web.id"
route_table_id = "$aws_route_table.web.id"
/* App subnet - private */
resource "aws_subnet" "app"
vpc_id = "$aws_vpc.vpc_poc.id"
cidr_block = "10.200.3.0/24"
/* Create route for DMZ Bastion */
resource "aws_route_table" "app"
vpc_id = "$aws_vpc.vpc_poc.id"
route
cidr_block = "0.0.0.0/0"
/* this send traffic to the bastion to pass off */
instance_id = "$aws_instance.nat_dmz.id"
/* Create route for App Bastion */
resource "aws_route_table" "app"
vpc_id = "$aws_vpc.vpc_poc.id"
route
cidr_block = "0.0.0.0/0"
/* this send traffic to the bastion to pass off */
instance_id = "$aws_instance.nat_web.id"
resource "aws_route_table_association" "app"
subnet_id = "$aws_subnet.app.id"
route_table_id = "$aws_route_table.app.id"
/* Default security group */
resource "aws_security_group" "default"
name = "default-sg"
description = "Default security group that allows inbound and outbound traffic from all instances in the VPC"
vpc_id = "$aws_vpc.vpc_poc.id"
ingress
from_port = "0"
to_port = "0"
protocol = "-1"
self = true
egress
from_port = "0"
to_port = "0"
protocol = "-1"
self = true
/* Security group for the nat server */
resource "aws_security_group" "nat"
name = "nat-sg"
description = "Security group for nat instances that allows SSH and *** traffic from internet. Also allows outbound HTTP[S]"
vpc_id = "$aws_vpc.vpc_poc.id"
ingress
from_port = 80
to_port = 80
protocol = "tcp"
/* this your private subnet cidr */
cidr_blocks = ["10.200.3.0/24"]
ingress
from_port = 443
to_port = 443
protocol = "tcp"
/* this is your private subnet cidr */
cidr_blocks = ["10.200.3.0/24"]
ingress
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
ingress
from_port = -1
to_port = -1
protocol = "icmp"
cidr_blocks = ["0.0.0.0/0"]
egress
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
egress
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
egress
from_port = 22
to_port = 22
protocol = "tcp"
/* this is the vpc cidr block */
cidr_blocks = ["10.200.0.0/16"]
egress
from_port = -1
to_port = -1
protocol = "icmp"
cidr_blocks = ["0.0.0.0/0"]
/* Security group for the web */
resource "aws_security_group" "web"
name = "web-sg"
description = "Security group for web that allows web traffic from internet"
vpc_id = "$aws_vpc.vpc_poc.id"
ingress
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
ingress
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
/* Install deploy key for use with all of our provisioners */
resource "aws_key_pair" "deployer"
key_name = "deployer-key"
public_key = "$file("~/.ssh/id_rsa")"
/* Setup NAT in DMZ subnet */
resource "aws_instance" "nat_dmz"
ami = "ami-67a54423"
availability_zone = "us-west-1a"
instance_type = "m1.small"
key_name = "$aws_key_pair.deployer.id"
/* Notice we are assigning the security group here */
security_groups = ["$aws_security_group.nat.id"]
/* this puts the instance in your public subnet, but translate to the private one */
subnet_id = "$aws_subnet.dmz.id"
/* this is really important for nat instance */
source_dest_check = false
associate_public_ip_address = true
/* Give NAT EIP In DMZ */
resource "aws_eip" "nat_dmz"
instance = "$aws_instance.nat_dmz.id"
vpc = true
/* Setup NAT in Web subnet */
resource "aws_instance" "nat_web"
ami = "ami-67a54423"
availability_zone = "us-west-1a"
instance_type = "m1.small"
key_name = "$aws_key_pair.deployer.id"
/* Notice we are assigning the security group here */
security_groups = ["$aws_security_group.nat.id"]
/* this puts the instance in your public subnet, but translate to the private one */
subnet_id = "$aws_subnet.web.id"
/* this is really important for nat instance */
source_dest_check = false
associate_public_ip_address = true
/* Give NAT EIP In DMZ */
resource "aws_eip" "nat_web"
instance = "$aws_instance.nat_web.id"
vpc = true
/* Install server in private subnet and jump host to it with terraform */
resource "aws_instance" "private_box"
ami = "ami-d1315fb1"
instance_type = "t2.large"
key_name = "$aws_key_pair.deployer.id"
subnet_id = "$aws_subnet.api.id"
associate_public_ip_address = false
/* this is what gives the box access to talk to the nat */
security_groups = ["$aws_security_group.nat.id"]
connection
/* connect through the nat instance to reach this box */
bastion_host = "$aws_eip.nat_dmz.public_ip"
bastion_user = "ec2-user"
bastion_private_key = "$file("keys/terraform_rsa")"
/* connect to box here */
user = "ec2-user"
host = "$self.private_ip"
private_key = "$file("~/.ssh/id_rsa")"
【讨论】：
【参考方案2】：除非堡垒主机也充当 NAT（我不建议您在同一个实例上组合角色），否则 Web 和应用子网将没有任何出站 Web 访问，否则看起来像 TF 一样好的路由将自动为 VPC 添加本地路由记录。
只要您拥有覆盖您的 VPC 范围的本地路由记录，那么路由应该没问题。获取您的 Terraform 配置文件（并添加最少的必要资源）允许我在所有 3 个子网中创建一些基本实例并在它们之间成功路由，因此您可能会丢失其他内容，例如安全组或 NACL。
【讨论】：
谢谢。同样，我遇到的问题是 terraform 无法通过堡垒 SSH 进入私有子网上的实例。我认为它是路由，但我会更深入地挖掘并在这里发布更新。【参考方案3】：您尚未提供完整的 Terraform，但您需要允许从堡垒 IP 或堡垒主机的 CIDR 块通过 SSH 进入您的“应用”VPC 实例，因此如下所示：
resource "aws_security_group" "allow_ssh"
name = "allow_ssh"
description = "Allow inbound SSH traffic"
ingress
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["$aws_instance.bastion.private_ip/32"]
然后在您的“应用”实例资源中，您需要添加安全组：
...
vpc_security_group_ids = ["$aws_security_group.allow_ssh.id"]
...
https://www.terraform.io/docs/providers/aws/r/security_group_rule.html
【讨论】：
这很有帮助，因为它向我展示了一种表达这一点的新方法，但遗憾的是，这不是解决办法。一定还有什么遗漏的。 如果您需要特定帮助，请提供模板的其余部分。这里没有足够的上下文。【参考方案4】：我没有看到堡垒主机的原因。
我使用 saltstack 有类似的东西，我只是使用 VPC 内的主服务器控制其余部分，为其分配特定的安全组以允许访问。
CIDR X/24
subnetX.0/26- subnet for control server. <aster server ip EC2-subnet1/32
subnetX.64/26 - private minions
subentX.128/26 - public minions
subnetX.192/26- private minions
然后为每个子网创建一个路由表，以满足您对隔离的热爱 将每条路由附加到各个子网。例如
rt-1 - subnetX.0/26
rt-2 - subnetX.64/26
rt-3 - subnetX.128/26
rt-4 - subnetX.192/26
确保您的路由表有这样的内容，以便 rt-1 实例可以连接到每个人的路由
destination: CIDR X/24 Target: local
然后通过安全组入站限制连接。例如 允许来自 EC2-subnet1/32 的 SSH
完成控制服务器的所有工作后，我可以删除在我的公共子网中显示 CIDR X/24 Target: local 的特定路由，因此它不再能够将流量路由到我的本地 CIDR。
我没有理由创建复杂的堡垒，因为我已经授权删除控制服务器中的路由。
【讨论】：
【参考方案5】：您应该使用 tcpdump 和其他调试工具检查网络问题。 请检查：
-
ips 可达且网络设置正确（如 10.200.2.X 可以访问堡垒主机的 ip）
iptables/另一个防火墙不会阻止您的流量
一个 ssh 服务器正在监听（从那些主机 ssh 到那些主机的一个 ip）
您拥有正确的主机安全组（您可以在 EC2 实例的属性中看到这一点）
尝试使用 tcpdump 嗅探流量
【讨论】：
以上是关于如何在带有 Terraform 的 AWS VPC 中的两个子网之间进行路由？的主要内容，如果未能解决你的问题，请参考以下文章
使用 terraform 在非默认 VPC 中创建 AWS RDS 实例