使用 boto3 创建自动缩放组 amazon ec2 时出现关键错误

Posted

技术标签:

【中文标题】使用 boto3 创建自动缩放组 amazon ec2 时出现关键错误【英文标题】:key error while creating autoscaling group amazon ec2 using boto3 【发布时间】:2019-02-07 21:47:16 【问题描述】:

我有一个亚马逊 ec2 实例,我正在尝试创建一个自动缩放组并在其上使用 cloudwatch 指标等。

import boto3
from boto3 import Session

session = Session()
credentials = session.get_credentials()
current_credentials = credentials.get_frozen_credentials()


if current_credentials.access_key is None:
    print("Access Key missing, use  `aws configure` to setup")
    exit()

if current_credentials.secret_key is None:
    print("Secret Key missing, use  `aws configure` to setup")
    exit()

# VPC design for multi az deployments
globalVars = 
globalVars['REGION_NAME']              = "ap-south-1"
globalVars['AZ1']                      = "ap-south-1a"
globalVars['AZ2']                      = "ap-south-1b"
globalVars['CIDRange']                 = "10.240.0.0/23"
globalVars['az1_pvtsubnet_CIDRange']   = "10.240.0.0/25"
globalVars['az1_pubsubnet_CIDRange']   = "10.240.0.128/26"
globalVars['az1_sparesubnet_CIDRange'] = "10.240.0.192/26"
globalVars['az2_pvtsubnet_CIDRange']   = "10.240.1.0/25"
globalVars['az2_pubsubnet_CIDRange']   = "10.240.1.128/26"
globalVars['az2_sparesubnet_CIDRange'] = "10.240.1.192/26"
globalVars['Project']                  =  'Key': 'Name',        'Value': 'test1'
globalVars['tags']                     = ['Key': 'Owner',       'Value': 'test1',
                                          'Key': 'Environment', 'Value': 'Test',
                                          'Key': 'Department',  'Value': 'TestD']
# EC2 Parameters

globalVars['EC2-Amazon-AMI-ID']        = "ami-00b6a8a2bd28daf19"
globalVars['EC2-InstanceType']         = "t2.micro"
globalVars['EC2-KeyName']              = "datastructutrekey.pem"

# AutoScaling Parameters
globalVars['ASG-LaunchConfigName']     = "ASG-Demo-LaunchConfig"
globalVars['ASG-AutoScalingGroupName'] = "ASG-Demo-AutoScalingGrp"


# Creating a VPC, Subnet, and Gateway
ec2       = boto3.resource('ec2', region_name=globalVars['REGION_NAME'])
ec2Client = boto3.client('ec2',   region_name=globalVars['REGION_NAME'])
vpc       = ec2.create_vpc(CidrBlock=globalVars['CIDRange'])
asgClient = boto3.client('autoscaling', region_name=globalVars['REGION_NAME'])
rds       = boto3.client('rds', region_name=globalVars['REGION_NAME'])

# AZ1 Subnets
az1_pvtsubnet   = vpc.create_subnet(CidrBlock=globalVars['az1_pvtsubnet_CIDRange'],   AvailabilityZone=globalVars['AZ1'])
az1_pubsubnet   = vpc.create_subnet(CidrBlock=globalVars['az1_pubsubnet_CIDRange'],   AvailabilityZone=globalVars['AZ1'])
az1_sparesubnet = vpc.create_subnet(CidrBlock=globalVars['az1_sparesubnet_CIDRange'], AvailabilityZone=globalVars['AZ1'])
# AZ2 Subnet
az2_pvtsubnet   = vpc.create_subnet(CidrBlock=globalVars['az2_pvtsubnet_CIDRange'],   AvailabilityZone=globalVars['AZ2'])
az2_pubsubnet   = vpc.create_subnet(CidrBlock=globalVars['az2_pubsubnet_CIDRange'],   AvailabilityZone=globalVars['AZ2'])
az2_sparesubnet = vpc.create_subnet(CidrBlock=globalVars['az2_sparesubnet_CIDRange'], AvailabilityZone=globalVars['AZ2'])

# Enable DNS Hostnames in the VPC
vpc.modify_attribute(EnableDnsSupport='Value': True)
vpc.modify_attribute(EnableDnsHostnames='Value': True)

# Create the Internet Gatway & Attach to the VPC
intGateway = ec2.create_internet_gateway()
intGateway.attach_to_vpc(VpcId=vpc.id)

# Create another route table for Public & Private traffic
routeTable = ec2.create_route_table(VpcId=vpc.id)
rtbAssn=[]
rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az1_pubsubnet.id))
rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az1_pvtsubnet.id))
rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az2_pubsubnet.id))
rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az2_pvtsubnet.id))

# Create a route for internet traffic to flow out
intRoute = ec2Client.create_route(RouteTableId=routeTable.id, DestinationCidrBlock='0.0.0.0/0', GatewayId=intGateway.id)

# Tag the resources
vpc.create_tags            (Tags=globalVars['tags'])
az1_pvtsubnet.create_tags  (Tags=globalVars['tags'])
az1_pubsubnet.create_tags  (Tags=globalVars['tags'])
az1_sparesubnet.create_tags(Tags=globalVars['tags'])
az2_pvtsubnet.create_tags  (Tags=globalVars['tags'])
az2_pubsubnet.create_tags  (Tags=globalVars['tags'])
az2_sparesubnet.create_tags(Tags=globalVars['tags'])
intGateway.create_tags     (Tags=globalVars['tags'])
routeTable.create_tags     (Tags=globalVars['tags'])

vpc.create_tags            (Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-vpc'])
az1_pvtsubnet.create_tags  (Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az1-private-subnet'])
az1_pubsubnet.create_tags  (Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az1-public-subnet'])
az1_sparesubnet.create_tags(Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az1-spare-subnet'])
az2_pvtsubnet.create_tags  (Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az2-private-subnet'])
az2_pubsubnet.create_tags  (Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az2-public-subnet'])
az2_sparesubnet.create_tags(Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az2-spare-subnet'])
intGateway.create_tags     (Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-igw'])
routeTable.create_tags     (Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-rtb'])

# Let create the Public & Private Security Groups
elbSecGrp = ec2.create_security_group(DryRun=False,
                                      GroupName='elbSecGrp',
                                      Description='ElasticLoadBalancer_Security_Group',
                                      VpcId=vpc.id
                                      )

pubSecGrp = ec2.create_security_group(DryRun=False,
                                      GroupName='pubSecGrp',
                                      Description='Public_Security_Group',
                                      VpcId=vpc.id
                                      )

pvtSecGrp = ec2.create_security_group(DryRun=False,
                                      GroupName='pvtSecGrp',
                                      Description='Private_Security_Group',
                                      VpcId=vpc.id
                                      )

elbSecGrp.create_tags(Tags=globalVars['tags'])
pubSecGrp.create_tags(Tags=globalVars['tags'])
pvtSecGrp.create_tags(Tags=globalVars['tags'])

elbSecGrp.create_tags(Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-elb-security-group'])
pubSecGrp.create_tags(Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-public-security-group'])
pvtSecGrp.create_tags(Tags=['Key': 'Name', 'Value': globalVars['Project']['Value'] + '-private-security-group'])

# Add a rule that allows inbound SSH, HTTP, HTTPS traffic ( from any source )
ec2Client.authorize_security_group_ingress(GroupId=elbSecGrp.id,
                                           IpProtocol='tcp',
                                           FromPort=80,
                                           ToPort=80,
                                           CidrIp='0.0.0.0/0'
                                           )

# Allow Public Security Group to receive traffic from ELB Security group
ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id,
                                           IpPermissions=['IpProtocol': 'tcp',
                                                           'FromPort': 80,
                                                           'ToPort': 80,
                                                           'UserIdGroupPairs': ['GroupId': elbSecGrp.id]
                                                           ]
                                           )
# Allow Private Security Group to receive traffic from Application Security group
ec2Client.authorize_security_group_ingress(GroupId=pvtSecGrp.id,
                                           IpPermissions=['IpProtocol': 'tcp',
                                                           'FromPort': 3306,
                                                           'ToPort': 3306,
                                                           'UserIdGroupPairs': ['GroupId': pubSecGrp.id]
                                                           ]
                                           )

ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id,
                                           IpProtocol='tcp',
                                           FromPort=80,
                                           ToPort=80,
                                           CidrIp='0.0.0.0/0'
                                           )
ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id,
                                           IpProtocol='tcp',
                                           FromPort=443,
                                           ToPort=443,
                                           CidrIp='0.0.0.0/0'
                                           )
ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id,
                                           IpProtocol='tcp',
                                           FromPort=22,
                                           ToPort=22,
                                           CidrIp='0.0.0.0/0'
                                           )



# The user defined code to install  WebServer & Configure them
userDataCode = """
#!/bin/bash
set -e -x
# Setting up the HTTP server 
yum install -y httpd 
service httpd start
chkconfig httpd on
groupadd www
usermod -a -G www ec2-user
cd /var/www/
# Set the permissions
chown -R root:www /var/www
chmod 2775 /var/www
find /var/www -type d -exec chmod 2775  +
find /var/www -type f -exec chmod 0664  +
# SE Linux permissive
# setsebool -P httpd_can_network_connect=1
service httpd restart
echo "<?php phpinfo(); ?>" > /var/www/html/phptestinfo.php
"""

# Create the  Public Instance
##### **DeviceIndex**:The network interface's position in the attachment order. For example, the first attached network interface has a DeviceIndex of 0
instanceLst = ec2.create_instances(ImageId=globalVars['EC2-Amazon-AMI-ID'],
                                   MinCount=1,
                                   MaxCount=2,
                                   KeyName="datastructutrekey.pem",
                                   UserData=userDataCode,
                                   InstanceType=globalVars['EC2-InstanceType'],
                                   NetworkInterfaces=[
                                       
                                           'SubnetId': az1_pubsubnet.id,
                                           'Groups': [pubSecGrp.id],
                                           'DeviceIndex': 0,
                                           'DeleteOnTermination': True,
                                           'AssociatePublicIpAddress': True,
                                       
                                   ]
                                   )


# Create the Launch Configuration
# InstanceId = 'string'
asgLaunchConfig = asgClient.create_launch_configuration(
    LaunchConfigurationName=globalVars['ASG-LaunchConfigName'],
    ImageId=globalVars['EC2-Amazon-AMI-ID'],
    KeyName=globalVars['EC2-KeyName'],
    SecurityGroups=[pubSecGrp.id],
    UserData=userDataCode,
    InstanceType=globalVars['EC2-InstanceType'],
    InstanceMonitoring='Enabled': False ,
    EbsOptimized=False,
    AssociatePublicIpAddress=False
)

# create Auto-Scaling Group
ASGSubnets = az1_pubsubnet.id + "," +az2_pubsubnet.id
asGroup=asgClient.create_auto_scaling_group(
    AutoScalingGroupName=globalVars['ASG-AutoScalingGroupName'],
    LaunchConfigurationName=globalVars['ASG-LaunchConfigName'],
    MinSize=1,
    MaxSize=3,
    DesiredCapacity=2,
    DefaultCooldown=120,
    HealthCheckType='EC2',
    HealthCheckGracePeriod=60,
    Tags=globalVars['tags'],
    VPCZoneIdentifier=ASGSubnets
    )

asgClient.create_or_update_tags(
    Tags=[
        
            'ResourceId': globalVars['ASG-AutoScalingGroupName'],
            'ResourceType': 'auto-scaling-group',
            'Key': 'Name',
            'Value': globalVars['Project']['Value'] + '-ASG-Group',
            'PropagateAtLaunch': True
        ,
    ]
)



###### Print to Screen ########
print("VPC ID                    : 0".format(vpc.id))
print("AZ1 Public Subnet ID      : 0".format(az1_pubsubnet.id))
print("AZ1 Private Subnet ID     : 0".format(az1_pvtsubnet.id))
print("AZ1 Spare Subnet ID       : 0".format(az1_sparesubnet.id))
print("Internet Gateway ID       : 0".format(intGateway.id))
print("Route Table ID            : 0".format(routeTable.id))
print("Public Security Group ID  : 0".format(pubSecGrp.id))
print("Private Security Group ID : 0".format(pvtSecGrp.id))
print("EC2 Key Pair              : 0".format(globalVars['EC2-KeyName']))
print("EC2 PublicIP              : 0".format(globalVars['EC2-KeyName']))
print("RDS Endpoint              : 0".format(globalVars['Endpoint']))
###### Print to Screen ########


"""
Function to clean up all the resources
"""
def cleanAll(resourcesDict=None):
    # Delete the instances
    ids = []
    for i in instanceLst:
        ids.append(i.id)

    ec2.instances.filter(InstanceIds=ids).terminate()

    # Wait for the instance to be terminated
    waiter = ec2Client.get_waiter('instance_terminated')
    waiter.wait(InstanceIds=[ids])
    ec2Client.delete_key_pair(KeyName=globalVars['EC2-KeyName'])

    # Delete Routes & Routing Table
    for assn in rtbAssn:
        ec2Client.disassociate_route_table(AssociationId=assn.id)

    routeTable.delete()

    # Delete Subnets
    az1_pvtsubnet.delete()
    az1_pubsubnet.delete()
    az1_sparesubnet.delete()

    # Detach & Delete internet Gateway
    ec2Client.detach_internet_gateway(InternetGatewayId=intGateway.id, VpcId=vpc.id)
    intGateway.delete()

    # Delete Security Groups
    pubSecGrp.delete()
    pvtSecGrp.delete()

    vpc.delete()

脚本运行良好,除了最后一行我得到以下输出

VPC ID                    : vpc-047839873c2b62b51
AZ1 Public Subnet ID      : subnet-0c8db93f160c952b3
AZ1 Private Subnet ID     : subnet-00e2068c36695cf47
AZ1 Spare Subnet ID       : subnet-0521b503114f92f97
Internet Gateway ID       : igw-020b3e284d591e977
Route Table ID            : rtb-0e83e29fed23f6773
Public Security Group ID  : sg-05bbc53b9cad2a6db
Private Security Group ID : sg-028d432258e825562
EC2 Key Pair              : datastructutrekey.pem
EC2 PublicIP              : datastructutrekey.pem
Traceback (most recent call last):
  File "D:\programming \autoscaling.py", line 281, in <module>
    print("RDS Endpoint              : 0".format(globalVars['Endpoint']))
KeyError: 'Endpoint'

我无法理解这里出现的最后一个错误 1) RDS 端点错误 为什么会出现。我登录到 https://ap-south-1.console.aws.amazon.com/vpc/home?region=ap-south-1#igws:sort=internetGatewayId 和 https://ap-south-1.console.aws.amazon.com/vpc/home?region=ap-south-1#vpcs:但我在这里找不到任何错误我已经创建了 test1-vpc,它显示给我.任何人都可以在这里指出为什么会出现此错误以及如何摆脱它。我正在创建的这些实例是 Amazon Linux 2 AMI (HVM)、SSD 卷类型,默认情况下 ap-south-1 不支持自动缩放,这可能是我出错的原因。

2)此外,当我进行控制台日志记录时,我看到 4 个实例正在运行。在 2 个实例名称为空白,而两个实例作为 t​​est1-ASG-Group 时,为什么 2 个实例通过上面的脚本具有空白名称。这是一个截图第2点 图像中的最后 4 个实例是创建的实例编号 2,3 没有任何名称。在我看来应该存在。

【问题讨论】:

【参考方案1】:

这里的问题是 Python 编程错误,而不是您使用 AWS 的问题。

您的 globarVars dict 没有键为“Endpoint”的值,因此当您尝试访问 globalVars['Endpoint'] 时 Python 会抛出 KeyError。您需要填充一个值。

此外,您可能需要考虑使用AWS CloudFormation,而不是编写 100 行代码来构建 AWS 基础设施,它允许您编写 YAML 模板来做同样的事情。

最后,填充 globalVars 字典的首选方法是:

globalVars = 
  'REGION_NAME': 'ap-south-1',
  'AZ1': 'ap-south-1a',
  'AZ2': 'ap-south-1b',
  # others here

【讨论】:

以上是关于使用 boto3 创建自动缩放组 amazon ec2 时出现关键错误的主要内容,如果未能解决你的问题,请参考以下文章

我想使用 python boto3 脚本将数据加载到 Amazon Redshift 集群中

OpsWorks基于负载的实例与自动扩展组?

AWS-CLI:如何过滤自动缩放组

具有弹性 IP 的 Amazon EC2 自动扩展实例

将自动缩放策略应用于 DynamoDB 表时出现 ObjectNotFoundException

Amazon Code Deploy 中从 ASCII-8BIT 到 UTF-8 的“\xCB”