如何使用cloudformation模板将两个EC2实例（安装AMI创建的Elasticsearch）作为多节点？

Posted 2023-03-04

【中文标题】如何使用cloudformation模板将两个EC2实例（安装AMI创建的Elasticsearch）作为多节点？

【英文标题】：How to make two EC2 instance (installed Elasticsearch created by AMI) as multi-node using cloudformation template?

【发布时间】：2021-07-05 06:45:40

【问题描述】：

我需要使用 AMI 创建两个 Ec2 实例，并使用 CloudFormation 模板将其作为多节点。 AMI 在其中安装了 elasticsearch。我需要将一个作为主节点，另一个作为数据节点。

我的 CF 模板脚本，

AWSTemplateFormatVersion: '2010-09-09'
#Transform: 'AWS::Serverless-2016-10-31'
Description: AWS CloudFormation Template with EC2InstanceWithSecurityGroup
Parameters:
  KeyName:
    Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: must be the name of an existing EC2 KeyPair.
  RemoteAccessLocation:
    Description: The IP address range that can be used to access to the EC2 instances
    Type: String
    MinLength: '9'
    MaxLength: '18'
    Default: 0.0.0.0/0
    AllowedPattern: (\d1,3)\.(\d1,3)\.(\d1,3)\.(\d1,3)/(\d1,2)
    ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.

Resources:
  ES1EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.2xlarge
      SecurityGroups:
        - !Ref 'InstanceSecurityGroup'
      KeyName: !Ref 'KeyName'
      ImageId: ami-xxxxxxxxxxxxxxxx
      #DependsOn: ES2EC2Instance
      UserData:
        Fn::Base64: !Sub |
              #!/bin/bash -ex

              cat > /etc/elasticsearch/elasticsearch.yml<<EOF1
              network.host: "$EC2_PRIVATE_IP"
              http.port: 9200
              http.max_content_length: 1gb
              node.name: node-1
              node.roles: [ master, data, ingest ]
              transport.port: 9300-9400
              discovery.seed_hosts: ["$ES1EC2Instance.PrivateIp", "$ES2EC2Instance.PrivateIp"]
              cluster.initial_master_nodes: ["node-1"]
              gateway.recover_after_nodes: 2
              EOF1

              ## Restart Elasticsearch
              sudo systemctl restart elasticsearch
  ES2EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.2xlarge
      SecurityGroups:
        - !Ref 'InstanceSecurityGroup'
      KeyName: !Ref 'KeyName'
      ImageId: ami-xxxxxxxxxxxxxxxx
      #DependsOn: ES1EC2Instance
      DependsOn: ES1EC2Instance
      UserData:
        Fn::Base64: !Sub |
              #!/bin/bash -ex

              cat > /etc/elasticsearch/elasticsearch.yml<<EOF1
              network.host: "$ES2EC2Instance.PrivateIp"
              http.port: 9200
              http.max_content_length: 1gb
              node.name: node-2
              node.roles: [ data, ingest ]
              transport.port: 9300-9400
              discovery.seed_hosts: ["$ES1EC2Instance.PrivateIp", "$ES2EC2Instance.PrivateIp"]
              cluster.initial_master_nodes: ["node-1"]
              gateway.recover_after_nodes: 2
              EOF1

              ## Restart Elasticsearch
              sudo systemctl restart elasticsearch

  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH (22), HTTP (8080),
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: !Ref 'RemoteAccessLocation'
        - CidrIp: 0.0.0.0/0
          FromPort: '8080'
          IpProtocol: tcp
          ToPort: '8080'
        - IpProtocol: tcp
          FromPort: '9200'
          ToPort: '9200'
          CidrIp: !Ref 'RemoteAccessLocation'

Outputs:
  AZ:
    Description: Availability Zone of the newly created EC2 instance for ES
    Value: !GetAtt 'ES1EC2Instance.AvailabilityZone'
  PublicDNS:
    Description: Public DNSName of the newly created EC2 instance for ES
    Value: !GetAtt 'ES1EC2Instance.PublicDnsName'
  PublicIP:
    Description: Public IP address of the newly created EC2 instance for ES
    Value: !GetAtt 'ES1EC2Instance.PublicIp'

如何使用 CloudFormation 模板更新 elasticsearch.yaml 以使其成为多节点？

【问题讨论】：

当您应用 cloudformation 时，您的 ES 集群的当前状态是什么？昨天我使用自动缩放组（一个带有主节点，另一个带有从节点）部署了一个 ES 集群，并启用了 AWS EC2 插件，一切正常。请注意，您的配置中缺少 cluster.name 属性。

它是使用单节点配置创建的。创建实例后，我需要使用多节点配置更新 elasticsearch.yml。我怎样才能做到这一点？

如果您使用单节点配置创建，那么您必须修改很多设置才能使集群正常工作。从具有当前设置的 cloudformation 中，您可能可以使用宏来解决问题。宏是对 cloudformation 中的 lambda 函数的调用，您甚至可以传递参数。然后在 lambda 中使用例如 boto3 并在实例中启用会话管理器，您可以连接到服务器并使用 AWS SSM RunCommand 执行一些命令，直到所需的配置生效。

谢谢米格尔！您有此 SSM RunCommand 的示例吗？

是的，我会像可能的答案一样添加代码。

【参考方案1】：

尝试将宏添加到您的 Cloudformation 模板。这是一个可以被宏调用的 lambda 示例。此功能使用 SSM RunCommand 向您的实例发送 bash 命令，在这种情况下，实例由自动缩放组过滤，但您可以按标签或任何其他属性过滤您的实例。此外，您还需要向具有 AmazonEC2RoleforSSM 权限的实例添加 IAM 角色。

import json
import boto3
import time
import os

ssm = boto3.client('ssm')
ec2 = boto3.client('ec2')
autoscalingg = boto3.client('autoscaling')

def lambda_handler(event, context):
    # TODO implement
    env = event['environment']
    kafka = event['kafka']
    print (env, kafka)
    status = check_autoscaling_group(env)
    if status:
        add_IP(env, kafka)
        return 
            'body': json.dumps('Function executed, please see the logs!')
        
    else:
        return 
            'body': json.dumps('The function was not executed, please see the logs!')
        

def check_autoscaling_group(env):
    response = autoscalingg.describe_auto_scaling_groups(
        AutoScalingGroupNames=[
            'elasticsearch-master-'+str(env)
        ]
    )
    stable = True
    for instance in response['AutoScalingGroups'][0]['Instances']:
        if instance['LifecycleState'] != "InService":
            stable = False
            message = "At least 1 master instance is not in Running status, please wait until all the masters nodes are stable."
            print (message)
            break
    return stable

def add_IP(env, kafka):
    response = ec2.describe_instances(
        Filters=[
            
                'Name': 'tag:Name',
                'Values': [
                    'elasticsearch-masters-server-'+str(env)
                ]
            ,
            
                'Name': 'instance-state-name',
                'Values': [
                    'running'
                ]
            ,
            
                'Name': 'tag:role',
                'Values': [
                    'master'
                ]
            
        ]
    )

    for instances in response['Reservations']:
        id = instances['Instances'][0]['InstanceId']
        ip = instances['Instances'][0]['PrivateIpAddress']
        
        #Add ES server IP to Kibana config
        command_to_execute = 'sed -i "s/ELASTICSEARCH_HOSTS: http:\/\/.*/ELASTICSEARCH_HOSTS: http:\/\/'+str(ip)+':9200/g" /home/ubuntu/Kibana/docker-compose.yml'
        execute_in_master(command_to_execute, id, env)
        
        #Add kafka server IP to logstash conf
        kafka = '\\"'+str(kafka)+':9092\\"'
        command_to_execute = 'sed -i "s/bootstrap_servers => .*/bootstrap_servers => ['+str(kafka)+']/g" /home/ubuntu/Logstash/config/logstash/pipeline/my_pipeline.conf'
        execute_in_master(command_to_execute, id, env)
        
        #Add ES server IP to logstash config
        ip = '\\"'+str(ip)+':9200\\"'
        command_to_execute = 'sed -i "s/hosts => .*/hosts => '+str(ip)+'/g" /home/ubuntu/Logstash/config/logstash/pipeline/my_pipeline.conf'
        execute_in_master(command_to_execute, id, env)
        
        #Restart services
        command_to_execute = 'cd /home/ubuntu/Kibana/ && docker-compose up -d'
        execute_in_master(command_to_execute, id, env)
        
        command_to_execute = 'cd /home/ubuntu/Logstash/ && docker-compose up -d'
        execute_in_master(command_to_execute, id, env)
        
        #Just select one master instance
        break
        
def execute_in_master(command_to_execute, id, env):
    print (command_to_execute)
    response = ec2.describe_instances(
        Filters=[
            
                'Name': 'tag:Name',
                'Values': [
                    'elasticsearch-masters-server-'+str(env)
                ]
            ,
            
                'Name': 'instance-state-name',
                'Values': [
                    'running'
                ]
            ,
            
                'Name': 'tag:role',
                'Values': [
                    'master'
                ]
            
        ]
    )
    
    for instances in response['Reservations']:
        instance_id = instances['Instances'][0]['InstanceId']
        if instance_id == id:
            response = ssm.send_command(
                InstanceIds=[instance_id],
                DocumentName="AWS-RunShellScript",
                Parameters='commands': [command_to_execute]
            )
            time.sleep(1)   

要在 cloudformation 模板中调用宏，您需要添加以下内容：

  ModifyInstances:
    Fn::Transform:
        Name: MacroSetUpCluster
        Parameters:
          env: !Ref MyEnv
          kafka: !Ref MyKafkaIP

堆栈 MacroSetUpCluster 需要事先使用 lambda 函数进行部署。

【讨论】：

非常感谢您的回复@Miguel！我已经添加了静态私有IP的设置并配置了多节点实例