谷歌云如何在使用 Python api 创建实例时启动服务？

Posted

技术标签:

【中文标题】谷歌云如何在使用 Python api 创建实例时启动服务？【英文标题】：Google cloud how to start service when creating instance using Python api? 【发布时间】：2021-10-06 14:58:42 【问题描述】：

我在创建实例并使用 Python Google Cloud 的 api 启动它们时遇到了很多麻烦。

基本上，我首先创建了所有虚拟机并设置了服务。然后，一旦一切正常，我停止了虚拟机并创建了Image，这样我就可以使用图像轻松创建实例了。

[编辑] 我觉得很高兴补充一下：

前端实例：nextjs 后端实例：nodejs 数据库实例：postgresql

这是我的 Python 脚本：

import googleapiclient.discovery
import argparse
import os
import time

backend_script = """
#! /bin/bash
cd /mse-tsm-cloudsys-lab1/api/
sudo npm install
sudo npx knex migrate:latest
sudo npx knex seed:run
sudo npm run start
"""

frontend_script = """
#! /bin/bash
cd /mse-tsm-cloudsys-lab1/app/
sudo npm install
sudo npm run build
sudo npm run start
"""

def create_instance(compute, project, zone, name, image, machine, ip, ip_public, tags=[], metadata=[]):
    config = 
        'name': name,
        'machineType': "zones/%s/machineTypes/%s" % (zone, machine),
        'disks': [
            
                'initializeParams': 
                    "sourceImage": image
                ,
                "boot": True
            
        ],
        'tags': 
            "items": tags
        ,
        "networkInterfaces": [
            
                'network': 'global/networks/default',
                'networkIP': ip,
                'accessConfigs': [
                    
                        'type': 'ONE_TO_ONE_NAT',
                        'name': 'External NAT',
                        'natIP': ip_public
                    
                ]
            
        ],
        "metadata": 
            "items": metadata
        
    
    return compute.instances().insert(
        project=project,
        zone=zone,
        body=config).execute()

def delete_instance(compute, project, zone, name):
    return compute.instances().delete(
        project=project,
        zone=zone,
        instance=name).execute()

def list_instances(compute, project, zone):
    result = compute.instances().list(project=project, zone=zone).execute()
    return result['items'] if 'items' in result else None

def wait_for_operation(compute, project, zone, operation):
    print('Waiting for operation to finish...')
    while True:
        result = compute.zoneOperations().get(
            project=project,
            zone=zone,
            operation=operation).execute()

        if result['status'] == 'DONE':
            print("done.")
            if 'error' in result:
                raise Exception(result['error'])
            return result

        time.sleep(1)

def main(mode, project, zone, wait=True):
    compute = googleapiclient.discovery.build('compute', 'v1')

    if(mode == 'list'):
        instances = list_instances(compute, project, zone)

        print('Instances in project %s and zone %s:' % (project, zone))
        for instance in instances:
            print(' - ' + instance['name'])

        return

    print("Creating Database")
    instance_name = "postgres-lab1"
    image = "projects/tsm-cloudsys-vial/global/images/image-postgres-lab1"
    machine = "e2-micro"
    ip = "10.132.0.2"
    ip_public = "34.79.195.77"
    tags = ["postgres-lab1"]
    operation1 = create_instance(compute, project, zone, instance_name, image, machine, ip, ip_public)

    print("Creating Backend")
    instance_name = "backend-lab1"
    image = "projects/tsm-cloudsys-vial/global/images/image-backend-lab1"
    machine = "e2-micro"
    ip = "10.132.0.4"
    ip_public = "34.79.111.217"
    metadata = [
        "key": "startup-script",
        "value": backend_script
    ]
    tags = ["backend-lab1","http-server","https-server"]
    operation2 = create_instance(compute, project, zone, instance_name, image, machine, ip, ip_public, metadata=metadata)

    print("Creating Frontend")
    instance_name = "frontend-lab1"
    image = "projects/tsm-cloudsys-vial/global/images/image-frontend-lab1"
    machine = "e2-micro"
    ip = "10.132.0.3"
    ip_public = "35.187.109.153"
    metadata = [
        "key": "startup-script",
        "value": frontend_script
    ]
    tags = ["frontend-lab1","http-server","https-server"]
    operation3 = create_instance(compute, project, zone, instance_name, image, machine, ip, ip_public, tags, metadata=metadata)

    print("Waiting for VMs to come online")
    wait_for_operation(compute, project, zone, operation1['name'])
    wait_for_operation(compute, project, zone, operation2['name'])
    wait_for_operation(compute, project, zone, operation3['name'])

    instances = list_instances(compute, project, zone)

    print('Instances in project %s and zone %s:' % (project, zone))
    for instance in instances:
        print(' - ' + instance['name'])

    if wait:
        input()

    print("Deleting DB")
    operation1 = delete_instance(compute, project, zone, "postgres-lab1")
    print("Deleting Backend")
    operation2 = delete_instance(compute, project, zone, "backend-lab1")
    print("Deleting Frontend")
    operation3 = delete_instance(compute, project, zone, "frontend-lab1")
    wait_for_operation(compute, project, zone, operation1['name'])
    wait_for_operation(compute, project, zone, operation2['name'])
    wait_for_operation(compute, project, zone, operation3['name'])

if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('--mode', help='Wanted mode', default='list')
    parser.add_argument('project_id', help='Your Google Cloud project ID.')
    parser.add_argument(
        '--zone',
        default='europe-west1-b',
        help='Compute Engine zone to deploy to.')

    args = parser.parse_args()

    main(args.mode, args.project_id, args.zone)

3 个实例已创建并正在运行，我可以 ping 通所有实例。 （是的，一个已关闭，因为我正在尝试重新启动它） 但我有两个主要问题：

    在 Google Cloud 仪表板上使用 SSH 连接到实例是不可能的，这需要很长时间... 我觉得我的startup-script 不正确，因为即使 3 个实例正在运行，我也无法像以前手动创建它们时那样访问它们。例如frontend-lab1 应该在端口 3000 上响应，但不是在我使用脚本创建它时

我主要关心的是第 2 个问题，因为我认为它会解决第 1 个问题。有人可以帮我解决我的startup-script 吗？

[编辑2] 这是日志：

ct  6 19:24:23 frontend-lab1 systemd[1]: Starting Google Compute Engine Startup Scripts...
Oct  6 19:24:23 frontend-lab1 google_metadata_script_runner[795]: 2021/10/06 19:24:23 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct  6 19:24:37 frontend-lab1 systemd[1]: systemd-fsckd.service: Succeeded.
Oct  6 19:24:51 frontend-lab1 systemd[1]: systemd-hostnamed.service: Succeeded.
Oct  6 19:25:21 frontend-lab1 GCEGuestAgent[539]: 2021-10-06T19:25:21.4654Z GCEGuestAgent Info: Removing user jeromevialhes.
Oct  6 19:25:22 frontend-lab1 google_guest_agent[539]: 2021/10/06 19:25:22 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct  6 19:25:22 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:22 GCEMetadataScripts: startup-script: /tmp/metadata-scripts111777770/startup-script: line 3: cd: /mse-tsm-cloudsys-lab1/app/: No such file or directory
Oct  6 19:25:23 frontend-lab1 google_metadata_script_runner[795]: 2021/10/06 19:25:23 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN saveError ENOENT: no such file or directory, open '/package.json'
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN enoent ENOENT: no such file or directory, open '/package.json'
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No description
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No repository field.
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No README data
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No license field.
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: 
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: up to date in 0.489s
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: found 0 vulnerabilities
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: 
Oct  6 19:25:27 frontend-lab1 google_metadata_script_runner[795]: 2021/10/06 19:25:27 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! code ENOENT
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! syscall open
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! path /package.json
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! errno -2
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! enoent ENOENT: no such file or directory, open '/package.json'
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! enoent This is related to npm not being able to find a file.
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! enoent 
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: 
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! A complete log of this run can be found in:
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR!     /root/.npm/_logs/2021-10-06T19_25_28_453Z-debug.log
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! code ENOENT
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! syscall open
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! path /package.json
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! errno -2
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! enoent ENOENT: no such file or directory, open '/package.json'
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! enoent This is related to npm not being able to find a file.
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! enoent 
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: 
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! A complete log of this run can be found in:
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR!     /root/.npm/_logs/2021-10-06T19_25_29_171Z-debug.log
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script exit status 254
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: Finished running startup scripts.
Oct  6 19:25:29 frontend-lab1 systemd[1]: google-startup-scripts.service: Succeeded.
Oct  6 19:25:29 frontend-lab1 systemd[1]: Finished Google Compute Engine Startup Scripts.

但是你可以看到仓库是存在的：

jeromevialhes@frontend-lab1:~$ ls
mse-tsm-cloudsys-lab1

【问题讨论】：

您是否在端口 22 (SSH) 上创建了防火墙规则，以便您可以连接到实例？在 VPC 网络 API 上？ @razimbres 是的，我为我需要的所有端口创建了我需要的所有规则（这就是我手动创建 VM 实例时它起作用的原因）但我真的认为问题来自我的startup-script 这也可能对您有所帮助：cloud.google.com/compute/docs/instance-templates/…，而不是图像或快照 我写了一篇关于如何计算启动时间的文章。这也将帮助您了解在哪里查找启动脚本问题：jhanley.com/google-compute-startup-script-total-execution-time 我不确定您使用的是哪个操作系统，了解要查看哪些日志文件很重要。 2) 安装 npm/node 需要时间，有时五分钟或更长时间。当您说您可以访问时，请准确定义其含义（测试和错误）。 @JohnHanley 我会看看你的文章！基本上，当我手动进行设置时，我打开浏览器并执行http://ipfrontend:3000 并且我可以访问前端，但是当我执行上面的脚本时，实例正在运行（如图所示）但是当我尝试访问时http://ipfrontend:3000 我的浏览器出现错误，因为前端没有运行 --> 这就是为什么我认为我的 script-frontend 写得不正确 【参考方案1】：

查看您在问题中包含的日志，您的custom 图像可能存在问题。在日志文件的第 7 行，它警告说找不到目录“/mse-tsm-cloudsys-labl1/app/”：

Oct  6 19:25:22 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:22 GCEMetadataScripts: startup-script: /tmp/metadata-scripts111777770/startup-script: line 3: cd: /mse-tsm-cloudsys-lab1/app/: No such file or directory

如果此目录用于包含您的应用程序文件，这将导致与 NPM 无法找到诸如 package.json 等文件和其他属于您的应用程序的文件相关的后续错误。

您应该确认应用程序路径是否正确或图像是否正确创建。您可以按照 this 指南从现有 VM 实例创建自定义映像，并使用 Node.js 和 Python 指南确保您遵循正确的步骤。

【讨论】：

以上是关于谷歌云如何在使用 Python api 创建实例时启动服务？的主要内容，如果未能解决你的问题，请参考以下文章

谷歌云视觉 API - Python

如何在谷歌云平台中使用私有 IP 从我的电脑连接到 sql 实例

如何在谷歌云数据流中停止流式传输管道

身份验证不适用于使用 NodeJS 进行的谷歌云 API 身份验证

如何在谷歌云功能中使用谷歌表格 API

如何将谷歌云自然语言实体情感响应转换为 Python 中的 JSON/dict？