terraform 将云作曲家从 1.0 升级到 2.0？

Posted 2023-03-24

【中文标题】terraform 将云作曲家从 1.0 升级到 2.0？

【英文标题】：terraform upgrade cloud composer from 1.0 to 2.0?

【发布时间】：2021-12-01 10:46:57

【问题描述】：

我尝试使用 terraform 将 composer 从 1.0 升级到 2.0。我收到以下错误。你能帮我解决问题吗？问题出在哪里？

Terraform 已成功初始化！ $ terraform apply -input=false $计划 module.composer.google_composer_environment.composer：正在创建...

│ Warning: Version constraints inside provider configuration blocks are deprecated
│ 
│   on provider.tf line 2, in provider "google":
│    2:     version = "~> 3.22"
│ 
│ Terraform 0.13 and earlier allowed provider version constraints inside the
│ provider configuration block, but that is now deprecated and will be
│ removed in a future version of Terraform. To silence this warning, move the
│ provider version constraint into the required_providers block.
╵
╷
│ Error: googleapi: Error 400: Found 2 problems:
│   1) Error validating property [core]store_dag_code. Overriding [core]store_dag_code is not permitted. 
│   2) Error validating property [core]store_serialized_dags. Overriding [core]store_serialized_dags is not permitted. , badRequest
│ 
│   with module.composer.google_composer_environment.composer,
│   on ../modules/composer/main.tf line 2, in resource "google_composer_environment" "composer":
│    2: resource "google_composer_environment" "composer" 
│ 
╵
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1

【参考方案1】：

目前不支持将 Composer 1.x 版本升级到 Composer 2.x。实际上，用户需要从头开始创建 Composer 2.x。

可以使用迁移工具将 DAG/任务的历史从 Composer1/Airflow 1 迁移到 Composer 2/Airflow 2 - 但这是一个手动步骤：https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/composer/tools/composer_db_transfer.md

【参考方案2】：

Terraform 错误，因为 GCP Cloud Composer 不允许从 Composer 1 升级到 Composer 2。见Composer upgrade limitations。

您只能在 相同的主要版本，例如 composer-1.12.4-airflow-1.10.10 到 composer-1.13.0-airflow-1.10.10.

升级自 composer-1.4.0-airflow-1.10.0 到 composer-2.0.0-airflow-1.10.0 不是 允许，因为 Cloud Composer 主要版本从 1 更改为 2.

升级的唯一解决方法是创建一个新的 Cloud Composer 2.0 实例。

【讨论】：

我正在使用创建一个新实例。地形后端。我不会替换现有实例 backend "gcs" bucket = "xxxxxxxxxx" prefix = "data-engineering-infrastructure/xxxx/xxxxxx/xxxxxxx"

terraform backend "gcs" bucket = "xxxxxxxx" prefix = "xxxxxxx/gcp/xxxxx/dev_upgrade"

@Dataflower 我明白了。标题有点误导，因为说升级到 2.0。在创建 2.0 实例时将问题的标题和内容更改为错误可能会更好？您还可以在问题中包含您的 terraform 代码。