从 Python Lambda 函数连接 Athena SQL 查询中的变量

Posted 2023-03-28

【中文标题】从 Python Lambda 函数连接 Athena SQL 查询中的变量

【英文标题】：Concatenate variable in Athena SQL query from Python Lambda function

【发布时间】：2022-01-21 03:36:06

【问题描述】：

我有一个 Python Lambda 函数，可以在 Athena 中创建一个 SQL 表。如何正确连接查询中的变量？当我设置 LOCATION 值时，我收到下面的错误响应。如果我对 LOCATION 值进行硬编码，该函数将成功运行。

LOCATION “”” + s3_bucket_test + “””

错误响应：

Response

  "errorMessage": "An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: line 1:8: mismatched input 'EXTERNAL'. Expecting: 'OR', 'SCHEMA', 'TABLE', 'VIEW'",
  "errorType": "InvalidRequestException",
  "stackTrace": [
    "  File \"/var/task/lambda_function.py\", line 34, in lambda_handler\n    queryStart = client.start_query_execution(\n",
    "  File \"/var/runtime/botocore/client.py\", line 386, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n",
    "  File \"/var/runtime/botocore/client.py\", line 705, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n"
  ]

Lambda 函数：

import boto3
import json
import time

database = ‘daily_reports’
s3_bucket = 's3://test/’
s3_bucket_results = 's3://test/results’

query = ("""
          CREATE EXTERNAL TABLE IF NOT EXISTS `reports` (
              `timestamp` bigint,
              `user_id` string,
              `name` string
            )
            ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' 
            WITH SERDEPROPERTIES (
              'serialization.format' = '1'
            ) LOCATION “”” + s3_bucket + “””
            TBLPROPERTIES ('has_encrypted_data'='false');
        """)        


def lambda_handler(event, context):
    client = boto3.client('athena')
    queryStart = client.start_query_execution(
        QueryString = query,
        QueryExecutionContext = 
            'Database': database
        ,
        ResultConfiguration = 
            'OutputLocation': s3_bucket_results
        
    )

谢谢。

【参考方案1】：

你试过用 Python 的格式化方法吗？像这样的

query = ("""
          CREATE EXTERNAL TABLE IF NOT EXISTS `reports` (
              `timestamp` bigint,
              `user_id` string,
              `name` string
            )
            ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' 
            WITH SERDEPROPERTIES (
              'serialization.format' = '1'
            ) LOCATION '' 
            TBLPROPERTIES ('has_encrypted_data'='false');
        """).format(s3_bucket)    

【讨论】：

成功了！谢谢。