在使用 BigQuery API 的 python 插入作业期间登录失败

Posted 2023-03-25

【中文标题】在使用 BigQuery API 的 python 插入作业期间登录失败

【英文标题】：Login failed during python Insert job with BigQuery API

【发布时间】：2016-06-10 17:15:10

【问题描述】：

我正在尝试通过设置服务器-服务器身份验证将本地文件加载到 bigquery。 我已经完成了以下步骤

已创建服务帐号 为此帐户创建 JSON 密钥文件

使用

激活服务帐户

gcloud auth activate-service-account 命令

用

登录

gcloud 认证登录

尝试执行 python 脚本将文件上传到 BigQuery

范围 =

['https://www.googleapis.com/auth/bigquery',
         'https://www.googleapis.com/auth/bigquery.insertdata']

credentials = ServiceAccountCredentials.from_json_keyfile_name(
    '/path/privatekey.json', scopes)
# Construct the service object for interacting with the BigQuery API.
service = build('bigquery', 'v2', credentials=credentials)

# Load configuration with the destination specified.
load_config = 
    'destinationTable': 
        'projectId': "project id",
        'datasetId': "data set id",
        'tableId': "table name"
    


# Setup the job here.
# load[property] = value
load_config['schema'] = 
    'fields': [
        <several field>
    ]



upload = MediaFileUpload('/path/to/csv/file',
                         mimetype='application/octet-stream',
                         # This enables resumable uploads.
                         resumable=True)
# End of job configuration.

run_load.start_and_wait(service.jobs(),
                        "my project id",
                        load_config,
                        media_body=upload)

结果是

   "error": 
    "errors": [
   
    "domain": "global",
    "reason": "required",
    "message": "Login Required",
    "locationType": "header",
    "location": "Authorization"
   
  ],
  "code": 401,
  "message": "Login Required"
 

但我有足够的权限来创建查询作业

query_request = service.jobs()
query_data = 
    'query': (
        'SELECT COUNT(*) FROM [dmrebg.testDay];')


query_response = query_request.query(
    projectId=project_id,
    body=query_data).execute()

print('Query Results:')
for row in query_response['rows']:
    print('\t'.join(field['v'] for field in row['f']))

---

我错过了什么？我以为我已经登录了。

【参考方案1】：

问题是任何调用 https://www.googleapis.com/bigquery/v2/projects/project_id/jobs/* 会导致同样的问题

  "error": 
    "errors": [
   
    "domain": "global",
    "reason": "required",
    "message": "Login Required",
    "locationType": "header",
    "location": "Authorization"
   
  ],
  "code": 401,
  "message": "Login Required"
 

原来是我的浏览器鉴权有问题，python鉴权好。

---

根本原因是我的 CSV 架构和数据不匹配。

Errors:
Too many errors encountered. (error code: invalid)