bigquery python 客户端：load_table_from_file 不适用于 csv 文件

Posted 2023-03-25

【中文标题】bigquery python 客户端：load_table_from_file 不适用于 csv 文件

【英文标题】：bigquery python client: load_table_from_file not working with csv file

【发布时间】：2020-04-03 22:06:49

【问题描述】：

我正在尝试从 csv 文件在现有 bigquery 表中追加新行。 csv是：

"sprotocol";"w5q53";"insertingdate";"closeddate";"sollectidate";"company";"companyid";"contact"
"20-22553";"DELETED";"2020-01-26;0000-01-01 00:00";"0000-01-01 00:00";"";"";"this is a ticket"

这是我的python函数：

job_config = bigquery.LoadJobConfig()
    job_config.source_format = 'text/csv'
    job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND
    job_config.source_format = bigquery.SourceFormat.CSV
    job_config.skip_leading_rows = 1
    job_config.autodetect = False
    job_config.schema = [
        bigquery.SchemaField("sprotocol", "STRING", mode="NULLABLE"),
        bigquery.SchemaField("w5q53", "STRING", mode="NULLABLE"),
        bigquery.SchemaField("insertingdate", "TIMESTAMP", mode="NULLABLE"),
        bigquery.SchemaField("closeddate", "STRING", mode="NULLABLE"),
        bigquery.SchemaField("sollectidate", "STRING", mode="NULLABLE"),
        bigquery.SchemaField("company", "STRING", mode="NULLABLE"),
        bigquery.SchemaField("companyid", "STRING", mode="NULLABLE"),
        bigquery.SchemaField("contact", "STRING", mode="NULLABLE")
    ]
    job_config.fieldDelimiter = ';'
    job_config.allow_quoted_newlines = True

    with open(file_path, "rb") as file:
        load_job = _connection.load_table_from_file(
            file,
            table_ref,
            job_config=job_config
        )  # API request
        print("Starting job ".format(load_job.job_id))

        load_job.result()  # Waits for table load to complete.
        print("Job finished.")
    file.close()

我收到以下错误：

['reason': 'invalid', 'message': 'Error while reading data, error message: CSV table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.', 'reason': 'invalid', 'message': 'Error while reading data, error message: CSV table references column position 55, but line starting at position:743 contains only 1 columns.']

我也尝试删除架构定义，但收到同样的错误。 有人可以帮助我吗？

【参考方案1】：

以上代码存在三个问题

使用field_delimiter 代替fieldDelimiter

job_config.field_delimiter = ';'

使用DATE 而不是TIMESTAMP，因为输入只包含日期

bigquery.SchemaField("insertingdate", "DATE", mode="NULLABLE"),

双引号不合适

"20-22553";"DELETED";"2020-01-26";"0000-01-01 00:00";"0000-01-01 00:00";"";"";"this is a ticket"

【讨论】：

您好，感谢您的回答。关于第 2 点，我必须参考 bigquery 表模式（是时间戳）。关于第 3 点，我删除了引用字段。问题与这一点有关。

@br1 很高兴解决了您的问题，您可以将其标记为答案