使用云功能将数据加载到大查询表中，它是附加到表中的，我需要它来替换

Posted 2023-02-22

【中文标题】使用云功能将数据加载到大查询表中，它是附加到表中的，我需要它来替换

【英文标题】：Using cloud function to load data into Big Query Table, it is appending to the table, I need it to replace

【发布时间】：2018-08-13 04:32:16

【问题描述】：

我有一个云函数，该函数当前将获取一个 .csv 文件，该文件登陆云存储并将该文件加载到大查询表中。问题是它正在附加它，我需要它来覆盖，我找到了一种使用命令行的方法 --replace 但不确定如何使用云函数在 .json 中执行此操作。以下是我当前的代码：

exports.ToBigQuery_Stage = (event, callback) => 
  const file = event.data;
  const context = event.context;

  const BigQuery = require('@google-cloud/bigquery');
  const Storage = require('@google-cloud/storage');

  const projectId = "gas-ddr";
  const datasetId = "gas_ddr_qc_stage";
  const bucketName = file.bucket;
  const filename = file.name;



  // Do not use the ftp_files Bucket to ensure that the bucket does not get crowded. 
  // Change bucket to gas_ddr_files_staging
  // Set the table name (TableId) to the full file name including date, 
  // this will give each table a new distinct name and we can keep a record of all of the files recieved.
  // This may not be the best way to do this... at some point we will need to archive and delete prior records. 
    const dashOffset = filename.indexOf('-');
  const tableId = filename.substring(0, dashOffset) + "_STAGE";


  console.log(`Load $filename into $tableId.`);

 // Instantiates clients
  const bigquery = new BigQuery(
    projectId: projectId,
  );

  const storage = Storage(
    projectId: projectId,
  );

  const metadata = 
  allowJaggedRows: true,
  skipLeadingRows: 1

 ;



  let job;

  // Loads data from a Google Cloud Storage file into the table
  bigquery
    .dataset(datasetId)
    .table(tableId)
    .load(storage.bucket(bucketName).file(filename),metadata)
    .then(results => 
      job = results[0];
      console.log(`Job $job.id started.`);

      // Wait for the job to finish
      return job;
    )
    .then(metadata => 
      // Check the job's status for errors
      const errors = metadata.status.errors;
      if (errors && errors.length > 0) 
        throw errors;
      
    )
    .then(() => 
      console.log(`Job $job.id completed.`);
    )
    .catch(err => 
      console.error('ERROR:', err);
    );

  callback();
;

【参考方案1】：

您可以将此添加到metadata：

const metadata = 
allowJaggedRows: true,
skipLeadingRows: 1,
writeDisposition: 'WRITE_TRUNCATE'
;

您可以在documentation 中找到更多信息。

【讨论】：

我想这就是我要找的。我会试一试的。