通过 Cloud Function 插入 BigQuery 不起作用

Posted 2023-02-22

【中文标题】通过 Cloud Function 插入 BigQuery 不起作用

【英文标题】：Insert into BigQuery via Cloud Function not working

【发布时间】：2019-04-16 15:01:29

【问题描述】：

我将以下 json 插入到预先创建的 BigQuery 表中。

  "Member_ID": 881230,
  "First_Name": "Dave2",
  "Last_Name": "Manin2",
  "Gender": "M",
  "Age": 53,
  "Height": "5,2",
  "Weight": 145,
  "Hours_Sleep": 4,
  "Calories_Consumed": 2497,
  "Exercise_Calories_Burned": 876,
  "Date": "2018-10-17"

当我将上面的行直接插入到表中时，它可以正常工作，但会通过函数输出错误（完整的错误消息见下文）。我错过了什么？任何帮助表示赞赏。

这是我的 node.js（代码后的错误信息）

exports.subscribe = function (event, callback) 
  const BigQuery = require('@google-cloud/bigquery');
  const projectId = "mydemo-221920"; //Enter your project ID here
  const datasetId = "mydemodata"; //Enter your BigQuery dataset name here
  const tableId = "memid"; //Enter your BigQuery table name here -- make sure it is setup correctly
  const PubSubMessage = event.data;
  // Incoming data is in JSON format
   const incomingData = PubSubMessage.data ? Buffer.from(PubSubMessage.data, 'base64').toString() : "'Member_ID':'na','First_Name':'na','Last_Name':'na','Gender':'na','Age':'na','Height':'na','Weight':'na','Hours_Sleep':'na','Calories_Consumed':'na','Exercise_Calories_Burned':'na','Date':'na'";
  console.log(`My log  PubSub en String est: $incomingData`);
  const jsonData = JSON.parse(incomingData);
var rows = [jsonData];

  console.log(`Uploading data: $JSON.stringify(rows)`);

  // Instantiates a client
  const bigquery = BigQuery(
    projectId: projectId
  );

  // Inserts data into memeid table
  bigquery
    .dataset(datasetId)
    .table(tableId)
    .insert(rows)
    .then((foundErrors) => 
      rows.forEach((row) => console.log('Inserted: ', row));

      if (foundErrors && foundErrors.insertErrors != undefined) 
        foundErrors.forEach((err) => 
            console.log('Error: ', err);
        )
      
    )
    .catch((err) => 
      console.error('ERROR:', err);
    );
  // [END bigquery_insert_stream]


  callback();

函数尝试插入时出现此错误

ERROR:  PartialFailureError: A failure occurred during this request. 
 at /user_code/node_modules/@google-cloud/bigquery/src/table.js:1213:13
 at Object.handleResp (/user_code/node_modules/@google-cloud/bigquery/node_modules/@google-cloud/common/src/util.js:135:3) 
 at /user_code/node_modules/@google-cloud/bigquery/node_modules/@google-cloud/common/src/util.js:465:12 
 at Request.onResponse [as _callback] (/user_code/node_modules/@google-cloud/bigquery/node_modules/retry-request/index.js:198:7)
 at Request.self.callback (/user_code/node_modules/@google-cloud/bigquery/node_modules/request/request.js:185:22)
 at emitTwo (events.js:106:13)
 at Request.emit (events.js:191:7)
 at Request.<anonymous> (/user_code/node_modules/@google-cloud/bigquery/node_modules/request/request.js:1161:10)
 at emitOne (events.js:96:13)
 at Request.emit (events.js:188:7) errors: [  errors: [Object], row: [Object]  ], response:  kind: 'bigquery#tableDataInsertAllResponse', insertErrors: [ [Object] ] , message: 'A failure occurred during this request.' 

函数正在读取这个 json

 insertId:  "000000-6dbc4c0f-fe15-4260-95c6-10afe7d0960b"  
 labels: …  
 logName:  "projects/mydemo-221920/logs/cloudfunctions.googleapis.com%2Fcloud-functions"  
 receiveTimestamp:  "2018-11-13T17:27:48.429240186Z"  
 resource: …  
 severity:  "INFO"  
 textPayload:  "My log  PubSub en String est: "Member_ID":881230,"First_Name":"Dave2","Last_Name":"Manin2","Gender":"M","Age":53,"Height":"5,2","Weight":145,"Hours_Sleep":4,"Calories_Consumed":2497,"Exercise_Calories_Burned":876,"Date":"2018-10-17""  
 timestamp:  "2018-11-13T17:27:42.143Z"  
 trace:  "projects/mydemo-221920/traces/cef3531fe182bf1d6da6e47aae3bbff3"  

【问题讨论】：

“错误输出”是什么意思？如果您看到错误消息，请编辑您的问题以具体说明您正在观察的内容。

请注意，您可以编辑您的问题以添加更多详细信息，而不是使用评论。然后，您可以格式化错误消息，使其更易于阅读。

搞定了。谢谢你的指点。边走边学 :)

【参考方案1】：

我在您的代码或 json 中看不到任何错误。我尝试了您提供的代码，并且能够将数据流式传输到 BigQuery 中而不会出现任何错误：

我尝试修改我的 BigQuery 表的架构以重现您的问题，但我得到了与您完全相同的错误：

这是架构应该可以解决您的问题：