使用java.Without使用作业加载数据，将json数据流式传输到Bigquery

Posted 2021-04-16

我是bigquery的新手，所以我不完全理解如何将数据流传输到bigquery，这是我的问题，我有jsonInString，从这样的对象映射

    String customerJsonInString = mapper.writeValueAsString(customer);

   {
  "id": "1",
  "first_name": "John",
  "last_name": "Doe",
  "dob": "1968-01-22",
  "addresses": [
    {
      "status": "current",
      "address": "123 First Avenue",
      "city": "Seattle",
      "state": "WA",
      "zip": "11111",
      "numberOfYears": "1"
    },
    {
      "status": "previous",
      "address": "456 Main Street",
      "city": "Portland",
      "state": "OR",
      "zip": "22222",
      "numberOfYears": "5"
    }
  ]
}

该表已使用正确的架构创建。现在我想将这些数据流式传输到bigquery（插入行），我正在使用（https://cloud.google.com/bigquery/streaming-data-into-bigquery#bigquery-stream-data-java）上的示例来使它适应我的，这是我试过的，

TableId tableId = TableId.of(DATASET_NAME,TABLE_NAME);
        Map<String, Object> recordsContent = new HashMap<>();
        recordsContent.put("Customer", customerJsonInString);
        InsertAllResponse response = bigquery.insertAll(InsertAllRequest.newBuilder(tableId)
                        .addRow("rowId", recordsContent)
                        .build());
        if (response.hasErrors()) {
            for (Entry<Long, List<BigQueryError>> entry : response.getInsertErrors().entrySet()) {
            }
        }

答案

我猜这张桌子的架构是id, first_name, last_name, ...而不是customer？在这种情况下，您需要逐个设置字段而不是整个json字符串，例如recordsContent.put("id", 1)。 检查example with comments。