如何使用 Nodejs 将本地 Avro 上传到谷歌大查询

Posted 2023-03-25

【中文标题】如何使用 Nodejs 将本地 Avro 上传到谷歌大查询

【英文标题】：How to upload a local Avro to google big query using Nodejs

【发布时间】：2016-06-09 11:21:01

【问题描述】：

我正在尝试使用 nodejs 将本地 Avro 上传到谷歌大查询

var gcloud = require('gcloud')(
  keyFilename: './config/keyfile.json',
  projectId: 'my-project'
);

var bigquery = gcloud.bigquery();


var schoolsDataset = bigquery.dataset('my_dataset');
var schoolsTable = schoolsDataset.table('person_data');


schoolsTable.import('./examples/yob1900.avro', function(err, job) 
console.log(job)
console.log(err)
);

我收到以下错误：

 [ApiError: No schema specified on job or table.]
  code: 400,
  errors: 
   [  domain: 'global',
       reason: 'invalid',
       message: 'No schema specified on job or table.'  ],
  response: undefined,
  message: 'No schema specified on job or table.' 

如何添加架构？ （“yob1900”文件是从 bigquery 文档中下载的 https://cloud.google.com/bigquery/loading-data#loading_json_files)

【参考方案1】：

答案如下：

var gcloud = require('gcloud')(
  keyFilename: '../config/keyfile.json',
  projectId: 'my-project'
);

var request = require('request');

var bigquery = gcloud.bigquery();


var dataset = bigquery.dataset('my_dataset');
var table = dataset.table('my_table');

var metadata = 
  sourceFormat: 'AVRO'
;

fs = require('fs');

fs.createReadStream('./yob1900.avro')
  .pipe(table.createWriteStream(metadata))
  .on('complete', function(job) 
   job
      .on('error', console.log)
      .on('complete', function(metadata) 
        console.log('job completed', metadata);
      );
  );