Pig AvroStorage + 记录中不支持的类型：类 org.apache.pig.data.DataByteArray

Posted 2023-04-18

【中文标题】Pig AvroStorage + 记录中不支持的类型：类 org.apache.pig.data.DataByteArray

【英文标题】：Pig AvroStorage + Unsupported type in record:class org.apache.pig.data.DataByteArray

【发布时间】：2014-11-14 22:49:23

【问题描述】：

我正在尝试读取 csv 文件作为输入数据并以 avro 格式写入输出。

注意：- Pig 版本 Apache Pig 版本 0.12.1.2.1.5.0-695

REGISTER /usr/lib/pig/lib/avro-1.7.4.jar; 
REGISTER /usr/lib/pig/lib/piggybank.jar;
REGISTER /usr/lib/pig/lib/jackson-mapper-asl-1.8.8.jar;
REGISTER /usr/lib/pig/lib/jackson-core-asl-1.8.8.jar;
REGISTER /usr/lib/pig/lib/json-simple-1.1.1.jar;

 A = LOAD '/data/raw/event'; 

store A into '/data/dev/raw/pig' 

using org.apache.pig.piggybank.storage.avro.AvroStorage('no_schema_check',
'schema', '   
  "name" : "EVENT",
  "type" : "record",  
  "fields" : [ 
    "name" : "evt",
    "type" : [ "long", "null" ]
  , 
    "name" : "mac",
    "type" : [ "int", "null" ]
  , 
    "name" : "sec",
    "type" : [ "int", "null" ]
   ]

'); 

我得到以下异常：

ERROR 2997: Unable to recreate exception from backed error: Error: org.apache.avro.file.DataFileWriter$AteException: java.lang.RuntimeException:
 Unsupported type in record:class org.apache.pig.data.DataByteArray

        at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263)
        at org.apache.pig.piggybank.storage.avro.PigAvroRecordWriter.write(PigAvroRecordWriter.java:49)
        at org.apache.pig.piggybank.storage.avro.AvroStorage.putNext(AvroStorage.java:749)


Caused by: java.lang.RuntimeException: Unsupported type in record:class org.apache.pig.data.DataByteArray
        at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.getField(PigAvroDatumWriter.java:385)
        at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.writeRecord(PigAvroDatumWriter.java:363)

如果我遗漏了任何事情或是否存在任何解决方法，请告诉我

【参考方案1】：

默认情况下，Pig 会将所有字段加载为 DataByteArray。 所以你必须使用如下模式加载数据

A = LOAD '/data/raw/event' as (evt:long,mac,int,sec:int)