尚硅谷电商数仓 Flume写入hdfs报错

Posted 2022-10-24 刘润森！

使用Flume消费kafka中数据，sink到hdfs中出现数据压缩格式支持错误；

java.lang.IllegalArgumentException: Unsupported compression codec lzop.  Please choose from: [None, BZip2Codec, DefaultCodec, DeflateCodec, GzipCodec, Lz4Codec, SnappyCodec, ZStandardCodec]
	at org.apache.flume.sink.hdfs.HDFSEventSink.getCodec(HDFSEventSink.java:345)
	at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:248)

锁定关键词 LZO

hadoop本身并不支持lzo压缩，故需要使用twitter提供的hadoop-lzo开源组件。

core-site.xml增加配置支持LZO压缩

<configuration>
    <property>
        <name>io.compression.codecs</name>
        <value>
            org.apache.hadoop.io.compress.GzipCodec,
            org.apache.hadoop.io.compress.DefaultCodec,
            org.apache.hadoop.io.compress.BZip2Codec,
            org.apache.hadoop.io.compress.SnappyCodec,
            com.hadoop.compression.lzo.LzoCodec,
            com.hadoop.compression.lzo.LzopCodec
        </value>
    </property>

    <property>
        <name>io.compression.codec.lzo.class</name>
        <value>com.hadoop.compression.lzo.LzoCodec</value>
    </property>
</configuration>

上传 hadoop-lzo-0.4.20.jar 到 hadoop-3.1.3/share/hadoop/common/,也可以将这个jar拷贝到自己flume的lib目录下