使用 Java 读取 parquet 文件，但它在本地机器上工作，在 docker 容器中不工作
Posted
技术标签:
【中文标题】使用 Java 读取 parquet 文件，但它在本地机器上工作，在 docker 容器中不工作【英文标题】：read a parquet file using Java, but it works in local machine, and doesn't work in docker container 【发布时间】：2021-10-27 22:16:19 【问题描述】：我需要在 Java 独立应用程序中读取 parquet 文件并发布到 Kafka。我有下面的代码来读取由 spark scala 应用程序生成的 parquet 文件。
public void readTest(Path path)
try
ParquetMetadata readFooter = ParquetFileReader.readFooter(conf, path, ParquetMetadataConverter.NO_FILTER);
List<BlockMetaData> blocks = readFooter.getBlocks();
System.out.println("Blocks size: "+blocks.size());
Map<String, String> keyValueMetaData = readFooter.getFileMetaData().getKeyValueMetaData();
System.out.println("K-v metadata: "+keyValueMetaData);
System.out.println("Created by: "+readFooter.getFileMetaData().getCreatedBy());
MessageType schema = readFooter.getFileMetaData().getSchema();
System.out.println("Schema: "+schema);
ParquetFileReader r = new ParquetFileReader(conf, path, readFooter);
PageReadStore pages = null;
try
while (null != (pages = r.readNextRowGroup()))
final long rows = pages.getRowCount();
System.out.println("Number of rows: " + rows);
final MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
final RecordReader recordReader = columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
for (int i = 0; i < rows; i++)
final Group g = (Group) recordReader.read();
System.out.println("group: "+g);
printGroup(g);
finally
r.close();
catch (Exception e)
System.out.println("Error reading parquet file.");
e.printStackTrace();
java 应用程序是一个 Spring Boot 非 Web 应用程序。 此代码在本地或 intelliJ IDE 中运行时运行良好，但当相同的文件和代码被 dockerized 时，我收到以下错误。
............Processing File name............: part-00000-2b69fe41-592a-485b-85e8-8971c5842155-c000.snappy.parquet
07:45:50.770 [main] WARN o.a.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-08-28 07:45:50 DEBUG Tracer:106 - sampler.classes = ; loaded no samplers
2021-08-28 07:45:51 DEBUG Tracer:128 - span.receiver.classes = ; loaded no span receivers
Blocks size: 1
K-v metadata: org.apache.spark.sql.parquet.row.metadata="type":"struct","fields":["name":"first_name","type":"string","nullable":true,"metadata":,"name":"last_name","type":"string","nullable":true,"metadata":,"name":"email","type":"string","nullable":true,"metadata":,"name":"gender","type":"string","nullable":true,"metadata":]
Created by: parquet-mr version 1.8.2 (build c6522788629e590a53eb79874b95f6c3ff11f16c)
Schema: message spark_schema
optional binary first_name (STRING);
optional binary last_name (STRING);
optional binary email (STRING);
optional binary gender (STRING);
07:45:52.073 [main] INFO o.a.hadoop.io.compress.CodecPool - Got brand-new decompressor [.snappy]
Number of rows: 1000
Error reading parquet file.
java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:275)
at org.xerial.snappy.Snappy.uncompress(Snappy.java:553)
at org.apache.parquet.hadoop.codec.SnappyDecompressor.decompress(SnappyDecompressor.java:71)
at org.apache.parquet.hadoop.codec.NonBlockedDecompressorStream.read(NonBlockedDecompressorStream.java:51)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:286)
at org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:237)
at org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.<init>(PlainValuesDictionary.java:91)
at org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.<init>(PlainValuesDictionary.java:74)
at org.apache.parquet.column.Encoding$1.initDictionary(Encoding.java:90)
at org.apache.parquet.column.Encoding$5.initDictionary(Encoding.java:163)
at org.apache.parquet.column.impl.ColumnReaderBase.<init>(ColumnReaderBase.java:413)
at org.apache.parquet.column.impl.ColumnReaderImpl.<init>(ColumnReaderImpl.java:46)
at org.apache.parquet.column.impl.ColumnReadStoreImpl.getColumnReader(ColumnReadStoreImpl.java:82)
at org.apache.parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:271)
at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:147)
at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:109)
at org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:177)
at org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:109)
at org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:80)
at com.csp.cdp.CPRemediationIntegrationPub.readers.NFSStorageReader.readTest(NFSStorageReader.java:131)
at com.csp.cdp.CPRemediationIntegrationPub.readers.NFSStorageReader.read(NFSStorageReader.java:59)
at com.csp.cdp.CPRemediationIntegrationPub.readers.NFSStorageReader.read(NFSStorageReader.java:38)
at com.csp.cdp.CPRemediationIntegrationPub.executors.AppExecutor.execute(AppExecutor.java:29)
at com.csp.cdp.CPRemediationIntegrationPub.CpRemediationIntegrationPubApplication.main(CpRemediationIntegrationPubApplication.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
我的 Dockerfile 供参考。
FROM openjdk:8-jdk-alpine
RUN mkdir /app
RUN mkdir /app/files/
COPY testDir /app/files/
COPY normal /app/files/
WORKDIR /app
COPY target/appname-0.0.1-SNAPSHOT.jar /app/app.jar
ENTRYPOINT ["java","-jar","app.jar"]
请帮忙。
【问题讨论】：
【参考方案1】：我把基础图片改成
FROM openjdk:8
它就像一个魅力。不确定早期的基本映像有什么问题。我认为镶木地板不支持特殊的 unicode 字符。
【讨论】：
我的错误是： 引起：java.lang.IllegalArgumentException: newLimit > capacity: (506 > 326) 我正在使用图像 eclipse-temurin:11-jre-alpine 我切换到 eclipse-temurin :11-jre-focal 现在没有问题了，谢谢以上是关于使用 Java 读取 parquet 文件，但它在本地机器上工作，在 docker 容器中不工作的主要内容，如果未能解决你的问题，请参考以下文章
如何使用 Parquet.net 从 Parquet 文件中仅读取列的一部分？
使用 java 反射运行时，HDFS Parquet 文件阅读器抛出 DistributedFileSystem.class 未找到