java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider 在 flink 启动时

Posted 2023-03-10

【中文标题】java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider 在 flink 启动时

【英文标题】：java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider when flink boot up

【发布时间】：2021-03-11 16:43:41

【问题描述】：

我在 k8s 集群上运行 Flink 1.11，尝试更新 log4j-console.properties 文件时出现以下错误：

Starting Task Manager
Enabling required built-in plugins
Linking flink-s3-fs-hadoop-1.11.1.jar to plugin directory
Successfully enabled flink-s3-fs-hadoop-1.11.1.jar
sed: couldn't open temporary file /opt/flink/conf/sedl2dH0X: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedPLYAzY: Read-only file system
/docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml: Permission denied
sed: couldn't open temporary file /opt/flink/conf/sede0G5LW: Read-only file system
/docker-entrypoint.sh: 120: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting taskexecutor as a console application on host flink-taskmanager-c765c947c-qx68t.
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ser/FilterProvider
    at org.apache.logging.log4j.core.layout.JsonLayout.<init>(JsonLayout.java:158)
    at org.apache.logging.log4j.core.layout.JsonLayout.<init>(JsonLayout.java:69)
    at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:102)
    at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:77)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
    at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<clinit>(TaskManagerRunner.java:89)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
    at java.base/java.lang.ClassLoader.loadClass(Unknown Source)

我的 log4j-console.properties：

rootLogger.level = INFO
#rootLogger.appenderRef.console.ref = ConsoleAppender

appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %dHH:mm:ss.SSS [%t] %-5level %logger36 - %msg%n

appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.topic = test
appender.kafka.layout.type = JsonLayout
appender.kafka.layout.complete = false
appender.kafka.additional1.type = KeyValuePair
appender.kafka.additional1.key=app
appender.kafka.additional1.value=TEST
appender.kafka.additional2.type = KeyValuePair
appender.kafka.additional2.key=subsystem
appender.kafka.additional2.value=TEST
appender.kafka.additional3.type = KeyValuePair
appender.kafka.additional3.key=deployment
appender.kafka.additional3.value=TEST
appender.kafka.property.bootstrap.servers=***

rootLogger.appenderRef.console.ref = STDOUT
rootLogger.appenderRef.kafka.ref = Kafka

我正在使用“flink:1.11.1-scala_2.11-java11”docker 镜像并验证所有 log4j2 依赖项都在类路径中。

我也尝试从上面的基础镜像创建一个新的 docker 镜像，并添加缺少的依赖项，但什么也没发生。

【问题讨论】：

所有部门都在类路径中，但它们是否是正确的版本。这似乎是 jackson.databind 的共同主题

为什么不是正确的版本？ Flink 带来了那些……

我们可以看看你的 pom 吗？

我正在使用 Terraform 构建它，并在作业开始之前获取异常。因此我不认为 pom 是相关的。我将添加错误之前得到的日志。

您找到解决方案了吗？一旦我使用 JsonLayout 作为布局类型，我就会遇到同样的问题

【参考方案1】：

我也遇到过这个错误。这里的问题是，当任务管理器和作业管理器启动时，它们使用修改后的类路径运行，而不是您通过构建系统构建的 JAR。

参见flink source code 中的constructFlinkClassPath。为了证明这一点，请还原 JSON 日志记录模式并在启动时检查 tm/jm 日志中的类路径。您会注意到您的 JAR 不在类路径中。

要解决此问题，您需要向 tm/jm 节点内的 lib 文件夹（lib 文件夹默认情况下包含在 flink 类路径中）。

如果您使用的是 docker，则可以在构建容器时执行此操作 (RUN wget...)。