Zeppelin - Flink 问题运行基础教程

Posted 2023-03-23

【中文标题】Zeppelin - Flink 问题运行基础教程

【英文标题】：Zeppelin - Flink problem running basic tutorial

【发布时间】：2022-01-01 16:55:39

【问题描述】：

我在 Ubuntu 20.04 LTS 中安装 Apache Flink 1.14.0 for Scala 2.12 和 Zeppelin 0.10.0， 是为了有一个用于简单开发的笔记本。我已按照以下所有说明进行操作，但默认笔记本示例无法执行：

Flink Notebook 错误

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to open FlinkInterpreter
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:833)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:741)
    at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
    at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
    at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to open FlinkInterpreter
    at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:80)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
    ... 8 more
Caused by: java.lang.IllegalArgumentException: argument type mismatch
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at org.apache.zeppelin.flink.FlinkInterpreter.loadFlinkScalaInterpreter(FlinkInterpreter.java:98)
    at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:74)
    ... 9 more

我使用的是默认 Java

java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

当我运行代码示例时，Flink 运行良好

./bin/flink run examples/streaming/WordCount.jar
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/home/flink/flink-1.14.0/lib/flink-dist_2.12-1.14.0.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Job has been submitted with JobID b35bc72fce8d34990f32de1af7bcdc6b

Program execution finished
Job with JobID b35bc72fce8d34990f32de1af7bcdc6b has finished.
Job Runtime: 495 ms

root@ecs-flink:/home/flink/flink-1.14.0# tail log/flink-*-taskexecutor-*.out
(nymph,1)
(in,3)
(thy,1)
(orisons,1)
(be,4)
(all,2)
(my,1)
(sins,1)
(remember,1)
(d,4)

Zeppelin 笔记本代码

%flink
val data = benv.fromElements("hello world", "hello flink", "hello hadoop")
data.flatMap(line => line.split("\\s"))
             .map(w => (w, 1))
             .groupBy(0)
             .sum(1)
             .print()

Zeppelin Flink 解释器配置

FLINK_HOME ==> /home/flink/flink-1.14.0 (I've tried /home/flink/flink-1.14.0/, /home/flink/ etc)
flink.execution.mode ==> local (also tried remote)
flink.execution.remote.host ==> localhost (or ip address)
flink.execution.remote.port ==> 8081 (default port)

############################################## ###########################

普通安装

# Upgrade and update
sudo apt update
sudo apt upgrade

# Install Java
sudo apt install default-jre -y
java -version

############################################## ###########################

FLINK 安装

mkdir /home/flink
cd /home/flink
wget https://dlcdn.apache.org/flink/flink-1.14.0/flink-1.14.0-bin-scala_2.12.tgz
tar -zxvf flink*.tgz
cd flink-1.14.0/


# Copy Flink Python from opt/ to lib/
cp opt/flink-python_2.12-1.14.0.jar lib/

# Start Flink
./bin/start-cluster.sh


# Submitting a Flink Job
./bin/flink run examples/streaming/WordCount.jar
tail log/flink-*-taskexecutor-*.out

############################################## ###########################

齐柏林飞艇安装

mkdir /home/zeppelin
cd /home/zeppelin
wget https://dlcdn.apache.org/zeppelin/zeppelin-0.10.0/zeppelin-0.10.0-bin-all.tgz
tar -zxvf zeppelin-0.10.0-bin-all.tgz
cd zeppelin-0.10.0-bin-all

# Install interpreters
./bin/install-interpreter.sh --all

# Configure Zeppelin
cp conf/zeppelin-site.xml.template conf/zeppelin-site.xml
vi conf/zeppelin-site.xml
    # To access a remote Zeppelin, you need to change zeppelin.server.addr to 0.0.0.0 in conf/zeppelin-site.xml.

# Configure Zeppelin Authentication
# See more details on https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html
cp conf/shiro.ini.template conf/shiro.ini
vi conf/shiro.ini

# Start Zeppelin
bin/zeppelin-daemon.sh start

############################################## ###########################

有用的链接

https://nightlies.apache.org/flink/flink-docs-release-1.14//docs/try-flink/local_installation/ https://flink.apache.org/downloads.html#apache-flink-1140

https://zeppelin.apache.org/docs/latest/quickstart/install.html http://zeppelin.apache.org/download.html

有什么想法吗？非常感谢！

【参考方案1】：

我也有同样的问题。显然这是一个齐柏林飞艇的错误。 使用 scala-2.11 尝试 zeppelin 0.9.0 和 Flink 1.12.5。