单机Flink TaskExecutor no more allocated slots 问题解决
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了单机Flink TaskExecutor no more allocated slots 问题解决相关的知识，希望对你有一定的参考价值。
参考技术A测试环境:
原因： 出现该问题的主要原因是因为在我们本地测试时，默认 jobmanager 配置的地址是 localhost ，但 taskmanager 在初始化时使用的ip却是本机局域网IP，如 192.168.*；10.* 等，造成任务执行过程中JobManager向TaskManager申请计算资源执行任务时无法正常获取 Slots 算子
官方配置说明文档 ，修改完成后重启集群
重新运行示例即可正常执行
Apache Flink 本地单机部署及简单操作
JDK安装版本要求8.0以上
tar xf jdk-8u161-linux-x64.tar.gz -C /usr/local/
mv /usr/local/{jdk1.8.0_161,jdk}
#vim /etc/profile.d/jdk.sh
export JAVA_HOME=/usr/local/jdk
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
# exec bash
#java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
下载安装包
wget https://archive.apache.org/dist/flink/flink-1.7.2/flink-1.7.2-bin-scala_2.11.tgz
解压
tar xf flink-1.7.2-bin-scala_2.11.tgz
进入目录启动
cd flink-1.7.2/bin/ [root@master bin]# ./start-cluster.sh Starting cluster. Starting standalonesession daemon on host master. Starting taskexecutor daemon on host master. [root@master bin]# ps -aux | grep java root 17762 168 7.7 4666352 298976 pts/1 Sl 13:58 0:42 /usr/local/jdk/bin/java -Xms1024m -Xmx1024m -Dlog.file=/root/flink-1.7.2/log/flink-root-standalonesession-0-master.log -Dlog4 j.configuration=file:/root/flink-1.7.2/conf/log4j.properties -Dlogback.configurationFile=file:/root/flink-1.7.2/conf/logback.xml -classpath /root/flink-1.7.2/lib/flink-python_2.11-1.7.2.jar:/root/flink-1.7.2/lib/log4j-1.2.17.jar:/root/flink-1.7.2/lib/slf4j-log4j12-1.7.15.jar:/root/flink-1.7.2/lib/flink-dist_2.11-1.7.2.jar::/usr/local/src/hadoop-2.6.1/etc/hadoop: org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint --configDir /root/flink-1.7.2/conf --executionMode clusterroot 18206 92.5 3.6 4580892 142368 pts/1 Sl 13:58 0:20 /usr/local/jdk/bin/java -XX:+UseG1GC -Xms922M -Xmx922M -XX:MaxDirectMemorySize=8388607T -Dlog.file=/root/flink-1.7.2/log/flin k-root-taskexecutor-0-master.log -Dlog4j.configuration=file:/root/flink-1.7.2/conf/log4j.properties -Dlogback.configurationFile=file:/root/flink-1.7.2/conf/logback.xml -classpath /root/flink-1.7.2/lib/flink-python_2.11-1.7.2.jar:/root/flink-1.7.2/lib/log4j-1.2.17.jar:/root/flink-1.7.2/lib/slf4j-log4j12-1.7.15.jar:/root/flink-1.7.2/lib/flink-dist_2.11-1.7.2.jar::/usr/local/src/hadoop-2.6.1/etc/hadoop: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /root/flink-1.7.2/conf
查看启动的默认端口
[root@master bin]# ss -lntp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
users:(("sshd",pid=9086,fd=3))LISTEN 0 100 127.0.0.1:25 *:*
users:(("master",pid=9290,fd=13))LISTEN 0 128 :::6123 :::*
users:(("java",pid=17762,fd=38))LISTEN 0 128 :::8081 :::*
users:(("java",pid=17762,fd=87))LISTEN 0 128 :::22 :::*
users:(("sshd",pid=9086,fd=4))LISTEN 0 100 ::1:25 :::*
users:(("master",pid=9290,fd=14))LISTEN 0 128 ::ffff:192.168.10.14:45594 :::*
users:(("java",pid=18206,fd=65))LISTEN 0 128 :::33466 :::*
users:(("java",pid=17762,fd=59))LISTEN 0 128 :::40387 :::*
users:(("java",pid=18206,fd=58))LISTEN 0 128 :::43075 :::*
users:(("java",pid=18206,fd=39))LISTEN 0 128 :::46372 :::*
users:(("java",pid=17762,fd=40))
浏览器访问地址：IP:8081
提交一个作业操作
#nc -l 9000 [root@master flink-1.7.2]# ./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000 Starting execution of program
浏览器里查看作业
在NC进程输入单词
[root@master bin]# nc -l 9000 chenxi qwe chenxi chenxi chenxi qwe hh \\ jhh jghgh ghghg gg kkk chenxi xhenxi^H^H^H xhenxi xhenxi
查看日志
[root@master log]# tail -f flink-root-taskexecutor-0-master.out chenxi : 1 qwe : 1 chenxi : 1 chenxi : 1 chenxi : 1 qwe : 1 hh : 1 jhh : 1 \\ : 1 jghgh : 1 ghghg : 1 gg : 1 kkk : 1 chenxi : 1 xhe : 1 xhenxi : 2
以上是关于单机Flink TaskExecutor no more allocated slots 问题解决的主要内容，如果未能解决你的问题，请参考以下文章
FlinkFlink 1.12.2 源码浅析 : TaskExecutor
FlinkDiscard registration from TaskExecutor SplitFetcher received unexpected exception