[Spark]Spark-streaming通过Receiver方式实时消费Kafka流程（Yarn-cluster）

Posted 2021-01-23 skyell

1.启动zookeeper

2.启动kafka服务（broker）

[[email protected] kafka_2.11-0.10.2.1]# ./bin/kafka-server-start.sh config/server.properties

3.启动kafka的producer（前提：已经创建好topic

[[email protected] kafka_2.11-0.10.2.1]# ./bin/kafka-console-producer.sh --broker-list master:9092 --topic test

4.启动kafka的consumer

[[email protected] kafka_2.11-0.10.2.1]#./bin/kafka-console-consumer.sh --zookeeper master:2181 --topic test --from-beginning

5.打jar包，将带有依赖的jar包上传到集群上

mvn clean assembly:assembly

6.编写启动脚本，启动任务 sh run_receiver.sh

/usr/local/src/spark-2.0.2-bin-hadoop2.6/bin/spark-submit        --class com.skyell.streaming.ReceiverFromKafka        --master yarn-cluster         --executor-memory 1G         --total-executor-cores 2         --files $HIVE_HOME/conf/hive-site.xml         ./Spark8Pro-2.0-SNAPSHOT-jar-with-dependencies.jar

监控任务及查看日志

http://master:8088/cluster

关闭spark streaming任务

yarn application -kill application_1539421032843_0093

数据驱动变革-云将 个人博客地址