build jar(sbt)

Posted 2020-10-08 satyrs

project 中遇到

1. log file & project file
2. build.sbt
3. startup.sh
4. then command line:　　> sh startup.sh

example:

.sbt

name := "Simple Project" //project name
version := "1.0"  //project version
scalaVersion := "2.11.8" //scala version
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.1.0"
libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.2.11"
//all Dependencies should be aware of the version compatibility

.sh

 /usr/local/spark/bin/spark-submit --driver-class-path /usr/local/spark/jars/*:/usr/local/spark/jars/kafka/* --class "org.apache.spark.examples.streaming.KafkaWordCount" /usr/local/spark/mycode/kafka/target/scala-2.11/simple-project_2.11-1.0.jar 127.0.0.1:2181 1 sex 1

spark-submit目录 
参数：

--driver-class-path　　spark中的所有jars以及kafka中所有jars

--class　　工程文件main所在class + 生成的jar的位置

127.0.0.1:2181 ： zookeeper-server kafka-server ip+port

1 sex 1： consumer group；topics； number of consumers

REFERENCE:
http://dblab.xmu.edu.cn