Spark Idea Maven 开发环境搭建
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Spark Idea Maven 开发环境搭建相关的知识,希望对你有一定的参考价值。
一、安装jdk
jdk版本最好是1.7以上,设置好环境变量,安装过程,略。
二、安装Maven
我选择的Maven版本是3.3.3,安装过程,略。
编辑Maven安装目录conf/settings.xml文件,
<!-- 修改Maven 库存放目录--> <localRepository>D:\maven-repository\repository</localRepository>
三、安装Idea
安装过程,略。
四、创建Spark项目
1、新建一个Spark项目,
2、选择Maven,从模板创建项目,
3、填写项目GroupId等,
4、选择本地安装的Maven和Maven配置文件。
5、next
6、创建完毕,查看新项目结构:
7、自动更新Maven pom文件
8、编译项目
如果出现这种错误,这个错误是由于Junit版本造成的,可以删掉Test,和pom.xml文件中Junit的相关依赖,
即删掉这两个Scala类:
和pom.xml文件中的Junit依赖:
<dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> </dependency>
9、刷新Maven依赖
10、引入Jdk和Scala开发库
11、在pom.xml加入相关的依赖包,包括Hadoop、Spark等
<dependency> <groupId>commons-logging</groupId> <artifactId>commons-logging</artifactId> <version>1.1.1</version> <type>jar</type> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> <version>3.1</version> </dependency> <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.9</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.7.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.7.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.7.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.5.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.10</artifactId> <version>1.5.1</version> </dependency>
然后刷新maven的依赖,
12、新建一个Scala Object。
测试代码为:
def main(args: Array[String]) { println("Hello World!") val sparkConf = new SparkConf().setMaster("local").setAppName("test") val sparkContext = new SparkContext(sparkConf) }
执行,
如果报了以下错误,
java.lang.SecurityException: class "javax.servlet.FilterRegistration"‘s signer information does not match signer information of other classes in the same package at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) at java.lang.ClassLoader.defineClass(ClassLoader.java:794) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at org.spark-project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:136) at org.spark-project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:129) at org.spark-project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:98) at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:110) at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:101) at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:78) at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:62) at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:62) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:62) at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:61) at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:74) at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:190) at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:141) at org.apache.spark.SparkContext.<init>(SparkContext.scala:466) at com.test.Test$.main(Test.scala:13) at com.test.Test.main(Test.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
可以把servlet-api 2.5 jar删除即可:
最好的办法是删除pom.xml中相关的依赖,即
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.7.1</version> </dependency>
最后的pom.xml文件的依赖是
<dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.7.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.7.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.5.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.10</artifactId> <version>1.5.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_2.10</artifactId> <version>1.5.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.10</artifactId> <version>1.5.2</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-mllib_2.10</artifactId> <version>1.5.2</version> </dependency> <dependency> <groupId>com.databricks</groupId> <artifactId>spark-avro_2.10</artifactId> <version>2.0.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.10</artifactId> <version>1.5.2</version> </dependency> </dependencies>
如果是报了这个错误,也没有什么问题,程序依旧可以执行,
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:272) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2084) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2084) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2084) at org.apache.spark.SparkContext.<init>(SparkContext.scala:311) at com.test.Test$.main(Test.scala:13) at com.test.Test.main(Test.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
最后看到的正常输出:
Hello World! Using Spark‘s default log4j profile: org/apache/spark/log4j-defaults.properties 16/09/19 11:21:29 INFO SparkContext: Running Spark version 1.5.1 16/09/19 11:21:29 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:272) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2084) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2084) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2084) at org.apache.spark.SparkContext.<init>(SparkContext.scala:311) at com.test.Test$.main(Test.scala:13) at com.test.Test.main(Test.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) 16/09/19 11:21:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/09/19 11:21:30 INFO SecurityManager: Changing view acls to: pc 16/09/19 11:21:30 INFO SecurityManager: Changing modify acls to: pc 16/09/19 11:21:30 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(pc); users with modify permissions: Set(pc) 16/09/19 11:21:30 INFO Slf4jLogger: Slf4jLogger started 16/09/19 11:21:31 INFO Remoting: Starting remoting 16/09/19 11:21:31 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:52500] 16/09/19 11:21:31 INFO Utils: Successfully started service ‘sparkDriver‘ on port 52500. 16/09/19 11:21:31 INFO SparkEnv: Registering MapOutputTracker 16/09/19 11:21:31 INFO SparkEnv: Registering BlockManagerMaster 16/09/19 11:21:31 INFO DiskBlockManager: Created local directory at C:\Users\pc\AppData\Local\Temp\blockmgr-f9ea7f8c-68f9-4f9b-a31e-b87ec2e702a4 16/09/19 11:21:31 INFO MemoryStore: MemoryStore started with capacity 966.9 MB 16/09/19 11:21:31 INFO HttpFileServer: HTTP File server directory is C:\Users\pc\AppData\Local\Temp\spark-64cccfb4-46c8-4266-92c1-14cfc6aa2cb3\httpd-5993f955-0d92-4233-b366-c9a94f7122bc 16/09/19 11:21:31 INFO HttpServer: Starting HTTP Server 16/09/19 11:21:31 INFO Utils: Successfully started service ‘HTTP file server‘ on port 52501. 16/09/19 11:21:31 INFO SparkEnv: Registering OutputCommitCoordinator 16/09/19 11:21:31 INFO Utils: Successfully started service ‘SparkUI‘ on port 4040. 16/09/19 11:21:31 INFO SparkUI: Started SparkUI at http://192.168.51.143:4040 16/09/19 11:21:31 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. 16/09/19 11:21:31 INFO Executor: Starting executor ID driver on host localhost 16/09/19 11:21:31 INFO Utils: Successfully started service ‘org.apache.spark.network.netty.NettyBlockTransferService‘ on port 52520. 16/09/19 11:21:31 INFO NettyBlockTransferService: Server created on 52520 16/09/19 11:21:31 INFO BlockManagerMaster: Trying to register BlockManager 16/09/19 11:21:31 INFO BlockManagerMasterEndpoint: Registering block manager localhost:52520 with 966.9 MB RAM, BlockManagerId(driver, localhost, 52520) 16/09/19 11:21:31 INFO BlockManagerMaster: Registered BlockManager 16/09/19 11:21:31 INFO SparkContext: Invoking stop() from shutdown hook 16/09/19 11:21:32 INFO SparkUI: Stopped Spark web UI at http://192.168.51.143:4040 16/09/19 11:21:32 INFO DAGScheduler: Stopping DAGScheduler 16/09/19 11:21:32 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 16/09/19 11:21:32 INFO MemoryStore: MemoryStore cleared 16/09/19 11:21:32 INFO BlockManager: BlockManager stopped 16/09/19 11:21:32 INFO BlockManagerMaster: BlockManagerMaster stopped 16/09/19 11:21:32 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 16/09/19 11:21:32 INFO SparkContext: Successfully stopped SparkContext 16/09/19 11:21:32 INFO ShutdownHookManager: Shutdown hook called 16/09/19 11:21:32 INFO ShutdownHookManager: Deleting directory C:\Users\pc\AppData\Local\Temp\spark-64cccfb4-46c8-4266-92c1-14cfc6aa2cb3 Process finished with exit code 0
至此,开发环境搭建完毕。
以上是关于Spark Idea Maven 开发环境搭建的主要内容,如果未能解决你的问题,请参考以下文章
IDEA+maven搭建scala开发环境(spark)(半转载)
走进大数据 | hadoop spark环境搭建及idea scala maven集成开发spark任务