spark 实战 1： 基于gettyimages spark docker image 创建spark 集群

Posted

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了spark 实战 1： 基于gettyimages spark docker image 创建spark 集群相关的知识，希望对你有一定的参考价值。


1、首先下载该镜像到本地。https://hub.docker.com/r/gettyimages/spark/

~$ docker pull gettyimages/spark


2、从https://github.com/gettyimages/docker-spark/blob/master/docker-compose.yml下载编写好的支持 Spark 集群的 docker-compose.yml 文件


启动它

$ docker-compose up

$ docker-compose up

Creating spark_master_1

Creating spark_worker_1

Attaching to spark_master_1, spark_worker_1

master_1  | 16/10/29 13:25:05 INFO master.Master: Started daemon with process name: [email protected]

master_1  | 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for TERM

master_1  | 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for HUP

master_1  | 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for INT

master_1  | 16/10/29 13:25:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing view acls to: root

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing modify acls to: root

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing view acls groups to:

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing modify acls groups to:

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()

worker_1  | 16/10/29 13:25:06 INFO worker.Worker: Started daemon with process name: [email protected]

worker_1  | 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for TERM

worker_1  | 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for HUP

worker_1  | 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for INT

master_1  | 16/10/29 13:25:07 INFO util.Utils: Successfully started service ‘sparkMaster‘ on port 7077.

master_1  | 16/10/29 13:25:07 INFO master.Master: Starting Spark master at spark://master:7077

master_1  | 16/10/29 13:25:07 INFO master.Master: Running Spark version 2.0.1

master_1  | 16/10/29 13:25:07 INFO util.log: Logging initialized @2542ms

master_1  | 16/10/29 13:25:07 INFO server.Server: jetty-9.2.z-SNAPSHOT

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/app,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/app/json,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/json,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/static,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/app/kill,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/driver/kill,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{0.0.0.0:8080}

master_1  | 16/10/29 13:25:07 INFO server.Server: Started @2742ms

master_1  | 16/10/29 13:25:07 INFO util.Utils: Successfully started service ‘MasterUI‘ on port 8080.

master_1  | 16/10/29 13:25:07 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://localhost:8080

master_1  | 16/10/29 13:25:07 INFO server.Server: jetty-9.2.z-SNAPSHOT

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{master:6066}

master_1  | 16/10/29 13:25:07 INFO server.Server: Started @2779ms

master_1  | 16/10/29 13:25:07 INFO util.Utils: Successfully started service on port 6066.

master_1  | 16/10/29 13:25:07 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066

worker_1  | 16/10/29 13:25:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing view acls to: root

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing modify acls to: root

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing view acls groups to:

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing modify acls groups to:

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()

master_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/metrics/master/json,null,AVAILABLE}

master_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/metrics/applications/json,null,AVAILABLE}

master_1  | 16/10/29 13:25:08 INFO master.Master: I have been elected leader! New state: ALIVE

worker_1  | 16/10/29 13:25:08 INFO util.Utils: Successfully started service ‘sparkWorker‘ on port 8881.

worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Starting Spark worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM

worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Running Spark version 2.0.1

worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Spark home: /usr/spark-2.0.1

worker_1  | 16/10/29 13:25:08 INFO util.log: Logging initialized @2625ms

worker_1  | 16/10/29 13:25:08 INFO server.Server: jetty-9.2.z-SNAPSHOT

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/logPage,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/logPage/json,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/json,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/static,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/log,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{0.0.0.0:8081}

worker_1  | 16/10/29 13:25:08 INFO server.Server: Started @2749ms

worker_1  | 16/10/29 13:25:08 INFO util.Utils: Successfully started service ‘WorkerUI‘ on port 8081.

worker_1  | 16/10/29 13:25:08 INFO ui.WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://localhost:8081

worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Connecting to master master:7077...

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/metrics/json,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO client.TransportClientFactory: Successfully created connection to master/172.17.0.2:7077 after 41 ms (0 ms spent in bootstraps)

master_1  | 16/10/29 13:25:09 INFO master.Master: Registering worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM

worker_1  | 16/10/29 13:25:09 INFO worker.Worker: Successfully registered with master spark://master:7077

本文出自 “鱼翔浅底” 博客，请务必保留此出处http://yanwe.blog.51cto.com/10031173/1869988

以上是关于spark 实战 1： 基于gettyimages spark docker image 创建spark 集群的主要内容，如果未能解决你的问题，请参考以下文章

大数据-基于Spark的机器学习-智能客户系统项目实战

基于Spark+Grafana可视化电商项目实战，好文收藏~

大数据-基于Spark的机器学习-智能客户系统项目实战

基于Python Spark的大数据分析_pyspark实战项目课程

Spark Streaming实时流处理项目实战Spark Streaming整合Flume实战二

Spark Streaming实时流处理项目实战Spark Streaming整合Flume实战二