数学之路-分布式计算-storm

Posted 2020-09-19 zhchoutai

5、安装zookeeper

本博客全部内容是原创。假设转载请注明来源

http://blog.csdn.net/myhaspl/

[email protected]:~/jzmq-master$cd ..

[email protected]:~$wget http://apache.fayea.com/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz

 

[email protected]:~$tar -xzvf zookeeper-3.3.6.tar.gz

$cd zookeeper-3.3.6

[email protected]:~/zookeeper-3.3.6$ cp./conf/zoo_sample.cfg ./conf/zoo.cfg

[email protected]:~/zookeeper-3.3.6$bin/zkServer.sh start

JMX enabled by default

[email protected]:~/zookeeper-3.3.6$ sudobin/zkServer.sh start

JMX enabled by default

Using config:/home/myhaspl/zookeeper-3.3.6/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[email protected]:~/zookeeper-3.3.6$bin/zkCli.sh -server 127.0.0.1:2181

Connecting to 127.0.0.1:2181

2014-12-17 16:22:17,187 - INFO  [main:[email protected]] - Clientenvironment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT

2014-12-17 16:22:17,207 - INFO  [main:[email protected]] - Clientenvironment:host.name=aaayun

2014-12-17 16:22:17,208 - INFO  [main:[email protected]] - Clientenvironment:java.version=1.7.0_71

2014-12-17 16:22:17,209 - INFO  [main:[email protected]] - Clientenvironment:java.vendor=Oracle Corporation

2014-12-17 16:22:17,210 - INFO  [main:[email protected]] - Clientenvironment:java.home=/usr/lib/java/jre

2014-12-17 16:22:17,212 - INFO  [main:[email protected]] - Clientenvironment:java.class.path=/home/myhaspl/zookeeper-3.3.6/bin/../build/classes:/home/myhaspl/zookeeper-3.3.6/bin/../build/lib/*.jar:/home/myhaspl/zookeeper-3.3.6/bin/../zookeeper-3.3.6.jar:/home/myhaspl/zookeeper-3.3.6/bin/../lib/log4j-1.2.15.jar:/home/myhaspl/zookeeper-3.3.6/bin/../lib/jline-0.9.94.jar:/home/myhaspl/zookeeper-3.3.6/bin/../src/java/lib/*.jar:/home/myhaspl/zookeeper-3.3.6/bin/../conf:.:/usr/lib/java/lib:/usr/lib/java/jre/lib

2014-12-17 16:22:17,213 - INFO  [main:[email protected]] - Clientenvironment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2014-12-17 16:22:17,276 - INFO  [main:[email protected]] - Clientenvironment:java.io.tmpdir=/tmp

2014-12-17 16:22:17,277 - INFO  [main:[email protected]] - Clientenvironment:java.compiler=<NA>

2014-12-17 16:22:17,278 - INFO  [main:[email protected]] - Clientenvironment:os.name=Linux

2014-12-17 16:22:17,278 - INFO  [main:[email protected]] - Clientenvironment:os.arch=amd64

2014-12-17 16:22:17,279 - INFO  [main:[email protected]] - Clientenvironment:os.version=3.2.0-53-generic

2014-12-17 16:22:17,279 - INFO  [main:[email protected]] - Clientenvironment:user.name=myhaspl

2014-12-17 16:22:17,280 - INFO  [main:[email protected]] - Clientenvironment:user.home=/home/myhaspl

2014-12-17 16:22:17,281 - INFO  [main:[email protected]] - Clientenvironment:user.dir=/home/myhaspl/zookeeper-3.3.6

2014-12-17 16:22:17,286 - INFO  [main:[email protected]] - Initiating clientconnection, connectString=127.0.0.1:2181 sessionTi[email protected]6c0dc01

2014-12-17 16:22:17,408 - INFO [main-SendThread():[email protected]] - Opening socketconnection to server /127.0.0.1:2181

Welcome to ZooKeeper!

2014-12-17 16:22:17,796 - INFO [main-SendThread(localhost:2181):[email protected]] - Socketconnection established to localhost/127.0.0.1:2181, initiating session

JLine support is enabled

2014-12-17 16:22:18,204 - INFO [main-SendThread(localhost:2181):[email protected]] - Sessionestablishment complete on server localhost/127.0.0.1:2181, sessionid =0x14a575462220000, negotiated timeout = 30000

 

WATCHER::

 

WatchedEvent state:SyncConnected type:Nonepath:null

[zk: 127.0.0.1:2181(CONNECTED) 0] ls /

[zookeeper]

[zk: 127.0.0.1:2181(CONNECTED) 1] quit

6、storm安装

[email protected]:~/zookeeper-3.3.6$ cd ..

[email protected]:~$ wget http://mirrors.cnnic.cn/apache/storm/apache-storm-0.9.3/apache-storm-0.9.3.tar.gz

[email protected]:~$ tar -xzvf  apache-storm-0.9.3.tar.gz

启动storm

[email protected]:~/apache-storm-0.9.3$ sudo  ./bin/storm nimbus&

[1] 25871

[email protected]:~/apache-storm-0.9.3$ sudo  ./bin/storm supervisor&

[2] 25882

编辑參数配置

[email protected]:~/apache-storm-0.9.3$

[email protected]:~/apache-storm-0.9.3$ cdconf

[email protected]:~/apache-storm-0.9.3/conf$ls

默认配置例如以下:

# Licensed to the Apache SoftwareFoundation (ASF) under one

# or more contributor licenseagreements.  See the NOTICE file

# distributed with this work for additionalinformation

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version2.0 (the

# "License"); you may not usethis file except in compliance

# with the License.  You may obtain a copy of the License at

#

#http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law oragreed to in writing, software

# distributed under the License isdistributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANYKIND, either express or implied.

# See the License for the specific languagegoverning permissions and

# limitations under the License.

 

 

########### These all have default valuesas shown

########### Additional configuration goesinto storm.yaml

 

java.library.path:"/usr/local/lib:/opt/local/lib:/usr/lib"

 

### storm.* configs are generalconfigurations

# the local dir is where jars are kept

storm.local.dir: "storm-local"

storm.zookeeper.servers:

    -"localhost"

storm.zookeeper.port: 2181

storm.zookeeper.root: "/storm"

storm.zookeeper.session.timeout: 20000

storm.zookeeper.connection.timeout: 15000

storm.zookeeper.retry.times: 5

storm.zookeeper.retry.interval: 1000

storm.zookeeper.retry.intervalceiling.millis:30000

storm.cluster.mode: "distributed"# can be distributed or local

storm.local.mode.zmq: false

storm.thrift.transport:"backtype.storm.security.auth.SimpleTransportPlugin"

storm.messaging.transport:"backtype.storm.messaging.netty.Context"

 

### nimbus.* configs are for the master

nimbus.host: "localhost"

nimbus.thrift.port: 6627

nimbus.thrift.max_buffer_size: 1048576

nimbus.childopts: "-Xmx1024m"

nimbus.task.timeout.secs: 30

nimbus.supervisor.timeout.secs: 60

nimbus.monitor.freq.secs: 10

nimbus.cleanup.inbox.freq.secs: 600

nimbus.inbox.jar.expiration.secs: 3600

nimbus.task.launch.secs: 120

nimbus.reassign: true

nimbus.file.copy.expiration.secs: 600

nimbus.topology.validator:"backtype.storm.nimbus.DefaultTopologyValidator"

 

### ui.* configs are for the master

ui.port: 8080

ui.childopts: "-Xmx768m"

 

logviewer.port: 8000

logviewer.childopts: "-Xmx128m"

logviewer.appender.name: "A1"

 

 

drpc.port: 3772

drpc.worker.threads: 64

drpc.queue.size: 128

drpc.invocations.port: 3773

drpc.request.timeout.secs: 600

drpc.childopts: "-Xmx768m"

 

transactional.zookeeper.root:"/transactional"

transactional.zookeeper.servers: null

transactional.zookeeper.port: null

 

### supervisor.* configs are for nodesupervisors

# Define the amount of workers that can berun on this machine. Each worker is assigned a port to use for communication

supervisor.slots.ports:

    -6700

    -6701

    -6702

    -6703

supervisor.childopts: "-Xmx256m"

#how long supervisor will wait to ensurethat a worker process is started

supervisor.worker.start.timeout.secs: 120

#how long between heartbeats untilsupervisor considers that worker dead and tries to restart it

supervisor.worker.timeout.secs: 30

#how frequently the supervisor checks onthe status of the processes it‘s monitoring and restarts if necessary

supervisor.monitor.frequency.secs: 3

#how frequently the supervisor heartbeatsto the cluster state (for nimbus)

supervisor.heartbeat.frequency.secs: 5

supervisor.enable: true

 

### worker.* configs are for task workers

worker.childopts: "-Xmx768m"

worker.heartbeat.frequency.secs: 1

 

task.heartbeat.frequency.secs: 3

task.refresh.poll.secs: 10

 

zmq.threads: 1

zmq.linger.millis: 5000

zmq.hwm: 0

 

 

storm.messaging.netty.server_worker_threads:1

storm.messaging.netty.client_worker_threads:1

storm.messaging.netty.buffer_size: 5242880#5MB buffer

storm.messaging.netty.max_retries: 30

storm.messaging.netty.max_wait_ms: 1000

storm.messaging.netty.min_wait_ms: 100

 

### topology.* configs are for specificexecuting storms

topology.enable.message.timeouts: true

topology.debug: false

topology.optimize: true

topology.workers: 1

topology.acker.executors: null

topology.tasks: null

# maximum amount of time a message has tocomplete before it‘s considered failed

topology.message.timeout.secs: 30

topology.skip.missing.kryo.registrations:false

topology.max.task.parallelism: null

topology.max.spout.pending: null

topology.state.synchronization.timeout.secs:60

topology.stats.sample.rate: 0.05

topology.builtin.metrics.bucket.size.secs:60

topology.fall.back.on.java.serialization:true

topology.worker.childopts: null

topology.executor.receive.buffer.size: 1024#batched

topology.executor.send.buffer.size: 1024#individual messages

topology.receiver.buffer.size: 8 # settingit too high causes a lot of problems (heartbeat thread gets starved, throughputplummets)

topology.transfer.buffer.size: 1024 #batched

topology.tick.tuple.freq.secs: null

topology.worker.shared.thread.pool.size: 4

topology.disruptor.wait.strategy:"com.lmax.disruptor.BlockingWaitStrategy"

topology.spout.wait.strategy:"backtype.storm.spout.SleepSpoutWaitStrategy"

topology.sleep.spout.wait.strategy.time.ms:1

topology.error.throttle.interval.secs: 10

topology.max.error.report.per.interval: 5

topology.kryo.factory:"backtype.storm.serialization.DefaultKryoFactory"

topology.tuple.serializer:"backtype.storm.serialization.types.ListDelegateSerializer"

topology.trident.batch.emit.interval.millis:500

 

dev.zookeeper.path:"/tmp/dev-storm-zookeeper"

Storm发行版本号解压文件夹下有一个conf/storm.yaml文件。用于配置Storm。默认配置在defaults.yaml能够查看。conf/storm.yaml中的配置选项将覆盖defaults.yaml中的默认配置。

启动后台ui

[email protected]:~/apache-storm-0.9.3$ sudo ./bin/stormui&

[3] 25893