IT十八掌 徐培成 大数据 课堂笔记 storm第二天

Posted 2020-08-30

storm

------------
stream计算.实时性。

master/slave
nimbus //master
supervisor //work node
//zk

spout //龙头,源源不断产生数据
nextTuple()

bolt //接头.

tuple //List<>

storm.yaml
--------------
zkserver:
local.dir=
slot.ports //槽位
- 6700 //worker process
- 6701
executor //执行线程
task //任务
topology //拓扑。hadoop job

Spout extends BaseRichSpout{
open(){
...
}

nextTuple(){

}

public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("site"));
}
}

Bolt extends BaseBasicBolt{
execute(Tuple input, BasicOutputCollector collector){
...
}

public void declareOutputFields(OutputFieldsDeclarer declarer) {
..
}
}

storm操作模式
---------------------
1.local
LocalCluster //本地集群
2.StormSubmitter

deactivate
-------------------
暂停top运行，不在产生新的tuple，但是已经发出的tuple继续运行。
$>storm deactivate com.it18zhang.storm2.MyApp

activate
-----------------
$>storm activate com.it18zhang.storm2.MyApp

kill
-----------------
//先进入deactive工作
$>storm kill LearningStormSingleNodeTopology

配置拓扑的并行
-----------------
1.worker prcess
jvm.
storm可以同时运行多个top.
一个worker process只隶属于一个top，并且可以中该top中的多个组件。
top之间无法共享worker process.

2.executor //执行线程
在worker process内部，可以有多个executor线程，他们可执行top的一部分。
executor只能执行一个组件。
每个executor都是一个单独的线程，只能按序执行指派给他的tasks.
在top执行期间可以动态修改执行线程的个数。
容易控制各种组件的并发度。

3.task
top执行过程的最小单位。
每个task是spout或者bolt的一个实例。
定义top时，可以指定spout/bolt的task数量。
定义之后，运行期间，task不能修改。
task可以单独执行，也可以和其他同类型的task执行。

配置并发程度
--------------------
1.worker process
Config conf = ...
conf.setNumberWorkers(3); //设置进程数
2.executor数
TopologyBuiler.setSpout(?,?,x); //x 为Spout执行线程数
TopologyBuiler.setBolt(?,?,x); //x bolt的执行线程数
3.task数
SpoutDeclarer spoutDeclare = builder.setSpout("LearningStormSpout", new MySpout(), 4);
spoutDeclare.setNumTasks(2);

BoltDeclarer boltDeclare = builder.setBolt("LearningStormBolt", new MyBolt(), 2);
boltDeclare.shuffleGrouping("LearningStormSpout");
boltDeclare.setNumTasks(4);

4.计算并发度
worker process‘s num = 3 ; //top

spout executor‘s num = 3 ;
bolt executor‘s num = 3 ;

bolt task‘s num = 6 ;

并发度 = spout‘task + bolt task

再平衡:rebalance
-------------------
1.如果增加新的supervisor之后，如果不进行再平衡处理，则新的supervisor是idle。
2.再平衡
a.web ui
b.cli
$>storm rebalance [TopologyName] -n [NumberOfWorkers] -e [Spout]=[NumberOfExecutos] -e [Bolt1]=[NumberOfExecutos] [Bolt2]=[NumberOfExecutos]
$>storm rebalance MyStormApp -n 2 -e spout=1 -e bolt=2
3.
4.

com.it18zhang.storm2.MyApp

 

流分组
--------------------
1.Suffle
随机选择一个bolt接收tuple。尽可能在bolt之间平均分配。
2.FieldsGroup
字段分组，是根据tuple中指定的字段值进行hash处理之后，进入指定的task。
MySpount.emit()
declear("site")

MyBolt.execute()

3.
4.
5.

-----xcall from s400 ------
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]
s400#7116#40-Thread-7-spout-executor[6 6]#[email protected]#[email protected]

-----xcall from s200 ------
s200#7360#46-Thread-13-bolt-executor[5 5]#[email protected]#[email protected]
s200#7360#46-Thread-13-bolt-executor[5 5]#[email protected]#[email protected]
s200#7360#46-Thread-13-bolt-executor[5 5]#[email protected]#[email protected]
s200#7360#46-Thread-13-bolt-executor[5 5]#[email protected]#[email protected]
s200#7360#46-Thread-13-bolt-executor[5 5]#[email protected]#[email protected]

s200#7360#40-Thread-7-bolt-executor[3 3]#[email protected]#[email protected]
s200#7360#40-Thread-7-bolt-executor[3 3]#[email protected]#[email protected]
s200#7360#40-Thread-7-bolt-executor[3 3]#[email protected]#[email protected]
s200#7360#40-Thread-7-bolt-executor[3 3]#[email protected]#[email protected]
s200#7360#40-Thread-7-bolt-executor[3 3]#[email protected]#[email protected]
s200#7360#40-Thread-7-bolt-executor[3 3]#[email protected]#[email protected]

 

-----xcall from s200 ------
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]01a1#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]
s200#7360#38-Thread-5-spout-executor[7 7]#[email protected]#[email protected]

-----xcall from s400 ------

s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]
s400#7116#44-Thread-11-bolt-executor[4 4]#[email protected]#[email protected]