55.storm 之 hello word（本地模式）

Posted 2020-09-18 司广孟

strom hello word

概述

然后卡一下代码怎么实现的：

1. 编写数据源类：Spout。可以使用两种方式：
   
   继承BaseRichSpout类
   
   实现IRichSpout接口
   
   主要需要实现或重写几个方法：open、nextTuple、declareOutputFields
   
   
2. 继续编写数据处理类：Bolt。可以使用两种方式：
   
   继承BaseBasicBolt类
   
   实现IRichBolt接口
   
   终点实现或重写几个方法：execute、declareOutputFields
   
   
3. 最后编写主函数（Topology）去进行提交一个任务
   
   在使用Topology的时候，Storm框架为我们提供了两种模式：本地模式和集群模式
   
   本地模式：（无需Storm集群，直接在java中即可运行，一般用于测试和开发阶段）执行main函数即可
   
   集群模式：（需要Storm集群，把实现java程序打包，然后Topology进行提交）需要把应用打成jar，使用Storm命令吧Topology提交到集群中去。

实际操作

先来看一下代码结构：

 

就如上图所说，数据从PWSpout流到PrintBolt，最后到WriteBolt写到文件。具体看一下这几个类的代码：

先看一本地模式的：

PWTopology1.java 拓扑结构构建

import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.topology.TopologyBuilder;
import bhz.bolt.PrintBolt;
import bhz.bolt.WriteBolt;
import bhz.spout.PWSpout;


public class PWTopology1 {

    public static void main(String[] args) throws Exception {
        //
        Config cfg = new Config();
        cfg.setNumWorkers(2);
        cfg.setDebug(true);
        
        
        TopologyBuilder builder = new TopologyBuilder();
        builder.setSpout("spout", new PWSpout());
        builder.setBolt("print-bolt", new PrintBolt()).shuffleGrouping("spout");
        builder.setBolt("write-bolt", new WriteBolt()).shuffleGrouping("print-bolt");
        
        
        //1 本地模式
        LocalCluster cluster = new LocalCluster();
        cluster.submitTopology("top1", cfg, builder.createTopology());
        Thread.sleep(10000);
        cluster.killTopology("top1");
        cluster.shutdown();
        
        //2 集群模式
//        StormSubmitter.submitTopology("top1", cfg, builder.createTopology());
        
    }
}

 

代码分析：

 

数据来源：

import java.util.HashMap;
import java.util.Map;
import java.util.Random;

import backtype.storm.spout.SpoutOutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseRichSpout;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Values;

public class PWSpout extends BaseRichSpout {

    private static final long serialVersionUID = 1L;
    private SpoutOutputCollector collector;
    
    private static final Map<Integer, String> map = new HashMap<Integer, String>();
    
    static {
        map.put(0, "java");
        map.put(1, "php");
        map.put(2, "groovy");
        map.put(3, "python");
        map.put(4, "ruby");
    }
    
    @Override
    public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
        //对spout进行初始化
        this.collector = collector;
        //System.out.println(this.collector);
    }
    
    /**
     * <B>方法名称：</B>轮询tuple<BR>
     * <B>概要说明：</B><BR>
     * @see backtype.storm.spout.ISpout#nextTuple()
     */
    @Override
    public void nextTuple() {
        //随机发送一个单词
        final Random r = new Random();
        int num = r.nextInt(5);
        try {
            Thread.sleep(500);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        this.collector.emit(new Values(map.get(num)));
    }

    /**
     * <B>方法名称：</B>declarer声明发送数据的field<BR>
     * <B>概要说明：</B><BR>
     * @see backtype.storm.topology.IComponent#declareOutputFields(backtype.storm.topology.OutputFieldsDeclarer)
     */
    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        //进行声明
        declarer.declare(new Fields("print"));
    }



}

代码解析：

整体结构

细入分析

---------------------------- open 方法---------------------------------------------------------

---------------------------------  nextTuple方法 --------------------------------------------------------------

---------------------------- declareOutputFields方法 ----------------------------------------------------

数据处理

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;

public class PrintBolt extends BaseBasicBolt {

    private static final Log log = LogFactory.getLog(PrintBolt.class);
    
    private static final long serialVersionUID = 1L;
    
    @Override
    public void execute(Tuple input, BasicOutputCollector collector) {
        //获取上一个组件所声明的Field
        String print = input.getStringByField("print");
        log.info("【print】： " + print);
        //System.out.println("Name of input word is : " + word);
        //进行传递给下一个bolt
        collector.emit(new Values(print));
        
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("write"));
    }

}

 

 

代码分析

 

import java.io.FileWriter;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import clojure.main;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Tuple;

public class WriteBolt extends BaseBasicBolt {

    private static final long serialVersionUID = 1L;

    private static final Log log = LogFactory.getLog(WriteBolt.class);
    
    private FileWriter writer ;
    @Override
    public void execute(Tuple input, BasicOutputCollector collector) {
        //获取上一个组件所声明的Field
        String text = input.getStringByField("write");
        try {
            if(writer == null){
                if(System.getProperty("os.name").equals("Windows 10")){
                    writer = new FileWriter("D:\\\\099_test\\\\" + this);
                } else if(System.getProperty("os.name").equals("Windows 8.1")){
                    writer = new FileWriter("D:\\\\099_test\\\\" + this);
                } else if(System.getProperty("os.name").equals("Windows 7")){
                    writer = new FileWriter("D:\\\\099_test\\\\" + this);
                } else if(System.getProperty("os.name").equals("Linux")){
                    System.out.println("----:" + System.getProperty("os.name"));
                    writer = new FileWriter("/usr/local/temp/" + this);
                }
            }
            log.info("【write】： 写入文件");
            writer.write(text);
            writer.write("\\n");
            writer.flush();
            
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        
    }
    


}

 

和PrintBolt 这个类很相似，都是在处理数据。不作过多解释