HBase链路监控：HTrace+Zipkin

Posted 2023-02-27 悟初境

HTrace是曾经apache的一个孵化项目，在18年孵化失败，停止开发了：https://incubator.apache.org/projects/htrace.html。

但在早期HBase和Hadoop版本中使用了HTrace，可以试试。

HTrace原始日志

在hbase的lib下已经自带了htrace的依赖：

[root@10 hbase-1.2.0]# ll lib/|grep htrace
-rw-rw-r-- 1 root root 1475955 Dec 27 2015 htrace-core-3.1.0-incubating.jar

启用trace功能，加入以下配置到 hbase-site.xml

<property>
    <name>hbase.htrace.sampler</name>
    <value>AlwaysSampler</value>
</property>
<property>
    <name>hbase.trace.spanreceiver.classes</name>
    <value>org.apache.htrace.impl.LocalFileSpanReceiver</value>
</property>
<property>
    <name>hbase.htrace.local-file-span-receiver.path</name>
    <value>/home/service/hbase/hbase-1.2.0/trace-log.out</value>
</property>

现在重新启动hbase，发现多出了 trace-out.log文件，但是没有任何东西。

使用客户端发起几个读请求，也没有日志输出。

原因是，trace需要客户端手动发起，下面演示其过程。

客户端引入htrace-core依赖：

<dependency>
    <groupId>org.apache.htrace</groupId>
    <artifactId>htrace-core</artifactId>
    <version>3.1.0-incubating</version>
</dependency>

修改读取代码如下：

public interface Callback 
    void execute(Table table) throws IOException, InterruptedException;

 
public static void execute(Callback callback) 
    Configuration c = HBaseConfiguration.create();
    c.set("hbase.zookeeper.quorum", "10.10.11.12");
    try (Connection connection = ConnectionFactory.createConnection(c)) 
        Table table = connection.getTable(TableName.valueOf(TABLE));
        callback.execute(table);
     catch (IOException | InterruptedException e) 
        e.printStackTrace();
    


@Test
public void traceRead() 
    TraceScope ts = Trace.startSpan("Get By RowKey", Sampler.ALWAYS);
    Random r = new Random();
    HBaseHelper.execute(table -> 
        Result res = table.get(new Get(String.format("%08d", 10000000 + r.nextInt(1000)).getBytes(StandardCharsets.UTF_8)));
        System.out.println("查到：" + res.toString());
    );
    ts.close();

运行traceRead，可以看到trace-log.out输出日志：

"i":"bbbad101a4ebe7cf","s":"9e0c47b7b700ead7","b":1655709104104,"e":1655709104104,"d":"HFileReaderV2.readBlock","r":"HRegionServer","p":["96e40556d27a742e"],"t":["t":1655709104104,"m":"blockCacheHit"]
"i":"bbbad101a4ebe7cf","s":"bae512e53e53b184","b":1655709104105,"e":1655709104105,"d":"HFileReaderV2.readBlock","r":"HRegionServer","p":["96e40556d27a742e"],"t":["t":1655709104105,"m":"blockCacheHit"]
"i":"bbbad101a4ebe7cf","s":"1e975395ee0fa49f","b":1655709104105,"e":1655709104105,"d":"HFileReaderV2.readBlock","r":"HRegionServer","p":["96e40556d27a742e"],"t":["t":1655709104105,"m":"blockCacheHit"]
"i":"bbbad101a4ebe7cf","s":"96e40556d27a742e","b":1655709104104,"e":1655709104106,"d":"ClientService.Scan","r":"HRegionServer","p":["3d9c2ea62c240122"],"t":["t":1655709104104,"m":"Creating MemStoreScanner"]
"i":"bbbad101a4ebe7cf","s":"c726cb50a801a177","b":1655709104136,"e":1655709104136,"d":"HFileReaderV2.readBlock","r":"HRegionServer","p":["8a1b6f5b206d5ac2"],"t":["t":1655709104136,"m":"blockCacheHit"]
"i":"bbbad101a4ebe7cf","s":"7ad3475cc637f1c2","b":1655709104136,"e":1655709104136,"d":"HFileReaderV2.readBlock","r":"HRegionServer","p":["8a1b6f5b206d5ac2"],"t":["t":1655709104136,"m":"blockCacheHit"]
"i":"bbbad101a4ebe7cf","s":"8a1b6f5b206d5ac2","b":1655709104135,"e":1655709104137,"d":"ClientService.Get","r":"HRegionServer","p":["62ab66b22e0eb820"],"t":["t":1655709104136,"m":"Creating MemStoreScanner"]

日志含义解释(见代码SpanSerializer里)：

• i：Trace ID
• s：Span ID
• b：开始时间
• e：结束时间
• d：描述信息
• r：process ID，节点或进程的唯一ID，比如可以用IP地址
• p：父Span ID，数组格式
• t：和这个span关联的注解信息，KV数组

ZipKin可视化链路

上面确实构成了完整调用链，但是并不直观。这时候就可以使用ZipKin来展示链路了。

下载zipkin-server：https://search.maven.org/remotecontent?filepath=io/zipkin/zipkin-server/2.14.1/zipkin-server-2.14.1-exec.jar

然后启动服务，开启zipkin-collector的RPC端口9410

SCRIBE_ENABLED=true nohup java -jar zipkin-server-2.14.1-exec.jar &

然后访问页面：http://10.10.11.12:9411/zipkin/

接下来是HBase端

默认没有zipkin的依赖，需要下载htrace-zipkin的jar包：https://repo1.maven.org/maven2/org/apache/htrace/htrace-zipkin/3.1.0-incubating/htrace-zipkin-3.1.0-incubating.jar，然后放到lib目录下。

接着修改hbase-site.xml配置：

<property>
  <name>hbase.trace.spanreceiver.classes</name>
  <value>org.apache.htrace.impl.ZipkinSpanReceiver</value>
</property>
<property>
  <name>hbase.htrace.zipkin.collector-hostname</name>
  <value>localhost</value>
</property>
<property>
  <name>hbase.htrace.zipkin.collector-port</name>
  <value>9410</value>
</property>

然后重启HBase。

再使用上面的代码请求下，可以从zipkin上面看到新增了一个服务：hregionserver。

总共6个Span。

如果是scan的代码

@Test
public void traceScan() 
    TraceScope ts = Trace.startSpan("Scan Trace", Sampler.ALWAYS);
    HBaseHelper.execute(table -> 
        byte[] start = "00000000".getBytes(StandardCharsets.UTF_8);
        byte[] end = "00010000".getBytes(StandardCharsets.UTF_8);
        ResultScanner scanner = table.getScanner(new Scan(start, end));
        int cnt = 0;
        for (Result r : scanner) 
            cnt++;
        
        System.out.println(cnt);
    );
    ts.close();

那么能得到下图调用链：

上面只有服务端的调用链，我们其实想看完整的链路，那么客户端也需要接入Zipkin。

客户端引入htrace-zipkin依赖：

<dependency>
    <groupId>org.apache.htrace</groupId>
    <artifactId>htrace-zipkin</artifactId>
    <version>3.1.0-incubating</version>
</dependency>
<dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>12.0.1</version>
</dependency>

然后修改测试代码，使用ZipkinSpanReceiver

@Test
public void traceToZipkin() 
    ImmutableMap<String, String> conf = ImmutableMap.of(
            "zipkin.collector-hostname", "10.10.11.12",
            "zipkin.collector-port", "9410"
    );
    Trace.addReceiver(new ZipkinSpanReceiver(HTraceConfiguration.fromMap(conf)));
    try (TraceScope ts = Trace.startSpan("Get By RowKey", Sampler.ALWAYS)) 
        Random r = new Random();
        HBaseHelper.execute(table -> 
            Result  res = table.get(new Get(String.format("%08d", 10000000 + r.nextInt(1000)).getBytes(StandardCharsets.UTF_8)));
            System.out.println("查到：" + res.toString());
        );
    

再次运行之后，可以发现完整链路:这里客户端是junitstarter, 有相应的客户端方法被拦截

链路