[centos7][nginx][tshark] 基于 tshark 的页面流量统计

Posted TesterHome

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了[centos7][nginx][tshark] 基于 tshark 的页面流量统计相关的知识，希望对你有一定的参考价值。

缘由

因为收到反馈说我们的app页面刷新慢，初步分析呢是因为运营的小伙伴上传了较大的图片导致的，然后就引发了一堆优化：

  • ERP上传限制 ==> 开发

  • 文件服务器针对大图片自动压缩 ==> 运维

  • app流量统计 ==> 测试

怎么做

既然问题来了，那么怎么做呢？
以前做过app性能获取的工具，但是不理想，兼容性差：

  • android 5.0的手机上没问题的方法可能到了6.0的版本不适用

  • 小米手机上成功的，可能在华为手机上不行

开启一个代理服务器，通过抓包过滤出被测设备产生的流量，进行统计相对靠谱的多

如何设置一个代理服务器

windows, linux, mac都可以设置代理服务器，我选择的是centos，
由于公司的服务目前是基于http协议的，所以需要搭建一个http的代理服务，我选择了nginx

安装nginx

yum install nginx

修改nginx配置

# 访问nginx安装目录cd /etc/nginx# 保存一份默认的配置文件cp nginx.conf nginx.origin.conf# 编辑配置文件，并增加http正向代理vi nginx.conf# 在server下面增加resolver，在location中增加proxy_pass与proxy_set_header
    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        resolver 8.8.8.8;
        resolver_timeout 5s;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
          proxy_pass $scheme://$host$request_uri;
          proxy_set_header Host $http_host;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

修改好配置文件后，可以测试一下，并启动nginx

# 测试nginx -t# 启动nginx# 重启启动nginxnginx -s reload# 查看nginx访问日志，这个文件位置是在nginx.conf里面配的tail -f /var/log/nginx/access.log

代理配置

这里我在win7的机器上打开浏览器，修改代理设置：centos机器ip，端口是上面配的80端口

修改代理后，成功访问163.com
注意了，我只代理了http协议，https不能正常访问是正常的

抓包工具的选择

wireshark
tcpdump
我希望最好是一个命令行工具，而不是GUI操作的工具，
tcpdump和tshark大部分场景基本一致，最后选择了wireshark自带的tshark

安装wireshark

yum install wireshark
yum install wireshark-gnome

查看帮助

tshark -hTShark 1.10.14 (Git Rev Unknown from unknown)Dump and analyze network traffic.
See http://www.wireshark.org for more information.

Copyright 1998-2015 Gerald Combs <gerald@wireshark.org> and contributors.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Usage: tshark [options] ...

Capture interface:  -i <interface>           name or idx of interface (def: first non-loopback)
  -f <capture filter>      packet filter in libpcap filter syntax  -s <snaplen>             packet snapshot length (def: 262144)
  -p                       don't capture in promiscuous mode  -I                       capture in monitor mode, if available  -B <buffer size>         size of kernel buffer (def: 4MB)  -y <link type>           link layer type (def: first appropriate)  -D                       print list of interfaces and exit  -L                       print list of link-layer types of iface and exit Capture stop conditions:  -c <packet count>        stop after n packets (def: infinite)  -a <autostop cond.> ...  duration:NUM - stop after NUM seconds                           filesize:NUM - stop this file after NUM KB                              files:NUM - stop after NUM files Capture output:  -b <ringbuffer opt.> ... duration:NUM - switch to next file after NUM secs                           filesize:NUM - switch to next file after NUM KB                              files:NUM - ringbuffer: replace after NUM files Input file:  -r <infile>              set the filename to read from (no stdin!) Processing:  -2                       perform a two-pass analysis  -R <read filter>         packet Read filter in Wireshark display filter syntax  -Y <display filter>      packet displaY filter in Wireshark display filter syntax  -n                       disable all name resolutions (def: all enabled)  -N <name resolve flags>  enable specific name resolution(s): "mnNtC"  -d <layer_type>==<selector>,<decode_as_protocol> ...                           "Decode As", see the man page for details                           Example: tcp.port==8888,http  -H <hosts file>          read a list of entries from a hosts file, which will                           then be written to a capture file. (Implies -W n) Output:  -w <outfile|->           write packets to a pcap-format file named "outfile"                           (or to the standard output for "-")  -C <config profile>      start with specified configuration profile  -F <output file type>    set the output file type, default is pcapng                           an empty "-F" option will list the file types  -V                       add output of packet tree        (Packet Details)  -O <protocols>           Only show packet details of these protocols, comma                           separated  -P                       print packet summary even when writing to a file  -S <separator>           the line separator to print between packets  -x                       add output of hex and ASCII dump (Packet Bytes)  -T pdml|ps|psml|text|fields                           format of text output (def: text)  -e <field>               field to print if -Tfields selected (e.g. tcp.port, col.Info);                           this option can be repeated to print multiple fields  -E<fieldsoption>=<value> set options for output when -Tfields selected:     header=y|n            switch headers on and off     separator=/t|/s|<char> select tab, space, printable character as separator     occurrence=f|l|a      print first, last or all occurrences of each field     aggregator=,|/s|<char> select comma, space, printable character as                           aggregator     quote=d|s|n           select double, single, no quotes for values  -t a|ad|d|dd|e|r|u|ud    output format of time stamps (def: r: rel. to first)  -u s|hms                 output format of seconds (def: s: seconds)  -l                       flush standard output after each packet  -q                       be more quiet on stdout (e.g. when using statistics)  -Q                       only log true errors to stderr (quieter than -q)  -g                       enable group read access on the output file(s)  -W n                     Save extra information in the file, if supported.                           n = write network address resolution information  -X <key>:<value>         eXtension options, see the man page for details  -z <statistics>          various statistics, see the man page for details Miscellaneous:  -h                       display this help and exit  -v                       display version info and exit  -o <name>:<value> ...    override preference setting  -K <keytab>              keytab file to use for kerberos decryption  -G [report]              dump one of several available reports and exit                           default report="fields"                           use "-G ?" for more help

查看IP地址

ifconfig

查询本机支持抓包的硬设

tshark -D

通过网卡

tshark -i 1
tshark -i enp2s0

抓包并输出到文件

tshark -i enp2s0 -w packets.pcap
tshark -i enp2s0 -w packets.pcap -c10  #只抓10个，抓满10即自动停止

查看packets.pcap文件中的内容

tshark -r packets.pcap
tshark -r packets.pcap -c5 #读取前5个

通过-V参数查看具体信息

tshark -r p.pcap -Vtshark -r p.pcap -V -c1  #结合之前的-c，可以查看第一个包的具体内容

根据mac地址，ip地址，端口等条件抓包

The following values are available when using this option:
m MAC address resolution
n Network address resolution
t Transport layer (port name) resolution
N Use external resolvers
C Concurrent DNS lookups
tshark -i enp2s0 -n

抓包过滤：-f

tshark -ni 5 -w pack.pcap -f "tcp port 9980"

显示过滤：-Y

tshark -r pack.pcap -Y "tcp.dstport == 9980"

显示时间格式

tshark -r p.pcap -t ad

查看统计帮助

tshark -z help
tshark -z helptshark: The available statistics for the "-z" option are:
     afp,srt
     ancp,tree
     ansi_a,
     bacapp_instanceid,tree
     bacapp_ip,tree
     bacapp_objectid,tree
     bacapp_service,tree
     bootp,stat,
     camel,counter
     camel,srt
     collectd,tree
     compare,
     conv,
     dcerpc,srt,
     dests,tree
     diameter,avp
     expert
     follow,ssl
     follow,tcp
     follow,udp
     gsm_a,
     h225,counter
     h225,srt
     hart_ip,tree
     hosts
     http,stat,
     http,tree
     http_req,tree
     http_srv,tree
     icmp,srt
     icmpv6,srt
     io,phs
     io,stat,
     ip_hosts,tree
     isup_msg,tree
     mac-lte,stat
     megaco,rtd
     mgcp,rtd
     plen,tree
     proto,colinfo,
     ptype,tree
     radius,rtd
     rlc-lte,stat
     rpc,programs
     rpc,srt,
     rtp,streams
     rtsp,stat,
     rtsp,tree
     sametime,tree
     scsi,srt,
     sctp,stat
     sip,stat
     smb,sids
     smb,srt
     smpp_commands,tree
     sv
     ucp_messages,tree
     wsp,stat,

http,tree Displays statistics related to HTTP requests and responses

流量统计

tshark -r pack.pcap -qz conv,tcp================================================================================TCP Conversations
Filter:<No Filter>
                                               |       <-      | |       ->      | |     Total     |    Relative    |   Duration   |
                                               | Frames  Bytes | | Frames  Bytes | | Frames  Bytes |      Start     |              |
192.168.1.189:57399   <-> 192.168.1.171:9980      1332   2008853     341     22849    1673   2031702     0.196315903         0.4847
192.168.1.189:57400   <-> 192.168.1.171:9980       376    565847     126      8770     502    574617     0.196795638         0.1422
192.168.1.189:57397   <-> 192.168.1.171:9980       231    340320      78      7273     309    347593     0.000000000         0.4854
192.168.1.189:57402   <-> 192.168.1.171:9980        86    123830      37      3437     123    127267     0.197292575         0.2728
192.168.1.189:57403   <-> 192.168.1.171:9980        59     85650      25      2711      84     88361     0.197415689         0.0599
192.168.1.189:57401   <-> 192.168.1.171:9980        50     71782      24      2660      74     74442     0.197162662         0.0567================================================================================

上面按tcp协议统计，将Total下面的Bytes那一列求和，可得总计使用的流量信息
下面按ip统计

tshark -r pack.pcap -qz conv,ip================================================================================IPv4 Conversations
Filter:<No Filter>
                                               |       <-      | |       ->      | |     Total     |    Relative    |   Duration   |
                                               | Frames  Bytes | | Frames  Bytes | | Frames  Bytes |      Start     |              |
192.168.1.171        <-> 192.168.1.189             631     47700    2134   3196282    2765   3243982     0.000000000         0.6811================================================================================

更多玩法

过滤sql语句

tshark -i 5 -n -f 'tcp dst port 3306' -T fields -e mysql.query
Running as user "root" and group "root". This could be dangerous.
Capturing on 'enp2s0'SELECT project_name as projectName FROM tb_project

根据mysql 默认的3306端口过滤sql语句

根据被测机器ip以及目标域名进行过滤

tshark -i enp2s0 -t ad -w 2018.pcap -f "ip src host 被测机器ip(例：192.168.1.189)"tshark -r 2018.pcap -R 'http.host==目标域名(例：www.xxx.com)' -qz conv,ip

拓展

  • 将本功能放到服务端，可在线配置，获取结果

  • 数据存库

    • 同品对比

    • 竞品对比

    • 可视化展示

请教

大家是怎么做的？

Reference


以上是关于[centos7][nginx][tshark] 基于 tshark 的页面流量统计的主要内容，如果未能解决你的问题，请参考以下文章

Tshark-如何仅使用tshark显示tcp播放负载的解剖树？

tshark使用说明

在 tshark 中定义 IP 范围

在linux上tshark怎么过滤

tshark的抓包和解析

tcpdump VS tshark用法(转)