awk分析nginx日志中响应时间的方法

Posted 2020-06-27

 

针对响应时间慢的问题，我们在nginx日志格式中增加响应时间，现在需要针对响应时间进行分析，查找出相对较慢的响应时间。

 

1、确认下日志文件格式

 

日志格式：

log_format main ‘$remote_addr - $remote_user [$time_local] "$request" ‘
                ‘$status $body_bytes_sent "$http_referer" ‘
                ‘"$http_user_agent" $http_x_forwarded_for "$request_time"‘;

 

2、输出日志内容：

 

118.187.188.188 - - [19/Jun/2015:09:26:08 +0800] "GET /wap/data/ipad/gsxw/1.json HTTP/1.1" 304 0 "-" "android-async-http/1.4.3 (http://loopj.com/android-async-http)" - "0.287"
61.179.109.132 - - [19/Jun/2015:09:26:08 +0800] "GET /wap/data/ipad/gsxw/1.json HTTP/1.1" 304 0 "-" "android-async-http/1.4.3 (http://loopj.com/android-async-http)" - "0.290"
61.179.107.103 - - [19/Jun/2015:09:26:08 +0800] "GET /wap/data/ipad/gsxw/1.json HTTP/1.1" 304 0 "-" "android-async-http/1.4.3 (http://loopj.com/android-async-http)" - "0.288"
218.92.220.57 - - [19/Jun/2015:09:26:08 +0800] "GET /wap/news/ipad2_gg.html HTTP/1.1" 200 1259 "-" "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B206" - "0.292"
182.140.237.207 - - [19/Jun/2015:09:26:08 +0800] "GET /wap/data/advImg.json?keytime=20150619092610 HTTP/1.1" 200 589 "-" "Dalvik/1.6.0 (Linux; U; Android 4.3; GT-I9308 Build/JSS15J)" - "0.276"

 

3、分析nginx日志

 

　　3.1、打印响应时间并去除引号

 

＃awk ‘{print $NF}‘ news.log | awk -F "\"" ‘{print $2}‘  >  time.txt

time.txt文件内容 

4.822
4.805
0.000
0.000
0.000
4.806
4.801
0.000
4.781
0.000
4.777

 

　　

　　3.2、合并日志文件，增加修改后的响应时间

 

paste  -d " " news.log time.txt > news1.log

合并后news1.log文件内容

 

218.92.220.156 - - [19/Jun/2015:09:58:25 +0800] "GET /wap/news/ipad2_kxcwsj.html HTTP/1.1" 304 0 "-" "Mozilla/5.0 (iPad; CPU OS 8_3 like Mac OS X) AppleWebKit/600.1.
4 (KHTML, like Gecko) Mobile/12F69" - "4.822" 4.822
61.179.107.29 - - [19/Jun/2015:09:58:25 +0800] "GET /wap/news/finance-dividends.html HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Linux; U; Android 4.3; zh-cn; SM-G7108V Buil
d/JLS36C) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30" - "4.805" 4.805
61.179.107.32 - - [19/Jun/2015:09:58:25 +0800] "GET /wap/data/1.json HTTP/1.1" 304 0 "-" "android-async-http/1.4.3 (http://loopj.com/android-async-htt
p)" - "0.000" 0.000
61.179.107.129 - - [19/Jun/2015:09:58:25 +0800] "GET /wap/data/index.json HTTP/1.1" 304 0 "-" "DZHTremereMobile/3.05 CFNetwork/609 Darwin/13.0.0" - "0.000"
0.000
101.254.206.41 - - [19/Jun/2015:09:58:25 +0800] "GET /wap/data/1.json HTTP/1.1" 304 0 "-" "android-async-http/1.4.3 (http://loopj.com/android-async-htt
p)" - "0.000" 0.000
61.179.107.132 - - [19/Jun/2015:09:58:25 +0800] "GET /wap/data/t_index.html?jsonname=rdht/200000441719.json&class=zxts HTTP/1.1" 200 2587 "/wap/news/1.html" "Mozilla/5.0 (iPad; CPU OS 8_2 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Mobile/12D508" - "4.806" 4.806
61.179.107.103 - - [19/Jun/2015:09:58:25 +0800] "GET /wap/data/1.json HTTP/1.1" 304 0 "-" "android-async-http/1.4.3 (http://loopj.com/android-async-htt
p)" - "4.801" 4.801

查询结果文件内容

218.92.220.156 [19/Jun/2015:09:58:25 "GET /wap/news/ipad2_kxcwsj.html 4.822
61.179.107.29 [19/Jun/2015:09:58:25 "GET /wap/news/finance-dividends.html 4.805
61.179.107.32 [19/Jun/2015:09:58:25 "GET /wap/data/t_index.html?jsonname=rdht/200000441719.json&class=zxts 4.806
61.179.107.103 [19/Jun/2015:09:58:25 "GET /wap/data/1.json 4.801
218.92.220.58 [19/Jun/2015:09:58:25 "GET /wap/news/ipad2_kxcwsj.html 4.781
115.231.82.33 [19/Jun/2015:09:58:25 "GET /wap/data/gsxw/1.json 4.777
61.179.107.129 [19/Jun/2015:09:58:25 "GET /wap/news/trader-index.html 4.773
222.88.95.136 [19/Jun/2015:09:58:25 "GET /wap/news/finance-indicators.html 4.771
218.92.220.59 [19/Jun/2015:09:58:25 "GET /wap/data/index.json 4.786
101.254.206.47 [19/Jun/2015:09:58:25 "GET /wap/data/1.json 4.767

4、可以将以上几个命令都加入到一个分析脚本

 

编写 loganalysis.sh 脚本

 1 #!/bin/bash
 2 
 3 if [ $1 ==" " ]
 4 then
 5        echo "Usage: ./loganalysis.sh logname"
 6 exit 0
 7 else
 8        cat $1 | awk ‘{print $NF}‘  | awk -F "\"" ‘{print $2}‘  >  time.txt
 9        echo "split request_time over!!!"
10 
11        paste  -d " " $1 time.txt > new.log
12        echo "build new logfile over!!!"
13 
14        awk ‘($NF>1){print$1" "$4" "$6" "$7" "$NF}‘ new.log > slowtime.txt
15        echo "please see slowtime in slowtime.txt!!!" 
16 
17        rm -f time.txt
18        rm -f new.log
19 fi