Python小程序

Posted 正在努力的BOY

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Python小程序相关的知识,希望对你有一定的参考价值。

需求:统计每分钟有多少行;及23:59有多少行,00:00有多少行,文件过大每个文件2G,共5个文件

文件:access_log.2017-11-06-00_00_00文件内容如下:

21.123.42.233 - - [05/Nov/2017:23:59:59 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 11637 1388               
21.123.42.233 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 7223 150                  
21.123.42.235 - - [05/Nov/2017:23:59:59 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 185121 135                           
21.123.42.235 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 7998 330                        
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 29833 1388               
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 20608 150                 
21.123.42.232 - - [05/Nov/2017:23:59:59 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 252234 139                           
21.123.42.232 - - [05/Nov/2017:23:59:59 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 230661 135                           
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 33536 1388               
21.123.42.233 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 138429 139                           
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 24368 150                 
21.123.42.233 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 7075 330                        
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 23221 330                       
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 19719 330                       
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 85528 135                            
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 22324 1388               
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 15338 150                 
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 20461 1388               
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 24474 330                       
21.123.42.235 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 10589 1388               
21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 24163 150                 
21.123.42.235 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 4449 150                  
21.123.42.236 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 345698 139                           
21.123.42.236 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 19886 330                       
21.123.42.233 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 5096 150                     
21.123.42.232 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 5755 150                     
21.123.42.235 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 7182 330                           
21.123.42.235 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 4752 150                     
21.123.42.233 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 101802 139                              
21.123.42.232 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 7666 330                           
21.123.42.236 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 11735 1388                  
21.123.42.232 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 5376 150                     
21.123.42.234 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 19922 1388                  
21.123.42.236 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 8779 1388                   
21.123.42.235 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 99890 135                               
21.123.42.234 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 5242 150                     
21.123.42.236 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 16658 1388                  
21.123.42.233 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 4977 150      
21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 5559 3
21.123.42.232 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 5176 3
21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME:
21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME:
21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME:
21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 
21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 
21.123.42.235 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 136333 135 
21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 8429 3
21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME:
21.123.42.236 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 147556 135 
21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 
21.123.42.235 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 119417 135 
21.123.42.233 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 194970 135 
21.123.42.235 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 76006 135  
21.123.42.236 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 5892 3
21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 4904 3
21.123.42.235 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35963Invoker HTTP/1.1" 200 TIME: 297515 369 
21.123.42.236 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 402771 135 
21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME:
21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 135743 135 
21.123.42.235 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 5853 3
21.123.42.235 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 14536 
21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 
21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME:
21.123.42.233 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 186202 135  

Python代码如下所示:

#!/usr/bin/env python
# -*- coding:utf-8 -*-
__author__ = 40kuai
li = [0 for i in range(1440)]
with open(/root/access_log.2017-11-06-00_00_00) as file:
    for line in file:
        hour = line.split(:)[1]
        mins = line.split(:)[2]
        haha =  int(hour)*60+int(mins)
        li[haha] +=1


for index,i in enumerate(li):
    try:
        a,b=divmod(index,60)
        print %s:%s % (a, b), i
    except ZeroDivisionError as e:
        print 00:00, i

又用Shell编了一个简单Shell脚本,脚本如下:

#!/bin/bash
for i in {00..23} ;do
  for j in {00..59} ;do 
    echo $i:$j  `awk ‘{print $4}‘ /root/access_log.2017-11-06-00_00_00|awk -F ‘:‘ ‘{print $2$3}‘|grep $i$j|wc -l`>>/lines.txt
  done
done

两个脚本的效率差别很大。。。哪位大神可以帮忙看看能不能优化一下Shell脚本,Shell主要问题为循环读文件进行过滤

 

以上是关于Python小程序的主要内容,如果未能解决你的问题,请参考以下文章

小程序各种功能代码片段整理---持续更新

Android获取各个应用程序的缓存文件代码小片段(使用AIDL)

提效小技巧——记录那些不常用的代码片段

python+spark程序代码片段

Android小部件,启动一个片段?

python小知识片段