Python日志监控系统处理日志（pyinotify）

Posted 2020-09-29

前言

最近项目中遇到一个用于监控日志文件的Python包pyinotify，结合自己的项目经验和网上的一些资料总结一下，总的原理是利用pyinotify模块监控日志文件夹，当日志到来的情况下，触发相应的函数进行处理，处理完毕后删除日志文件的过程，下面就着重介绍下pyinotify

pyinotify

Pyinotify是一个Python模块，用来监测文件系统的变化。 Pyinotify依赖于Linux内核的功能—inotify（内核2.6.13合并）。 inotify的是一个事件驱动的通知器，其通知接口通过三个系统调用从内核空间到用户空间。pyinotify结合这些系统调用，并提供一个顶级的抽象和一个通用的方式来处理这些功能。

• pyinotify 说百了就是通过 调用系统的inotify来实现通知的
• inotify 既可以监视文件，也可以监视目录
• Inotify 使用系统调用而非 SIGIO 来通知文件系统事件。

Inotify 可以监视的文件系统事件包括：

Event Name	Is an Event	Description
IN_ACCESS	Yes	file was accessed.
IN_ATTRIB	Yes	metadata changed.
IN_CLOSE_NOWRITE	Yes	unwrittable file was closed.
IN_CLOSE_WRITE	Yes	writtable file was closed.
IN_CREATE	Yes	file/dir was created in watched directory.
IN_DELETE	Yes	file/dir was deleted in watched directory.
IN_DELETE_SELF	Yes	自删除，即一个可执行文件在执行时删除自己
IN_DONT_FOLLOW	No	don‘t follow a symlink (lk 2.6.15).
IN_IGNORED	Yes	raised on watched item removing. Probably useless for you, prefer instead IN_DELETE*.
IN_ISDIR	No	event occurred against directory. It is always piggybacked to an event. The Event structure automatically provide this information (via .is_dir)
IN_MASK_ADD	No	to update a mask without overwriting the previous value (lk 2.6.14). Useful when updating a watch.
IN_MODIFY	Yes	file was modified.
IN_MOVE_SELF	Yes	自移动，即一个可执行文件在执行时移动自己
IN_MOVED_FROM	Yes	file/dir in a watched dir was moved from X. Can trace the full move of an item when IN_MOVED_TO is available too, in this case if the moved item is itself watched, its path will be updated (see IN_MOVE_SELF).
IN_MOVED_TO	Yes	file/dir was moved to Y in a watched dir (see IN_MOVE_FROM).
IN_ONLYDIR	No	only watch the path if it is a directory (lk 2.6.15). Usable when calling .add_watch.
IN_OPEN	Yes	file was opened.
IN_Q_OVERFLOW	Yes	event queued overflowed. This event doesn‘t belongs to any particular watch.
IN_UNMOUNT	Yes	宿主文件系统被 umount

IN_ACCESS，即文件被访问
IN_MODIFY，文件被write
IN_ATTRIB，文件属性被修改，如chmod、chown、touch等
IN_CLOSE_WRITE，可写文件被close
IN_CLOSE_NOWRITE，不可写文件被close
IN_OPEN，文件被open
IN_MOVED_FROM，文件被移走,如mv
IN_MOVED_TO，文件被移来，如mv、cp
IN_CREATE，创建新文件
IN_DELETE，文件被删除，如rm
IN_DELETE_SELF，自删除，即一个可执行文件在执行时删除自己
IN_MOVE_SELF，自移动，即一个可执行文件在执行时移动自己
IN_UNMOUNT，宿主文件系统被umount
IN_CLOSE，文件被关闭，等同于(IN_CLOSE_WRITE | IN_CLOSE_NOWRITE)
IN_MOVE，文件被移动，等同于(IN_MOVED_FROM | IN_MOVED_TO)

pyinotify使用例子

#!/usr/bin/python
# coding:utf-8

import os
from pyinotify import WatchManager, Notifier,ProcessEvent,IN_DELETE, IN_CREATE,IN_MODIFY
  
class EventHandler(ProcessEvent):
 """事件处理"""
 def process_IN_CREATE(self, event):
  print  "Create file: %s " %  os.path.join(event.path,event.name)
  
 def process_IN_DELETE(self, event):
  print  "Delete file: %s " %  os.path.join(event.path,event.name)
 
 def process_IN_MODIFY(self, event):
   print  "Modify file: %s " %  os.path.join(event.path,event.name)
  
def FSMonitor(path=‘.‘):
  wm = WatchManager() 
  mask = IN_DELETE | IN_CREATE |IN_MODIFY
  notifier = Notifier(wm, EventHandler())
  wm.add_watch(path, mask,auto_add=True,rec=True)
  print ‘now starting monitor %s‘%(path)
  while True:
   try:
     notifier.process_events()
     if notifier.check_events():
       notifier.read_events()
   except KeyboardInterrupt:
     notifier.stop()
     break
  
if __name__ == "__main__":
 FSMonitor(‘/root/softpython/apk_url‘)