python 多进程下的日志打印

Posted 2020-12-12 youxiaogeo

python 使用多线程效果提升不高，因为只会占用一个cpu核，为了提升，需要使用多进程（比如 uwsgi 做服务）

这时候，在日志更新时就会出问题，通常日志更新是按天的，这时候多个进程就会重新去更新，比如：

进程a切换到 log.2020.06.22,进程b在切换的时候发现这个文件存在了就会去删除，导致旧的数据被删除。解决方式，

修改

logging.handlers.TimedRotatingFileHandler

的类，使得进程b在切换的时候发现这个文件存在了（说明其他进程创建了）就不删除旧的，同时新建的时候，文件writemode 采用 a+ 而不是w（用w会删除已经有的文件，导致进程a打的日志丢失），

参考：https://www.jianshu.com/p/d615bf01e37b

创建新类继承  TimedRotatingFileHandler

class SafeRotatingFileHandler(TimedRotatingFileHandler):
    def __init__(self, filename, when=‘h‘, interval=1, backupCount=0, encoding=None, delay=False, utc=False):
        TimedRotatingFileHandler.__init__(self, filename, when, interval, backupCount, encoding, delay, utc)
    """
    Override doRollover
    lines commanded by "##" is changed by cc
    """
    def doRollover(self):
        """
        do a rollover; in this case, a date/time stamp is appended to the filename
        when the rollover happens.  However, you want the file to be named for the
        start of the interval, not the current time.  If there is a backup count,
        then we have to get a list of matching filenames, sort them and remove
        the one with the oldest suffix.
        
        Override,   1. if dfn not exist then do rename
                    2. _open with "a" model
        """
        if self.stream:
            self.stream.close()
            self.stream = None
        # get the time that this sequence started at and make it a TimeTuple
        currentTime = int(time.time())
        dstNow = time.localtime(currentTime)[-1]
        t = self.rolloverAt - self.interval
        if self.utc:
            timeTuple = time.gmtime(t)
        else:
            timeTuple = time.localtime(t)
            dstThen = timeTuple[-1]
            if dstNow != dstThen:
                if dstNow:
                    addend = 3600
                else:
                    addend = -3600
                timeTuple = time.localtime(t + addend)
        dfn = self.baseFilename + "." + time.strftime(self.suffix, timeTuple)
##        if os.path.exists(dfn):
##            os.remove(dfn)
            
        # Issue 18940: A file may not have been created if delay is True.
##        if os.path.exists(self.baseFilename):
        if not os.path.exists(dfn) and os.path.exists(self.baseFilename):
            os.rename(self.baseFilename, dfn)
        if self.backupCount > 0:
            for s in self.getFilesToDelete():
                os.remove(s)
        if not self.delay:
            self.mode = "a"
            self.stream = self._open()
        newRolloverAt = self.computeRollover(currentTime)
        while newRolloverAt <= currentTime:
            newRolloverAt = newRolloverAt + self.interval
        #If DST changes and midnight or weekly rollover, adjust for this.
        if (self.when == ‘MIDNIGHT‘ or self.when.startswith(‘W‘)) and not self.utc:
            dstAtRollover = time.localtime(newRolloverAt)[-1]
            if dstNow != dstAtRollover:
                if not dstNow:  # DST kicks in before next rollover, so we need to deduct an hour
                    addend = -3600
                else:           # DST bows out before next rollover, so we need to add an hour
                    addend = 3600
                newRolloverAt += addend
        self.rolloverAt = newRolloverAt