python代码异常时重新执行函数

Posted 2021-10-13 sysu_lluozh

有小伙伴在使用接口平台时反馈测试计划在执行时会概率性出现部分用例未执行的情况

一、初步定位

在定位问题时发现偶发性出现如下异常

Process CaseProcess-9:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/Users/lluozh/work/git/swapi/httpRequest/controller/planProcess/caseProcess.py", line 43, in run
    self.execute_one_case(in_case_list)
  File "/Users/lluozh/work/git/swapi/httpRequest/controller/planProcess/caseProcess.py", line 55, in execute_one_case
    _interface = Interface(in_id)
  File "/Users/lluozh/work/git/swapi/httpRequest/serializers/interface.py", line 20, in __init__
    self.setup = self.get_interface_setup()
  File "/Users/lluozh/work/git/swapi/httpRequest/serializers/interface.py", line 33, in get_interface_setup
    return self.interface_info.get('in_setup')
AttributeError: 'NoneType' object has no attribute 'get'

即interface_info为None导致，然而interface_info是通过数据库查询直接获取，在多次试验后发现该问题出现的原因是在执行用例从数据库获取数据时，数据库执行查询操作概率性出现异常导致

二、问题分析

那数据库执行的异常的原因是？

数据库执行查询的逻辑

@classmethod
def query(cls, sql, param):
    try:
        conn, cursor = cls.open()
        cursor.execute(sql, param)
        result = cursor.fetchall()
        cls.close(conn, cursor)
        return result
    except Exception as e:
        traceback.print_exc(e)

@classmethod
def close(cls, conn, cursor):
    try:
        conn.commit()
        cursor.close()
        conn.close()
    except Exception as e:
        traceback.print_exc()

在执行close函数时抛出异常

Traceback (most recent call last):
  File "/Users/lluozh/work/git/swapi/util/DBTool/sqlUtil.py", line 35, in close
    conn.commit()
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/DBUtils/SteadyDB.py", line 423, in commit
    raise error  # reraise the original error
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/DBUtils/SteadyDB.py", line 414, in commit
    self._con.commit()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymysql/connections.py", line 470, in commit
    self._read_ok_packet()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymysql/connections.py", line 443, in _read_ok_packet
    pkt = self._read_packet()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymysql/connections.py", line 692, in _read_packet
    packet_header = self._read_bytes(4)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymysql/connections.py", line 749, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query"
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

可以发现概率性出现close操作将数据库连接放回连接池时异常导致直接返回null

三、问题解决

3.1 手动捕获异常迭代执行

 @classmethod
 def query(cls, sql, param, reTry=0):
     try:
         conn, cursor = cls.open()
         cursor.execute(sql, param)
         result = cursor.fetchall()
         cls.close(conn, cursor)
         return result
     except (OperationalError, InternalError) as e:
         if reTry <= 2:
             reTry += 1
             logger.info(str(conn) + "Try to reconnect Database query {} times!".format(reConn))
             cls.query_one(sql, param, reTry=reTry)
     except Exception as e:
         if reTry <= 2:
             reTry += 1
             logger.info("sqlError：Try to redo!"+ str(sql) + str(param))
             cls.query_one(sql, param, reTry=reTry)

但是出现异常重新执行但同样并未返回结果，需要将数据return，而且这样写的代码过于臃肿

3.2 retrying实现重试机制

retrying是一个python的重试包，可以用来自动重试一些可能运行失败的程序段

@classmethod
def query(cls, sql, param):
    try:
        return cls._query(sql, param)
    except Exception as e:
        logger.info("sqlError：" + str(sql) + str(param))
        traceback.print_exc(e)
        abort(417,"数据库执行异常")


@classmethod
@retry(stop_max_attempt_number=3)
def _query(cls, sql, param):
    conn, cursor = cls.open()
    cursor.execute(sql, param)
    result = cursor.fetchall()
    cls.close(conn, cursor)
    return result

四、retrying

4.2 Retrying类定义

查看Retrying类定义

class Retrying(object):

    def __init__(self,
                 stop=None, wait=None,
                 stop_max_attempt_number=None,
                 stop_max_delay=None,
                 wait_fixed=None,
                 wait_random_min=None, wait_random_max=None,
                 wait_incrementing_start=None, wait_incrementing_increment=None,
                 wait_exponential_multiplier=None, wait_exponential_max=None,
                 retry_on_exception=None,
                 retry_on_result=None,
                 wrap_exception=False,
                 stop_func=None,
                 wait_func=None,
                 wait_jitter_max=None):

        self._stop_max_attempt_number = 5 if stop_max_attempt_number is None else stop_max_attempt_number
        self._stop_max_delay = 100 if stop_max_delay is None else stop_max_delay
        self._wait_fixed = 1000 if wait_fixed is None else wait_fixed
        self._wait_random_min = 0 if wait_random_min is None else wait_random_min
        self._wait_random_max = 1000 if wait_random_max is None else wait_random_max
        self._wait_incrementing_start = 0 if wait_incrementing_start is None else wait_incrementing_start
        self._wait_incrementing_increment = 100 if wait_incrementing_increment is None else wait_incrementing_increment
        self._wait_exponential_multiplier = 1 if wait_exponential_multiplier is None else wait_exponential_multiplier
        self._wait_exponential_max = MAX_WAIT if wait_exponential_max is None else wait_exponential_max
        self._wait_jitter_max = 0 if wait_jitter_max is None else wait_jitter_max

4.2 参数

从类定义中可以retry可以接受一些参数，这个从源码中Retrying类的初始化函数可以看到可选的参数：

• stop_max_attempt_number：用来设定最大的尝试次数，超过该次数就停止重试
• stop_max_delay：比如设置成10000，那么从被装饰的函数开始执行的时间点开始，到函数成功运行结束或者失败报错中止的时间点，只要这段时间超过10秒，函数就不会再执行了
• wait_fixed：设置在两次retrying之间的停留时间
• wait_random_min和wait_random_max：用随机的方式产生两次retrying之间的停留时间
• wait_exponential_multiplier和wait_exponential_max：以指数的形式产生两次retrying之间的停留时间，产生的值为2^previous_attempt_number * wait_exponential_multiplier，previous_attempt_number是前面已经retry的次数，如果产生的这个值超过了wait_exponential_max的大小，那么之后两个retrying之间的停留值都为wait_exponential_max。这个设计可以减轻阻塞的情况

4.3 使用示例

4.3.1 指定异常重试

可以指定要在出现哪些异常的时候再去retry，这个要用retry_on_exception传入一个函数对象：

def retry_if_io_error(exception):
 return isinstance(exception, IOError)
  
@retry(retry_on_exception=retry_if_io_error)
def read_a_file():
 with open("file", "r") as f:
  return f.read()

在执行read_a_file函数的过程中，如果报出异常，那么这个异常会以形参exception传入retry_if_io_error函数中，如果exception是IOError那么就进行retry，如果不是就停止运行并抛出异常

4.3.2 指定结果重试

还可以指定要在得到哪些结果的时候去retry，这个要用retry_on_result传入一个函数对象：

def retry_if_result_none(result):
    print("******************2")
    return result is None


@retry(retry_on_result=retry_if_result_none, stop_max_attempt_number=3)
def get_result():
    print("******************1")
    return None

get_result()

执行结果

******************1
******************2
******************1
******************2
******************1  
******************2
Traceback (most recent call last):
	File "/Users/lluozh/work/git/swapi/xyz/tt.py", line 50, in <module>
		get_result()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f
    	return Retrying(*dargs, **dkw).call(f, *args, **kw)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/retrying.py", line 214, in call
    	raise RetryError(attempt)
retrying.RetryError: RetryError[Attempts: 3, Value: None]

在执行get_result成功后，会将函数的返回值通过形参result的形式传入retry_if_result_none函数中，如果返回值是None那么就进行retry，否则就结束并返回函数值

五、重试装饰器

5.1 retry实现源码

查看retry的源码

def retry(*dargs, **dkw):
    """
    Decorator function that instantiates the Retrying object
    @param *dargs: positional arguments passed to Retrying object
    @param **dkw: keyword arguments passed to the Retrying object
    """
    # support both @retry and @retry() as valid syntax
    if len(dargs) == 1 and callable(dargs[0]):
        def wrap_simple(f):

            @six.wraps(f)
            def wrapped_f(*args, **kw):
                return Retrying().call(f, *args, **kw)

            return wrapped_f

        return wrap_simple(dargs[0])

    else:
        def wrap(f):

            @six.wraps(f)
            def wrapped_f(*args, **kw):
                return Retrying(*dargs, **dkw).call(f, *args, **kw)

            return wrapped_f

        return wrap

retrying提供一个装饰器函数retry，被装饰的函数就会在运行失败的条件下重新执行，默认只要一直报错就会不断重试

5.2 重试装饰器

根据上面retry装饰器函数，其实可以自定义重试装饰器

5.2.1 重试装饰器demo

可以自己实现失败重试装饰器的demo

def retry(loop_num=2):
    def wrapper(func):
        def _wrapper(*args, **kwargs):
            raise_ex = 0
            for _ in range(loop_num):
                print(f"第{_}次操作")
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    raise_ex += 1
                    if raise_ex == loop_num:
                        raise e

        return _wrapper

    return wrapper


@retry(loop_num=3)
def demo():
    print('a')
    print(a)
    # raise NameError

demo()

执行结果

第0次操作
a
第1次操作
a
第2次操作
a
Traceback (most recent call last):
  File "/Users/lluozh/work/git/swapi/xyz/tt.py", line 55, in <module>
    demo()
  File "/Users/lluozh/work/git/swapi/xyz/tt.py", line 42, in _wrapper
    raise e
  File "/Users/lluozh/work/git/swapi/xyz/tt.py", line 38, in _wrapper
    return func(*args, **kwargs)
  File "/Users/lluozh/work/git/swapi/xyz/tt.py", line 52, in demo
    print(a)
NameError: name 'a' is not defined

5.2.2 增加等待时间定制逻辑

可以在自定义的装饰器函数中增加等待时间等根据业务需求的定制逻辑

import time

def retry(loop_num=2, wait_time=1):
    """
    :param loop_num: 循环次数，默认2次
    :param wait_time: 等待时间，默认1s
    :return:
    """
    def wrapper(func):
        def _wrapper(*args, **kwargs):
            raise_ex = 0
            for i in range(1, loop_num + 1):
                print(f"第{i}次操作")
                try:
                    func(*args, **kwargs)
                except Exception as e:
                    time.sleep(wait_time)
                    raise_ex += 1
                    if raise_ex == loop_num:
                        raise e

        return _wrapper

    return wrapper


@retry()
def demo_():
    print('开始重试')
    print(a)


if __name__ == '__main__':
    demo_()

执行结果

第1次操作
开始重试
第2次操作
开始重试
Traceback (most recent call last):
  File "/Users/lluozh/work/git/swapi/xyz/tt.py", line 65, in <module>
    demo_()
  File "/Users/lluozh/work/git/swapi/xyz/tt.py", line 51, in _wrapper
    raise e
  File "/Users/lluozh/work/git/swapi/xyz/tt.py", line 46, in _wrapper
    func(*args, **kwargs)
  File "/Users/lluozh/work/git/swapi/xyz/tt.py", line 61, in demo_
    print(a)
NameError: name 'a' is not defined