当其中一个线程(chrome)异常时,Uwsgi挂起(10-60秒)
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了当其中一个线程(chrome)异常时,Uwsgi挂起(10-60秒)相关的知识,希望对你有一定的参考价值。
我有一台服务器,可以为大约100个客户服务。我使用sqlite本地数据库来存储和检索数据。
码:
class myThread (threading.Thread):
def __init__(self, threadID, name, q, q_imme):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.q = q
self.q_imme = q_imme
def run(self):
print ("Starting " + self.name)
worker(self.name, self.q, self.q_imme)
print ("Exiting " + self.name)
def worker(name, q, q_imme):
try:
# Do some chrome-selenium-automation stuff
while not exitFlag:
username = None
donelist = None
type = 0
while username is None:
sleep(1)
if not q_imme.empty():
usernameAndDonelistTuple = q_imme.get(timeout=1)
username = usernameAndDonelistTuple[0]
type = usernameAndDonelistTuple[1]
donelist = usernameAndDonelistTuple[2]
elif not q.empty():
username = q.get(timeout=1)
print('{}: start grabbing for {} type: {}'.format(name, username, type))
time_element_in_dict = {'timestamp': str(datetime.now())}
if type == 0: # regular data
res, data = get_api_info(session, username)
if res is True:
data.update({'status': 'ok'})
elif res is None:
data = {'status': 204} # No Content
elif res is False:
data = {'status': 503} # exception
# TODO: async update database
data.update(time_element_in_dict)
userDb[username] = data
print('{}: done grabbing for {} with status {}'.format(name, username, data['status']))
if donelist: # means we need to confirm
status = data
donelist.put(status)
except:
# when we reach here the application hangs!!!, I also manually killed all chrome proceess and it hangs.
import time
class user_info(Resource):
def get(self):
username = request.args.get('username')
start_time = time.time()
print('username:{} ~ {}'.format(username, start_time))
return_data = dict()
username = username.lower()
try:
# first try to grab from database
#lock.acquire()
return_data[username] = userDb[username]
#lock.release()
print('username: {}'.format(time.time() - start_time))
except KeyError:
# grab using chrome
doneList = doneListsPool.get()
type = 0
usernameAndDoneListTuple = (username, type, doneList)
workQueue_imme.put(usernameAndDoneListTuple)
#lock.release()
return_data[username] = doneList.get() # wait for a thread to finish and confirm
doneListsPool.put(doneList) # return doneList back to pool
return return_data
# threading
exitFlag = 0
# database
userDb = SqliteDict('./usernames.sqlite', encode=json.dumps, decode=json.loads, autocommit=True)
print("1")
threadList = ['1', '2', '3', '4', '5', '6']
NUMBER_OF_THREADS = len(threadList)
NUMBER_OF_PENDING_QUEUES = NUMBER_OF_THREADS*2
queue#lock = threading.#lock()
workQueue = queue.Queue(2000)
workQueue_imme = queue.Queue(100)
threads = []
threadID = 1
# Create new threads
for tName in threadList:
thread = myThread(threadID, tName, workQueue, workQueue_imme)
thread.start()
threads.append(thread)
threadID += 1
sleep(5)
# create pool of done lists
doneListsPool = queue.Queue()
for i in range(NUMBER_OF_PENDING_QUEUES):
doneListsPool.put(queue.Queue())
app = Flask(__name__)
api = Api(app)
api.add_resource(user_info_list, '/userinfolist') # Route_1
api.add_resource(user_info, '/userinfo') # Route_2
if __name__ == '__main__':
app.run(host='0.0.0.0', port='5007')
- 甚至1个工作者(线程“工作者”函数)都失败了,除了它结束了chrome进程,但在此期间没有处理请求,也请求只需要数据库,我没有看到“print('用户名:{} 〜{}'.format(username,start_time))“行打印,直到chrome重新启动。
- Localhost请求也失败
我像这样启动我的服务器:
uwsgi --socket 0.0.0.0:5077 --protocol=http -w wsgi:app --enable-threads --ignore-sigpipe --ignore-write-errors --disable-write-exception --listen 120 --threads 2
- 我认为这个问题在我添加了donelist以确认线程工作者完成的“user_info”函数时就开始了。
答案
您的完成列表会因工作者中的异常而丢失。加:
if donelist: # means we need to confirm
status = data
donelist.put(status)
到例外。
以上是关于当其中一个线程(chrome)异常时,Uwsgi挂起(10-60秒)的主要内容,如果未能解决你的问题,请参考以下文章
当其中一个线程中存在未捕获的异常时,Python 多线程程序不会退出
切换前置罕见摄像头时,HTML5 视频元素请求永远保持挂起状态(在移动设备上的 chrome 上)