pythonCelery无法运行的问题

Posted 2021-12-25 sysu_lluozh

部署代码后，进入项目目录，执行启动Celery

celery -A tasks.xrayTask  worker -l info

一、报错信息

出现如下报错：

[2021-11-30 10:08:11,922: CRITICAL/MainProcess] Unrecoverable error: SyntaxError('invalid syntax', ('/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/backends/redis.py', 22, 19, 'from . import async, base\\n'))
Traceback (most recent call last):
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/bootsteps.py", line 115, in start
    self.on_start()
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/apps/worker.py", line 139, in on_start
    self.emit_banner()
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/apps/worker.py", line 154, in emit_banner
    ' \\n', self.startup_info(artlines=not use_image))),
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/apps/worker.py", line 217, in startup_info
    results=self.app.backend.as_uri(),
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/app/base.py", line 1196, in backend
    return self._get_backend()
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/app/base.py", line 914, in _get_backend
    self.loader)
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/app/backends.py", line 70, in by_url
    return by_name(backend, loader), url
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/app/backends.py", line 50, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/kombu/utils/imports.py", line 57, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 860, in get_code
  File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/lluozh/Library/Python/3.7/lib/python/site-packages/celery/backends/redis.py", line 22
    from . import async, base
                      ^
SyntaxError: invalid syntax

二、原因分析

这个问题有些奇怪，查看issue找到对应的问题

查看当前python的版本

 python -V

#Python 3.7.3

查看当前celery的版本

pip3 list

# celery  4.2.1

刚好是出现问题的对应版本

三、问题原因

celery中有一个文件名命名为async
在python3.7的更新中，新增两个关键字，其中一个恰好就是async，另一个是await

四、解决问题

celery的作者也在issue中表示，后续版本会将async这个文件改为asynchronous，并且在新版本中解决该问题

问题修复

所以更新celery版本至最新版本

pip3 install --upgrade celery

查看版本

pip3 list
celery       5.2.1

执行启动命令

celery -A tasks.xrayTask  worker -l info

可正常启动

 -------------- [e-mail] v5.2.1 (dawn-chorus)
--- ***** ----- 
-- ******* ---- Darwin-19.6.0-x86_64-i386-64bit 2021-11-30 10:40:48
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x1055fc668
- ** ---------- .> transport:   redis://:**@172.20.117.53:6379/0
- ** ---------- .> results:     redis://:**@172.20.117.53:6379/1
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . xray.build

[2021-11-30 10:40:48,427: INFO/MainProcess] Connected to redis://:**@172.20.117.53:6379/0
[2021-11-30 10:40:48,446: INFO/MainProcess] mingle: searching for neighbors
[2021-11-30 10:40:49,488: INFO/MainProcess] mingle: all alone
[2021-11-30 10:40:49,525: INFO/MainProcess] [e-mail] ready.

启动成功

五、拓展

flask + celery + redis的单文件和工厂模式demo，可参考flask_celery_redis_demo