centos6u3 安装 celery 总结

Posted 2020-08-23 照夜白

耗时大概6小时.

执行 pip install celery 之后, 在 mac 上 celery 可以正常运行, 在 centos 6u3 上报错如下:

Traceback (most recent call last):
  File "/home/work/collie/venv/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/home/work/collie/venv/lib/python2.7/site-packages/celery/__main__.py", line 13, in main
    from celery.bin.celery import main as _main
  File "/home/work/collie/venv/lib/python2.7/site-packages/celery/bin/__init__.py", line 2, in <module>
    from .base import Option
  File "/home/work/collie/venv/lib/python2.7/site-packages/celery/bin/base.py", line 17, in <module>
    from celery import VERSION_BANNER, Celery, maybe_patch_concurrency
  File "/home/work/collie/venv/lib/python2.7/site-packages/celery/local.py", line 509, in __getattr__
    module = __import__(self._object_origins[name], None, None, [name])
  File "/home/work/collie/venv/lib/python2.7/site-packages/celery/app/__init__.py", line 5, in <module>
    from celery import _state
  File "/home/work/collie/venv/lib/python2.7/site-packages/celery/_state.py", line 15, in <module>
    from celery.utils.threads import LocalStack
  File "/home/work/collie/venv/lib/python2.7/site-packages/celery/utils/__init__.py", line 10, in <module>
    from .nodenames import worker_direct, nodename, nodesplit
  File "/home/work/collie/venv/lib/python2.7/site-packages/celery/utils/nodenames.py", line 7, in <module>
    from kombu.entity import Exchange, Queue
  File "/home/work/collie/venv/lib/python2.7/site-packages/kombu/entity.py", line 6, in <module>
    from .abstract import MaybeChannelBound, Object
  File "/home/work/collie/venv/lib/python2.7/site-packages/kombu/abstract.py", line 6, in <module>
    from .connection import maybe_channel
  File "/home/work/collie/venv/lib/python2.7/site-packages/kombu/connection.py", line 15, in <module>
    from kombu import exceptions
  File "/home/work/collie/venv/lib/python2.7/site-packages/kombu/exceptions.py", line 6, in <module>
    from amqp import ChannelError, ConnectionError, ResourceError
  File "/home/work/collie/venv/lib/python2.7/site-packages/amqp/__init__.py", line 47, in <module>
    from .connection import Connection  # noqa
  File "/home/work/collie/venv/lib/python2.7/site-packages/amqp/connection.py", line 29, in <module>
    from . import spec
ImportError: cannot import name spec

遇到这个问题时, 心态不好, 没有深入的分析问题, 而是寻求在生产环境服务器上复现与 mac 一样的环境, 以求问题可以自己解决.

最开始以为是 rabbitMQ 没有安装导致, 转而寻求安装 rabbitMQ, 生产环境 10KB 的下载速度, 浪费了很多时间.

然后怀疑是 virtualenv 的问题, 然后在另一台服务器上采用绝对路径安装了 python 2.7.12, pip 之后发现问题依旧.

进一步怀疑是 pip 下载的文件有损, 开始手动下载相关包文件安装, 问题依旧.

最后回到报错的信息, 模块加载失败, 进入到 amqp 包文件目录, 解释器下加载 spec, 没有报警.

执行 from .connection import Connection, 报错信息如下:

>>> from connection import Connection
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "connection.py", line 26, in <module>
    from vine import ensure_promise
  File "/usr/local/lib/python2.7/site-packages/vine/__init__.py", line 8, in <module>
    from .abstract import Thenable
  File "/usr/local/lib/python2.7/site-packages/vine/abstract.py", line 8, in <module>
    from .five import with_metaclass
  File "/usr/local/lib/python2.7/site-packages/vine/five.py", line 67, in <module>
    import platform
  File "platform.py", line 16, in <module>
    int, platform.release().split(‘-‘)[0].split(‘.‘)))
AttributeError: ‘module‘ object has no attribute ‘release‘

报错 platform 没有 release 方法, 这个是不对的, platform 是有这个方法的.

发现原因, amqp 2.1.2 版本之后, 增加了 platform.py 的同名文件

# platform.py
"""Platform compatibility."""
from __future__ import absolute_import, unicode_literals

import sys
import platform

# Jython does not have this attribute
try:
    from socket import SOL_TCP
except ImportError:  # pragma: no cover

看起来, 当 vine 包中 five.py 执行到 platform.release 时, 解释器混淆了两个 platform, 导致报错.

不过问题到这里并不能结束, 虽然问题可以在 centos 6, 非 /usr/bin/ 路径安装的 Python 2.7.11 下复现, 为什么 mac 下就没有这个问题.

mac 下的默认 Python 也不是系统自带版本, brew install 与手动安装原理没有区别, 考虑到 amqp 2.1.2 发布时间已经一段时间, 并且在 celery 的 github 上也没有 issue 报出, 还需要想一下, 为什么偏偏在这些机器上出现了这个问题.

顺便, 在寻找 amqp 的历史版本时, 发现无从下手, pypi 页面只提供最新版, 最后在手动安装的过程中, 发现原来 pypi 上是有存储的.

地址 https://pypi.python.org/simple/amqp/

最后, 直面问题的态度, virtualenv 的原理是否完全理解了, import 相对路径绝对路径是否清楚, 异常信息是否认真看过, 都是些很基础的事情, 这个需要加强. 项目上线之后需要再总结一下.