RQ: redis.exceptions.ResponseError: Command # 3 ... of pipeline 导致错误：OOM command not allowed when us

Posted 2023-03-04

【中文标题】RQ: redis.exceptions.ResponseError: Command # 3 ... of pipeline 导致错误：OOM command not allowed when used memory > \'maxmemory\'

【英文标题】：RQ: redis.exceptions.ResponseError: Command # 3 ... of pipeline caused error: OOM command not allowed when used memory > 'maxmemory'RQ: redis.exceptions.ResponseError: Command # 3 ... of pipeline 导致错误：OOM command not allowed when used memory > 'maxmemory'

【发布时间】：2017-11-29 16:25:47

【问题描述】：

我正在使用 RQ 在我的 django 应用程序中运行后台任务，因为它应该是完成工作的最简单方法之一。该任务包括检查一些 API 是否有任何信息已更新，并在我自己的数据库中插入任何新信息。

直到几天前，它还可以正常工作，但我现在遇到了一个我无法解决的错误。

从错误信息的最后几行开始：

OOM command not allowed when used memory > 'maxmemory'

一开始我认为我向工作人员传递了太多数据。但是，我最终减少了传递给具有 5 个键值对的单个字典的数据，但我仍然收到错误消息（请参阅底部的完整消息）。然而，直到上周，我才通​​过了 20 多个字典，每个字典都包含更多元素，而且运行良好。

我检查了here和here，但似乎和我的问题不一样。

知道为什么我会收到此错误以及如何解决它吗？

---

错误信息：

Internal Server Error: /en/results/
Traceback (most recent call last):
  File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2518, in _execute_transaction
    response = self.parse_response(connection, '_')
  File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2584, in parse_response
    self, connection, command_name, **options)
  File "/my/path/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
    response = connection.read_response()
  File "/my/path/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
    raise response
redis.exceptions.ExecAbortError: Transaction discarded because of previous errors.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/my/path/lib/python3.5/site-packages/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/my/path/lib/python3.5/site-packages/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/my/path/app/results/views.py", line 259, in results_list
    price_update_to_queue(outbound_trips)
  File "/my/path/app/results/views.py", line 331, in price_update_to_queue
    queue.enqueue(update_prices, queries)
  File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 274, in enqueue
    job_id=job_id, at_front=at_front, meta=meta)
  File "/my/path/lib/python3.5/site-packages/django_rq/queues.py", line 60, in enqueue_call
    return self.original_enqueue_call(*args, **kwargs)
  File "/my/path/lib/python3.5/site-packages/django_rq/queues.py", line 56, in original_enqueue_call
    return super(DjangoRQ, self).enqueue_call(*args, **kwargs)
  File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 227, in enqueue_call
    job = self.enqueue_job(job, at_front=at_front)
  File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 298, in enqueue_job
    pipe.execute()
  File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2626, in execute
    return execute(conn, stack, raise_on_error)
  File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2523, in _execute_transaction
    raise errors[0][1]
  File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2510, in _execute_transaction
    self.parse_response(connection, '_')
  File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2584, in parse_response
    self, connection, command_name, **options)
  File "/my/path/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
    response = connection.read_response()
  File "/my/path/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
    raise response
redis.exceptions.ResponseError: Command # 3 (HMSET b'rq:job:8df23896-d52d-4585-aa60-9f5f9a39292a' created_at 2017-06-26T13:13:33Z timeout 500 origin default data b'\x80\x04\x95\xff\x00\x00\x00\x00\x00\x00\x00(\x8c\x1bresults.tasks.update_prices\x94N]\x94\x94(\x8c\x03arr\x94\x8c\x0bDE-CGN-00-0\x94\x8c\x03dep\x94\x8c\x0bDE-BER-02-0\x94\x8c\x03rtn\x94K\x00\x8c\x06seller\x94\x8c\x04KIWI\x94\x8c\x06dep_dt\x94\x8c\x08datetime\x94\x8c\x08datetime\x94\x93\x94C\n\x07\xe1\x07\x18\x00\x00\x00\x00\x00\x00\x94\x8c\x0bpsycopg2.tz\x94\x8c\x13FixedOffsetTimezone\x94\x93\x94KxN\x86\x94R\x94\x94\x8c\x07_offset\x94\x8c\x08datetime\x94\x8c\ttimedelta\x94\x93\x94K\x00M \x1cK\x00\x87\x94R\x94sb\x86\x94R\x94ua\x85\x94\x94t\x94.' description results.tasks.update_prices(['arr': 'DE-CGN-00-0', 'dep': 'DE-BER-02-0', 'rtn': 0, 'seller': 'KIWI', 'dep_dt': datetime.datetime(2017, 7, 24, 0, 0, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=120, name=None))]) enqueued_at 2017-06-26T13:13:33Z status queued) of pipeline caused error: OOM command not allowed when used memory > 'maxmemory'.
[26/Jun/2017 13:13:33] "GET /en/results/?From=Berlin&To=Cologne&OutboundDate=24-07-2017&ReturnDate= HTTP/1.1" 500 174583

这就是我传递任务的方式

#views.py
import django_rq
from .tasks import update_prices

...
queue = django_rq.get_queue('default')
queue.enqueue(update_prices, queries)

和

#tasks.py
from django_rq import job

@job
def update_prices(queries):

    ...
    return queries_to_background(queries)

【参考方案1】：

之前遇到过类似的问题。我的 Flask 应用程序部署到 Heroku 上，我在 Heroku 上使用了 redistogo，免费计划的内存非常少，这导致了超出 maxmemory 错误。然后我切换到heroku-redis，问题就解决了。

【讨论】：

我最终在没有 Redis 的情况下使用了一种完全不同的方法，但我也在使用 Redis To Go，所以切换到 heroku-redis 可能已经解决了这个问题。