带有 Redis 队列的标签 5.x 生成大量日志

Posted 2023-03-25

技术标签:

【中文标题】带有 Redis 队列的标签 5.x 生成大量日志【英文标题】：Lavel 5.x with Redis Queues generating huge logs 【发布时间】：2019-02-12 14:40:55 【问题描述】：

我有一个使用 Laravel 和 Redis 队列设置的项目。应用程序配置为在收到消息时发送推送通知。收到消息后，立即将其存储在数据库中并生成推送：

dispatch(new CreateMessageNotification($message));

项目正在使用 Redis 队列和服务提供程序。问题是如果我们不执行以下命令，队列调度不起作用：

php artisan queue:work redis --daemon

但是，当我们运行此命令时，应用程序开始在下面生成错误日志。似乎某些例程卡在某个地方，我们无法弄清楚在哪里，为什么以及如何？如果我们不运行此脚本，推送通知将不起作用。如果我们确实运行此脚本，推送通知会延迟工作，并且会生成 ERROR 日志，并且日志大小会在一个小时内达到 GB。我们在这里做错了什么？

我们正在使用 Redis/Predis：

'redis' => [
    'client' => 'predis',
    'cluster' => env('REDIS_CLUSTER', false),
    'default' => [
        'host' => env('REDIS_HOST', 'localhost'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', 6379),
        'database' => env('REDIS_DB', 0),
        'timeout' => 15.0,
    ],
],

对于广播，我们使用 Pusher：

'default' => env('BROADCAST_DRIVER', 'pusher'),

对于队列，我们使用 Redis：

'default' => env('QUEUE_DRIVER', 'sync'),

QUEUE_DRIVER=redis

对于缓存，我们使用 Redis：

'default' => env('CACHE_DRIVER', 'file'),

CACHE_DRIVER=redis

CreateMessageNotification

<?php
namespace App\Jobs\EventHandlers;

use App\Events\Notifications\NewMessageCreatedEvent;
use App\Repositories\UserDevicesRepository;
use Event;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class CreateMessageNotification implements ShouldQueue

    use InteractsWithQueue, Queueable, SerializesModels;

    private $message;

    public function __construct($message)
    
        $this->message = $message;
    

    public function handle()
    
        Event::fire(new NewMessageCreatedEvent($this->message));

NewMessageCreatedEvent

<?php
namespace App\Events\Notifications;

use App\Models\Chat\Message;
use Event;
use Illuminate\Queue\SerializesModels;

class NewMessageCreatedEvent extends Event

    use SerializesModels;

    public function __construct(Message $message)
    
        $message->load(['other', 'chat', 'chat.chatUsers']);

        foreach ($message->chat->chatUsers as $chatUser) 
            if ($chatUser->user_id !== $message->other->id) 
                Event::fire(new NewMessageNotificationEvent($message, $chatUser->user_id));

错误日志

[2018-09-06 06:28:55] local.ERROR: exception 'Illuminate\Broadcasting\BroadcastException' with message 'API call failed since (daily) message quota exceeded
' in /var/www/html/project/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:106
Stack trace:
#0 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastEvent.php(49): Illuminate\Broadcasting\Broadcasters\PusherBroadcaster->broadcast(Array, 'new_$
#1 [internal function]: Illuminate\Broadcasting\BroadcastEvent->handle(Object(Illuminate\Broadcasting\Broadcasters\PusherBroadcaster))
#2 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(30): call_user_func_array(Array, Array)
#3 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\Container\BoundMethod::Illuminate\Container\closure()
#4 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Appli$
#5 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/Container.php(539): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Arr$
#6 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\Container\Container->call(Array)
#7 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(114): Illuminate\Bus\Dispatcher->Illuminate\Bus\closure(Object(Illuminate\Broadcasting\Broa$
#8 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\closure(Object(Illuminate\Broadcast$
#9 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#10 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(43): Illuminate\Bus\Dispatcher->dispatchNow(Object(Illuminate\Broadcasting\BroadcastEv$
#11 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(69): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\RedisJob), Array)
#12 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(317): Illuminate\Queue\Jobs\Job->fire()
#13 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(267): Illuminate\Queue\Worker->process('redis', Object(Illuminate\Queue\Jobs\RedisJob), Object(Il$
#14 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(113): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\RedisJob), 'redis', Object(Ill$
#15 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(102): Illuminate\Queue\Worker->daemon('redis', 'default', Object(Illuminate\Queue\Wo$
#16 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(86): Illuminate\Queue\Console\WorkCommand->runWorker('redis', 'default')
#17 [internal function]: Illuminate\Queue\Console\WorkCommand->fire()
#18 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(30): call_user_func_array(Array, Array)
#19 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\Container\BoundMethod::Illuminate\Container\closure()
#20 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Appl$
#21 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/Container.php(539): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Ar$
#22 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Illuminate\Container\Container->call(Array)
#23 /var/www/html/project/vendor/symfony/console/Command/Command.php(252): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\$
#24 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Console/Command.php(168): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\Ar$
#25 /var/www/html/project/vendor/symfony/console/Application.php(946): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\C$
#26 /var/www/html/project/vendor/symfony/console/Application.php(248): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symf$
#27 /var/www/html/project/vendor/symfony/console/Application.php(148): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfon$
#28 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(122): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\In$
#29 /var/www/html/project/artisan(36): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\Conso$
#30 main

[2018-09-06 06:28:55] local.ERROR: exception 'ErrorException' with message 'Trying to get property of non-object' in /var/www/html/project/app/Events/Notifications/NewMessageCreate$
Stack trace:
#0 /var/www/html/project/app/Events/Notifications/NewMessageCreatedEvent.php(27): Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(8, 'Trying to get p...', '/var/www/h$
#1 /var/www/html/project/app/Jobs/EventHandlers/CreateMessageNotification.php(38): App\Events\Notifications\NewMessageCreatedEvent->__construct(Object(App\Models\Chat\Message))
#2 [internal function]: App\Jobs\EventHandlers\CreateMessageNotification->handle()
#3 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(30): call_user_func_array(Array, Array)
#4 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\Container\BoundMethod::Illuminate\Container\closure()
#5 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Appli$
#6 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/Container.php(539): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Arr$
#7 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\Container\Container->call(Array)
#8 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(114): Illuminate\Bus\Dispatcher->Illuminate\Bus\closure(Object(App\Jobs\EventHandlers\Creat$
#9 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\closure(Object(App\Jobs\EventHandle$
#10 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#11 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(43): Illuminate\Bus\Dispatcher->dispatchNow(Object(App\Jobs\EventHandlers\CreateMessag$
#12 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(69): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\RedisJob), Array)
#13 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(317): Illuminate\Queue\Jobs\Job->fire()
#14 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(267): Illuminate\Queue\Worker->process('redis', Object(Illuminate\Queue\Jobs\RedisJob), Object(Il$
#15 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(113): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\RedisJob), 'redis', Object(Ill$
#16 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(102): Illuminate\Queue\Worker->daemon('redis', 'default', Object(Illuminate\Queue\Wo$
#17 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(86): Illuminate\Queue\Console\WorkCommand->runWorker('redis', 'default')
#18 [internal function]: Illuminate\Queue\Console\WorkCommand->fire()
#19 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(30): call_user_func_array(Array, Array)
#20 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\Container\BoundMethod::Illuminate\Container\closure()
#21 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Appl$
#22 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Container/Container.php(539): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Ar$
#23 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Illuminate\Container\Container->call(Array)
#24 /var/www/html/project/vendor/symfony/console/Command/Command.php(252): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\$
#25 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Console/Command.php(168): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\Ar$
#26 /var/www/html/project/vendor/symfony/console/Application.php(946): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\C$
#27 /var/www/html/project/vendor/symfony/console/Application.php(248): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symf$
#28 /var/www/html/project/vendor/symfony/console/Application.php(148): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfon$
#29 /var/www/html/project/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(122): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\In$
#30 /var/www/html/project/artisan(36): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\Conso$

【问题讨论】：

【参考方案1】：

对我有用的是从队列中删除所有失败的作业（在尝试再试一次之后）：

php artisan queue:restart
php artisan queue:work redis --daemon --tries=1

重新运行队列：

php artisan queue:work redis --daemon

此外，我必须摆脱对 Pusher 的调用，这些调用未能使其正常工作。

【讨论】：

以上是关于带有 Redis 队列的标签 5.x 生成大量日志的主要内容，如果未能解决你的问题，请参考以下文章

PHP(Mysql/Redis)消息队列的介绍及应用场景案例--转载

Redis实现消息队列的4种方案

Filebeat +Redis+ELK处理Nginx日志系统

redis 日志文件

尽管队列看起来是空的，但 Celery Redis 实例已满

Redis实现简单的消息队列